Hi,
I have a requirement to load 25M records in a single collection and I did that. When I access through my REST / Phalcon API Call, the server getting shutdown and it through the following error message from the cursor.
"Failed to connect to: localhost:27017: Remote server has closed the connection 500 Internal Server Error - MongoConnectionException"
Can you please suggest me, how to do I access large sets ? How do I split the database if incase.
Thanks, Raja K
25M is a large set, if you try to read all of the rows intoa json response, your server will run out of memory. Create a pagination logic and only serve a few hundred records per request.
DB partitioning is also useful, but I dont know if mongo supports it
Hello there...
Not familiar with MongoDB but I found a very useful and accurate article called SSIS MongoDB Integration Where all points are described clearly in deep manner.
That's really not going to work. 25 million of something seems like a batch processing you want to have, and not live API response.
Client -> API -> MongoDB on that pipeline all three must be able to process that many data.
If you really need to do that - use some form of buffering. So pipeline will process only chunks of data and dispatch it directly to the client (persistent connection).
