Meteor server restarts itself upon slow requests - javascript

I have a Meteor app that is performing some calls that are currently hanging. I'm processing a lot of items in a loop that is then upserting to server-side Mongo. (I think this is done asynchronously) I understand the upserting in a loop is not good .
This whole functionality seems to make the app hang for a while. I'm even noticing sock.js and websocket error out in the console. I think this is all due to DDP, async Mongo upserts, and the slow requests.
Here's some pseduocode to what I'm talking about
for (1..A Lot of records) {
//Is this async?
Collection.upsert(record)
}
Eventually this function will complete. However, I'll notice that Meteor "Restarts" (I think this is true because I see Accounts.onLogin being called again. It's almost like the client refreshes after the slow request has actually finished. This results in something that appears like an infinite loop.
My question is why the app is "restarting". Is this due to something in the framework and how it handles slow requests? I.e. does it queue up all bad requests and then eventually retry them automatically?

I am not sure about what exactly is going on here, but it sounds like the client isn't able to reach the server while it is "busy", and then the client connection over DDP times out, and ends up with a client refresh. The server process probably doesn't restart.
One technique for improving this is to implement a queue in your database. One piece of code detects there are a bunch of database upserts to do, so it records the information in a table which is used as a queue.
You set up a cron job (using eg npm module node-cron) that looks for things in the queue on a regular basis - when it finds an unprocessed record, it does the upsert work needed, and then either updates a status value in the queue record to 'done', or simply deletes it from the queue. You can decide how many records to process at a time to minimise interruptions.
Another approach is to do the processing in another node process on your server, basically like a worker process. If this process is busy, it is not going to impact your front end. The same queueing technique can be used to make sure this doesn't get bogged down either.
You lose a little reactivity this way, but given it's some kind of bulk process, that shouldn't matter.

Related

Python-SocketIO server takes a long time to fire event

I have a python websocket server attempting to communicate with a javascript websocket client (embedded in HTML). The events are being emited from the server immediately, but it takes upwards of 30 seconds for the server to send the event trigger, despite both the client and server being locally hosted.
Here is the relavent code for the server:
sio = socketio.AsyncServer(cors_allowed_origins='*')
app = web.Application() #aiohttp web server
loop = asyncio.get_event_loop()
sio.attach(app)
async def index(request):
with open('./index.html') as f:
return web.Response(text=f.read(), content_type='text/html')
app.router.add_get('/', index)
app.router.add_get('/index.html', index)
if __name__ == '__main__':
web.run_app(app)
the event is being fired like so (edit, this must be done with event loops, as emit is an asynchronous function being run from a synchronous one.):
print('Starting event')
loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop)
loop.run_until_complete(sio.emit('ChangeProgressState'))
loop.close()
print('Event has been fired.')
However, the print statements show up immediately. On the client end, I am connecting and trying to consume the event like this:
const socket = io.connect("http://localhost:8080", {
transports: ['websocket']
})
socket.on("ChangeProgressState", function (data) {
console.log("got event.")
//some code here...
});
However, from the time it takes for the event to fire, and the time it takes for the javascript socket to notice can be a very long time, from 30 seconds to sometimes a few minutes. Is there something I'm doing wrong here?
It should be noted, there are very little (2%-5%) resources being consumed (both memory and CPU), so I do not currently think that is the issue. Any help would be much appreciated.
EDIT 11/15/2019: I have tried looking at the networking tab of the application (chromium-browser on raspberry pi). It seems to show the initial socket connection, but it doesn't show anything in terms of communication between sockets, even after the event eventually fires.
EDIT 2: This definitely seems to be an issue server-side. I can send events from the JS client to the python server essentially immediately, but going in the other direction is when it takes a long time to arrive. I'm not quite sure why though.
Ah ok, so my gut said it sounds like the client is long polling. Many socket libraries first establish long-polling and then upgrade to ws connections.
After taking a look at Socket.io:
... which first establishes a long-polling connection, then tries to upgrade to better transports that are “tested” on the side, like WebSocket. ...
So I don't believe you're doing anything wrong, it's just the initialization process of establishing the WebSocket connection.
As for the python part, I'll be honest that's a tad more fuzzy to me. My first guess is that the loop code doesn't block the print statement from being executed -- but I'm more familiar with JavaScript than Python, so not completely certain on that front. My second guess is that I do know from other pub/sub libraries that the server side engine sometimes makes use of a middle layer of sorts (sometimes a cache, sometimes a queue) that helps ensure messages are sent/received, that's also a possibility.
extra tidbit: I suspect if you look at the network tab of your browser's dev tools, it'd display that behavior, some form of HTTP requests, and then eventually you'd see the socket connection. Playing around with turning your Python server/service off/on would also demonstrate the robustness of socket.io in the browser and edge cases for how it handles unstable networking when communicating with respect to various internet communication protocols.
Thank you to everyone that helped answering this question! I finally found a solution that is a bit unorthodox, so I'll explain the whole situation here.
Essentially, in order to run an async method in a synchronous context, you must use the asyncio's run_until_complete method on an event loop. This is how I was doing it when this question was asked. However, after talking to the creator of the python-socketio library, it seems that you must run this in the same event loop as the one the server is running in.
However, this creates a different problem. If an event loop is already running, python does not allow you to use run_until_complete on it, giving you an error: RuntimeError: This event loop is already running.
So, this things sound contradictory right? And you would be correct. However, this problem is prevalent enough that another library exists for the sole purpose of monkey-patching the python asyncio library to fix this problem. I found this library here .
After installing and utilizing that library, I can now do this, which fixes my problem completely:
main_event_loop = asyncio.get_event_loop()
main_event_loop.run_until_complete(sio.emit("ChangeProgressState"))
Now the program runs as expected, and the messages are being sent/arriving immediately.

RxJS share memory across node.js instances using Redis

We're working on a project where we are creating an event processor using RxJS. We have a few 'rules', so to speak, where input is provided from a few different source and output has to be generated based on the number of times an input is above a set value (simple rule).
Now, all this works without any problems, but we want to move the project from beta to production. This means running multiple instances of Node.JS with RxJS on top of it.
We're wondering if it's possible for RxJS to share its memory using Redis for example. This way when one of the instances dies for whatever reason, another one can pick up where the dead one stopped. Ensuring that the amount of times the value was above the set value is retained.
This would also allow us to spread the load over multiple instances if the 'rules' get more complex and the amount of data increases.
Is something like this possible with RxJS, or should we build our own administration around it?
You can't share memory between node.js processes, as far as I know. Doing so would be super-unsafe, since then you're dealing with concurrency problems that can't be mitigated with javascript (what happens when one process interrupts another?)
That said, you can pass messages back and forth with redis. Generally, what I do is establish a work queue as a redis queue. Some servers push into the work queue. Some workers will pull from the queue and process data and deal with the results as needed.
A concrete example is generating outbound emails in response to some REST event (new post or whatever). The webapp does a LPUSH to a known queue. The worker process can use BRPOPLPUSH to atomically pull an entry from the work queue and push it into an "in process" queue. Once the mail is sent, it can be removed from the in process queue. If there's a server crash or long timeout, entries in the in process queue can be pushed back into the work queue and re-tried.
But you're not going to get a fancy shared-memory solution here, I don't think at least.

Save to 3 firebase locations with a slow internet connection

Sometimes I'm having issues with firebase when the user is on a slow mobile connection. When the user saves an entry to firebase I actually have to write to 3 different locations. Sometimes, the first one works, but if the connection is slow the 2nd and 3rd may fail.
This leaves me with entries in the first location that I constantly need to clean up.
Is there a way to help prevent this from happening?
var newTikiID = ref.child("tikis").push(tiki, function(error){
if(!error){
console.log("new tiki created")
var tikiID = newTikiID.key()
saveToUser(tikiID)
saveToGeoFire(tikiID, tiki.tikiAddress)
} else {
console.log("an error occurred during tiki save")
}
});
There is no Firebase method to write to multiple paths at once. Some future tools planned by the team (e.g. Triggers) may resolve this in the future.
This topic has been explored before and the firebase-multi-write README contains a lot of discussion on the topic. The repo also has a partial solution to client-only atomic writes. However, there is no perfect solution without a server process.
It's important to evaluate your use case and see if this really matters. If the second and third writes failed to write to a geo query, chances are, there's really no consequence. Most likely, it's essentially the same as if the first write had failed, or if all writes had failed; it won't appear in searches by geo location. Thus, the complexity of resolving this issue is probably a time sink.
Of course, it does cost a few bytes of storage. If we're working with millions of records, that may matter. A simple solution for this scenario would be to run and audit report that detects broken links between the data and geofire tables and cleans up old data.
If an atomic operation is really necessary, such as gaming mechanics where fairness or cheating could be an issue, or where integrity is lost by having partial results, there are a couple options:
1) Master Record approach
Pick a master path (the one that must exist) and use security rules to ensure other records cannot be written, unless the master path exists.
".write": "root.child('maste_path').child(newData.child('master_record_id')).exists()"
2) Server-side script approach
Instead of writing the paths separately, use a queue strategy.
Create an single event by writing a single event to a queue
Have a server-side process monitor the queue and process events
The server-side process does the multiple writes and ensures they
all succeed
If any fail, the server-side process handles
rollbacks or retries
By using the server-side queue, you remove the risk of a client going offline between writes. The server can safely survive restarts and retry events or failures when using the queue model.
I have had the same problem and I ended up choosing to use condition Conditional Request with the Firebase REST API in order to write data transactionally. See my question and answer. Firebase: How to update multiple nodes transactionally? Swift 3 .
If you need to write concurrently (but not transactionally) to several paths, you can do that now as Firebase supports multi-path updates. https://firebase.google.com/docs/database/rest/save-data
https://firebase.googleblog.com/2015/09/introducing-multi-location-updates-and_86.html

NodeJS: Why pm2 list shows memory keeps on increasing?

I am using HAPI.JS framework with NodeJS and created a proxy. Think that proxy means i am just maintaining session in redis. Other than that i am not doing anything in the code. May be only thing is i am using setInterval to log my process.memoryUsage() for every 3 mintues.
My Questions:
Why my Memory Keeps on Increasing?
Will it get down?
Is this occurs due to setInterval keeps on logging the process usage?
Is this occurs due to console logging of every request and response?
My Redis Database is kept open till my server crashes, it this causes this ?
Do i need use process mananger like new relic or strong loop to identify this?
So how long this memory will keep on increasing, at some point it must stop (i want to know which point is that?)
I am using sequelize of MSSQL transaction using pooling concept? Does pooling makes this?
P.S I am new to node JS.
Why my Memory Keeps on Increasing?
You got a memory leak
Will it get down?
Sometimes GC kicks in and cleans up some things (that are not leaking)
Is this occurs due to setInterval keeps on logging the process usage?
Usually not, but w/o seeing the code I can't say this for sure
Is this occurs due to console logging of every request and response?
Usually not, but w/o seeing the code I can't say this for sure
My Redis Database is kept open till my server crashes, it this causes this ?
Should not be a problem.
Do i need use process mananger like new relic or strongloop to identify this?
It is one way to do it ... but there are also others.
So how long this memory will keep on increasing, at some point it must stop (i want to know which point is that?)
Depends on the server setup. How much RAM + what else is running etc.
I am using sequelize of MSSQL transaction using pooling concept? Does pooling makes this?
Usually not, but w/o seeing the code I can't say this for sure
Maybe this post helps you find the leak:
https://www.nearform.com/blog/how-to-self-detect-a-memory-leak-in-node/

How to prevent HTML5 Web Workers from locking up thus correctly responding to messages from parent

I'm using web workers to do some CPU intensive work but have the requirement that the worker will respond to messages from the parent script while the worker is still processing.
The worker however will not respond to messages while it is locked in a processing loop, and I have not found a way to say poll the message queue. Thus it seems like the only solution is to break processing at an interval to allow any messages in the queue to be serviced.
The obvious options are to use a timer (say with setInterval) however I have read that the minimum delay between firings is quite long (http://ajaxian.com/archives/settimeout-delay) which is unfortunate as it will slow down processing alot.
What are other peoples thoughts on this? I'm going to try have the worker dispatch onmessage to itself at the end of each onmessage, thus effectively implementing one step of the processing loop per event received from itself, but just wanted to see if anyone had any ideas about this.
Thanks,
A worker can spawn sub workers. You can have your main worker act as your message queue, and when it receives a request for a long running operation, spawn a sub worker to process that data. The sub worker can then send the results back to the main worker to remove the event from the queue and return the results to the main thread. That way your main worker will always be free to listen for new messages and you have complete control over the queue.
--Nick
I ran into this issue myself when playing with workers for the first time. I also debated using setInterval, but I felt that this would be a rather hacky approach to the problem (and I had already went this way for my emulated multithreading). Instead, I settled on terminating the workers from the main thread (worker.terminate()) and recreating them if the task that they are involved in needs to be interrupted. Garbage collection etc seemed to be handled in my testing.
If there is data from these tasks that you want to save, you can always post it back to the main thread for storage at regular intervals, and if there is some logic you wish to implement regarding whether they are terminated or not, you can post the relevant data back at regular enough intervals to allow it.
Spawning subworkers would lead to the same set of issues anyway; you'd still have to terminate the subworkers (or create new ones) according to some logic, and I'm not sure it's as well supported (on chrome for example).
James
Having the same problem I searched the web workers draft and found something in the Processing model section, steps from 9 to 12. As far as I have understood, a worker that starts processing a task will not process another one until the first is completed. So, if you don't care about stopping and resuming a task, nciagra's answer should give better performances than rescheduling each iteration of the task.
Still investigating, though.

Categories