Queue task for every user in nodejs - javascript

The requirement is the user can take many actions at the same time (call API), but there some actions need to run sequentially for each user.
etc: user hit button add item, and clear cart the same time, but on the backend we need do it's sequentially, done every phase in action add item before start any new phase of action clear cart.
A phase can be called rest API to another service, query database or execute some javascripts code.
I am thinking about creating a queue job for every user's to archive this. Is that a good idea? What is the best way to do it, and are there any libraries for this (I have already searched but couldn't find anything)?
P/s: I'm using expressJS and MongoDB (> 4.0).
Edited:
If using a queue, how can i detect that user already has a queue or not to create a new one? If using Redis we can meet the race condition, or variable in ram can't help when I run nodejs as multi-cluster.

bluebird will solve this problem
code example:
action1()
.then(phase1)
.then(phase2)
.then(phase3);
action2()
.then(phase1)
.then(phase2);

Related

Detecting changes on database table column status

I am having a project in Laravel. In database I have a status column, which shows if exam is started or not. I had an idea in the waiting room checking every single second if the status was changed or not, if changed to 1, when the exam starts, but I am so new to Laravel and everything else, that I even don't get the main idea how I could do this, I don't ask for any code, just for the lead, to move on. yeah, hope someones gets me. Thanks if someone answers me.
Check about laravel cron jobs. You will need a class implementing ShouldQueue interface and using Dispatchable, InteractsWithQueue, Queueable, SerializesModels;
With regards to the storage of the jobs i do recommend Redis or SQS.
In order to keep monitoring the queue in production think about installing supervisor.
Further information here: Queues
Your plan can work, it is called polling.
Basically, you will want to call
setInterval(function() {
//your code here
}, 1000);
setInterval is a function that receives two parameter. The first is a callback function, that will periodically be executed and the second is the length of the period in milliseconds (1000 milliseconds is a second).
Now, you will need to implement your callback function (Javascript, of course) to send an AJAX request to a Laravel action. You will need to look into XMLHttpRequest and its usages, or you can use some libraries to simplify your task, like jQuery or Axios.
On Laravel's side you will need to implement an action and a Route for it. (read this: https://appdividend.com/2022/01/22/laravel-ajax/)
Your Laravel will need to load data from your database, you can use Eloquent for this purpose or raw queries and then respond the POST request with the result.
Now, in your Javascript at the AJAX request's code you will need to have a callback function (yes, a callback inside a callback) which will handle the response and the the changes.
What about leveraging Observers? Also instead of having a status boolean, you could take a similar approach that Laravel has done for soft deletes and set exam_started_at. This way you can also keep track of time stamp and state all in one column. Also, observers are immediate rather than pushing them into a queue. Then generate a websocket event that can report back to your front end, if needed.
check out Laravel observer and soft delete documentation.
I know you specified "when the column on db changes..." but if it's not a strict-requirement you might want to consider implementing event-based architecture. Laravel has support for model events, which essentially allows you to run certain assertions and controls when a model created, updated, deleted etc.
class Exam extends Model
protected static function booted()
{
static::updated(function ($exam) {
if($exam->status=='your-desired-status'){
//your actions
}
//you can even in cooperate change controls
if ($exam->isDirty('status')){
//means status column changed
});
}
}
Of course this solution applies only if Database in question is in Laravel's reach. If database data changes outside the Laravel application these event listeners won't help at all.

Meteor server restarts itself upon slow requests

I have a Meteor app that is performing some calls that are currently hanging. I'm processing a lot of items in a loop that is then upserting to server-side Mongo. (I think this is done asynchronously) I understand the upserting in a loop is not good .
This whole functionality seems to make the app hang for a while. I'm even noticing sock.js and websocket error out in the console. I think this is all due to DDP, async Mongo upserts, and the slow requests.
Here's some pseduocode to what I'm talking about
for (1..A Lot of records) {
//Is this async?
Collection.upsert(record)
}
Eventually this function will complete. However, I'll notice that Meteor "Restarts" (I think this is true because I see Accounts.onLogin being called again. It's almost like the client refreshes after the slow request has actually finished. This results in something that appears like an infinite loop.
My question is why the app is "restarting". Is this due to something in the framework and how it handles slow requests? I.e. does it queue up all bad requests and then eventually retry them automatically?
I am not sure about what exactly is going on here, but it sounds like the client isn't able to reach the server while it is "busy", and then the client connection over DDP times out, and ends up with a client refresh. The server process probably doesn't restart.
One technique for improving this is to implement a queue in your database. One piece of code detects there are a bunch of database upserts to do, so it records the information in a table which is used as a queue.
You set up a cron job (using eg npm module node-cron) that looks for things in the queue on a regular basis - when it finds an unprocessed record, it does the upsert work needed, and then either updates a status value in the queue record to 'done', or simply deletes it from the queue. You can decide how many records to process at a time to minimise interruptions.
Another approach is to do the processing in another node process on your server, basically like a worker process. If this process is busy, it is not going to impact your front end. The same queueing technique can be used to make sure this doesn't get bogged down either.
You lose a little reactivity this way, but given it's some kind of bulk process, that shouldn't matter.

How can I check if there is an instance running for a certain block of code in Cloud Functions for Firebase?

I would like to know if it is possible to detect that a thread is already running a Cloud Functions, and if possible to also detect if it is running on a particular ID's data. I think I could have a variable stored in firebase memory of the ID in Firebase Database that the function is being run on from the Database, and to remove the variable when the function is done running,but the concern is of two writes to the database happening subsequently and very rapidly, causing the initial thread to not be able to write to memory fast enough before the second thread checks if the variable is there, especially on a cold start from the firebase thread - which in my understanding is a variable amount of time in which either thread could potentially spin up first.
My use case is this:
Let's say a write to the realtime database happens from the client side that causes a trigger for Cloud Functions to run a handler. This handlers job is to loop through and do work with the snapshot of records that was just written to by the client, and using a loop will parse each record in the snapshot, and when it is done, delete them. The handler works great until another record is written to the same group of records in the database before the handler's job is done, which causes a second handler thread to spin up, and start moving through the records in the same group of records, which would cause records to be iterated over twice, and possibly the data to be handled twice.
I have other solutions for my particular case which I can use instead, but it involves just allowing each record to trigger a separate thread like normal.
Thanks in advance!
There is no way to track running instances "in-memory" for Cloud Functions, as each function invocation may be running in entirely different virtual infra. Instead, what you'd most likely want to do here is have some kind of lock persisted in e.g. the Firebase Realtime Database using a transaction. So you'd do something like:
When the function invocation starts, generate a random "worker ID".
Run a transaction to check a DB path derived from the file you're processing. If it's empty, or populated with a timestamp that is older than a function timeout, write your worker ID and the current timestamp to the location. If it's not empty or the timestamp is fresh, exit your function immediately because there's already an active worker.
Do your file processing.
Run another transaction that deletes the lock from the DB if the worker ID in the DB still matches your worker ID.
This will prevent two functions from processing the same file at the same time. It will mean, however, that any functions that execute while a path is locked will be discarded (which may or may not be what you want).

RxJS share memory across node.js instances using Redis

We're working on a project where we are creating an event processor using RxJS. We have a few 'rules', so to speak, where input is provided from a few different source and output has to be generated based on the number of times an input is above a set value (simple rule).
Now, all this works without any problems, but we want to move the project from beta to production. This means running multiple instances of Node.JS with RxJS on top of it.
We're wondering if it's possible for RxJS to share its memory using Redis for example. This way when one of the instances dies for whatever reason, another one can pick up where the dead one stopped. Ensuring that the amount of times the value was above the set value is retained.
This would also allow us to spread the load over multiple instances if the 'rules' get more complex and the amount of data increases.
Is something like this possible with RxJS, or should we build our own administration around it?
You can't share memory between node.js processes, as far as I know. Doing so would be super-unsafe, since then you're dealing with concurrency problems that can't be mitigated with javascript (what happens when one process interrupts another?)
That said, you can pass messages back and forth with redis. Generally, what I do is establish a work queue as a redis queue. Some servers push into the work queue. Some workers will pull from the queue and process data and deal with the results as needed.
A concrete example is generating outbound emails in response to some REST event (new post or whatever). The webapp does a LPUSH to a known queue. The worker process can use BRPOPLPUSH to atomically pull an entry from the work queue and push it into an "in process" queue. Once the mail is sent, it can be removed from the in process queue. If there's a server crash or long timeout, entries in the in process queue can be pushed back into the work queue and re-tried.
But you're not going to get a fancy shared-memory solution here, I don't think at least.

Call NodeJs function at a specific moment in time

Is there a way to automate functions and call them at a specific mooment in time with Node.Js ? More precisely, I would like to create a reminder with node js. When I receive the date from the user, I have to send notification to remind him/her of something.
I know setTimeout() function but is it really good idea when I have large database of users ?
Thank you.
You can use setTimeout() and keep all the work inside your server, but for a large number of users, you would not necessarily want to set a timeout for every single user. All you really need is a single timeout for the next user that needs to be notified. When that timer fires, you then set a timer for the next user that needs to be notified and so on.
This can be done with an array of objects that you sort by the notification time. Each time you add something to the array, you cancel your current timer, add the new notification to the array, sort the array and set a timer for the earliest notification.
When a timer fires, you remove that item from the array and schedule the next one.
And, since you likely want this to survive a server restart, you save the notification array (probably in JSON format) to a file or to a database each time you modify it so you can reload that data upon a server restart.
FYI, there are scheduling modules for node.js that already offer this type of functionality if you'd prefer to pick up code someone else has already written.
Examples of some of these modules:
node-cron
agenda
node-schedule
What you are looking for is called a Scheduler or a cron job (its origin is from Linux OS).
In NodeJS you can find the node-schedule that implement the same idea.

Categories