I have a use case where I have to create a record in the database for users repeatedly on a scheduled basis. Let's say every Monday Weekly/BiWeekly. There are two ways with which i can achieve it.
Using Database Triggers to Create a record on the time. But I don't know how to repeat it. I have to create a trigger for the next schedule when this trigger runs, which i don't think is right approach.
Using Queues to handle the scheduling and executing the repeated jobs. But adding a job for each user is not a good idea I guess. I might be wrong but there is no other way to achieve my goal.
I am confused on what to choose between the two. Let's say i have to do this for 1 million users every week Monday at 9.00 a.m.
Which approach will scale?
I am using nodejs as my backend and using Bull-Queue for the queue and postgres as my Database.
Using Database Triggers to Create a record on the time. But I don't
know how to repeat it. I have to create a trigger for the next
schedule when this trigger runs, which i don't think is right
approach.
- Not a right approach based on the so many factors like - memory, number of requests and code quality.
So I went with the second approach:
Using Queues to handle the scheduling and executing the repeated jobs.
But adding a job for each user is not a good idea I guess. I might be
wrong but there is no other way to achieve my goal.
Related
I'm developing an email reminder system to ping users 24, 8, 3, and 1 hour before a task is due.
I have a server that runs on Node.js. My first idea was to set four separate setTimeout() each time a user is assigned a task. However, I assume that having hundreds of setTimeout() idling on a server wouldn't be best performance-wise.
As such, would I be better off polling for incomplete tasks every five minutes and sending reminders to users who have tasks with approaching deadlines? The downside here is that I would be reading an entire MongoDB collection every five minutes.
Nodejs is very efficient with lots and lots of timers. You can easily have tens of thousands of timers with no meaningful ramifications. It uses a sorted, linked list that takes only a tiny amount of time to insert a new timer and costs nothing once inserted.
Only the next timer to fire at the start of the list is regularly compared in the event loop. When it fires, it's removed from the front of the linked list and the next timer in the list is now at the head. Because it's a linked list, the time to fire a timer and remove it from the start of the linked list is independent of how long the list is (e.g. it's not an array that has to be copied down).
So, for your specific application, it is far, far more important to be efficient with your database (as few requests as possible) than it is to minimize the number of timers. So, whichever timer design/implementation optimizes your database load is what I would recommend.
FYI, if you want to remind a user 4 times about an approaching due task, you can still only have one timer live at a time per task. Set the first timer to fire, then when it fires to notify, you do a little time calculation on the due date/time that you previously saved and see when to set the next timer for. That would leave you with just one timer per task rather than four.
But, the main point here is still that you should first optimize the design for efficient use of the database.
And, timers are not persistent so if your server restarts, you need a mechanism for recreating the appropriate timers upon server startup (probably a database query that provides you any tasks pending within a certain time).
I am solo developer and freelancer , I got simple local e-commerce project in my city .
I have developed all the code in PHP MySQL and with other front end technology . But I thought what happens if there is only one item left and two or more person order that same item at the same time . I have used algorithm if item is out of stock than order will be panding or ask customer to cancel project so which customer will have to cancel his project or which customer should get place order .
I know that question seems silly but , think about other project in other scenario when if I make random chatting web app than how state will affect . If a person is online and not chatting with anyone how to match accounts .
I have tried something for this but it did not worked . I have 3 computers 2 PC and one mac I try to make request at the same time but due to speed of internet only one could place order and other get message of out of order after 2 minutes. So what happens if speed was same .
Do I need to add any kind of algorithm that will place order of person and at that time other person hold for that order . I was googling but found nothing . And as I said before I am solo developer so there is no one to ask what to do or MySql will handle it .
Oh , if you are thinking to make this kind of project with WordPress or framework like laravel . I also agree with you but client said to use only plain programming language not even library .
Thank You For Reading .
What you are describing is a race condition type of situation and the way you handle it depends on the business logic. For instance, with the out-of-stock product, first you need to decide if you want to automatically back-order the product or if you want to abort the purchase.
If you want to abort the purchase you need to use some type of database locking mechanism. Locking can block access to a database table or record momentarily. The idea is that you lock the stock information at the beginning of the fulfillment process and unlocks it only after the purchase has been processed.
When a database request encounters a lock, it will wait until the lock is released to continue the operation (or until a timeout happens). So the request that gets the lock first, is the first one to have the chance to finish the purchase.
But locking alone might not be enough, to prevent inconsistencies you want to perform all database operations inside a database transaction. A transaction can ensure that all operations needed to process a purchase happen successfully, otherwise everything is rolled back. For instance, using transactions could prevent you from updating the stock quantity without actually finishing the purchase.
The goal is that when two requests come at almost the same time (since nothing happens at the same time), one of them will get a lock first, then finish and commit the purchase transaction. When the second request finally gets a chance to process is, the stock will be zero and the transaction will fail and a roll-back occurs (given that you did the proper checks).
The specific queries and checks you need vary from application to application, but the same concepts apply. In your case we don't have any specifics to guide you further, so I suggest you check the links provided to gather more information.
Good luck!
Have a counter to track remaining item count in the database. When placing an order apply a WRITE LOCK to the table before updating the available items count. This was two requests can't update the table at once.
https://www.mysqltutorial.org/mysql-table-locking/
In my page, I have some logic that searches through the large array of items by given term.
These items are the large list of data on the client side.
For example, I search 'user' and it filters out 'user.edit', 'user.save' and so on.
When the list gets very large, it blocks the search input.
When I write 'a' it starts searching, and if I type something, it gets rendered when filtering is complete.
I want the data to be on the client side for a couple of reasons.
Is there the best way to solve the issue:
My current suggestions are these:
1) Filter the items by batches of 2000(or 5000) whatever makes sense.
Filter the first 2000 records show filtered results, after that filter the next 2000, show them and so until all items are iterated.
2) Using setTimeout() for each batch - but this could cause an issue because I don't know how long will take to filter each batch.
3) Using setImmediate - "This method is used to break up long-running operations" - its solution maybe but it is Non-standard, and I don't know if its gonna break sometime in the future.
4) Using promises somehow - I think it will be hard with promises because the code is sync (uses indexOf for example) and with or without the promises it will block the UI.
Can you recommend me something? I avoid large libraries or webworkers.
Thank you.
This does sound like a good use case for web workers. As they are off thread, it would not block the user interaction.
If I understand correctly, the data is already loaded and it's searching large datasets what is causing the delay. If this is correct:
I think the general answer to your question is using better data structures and algorithms to reduce the complexity.
Assuming that it does not need to match, but simply "start with":
You could store data in a Trie and run the tree until the point and return all the children.
If the data is ordered, you could implement a variation of binary search to look for the the index-range of elements.
If the issue is on handling the large dataset. Yes, loading it progressively would be best. APIs, for example, usually have a next page token for you to use this to call it again. You could do something like that, load a batch, complete the process and, when completed, call the same operations on the next batch.
All 1)-4) are valid points. But mainly the optimization of the search depends on your implementation. For example, if you search for strings starting with a given query then you could build a suffix tree https://www.geeksforgeeks.org/pattern-searching-using-suffix-tree/ to decrease complexity.
Also if you are researching thru that array every time user types a letter then I would debounce (https://underscorejs.org/#debounce) search function to be executed only once he stops typing.
I am building a project using node.js that is integrated with 4 other systems that keeps sending data from sensors every 1 second. I am trying to have like a timeline so I need to save that data, but I don't feel it's correct to hit a couple of insert statements every one second.
what is the best way to save data that is that redundant. I was thinking about having some log files and then insert data in bulk. Any suggestions?
Thank you.
This would make it a premature optimization. I've bench-marked PostgreSQL under Node.js many times. And at any given moment inserting several records per second will take under 10ms, i.e. less than 1% of your app's load, if you do it every second.
The only worthwhile optimization you should do from start - use multi-row insert, even if you insert only 2 rows at a time. The reasons for this are as follows:
Node.js IO is a valuable resource, so the fewer round trips you do the better
Multi-row inserts are tremendously faster than separate insert queries
Separate inserts typically require a transaction, and a single multi-row insert doesn't.
You can find a good example here: Multi-row insert with pg-promise.
Is it possible on Firebase or Parse to set up something kinda like a cron job?
Is there a way to set up some sort of timed operation that runs over the stored user data?
For example, I'm writing a program that allows people to RSVP for lunch everyday. If you have RSVPed by noon, then you get paired up with somebody else who has also RSVPed. Using JavaScript, the user can submit their RSVP in the browser.
The question is, can Firebase/Parse execute the code to match everyone at 12:00pm every day?
Yes, this can be done with Parse. You'll need to write your matching function as a background job in cloud code, and then you'll need to schedule the task in the dashboard. In terms of the flexibility in scheduling, it's not as flexible as cron but you can definitely run a task at the same time every day, or every x minutes/hours.
Tasks can take 15 mins max to execute before they're killed, so depending on the size of your database or the complexity of your task, you may need to break it up into different tasks or make it resumable.
Just to confirm about Firebase:
As #rickerbh said, it can be done with Parse, but currently there is no way for you to run your code on Firebase's server. There are 2 options for you 2 solve this:
You could use Firebase Queue and run your code in Node.js
You could use a different library such as Microsoft Azure (I still haven't tried this yet, I'm not sure if it provides Job Scheduling for Android)
However, Firebase is working on something called Firebase Trigger, which will solve our problem, however it is still not released with no confirmed release date.