I have a configuration page where the user can select at what time to receive certain information from my API, for example they could select to get the information every day at 5PM or every Friday at 5PM. After this is set, the user should receive a Slack message at the time they defined, for example, every day at 5PM they would receive a message...
I save the user's timezone so that I can send the message to them in Slack at the correct time for them.
That being said, how can I schedule this message to send out from my node.js app? I would have a few users in the system, all who would have likely chosen different times so I would need a timer per user?
The only thing I can think of is scanning all users in the system, getting their selected time and then sending the message to them...but this doesn't seem scalable.
I'm not looking for a complete solution for this, just some pointers for how to design this sort of functionality.
I also looked into Slack scheduled messages but this isn't exactly what I'm looking for. Note: I can already send messages to Slack, I'm more interested in how to build the timer mechanism.
Thanks in advance!
EDIT:
Did a bit more research and it looks like node-schedule could be an option to schedule jobs: https://github.com/node-schedule/node-schedule#readme
With using this package, is the approach that I scan all users in my database at let's say midnight everyday and schedule jobs based on their settings...then those jobs execute at their scheduled time and the user receives the message in Slack. Is this a good approach?
For the core logic I would suggest something like this:
Store the timing of delivery (e.g. 5 AM ever Friday) for every user in your database
Then have a worker process that is running on a regular basis, e.g.
every 5 minutes
When it runs it checks if there are any due messages to be delivered
If yes it sends the message with your API information to the user and store the last time of sending for the user
This approach is resilient to downtime. It will just resume sending due messages once the worker process is running again after a downtime.
It is also scaleable: If needed you can run multiple worker processes (make sure to design your workers to support concurrent processing, e.g. with transactions)
Some additional things to consider:
Would limit the number of messages sent per run to avoid timeouts and having too many workers running in parallel
You need some error handling if sending message to Slack fails
To avoid timezone complexities I would suggest to convert all timings to UTC for processing in your app
Is this a regular task? In other words, it's executed every day at the same time for user X? If so, node-schedule seems fine, and it can run the same job regularly, you just have to set it up properly via a cron-like string (see the instructions in the README). If a user changes their setting, you then modify the previous job. The downside with node-schedule is that you need to set it up everytime your application is loaded, which can take a while and consume lots of resources if you have too many users.
Alternatively, if the number of users is big or you prefer to keep your application stateless, you can set a number of slots for sending these messages (and run that in a separate process from your main application). Let's say, a slot every 30 min. Then you set timers for those time slots (using node-schedule if you like, it will be just 48 timers), fetch the list of users for that time slot from the database, and send the messages.
Overall, NodeJS/JavaScript is pretty efficient with this sort of timer-based scheduling. If you want an in-depth dive into the reasons, see this: https://nodejs.org/de/docs/guides/event-loop-timers-and-nexttick/
You also need to consider what happens in case your application suffers from downtime. Should users be guaranteed to receive those messages, even if they are late? But that's another story :-)
Related
I'm currently using cluster with Node.js. The user will be able to send a request to the server to check on something. What I'm struggling to figure out is how to approach this.
One solution would be for the user to request the information and the server gets it for them. So every 10 minutes the user would send a request to the server. This way, it's handled automatically with express and cluster, but it's not exactly what I would prefer. I would prefer the server to have cron-like tasks that spread around the cluster.
I was thinking another solution might be to have a database with the list of user tasks to do. A cluster periodically checks the database, checks all tasks and their "last ran" time stamp, then does a task if the last ran timestamp was 10 minutes ago. I feel like there must be a cleaner solution, though.
I want to make a custom tracking system for web events. I have looked into multiple per-excsiting systems, but I want something terribly simple - yet very accurate.
I want to be able to track the following:
Page view even
Time on that page
or:
Video started playing event
Time of video watched
My first initial thought was to do a simple javascript reporting back to the server, but what happens if the user closes the window? How do I know they stopped viewing? And how can I get accurate measurements down to 1/10th of a second? So I thought of a websocket solution, as it know when a user has discounted. I ended up with Socket.io, but I want to make sure there is no better or smarter way to achieve this?
How would you approach his challenge? What is the smartest way to engineer this?
A Websocket connection which reports back to the server frequently was my first thought as well, but if you send 10 messages every second, even that might be too much for a websocket, especially when connectivity isn't top-notch.
Since the server doesn't require the information absolutely immediately, consider batching requests instead - save/update the information into Local Storage every 0.1 seconds, but don't send it to the server then - instead, every 30 or 60 seconds, or on pageload, take the current data in Local Storage and send it to the server, and clear Local Storage so that the next request a minute from now doesn't send duplicate data.
So I've been messing around with node.js and twilio these past couple of days. I managed to get it running and create a nice little system of keyword checks and responses. The problem is I need a way to prevent a conversation from going beyond X amount of exchanged messages. Anything that can get the user to stop messaging or get twilio to stop accepting them would be great.
I've looked into things like blocking their number, or even trying to activate the STOP keyword for them but nothing came up. The best I thought of was creating a counter of inbound and outbound messages and sending a final messaging stating "You've reached the conversation limit. You'll be charged X amount per every new message sent and received" to scare them off.I'm not really sure if that option is ethical or even legal.
I don't want to wake up one day and find that I owe $5000 in SMS fees because my numbers were spammed even after users have already gone through every possible dialogue option. Any help is appreciated.
EDIT : I think I've found my answer so I'll post it here in case anyone else has the same thought. https://support.twilio.com/hc/en-us/articles/223181648-Is-there-a-way-to-block-incoming-SMS-on-my-Twilio-phone-number-
Twilio billing is prepaid (for me), so i don't think you will "owe $5000". They could burn thru your credit and if you are set up for auto top up, that could be an issue.
That being said there are different ways to keep on top of spending without coding. I would recommend taking a look at the following:
Account Triggers - Get notifications
Projects - Isolate your numbers/apps that are in production
Some Coding Ideas
I am guessing you are using a twilio number to bridge two users that gives anonymity. What you proposed should be possible.
Use a function to query your logs and rate limit with your own logic
Docs: Usage Records
Use a database, record transactions, and query the data base with your own logic
Note: if the incoming message is generated by an incoming sms/mms, an incoming fee will be charged regardless of if the message is relayed or not.
Additionally, there are rate limiting npm modules available (I have not explored them). ex// npm limiter
As mentioned in the comments, this specific question is off-topic. However, I'll state few things.
As of now, you can't control incoming messages. Twilio does that for you. For example, twilio charge you only for numbers validated as spam-free. You can get more information from their support agents.
From your side, you can try few other things.
Control the outgoing messages (For instance, detect spam, and do not send automatic replies if any)
Stick into a basic plan
For controlling the incoming load to your application, there is a library called queuep(npm install queuep). You can easily write the spam detection logic and avoid a load of messages. There are other benefits such as throttling and memoization.
But this does not guarantee that you are not charged for the load if any. That is because, charging happens even before the message is received by the NodeJs application. Only possible place to control this is through the twilio admin panel
I have created a chat with node.js and socket.io.
When a user sends a message, I insert it directly in the DOM and emit the message to the server, so it can be emitted to all the other clients.
The problem is that it seems the timestamp set with Date.now() on the server is different from the timestamp set on the client with exact same command.
This makes the interface a bit weird because a message sent at a later point in time can show a timestamp prior to previously sent messages.
One solution would be to calculate the time difference when the user joins the chat room and subtract this difference when a new message is added, but should this really be necessary or is this the common way to solve this problem? Could this also be the solution to cope with timezones etc?
Different time zones between the client and the server possibly. It will not work with multiple users across the world. This one will be helpful:
How to ignore user's time zone and force Date() use specific time zone
Also, if you append the timestamp to the DOM immediately, and then get the date again from the server, there will be a difference in any case because you have to consider the time that the request needs to reach the server.
I am creating an auction site in which the auction has a particular end date/time.
I have a javascript timer displaying the remaining time to the user and when this countsdown it fires and event to update the back end mongodb database to say the auction has completed and it informs the winning user and fires a CLOSE function.
How would you recommend doing this for auctions that aren't physically open in a browser at the time so the Timer event never creates this CLOSE event.
I wouldn't recommend this approach at all. Once I was working on a website and I needed to implement auctions. The project was in php so I wrote a php script which checks all auction rows in the database and looks for those with the timeEnd >= current time and sets their status to closed(The auctions table had int column for the status).
Then I set that php script to run as a cron job once every hour. So now the server automatically runs this server side script every hours and checks for out-dated auctions. The interval depends on the business logic of the app. For this project the auctions could only end or start to the beginning of every hour. This approach is far better than using javascript code that triggers the server script. One reason is that you can't trust client side code. Hackers could potentially get access to that javascript file and easily modify it. You should never let your server code depend on your client side code.
However, note that my approach is not the most ideal because depending on how much auctions your db have, the server script will still need time to process it and might take from a few seconds to couple of minutes to execute it.
For example if you have some auction that ends at 10:00:00 and the server script start executing at 10:00:00 and it takes 40 seconds to execute, the users could potentially find a way to place bid on those auctions in the interval of that 40 seconds. Your client side code should only take care for resetting the interface right at 10:00:00 so that users are not able to place bids. However, you should also make sure that the server-side code that handles your POST requests for placing bids, should also check if the auction end time is in the past before proceeding. If it only checks the status of the auction (opened or closed) it might get auctions that are ended with their status set to opened. The reason is that the cron job might still be processing the auctions and changing their status.
Another similar approach is to create service that runs on operative system level (probably c or c++ app) that would run constantly in the background and do the checks.
The good thing with the first approach is that most of the hosting companies already offer setting up cron jobs. One example is Bluehost.
For setting up windows based "cron jobs" read additional info on this post
I hope this makes it more clear to you how to handle the auctions.