Well I (naively) tried to get bull working in a sails application: ultimatelly I wish to have a queue to which I can add/remove/check tasks based on incoming routes.
Now as I understand sails to create a queueing system that works globaly I would have to add this setup in bootstrap.js.
/**
* Bootstrap
* (sails.config.bootstrap)
*
* An asynchronous bootstrap function that runs before your Sails app gets lifted.
* This gives you an opportunity to set up your data model, run jobs, or perform some special logic.
*
* For more information on bootstrapping your app, check out:
* https://sailsjs.com/config/bootstrap
*/
module.exports.bootstrap = function(done) {
// It's very important to trigger this callback method when you are finished
// with the bootstrap! (otherwise your server will never lift, since it's waiting on the bootstrap)
let Queue = require('bull');
let q = new Queue('test queue');
q.process(function(job, done){
console.log("starting job");
for(let i = 0; i<job.value; i+=1) {
console.log(i);
}
done();
});
q.add({'value':10});
global.DirectUpdateQueue = q;
return done();
};
Given above code, sails launches perfectly fine, and in the routes I can see the global.DirectUpdateQueue existing.
What does however not work is that the queued tasks are executed. - I do not see any log in the console ("starting job" is expected at least). Nor does the code break whe nI put a breakpoint in the processing function.
So what is going on here?
EDIT: can this be due to me not having set up a (local) redis server? - I don't find any information on this subject but I expected/hoped bull.js to actually handle this server internally and (even more importantly) not be limited to a specific (OS) environment.
So, first of all, you have to make sure you have Redis installed on your server.
When creating a queue, you can pass Redis config in my example below it's the default.
Then in bootsrap.js:
var Queue = require('bull');
var testQueue = new Queue('Website Queue', 'redis://127.0.0.1:6379');
testQueue.process(function(job, done){
console.log('job started');
setTimeout(function () {
console.log('10 seconds later');
console.log(job.data);
}, 10000)
done();
});
global.testQueue = testQueue;
then from action/controller you can do this:
testQueue.add({'value':10});
First you must connect to a Redis server
var testQueue = new Queue('test', {
redis: {
port: 6379,
host: '127.0.0.1',
password: 'secret'
}
});
According to the doc :
If the queue is empty the job will be executed directly, otherwise it will be placed in the queue and executed as soon as possible.
To access data in job, use job.data object :
testQueue.process((job) => {
console.log("job with data 'foo' :", job.data.foo);
// example with Promise
return asynchTreatment()
.then(() => { console.log('treatment ok'); })
.catch((err) => { console.log('treatment ko :', err); }
}).on('completed', (job, result) => {
// Job completed with output result!
console.log('result :', result);
});
testQueue.add({ foo : 'bar' });
EDIT 1 :
The doc says :
It creates a new Queue that is persisted in Redis. Everytime the same queue is instantiated it tries to process all the old jobs that may exist from a previous unfinished session.
So if the server restart, you don't lose your jobs.
Just use job.data.value in your for loop
for(let i = 0; i<job.data.value; i+=1) {
console.log(i);
}
Related
The function I would like this function to run by itself at time intervals. As it is now I have to visit the '/getCompanyInfo' path to trigger it. I would like it to run every minute as if I was visiting the '/getCompanyInfo' path each minute. The app is on Heroku and I would like the function to execute without any pages open.
The original function that is triggered by visiting the path.
const express = require('express');
const app = express();
/**
* getCompanyInfo ()
*/
app.get('/getCompanyInfo', function(req,res){
const companyID = oauthClient.getToken().realmId;
console.log(companyID)
const url = OAuthClient.environment.production ;
oauthClient.makeApiCall({url: url + 'v3/company/0000000000/salesreceipt/8?minorversion=41'})
.then(function(authResponse){
console.log("The response for API call is :"+JSON.parse(JSON.stringify(authResponse)));
res.send(authResponse);
})
.catch(function(e) {
console.error(e);
});
});
One of my attempts here was to put it in a function that executes each minute using node-schedule.
This one doesn't do anything other than print 'This will run once a minute.' to the console.
I tried removing
app.get(function(req,res){
and the
})
below it but that made the app (hosted on Heroku) fail to build.
const express = require('express');
const app = express();
var schedule = require('node-schedule');
var j = schedule.scheduleJob('* * * * *', function(){
console.log('This will run once a minute.');
app.get(function(req,res){
const companyID = oauthClient.getToken().realmId;
console.log(companyID)
const url = OAuthClient.environment.production ;
oauthClient.makeApiCall({url: url + 'v3/company/0000000000/salesreceipt/8?minorversion=41'})
.then(function(authResponse){
console.log("The response for API call is :"+JSON.parse(JSON.stringify(authResponse)));
res.send(authResponse);
})
.catch(function(e) {
console.error(e);
});
});
});
More Context:
It is inside an app I have on Heroku. I would like to set the app to make a requests for JSON data from the API every x time without me having to touch it.
app.get initializes api handler - e.g. this is your api route definition - the thing that will respond when you call GET /getCompanyInfo via web browser or some other client. You should not redefine it regularly with your scheduled action.
The failed build after you've removed the route handler is probably because of the res.send(authResponse); left behind.
You could have something like:
// function that will be used to get the data
const getCompanyInfo = (done) => {
const companyID = oauthClient.getToken().realmId
console.log(companyID)
const url = OAuthClient.environment.production
oauthClient.makeApiCall({url: url + 'v3/company/0000000000/salesreceipt/8?minorversion=41'})
.then((authResponse) => {
console.log("The response for API call is :"+JSON.parse(JSON.stringify(authResponse)))
done(authResponse)
})
.catch((e) => {
console.error(e)
})
}
// this will trigger the function regularly on the specified interval
const j = schedule.scheduleJob('* * * * *', () => {
getCompanyInfo((companyInfo) => {
// ...do whatever you want with the info
})
})
// this will return you the data by demand, when you call GET /getCompanyInfo via browser
app.get('/getCompanyInfo', function(req,res) {
getCompanyInfo((companyInfo) => {
res.send(companyInfo)
})
})
Heroku has an add on called Heroku Scheduler that does what you want. The node-schedule npm package might do the job, but as you mentioned, you probably aren't going to be able to see the execution/results/logs of your jobs that run every 24 hours without making some interface for it on your own.
For your issue, calling app.get doesn't make a lot of sense. That's just telling node about the route. Assuming you have your /getCompanyInfo route up and running, you just need to call it in your scheduled job, not re-register it every time.
You could also just do this (http being the http client you're using):
var j = schedule.scheduleJob('* * * * *', async function(){
console.log('This will run once a minute.');
const result = await http.get('/getCompanyInfo');
console.log(result);
});
I am currently trying to setup a server stream with the gRPC Node.js API. For that I want to achieve that when I write on server side to the stream that the client immediately receives the data event.
At the moment I don't receive anything on client side if I only call write on server side. However as soon as I call the end function on the server the client receives all data events.
To test this I used an endless while loop for writing messages on server side. Then the client does not receive messages (data events). If instead I use a for loop and call end afterwards the client receives all the messages (data events) when end is called.
My .proto file:
syntax = "proto3";
message ControlMessage {
enum Control {
Undefined = 0;
Start = 1;
Stop = 2;
}
Control control = 1;
}
message ImageMessage {
enum ImageType {
Raw = 0;
Mono8 = 1;
RGB8 = 2;
}
ImageType type = 1;
int32 width = 2;
int32 height = 3;
bytes image = 4;
}
service StartImageTransmission {
rpc Start(ControlMessage) returns (stream ImageMessage);
}
On the server side I implement the start function and try to endlessly write messages to the call:
function doStart(call) {
var imgMsg = {type: "Mono8", width: 600, height: 600, image: new ArrayBuffer(600*600)};
//for(var i = 0; i < 10; i++) {
while(true) {
call.write(imgMsg);
console.log("Message sent");
}
call.end();
}
I register the function as service in the server:
var server = new grpc.Server();
server.addService(protoDescriptor.StartImageTransmission.service, {Start: doStart});
On client side I generate an appropriate call and register the data and end event:
var call = client.Start({control: 0});
call.on('data', (imgMessage) => {
console.log('received image message');
});
call.read();
call.on('end', () => {console.log('end');});
I also tried to write the server side in python. In this case the node client instantly receives messages and not only after stream was ended on server side. So I guess this should be also possible for the server written with the Node API.
It seems that the problem was that the endless while loop is blocking all background tasks in node. A possible solution is to use setTimeout to create the loop. The following code worked for me:
First in the gRPC call store the call object in an array:
function doStart(call) {
calls.push(call);
}
For sending to all clients I use a setTimeout:
function sendToAllClients() {
calls.forEach((call) => {
call.write(imgMsg);
});
setTimeout(sendToAllClients, 10);
}
setTimeout(sendToAllClients, 10);
Helpful stackoverflow atricle: Why does a while loop block the event loop?
I was able to use uncork which comes from Node.js's Writable.
Here is an example. Pseudocode, but pulled from across a working implementation:
import * as grpc from '#grpc/grpc-js';
import * as proto from './src/proto/generated/organizations'; // via protoc w/ ts-proto
const OrganizationsGrpcServer: proto.OrganizationsServer = {
async getMany(call: ServerWritableStream<proto.Empty, proto.OrganizationCollection>) {
call.write(proto.OrganizationCollection.fromJSON({ value: [{}] }));
call.uncork();
// do some blocking stuff
call.write(proto.OrganizationCollection.fromJSON({ value: [{}] }));
call.uncork();
// call.end(), or client.close() below, at some point?
},
ping(call, callback) {
callback(null);
}
};
const client = new proto.OrganizationsClient('127.0.0.1:5000', grpc.credentials.createInsecure());
const stream = client.getMany(null);
stream.on('data', data => {
// this cb should run twice
});
export default OrganizationsGrpcServer;
//.proto
service Organizations {
rpc GetMany (google.protobuf.Empty) returns (stream OrganizationCollection) {}
}
message OrganizationCollection {
repeated Organization value = 1;
}
Versions:
#grpc/grpc-js 1.4.4
#grpc/proto-loader 0.6.7
ts-proto 1.92.1
npm 8.1.4
node 17
I have a MongoDB instance and two JavaScript services running on a Linux server. The first service, moscaService.js, listens to MQTT topics on the server, and records what is sent in a MongoDB collection. The second service, integrationService.js, runs every second, reading data on the same MongoDB collection and, if there's a new register (or more), sends it to Ubidots.
The problem is that both services work on the same IP/port: localhost:27017; and, if there ever is an occasion in which both of them are active simultaneously (say, moscaService.js is recording something and then the integrationService.js tries to connect), there will be a connection error and the service will restart.
Here are the connection parts of both services:
var MongoClient = require('mongodb').MongoClient;
var url = 'mongodb://127.0.0.1:27017/myGateway';
//integrationService.js
var job1 = new CronJob('*/1 * * * * *', function() {
MongoClient.connect(url, function(err, db) {
if(err != null) {
logger.error({message: 'Connection error: ' + err});
process.exit(0);
} else {
executeService();
}
function executeService() {
// execution block
}
});
}, null, true, timeZone);
//moscaService.js
server.on('published', function(packet, client) {
//the packet is read here
MongoClient.connect(url, function(err, db) {
if(err != null) {
logger.error({message: 'Connection error: ' + err});
process.exit(0);
} else {
executeService();
}
function executeService() {
// execution block
}
});
});
What I need is a way to properly handle the err instead of just exiting the service, because if there are new messages being published while the service is restarting, they will be lost. Something like testing if the port is open before connecting, or open a different port.
I tried creating another instance of MongoDB on a different port, in order to have each service listen to one, but it looks like Mongo locks more than one instance if it's trying to connect to the same database.
The code snippets here are just a small part; if anyone needs more parts to answer, just say so and I'll add them.
I have made an alteration and it solved this issue. I altered the code in a way that integrationService connects to MongoDB before starting the CronJob; that way, it only connects once and it keeps the connection alive.
Here's the connection part of the code:
var MongoClient = require('mongodb').MongoClient;
var url = 'mongodb://127.0.0.1:27017/myGateway';
//integrationService.js
MongoClient.connect(url, function(err, db) {
var job1 = new CronJob('*/1 * * * * *', function() {
if(err != null) {
logger.error({message: 'Connection error: ' + err});
process.exit(0);
} else {
executeService();
}
function executeService() {
// execution block
}
}, null, true, timeZone); // end CronJob
}); // end MongoClient.connect
Since this has solved the problem, I've left the err treatment as is was (although a more elegant way to treat it is still desirable).
Solving the problem on integrationService has solved it on moscaService as well, but I plan to make the same alteration on the second service too.
I would like to separate server high consuming CPU task from user experience:
./main.js:
var express = require('express');
var Test = require('./resources/test');
var http = require('http');
var main = express();
main.set('port', process.env.PORT || 3000);
main.set('views', __dirname + '/views');
main.use(express.logger('dev'));
main.use(express.bodyParser());
main.use(main.router);
main.get('/resources/test/async', Test.testAsync);
main.configure('development', function() {
main.use(express.errorHandler());
});
http.createServer(main).listen(main.get('port'), function(){
console.log('Express server app listening on port ' + main.get('port'));
});
./resources/test.js:
function Test() {}
module.exports = Test;
Test.testAsync = function(req, res) {
res.send(200, "Hello world, this should be sent inmediately");
process.nextTick(function() {
console.log("Simulating large task");
for (var j = 0; j < 1000000000; j++) {
// Simulate large loop
}
console.log("phhhew!! Finished!");
});
};
When requesting "localhost:3000/resources/test/async" I would expect the browser rendering "Hello world, this should be sent inmediately" really fast and node.js to continue processing, and after a while in console appearing "finished" message.
Instead, browser keeps waiting until node.js finishes large task and then renders the content. I've tried with res.set({ 'Connection': 'close' }); and also res.end(); but nothing works as expected. I've also googled with no luck.
How should it be to send the response to client immediately and server continue with tasks?
EDIT
posted fork method in solution
Try waiting instead of hogging the CPU:
res.send("Hello world, this should be sent inmediately");
console.log("Response sent.");
setTimeout(function() {
console.log("After-response code running!");
}, 3000);
node.js is single-threaded. If you lock up the CPU with a busy loop, the whole thing grinds to a halt until that is done.
Thakns for Peter Lyons help, finally the main problem was firefox buffer: response was not so long as to flush it (so firefox kept waiting).
Anyway, for hight CPU performing tasks, node would keep hanged until finishing, so will not be attending new requests. If someone needs it, it can be achieved by forking (with child_process, see sample in http://nodejs.org/api/child_process.html)
Have to say that change of context by forking could take longer than splitting the task in different ticks.
./resources/test.js:
var child = require('child_process');
function Test() {}
module.exports = Test;
Test.testAsync = function(req, res) {
res.send(200, "Hello world, this should be sent inmediately");
var childTask = child.fork('child.js');
childTask.send({ hello: 'world' });
};
./resources/child.js:
process.on('message', function(m) {
console.log('CHILD got message:', m);
});
A good solution is to use child_process.fork(): it allows you to execute another JavaScript file of your app in a different Node instance, and thus in a different event loop. Of course, you can still communicate between the two processes by sending messages: so, from your UI process, you can send a message to the forked process to ask it to execute something.
For example, in ui.js:
var ChildProcess = require('child_process');
var heavyTaskWorker = ChildProcess.fork('./heavyTaskWorker.js');
...
var message = {
operation: "longOperation1",
parameters: {
param1: "value1",
...
}
};
heavyTaskWorker.send(message);
And in heavyTaskWorker.js:
process.on('message', function (message) {
switch (message.operation) {
case 'longOperation1':
longOperation1.apply(null, message.parameters);
break;
...
}
});
Tested here, and it works fine!
Hope that helps!
I have app in Node.js and Express. I need to write tests for it. I have a problem with handling Express app errors. I found this How do I catch node.js/express server errors like EADDRINUSE?, but it doesn't work for me, I don't know why. I want to handle errors, which can occured while expressApp.listen() is executing (EADDRINUSE, EACCES etc.).
express = require('express')
listener = express()
#doesn't work for me
listener.on('uncaughtException', (err) ->
#do something
)
#doesn't work too
listener.on("error", (err) ->
#do something
)
#this works, but it caughts all errors in process, I want only in listener
process.on('uncaughtException', (err) ->
#do something
)
listener.listen(80) #for example 80 to get error
Any ideas?
This should do the trick:
listener.listen(80).on('error', function(err) { });
What listener.listen actually does is create a HTTP server and call listen on it:
app.listen = function(){
var server = http.createServer(this);
return server.listen.apply(server, arguments);
};
First off, expressJS does not throw the uncaughtException event, process does, so it's no surprise your code doesn't work.
So use: process.on('uncaughtException',handler) instead.
Next, expressJS already provides a standard means of error handling which is to use the middleware function it provides for this purpose, as in:
app.configure(function(){
app.use(express.errorHandler({ dumpExceptions: true, showStack: true }));
});
This function returns an error message to the client, with optional stacktrace, and is documented at connectJS errorHandler.
(Note that errorHandler is actually part of connectJS and is only exposed by expressJS.)
If the behavior the existing errorHandler provides is not sufficient for your needs, its source is located at connectJS's errorHandler middleware and can be easily modified to suit your needs.
Of course, rather than modifying this function directly, the "correct" way to do this is to create your own errorHandler, using the connectJS version as a starting point, as in:
var myErrorHandler = function(err, req, res, next){
...
// note, using the typical middleware pattern, we'd call next() here, but
// since this handler is a "provider", i.e. it terminates the request, we
// do not.
};
And install it into expressJS as:
app.configure(function(){
app.use(myErrorHandler);
});
See Just Connect it, Already for an explanation of connectJS's idea of filter and provider middleware and How To Write Middleware for Connect/Express for a well-written tutorial.
You might also find these useful:
How to handle code exceptions in node.js?
Recover from Uncaught Exception in Node.JS
Finally, an excellent source of information regarding testing expressJS can be found in its own tests.
Mention: Marius Tibeica answer is complete and great, also david_p comment is. As too is Rob Raisch answer (interesting to explore).
https://stackoverflow.com/a/27040451/7668448
https://stackoverflow.com/a/13326769/7668448
NOTICE
This first method is a bad one! I leave it as a reference! See the Update section! For good versions! And also for the explanation for why!
Bad version
For those who will find this useful, here a function to implement busy port handling
(if the port is busy, it will try with the next port, until it find a no busy port)
app.portNumber = 4000;
function listen(port) {
app.portNumber = port;
app.listen(port, () => {
console.log("server is running on port :" + app.portNumber);
}).on('error', function (err) {
if(err.errno === 'EADDRINUSE') {
console.log(`----- Port ${port} is busy, trying with port ${port + 1} -----`);
listen(port + 1)
} else {
console.log(err);
}
});
}
listen(app.portNumber);
The function listen is recursively calling itself. In case of port busy error. Incrementing the port number each time.
update Completely re-done
Callback full version
First of all this version is the one that follow the same signature as nodejs http.Server.listen() method!
function listen(server) {
const args = Array.from(arguments);
// __________________________________ overriding the callback method (closure to pass port)
const lastArgIndex = arguments.length - 1;
let port = args[1];
if (typeof args[lastArgIndex] === 'function') {
const callback = args[lastArgIndex];
args[lastArgIndex] = function () {
callback(port);
}
}
const serverInstance = server.listen.apply(server, args.slice(1))
.on('error', function (err) {
if(err.errno === 'EADDRINUSE') {
console.log(`----- Port ${port} is busy, trying with port ${port + 1} -----`);
port += 1;
serverInstance.listen.apply(serverInstance, [port].concat(args.slice(2, lastArgIndex)));
} else {
console.log(err);
}
});
return serverInstance;
}
Signature:
listen(serverOrExpressApp, [port[, host[, backlog]]][, callback])
just as per
https://nodejs.org/api/net.html#net_server_listen_port_host_backlog_callback
The callback signature is changed to
(port) => void
usage:
const server = listen(app, 3000, (port) => {
console.log("server is running on port :" + port);
});
// _____________ another example port and host
const server = listen(app, 3000, 'localhost', (port) => {
console.log("server is running on port :" + port);
});
Explanation
In contrary to the old example! This method doesn't call itself!
Key elements:
app.listen() first call will return a net.Server instance
After binding an event once, calling listen again into the same net.Server instance will attempt reconnecting!
The error event listener is always there!
each time an error happen we re-attempt again.
the port variable play on the closure to the callback! when the callback will be called the right value will be passed.
Importantly
serverInstance.listen.apply(serverInstance, [port].concat(args.slice(2, lastArgIndex)));
Why we are skipping the callback here!?
The callback once added! It's hold in the server instance internally on an array! If we add another! We will have multiple triggers! On the number of (attempts + 1). So we only include it in the first attempt!
That way we can have the server instance directly returned! And keep using it to attempt! And it's done cleanly!
Simple version port only
That's too can help to understand better at a glimpse
function listen(server, port, callback) {
const serverInstance = server.listen(port, () => { callback(port) })
.on('error', function (err) {
if(err.errno === 'EADDRINUSE') {
console.log(`----- Port ${port} is busy, trying with port ${port + 1} -----`);
port += 1;
serverInstance.listen(port);
} else {
console.log(err);
}
});
return serverInstance;
}
Here the parameter port variable play on the closure!
ES6 full version
function listen(server, ...args) {
// __________________________________ overriding the callback method (closure to pass port)
const lastArgIndex = args.length - 1;
let port = args[0];
if (typeof args[lastArgIndex] === 'function') {
const callback = args[lastArgIndex];
args[lastArgIndex] = function () {
callback(port);
}
}
const serverInstance = server.listen(server, ...args)
.on('error', function (err) {
if(err.errno === 'EADDRINUSE') {
console.log(`----- Port ${port} is busy, trying with port ${port + 1} -----`);
port += 1;
serverInstance.listen(...[port, ...args.slice(1, lastArgIndex)])
} else {
console.log(err);
}
});
return serverInstance;
}
Why the old version is bad
To say right it's not really! But with the first version! We call the function itself at every failure! And each time it create a new instance! The garbage collector will budge some muscles!
It doesn't matter because this function only execute once and at start!
The old version didn't return the server instance!
Extra (for #sakib11)
You can look at #sakib11 comment to see what problem he fall in! It can be thoughtful!
Also in the comment i mentioned promise version and closure getter pattern! I don't deem them interesting! The way above just respect the same signature as nodejs! And too callback just do fine! And we are getting our server reference write away! With a promise version! A promise get returned and at resolution we pass all the elements! serverInstance + port!
And if you wonder for the closure getter pattern! (It's bad here)
Within our method we create a ref that reference the server instance! If we couldn't return the server instance as we are doing (imaging it was impossible! So each time a new instance is created! The pattern consist of creating a closure (method at that scope) And return it!
so for usage
const getServer = listen(port, () => {
console.log('Server running at port ' + getServer().address().port);
const io = socketIo(getServer(), {});
});
But it's just overhead specially we need to wait for the server to be done!
Unless we set it in a way that it use a callback! or return a promise!
And it's just over complicating! And not good at all!
It's just because i mentioned it!
And the method above can be tweaked! To add number of attempts limit! And add some events or hooks! But well! Generally we only need a simple function that just attempt and make it! For me the above is more then sufficient!
Good links
https://nodejs.org/api/http.html#http_http_createserver_options_requestlistener
https://nodejs.org/api/http.html#http_class_http_server
https://expressjs.com/en/4x/api.html#app.listen
From the doc
The app.listen() method returns an http.Server object and (for HTTP) is a convenience method for the following:
app.listen = function () {
var server = http.createServer(this)
return server.listen.apply(server, arguments)
}