We have a node server which doesn't handle post requests properly when they are made asynchronously. When the requests are made synchronously, it handles them fine.
There is a node api server and to mimic a client, there is node script which makes a post request to the server.
While making a single post request or post requests in a loop synchronously, everything works as expected.
While making asynchronous post requests in a loop to the server, the code doesn't execute properly.
Here is the code on the server side. This method is called from router.post() method.
async insert(params) {
let account = new Account();
try {
let totalLicenses = await this.getLicenses(params);
if (totalLicenses === 0) throw new AccountError('NO_AVAILABLE_LICENSE');
let accountResponse = await account.insert(params);
let useLicense = await license.use(accountResponse, params);
/*
Do other account setup stuff here
*/
return accountResponse;
} catch(err) {
throw err;
}
}
getLicenses(params) is a async function that prepares the sql query and awaits the response from a queryDb method which wraps the callback in a promise. Here is the code:
getLicense(params) {
let vals = [...arguments],
sql = 'my sql query';
try {
return await queryDb(sql, val);
} catch (err) {
throw new Error()
}
}
We are using mysql package to talk to db. This is what queryDb method looks like.
queryDb(query, vals) {
return new Promise( (resolve, reject) => {
connection.query(query, vals, (err, results, fields) => {
if (err) reject(err);
resolve(results);
}
}
}
While making asynchronous post requests in a loop to the server, the insert method above executes this.getLicenses(params) method for the requests and then calls the account.insert(params) for the requests and then license.use(accountResponse, params) for the requests.
This becomes a problem when let's say a client has 3 licenses available and they send 5 asynchronous post requests. Technically, it should throw error for the last 2 requests but that is not the case. What ends up happening is, it inserts all 5 accounts since it calls this.getLicenses(params) for all 5 requests before proceeding to insert the account.
Any help is appreciated!
Related
I have the following code:
await axiosAPICall(dummyData); // works
const sqlQuery = `SELECT column1, column2 FROM table`;
const queryStream = mySqlConnectionInstance.query(sqlQuery, []);
queryStream
.on('error', function (err) {
// Handle error, an 'end' event will be emitted after this as well
})
.on('result', async (actualData) => {
// axios api call, the api callback goes to callstack with any of the following errors:
// 1. read ECONNRESET
// 2. Client network socket disconnected before secure TLS connection was established
await axiosAPICall(actualData); // breaks
})
.on('end', function () {
// all rows have been received
});
As you can see I'm getting all the rows from a table in a MySQL database stream. When the data comes from the database stream, I'm passing that data to the axios API call.
The API call works perfectly fine when called outside of the stream logic but when I call the API inside the streaming logic, it breaks all the time.
I am hitting API calls as fast as each on('result') gets called (the async/await does NOT slow down the request rate, i.e. I end up with multiple requests in parallel.
Does anyone know why is API calls not working inside the streaming logic section?
If the question needs any clarifications please comment.
Based on a comment that suggests the error is due to making "too many requests" at once - this is a simple and naive way to wait for the previous request before making the next
const sqlQuery = `SELECT column1, column2 FROM table`;
const queryStream = mySqlConnectionInstance.query(sqlQuery, []);
const wait = (ms) => new Promise(resolve => setTimeout(resolve, ms));
let previous = Promise.resolve();
queryStream
.on('error', function (err) {
// Handle error, an 'end' event will be emitted after this as well
})
.on('result', async (actualData) => {
// wait until previous request has completed
await previous;
// optional, add a delay,
// 100ms for this example -
// for example if there is a limit of 10 requests per second
// adjust (or remove) as required
await wait(100);
// set "previous" for next request
previous = axiosAPICall(actualData);
})
.on('end', function () {
// if you `await previous` here,
// you can truly wait until all rows are processed
// all rows have been received
});
I am trying to writ data to Amazon Kinesis using AWS SDK for NodeJS. I am using putRecords method which I want to use with async/await. I need to preform few other steps once the records are written to Kinesis. I want this to be a synchronous process, so I cannot use just callbacks.
I have tried few different ways using promise, async/await but none of them seem to be working.
With Async/Await:
let response = await kinesisPutRecords({
Records: //records,
StreamName: //streamName
});
async function kinesisPutRecords(recordsParams){
return new Promise((resolve, reject)=>{
kinesisClient.putRecords(recordsParams,function (err, data) {
if (err)
reject(err);
resolve();
}
});
}
With Promise
let response = await kinesisClient.putRecords(recordsParams).promise();
console.log(response);
This approaches are used inside a loop, the loop continues without waiting for this call to finish.
These two approaches work fine with S3.getObject();
Any help is appreciated. Thank you.
Update 1:
I found the problem with the above code. Its just missing try...catch block is causing the issue. In my full code, I still have this issue but I was able to find the root cause.
I am using csv tarnsform with NodeJS streams. Inside the transform function, await is not working. May be because of the fact that stream is async and do not pause?
Code close to my scenario:
const transformer = transform(function (record) {
//add records to an array until a threshold is met and then push to kinesis.
let response = await kinesisPutRecords();
console.log(response);
}, function(err, output){
//logic to push remaining records
});
fileReadStream
.pipe(gunzip) //unzip the file
.pipe(csvParser) //to parse the csv
.pipe(transformer);
async function kinesisPutRecords(recordsParams){
return new Promise((resolve, reject)=>{
try{
kinesisClient.putRecords(recordsParams,function (err, data) {
if (err)
reject(err);
resolve();
});
}
catch(exception ex){
reject();
}
});
}
I'm trying to implement a Sequence of HTTP-Requests for training purposes in express.js. In order to do that i worked with async/await to wait for my requests and wrapped them in a try catch block. Everything works as aspected, since i added some console.log to check the flow. When everything is done, i want to render my page with the built in res.render() function but it throws an error when calling res.render()[res.render is not a function].
I could load the data via AJAX but is this really necessary?
/* Express Functions */
router
.get('/', async function (res,req,next){
try{
doSynchronousThings();
const ParticipantIds = await xingRequests.getParticipantIds();
console.log('...getParticipantIds response:');
console.log(ParticipantIds);
finalLog();
const ParticipantData = await xingRequests.getParticipantData(ParticipantIds[0]);
console.log('...getParticipantData response:');
console.log(ParticipantData);
console.log('...all done');
}catch(err){
console.error(err);
next(err);
}
res.render('test', {
page: {
title:'Xing Event API',
message: 'Happy Hacking!'
}
});
});
The xingRequest methods just fire a request-promise based request and return the response objects.
In Node.js (which I'm new to) I am trying to perform a series of tasks after receiving a response. However, I want to make the response time as fast as possible. I don't need to return the results of these tasks to the client, so I'm trying to return the response immediately.
My current implementation is roughly:
var requestTime = Date.now;
app.post('/messages', function (req, res) {
console.log("received request");
// handle the response
var body = res.body;
res.send('Success');
res.end();
console.log("sent response");
performComplexTasks(body)
})
function performComplexTasks(body){
// perform data with body data here;
console.log("finished tasks:", Date.now()-requestTime, "ms");
}
// -------LOG-----------
// received request
// POST /api/messages 200 3.685 ms - 59
// sent response
// finished tasks: 2500ms
The client making the request seems to hang until performComplexTasks() is finished. (The POST finishes in 3.685ms, but the response takes 2500ms to finish.)
Is there a way to send the response immediately and complete other tasks without having the client wait/hang? (In my case, the client cannot make multiple API calls.)
If your job is not super-CPU-intense and you can tolerate some work on the main server thread, then just use await to break the execution so that the request can be properly sent. You can use setTimeout or await.
// This line is wrong too - ask a separate question if needed
var requestTime = Date.now;
app.post('/messages', async function (req, res) {
console.log("received request");
// handle the response
var body = res.body;
res.status(200).send({ success: true });
console.log("sent response");
// Method 1:
await performComplexTasks(body)
// Method 2:
setTimeout(() => performComplexTasks(body), 0);
})
async function performComplexTasks(body){
// The line below is required if using the `await` method -
// it breaks execution and allows your previous function to
// continue, otherwise we will only resume the other function after
// this function is completed.
await new Promise(resolve => setTimeout(resolve, 1));
// perform data with body data here;
console.log("finished tasks:", Date.now()-requestTime, "ms");
}
This isn't really a fantastic solution and you'd need to use worker threads for long operations.
Am I right that you're trying to execute a CPU-intensive job in performComplexTasks? If so, then event loop is being locked by that task and new requests are waiting until the job is finished.
It's a bad practice in node.js to execute such 'complex' tasks in the same process as http server. Consider using background workers, queues or something like that.
See this topic for details: Node.js and CPU intensive requests
I am currently using node-serialport module for serial port communication. I will send a command ATEC and it will respond with ECHO.
However, this process of sending and receiving data is async(after i send the data, i will not know when the data will arrive in the data event), the example code is below:
//Register the data event from the serial port
port.on('data', (data) => {
console.log(data);
});
//Send data using serialport
port.write('ATEC');
Is there anyway I could write it in this way?
//When i send the command, I could receive the data
port.write('ATEC').then((data)=> {
console.log(data);
});
Is this possible to achieve?
In http communication using request client, we could do something like
request.get('http:\\google.com')
.on('response', (res) => {
console.log(res);
});
I want to replicate the same behaviour using serialport
I wrap a promise in the serial data receive
function sendSync(port, src) {
return new Promise((resolve, reject) => {
port.write(src);
port.once('data', (data) => {
resolve(data.toString());
});
port.once('error', (err) => {
reject(err);
});
});
}
Please take note, the event is using once instead of on to prevent event from stacking (please check the comments below for more information - thanks #DKebler for spotting it)
Then, I could write the code in sync as below
sendSync(port, 'AThello\n').then((data) => {
//receive data
});
sendSync(port, 'ATecho\n').then((data) => {
//receive data
});
or I could use a generator, using co package
co(function* () {
const echo = yield sendSync(port, 'echo\n');
const hello = yield sendSync(port, 'hello 123\n');
return [echo, hello]
}).then((result) => {
console.log(result)
}).catch((err) => {
console.error(err);
})
We have a similar problem in a project I'm working on. Needed a synchronous send/receive loop for serial, and the serialport package makes that kinda weird.
Our solution is to make some sort of queue of functions/promises/generators/etc (depends on your architecture) that the serial port "data" event services. Every time you write something, put a function/promise/etc into the queue.
Let's assume you're just throwing functions into the queue. When the "data" event is fired, it sends the currently aggregated receive buffer as a parameter into the first element of the queue, which can see if it contains all of the data it needs, and if so, does something with it, and removes itself from the queue somehow.
This allows you to handle multiple different kinds of architecture (callback/promise/coroutine/etc) with the same basic mechanism.
As an added bonus: If you have full control of both sides of the protocol, you can add a "\n" to the end of those strings and then use serialport's "readline" parser, so you'll only get data events on whole strings. Might make things a bit easier than constantly checking input validity if it comes in pieces.
Update:
And now that code has been finished and tested (see the ET312 module in http://github.com/metafetish/buttshock-js), here's how I do it:
function writeAndExpect(data, length) {
return new Promise((resolve, reject) => {
const buffer = new Buffer(length);
this._port.write(data, (error) => {
if (error) {
reject(error);
return;
}
});
let offset = 0;
let handler = (d) => {
try {
Uint8Array.from(d).forEach(byte => buffer.writeUInt8(byte, offset));
offset += d.length;
} catch (err) {
reject(err);
return;
}
if (offset === length) {
resolve(buffer);
this._port.removeListener("data", handler);
};
};
this._port.on("data", handler);
});
}
The above function takes a list of uint8s, and an expected amount of data to get back, returns a promise. We write the data, and then set ourselves up as the "data" event handler. We use that to read until we get the amount of data we expect, then resolve the promise, remove ourselves as a "data" listener (this is important, otherwise you'll stack handlers!), and finish.
This code is very specific to my needs, and won't handle cases other than very strict send/receive pairs with known parameters, but it might give you an idea to start with.