I am trying to writ data to Amazon Kinesis using AWS SDK for NodeJS. I am using putRecords method which I want to use with async/await. I need to preform few other steps once the records are written to Kinesis. I want this to be a synchronous process, so I cannot use just callbacks.
I have tried few different ways using promise, async/await but none of them seem to be working.
With Async/Await:
let response = await kinesisPutRecords({
Records: //records,
StreamName: //streamName
});
async function kinesisPutRecords(recordsParams){
return new Promise((resolve, reject)=>{
kinesisClient.putRecords(recordsParams,function (err, data) {
if (err)
reject(err);
resolve();
}
});
}
With Promise
let response = await kinesisClient.putRecords(recordsParams).promise();
console.log(response);
This approaches are used inside a loop, the loop continues without waiting for this call to finish.
These two approaches work fine with S3.getObject();
Any help is appreciated. Thank you.
Update 1:
I found the problem with the above code. Its just missing try...catch block is causing the issue. In my full code, I still have this issue but I was able to find the root cause.
I am using csv tarnsform with NodeJS streams. Inside the transform function, await is not working. May be because of the fact that stream is async and do not pause?
Code close to my scenario:
const transformer = transform(function (record) {
//add records to an array until a threshold is met and then push to kinesis.
let response = await kinesisPutRecords();
console.log(response);
}, function(err, output){
//logic to push remaining records
});
fileReadStream
.pipe(gunzip) //unzip the file
.pipe(csvParser) //to parse the csv
.pipe(transformer);
async function kinesisPutRecords(recordsParams){
return new Promise((resolve, reject)=>{
try{
kinesisClient.putRecords(recordsParams,function (err, data) {
if (err)
reject(err);
resolve();
});
}
catch(exception ex){
reject();
}
});
}
Related
I want to make an openwhisk action that does something really simple: performs a find query in a mongodb and returns the result. However, since I am new in those technologies, I cannot find a solution, cause I am always recieving no results. I have connected openwhisk with the mongodb correctly. Can anyone help with a code example?
My code so far is this:
function main(){
var MongoClient = require('mongodb').MongoClient
var url = 'mongodb://192.168.1.14:27017/'
MongoClient.connect(url, (err, db) => {
db.db('yelp').collection('Review').find({stars:5}).limit(100).toArray().then((docs) => {
return docs;
db.close();
}).catch((err) => {
console.log(err.stack);
});
})
}
I am recieving null as a result. Any suggestions?
You need to use something like Promises or async/await to deal with async, and wait until the respond is back from the DB to end the action execution
See this question on how to use promises with mongodb client in nodejs
How to use MongoDB with promises in Node.js?
I'm new to nodejs and javascript in general. I believe this is an issue with the scope that I'm not understanding.
Given this example:
...
...
if (url == '/'){
var request = require('request');
var body_text = "";
request('http://www.google.com', function (error, response, body) {
console.log('error:', error);
console.log('statusCode:', response && response.statusCode);
console.log('body:', body);
body_text=body;
});
console.log('This is the body:', body_text)
//I need the value of body returned from the request here..
}
//OUTPUT
This is the body: undefined
I need to be able to get the body from response back and then do some manipulation and I do not want to do all the implementation within the request function. Of course, if I move the log line into:
request( function { //here })
It works. But I need to return the body in some way outside the request. Any help would be appreciated.
You can't do that with callbacks because this will works asynchronously.
Work with callbacks is kind of normal in JS. But you can do better with Promises.
You can use the request-promise-native to do what you want with async/await.
async function requestFromClient(req, res) {
const request = require('request-promise-native');
const body_text = await request('http://www.google.com').catch((err) => {
// always use catches to log errors or you will be lost
})
if (!body_text) {
// sometimes you won't have a body and one of this case is when you get a request error
}
console.log('This is the body:', body_text)
//I need the value of body returned from the request here..
}
As you see, you always must be in a function scope to use the async/await in promises.
Recommendations:
JS the right way
ES6 Fetures
JS clean coding
More best practices...
Using promises
We have a node server which doesn't handle post requests properly when they are made asynchronously. When the requests are made synchronously, it handles them fine.
There is a node api server and to mimic a client, there is node script which makes a post request to the server.
While making a single post request or post requests in a loop synchronously, everything works as expected.
While making asynchronous post requests in a loop to the server, the code doesn't execute properly.
Here is the code on the server side. This method is called from router.post() method.
async insert(params) {
let account = new Account();
try {
let totalLicenses = await this.getLicenses(params);
if (totalLicenses === 0) throw new AccountError('NO_AVAILABLE_LICENSE');
let accountResponse = await account.insert(params);
let useLicense = await license.use(accountResponse, params);
/*
Do other account setup stuff here
*/
return accountResponse;
} catch(err) {
throw err;
}
}
getLicenses(params) is a async function that prepares the sql query and awaits the response from a queryDb method which wraps the callback in a promise. Here is the code:
getLicense(params) {
let vals = [...arguments],
sql = 'my sql query';
try {
return await queryDb(sql, val);
} catch (err) {
throw new Error()
}
}
We are using mysql package to talk to db. This is what queryDb method looks like.
queryDb(query, vals) {
return new Promise( (resolve, reject) => {
connection.query(query, vals, (err, results, fields) => {
if (err) reject(err);
resolve(results);
}
}
}
While making asynchronous post requests in a loop to the server, the insert method above executes this.getLicenses(params) method for the requests and then calls the account.insert(params) for the requests and then license.use(accountResponse, params) for the requests.
This becomes a problem when let's say a client has 3 licenses available and they send 5 asynchronous post requests. Technically, it should throw error for the last 2 requests but that is not the case. What ends up happening is, it inserts all 5 accounts since it calls this.getLicenses(params) for all 5 requests before proceeding to insert the account.
Any help is appreciated!
I am currently using node-serialport module for serial port communication. I will send a command ATEC and it will respond with ECHO.
However, this process of sending and receiving data is async(after i send the data, i will not know when the data will arrive in the data event), the example code is below:
//Register the data event from the serial port
port.on('data', (data) => {
console.log(data);
});
//Send data using serialport
port.write('ATEC');
Is there anyway I could write it in this way?
//When i send the command, I could receive the data
port.write('ATEC').then((data)=> {
console.log(data);
});
Is this possible to achieve?
In http communication using request client, we could do something like
request.get('http:\\google.com')
.on('response', (res) => {
console.log(res);
});
I want to replicate the same behaviour using serialport
I wrap a promise in the serial data receive
function sendSync(port, src) {
return new Promise((resolve, reject) => {
port.write(src);
port.once('data', (data) => {
resolve(data.toString());
});
port.once('error', (err) => {
reject(err);
});
});
}
Please take note, the event is using once instead of on to prevent event from stacking (please check the comments below for more information - thanks #DKebler for spotting it)
Then, I could write the code in sync as below
sendSync(port, 'AThello\n').then((data) => {
//receive data
});
sendSync(port, 'ATecho\n').then((data) => {
//receive data
});
or I could use a generator, using co package
co(function* () {
const echo = yield sendSync(port, 'echo\n');
const hello = yield sendSync(port, 'hello 123\n');
return [echo, hello]
}).then((result) => {
console.log(result)
}).catch((err) => {
console.error(err);
})
We have a similar problem in a project I'm working on. Needed a synchronous send/receive loop for serial, and the serialport package makes that kinda weird.
Our solution is to make some sort of queue of functions/promises/generators/etc (depends on your architecture) that the serial port "data" event services. Every time you write something, put a function/promise/etc into the queue.
Let's assume you're just throwing functions into the queue. When the "data" event is fired, it sends the currently aggregated receive buffer as a parameter into the first element of the queue, which can see if it contains all of the data it needs, and if so, does something with it, and removes itself from the queue somehow.
This allows you to handle multiple different kinds of architecture (callback/promise/coroutine/etc) with the same basic mechanism.
As an added bonus: If you have full control of both sides of the protocol, you can add a "\n" to the end of those strings and then use serialport's "readline" parser, so you'll only get data events on whole strings. Might make things a bit easier than constantly checking input validity if it comes in pieces.
Update:
And now that code has been finished and tested (see the ET312 module in http://github.com/metafetish/buttshock-js), here's how I do it:
function writeAndExpect(data, length) {
return new Promise((resolve, reject) => {
const buffer = new Buffer(length);
this._port.write(data, (error) => {
if (error) {
reject(error);
return;
}
});
let offset = 0;
let handler = (d) => {
try {
Uint8Array.from(d).forEach(byte => buffer.writeUInt8(byte, offset));
offset += d.length;
} catch (err) {
reject(err);
return;
}
if (offset === length) {
resolve(buffer);
this._port.removeListener("data", handler);
};
};
this._port.on("data", handler);
});
}
The above function takes a list of uint8s, and an expected amount of data to get back, returns a promise. We write the data, and then set ourselves up as the "data" event handler. We use that to read until we get the amount of data we expect, then resolve the promise, remove ourselves as a "data" listener (this is important, otherwise you'll stack handlers!), and finish.
This code is very specific to my needs, and won't handle cases other than very strict send/receive pairs with known parameters, but it might give you an idea to start with.
I am building a NodeJS server using Express4. I use this server as a middleman between frontend angular app and 3rd party API.
I created a certain path that my frontend app requests and I wish on that path to call the API multiple times and merge all of the responses and then send the resulting response.
I am not sure how to do this as I need to wait until each API call is finished.
Example code:
app.post('/SomePath', function(req, res) {
var merged = [];
for (var i in req.body.object) {
// APIObject.sendRequest uses superagent module to handle requests and responses
APIObject.sendRequest(req.body.object[i], function(err, result) {
merged.push(result);
});
}
// After all is done send result
res.send(merged);
});
As you can see Im calling the API within a loop depending on how many APIObject.sendRequest I received within request.
How can I send a response after all is done and the API responses are merged?
Thank you.
Check out this answer, it uses the Async module to make a few requests at the same time and then invokes a callback when they are all finished.
As per #sean's answer, I believe each would fit better than map.
It would then look something like this:
var async = require('async');
async.each(req.body.object, function(item, callback) {
APIObject.sendRequest(item, function(err, result)) {
if (err)
callback(err);
else
{
merged.push(result);
callback();
}
}
}, function(err) {
if (err)
res.sendStatus(500); //Example
else
res.send(merged);
});
First of all, you can't do an async method in a loop, that's not correct.
You can use the async module's map function.
app.post('/SomePath', function(req, res) {
async.map(req.body.object, APIObject.sendRequest, function(err, result) {
if(err) {
res.status(500).send('Something broke!');
return;
}
res.send(result);
});
});