I'm trying to wrap http.get with a Promise. Here's what I've got:
import Http from 'http';
export function get(url) {
let req;
return new Promise((resolve, reject) => {
req = Http.get(url, async res => {
if(res.statusCode !== 200) {
return reject(new Error(`Request failed, got status ${res.statusCode}`));
}
let contentLengthStr = res.headers['content-length'];
if(contentLengthStr) {
let contentLength = parseInt(contentLengthStr, 10);
if(contentLength >= 0 && contentLength <= 2*1024**3) {
let buf = Buffer.allocUnsafe(contentLength);
let offset = 0;
res.on('data', chunk => {
if(chunk.length + offset > contentLength) {
return reject(new Error(`Received too much data, expected ${contentLength} bytes`));
}
chunk.copy(buf, offset);
offset += chunk.length;
});
res.on('end', () => {
if(offset === contentLength) {
resolve(buf);
} else {
return reject(new Error(`Expected ${contentLength} bytes, received ${offset}`));
}
})
} else {
return reject(new Error(`Bad Content-Length header: ${contentLengthStr}`));
}
} else {
return reject(new Error(`Missing Content-Length header not supported`));
}
});
}).catch(err => {
req.abort();
throw err;
})
}
It seems to work OK, but it feels a bit clunky.
Firstly, async/await appear to be of no help here. I can't throw nor return inside of res.on('end'. Returning just returns out of the end callback, but there's no real way for me to break out of the Http.get(url, res => { function from inside there. throwing doesn't "bubble up" to the Promise I created either because the data/end events don't fire synchronously. I have to call reject manually.
The part that really bothers me though is that if the server sends me more data than they said they were going to send me via the Content-Length header, I want to abort the request and stop listening for events. To do that I've rejected the Promise, and then immediately catch it so that I can abort the request, and then I rethrow the error so that the caller can handler it. To do this I have to declare the req variable above the Promise, and then initialise it inside the Promise, so that I can access it after the Promise.
The flow of everything just feels really clunky. Is there a nicer way to write this? (Assume I have available all ES6/ES2017+/next features)
Here is how I would approach the problem:
"promisify' Http.get, meaning converting the API to use promises.
Separate the stream logic into its own separate promise.
Instrument the error logic in a clear flat async function.
This looks something like the following:
import Http from 'http';
// promisify Http.get
const http = (url, abort) => new Promise((resolve, reject) => {
let req = Http.get(url, res => {
if(res.statusCode !== 200) reject(new Error(`Request failed, got status ${res.statusCode}`));
else resolve(res);
});
abort(() => req.abort()); // allow aborting via passed in function
});
export async function get(url) {
let abort = null;
let res = await http(url, (onabort) => { abort = onabort; });
let contentLengthStr = res.headers['content-length'];
if(!contentLengthStr) new Error(`Missing Content-Length header not supported`);
if(contentLengthStr < 0 || contentLengthStr > (2*1024**3 +1)) {
throw new Error(`Bad Content-Length header: ${contentLengthStr}`);
}
try {
return await read(res, contentLength);
} catch (e) {
abort();
throw e;
}
}
function read(res, contentLength) {
let buf = Buffer.allocUnsafe(contentLength), offset = 0;
return new Promise((resolve, reject) => {
res.on('data', chunk => {
if(chunk.length + offset > contentLength) {
return reject(new Error(`Received too much data, expected ${contentLength} bytes`));
}
chunk.copy(buf, offset);
offset += chunk.length;
});
res.on('end', () => {
if(offset === contentLength) resolve(buf);
else return reject(new Error(`Expected ${contentLength} bytes, received ${offset}`));
});
});
}
Technically this should be on code review, but it's a different audience.
First observation. Correct me if I am wrong, but HTTP.get does not document usage of a return value from the call back. So unless function code uses the await keyword, leaving the callback defined with the async keyword is confusing (you did mention it was of no help).
A second source of confusion for readers is usage of the return reject( ...... ); construct. reject doesn't return a value, and I am unaware of the return value of event listeners being used. So the the order can be reversed by placing the return statement (if actually needed) after the reject call.
Aborting the request can be done in scope of the promise executor by moving let req; back into the executor and explicitly calling .abort as required. Catching the promise should no longer be required. Refactored, the event emitter callbacks could, for example, look like
res.on('data', chunk => {
if(chunk.length + offset > contentLength) {
reject(new Error(`Received too much data, expected ${contentLength} bytes`));
req.abort();
} else {
chunk.copy(buf, offset);
offset += chunk.length;
}
});
res.on('end', () => {
if(offset === contentLength) {
resolve(buf);
} else {
reject( new Error(`Expected ${contentLength} bytes, received ${offset}`));
req.abort();
}
});
Which just leaves the last two return reject.... usages. You could either abort the request in each case, or set an error variable which is checked later:
req = Http.get(url, res => {
let error = null;
// ....
} else {
error = new Error(`Bad Content-Length header: ${contentLengthStr}`);
}
} else {
error = new Error(`Missing Content-Length header not supported`));
}
if( error) {
reject( error);
req.abort();
}
});
At this stage catching promise rejection and re-throwing the error should no longer be needed. Naturally this is untested but hopefully provides some useful feedback.
Related
NB: this javascript uses the openui5 libraries
attachOnceAsync: function(binding, callback) {
var args = [...arguments]
args.splice(0, 2)
return new Promise((resolve, reject) => {
var dataReceived = function(oEvent) {
if (typeof callback == 'function') {
if (callback.constructor.name === "AsyncFunction") {
callback(oEvent, ...args).then(() =>
resolve(oEvent)
).catch((err) => {
reject(err)
})
} else {
try {
callback(oEvent, ...args);
resolve(oEvent)
} catch (err) {
reject(err)
}
}
} else {
resolve(oEvent, ...args);
}
}
binding.attachEventOnce("dataReceived", dataReceived);
})
}
There aren't any Promise versions of events like "dataReceived" so this is my attempt to wrap one up.
In this case, callback is undefined (it shouldn't be, but that's a different issue). The line resolve(oEvent, ...args); is hit instead, only it never returns out of the await!
I changed that line to just resolve(oEvent) but still no joy.
Any ideas what I'm doing wrong?
Thanks
PS: I'll keep an eye on this question so I can offer any extra info required
here's how I call it:
handleAsync: async function(controller) {
/* Lots of setup */
var response = await fetch(/*redacted*/ )
if (response.status === 400) {
console.log(await response.json());
throw new BadRequestException();
}
if (response.status === 204){
/* more setup */
context = oPanel.getBindingContext("odata");
var updatePurchaseOrderDocTotalFunc = this.updatePurchaseOrderDocTotal
var callback = async function() {
var docTotalUpdate = await updatePurchaseOrderDocTotalFunc(oView, context, oModel, agreedPrice);
if (docTotalUpdate && docTotalUpdate.status === 204) {
await context.refresh();
}
}
//-- HERE --
await AsyncBinding.attachOnceAsync(binding,callback)
await context.refresh();
}
return true;
}
Ok - The answer was staring at me.
Sorry I wasn't able to provide a working snippet but the ui5 scripts are quite extensive.
I have several other functions that return a Promise so I can use the await/async approach, but with this one I forgot to include the action that causes the event to fire.
I renamed the function to show what it now does, and included the action that causes the event to fire:
RefreshAsync : function(context,binding, callback){
var args = [...arguments]
args.splice(0,2)
return new Promise((resolve,reject)=>{
var dataReceived = function(oEvent){
if(typeof callback == 'function' ){
if(callback.constructor.name==="AsyncFunction"){
callback(oEvent, ...args).then(()=>{
resolve(oEvent)
}).catch((err)=>{
reject(err)
})
}else{
try{
callback(oEvent, ...args);
resolve(oEvent)
}catch(err){
reject(err)
}
}
}else{
resolve([oEvent, ..args])
}
}
binding.attachEventOnce("dataReceived", dataReceived);
//This is the action I missed
context.Refresh()
})
},
I have been trying to create an api like this where I tried different things like using array methods like map/filter/reduce where either I get pending promises or result returned before execution of api call.
So my doubt is -->
How do I get total number of drawn matches of all pages ( so I need to add data.total from all pages).
How to better understand this behaviour.
async function getNumDraws(year) {
const goals = [...Array(11).keys()];
let result = 0;
console.log(`before loop ${new Date()}`);
for(let goal of goals){
console.log(`in loop before await ${new Date()}`);
await require('https').get(`https://jsonmock.hackerrank.com/api/football_matches?year=${year}&team1goals=${goal}&team2goals=${goal}`,res=>{
let data="";
res.on('data', (chunk) => {
data += chunk;
});
// The whole res has been received. Print out the result.
res.on('end', () => {
data=JSON.parse(data);
console.log(result,data.total)
result= result + data.total;
});
})
console.log(`in loop after await ${new Date()}`);
}
console.log(`after loop ${new Date()}`);
return result;
}
console.log(getNumDraws(2011));
https.get is a callbacked function so await won't work. You should promisify it first like they did in this other SO question;
const https = require("https"); // only require this once
const getJSONAsync = url => new Promise((resolve, reject) => { // define a function getJSONAsync which returns a Promise that wraps the https.get call, getJSONAsync is awaitable
let req = https.get(url, res => { // make the https.get request
if(res.statusCode < 200 || res.statusCode >= 300) { // if the status code is an error one
return reject(new Error('statusCode=' + res.statusCode)); // reject and skip the rest
}
let body = []; // otherwise accumulate..
res.on('data', chunk => body.push(chunk)); // ..the data
res.on('end', () => { // on end
try {
body = JSON.parse(Buffer.concat(body).toString()); // try to JSON.parse the data
} catch(e) {
reject(e); // reject if an error occurs
}
resolve(body); // resolve the parsed json object otherwise
});
});
req.on("error", error => reject(error)); // reject if the request fails too (if something went wrong before the request is sent for example)
});
async function getNumDraws(year) {
let result = 0;
for(let goal = 0; goal < 11; goal++) {
let data = await getJSONAsync(`https://jsonmock.hackerrank.com/api/football_matches?year=${year}&team1goals=${goal}&team2goals=${goal}`);
result += data.total;
}
return result;
}
Note: getJSONAsync is not specific to getNumDraws, you can use it somewhere else if you need it, and since it returns a Promise you can either await it like getNumDraws does or use it with then/catch blocks like so:
getJSONAsync("url")
.then(data => {
// data is the parsed json returned by the request
})
.catch(error => {
// the error message if something fails
})
Hopefully someone can point me to the right direction. I read up on waiting for functions to complete before continuing and I resolved myself to using await/async but I am just stuck now.
I tried to get the Async/Await process to work, tried to inject the await in various locations, with adjusting the functions to be async, but i can not get the PSA_Resultbody to return to the original request. Any pointers would be appreciated.
Thank you,
CE
PSA_Resultbody = ProcessPSAAPI(xmlpackage, PSA_Action);
console.log("3 - Returned data:" + PSA_Resultbody);
calls the below:
async function ProcessPSAAPI(xmlpackage, PSA_Action) { //psa action is part of the options
var options = {...};
var req = https.request(options, function (res) {
var chunks = [];
res.on("data", function (chunk) {
chunks.push(chunk);
});
res.on("end", function (chunk) {
var body = Buffer.concat(chunks);
console.log('0 - Start '+body.toString());
if(res.statusCode != 200) {
PSA_Resultcode = "Error: " +res.statusCode +" - "+ res.statusMessage;
} else {
PSA_Resultcode = "Success: " +res.statusCode +" - "+ res.statusMessage;
PSA_Resultbody = ParseResults(body.toString()); //parse the results for later use --SCRIPT NEEDS TO WAIT FOR RESULTBODY TO COMPLETE
console.log("1 -PSA_Resultbody as part of RES = "+PSA_Resultbody);
}
});
res.on("error", function (error) {
console.error(error);
PSA_Resultcode = res.statusCode +" - "+ res.statusMessage;
});
});
console.log('2 -RESULT BODY BEFORE SENDING BACK TO INITIATING FUNCTION: '+PSA_Resultbody);
req.write(xmlpackage);
req.end();
return PSA_Resultbody;
Based on the above, my console log order is: 3,2,0,1 in stead of 0,1,2,3.
0 and 1 will have the correct data, so the API Call does work, but 2 will be "undefined" and should have the same data that is in 1.
There's no way to await an event emitter, so using async in this case isn't going to be useful. You cannot "return" from inside an event either.
The solution here is to return a new custom promise and to use resolve() inside of the "end" event of your emitter.
It will look something like this:
function ProcessPSAAPI(xmlpackage, PSA_Action) {
return new Promise( (resolve, reject) => {
// other code
res.on("end", function (chunk) {
// other code
resolve(PSA_Resultbody);
});
res.on("error", function (error) {
// other code
reject(error);
});
});
}
Here's a quick tutorial on creating your own promises, which I've written to simplify comprehension of the subject (official docs are somewhat dry and complex imho).
I did not change your code. I just put the appropriate promise structure in to get you started. This should really be a lesson in promises. async await is a shorthand promise structure. A Promise is one way you wait on code. It can be thought of as an array of callbacks that will be executed when the Promise is resolved.
A simple promise works like this:
const myPromise = new Promise(function(resolve, reject) {
/* Your logic goes in here. It can be anything.
* But the important part to remember is that when you have success, resolve it.
* When you have a failure, reject it.
*/
someCallBackPattern(function(error, data) {
if(error) {
reject(error);
} else {
resolve(data);
}
});
});
// To get the data out you use 'then', and 'catch'. then has two arguments.
myPromise.then(function(data) {
// The first argument is the result from resolve.
}, function(err) {
// The second argument is the result from reject.
}).catch((err) => {
// you can also get to the error from the catch callback
});
This is kinda messy and complex. So there is async await.
async function() {
try {
const result = await myFunctionThatReturnsAPromise();
// result is the resolved data
} catch (err) {
// err is the rejected Error
}
}
function myFunctionThatReturnsAPromise() {
return new Promise((resolve, reject) => {
// your code
})
}
And thats how it works.
async function someFunction () { // You can not wait on results unless you are in an await function
PSA_Resultbody = await ProcessPSAAPI(xmlpackage, PSA_Action); // await on your results.
console.log("3 - Returned data:" + PSA_Resultbody);
}
function ProcessPSAAPI(xmlpackage, PSA_Action) { // This does not need to be async. Unless you are awaiting in it.
return new Promise((resolve, reject) => { // async await is a shorthand promise structure. Although you do not need to use promises. It really helps to get the structure correct.
var options = {...};
var req = https.request(options, function (res) {
var chunks = [];
res.on("data", function (chunk) {
chunks.push(chunk);
});
res.on("end", function (chunk) {
var body = Buffer.concat(chunks);
console.log('0 - Start '+body.toString());
if(res.statusCode != 200) {
PSA_Resultcode = "Error: " +res.statusCode +" - "+ res.statusMessage;
reject(new Error(PSA_Resultcode)); // Reject you errors
} else {
PSA_Resultcode = "Success: " +res.statusCode +" - "+ res.statusMessage;
PSA_Resultbody = ParseResults(body.toString()); //parse the results for later use --SCRIPT NEEDS TO WAIT FOR RESULTBODY TO COMPLETE
console.log("1 -PSA_Resultbody as part of RES = "+PSA_Resultbody);
resolve(PSA_Resultbody); // Resolve your result
}
});
res.on("error", function (error) {
console.error(error);
PSA_Resultcode = res.statusCode +" - "+ res.statusMessage;
reject(new Error(PSA_Resultcode)); // Reject you errors
});
});
console.log('2 -RESULT BODY BEFORE SENDING BACK TO INITIATING FUNCTION: '+PSA_Resultbody);
req.write(xmlpackage);
req.end();
})
}
I'm doing a recursive request with async/await whenever the received response has length === 0. The problem is that when some request returns the desired data, the resolve(data); part of the promise doesn't seem to work.
So, in my code, I have reached the point where I get to make multiple recursive calls and, finally, receive a response whose length is not 0.
Note: there are plenty of API keys published in Github if you want to test the code.
var apiKey = "yourApiKey";
var url = "https://api.nasa.gov/mars-photos/api/v1/rovers/curiosity/photos?sol=";
function requestData(url) {
return fetch(url).then(response => {
if(response.ok) {
return response.json().then(data => {
return Promise.resolve(data);
});
} else {
return Promise.reject(response.status);
}
});
}
function NasaRequest(sun, limit, frecuency) {
return new Promise(async (resolve, reject) => {
var data = await requestData(url + sun + "&api_key=" + apiKey);
if(data.photos.length === 0 && !limit) {
setTimeout(async () => {
console.log("Delay for next request (sun " + sun + "): ", frecuency);
return await NasaRequest(sun - 1, limit, frecuency);
}, frecuency);
} else {
console.log("Resolve data:", data); // Code acutally reaches this point
resolve(data); // But this doesn't seem to work
}
});
};
async function init() {
try {
const currentValue = await NasaRequest(2175, false, 2000);
console.log("currentValue:", currentValue); // I want to reach this point, but is like the promise never returns
}catch(err){
console.error(err);
}
}
init();
In that moment, I want to return the data in the response to the calling init() function, for what I use resolve(data);. But it doesn't seem to work.
What am I doing wrong?
The problem is on setTimeout. When you calling setTimeout it returns right away and implicitly return undefined. The subsequence return doesn't matter at that point. If all you want to do, is to pause, and then proceed try something like this
async function requestData(url) {
var response = await fetch(url);
if (response.ok) {
return response.json()
} else {
throw new Error(response.status);
}
}
function sleep(ms) {
return new Promise((resolve) => {
setTimeout(resolve, ms);
});
}
async function nasaRequest(sun, limit, freq) {
var data = await requestData(url + sun + "&api_key=" + apiKey);
if (data.photos.length === 0 && !limit) {
await sleep(freq);
console.log("Delay for next request (sun " + sun + "): ", freq);
return await nasaRequest(sun - 1, limit, freq);
} else {
console.log("Resolve data:", data);
return data;
}
};
async function init() {
try {
const currentValue = await nasaRequest(2175, false, 2000);
console.log("currentValue:", currentValue);
} catch (err) {
console.error(err);
}
}
init();
I added a simple sleep function to handle the pause. I also modified requestData (removed then and Promise parts).
Note that using this recursive approach you may run into stack overflow. To avoid that problem you can simply use a loop and check against your limit variable.
I am confused by this async behavior.
When token is false, refreshToken() function runs but the createTokenFile() doesn't wait for it to finish.
Shouldn't var tokenDate = new Date(token.expires); wait after callApiToken().then(function() {refreshToken();}) to finish before executing?
function createTokenFile() {
console.log("No token-file.json file found. " .red +
"Please complete for a new one." .red);
return callApiToken().then(function() {
refreshToken();
});
}
function checkExpiredToken() {
return new Promise(function(resolve, reject) {
if (!token) {
refreshToken();
}
var tokenDate = new Date(token.expires);
var utc = new Date().toUTCString();
var now = new Date(utc);
}
function refreshToken() {
try {
var tokenFile = path.join(__dirname, 'token-file.json');
console.log(tokenFile);
return token = JSON.parse(fs.readFileSync(tokenFile, {encoding: 'utf-8'}));
} catch (err) {
if (err.code !== 'ENOENT') {
throw err;
} else {
return createTokenFile();
}
}
}
UPDATED with refreshToken()
Shouldn't var tokenDate = new Date(token.expires); wait after callApiToken().then(function() {refreshToken();}) to finish before executing?
No - it's not in a .then() callback that would wait for the promise to resolve. It only waits until the promise is created - but the promise resolution (that you call "finish") is asynchronous. Notice that promises are not magic, they're just callbacks.
To fix your code,
in createTokenFile you need to return the refreshToken() from the then callback
checkExpiredToken should not use the Promise constructor
refreshToken should always return a promise
there's no reason why refreshToken would read the file synchronously
you shouldn't cache the token as a global variable containing the value
function createTokenFile() {
console.log("No token-file.json file found. " +
"Please complete for a new one.");
return callApiToken();
}
function checkExpiredToken() {
return (tokenPromise || refreshToken())
.then(function(token) {
var tokenDate = new Date(token.expires);
var utc = new Date().toUTCString();
var now = new Date();
});
}
function refreshToken() {
var tokenFile = path.join(__dirname, 'token-file.json');
console.log(tokenFile);
return tokenPromise = readFileAsync(tokenFile, {encoding: 'utf-8'}))
.then(JSON.parse)
.catch(function(err) {
if (err.code !== 'ENOENT') {
throw err;
} else {
return createTokenFile().then(refreshToken);
}
});
}
(where readFileAsync is a promisified version of fs.readFile)
Promise do not de-synchronize code. An function that is asynchronous will always be so. Therefore if refreshToken is asynchronous then it's use above will not wait for it to complete before moving on.
Your code sample leaves too much to the imagination (not to mention syntactically incorrect) so a better answer is not available. Perhaps try recreating the situation in jsbin.com or jsfiddle.net.