I am trying to use http.request on node using the koajs framework. Is there a way I can utilize it as show below?
var http = require('http');
var result = yield http.request(options);
Presumably the problem you're facing is that http.request takes a callback rather than returning a promise, so you can't yield it from koa. You need to wrap http.request in a function that returns a promise and hook the promise resolve into the callback, while also hooking the promise reject into the error handler.
function request(opts, body) {
return new Promise((resolve, reject) => {
body.pipe(http.request(opts, resolve))
.on('error', reject);
});
}
...later in your koa function...
var response = yield request(opts, body);
There are so many possible variations on this that I couldn't come close to listing them all, but that's the basic idea :)
Related
I have a large number of images that I want to download. I'm using the request-promise package to download the images from their URL(s). Since there were a large number of images, the server was getting overloaded and the downloads would hang or get corrupted, so I decided to use the promise-limit library to set the concurrency limit. Here is my code :
const fs = require('fs');
const request = require('request-promise');
const promiseLimit = require('promise-limit');
var limit = promiseLimit(30);
async function download(data) {
console.log(data);
return Promise.all(data.map( (obj) => {
return limit(() => downloadRequest(obj))
})).then(() => {
console.log("All images downloaded.")
})
function downloadRequest(obj) {
var img = obj.dest;
var url = obj.url;
var req = request(url);
req.pipe(fs.createWriteStream(img));
}
I replicated the sample exactly as is given in the github page, but the method returns without ever having the Promise.all() fulfilled. I do not understand what I am doing wrong, I thought that Promise.all() will definitely wait till it resolves all promises.
In the backend, this call is used as
...
.then((data) => {
return downloader.download(data);
})
.then(() => {
var filename = "Sample.pdf";
// Pages.json is written early in the chain, contains URL and corresponding image paths
return generator.generate('./Pages.json', filename);
});
Does this mean NodeJS is already trying to generate the file out of pages.json? How can I make this part of the code synchronous to download?
Your function downloadRequest() does not return a promise. It must return a promise that is tied to the asynchronous operation in contains such that the promise is resolved when that asynchronous operation is complete or rejected when that asynchronous operation has an error. Only when it does that can the limit() package properly do its job.
Since you're using a stream and piping it in downloadRequest(), you will have to manually construct a promise and then monitor the various events in the stream to know when it's done or has an error so you can resolve or reject that promise.
Here's an idea how to make downloadRequest() properly return a promise:
function downloadRequest(obj) {
return new Promise((resolve, reject) => {
const img = obj.dest;
const url = obj.url;
const req = request(url);
req.on('error', reject);
const ws = fs.createWriteStream(img);
ws.on('error', reject);
req.pipe(ws).on('finish', resolve);
});
}
And, it is now recommended to use the pipeline() function instead of .pipe() because it does more complete cleanup in error conditions and there is also a promise version of that built-in:
const { pipeline } = require('stream/promises');
function downloadRequest(obj) {
return pipeline(request(obj.url), fs.createWriteStream(obj.dest));
}
P.S. In case you didn't know, the request() library has been deprecated and it is recommended that you not use it in new projects any more. There is a list of alternative libraries to choose from here, all of which also have built-in promise support. I've looked at the various choices, tried several and decided I'm using got() in my work.
I am using request module in my NodeJS application, for making server-to-server API calls. I am making the API call like this:
request(options, function (error, response, body) {
if( error ){
// return error response
}
// return success response here
});
For some reason, I need to not use this asynchronous way of making call, but do it synchronously. So, is there any way to make this call in synchronous manner. I tried and found some other modules for this, but I need to use this same module.
Thanks
No you cannot not. Request will return you promise and you have to handle it somewhere using .then() or calling the function with async/await pattern.
Because an HTTP request is asynchronous by nature, you cannot do it synchronously. However, you can use ES6+ Promises and async/await like so:
// First, encapsulate into a Promise
const doRequest = () => new Promise((resolve, reject) => request(options, function (error, response, body) {
if( error ){
reject(error)
}
resolve(response)
});
// And then, use async/await
const x = 1 + 1
const response = await myRequest()
console.log(response)
More info: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Promise
As indicated by #Errorname, promises are probably what you are looking for. Instead of writing the code by hand, you could also use the package request-promise: https://www.npmjs.com/package/request-promise
If you want a strongly-typed, synchronous client, you can try out ts-sync-request.
NPM: https://www.npmjs.com/package/ts-sync-request
This library is a wrapper around sync-request.
You can attach a header & make a request like below:
import { SyncRequestClient } from 'ts-sync-request/dist'
let url = "http://someurl.com";
let response = new SyncRequestClient()
.addHeader("content-type", "application/x-www-form-urlencoded")
.post<string, MyResponseModel>(url, "city=Dubai");
I have a Meteor.method() that server side returns a promise from oracledb. Client side I have:
Meteor.call('myMethod', (error, result) => {
result.then() // err -> no .then() method?,
});
So what is result? It does not have a .then() method, so it is not a promise?
Meteor does not "send" the promise to the client.
The server returns a result value to the client (which triggers the callback) once the promise is resolved (or rejected) on the server, and not at the moment the promise is returned from the method itself (unless it is already settled when returned).
You can also use async/await to simplify the code.
Here is a blog post with more details about the using asynchronous code in methods.
Note:
The value sent from the server is serialized using EJSON. Object methods, getters, etc. are stripped from it, unless you create a custom serializer. In some cases, serialization might even fail (I think it happened with certain moment objects) and result in undefined being returned.
Meteor is not using promises by default, however, you can wrap your Meteor.calls into a promise function as below
const callWithPromise = (method, myParameters) => {
return new Promise((resolve, reject) => {
Meteor.call(method, myParameters, (err, res) => {
if (err) reject('Something went wrong');
resolve(res);
});
});
}
(async function() {
const myValue1 = await callWithPromise('myMethod1', someParameters);
const myValue2 = await callWithPromise('myMethod2', myValue1);
})();
Sample code has copied from Meteor forum.
Also, this topic gives you a better insight in taking advantages of Aysnc/Await syntax or Promises in Meteor calls.
I would like to know what do you think about this kind of extension for ES6 Promise (https://developer.mozilla.org/en/docs/Web/JavaScript/Reference/Global_Objects/Promise):
Promise.create = function() {
var deferred;
var promise = new Promise(function (resolve, reject) {
deferred = {
resolve: resolve,
reject: reject
};
});
promise.deferred = deferred;
return promise;
}
Like that it avoid using the first callback to get cleaner code:
var requestsdeferred = Promise.create();
obj.myFunctionWithCallback(function(){
obj.mySecondFunctionWithCallback(function(){
requestsdeferred.resolve('all done!');
});
});
requestsdeferred.then(function(result) {
});
instead of:
var p = new Promise(function(resolve, reject){
obj.myFunctionWithCallback(function(){
obj.mySecondFunctionWithCallback(function(){
resolve('all done!');
});
});
});
p.then(function(){
});
Which need a callback.
What do you think ?
Neither is the correct/regular usage of a promise. The usual way to use a promise would be:
ajax.get('/get').then(function(){
return ajax.get('/cart');
}).then(function(){
alert('all done!');
});
Promises chain, you can return a promise from a then handler and it will cause the returned promise from the then to wait for its completion and assume its state.
Of course, unless the promises depend you can (and likely should) execute them concurrently:
Promise.all([ajax.get("/get"), ajax.get("/cart")]).then(function(results){
// both done here, concurrently
});
Avoid the explicit construction anti-pattern, no need to create a deferred. The reason the API does not look the way you describe it is so that if you throw synchronously it will get converted to a rejection and you won't have to add both a .catch and a catch (e){ handler to your code which is error prone.
It's just an obfuscation of the ES6 standard, and I don't see much benefit to your addition. It's just a layer of abstraction that isn't really needed.
I'd suggest a different approach that plays nice with ES6 promises...
The following is correct, but irrelevant (see comments) :
Any "thenable" can be turned into a standard Promise with Promise.resolve, so if your ajax is (for instance) created with jQuery's $.ajax, you might:
var prom1 = Promise.resolve(ajax.get('/get'));
var prom2 = Promise.resolve(ajax.get('/cart'));
We can create these together (so the requests run concurrently), then wait for all of the promises to complete with Promise.all:
Promise.all([req1, req2])
.then(function(vals){
//and get the results in the callback
var getData = vals[0];
var cartData = vals[1];
});
Recently I work on a new project and this project use JavaScript callbacks in nodejs. Now we use KOA but the problem happens when we try to use ES6 Generators and callbacks.
//Calback function
function load(callback){
result = null;
//Do something with xmla4js and ajax
callback(result);
return result;
}
Now in KOA I need to call load and response json to client so i use this code below :
router= require('koa-router');
app = koa();
app.use(router(app));
app.get('load',loadjson);
function *loadJson(){
var that = this;
load(function(result){
that.body = result;
});
}
but i get this error :
_http_outgoing.js:331
throw new Error('Can\'t set headers after they are sent.');
^
Error: Can't set headers after they are sent.
at ServerResponse.OutgoingMessage.setHeader (_http_outgoing.js:331:11)
at Object.module.exports.set (G:\NAP\node_modules\koa\lib\response.js:396:16)
at Object.length (G:\NAP\node_modules\koa\lib\response.js:178:10)
at Object.body (G:\NAP\node_modules\koa\lib\response.js:149:19)
at Object.body (G:\NAP\node_modules\koa\node_modules\delegates\index.js:91:31)
at G:\NAP\Server\OlapServer\index.js:40:19
at G:\NAP\Server\OlapServer\OLAPSchemaProvider.js:1599:9
at _LoadCubes.xmlaRequest.success (G:\NAP\Server\OlapServer\OLAPSchemaProvider.js:1107:13)
at Object.Xmla._requestSuccess (G:\NAP\node_modules\xmla4js\src\Xmla.js:2113:50)
at Object.ajaxOptions.complete (G:\NAP\node_modules\xmla4js\src\Xmla.js:2024:34)
Just to clarify things, let's write your callback as
//Calback function
function load(callback){
setTimeout(function() {
var result = JSON.stringify({ 'my': 'json'});
callback(/* error: */ null, result);
}, 500);
}
in Koa world, this is called a thunk, meaning that it is an asynchronous function that takes only one argument: a callback with the prototype (err, res). you can check https://github.com/visionmedia/node-thunkify for a better explanation.
now you have to write your middleware with
function *loadJson(){
this.type = 'application/json';
this.body = yield load;
}
this is mainly because KOA is generator based, if your on the top of the middleware it does not support callbacks. so its not waiting for the function to finish. best solution would be to convert your function into a promise. promise works great with KOA.
I had a very similar problem using braintree (regular callbacks) and koa. Based on your code, the only change I needed to do was with the load function and how it was called.
router = require('koa-router');
app = koa();
app.use(router(app));
app.get('/load',loadjson);
function *loadJson(){
this.body = yield load;
}
// Callback function
function load(callback) {
// Prepare some data with xmla4js and ajax
whatever_inputs = {...};
final_method(whatever_inputs, callback);
}
The explanation by Jerome and Evan above is absolutely correct, and thunkify looks like a suitable process for automatically doing it.
While thunks were a nice idea, in my view a Promise is a better long-term approach. Many libraries are already moving to promises for async instead of the old node standard callback(err, data), and they're dead-simple to wrap around any async code to make a promise. Other devs will have experiences with Promises and naturally understand your code, while most would have to look up what a "thunk" is.
e.g. here I am wrapping the not-yet-promise-based jsdom up in a promise, so I can yield it in my koa generator.
const jsdom = require('node-jsdom');
const koa = require('koa');
const app = koa();
app.use(function *() {
this.body = yield new Promise((resolve, reject) => jsdom.env({
url: `http://example.org${this.url}`,
done(errors, { document }) {
if (errors) reject(errors.message);
resolve(`<html>${document.body.outerHTML}</html>`);
},
}));
});
app.listen(2112);
Semantically, promises and generators go hand-in-hand to really clarify async code. A generator can be re-entered many times and yield several values, while a promise means "I promise I'll have some data for you later". Combined, you get one of the most useful things about Koa: the ability to yield both promises and synchronous values.
edit: here's your original example wrapped with a Promise to return:
const router = require('koa-router');
const { load } = require('some-other-lib');
const app = koa();
app.use(router(app));
app.get('load', loadjson);
function* loadJson() {
this.body = yield new Promise(resolve => {
load(result => resolve(result));
});
}
To bypass Koa's built-in response handling, you may explicitly set this.respond = false;. Use this if you want to write to the raw res object instead of letting Koa handle the response for you.
Header is already written by built-in response handling before your callback is invoked.