Where is the documentation on res.on('end' - javascript

Just cant find it! looking for documentation on the following on error, data, and end
return new Promise((resolve, reject) => {
https.get(setUrl(substr), (res) => {
let data = ''
res.on('data', (d) => {
data += d
});
res.on('end', () => resolve(data));
}).on('error', reject);
});

In the doc for https.get(), it says to go look at the doc for http.get() for the callback parameters.
There it says:
The callback is invoked with a single argument that is an instance of http.IncomingMessage.
And, if you then go look at the doc for http.IncomingMessage, you find that:
It implements the Readable Stream interface, as well as the following additional events, methods, and properties.
In that doc, you will find a description of the data and end events.
The https.get() itself returns an http.ClientRequest object and in the doc for http.request() which https.get() is derived from, it explains:
If any error is encountered during the request (be that with DNS resolution, TCP level errors, or actual HTTP parse errors) an 'error' event is emitted on the returned request object. As with all 'error' events, if no listeners are registered the error will be thrown.
Yes, it is work to follow all this. This is one of the aspects of object oriented design with lots of derived and common objects. You have to find the object being used or perhaps even the base object and then go look in its doc for find out how to use it.

Related

How to use Node Worker in Wakanda

Up until now for me the concept of a Node Worker has been one of those things that sounds interesting and I will find out about one day.
Well that day has come and I am asking for some help.
I need to call a web service and then process the data returned. I can call the service with an XMLHttpRequest, but then I have to get useful data out of it.
There is a nice node module that both calls the service and returns the data in a useful form with one call.
I can set up Node worker (in Wakanda) to do this and verify that it works.
My problem is handling the asynchronous call in the proxy.
The call to the node module looks like this:
myModule.getData(param, (err, data) => {
// data is an object containing everything I want.
// if I get data I want to return it to the proxy
// if I get an err I want to return the error to the proxy
});
So my wrapper code looks something like this:
function doSomething(param){
// call proxy with param
// wait for result
// return result
}
This all sounds like something I should know how to do. However I think I am struggling with too many new things and getting myself absolutely confused.
PS: I did try Threadify but couldn't work out how to get the worker to return the error it received.
I would really appreciate any help or pointers here.
If I am correctly understanding your issue, you cannot return a value from a function AFTER an asynchronous call completes. You need to handle the data within the myModule.getData callback.
If you would rather handle it in a calling function (like doSomething), you can use a promise to "return" a value.
function myProxy(param) {
return new Promise((resolve, reject) => {
myModule.getData(param, (err, data) => {
if (!err) { // or however you determine an error occurred.
resolve(data); // if I get data I want to return it to the proxy
} else {
reject(err); // if I get an err I want to return the error to the proxy
}
});
});
}
function doSomething(param){
myProxy(param).then(data => {
// Handle the data here.
}).catch(err => {
// Handle the error here.
});
}

Achieve error differentiation in Promises

Background
I have a REST API using MongoDB, Node.js and Express that makes a request to my NoSQL DB and depending on different results, I want to differentiate which error I send the customer.
Problem
The current version of my code has a generic error handler and always sends the same error message to the client:
api.post("/Surveys/", (req, res) => {
const surveyJSON = req.body;
const sender = replyFactory(res);
Survey.findOne({_id: surveyJSON.id})
.then(doc => {
if(doc !== null)
throw {reason: "ObjectRepeated"};
//do stuff
return new Survey(surveyJSON).save();
})
.then(() => sender.replySuccess("Object saved with success!"))
.catch(error => {
/*
* Here I don't know if:
* 1. The object is repeated
* 2. There was an error while saving (eg. validation failed)
* 3. Server had a hiccup (500)
*/
sender.replyBadRequest(error);
});
});
This is a problem, because the client will always get the same error message, no matter what and I need error differentiation!
Research
I found a possible solution, based on the division of logic and error/response handling:
Handling multiple catches in promise chain
However, I don't understand a few things:
I don't see how, at least in my example, I can separate the logic from the response. The response will depend on the logic after all!
I would like to avoid error sub-classing and hierarchy. First because I don't use bluebird, and I can't subclass the error class the answer suggests, and second because I don't want my code with a billion different error classes with brittle hierarchies that will change in the future.
My idea, that I don't really like either
With this structure, if I want error differentiation, the only thing I can do is to detect an error occurred, build an object with that information and then throw it:
.then(doc => {
if(doc === null)
throw {reason: "ObjectNotFound"};
//do stuff
return doc.save();
})
.catch(error => {
if(error.reason === "ObjectNotFound")
sendJsonResponse(res, 404, err);
else if(error.reason === "Something else ")
sendJsonResponse(/*you get the idea*/);
else //if we don't know the reasons, its because the server likely crashed
sendJsonResponse(res, 500, err);
});
I personally don't find this solution particularly attractive because it means I will have a huge if then else chain of statements in my catch block.
Also, as mentioned in the previous post, generic error handlers are usually frowned upon (and for a good reason imo).
Questions
How can I improve this code?
Objectives
When I started this thread, I had two objectives in mind:
Having error differentiation
Avoid an if then else of doom in a generic catcher
I have now come up with two radically distinct solutions, which I now post here, for future reference.
Solution 1: Generic error handler with Errors Object
This solution is based on the solution from #Marc Rohloff, however, instead of having an array of functions and looping through each one, I have an object with all the errors.
This approach is better because it is faster, and removes the need for the if validation, meaning you actually do less logic:
const errorHandlers = {
ObjectRepeated: function(error){
return { code: 400, error };
},
SomethingElse: function(error){
return { code: 499, error };
}
};
Survey.findOne({
_id: "bananasId"
})
.then(doc => {
//we dont want to add this object if we already have it
if (doc !== null)
throw { reason: "ObjectRepeated", error:"Object could not be inserted because it already exists."};
//saving empty object for demonstration purposes
return new Survey({}).save();
})
.then(() => console.log("Object saved with success!"))
.catch(error => {
respondToError(error);
});
const respondToError = error => {
const errorObj = errorHandlers[error.reason](error);
if (errorObj !== undefined)
console.log(`Failed with ${errorObj.code} and reason ${error.reason}: ${JSON.stringify(errorObj)}`);
else
//send default error Obj, server 500
console.log(`Generic fail message ${JSON.stringify(error)}`);
};
This solution achieves:
Partial error differentiation (I will explain why)
Avoids an if then else of doom.
This solution only has partial error differentiation. The reason for this is because you can only differentiate errors that you specifically build, via the throw {reaon: "reasonHere", error: "errorHere"} mechanism.
In this example, you would be able to know if the document already exists, but if there is an error saving the said document (lets say, a validation one) then it would be treated as "Generic" error and thrown as a 500.
To achieve full error differentiation with this, you would have to use the nested Promise anti pattern like the following:
.then(doc => {
//we dont want to add this object if we already have it
if (doc !== null)
throw { reason: "ObjectRepeated", error:"Object could not be inserted because it already exists." };
//saving empty object for demonstration purposes
return new Survey({}).save()
.then(() => {console.log("great success!");})
.catch(error => {throw {reason: "SomethingElse", error}});
})
It would work... But I see it as a best practice to avoid anti-patterns.
Solution 2: Using ECMA6 Generators via co.
This solution uses Generators via the library co. Meant to replace Promises in near future with a syntax similar to async/await this new feature allows you to write asynchronous code that reads like synchronous (well, almost).
To use it, you first need to install co, or something similar like ogen. I pretty much prefer co, so that is what i will be using here instead.
const requestHandler = function*() {
const survey = yield Survey.findOne({
_id: "bananasId"
});
if (survey !== null) {
console.log("use HTTP PUT instead!");
return;
}
try {
//saving empty object for demonstration purposes
yield(new Survey({}).save());
console.log("Saved Successfully !");
return;
}
catch (error) {
console.log(`Failed to save with error: ${error}`);
return;
}
};
co(requestHandler)
.then(() => {
console.log("finished!");
})
.catch(console.log);
The generator function requestHandler will yield all Promises to the library, which will resolve them and either return or throw accordingly.
Using this strategy, you effectively code like you were coding synchronous code (except for the use of yield).
I personally prefer this strategy because:
Your code is easy to read and it looks synchronous (while still have the advantages of asynchronous code).
You do not have to build and throw error objects every where, you can simply send the message immediately.
And, you can BREAK the code flow via return. This is not possible in a promise chain, as in those you have to force a throw (many times a meaningless one) and catch it to stop executing.
The generator function will only be executed once passed into the library co, which then returns a Promise, stating if the execution was successful or not.
This solution achieves:
error differentiation
avoids if then else hell and generalized catchers (although you will use try/catch in your code, and you still have access to a generalized catcher if you need one).
Using generators is, in my opinion, more flexible and makes for easier to read code. Not all cases are cases for generator usage (like mpj suggests in the video) but in this specific case, I believe it to be the best option.
Conclusion
Solution 1: good classical approach to the problem, but has issues inherent to promise chaining. You can overcome some of them by nesting promises, but that is an anti pattern and defeats their purpose.
Solution 2: more versatile, but requires a library and knowledge on how generators work. Furthermore, different libraries will have different behaviors, so you should be aware of that.
I think a good improvement would be creating an error utility method that takes the error message as a parameter, then does all your ifs to try to parse the error (logic that does have to happen somewhere) and returns a formatted error.
function errorFormatter(errMsg) {
var formattedErr = {
responseCode: 500,
msg: 'Internal Server Error'
};
switch (true) {
case errMsg.includes('ObjectNotFound'):
formattedErr.responseCode = 404;
formattedErr.msg = 'Resource not found';
break;
}
return formattedErr;
}

Split observable on error in RxJS

With promises, we can use a variant of .then to split up the chain when an error occurs. Here is an example using fetch
fetch('http://website.com').then(
// Perform some logic
(response) => response.json().then(({ answer }) => `Your answer: ${answer}`),
// Skip json parsing when an error occurs
(error) => 'An error occurred :(',
).then(console.log);
This allows me to skip the response processing logic and only respond to errors that were raised in the original fetch statement. Something similar in RxJS might look like this:
Observable.fromPromise(fetch('http://website.com'))
// if I put .catch here, the result will be piped into flatMap and map
.flatMap(response => response.json())
.map(({ answer }) => `Your answer: ${answer}`)
// if I put .catch here, errors thrown in flatMap and map will also be caught
.subscribe(console.log);
As the comments in the code state, I can't simply put a catch operator in as it doesn't have the same behaviour as my promise chain.
I know I can get it with custom operators involving materialising, or merging an error catching observable with this one, but it all seems like pretty major overkill. Is there a simple way to achieve the promise chain behaviour?
Actually, if I was in your situation I wouldn't be worried about catching errors from flatMap and map. When the source Observable throws an error then it'll be propagated to the observer. So I'd just use an error handler when calling subscribe (otherwise the error is rethrown):
.subscribe(console.log, err => console.log('error:', err));
Note that when an error occurs in the source Observable (a Promise in your case) that it's propagated as error notifications, not as standard next notifications. This means that flatMap() and map() won't have absolutely any effect on the error message. If you used catch() or materialize() that both operators (flatMap and map) would have to be able to handle this type of data (and not throw yet another error).
Anyway you can always use share() or publish() and make two different subscriptions where each handles only one type of signals:
let source = Observable.fromPromise(fetch('http://website.com')).publish();
source
.subscribe(undefined, err => console.log(err));
source
.flatMap(...)
.map(...)
.subscribe(console.log, () => {});
source.connect();
Now I have a separate observer only for errors.
Note that I had to make an empty callback with () => {} so the error will be silently ignored. Also note that when using multicasting (the publish() operator) then the Subject inside might have some specific behavior I should be aware of but maybe it doesn't matter in your use-case.

nodejs: API differences fs.readFile vs. http.get

Node.js has no built-in promises support. Eventual promise implementations are up to the user. The promise library Q supports some helper functions to convert node-style (err, succ) functions to promise objects.
The q documentation shows this with the fs.readFile function like this:
var fs = require("fs");
var q = require("q");
q.nfcall(fs.readFile, "/path/file")
.then(_ => console.log("succ"))
.fail(_ => console.log("err"))
.done();
This works well. Yet transferring this structure to, say, the http.get function is not possible, since it has natively another structure:
var http = require("http");
http.get("http://url..", succ_funct).on("error", err_funct);
This is already a "sort of" (!) promise style notation.
I'm asking myself if there is a reason for those different API types in node.js . Considering the examples shown above, it would be more consistent to write something like this:
q.nfcall(http.get, "http://url..")
.then(succ_f)
.fail(err_f)
.done();
fs.readFile() uses the common Node callback idiom where a callback gets called with an error object as first argument (if there were errors), or the result of the function (the file data) as second argument.
The optional function that gets passed to http.get() (only one, not two as you're suggesting) is a shortcut for adding an event handler function for the response event.
In other words, #1 is a shortcut for #2:
// #1
var client = http.get(URL, function(res) { ... });
// #2
var client = http.get(URL);
client.on('response', function(res) { ... });
Event handler functions and "regular" callbacks are different beasts. Event handlers are commonly used for streams (http.get() returns a stream).
Errors on streams may occur somewhere during the lifetime of the stream, so you don't use regular callbacks for streams (which imply that once the callback gets called, any errors have already occurred). Also, the result isn't necessarily available immediately (which requires reading the stream data "manually").
There are also stream-based API's for files (like fs.createReadStream()).

Is there a Node module for an async JSON parser that does not load the entire JSON string into memory?

I realize that there are a ton of Node modules that provide an async API for parsing JSON, but many of them seem to read the entire file or stream into memory, construct a giant string, and then pass it to JSON.parse(). This is what the second answer to "How to parse JSON using NodeJS?" suggests, and is exactly what the jsonfile module does.
Constructing a giant string is exactly what I want to avoid. I want an API like:
parseJsonFile(pathToJsonFile): Promise
where the Promise that is returned resolves to the parsed JSON object. This implementation should use a constant amount of memory. I'm not interested in any sort of SAX-like thing that broadcasts events as various pieces are parsed: just the end result.
I think jsonparse may do what I want (it clearly includes logic for parsing JSON without using JSON.parse()), but there is no simple example in the README.md, and the one file in the examples directory seems overly complicated.
I've written a module that does this: BFJ (Big-Friendly JSON). It exports a bunch of functions that operate at different levels of abstraction, but are all asynchronous and streaming at their core.
At the highest level are two functions for reading from and writing to the file system, bfj.read and bfj.write. They each return a promise, so you call them like this:
var bfj = require('bfj');
// Asynchronously read from a JSON file on disk
bfj.read(path)
.then(data => {
// :)
})
.catch(error => {
// :(
});
// Asynchronously write to a JSON file on disk
bfj.write(path, data)
.then(data => {
// :)
})
.catch(error => {
// :(
});
Also at this level is a function for serializing data to a JSON string, called bfj.stringify:
// Asynchronously serialize data to a JSON string
bfj.stringify(data)
.then(json => {
// :)
})
.catch(error => {
// :(
});
Beneath those are two more generic functions for reading from and writing to streams, bfj.parse and bfj.streamify. These serve as foundations for the higher level functions, but you can also call them directly:
// Asynchronously parse JSON from a readable stream
bfj.parse(readableStream).
.then(data => {
// :)
})
.catch(error => {
// :(
});
// Asynchronously serialize data to a writable stream of JSON
bfj.streamify(data).
.pipe(writableStream);
At the lowest level there are two functions analagous to SAX parsers/serializers, bfj.walk and bfj.eventify. It's unlikely you'd want to call these directly, they're just the guts of the implementation for the higher levels.
It's open-source and MIT-licensed. For more information, check the readme.
jsonparse is a streamed json parser, the sample code already shown the minimum of using Node stream.
Change client.request() to fs.createReadStream().
Setup on('data') listeners on the file read stream similar to what's in on('response') in the example.

Categories