I'm looking to see if there is a way of taking a generic async function, such as AJAX, or web worker on message, and forcing it to behave synchronously.
First off - I really do need this behaviour, I am aware of all the arguments regarding async callbacks vs returns - but in this case, I truly need to wait for a callback.
If there is no generic solution available I'd settle for one that works purely on the Worker.prototype.onmessage as that is the primary use case here.
I'm fairly sure the answer will be "thats not possible" but I wanted to check before surrendering.
I've already looked into promises and generators, I'd prefer not to use a third party library (I'm aware of fiber/futures for Node).
A potential use case:
function main(){
worker = new Worker("someWorker.js")
var result = worker.onmessage(function(event){
return event.data;
})
//wait for worker.onmessage
return result;
}
If you want to preserve synchronous flow of function, the generator function is the only possible solution. You'll send out all async tasks like var result = yield new Worker('someWorker.js').
Outside the function* main you'll get the worker with mainFn.next().value and then you attach onmessage listener which will return resolved data into function* main and the function will be resumed.
The complete generator example looks like this
// write function as generator
function* main () {
var result = yield new Worker("someWorker.js");
return result;
}
// instantiate the generator and get the first yield value
var mainFn = main()
, worker = mainFn.next().value
// set onmessage listener, which is callback, that sends data back to *main()
worker.onmessage = function(event) {
mainFn.next(event.data);
}
This way your main() function will remain almost similar to original form, so you can reuse the original code, but you need to write surrounding logic, which wouldn't be simple and which would use callbacks anyway.
You seem to be aware what generators are and that it's not widely supported in the browsers wild, so if you want run this code on web page, you'll need ES6 transpiler Babel.
Related
How to call Async function in recursion actually I have Async method - async function getSSN(req,res,next) and inside this function block I need to call this function but when I call this method by code - return await getSSN(req,res,next) this code is not working for me in node js. Can anyone give me solution for that.?
So, you can't mix async/await with plain asynchronous callbacks. They do not work properly. Your async function getSSN() is not properly resolving when your request.get() finishes because that's just a plain asynchronous callback which the async function knows nothing about.
Instead, you need to use a promise-based request() option. Since, the request library is now deprecated and should not be used for new projects, I would suggest one of the alternatives that are all based on promises already. My favorite is the got() library, but you can look at the list of alternatives and pick the one that has a programming style you like.
In addition, your getSSN() function has a bunch of side effects (modifying higher scoped variables). This is not a good way to program, particularly for asynchronous operations because the outside world generally doesn't know when things are done and when the higher scoped variables have the values in them you want them to. You should make getSSN() return a promise that resolves to its result and then have the caller use that result to modify the higher scoped variables. Again, if you showed us the rest of the coding context here, we could suggest a better overall way to do this. Please notice here that you're just not providing enough code for us to be able to show you the best way to write this code. That should be a general lesson for stackoverflow. Err on the side of giving us more than you think and we can then often make suggestions and improvements far beyond what you even know to ask about.
Here's your function with all the ill-adivsed side effects still in it (because you haven't shown us the rest of the code) using got():
async function getSSN(next) {
const result = await got('idms.dealersocket.com/api/account/…').json();
if (count <= totalpage) {
const data = Object.assign({}, result.Data);
const len = Object.keys(data).length;
for (var i = 0; i <= len; i++) {
ssn.push(data[i].Row.Borrower1SSN);
}
count++;
}
}
Now, getSSN() will return a promise that resolves or rejects when the network request is finished.
I have a long, complicated asynchronous process in TypeScript/JavaScript spread across many libraries and functions that, when it is finally finished receiving and processing all of its data, calls a function processComplete() to signal that it's finished:
processComplete(); // Let the program know we're done processing
Right now, that function looks something like this:
let complete = false;
function processComplete() {
complete = true;
}
In order to determine whether the process is complete, other code either uses timeouts or process.nextTick and checks the complete variable over and over again in loops. This is complicated and inefficient.
I'd instead like to let various async functions simply use await to wait and be awoken when the process is complete:
// This code will appear in many different places
await /* something regarding completion */;
console.log("We're done!");
If I were programming in Windows in C, I'd use an event synchronization primitive and the code would look something like this:
Event complete;
void processComplete() {
SetEvent(complete);
}
// Elsewhere, repeated in many different places
WaitForSingleObject(complete, INFINITE);
console.log("We're done!");
In JavaScript or TypeScript, rather than setting a boolean complete value to true, what exactly could processComplete do to make wake up any number of functions that are waiting using await? In other words, how can I implement an event synchronization primitive using await and async or Promises?
This pattern is quite close to your code:
const processComplete = args => new Promise(resolve => {
// ...
// In the middle of a callback for a async function, etc.:
resolve(); // instead of `complete = true;`
// ...
}));
// elsewhere
await processComplete(args);
console.log("We're done!");
More info: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Promise.
It really depends on what you mean by "other code" in this scenario. It sounds like you want to use some variation of the delegation pattern or the observer pattern.
A simple approach is to take advantage of the fact that JavaScript allows you to store an array of functions. Your processComplete() method could do something like this:
function processComplete(){
arrayOfFunctions.forEach(fn => fn());
}
Elsewhere, in your other code, you could create functions for what needs to be done when the process is complete, and add those functions to the arrayOfFunctions.
If you don't want these different parts of code to be so closely connected, you could set up a completely separate part of your code that functions as a notification center. Then, you would have your other code tell the notification center that it wants to be notified when the process is complete, and your processComplete() method would simply tell the notification center that the process is complete.
Another approach is to use promises.
I have a long, complicated asynchronous process in TypeScript/JavaScript spread across many libraries and functions
Then make sure that every bit of the process that is asynchronous returns a promise for its partial result, so that you can chain onto them and compose them together or await them.
When it is finally finished receiving and processing all of its data, calls a function processComplete() to signal that it's finished
It shouldn't. The function that starts the process should return a promise, and when the process is finished it should fulfill that promise.
If you don't want to properly promisify every bit of the whole process because it's too cumbersome, you can just do
function startProcess(…);
… // do whatever you need to do
return new Promise(resolve => {
processComplete = resolve;
// don't forget to reject when the process failed!
});
}
In JavaScript or TypeScript, rather than setting a boolean complete value to true, what exactly could processComplete do to make wake up any number of functions that are waiting using await?
If they are already awaiting the result of the promise, there is nothing else that needs to be done. (The awaited promise internally has such a flag already). It's really just doing
// somewhere:
var resultPromise = startProcess(…);
// elsewhere:
await resultPromise;
… // the process is completed here
You don't even need to fulfill the promise with a useful result if all you need is to synchronise your tasks, but you really should. (If there's no data they are waiting for, what are they waiting for at all?)
TL;DR: Is there any way to rewrite this callback-based JavaScript code to use promises and generators instead?
Background
I have a Firefox extension written using the Firefox Add-on SDK. As usual for the SDK, the code is split into an add-on script and a content script. The two scripts have different kinds of privileges: add-on scripts can do fancy things such as, for example, calling native code through the js-ctypes interface, while content scripts can interact with web pages. However, add-on scripts and content scripts can only interact with each other through an asynchronous message-passing interface.
I want to be able to call extension code from a user script on an ordinary, unprivileged web page. This can be done using a mechanism called exportFunction that lets one, well, export a function from extension code to user code. So far, so good. However, one can only use exportFunction in a content script, not an add-on script. That would be fine, except that the function I need to export needs to use the aforementioned js-ctypes interface, which can only be done in an add-on script.
(Edit: it turns out to not be the case that you can only use exportFunction in a content script. See the comment below.)
To get around this, I wrote a "wrapper" function in the content script; this wrapper is the function I actually export via exportFunction. I then have the wrapper function call the "real" function, over in the add-on script, by passing a message to the add-on script. Here's what the content script looks like; it's exporting the function lengthInBytes:
// content script
function lengthInBytes(arg, callback) {
self.port.emit("lengthInBytesCalled", arg);
self.port.on("lengthInBytesReturned", function(result) {
callback(result);
});
}
exportFunction(lengthInBytes, unsafeWindow, {defineAs: "lengthInBytes",
allowCallbacks: true});
And here's the add-on script, where the "real" version of lengthInBytes is defined. The code here listens for the content script to send it a lengthInBytesCalled message, then calls the real version of lengthInBytes, and sends back the result in a lengthInBytesReturned message. (In real life, of course, I probably wouldn't need to use js-ctypes to get the length of a string; this is just a stand-in for some more interesting C library call. Use your imagination. :) )
// add-on script
// Get "chrome privileges" to access the Components object.
var {Cu, Cc, Ci} = require("chrome");
Cu.import("resource://gre/modules/ctypes.jsm");
Cu.import("resource://gre/modules/Services.jsm");
var pageMod = require("sdk/page-mod");
var data = require("sdk/self").data;
pageMod.PageMod({
include: ["*", "file://*"],
attachTo: ["existing", "top"],
contentScriptFile: data.url("content.js"),
contentScriptWhen: "start", // Attach the content script before any page script loads.
onAttach: function(worker) {
worker.port.on("lengthInBytesCalled", function(arg) {
let result = lengthInBytes(arg);
worker.port.emit("lengthInBytesReturned", result);
});
}
});
function lengthInBytes(str) {
// str is a JS string; convert it to a ctypes string.
let cString = ctypes.char.array()(str);
libc.init();
let length = libc.strlen(cString); // defined elsewhere
libc.shutdown();
// `length` is a ctypes.UInt64; turn it into a JSON-serializable
// string before returning it.
return length.toString();
}
Finally, the user script (which will only work if the extension is installed) looks like this:
// user script, on an ordinary web page
lengthInBytes("hello", function(result) {
console.log("Length in bytes: " + result);
});
What I want to do
Now, the call to lengthInBytes in the user script is an asynchronous call; instead of returning a result, it "returns" its result in its callback argument. But, after seeing this video about using promises and generators to make async code easier to understand, I'm wondering how to rewrite this code in that style.
Specifically, what I want is for lengthInBytes to return a Promise that somehow represents the eventual payload of the lengthInBytesReturned message. Then, in the user script, I'd have a generator that evaluated yield lengthInBytes("hello") to get the result.
But, even after watching the above-linked video and reading about promises and generators, I'm still stumped about how to hook this up. A version of lengthInBytes that returns a Promise would look something like:
function lengthInBytesPromise(arg) {
self.port.emit("lengthInBytesCalled", arg);
return new Promise(
// do something with `lengthInBytesReturned` event??? idk.
);
}
and the user script would involve something like
var result = yield lengthInBytesPromise("hello");
console.log(result);
but that's as much as I've been able to figure out. How would I write this code, and what would the user script that calls it look like? Is what I want to do even possible?
A complete working example of what I have so far is here.
Thanks for your help!
A really elegant solution to this problem is coming in the next next version of JavaScript, ECMAScript 7, in the form of async functions, which are a marriage of Promises and generators that sugars over the warts of both. More on that at the very bottom of this answer.
I'm the author of Regenerator, a transpiler that supports async functions in browsers today, but I realize it might be overkill to suggest you introduce a compilation step into your add-on development process, so I'll focus instead on the questions you're actually asking: how does one design a sensible Promise-returning API, and what is the nicest way to consume such an API?
First of all, here's how I would implement lengthInBytesPromise:
function lengthInBytesPromise(arg) {
self.port.emit("lengthInBytesCalled", arg);
return new Promise(function(resolve, reject) {
self.port.on("lengthInBytesReturned", function(result) {
resolve(result);
});
});
}
The function(resolve, reject) { ... } callback is invoked immediately when the promise is instantiated, and the resolve and reject parameters are callback functions that can be used to provide the eventual value for the promise.
If there was some possibility of failure in this example, you could pass an Error object to the reject callback, but it seems like this operation is infallible, so we can just ignore that case here.
So that's how an API creates promises, but how do consumers consume such an API? In your content script, the simplest thing to do is to call lengthInBytesPromise and interact with the resulting Promise directly:
lengthInBytesPromise("hello").then(function(length) {
console.log(result);
});
In this style, you put the code that depends on the result of lengthInBytesPromise in a callback function passed to the .then method of the promise, which may not seem like a huge improvement over callback hell, but at least the indentation is more manageable if you're chaining a longer series of asynchronous operations:
lengthInBytesPromise("hello").then(function(length) {
console.log(result);
return someOtherPromise(length);
}).then(function(resultOfThatOtherPromise) {
return yetAnotherPromise(resultOfThatOtherPromise + 1);
}).then(function(finalResult) {
console.log(finalResult);
});
Generators can help reduce the boilerplate here, but additional runtime support is necessary. Probably the easiest approach is to use Dave Herman's task.js library:
spawn(function*() { // Note the *; this is a generator function!
var length = yield lengthInBytesPromise("hello");
var resultOfThatOtherPromise = yield someOtherPromise(length);
var finalResult = yield yetAnotherPromise(resultOfThatOtherPromise + 1);
console.log(finalResult);
});
This code is a lot shorter and less callback-y, that's for sure. As you can guess, most of the magic has simply been moved into the spawn function, but its implementation is actually pretty straightforward.
The spawn function takes a generator function and invokes it immediately to get a generator object, then invokes the gen.next() method of the generator object to get the first yielded promise (the result of lengthInBytesPromise("hello")), then waits for that promise to be fulfilled, then invokes gen.next(result) with the result, which provides a value for the first yield expression (the one assigned to length) and causes the generator function to run up to the next yield expression (namely, yield someOtherPromise(length)), producing the next promise, and so on, until there are no more promises left to await, because the generator function finally returned.
To give you a taste of what's coming in ES7, here's how you might use an async function to implement exactly the same thing:
async function process(arg) {
var length = await lengthInBytesPromise(arg);
var resultOfThatOtherPromise = await someOtherPromise(length);
var finalResult = await yetAnotherPromise(resultOfThatOtherPromise + 1);
return finalResult;
}
// An async function always returns a Promise for its own return value.
process(arg).then(function(finalResult) {
console.log(finalResult);
});
All that's really happening here is that the async keyword has replaced the spawn function (and the * generator syntax), and await has replaced yield. It's not a huge leap, but it will be really nice to have this syntax built into the language instead of having to rely on an external library like task.js.
If you're excited about using async functions instead of task.js, then by all means check out Regenerator!
I think the Promise is built by wrapping your original callback inside the resolve/reject function:
function lengthInBytesPromise(arg) {
self.port.emit("lengthInBytesCalled", arg);
let returnVal = new Promise(function(resolve, reject) {
self.port.on("lengthInBytesReturned", function(result) {
if (result) { // maybe some kind of validity check
resolve(result);
} else {
reject("Something went wrong?");
}
}
});
return returnVal;
}
Basically, it'd create the Promise and return it immediately, while the inside of the Promise kicks off and then handles the async task. I think at the end of the day someone has to take the callback-style code and wrap it up.
Your user would then do something like
lengthInBytesPromise(arg).then(function(result) {
// do something with the result
});
I'm attempting to understand javascript generators in node.js v8, without using any third party libraries.
I want to try having a generator that invokes an asynchronous callback (something like a web request) and returns the value that is called back.
My code (using Node v0.11.12) looks like:
var callbackWrapper = function (){
return function(callback) {
callback(null, 5);
};
};
var result = null;
var generator = (function *(){
result = yield callbackWrapper();
})();
var nextIteration = generator.next();
At the end of this code, nextIteration.value is a function, and result is still null.
It was my understanding that if yield is given a method with a callback, it invokes that callback.
My goal is to somehow get the value, 5, that is called back.
What is the way to do this? Do I have to pass some function to nextIteration.value and invoke it?
Let's clarify some things.
It was my understanding that if yield is given a method with a callback, it invokes that callback.
yield does not invoke anything. The purpose of yield is, well, to yield... When you define a generator you define an entity which yields values. It has nothing to do with async code. We can, however, leverage an important property of generators to handle async code.
Generators, by their definition, yield values. Here's the important part - between one yield to another the generator is suspended. In other words, it waits until it is asked to yield the next value. The internal state of the generator is kept in memory (on the stack) until the generator is exhausted (no more values to yield).
Another important property of generators is that we can send values back, thus changing their internal state.
So if we can suspend generators (make them wait) and we can send values back; we can essentially make them wait for some async operation to complete and then send the result back.
What is the way to do this? Do I have to pass some function to nodeIteration.value and invoke it?
Basically you need to wrap you code with a generator. Every time you start an async operation you make the generator wait by using yield. When the async operation completes you resume your generator by sending the result back with next(result).
I wrote an extensive post to explain this issue, which I'm sure will help you understand. Take a look: http://eyalarubas.com/javascript-generators-and-callbacks.html
Generators don't handle node style callbacks on their own. Instead it's returning the function that you wrapped inside of the callbackWrapper thunk. As yield only returns a value and then pauses execution at that point in time.
Generators weren't really designed for control flow but you can build on top of them to create control flow libraries like co, suspend, etc..
Basically what these libraries do (I'm oversimplifying here), is take your generator function and recursively call it until it tells them that it has finished.
Each of these libraries handles the internal yields in different ways, for example co turns everything it can handle into thunks internally. While suspend uses node-style callbacks for everything internally.
At each yield they check to see what was yielded to them a thunk, promise, generator, or whatever constructs that library handles, and abstracts the control out based on when they are completed.
You can build a structure around generators to handle asynchronous thunked functions but it's not for the feint of heart. Basically it would go something like this (Note: don't use this other than for playing around as its missing all the normal checks, error handling, etc..):
function controlFlow(genFunc){
// check to make sure we have a generator function otherwise explode
var generator; // reference for an initialized generator
// return a funcion so that execution time can be postponed
return function(){
runGen(genFunc());
}
function runGen(generator){
var ret = generator.next();
// here the generator is already finished we we'll return the final value
if(ret.done){
return ret.value
}
// here we'll handle the yielded value no matter what type it is
// I'm being naive here for examples sake don't do this
if(typeof ret.value === 'function'){
// oh look we have a regular function (very faulty logic)
// we wouldn't do this either but yeah
ret.value(function(err, result){
console.log(result);
});
}
// oh we got another item like an array or object that means parallelism or serialization depending on what we got back
// turn array, object, whatever into callback mechanisms and then track their completion
// we're just going to fake it here and just handle the next call
runGen(generator);
}
}
function thunked(callback){
return function(){
callback(null, 5);
};
};
function regular(callback){
console.log('regular function call');
callback(null, 'value');
};
controlFlow(function *(){
yield thunked(function(err, result){
console.log(err);
console.log(result);
});
yield regular;
yield thunked(function(err, result){
console.log('Another Thunked');
});
yield regular(function(err, result){
console.log(err);
console.log(result);
});
})();
result won’t get assigned until you send a value back to the generator by calling next again with the value you want to assign to result. In your example that would look like this:
nextIteration.value(function (error, value) {
generator.next(value);
});
I read some questions about this topic here at stackoverflow but none of them seems to answer my doubts.
I know how to create async functions using process.nextTick:
var async_function = function(val, callback){
process.nextTick(function(){
callback(val);
});
};
I've also read about Promise, but how do you write async non-blocking functions without using libraries such as process, Promise, etc?
What's the native way? I thought first on this approach:
var async = function(val, cb) {
cb(val);
}
However, that function blocks:
async(3, function(val) {
console.log(val);
});
console.log(4);
// returns:
3
4
So what's the correct implementation? How could I write async functions without depending on those libraries?
You can use setTimeout - it's native function like a delay.
On Node.js, you need to be using process.nextTick. In the browser, you can use the postMessage hack which is much more efficient than setTimeout(function(){},0).
Here is a cross-browser implementation of the postMessage hack: https://github.com/timoxley/next-tick/blob/master/index.js
Break up your function's work into chunks and use process.nextTick to queue the next chunk once the current one has completed. That way you allow other queued callbacks to be executed between each chunk of work. When all chunks have completed, call the callback that was passed into your function as a parameter.
The above is assuming your function is not calling existing async APIs which would naturally make your function async without doing anything special.