Understanding Node JS Generators with fs Module - javascript

I've been very excited about Node JS for awhile. I finally decided to knuckle down and write a test project to learn about generators in the latest Harmony build of Node.
Here is my very simple test project:
https://github.com/kirkouimet/project-node
To run my test project, you can easily pull the files from Github and then run it with:
node --harmony App.js
Here's my problem - I can't seem to get Node's asynchronous fs.readdir method to run inline with generators. Other projects out there, such as Galaxy and suspend seem to be able to do it.
Here is the block of code I need to fix. I want to be able to instantiate an object of type FileSystem and call the .list() method on it:
https://github.com/kirkouimet/project-node/blob/4c77294f42da9e078775bb84c763d4c60f21e1cc/FileSystem.js#L7-L11
FileSystem = Class.extend({
construct: function() {
this.currentDirectory = null;
},
list: function*(path) {
var list = yield NodeFileSystem.readdir(path);
return list;
}
});
Do I need to do something ahead of time to convert Node's fs.readdir into a generator?
One important note, I am parsing all class functions as they are created. This lets me handle generator functions differently than normal functions:
https://github.com/kirkouimet/project-node/blob/4c77294f42da9e078775bb84c763d4c60f21e1cc/Class.js#L31-L51
I've been really stumped with this project. Would love any assistance!
Here is what I am trying to accomplish:
Heavy use of classes with a modified version of John Resig's JavaScript Class support with inheritance
Using generators to get inline support for Node's stock async calls
Edit
I've tried to implement your example function and I am running into some trouble.
list: function*(path) {
var list = null;
var whatDoesCoReturn = co(function*() {
list = yield readdir(path);
console.log(list); // This shows an array of files (good!)
return list; // Just my guess that co should get this back, it doesn't
})();
console.log(whatDoesCoReturn); // This returns undefined (sad times)
// I need to use `list` right here
return list; // This returns as null
}

First and foremost, it is important to have a good model in your head of exactly what a generator is. A generator function is a function that returns a generator object, and that generator object will step through yield statements within the generator function as you call .next() on it.
Given that description, you should notice that asynchronous behavior is not mentioned. Any action on a generator on its own is synchronous. You can run to the first yield immediately and then do a setTimeout and then call .next() to go to the next yield, but it is the setTimeout that causes asynchronous behavior, not the generator itself.
So let's cast this in the light of fs.readdir. fs.readdir is an async function, and using it in a generator on its own will have no effect. Let's look at your example:
function * read(path){
return yield fs.readdir(path);
}
var gen = read(path);
// gen is now a generator object.
var first = gen.next();
// This is equivalent to first = fs.readdir(path);
// Which means first === undefined since fs.readdir returns nothing.
var final = gen.next();
// This is equivalent to final = undefined;
// Because you are returning the result of 'yield', and that is the value passed
// into .next(), and you are not passing anything to it.
Hopefully it makes it clearer that what you are still calling readdir synchronously, and you are not passing any callback, so it will probably throw an error or something.
So how do you get nice behavior from generators?
Generally this is accomplished by having the generator yield a special object that represents the result of readdir before the value has actually been calculated.
For (unrealistic) example, yielding a function is a simple way to yield something that represents the value.
function * read(path){
return yield function(callback){
fs.readdir(path, callback);
};
}
var gen = read(path);
// gen is now a generator object.
var first = gen.next();
// This is equivalent to first = function(callback){ ... };
// Trigger the callback to calculate the value here.
first(function(err, dir){
var dirData = gen.next(dir);
// This will just return 'dir' since we are directly returning the yielded value.
// Do whatever.
});
Really, you would want this type of logic to continue calling the generator until all of the yield calls are done, rather than hard-coding each call. The main thing to notice with this though, is now the generator itself looks synchronous, and everything outside the read function is super generic.
You need some kind of generator wrapper function that handles this yield value process, and your example of the suspend does exactly this. Another example is co.
The standard method for the method of "return something representing the value" is to return a promise or a thunk since returning a function like I did is kind of ugly.
With the thunk and co libraries, you with do the above without the example function:
var thunkify = require('thunkify');
var co = require('co');
var fs = require('fs');
var readdir = thunkify(fs.readdir);
co(function * (){
// `readdir` will call the node function, and return a thunk representing the
// directory, which is then `yield`ed to `co`, which will wait for the data
// to be ready, and then it will start the generator again, passing the value
// as the result of the `yield`.
var dirData = yield readdir(path, callback);
// Do whatever.
})(function(err, result){
// This callback is called once the synchronous-looking generator has returned.
// or thrown an exception.
});
Update
Your update still has some confusion. If you want your list function to be a generator, then you will need to use co outside of list wherever you are calling it. Everything inside of co should be generator-based and everything outside co should be callback-based. co does not make list automatically asynchronous. co is used to translate a generator-based async flow control into callback-based flow control.
e.g.
list: function(path, callback){
co(function * (){
var list = yield readdir(path);
// Use `list` right here.
return list;
})(function(err, result){
// err here would be set if your 'readdir' call had an error
// result is the return value from 'co', so it would be 'list'.
callback(err, result);
})
}

#loganfsmyth already provides a great answer to your question. The goal of my answer is to help you understand how JavaScript generators actually work, as this is a very important step to using them correctly.
Generators implement a state machine, the concept which is nothing new by itself. What's new is that generators allow to use the familiar JavaScript language construct (e.g., for, if, try/catch) to implement a state machine without giving up the linear code flow.
The original goal for generators is to generate a sequence of data, which has nothing to do with asynchrony. Example:
// with generator
function* sequence()
{
var i = 0;
while (i < 10)
yield ++i * 2;
}
for (var j of sequence())
console.log(j);
// without generator
function bulkySequence()
{
var i = 0;
var nextStep = function() {
if ( i >= 10 )
return { value: undefined, done: true };
return { value: ++i * 2, done: false };
}
return { next: nextStep };
}
for (var j of bulkySequence())
console.log(j);
The second part (bulkySequence) shows how to implement the same state machine in the traditional way, without generators. In this case, we no longer able to use while loop to generate values, and the continuation happens via nextStep callback. This code is bulky and unreadable.
Let's introduce asynchrony. In this case, the continuation to the next step of the state machine will be driven not by for of loop, but by some external event. I'll use a timer interval as a source of the event, but it may as well be a Node.js operation completion callback, or a promise resolution callback.
The idea is to show how it works without using any external libraries (like Q, Bluebird, Co etc). Nothing stops the generator from self-driving itself to the next step, and that's what the following code does. Once all steps of the asynchronous logic have completed (the 10 timer ticks), doneCallback will be invoked. Note, I don't return any meaningful data with yield here. I merely use it to suspend and resume the execution:
function workAsync(doneCallback)
{
var worker = (function* () {
// the timer callback drivers to the next step
var interval = setInterval(function() {
worker.next(); }, 500);
try {
var tick = 0;
while (tick < 10 ) {
// resume upon next tick
yield null;
console.log("tick: " + tick++);
}
doneCallback(null, null);
}
catch (ex) {
doneCallback(ex, null);
}
finally {
clearInterval(interval);
}
})();
// initial step
worker.next();
}
workAsync(function(err, result) {
console.log("Done, any errror: " + err); });
Finally, let's create a sequence of events:
function workAsync(doneCallback)
{
var worker = (function* () {
// the timer callback drivers to the next step
setTimeout(function() {
worker.next(); }, 1000);
yield null;
console.log("timer1 fired.");
setTimeout(function() {
worker.next(); }, 2000);
yield null;
console.log("timer2 fired.");
setTimeout(function() {
worker.next(); }, 3000);
yield null;
console.log("timer3 fired.");
doneCallback(null, null);
})();
// initial step
worker.next();
}
workAsync(function(err, result) {
console.log("Done, any errror: " + err); });
Once you understand this concept, you can move on with using promises as wrappers for generators, which takes it to the next powerful level.

Related

How to transform an asynchronous function into a synchronous function in javascript? [duplicate]

First, this is a very specific case of doing it the wrong way on-purpose to retrofit an asynchronous call into a very synchronous codebase that is many thousands of lines long and time doesn't currently afford the ability to make the changes to "do it right." It hurts every fiber of my being, but reality and ideals often do not mesh. I know this sucks.
OK, that out of the way, how do I make it so that I could:
function doSomething() {
var data;
function callBack(d) {
data = d;
}
myAsynchronousCall(param1, callBack);
// block here and return data when the callback is finished
return data;
}
The examples (or lack thereof) all use libraries and/or compilers, both of which are not viable for this solution. I need a concrete example of how to make it block (e.g. NOT leave the doSomething function until the callback is called) WITHOUT freezing the UI. If such a thing is possible in JS.
"don't tell me about how I should just do it "the right way" or whatever"
OK. but you should really do it the right way... or whatever
" I need a concrete example of how to make it block ... WITHOUT freezing the UI. If such a thing is possible in JS."
No, it is impossible to block the running JavaScript without blocking the UI.
Given the lack of information, it's tough to offer a solution, but one option may be to have the calling function do some polling to check a global variable, then have the callback set data to the global.
function doSomething() {
// callback sets the received data to a global var
function callBack(d) {
window.data = d;
}
// start the async
myAsynchronousCall(param1, callBack);
}
// start the function
doSomething();
// make sure the global is clear
window.data = null
// start polling at an interval until the data is found at the global
var intvl = setInterval(function() {
if (window.data) {
clearInterval(intvl);
console.log(data);
}
}, 100);
All of this assumes that you can modify doSomething(). I don't know if that's in the cards.
If it can be modified, then I don't know why you wouldn't just pass a callback to doSomething() to be called from the other callback, but I better stop before I get into trouble. ;)
Oh, what the heck. You gave an example that suggests it can be done correctly, so I'm going to show that solution...
function doSomething( func ) {
function callBack(d) {
func( d );
}
myAsynchronousCall(param1, callBack);
}
doSomething(function(data) {
console.log(data);
});
Because your example includes a callback that is passed to the async call, the right way would be to pass a function to doSomething() to be invoked from the callback.
Of course if that's the only thing the callback is doing, you'd just pass func directly...
myAsynchronousCall(param1, func);
Async functions, a feature in ES2017, make async code look sync by using promises (a particular form of async code) and the await keyword. Also notice in the code examples below the keyword async in front of the function keyword that signifies an async/await function. The await keyword won't work without being in a function pre-fixed with the async keyword. Since currently there is no exception to this that means no top level awaits will work (top level awaits meaning an await outside of any function). Though there is a proposal for top-level await.
ES2017 was ratified (i.e. finalized) as the standard for JavaScript on June 27th, 2017. Async await may already work in your browser, but if not you can still use the functionality using a javascript transpiler like babel or traceur. Chrome 55 has full support of async functions. So if you have a newer browser you may be able to try out the code below.
See kangax's es2017 compatibility table for browser compatibility.
Here's an example async await function called doAsync which takes three one second pauses and prints the time difference after each pause from the start time:
function timeoutPromise (time) {
return new Promise(function (resolve) {
setTimeout(function () {
resolve(Date.now());
}, time)
})
}
function doSomethingAsync () {
return timeoutPromise(1000);
}
async function doAsync () {
var start = Date.now(), time;
console.log(0);
time = await doSomethingAsync();
console.log(time - start);
time = await doSomethingAsync();
console.log(time - start);
time = await doSomethingAsync();
console.log(time - start);
}
doAsync();
When the await keyword is placed before a promise value (in this case the promise value is the value returned by the function doSomethingAsync) the await keyword will pause execution of the function call, but it won't pause any other functions and it will continue executing other code until the promise resolves. After the promise resolves it will unwrap the value of the promise and you can think of the await and promise expression as now being replaced by that unwrapped value.
So, since await just pauses waits for then unwraps a value before executing the rest of the line you can use it in for loops and inside function calls like in the below example which collects time differences awaited in an array and prints out the array.
function timeoutPromise (time) {
return new Promise(function (resolve) {
setTimeout(function () {
resolve(Date.now());
}, time)
})
}
function doSomethingAsync () {
return timeoutPromise(1000);
}
// this calls each promise returning function one after the other
async function doAsync () {
var response = [];
var start = Date.now();
// each index is a promise returning function
var promiseFuncs= [doSomethingAsync, doSomethingAsync, doSomethingAsync];
for(var i = 0; i < promiseFuncs.length; ++i) {
var promiseFunc = promiseFuncs[i];
response.push(await promiseFunc() - start);
console.log(response);
}
// do something with response which is an array of values that were from resolved promises.
return response
}
doAsync().then(function (response) {
console.log(response)
})
The async function itself returns a promise so you can use that as a promise with chaining like I do above or within another async await function.
The function above would wait for each response before sending another request if you would like to send the requests concurrently you can use Promise.all.
// no change
function timeoutPromise (time) {
return new Promise(function (resolve) {
setTimeout(function () {
resolve(Date.now());
}, time)
})
}
// no change
function doSomethingAsync () {
return timeoutPromise(1000);
}
// this function calls the async promise returning functions all at around the same time
async function doAsync () {
var start = Date.now();
// we are now using promise all to await all promises to settle
var responses = await Promise.all([doSomethingAsync(), doSomethingAsync(), doSomethingAsync()]);
return responses.map(x=>x-start);
}
// no change
doAsync().then(function (response) {
console.log(response)
})
If the promise possibly rejects you can wrap it in a try catch or skip the try catch and let the error propagate to the async/await functions catch call. You should be careful not to leave promise errors unhandled especially in Node.js. Below are some examples that show off how errors work.
function timeoutReject (time) {
return new Promise(function (resolve, reject) {
setTimeout(function () {
reject(new Error("OOPS well you got an error at TIMESTAMP: " + Date.now()));
}, time)
})
}
function doErrorAsync () {
return timeoutReject(1000);
}
var log = (...args)=>console.log(...args);
var logErr = (...args)=>console.error(...args);
async function unpropogatedError () {
// promise is not awaited or returned so it does not propogate the error
doErrorAsync();
return "finished unpropogatedError successfully";
}
unpropogatedError().then(log).catch(logErr)
async function handledError () {
var start = Date.now();
try {
console.log((await doErrorAsync()) - start);
console.log("past error");
} catch (e) {
console.log("in catch we handled the error");
}
return "finished handledError successfully";
}
handledError().then(log).catch(logErr)
// example of how error propogates to chained catch method
async function propogatedError () {
var start = Date.now();
var time = await doErrorAsync() - start;
console.log(time - start);
return "finished propogatedError successfully";
}
// this is what prints propogatedError's error.
propogatedError().then(log).catch(logErr)
If you go here you can see the finished proposals for upcoming ECMAScript versions.
An alternative to this that can be used with just ES2015 (ES6) is to use a special function which wraps a generator function. Generator functions have a yield keyword which may be used to replicate the await keyword with a surrounding function. The yield keyword and generator function are a lot more general purpose and can do many more things then just what the async await function does. If you want a generator function wrapper that can be used to replicate async await I would check out co.js. By the way co's function much like async await functions return a promise. Honestly though at this point browser compatibility is about the same for both generator functions and async functions so if you just want the async await functionality you should use Async functions without co.js.
(I recommend just using async/await it's pretty widely supported in most environments that the above strikethrough is supported in.)
Browser support is actually pretty good now for Async functions (as of 2017) in all major current browsers (Chrome, Safari, and Edge) except IE.
Take a look at JQuery Promises:
http://api.jquery.com/promise/
http://api.jquery.com/jQuery.when/
http://api.jquery.com/deferred.promise/
Refactor the code:
var dfd = new jQuery.Deferred();
function callBack(data) {
dfd.notify(data);
}
// do the async call.
myAsynchronousCall(param1, callBack);
function doSomething(data) {
// do stuff with data...
}
$.when(dfd).then(doSomething);
You can force asynchronous JavaScript in NodeJS to be synchronous with sync-rpc.
It will definitely freeze your UI though, so I'm still a naysayer when it comes to whether what it's possible to take the shortcut you need to take. It's not possible to suspend the One And Only Thread in JavaScript, even if NodeJS lets you block it sometimes. No callbacks, events, anything asynchronous at all will be able to process until your promise resolves. So unless you the reader have an unavoidable situation like the OP (or, in my case, are writing a glorified shell script with no callbacks, events, etc.), DO NOT DO THIS!
But here's how you can do this:
./calling-file.js
var createClient = require('sync-rpc');
var mySynchronousCall = createClient(require.resolve('./my-asynchronous-call'), 'init data');
var param1 = 'test data'
var data = mySynchronousCall(param1);
console.log(data); // prints: received "test data" after "init data"
./my-asynchronous-call.js
function init(initData) {
return function(param1) {
// Return a promise here and the resulting rpc client will be synchronous
return Promise.resolve('received "' + param1 + '" after "' + initData + '"');
};
}
module.exports = init;
LIMITATIONS:
These are both a consequence of how sync-rpc is implemented, which is by abusing require('child_process').spawnSync:
This will not work in the browser.
The arguments to your function must be serializable. Your arguments will pass in and out of JSON.stringify, so functions and non-enumerable properties like prototype chains will be lost.
There is one nice workaround at http://taskjs.org/
It uses generators which are new to javascript. So it's currently not implemented by most browsers. I tested it in firefox, and for me it is nice way to wrap asynchronous function.
Here is example code from project GitHub
var { Deferred } = task;
spawn(function() {
out.innerHTML = "reading...\n";
try {
var d = yield read("read.html");
alert(d.responseText.length);
} catch (e) {
e.stack.split(/\n/).forEach(function(line) { console.log(line) });
console.log("");
out.innerHTML = "error: " + e;
}
});
function read(url, method) {
method = method || "GET";
var xhr = new XMLHttpRequest();
var deferred = new Deferred();
xhr.onreadystatechange = function() {
if (xhr.readyState === 4) {
if (xhr.status >= 400) {
var e = new Error(xhr.statusText);
e.status = xhr.status;
deferred.reject(e);
} else {
deferred.resolve({
responseText: xhr.responseText
});
}
}
};
xhr.open(method, url, true);
xhr.send();
return deferred.promise;
}
What you want is actually possible now. If you can run the asynchronous code in a service worker, and the synchronous code in a web worker, then you can have the web worker send a synchronous XHR to the service worker, and while the service worker does the async things, the web worker's thread will wait. This is not a great approach, but it could work.
let result;
async_function().then(r => result = r);
while (result === undefined) // Wait result from async_function
require('deasync').sleep(100);
In Node.js it's possible to write synchronous code which actually invokes asynchronous operations.
node-fibers allows this. It's a 3rd party native extension provided as an npm module.
It implements fibers/coroutines, so when a specific fiber is blocked waiting for asynchronous operation, the whole program events loop doesn't block - another fiber (if exists) continues its job.
With fibers your code would look like this:
var Fiber = require('fibers');
function doSomething() {
var fiber = Fiber.current;
function callBack(data) {
fiber.run(data);
}
myAsynchronousCall(param1, callBack);
// execution blocks here
var data = Fiber.yield();
return data;
}
// The whole program must be wrapped with Fiber
Fiber(function main() {
var data = doSomething();
console.log(data);
}).run();
Note, that you should avoid it and use async/await instead. See below a note from the project readme https://github.com/laverdet/node-fibers:
NOTE OF OBSOLESCENCE -- The author of this project recommends you avoid its use if possible. The original version of this module targeted nodejs v0.1.x in early 2011 when JavaScript on the server looked a lot different. Since then async/await, Promises, and Generators were standardized and the ecosystem as a whole has moved in that direction.
I'll continue to support newer versions of nodejs as long as possible but v8 and nodejs are extraordinarily complex and dynamic platforms. It is inevitable that one day this library will abruptly stop working and no one will be able to do anything about it.
I'd like to say thank you to all the users of fibers, your support over the years has meant a lot to me.
Using Node 16's worker threads actually makes this possible, The following example the main thread is running the asynchronous code while the worker thread is waiting for it synchronously.
Not that is is very useful, but it at least does vaguely what the original question asked by waiting for asynchronous code synchronously.
const {
Worker, isMainThread, parentPort, receiveMessageOnPort
} = require('worker_threads');
if (isMainThread) {
const worker = new Worker(__filename);
worker.on('message', async () => {
worker.postMessage(await doAsyncStuff());
});
} else {
console.log(doStuffSync());
}
function doStuffSync(){
parentPort.postMessage({fn: 'doStuff'});
let message;
while (!message) {
message = receiveMessageOnPort(parentPort)
}
return message;
}
function doAsyncStuff(){
return new Promise((resolve) => setTimeout(() => resolve("A test"), 1000));
}
One thing people might not consider: If you control the async function (which other pieces of code depend on), AND the codepath it would take is not necessarily asynchronous, you can make it synchronous (without breaking those other pieces of code) by creating an optional parameter.
Currently:
async function myFunc(args_etcetc) {
// you wrote this
return 'stuff';
}
(async function main() {
var result = await myFunc('argsetcetc');
console.log('async result:' result);
})()
Consider:
function myFunc(args_etcetc, opts={}) {
/*
param opts :: {sync:Boolean} -- whether to return a Promise or not
*/
var {sync=false} = opts;
if (sync===true)
return 'stuff';
else
return new Promise((RETURN,REJECT)=> {
RETURN('stuff');
});
}
// async code still works just like before:
(async function main() {
var result = await myFunc('argsetcetc');
console.log('async result:', result);
})();
// prints: 'stuff'
// new sync code works, if you specify sync mode:
(function main() {
var result = myFunc('argsetcetc', {sync:true});
console.log('sync result:', result);
})();
// prints: 'stuff'
Of course this doesn't work if the async function relies on inherently async operations (network requests, etc.), in which case the endeavor is futile (without effectively waiting idle-spinning for no reason).
Also this is fairly ugly to return either a value or a Promise depending on the options passed in.
("Why would I have written an async function if it didn't use async constructs?" one might ask? Perhaps some modalities/parameters of the function require asynchronicity and others don't, and due to code duplication you wanted a monolithic block rather than separate modular chunks of code in different functions... For example perhaps the argument is either localDatabase (which doesn't require await) or remoteDatabase (which does). Then you could runtime error if you try to do {sync:true} on the remote database. Perhaps this scenario is indicative of another problem, but there you go.)
This ability of promises includes two key features of synchronous operations as follows (or then() accepts two callbacks).
When you get the result, call resolve() and pass the final result.
In case of error, call reject().
The idea is that the result is passed through the chain of .then() handlers.
const synchronize = (() => {
let chain = Promise.resolve()
return async (promise) => {
return chain = chain.then(promise)
}
})()
I wondered the same thing and noticed that the currently best answer contains the right idea in my mind for most use cases, but forgets to mention a couple of things. When using a global variable to lock execution, we're talking about Semaphores, and there are some packages which implement those (my recommendation: async-sema). I think this makes it a little simpler and cleaner.
import { Sema } from 'async-sema'
const sema = new Sema(1) // allow only one concurrent process
async function doSomething() {
var data;
await sema.acquire();
// only one process gets inside here
data = await myAsynchronousCall(param1);
sema.release();
return data;
}
The advantage is obviously that the rest of your program can still do other things asynchronously, only the single block is kind of forced to be synchronously. Disadvantage is that you have to be careful what and where to lock, try/catch/finally possible errors, etc.
You can also convert it into callbacks.
function thirdPartyFoo(callback) {
callback("Hello World");
}
function foo() {
var fooVariable;
thirdPartyFoo(function(data) {
fooVariable = data;
});
return fooVariable;
}
var temp = foo();
console.log(temp);
The idea that you hope to achieve can be made possible if you tweak the requirement a little bit
The below code is possible if your runtime supports the ES6 specification.
More about async functions
async function myAsynchronousCall(param1) {
// logic for myAsynchronous call
return d;
}
function doSomething() {
var data = await myAsynchronousCall(param1); //'blocks' here until the async call is finished
return data;
}

What does this generator syntax do?

What does the following syntax mean? I don’t understand the use of a * after a yield. I’m new to generators, redux, and sagas, so I would appreciate some help understanding what the syntax *, takeEvery(), and the return function *(action) { do:
var MIDDLEWARES = []
function builder( ) {
const LOAD_DATA = "POI_LOADER/LOAD_POIS"
MIDDLEWARES.push( function *sagaFunction() {
yield *takeEvery( LOAD_DATA, loadData( statusField) )
} )
}
const loadData = (statusField) => {
return function *(action) {
console.log("action.venueId = " + action.venueId)
}
}
There are several questions here:
What does the * in yield *takeEvery() mean? Seems to be answered by Delegated yield (yield star, yield *) in generator functions.
Why doesn't the builder() function need a * to make it a generator given that it contains a yield * statement? Is that because the yield *takeEvery() is wrapped in the generator function sagaFunction()?
What does the takeEvery() function do, especially given that it has a * in front of it? I think takeEvery(). Based on its documentation, I think it applies loadData() to everything in LOAD_DATA. But if LOAD_DATA isn't an array, is takeEvery() needed in this code?
How come the declaration return function *(action) seems to have no name for the function? Is it declaring a generator with input parameter action and assigning that generator for a const called loadData?
Does the Saga library call next() on the generators created in this code?
So generators allow to return a value from a generator function by using yield and then resume execution from there on the next call.
yield* is used to indicate that the value returned is coming from another generator - so the generator function that calls yield* is delegating the actual value creation to another generator function in this case.
See MDN on this for more info.

What is the correct pattern with generators and iterators for managing a stream

I am trying to figure out how to arrange a pair of routines to control writing to a stream using the generator/iterator functions in ES2015. Its a simple logging system to use in nodejs
What I am trying to achieve is a function that external processes can call to write to a log.I am hoping that the new generator/iterator functions means that if it needs to suspend/inside this routine that is transparent.
stream.write should normally return immediately, but can return false to say that the stream is full. In this case it needs to wait for stream.on('drain',cb) to fire before returning
I am thinking that the actual software that writes to the stream is a generator function which yields when it is ready to accept another request, and that the function I provide to allow external people to call the stream is an interator, but I might have this the wrong way round.
So, something like this
var stopLogger = false;
var it = writer();
function writeLog(line) {
it.next(line);
})
function *writer() {
while (!stopLogger) {
line = yield;
if(!stream.write) {
yield *WaitDrain(); //can't continue until we get drain
}
}
});
function *waitDrain() {
//Not sure what do do here to avoid waiting
stream.on('drain', () => {/*do I yield here or something*/});
I found the answer here https://davidwalsh.name/async-generators
I have it backwards.
The code above should be
var stopLogger = false;
function *writeLog(line) {
yield writer(line)
})
var it = writeLog();
function writer(line) {
if (stopLogger) {
setTimeout(()=>{it.next();},1};//needed so can get to yield
} else {
if(stream.write(line)) {
setTimeout(()=>{it.next();},1}; //needed so can get to yeild
}
}
}
stream.on('drain', () => {
it.next();
}
I haven't quite tried this, just translated from the above article, and there is some complication around errors etc which the article suggests can be solved by enhancing the it operator to return a promise which can get resolved in a "runGenerator" function, But it solved my main issue, which was about how should the pattern work.

How can I invoke a callback given to a yield in javascript v8 generator code?

I'm attempting to understand javascript generators in node.js v8, without using any third party libraries.
I want to try having a generator that invokes an asynchronous callback (something like a web request) and returns the value that is called back.
My code (using Node v0.11.12) looks like:
var callbackWrapper = function (){
return function(callback) {
callback(null, 5);
};
};
var result = null;
var generator = (function *(){
result = yield callbackWrapper();
})();
var nextIteration = generator.next();
At the end of this code, nextIteration.value is a function, and result is still null.
It was my understanding that if yield is given a method with a callback, it invokes that callback.
My goal is to somehow get the value, 5, that is called back.
What is the way to do this? Do I have to pass some function to nextIteration.value and invoke it?
Let's clarify some things.
It was my understanding that if yield is given a method with a callback, it invokes that callback.
yield does not invoke anything. The purpose of yield is, well, to yield... When you define a generator you define an entity which yields values. It has nothing to do with async code. We can, however, leverage an important property of generators to handle async code.
Generators, by their definition, yield values. Here's the important part - between one yield to another the generator is suspended. In other words, it waits until it is asked to yield the next value. The internal state of the generator is kept in memory (on the stack) until the generator is exhausted (no more values to yield).
Another important property of generators is that we can send values back, thus changing their internal state.
So if we can suspend generators (make them wait) and we can send values back; we can essentially make them wait for some async operation to complete and then send the result back.
What is the way to do this? Do I have to pass some function to nodeIteration.value and invoke it?
Basically you need to wrap you code with a generator. Every time you start an async operation you make the generator wait by using yield. When the async operation completes you resume your generator by sending the result back with next(result).
I wrote an extensive post to explain this issue, which I'm sure will help you understand. Take a look: http://eyalarubas.com/javascript-generators-and-callbacks.html
Generators don't handle node style callbacks on their own. Instead it's returning the function that you wrapped inside of the callbackWrapper thunk. As yield only returns a value and then pauses execution at that point in time.
Generators weren't really designed for control flow but you can build on top of them to create control flow libraries like co, suspend, etc..
Basically what these libraries do (I'm oversimplifying here), is take your generator function and recursively call it until it tells them that it has finished.
Each of these libraries handles the internal yields in different ways, for example co turns everything it can handle into thunks internally. While suspend uses node-style callbacks for everything internally.
At each yield they check to see what was yielded to them a thunk, promise, generator, or whatever constructs that library handles, and abstracts the control out based on when they are completed.
You can build a structure around generators to handle asynchronous thunked functions but it's not for the feint of heart. Basically it would go something like this (Note: don't use this other than for playing around as its missing all the normal checks, error handling, etc..):
function controlFlow(genFunc){
// check to make sure we have a generator function otherwise explode
var generator; // reference for an initialized generator
// return a funcion so that execution time can be postponed
return function(){
runGen(genFunc());
}
function runGen(generator){
var ret = generator.next();
// here the generator is already finished we we'll return the final value
if(ret.done){
return ret.value
}
// here we'll handle the yielded value no matter what type it is
// I'm being naive here for examples sake don't do this
if(typeof ret.value === 'function'){
// oh look we have a regular function (very faulty logic)
// we wouldn't do this either but yeah
ret.value(function(err, result){
console.log(result);
});
}
// oh we got another item like an array or object that means parallelism or serialization depending on what we got back
// turn array, object, whatever into callback mechanisms and then track their completion
// we're just going to fake it here and just handle the next call
runGen(generator);
}
}
function thunked(callback){
return function(){
callback(null, 5);
};
};
function regular(callback){
console.log('regular function call');
callback(null, 'value');
};
controlFlow(function *(){
yield thunked(function(err, result){
console.log(err);
console.log(result);
});
yield regular;
yield thunked(function(err, result){
console.log('Another Thunked');
});
yield regular(function(err, result){
console.log(err);
console.log(result);
});
})();
result won’t get assigned until you send a value back to the generator by calling next again with the value you want to assign to result. In your example that would look like this:
nextIteration.value(function (error, value) {
generator.next(value);
});

Node.js async to sync

How can I make this work
var asyncToSync = syncFunc();
function syncFunc() {
var sync = true;
var data = null;
query(params, function(result){
data = result;
sync = false;
});
while(sync) {}
return data;
}
I tried to get sync function from async one,
I need it to use FreeTds async query as sync one
Use deasync - a module written in C++ which exposes Node.js event loop to JavaScript. The module also exposes a sleep function that blocks subsequent code but doesn't block entire thread, nor incur busy wait. You can put the sleep function in your while loop:
var asyncToSync = syncFunc();
function syncFunc() {
var sync = true;
var data = null;
query(params, function(result){
data = result;
sync = false;
});
while(sync) {require('deasync').sleep(100);}
return data;
}
Nowadays this generator pattern can be a fantastic solution in many situations:
// nodejs script doing sequential prompts using async readline.question function
var main = (function* () {
// just import and initialize 'readline' in nodejs
var r = require('readline')
var rl = r.createInterface({input: process.stdin, output: process.stdout })
// magic here, the callback is the iterator.next
var answerA = yield rl.question('do you want this? ', res=>main.next(res))
// and again, in a sync fashion
var answerB = yield rl.question('are you sure? ', res=>main.next(res))
// readline boilerplate
rl.close()
console.log(answerA, answerB)
})() // <-- executed: iterator created from generator
main.next() // kick off the iterator,
// runs until the first 'yield', including rightmost code
// and waits until another main.next() happens
You can do it with node-sync lib
var sync = require('sync');
sync(function(){
var result = query.sync(query, params);
// result can be used immediately
})
Notice: your query must use standart callback call (with error first): callback(error, result).
If you can't change query method, just create .async() wrapper (see github link).
I've been using syncrhonize.js with great success. There's even a pending pull request (which works quite well) to support async functions which have multiple parameters. Far better and easier to use than node-sync imho. Added bonus that it has easy-to-understand and thorough documentation, whereas node-sync does not.
The issue you are having is that your tight while loop is blocking. So I don't think your query callback will ever be run. I think you need to use setTimeout or the like to prevent the function from blocking, but should you do so, the function will return before the callback is called. This functionality must be implemented at a lower level.
If you are in the browser, you might check out this article. In node you have to rely on the implementation of whatever you're querying. It may or may not provide synchronous methods.

Categories