I've looked high and low, and can only find how to write async functions, which I already understand.
What I am trying to do is run an async method in a triggered event [EventEmitter], but such a simple thing seems to be just simply not possible as I can find.
Consider the following...
// Your basic async method..
function doSomething(callback) {
var obj = { title: 'hello' };
// Fire an event for event handlers to alter the object.
// EvenEmitters are called synchronously
eventobj.emit('alter_object', obj);
callback(null, obj);
}
// when this event is fired, I want to manipulate the data
eventobj.on('alter_object', function(obj) {
obj.title += " world!";
// Calling this async function here means that our
// event handler will return before our data is retrieved.
somemodule.asyncFunction(callback(err, data) {
obj.data = data;
});
});
As you can see in the last few lines, the event handler will finish before the object's data property is added.
What I need is something where I can turn the async function into an sync function and get the results there and then. so for example...
obj.data = somemodule.asyncFunction();
I've looked at the wait.for module, the async module, and none of these will not work. I've even looked into the yield method, but it seems not yet fully implemented into the V8 engine.
I've also tried using a while loop too wait for data to populate, but this just brings with it the CPU overload issue.
Has anyone experienced this and found a design pattern to get around this?
You cannot turn an async function into a synchronous one in node.js. It just cannot be done.
If you have an asynchronous result, you cannot return it synchronously or wait for it. You will have to redesign the interface to use an asynchronous interface (which nearly always involves passing in a callback that will be called when the result is ready).
If you're wanting to do something after you .emit() an event that is itself going to do something asynchronously and you want to wait until after the async thing finished, then the event emitter is probably not the right interface. You'd rather have a function call that returns a promise or takes a callback as an argument. You could manipulate an eventEmitter to use this, but you'd have to post back a second event when the async operation finished and have the original caller not do it's second part until it receives the second event (which is really not a good way to go).
Bottom line - you need a different design that works with async responses (e.g. callbacks or promises).
Seems what I wanted to do is just not possible and the two models conflict. To achieve what I wanted in the end, I encapsulated my modules into an array-like object with some event methods that will pass it on to the module's object, which inherited the async-eventemitter class.
So, think of it like so...
My custom app modules may inherit the async-eventemitter module so they have the .on() and .emit(), etc. methods.
I create a customised array item, that will allow me to pass an event on to the module in question that will work asynchronously.
The code I created (and this is by no means complete or perfect)...
// My private indexer for accessing the modules array (below) by name.
var module_dict = {};
// An array of my modules on my (express) app object
app.modules = [];
// Here I extended the array with easier ways to add and find modules.
// Haven't removed some code to trim down this. Let me know if you want the code.
Object.defineProperty(app.modules, 'contains', { enumerable: false, ... });
Object.defineProperty(app.modules, 'find', { enumerable: false, ... });
// Allows us to add a hook/(async)event to a module, if it exists
Object.defineProperty(app.modules, 'on', { enumerable: false, configurable: false, value: function(modulename, action, func) {
if (app.modules.contains(modulename)) {
var modu = app.modules.find(modulename);
if (modu.module && modu.module['on']) {
// This will pass on the event to the module's object that
// will have the async-eventemitter inherited
modu.module.on(action, func);
}
}
} });
Object.defineProperty(app.modules, 'once', { enumerable: false, configurable: false, value: function(modulename, action, func) {
if (app.modules.contains(modulename)) {
var modu = app.modules.find(modulename);
if (modu.on) {
modu.on(action, func);
}
}
} });
This then allows me to bind an event handler to a module by simply calling something like the following... .on(module_name, event_name, callback)
app.modules.on('my_special_module_name', 'loaded', function(err, data, next) {
// ...async stuff, that then calls next to continue to the next event...
if (data.filename.endsWith('.jpg'))
data.dimensions = { width: 100, height: 100 };
next(err, data);
});
And then to execute it I would do something like (express)...
app.get('/foo', function(req, res, next) {
var data = {
filename: 'bar.jpg'
};
// Now have event handlers alter/update our data
// (eg, extend an object about a file with image data if that file is an image file).
my_special_module.emit('loaded', data, function(err, data) {
if (err) next(err);
res.send(data);
next();
});
});
Again, this is just an example of what I did, so i've probably missed something in my copy above, but effectively it's the design I ended up using and it worked like a treat, and I was able to extend data on an object before being pushed out to my HTTP response, without having to replace the main [expressjs] object's standard EventEmitter model.
(eg, I added image data for files that we're loaded that we're image files. If anyone wants the code, let me know, I am more than happy to share what I did)
Related
Short backstory: I am trying to create a Readable stream based on data chunks that are emitted back to my server from the client side with WebSockets. Here's a class I've created to "simulate" that behavior:
class DataEmitter extends EventEmitter {
constructor() {
super();
const data = ['foo', 'bar', 'baz', 'hello', 'world', 'abc', '123'];
// Every second, emit an event with a chunk of data
const interval = setInterval(() => {
this.emit('chunk', data.splice(0, 1)[0]);
// Once there are no more items, emit an event
// notifying that that is the case
if (!data.length) {
this.emit('done');
clearInterval(interval);
}
}, 1e3);
}
}
In this post, the dataEmitter in question will have been created like this.
// Our data is being emitted through events in chunks from some place.
// This is just to simulate that. We cannot change the flow - only listen
// for the events and do something with the chunks.
const dataEmitter = new DataEmitter();
Right, so I initially tried this:
const readable = new Readable();
dataEmitter.on('chunk', (data) => {
readable.push(data);
});
dataEmitter.once('done', () => {
readable.push(null);
});
But that results in this error:
Error [ERR_METHOD_NOT_IMPLEMENTED]: The _read() method is not implemented
So I did this, implementing read() as an empty function:
const readable = new Readable({
read() {},
});
dataEmitter.on('chunk', (data) => {
readable.push(data);
});
dataEmitter.once('done', () => {
readable.push(null);
});
And it works when piping into a write stream, or sending the stream to my test API server. The resulting .txt file looks exactly as it should:
foobarbazhelloworldabc123
However, I feel like there's something quite wrong and hacky with my solution. I attempted to put the listener registration logic (.on('chunk', ...) and .once('done', ...)) within the read() implementation; however, read() seems to get called multiple times, and that results in the listeners being registered multiple times.
The Node.js documentation says this about the _read() method:
When readable._read() is called, if data is available from the resource, the implementation should begin pushing that data into the read queue using the this.push(dataChunk) method. _read() will be called again after each call to this.push(dataChunk) once the stream is ready to accept more data. _read() may continue reading from the resource and pushing data until readable.push() returns false. Only when _read() is called again after it has stopped should it resume pushing additional data into the queue.
After dissecting this, it seems that the consumer of the stream calls upon .read() when it's ready to read more data. And when it is called, data should be pushed into the stream. But, if it is not called, the stream should not have data pushed into it until the method is called again (???). So wait, does the consumer call .read() when it is ready for more data, or does it call it after each time .push() is called? Or both?? The docs seem to contradict themselves.
Implementing .read() on Readable is straightforward when you've got a basic resource to stream, but what would be the proper way of implementing it in this case?
And also, would someone be able to explain in better terms what the .read() method is on a deeper level, and how it should be implemented?
Thanks!
Response to the answer:
I did try registering the listeners within the read() implementation, but because it is called multiple times by the consumer, it registers the listeners multiple times.
Observing this code:
const readable = new Readable({
read() {
console.log('called');
dataEmitter.on('chunk', (data) => {
readable.push(data);
});
dataEmitter.once('done', () => {
readable.push(null);
});
},
});
readable.pipe(createWriteStream('./data.txt'));
The resulting file looks like this:
foobarbarbazbazbazhellohellohellohelloworldworldworldworldworldabcabcabcabcabcabc123123123123123123123
Which makes sense, because the listeners are being registered multiple times.
Seems like the only purpose of actually implementing the read() method is to only start receiving the chunks and pushing them into the stream when the consumer is ready for that.
Based on these conclusions, I've come up with this solution.
class MyReadable extends Readable {
// Keep track of whether or not the listeners have already
// been added to the data emitter.
#registered = false;
_read() {
// If the listeners have already been registered, do
// absolutely nothing.
if (this.#registered) return;
// "Notify" the client via websockets that we're ready
// to start streaming the data chunks.
const emitter = new DataEmitter();
const handler = (chunk: string) => {
this.push(chunk);
};
emitter.on('chunk', handler);
emitter.once('done', () => {
this.push(null);
// Clean up the listener once it's done (this is
// assuming the #emitter object will still be used
// in the future).
emitter.off('chunk', handler);
});
// Mark the listeners as registered.
this.#registered = true;
}
}
const readable = new MyReadable();
readable.pipe(createWriteStream('./data.txt'));
But this implementation doesn't allow for the consumer to control when things are pushed. I guess, however, in order to achieve that sort of control, you'd need to communicate with the resource emitting the chunks to tell it to stop until the read() method is called again.
The following is a simple scenario for some GNOME extension:
Enable the extension. The extension is a class which extends
Clutter.Actor.
It creates an actor called myActor and adds it:
this.add_child(myActor).
Then it calls an asynchronous
time-consuming function this._tcFunction() which in the end does something
with myActor.
Here's where I run into a problem:
We disable (run this.destroy()) the extension immediately after enabling it.
On disabling, this.destroy() runs GObject's this.run_dispose() to
collect garbage. However, if this._tcFunction() has not finished running,
it'll later try to access myActor which might have been already
deallocated by this.run_dispose().
One way to go about this, is to define a boolean variable in this.destroy()
destroy() {
this._destroying = true
// ...
this.run_dispose;
}
and then add a check in this._tcFunction(), e.g.
async _tcFunction() {
await this._timeConsumingStuff();
if (this._destroying === true) { return; }
myActor.show();
}
My question: is there a nicer way to deal with these situations? Maybe with Gio.Cancellable()? AFAIK, there's no easy way to stop an async function in javascript...
Firstly, two things to be aware of:
Avoid calling low-level memory management functions like GObject.run_dispose() as there are a cases in the C libraries where these objects are being cached for reuse and aren't actually being disposed when you think they are. There is also no dispose signal and other objects may need notification.
Avoid overriding functions that trigger disposal like Clutter.Actor.destroy() and instead use the destroy signal. GObject signal callback always get the emitting object as the first argument and it is safe to use that in a destroy callback.
There are a couple ways I could think of solving this, but it depends on the situation. If the async function is a GNOME library async function, it probably does have a cancellable argument:
let cancellable = new Gio.Cancellable();
let actor = new Clutter.Actor();
actor.connect('destroy', () => cancellable.cancel());
Gio.File.new_for_path('foo.txt').load_contents_async(cancellable, (file, res) => {
try {
let result = file.load_contents_finish(res);
// This shouldn't be necessary if the operation succeeds (I think)
if (!cancellable.is_cancelled())
log(actor.width);
} catch (e) {
// We know it's not safe
if (e.matches(Gio.IOErrorEnum, Gio.IOErrorEnum.CANCELLED))
log('it was cancelled');
// Probably safe, but let's check
else if (!cancellable.is_cancelled())
log(actor.width);
}
});
// The above function will begin but not finish before the
// cancellable is triggered
actor.destroy();
Of course, you can always use a cancellable with a Promise, or just a regular function/callback pattern:
new Promise((resolve, reject) => {
// Some operation
resolve();
}).then(result => {
// Check the cancellable
if (!cancellable.is_cancelled())
log(actor.width);
});
Another option is to null out your reference, since you can safely check for that:
let actor = new Clutter.Actor();
actor.connect('destroy', () => {
actor = null;
});
if (actor !== null)
log(actor.width);
I have created a web server i node.js using express and passport. It authenticates using an oauth 2.0 strategy (https://www.npmjs.com/package/passport-canvas). When authenticated, I want to make a call such as:
app.get("/api/courses/:courseId", function(req, res) {
// pass req.user.accessToken implicitly like
// through an IIFE
createExcelToResponseStream(req.params.courseId, res).catch(err => {
console.log(err);
res.status(500).send("Ops!");
});
});
My issue is that i would like, in all subsequent calls from createExcelToResponseStream, to have access to my accessToken. I need to do a ton of api calls later in my business layer. I will call a method that looks like this:
const rq = require("request");
const request = url => {
return new Promise(resolve => {
rq.get(
url,
{
auth: {
bearer: CANVASTOKEN // should be req.user.accessToken
}
},
(error, response) => {
if (error) {
throw new Error(error);
}
resolve(response);
}
);
});
};
If i try to create a global access to the access token, i will risk
race conditions (i think) - i.e. that people get responses in the context of another persons access token.
If i pass the context as a variable i have to refactor a
lof of my code base and a lot of business layer functions have to
know about something they don't need to know about
Is there any way in javascript where i can pass the context, accross functions, modules and files, through the entire callstack (by scope, apply, bind, this...). A bit the same way you could do in a multithreaded environment where you have one user context per thread.
The only thing you could do would be
.bind(req);
But that has has to be chained into every inner function call
somefunc.call(this);
Or you use inline arrow functions only
(function (){
inner=()=>alert(this);
inner();
}).bind("Hi!")();
Alternatively, you could apply all functions onto an Object, and then create a new Instance:
var reqAuthFunctions={
example:()=>alert(this.accessToken),
accessToken:null
};
instance=Object.assign(Object.create(reqAuthFunctions),{accessToken:1234});
instance.example();
You could use a Promise to avoid Race conditions.
Let's have this module:
// ContextStorage.js
let gotContext;
let failedGettingContext;
const getContext = new Promise((resolve,reject)=>{
gotContext = resolve;
failedGettingContext = reject;
}
export {getContext,gotContext, failedGettingContext};
And this inititalization:
// init.js
import {gotContext} from './ContextStorage';
fetch(context).then(contextIGot => gotContext(contextIGot));
And this thing that needs the context:
// contextNeeded.js
import {getContext} from './ContextStorage';
getContext.then(context => {
// Do stuff with context
}
This is obviously not very usable code, since it all executes on load, but I hope it gives you a framework of how to think about this issue with portals... I mean Promises...
The thing that happens when you call the imported 'gotContext', you actually resolve the promise returned by 'getContext'. Hence no matter the order of operations, you either resolve the promise after the context has been requested setting the dependent operation into motion, or your singleton has already a resolved promise, and the dependent operation will continue synchronously.
On another note, you could easily fetch the context in the 'body' of the promise in the 'ContextStorage' singleton. However that's not very modular, now is it. A better approach would be to inject the initializing function into the singleton in order to invert control, but that would obfuscate the code a bit I feel hindering the purpose of the demonstration.
I am new to node js, and have a method in my code which saves information in the following way:
var saveInformation = function(query, update, options) {
asyncTasks.push(function(done) {
what.findOneAndUpdate(query, update, options, done)
})
}
saveInformation(query, update, options)
I call the saveInformation method on several occasions, and need to write to the log files every time this method is called. What is the easiest way of doing this?
I tried just adding my log (I am using winston module) right after, like this:
saveInformation(query, update, options)
logger.info('Hello')
However this sometimes saves and sometimes not (i.e. it will only log this once, however this method is run several times throughout my program).
So I am wondering, is there a way to add another method to the saveInformation line to ensure that they are both run at the same time, so when saveInformation is saved to the dbs, I get a log created too?
The call to the logging function should be in the callback of the function what.findeOneAndUpdate which is done in this case, this way it will execute after each operation.
// callback for findOneAndUpdate
var done = function(err, result) {
if(err) {
throw err
}
logger.info('Result of findOneAndUpdate: ', result)
}
//
// original code
var saveInformation = function(query, update, options) {
asyncTasks.push(function(done) {
what.findOneAndUpdate(query, update, options, done)
})
}
saveInformation(query, update, options)
This way whenever saveInformation is called the logger will be called if the callback is called with no error, you can also modify that to log errors as well if you like.
Probably asked before, but after the serious searching I'm still not able to find a proper solution. Please consider something like this:
function compute() {
asyncCall(args, function(err, result) {
});
/* 'join thread here' */
}
Even though asyncCall is asynchronous I'd like to use the result and return it from the function compute synchronously. asyncCall is a library call and I can't modify it in any way.
How to wait properly for the asynchronous result without setTimeout and watching a conditional variable? This is possible but suboptimal.
not sure how you can really use something that doesn't exist yet, but it's easy enough to return a slot where the result will be:
function compute() {
var rez=[];
asyncCall(args, function(err, result) {
rez[0]=result;
if(rez.onchange){ rez.onchange(result); }
});
/* 'join thread here' */
return rez;
}
now, you can refer to the [0] property of the return, and once the callback comes in, compute()[0] will have the result. It will also fire an event handler you can attach to the returned array that will fire when the data updates inside the callback.
i would use something more formal like a promise or secondary callback, but that's me...
EDIT: how to integrate a callback upstream:
// sync (old and busted):
function render(){
var myView=compute();
mainDiv.innerHTML=myView;
}
//async using my re-modified compute():
function render(){
var that=compute();
that.onchange=function(e){ mainDiv.innerHTML=e; }
}
see how making it wait only added a single wrapper in the render function?
There's no await syntax in browsers that is widely available. Your options are generally limited to Callback patterns or Promises.
NodeJS follows a callback pattern for most async methods.
function someAsyncMethod(options, callback) {
//callback = function(error, data)
// when there is an error, it is the first parameter, otherwise use null
doSomethingAsync(function(){
callback(null, response);
});
}
....
someAsyncMethod({...}, function(err, data) {
if (err) return alert("OMG! FAilZ!");
// use data
});
Another common implementation is promises, such as jQuery's .ajax() method...
var px = $.ajax({...});
px.data(function(data, xhr, status){
//runs when data returns.
});
px.fail(function(err,xhr, status){
//runs when an error occurs
});
Promises are similar to events...
Of the two methods above, the callback syntax tends to be easier to implement and follow, but can lead to deeply nested callback trees, though you can use utility patterns, methods like async to overcome this.