We Keep Coding

Multiple fs.writeFile on Node.js

Straight to the point, I am running an http server in Node.js managing a hotel's check-in/out info where I write all the JSON data from memory to the same file using "fs.writeFile".
The data usually don't exceed 145kB max, however since I need to write them everytime that I get an update from my DataBase, I have data loss/bad JSON format when calls to fs.writeFile happen one after each other immediately.
Currently I have solved this problem using "fs.writeFileSync" however I would like to hear for a more sophisticated solution and not using the easy/bad solution of sync function.
Using fs.promises results in the same error since again I have to make multiple calls to fs.promises.
According to Node's documentation , calling fs.writefile or fs.promises multiple times is not safe and they suggest using a filestream, however this is not currently an option.
To summarize, I need to wait for fs.writeFile to end normally before attempting any repeated write action, and using the callback is not useful since I don't know a priori when a write action needs to be done.
Thank you very much in advance

I assume you mean you are overwriting or truncating the file while the last write request is still being written. If I were you, I would use the promises API and heed the warning from the documentation:
It is unsafe to use fsPromises.writeFile() multiple times on the same file without waiting for the promise to be settled.
You can await the result in a traditional loop, or very carefully use .then() to "synchronize" your callbacks, but if you're not doing anything else in your event loop except reading from your database and writing to this file, you might as well just use writeFileSync to keep things simple/safe. The asynchronous APIs (callback and Promises) are intended to allow your program to do other things in the meantime; if this is not necessary and the async APIs add troublesome complexity for your code, just use the synchronous APIs. That's true for any node API or library function, not just fs.writeFile.
There are also libraries that will perform atomic filesystem operations for you and abstract away the implementation details, but I think these are probably overkill for you unless you describe your use case in more detail. For example, why you're dumping a database to disk as JSON as fast/frequently as you can, rather than keeping things in memory or using event-based incremental updates (e.g. a real, local database with atomicity and consistency guarantees).

thank you for your response!
Since my app is mainly an http server,yes I do other things rather than simply input/output, although with not a great amount of requests. I will review again the promises solution but the first time I had no luck.
To explain more I have a:function updateRoom(data){ ...update things in memory... writetoDisk(); }
and the function writetoDisk(){
fsWriteFile(....)
}
Making the function writetoDisk an async function and implementing "await" inside it still does not solve the problem since the updateRoom function will call the writetoDisk without waiting for it to end.
The ".then" approach can not be implemented since my updateRoom is being called constantly and dynamically .
If you happen to know 1-2 thing about async-await you are more than welcome to explain me a bit more, thanks again nevertheless!

Control Flow. Logic behind callbacks, promises, async/await. Kinda what's the point?

I think I know the answer to this but not sure. Wanting to confirm more than anything. The question applies (I think equally) to all three async approaches (callbacks, promises, async/await), but I'll ask it in the context of promises.
As I understand it, asynchronous programming is intended (at least) for event-driven applications to perform tasks in response to events, without the processes of doing those tasks blocking the ability to do any other tasks (presumably in response to other events) the application. eg. one event might trigger this series of tasks:
Query a database for some data
Wait for the response
Manipulate the data from the response (or handle errors as needed)
Write changes back to the database
done.
In more traditional programming (eg. C/C++) there'd be a main function that calls all that, and might get something back from it. But generally, the process of that main function, sitting around waiting for that to come back, blocks other operations (unless you start manually manipulating threads, or other stuff, that this JS async programming is presumably supposed to spare us from, right?).
But in the above example, supposedly, the (conceptual) "main function" doesn't need anything back from it. Step 5 isn't "return some result I need to act on", it's "done", end of story. If I understand correctly, the above is more likely called by a listener. That listener was set up by JS/node's equivalent of a "main function" -- the code that runs from the entry point of the app -- which has long since ended and the listeners are now running the show. Anything else this entire app needs to do will be triggered by some other independent event, caught by a listener. (Not that this is the only way but as I understand it this is pretty common). So perhaps the user sees the results, and hits another button or whatever to initiate some other, separate, independent, task with it. The above task is long deceased.
Ok... if all that's correct then there's this: For each of those 5 steps, we need to call them in succession. Each step relies on something provided by the one before it. So that process basically needs to be synchronous.
In promise code I believe it looks something like this:
askDbForData() // step 1
.then(responseFromDB => { // step 2
makeTheDesiredChangesToTheData(responseFromDB) // step 3
})
.then(changedData => writeBackToDB(changedData)) // step 4
Looks to me like ultimately this is chaining functions one after the other to perform what is otherwise essentially a synchronous task.
But but but... Synchronous... = blocking?
I've just realized how this doesn't appear to be very clear to me in most of the documentation/articles I've read on this. This is what I need to clear up...
I think the point is: the part where .next(...) picks up the result and sends it off to the next piece in the chain -- that part is "blocking" (though it happens in the blink of an eye so it's kinda moot), but each of those (presumably time consuming) functions (like askDbForData()) -- which are supposed to be asynchronous, and return promises -- do their thing separately and independently of any other control flow, etc, thus not blocking anything.
In other words, the promise chain itself is synchronous, but each piece along the way is asynchronous. The kicker, I think: Not asynchronous to other tasks in the same chain, but asynchronous to everything else the app is doing in other chains initiated by other events.
It might be that anyone reading this will read it and go "yeah well duh, that's the whole point". Perhaps that's what I'm hoping. But if it is, it hasn't been clear in any of my research so far, so it's be great to get it clear from someone(s) who "get" it. I think if all this IS correct, then it pretty much clears up nearly every other confusion I've had with this topic.
So... Yes this is long, but it is only one question. The question is:
Is that it -- is that the point -- or if not, what am I missing?
Thanks!

Looks to me like ultimately this is chaining functions one after the other to perform what is otherwise essentially a synchronous task
No, the code you show there only performs the synchronous part of askDbForData(), then returns. If properly coded, this first part is usually nothing.
I think the point is: the part where .next(...) picks up the result and sends it off to the next piece in the chain -- that part is "blocking" (though it happens in the blink of an eye so it's kinda moot)
Again no, the continuations are only called once the previous step is done. Once a continuation happens, the same split happens, first the synchronous (and hopefully minimal or non-existent) part runs then the rest of the function is registered as a continuation.
In other words, the promise chain itself is synchronous, but each piece along the way is asynchronous.
Absolutely not, the chain of promises is asynchronous. .next() stores a function reference to be called at a later date, it neither calls nor waits for the call of that function code.
The kicker, I think: Not asynchronous to other tasks in the same chain, but asynchronous to everything else the app is doing in other chains initiated by other events.
I've tried to avoid the word "chain" because it seems it's confusing you even more. It's not a chain, it's a tree, you can have promises executed in parallel or sequentially. So again, no.

Consider this.
You have a function A that returns a promise..a promise for something to come.
You also have a function B that does something else.
FunctionA().then(result=>{console.log(result)}).catch();
FunctionB();
If the promise in function A takes a long time to resolve..why not go ahead and do FunctionB while we wait?
Get it?

Abstracting away asynchronicity/promises in javascript?

Suppose I have written a library class:
class ComplexThingDoer implements ThingDoer {
...
doComplexThing(arg1, arg2) {
// lots of complex code here
return finalResult;
}
...
}
And I have made this class available to others, who are using it as intended, completely ignorant of how doComplexThing actually does the complex thing and making use of the finalResult for whatever they're using it for.
Now, suppose I decide that I actually want to reimplement doComplexThing so that instead of synchronously running a lot of complex code here, it makes a request to a specialized server that returns the final result. Or makes a call to a database in which results of the complex thing have been precalculated. Or does one of those things the first time it's called with a particular set of arguments, and caches the result so that the next time it's called with the same arguments it can just return the previously-calculated value.
As far as I can tell, whether I use callbacks, Promises, or async/await, I am screwed - I can't do any of those things without breaking my interface. I have to return a Promise, or accept a callback, or something - there doesn't seem to be any way to just not return until I have the result, the way I've always done before. My callers don't care how I'm getting the result; they just want the result.
Am I missing something? Is there in fact a way to "de-promisify" a function so that its caller doesn't have to know that it's performing an asynchronous operation? Should I be pre-emptively writing everything as Promises just in case someday I might want to reimplement it in an asynchronous way (sounds like a terrible idea)?
Thanks!

Nope, you're not missing anything. If you make your function asynchronous, then every caller of it will have to treat it as such and also be asynchronous itself. The result of an asynchronous call will be available sometime later, on subsequent iterations of the underlying event loop. There is absolutely no way to block synchronous code to wait for that, because if you block in code the event loop won't advance and the result will never become available.
Switching an interface from synchronous to asynchronous is a breaking change.

Node.js asynchronous use cases

Recently, I have been developing web application and I realize that I am not making use of the asynchronous property at all. Hence I am ending up with a lot of nested callbacks.
For example, if the user want to get a file from the server through a particular API, I will have code similar to this,
db.query(<select list of permitted files_names>, function(err, filenames) {
async.each(file_names, function(name, next) {
//open each file to put into array
});
})
This code needs to query database to get a list of file names before looping asynchronously and putting each file content into an array. Finally it will return the finished array to the client.
With the nested callback, and async library, this code is behaving like a synchronous code.
names = db.querySync(//select list of permitted files_names);
for(name in names) {
//open each file to put into array
}
I am better off writing synchronous code like this since it is much neater. My use case might be a little strange but most of my api behaves in similar manner and that makes me think why do I even need asynchronous function?
Can someone please enlighten me if there are any differences between these two codes in term of performance? How do I make use of non-blocking property to enhance the performance in this use case?

If you're writing callback functions you're using by definition using async calls. The callback function fires only when the operation is complete or has errored out. You don't need a fancy library to use these, this is the backbone of how Node's event-loop driven subsystem operates.
Node strongly advises against using "Sync" calls. The Node core only includes a handful as a convenience, they're there as last-resort tools. Many libraries don't even support them so you absolutely must get used to writing async code. In the browser environment, for example, you simply cannot use blocking calls without jamming up the JavaScript runtime and stalling the page.
I prefer using Promises line Bluebird implements to keep code orderly. There are other ways, like the async library, which can help manage otherwise complicated nesting patterns.
Some of the perks include things like Promise.all method runs a series of promises to completion and then triggers a next step, and Promise.map which iterates over a list, running async code for each element, then advancing when the list is complete.
If you're disciplined about organizing your code it's not too bad. Node does require a lot more attention being paid to the order of operations than in a traditional sync-by-default language like Ruby, Python or Java, but you can get used to it. Once you start working with async code rather than fighting it you can often do a ton of work quickly, efficiently, and with a minimum of fuss, in many cases more effectively than in other languages where you must juggle threads plus locking and/or deal with IPC.

Yes, there is a difference in the two codes in terms of performance.
In synchronous code:
names = db.querySync(//select list of permitted files_names);
you are calling the database here to give list of names. Assume , this takes 10 sec. So for this time, nodeJS as it is single threaded gos into blocking state. After 10 sec, it executes the rest of the code . Assume this for loop takes 5 sec and some code takes 5 sec.
for(name in names) {
//open each file to put into array
}
//some code
Therefore it takes a total time of 20 sec.
whereas in Asynchronous code:
db.query(<select list of permitted files_names>, function(err, filenames) {
NodeJs will ask the database to give list of names to a callback. Assume that it takes 10 sec. And immediately it goes into the next step(some code), but not into the blocking state. Assume that some code takes 5 sec.
async.each(file_names, function(name, next) {
//open each file to put into array
});
})
//some code.
After 5 sec, it will check whether it has an i/o operations to be performed. Once the call back is returned. It will execute the function(name, next) {..} for the 5 sec.
So the total time here is 15sec.
In this manner the performance is improved.
If the asynchronous code should be clear and neat then make use of closures & promises.
For ex: Above asynchronous code can be written as
fun = function(err, filenames) {
async.each(file_names, function(name, next) {
//open each file to put into array
}
db.query(<select list of permitted files_names>, fun);

The benefit is simple: By using asynchronous code, the current thread (remember, Node.js is single-threaded) is able to handle other requests while the current request is waiting on something (like a database query) to return.
If you use synchronous code instead, the current thread will block while it waits, and it won't be able to handle other requests in the meantime. In other words, you lose concurrency.
To keep your asynchronous code clean, look into promises (to avoid deeply nested callbacks) and ES7 async/await (to avoid callbacks at all and write asynchronous code that looks just like synchronous code).

Forever loop while waiting for asynchronous task?

I'm wondering if there's a way to cause JavaScript to wait for some variable-length code execution to finish before continuing using events and loops. Before answering with using timeouts, callbacks or referencing this as a duplicate, hear me out.
I want to expose a large API to a web worker. I want this API to feel 'native' in the sense that you can access each member using a getter which gets the information from the other thread. My initial idea was to compile the API and rebuild the entire object on the worker. While this works (and was a really fun project), it's slow at startup and cannot show changes made to the API without it being sent to the worker again after modification. Observers would solve part of this, and web workers transferrable objects would solve all, but they aren't adopted widely yet.
Since worker round-trip calls happen in a matter of milliseconds, I think stalling the thread for a few milliseconds may be an alright solution. Of course I would think about terminating in cases where calls take too long, but I'm trying to create a proof of concept first.
Let's say I want to expose the api object to the worker. I would define a getter for self.api which would fetch the first layer of properties. Each property would then be another getter and the process would continue until the final object is found.
worker.js
self.addEventListener('message', function(event) {
self.dataRecieved = true;
self.data = event.data; // would actually build new getters here
});
Object.defineProperty(self, 'api', {
get: function() {
self.dataRecieved = false;
self.postMessage('request api first-layer properties');
while(!self.dataRecieved);
return self.data; // whatever properties were received from host
}
});
For experimentation, we'll do a simple round-trip with no data processing:
index.html (only JS part)
var worker = new Worker("worker.js");
worker.onmessage = function() {
worker.postMessage();
};
If onmessage would interrupt the loop, the script should theoretically work. Then the worker could access objects like window.document.body.style on the fly.
My question really boils down to: is there a way to guarantee that an event will interrupt an executing code block?
From my understanding of events in JavaScript, I thought they did interrupt the current thread. Does it not because it's executing a blank statement over and over? What if I generated code to be executed and kept doing that until the data returned?

is there a way to guarantee that an event will interrupt an executing code block
As #slebetman suggests in comments, no, not in Javascript running in a browser's web-worker (with one possible exception that I can think of, see suggestion 3. below).
My suggestions, in decreasing order of preference:
Give up the desire to feel "native" (or maybe "local" might be a better term). Something like the infinite while loop that you suggest also seems to be very much fighting agains the cooperative multitasking environment offered by Javascript, including when thinking about a single web worker.
Communication between workers in Javascript is asynchronous. Perhaps it can fail, take longer than just a few milliseconds. I'm not sure what your use case is, but my feeling is that when the project grows, you might want to use those milliseconds for something else.
You could change your defined property to return a promise, and then the caller would do a .then on the response to retrieve the value, just like any other asynchronous API.
Angular Protractor/Webdriver has an API that uses a control flow to simulate a synchronous environment using promises, by always passing promises about. Taking the code from https://stackoverflow.com/a/22697369/1319998
browser.get(url);
var title = browser.getTitle();
expect(title).toEqual('My Title');
By my understanding, each line above adds a promise to the control flow to execute asynchronously. title isn't actually the title, but a promise that resolves to the title for example. While it looks like synchronous code, the getting and testing all happens asynchronously later.
You could implement something similar in the web worker. However, I do wonder whether it will be worth the effort. There would be a lot of code to do this, and I can't help feeling that the main consequence would be that it would end up harder to write code using this, and not easier, as there would be a lot of hidden behaviour.
The only thing that I know of that can be made synchronous in Javascript, is XMLHttpRequest when setting the async parameter to false https://developer.mozilla.org/en-US/docs/Web/API/XMLHttpRequest#Parameters. I wonder if you could come up with some sort of way to request to the server that maintains a connection with the main thread and pass data along that way. I have to say, my instinct is that this is quite an awful idea, and would be much slower than just requesting data from the main thread.

For what I know, there is not something native in JS to do this but it is relatively easy to do something similar. I made one some time ago for myself: https://github.com/xpy/whener/blob/master/whener.js .
You use it like when( condition, callback ) where condition is a function that should return true when your condition is met, and callback is the function that you want to execute at that time.