We Keep Coding

Multiple fs.writeFile on Node.js

Straight to the point, I am running an http server in Node.js managing a hotel's check-in/out info where I write all the JSON data from memory to the same file using "fs.writeFile".
The data usually don't exceed 145kB max, however since I need to write them everytime that I get an update from my DataBase, I have data loss/bad JSON format when calls to fs.writeFile happen one after each other immediately.
Currently I have solved this problem using "fs.writeFileSync" however I would like to hear for a more sophisticated solution and not using the easy/bad solution of sync function.
Using fs.promises results in the same error since again I have to make multiple calls to fs.promises.
According to Node's documentation , calling fs.writefile or fs.promises multiple times is not safe and they suggest using a filestream, however this is not currently an option.
To summarize, I need to wait for fs.writeFile to end normally before attempting any repeated write action, and using the callback is not useful since I don't know a priori when a write action needs to be done.
Thank you very much in advance

I assume you mean you are overwriting or truncating the file while the last write request is still being written. If I were you, I would use the promises API and heed the warning from the documentation:
It is unsafe to use fsPromises.writeFile() multiple times on the same file without waiting for the promise to be settled.
You can await the result in a traditional loop, or very carefully use .then() to "synchronize" your callbacks, but if you're not doing anything else in your event loop except reading from your database and writing to this file, you might as well just use writeFileSync to keep things simple/safe. The asynchronous APIs (callback and Promises) are intended to allow your program to do other things in the meantime; if this is not necessary and the async APIs add troublesome complexity for your code, just use the synchronous APIs. That's true for any node API or library function, not just fs.writeFile.
There are also libraries that will perform atomic filesystem operations for you and abstract away the implementation details, but I think these are probably overkill for you unless you describe your use case in more detail. For example, why you're dumping a database to disk as JSON as fast/frequently as you can, rather than keeping things in memory or using event-based incremental updates (e.g. a real, local database with atomicity and consistency guarantees).

thank you for your response!
Since my app is mainly an http server,yes I do other things rather than simply input/output, although with not a great amount of requests. I will review again the promises solution but the first time I had no luck.
To explain more I have a:function updateRoom(data){ ...update things in memory... writetoDisk(); }
and the function writetoDisk(){
fsWriteFile(....)
}
Making the function writetoDisk an async function and implementing "await" inside it still does not solve the problem since the updateRoom function will call the writetoDisk without waiting for it to end.
The ".then" approach can not be implemented since my updateRoom is being called constantly and dynamically .
If you happen to know 1-2 thing about async-await you are more than welcome to explain me a bit more, thanks again nevertheless!

Where does the Javascript asynchronicity REALLY comes from?

I mean, this may seem like a silly question, but I don't think it really is. In programming languages where we can use threads, the concept is very simple. But Javascript is single-threaded and handles events through callbacks. What really makes something "branch-out" the execution process (I know this is the wrong technical term but I hope you understand my point). Is it the timers? Like setTimeout and setInterval?
I mean, if I were to write a Database with JS (I know JS wouldn't fit it but please bear with me for the sake of the example) and I need to check for some hardware operations to be completed before calling a callback function to handle the completion of such operation I cannot for the sake of me think in a possible way to actually make this 'pseudo-code' async. I mean:
function op() {
while (someHardwareOpIsNotCompleted()) doWhatever();
callback("done");
}
How can I make this function run asynchronously without using some form of timer that refers and 'counts' to a different thread? In my mind the only way around it is to pass to setTimeout the function reference as such setTimeout(op). From here on I can implement my callbacks and listeners, or the syntax-sugar Promise or even the syntax sugar-sugar "async".
Hopefully I was able to get my point across and I don't get replies like: "Use promises!" or something of the sort. If the real structure has nothing to do with timers but only simple callbacks, can someone give me an example on how to achieve a non-blocking execution of some piece of code without timers?
Thanks

A function is asynchronous for one of two reasons.
It needs to happen later (e.g. when a timeout has finished or when a click happens)
It takes a long time and shouldn't tie up the JS engine while it runs (e.g. accessing a database / file / network resource)
The asynchronous parts of the latter functions are not written in JavaScript. They are provided by the host environment and are usually written in the same language a that (e.g. C). The host environment exposes an API to provide access to them to the JavaScript.
For example, the source code for Chrome's implementation of XMLHttpRequest is written in C++.
If you need to poll something, then testing it on a timer is the usual way.

Performance impact during the multiple callback functions after sending response

Can anyone help me understand the function of NodeJS and performance impact for the below scenario.
a. Making the request to Rest API end point "/api/XXX". In this request, i am returning the response triggering the asynchronous function like below.
function update(req, res) {
executeUpdate(req.body); //Asynchronous function
res.send(200);
}
b. In this, I send the response back without waiting for the function to complete and this function executing four mongodb updates of different collection.
Questions:
As I read, the NodeJS works on the single thread, how this
asynchronous function is executing?
If there are multiple requests for same end point, how will be the
performance impact of NodeJS?
How exactly the NodeJS handles the asynchronous function of each
request, because as the NodeJS is runs on the single thread, is there
any possibility of the memory issue?

In short, it depends on what you are doing in your function.
The synchronous functions in node are executed on main thread, thus,
they will not preempt and execute until end of the function or until
return statement is encountered.
The async functions, on the other hand, are removed from main thread,
and will only be executed when async tasks are completed on a
separate worker thread.

There are, I think, two different parts in the answer to your question.
Actual Performance - which includes CPU & memory performance. It also obviously includes speed.
Understanding as the previous poster said, Sync and Async.
In dealing with #1 - actual performance the real only way to test it is to create or use a testing environment on your code. In a rudimentary way based upon the system you are using you can view some of the information in top (linux) or Glances will give you a basic idea of performance, but in order to know exactly what is going on you will need to apply some of the various testing environments or writing your own tests.
Approaching #2 - It is not only sync and async processes you have to understand, but also the ramifications of both. This includes the use of callbacks and promises.
It really all depends on the current process you are attempting to code. For instance, many Node programmers seem to prefer using promises when they make calls to MongoDB, especially when one requires more than one call based upon the return of the cursor.
There is really no written-in-stone formula for when you use sync or async processes. Avoiding callback hell is something all Node programmers try to do. Catching errors etc. is something you always need to be careful about. As I said some programmers will always opt for Promises or Async when dealing with returns of data. The famous Async library coupled with Bluebird are the choice of many for certain scenarios.
All that being said, and remember your question is general and therefore so is my answer, in order to properly know the implications on your performance, in memory, cpu and speed as well as in return of information or passing to the browser, it is a good idea to understand as best as you can sync, async, callbacks, promises and error catching. You will discover certain situations are great for sync (and much faster), while others do require async and/or promises.
Hope this helps somewhat.

Javascript + selenium, writing readable tests

I'm writing tests for my web application with selenium webdriver for javascript. The problem that I'm facing is that because many operations are asynchronous, the indentation goes crazy. For example, for reading a value of an element, I'm using a structure
driver.findElement(By.id('my-element')).then(function(elem) {
elem.getAttribute('innerHTML').then(function(text) {
// Some operation
// Read next element with same structure
});
});
As you can see, if I need to read values of multiple elements, the indentation gets very deep quickly. Are there some kind of best practices to avoid this kind of issue? Is using "then" the only way to read values from elements?

If your concern is all about writing readable test then you should have some kind of code quality control tool in your codebase.
I prefer JSLint, which helps me to write cleaner code. It follows the standards of ECMAScript Programming Language and complains me if find any issues.

Node app.js - Reading files synchronously?

I've been reading through example applications try to learning node. And I've noticed that several use the readdirSync method when loading models and controllers on boot().
For instance:
var models_path = __dirname + '/app/models'
var model_files = fs.readdirSync(models_path)
model_files.forEach(function(file){
if (file == 'user.js')
User = require(models_path+'/'+file)
else
require(models_path+'/'+file)
})
This seems anti-node to me. It's the opposite of the "try-to-make-everything-async" that node favors.
When and why might synchronous file reads like this be a good idea?

More than likely, to make initialisation more simple—when asynchronicity for speed's sake doesn't matter; we're not trying to service many concurrent requests.
Similarly, if you need access to some variable which you're initialising on startup, which will be used for the life-time of the application, you don't want to have to wrap your entire app in a callback!

Synchronous reads are needed when you have to be certain that all of the data is available before you continue AND you need to keep a sequence in order. In other words if you NEED to have the process blocked and unable to do anything else (for anyone) such as when you are starting up a server (e.g. reading the certificate files for HTTPS).
Synchronous reads may be desirable at other time to keep the coding simpler as Len has suggested. But then you are trading off simplicity with performance as you suggest. In fact, it is better to make use of one of the many sequencing helper libraries in this case. These greatly simplify your code by taking care of nested callbacks and sequence issues.
And of course, the code you provide as an example is rather dangerous - what happens if the read fails?
Here are 3 of the libraries:
Streamline.js Allows you to write async js/coffeescript as though it were sync. Simply replace the callbacks with '_'. BUT you either have to compile your scripts or run them via a loader.
async - seems to be about the best thought out and documented and is recommended by a couple of people who have built real-world apps.
async.js - Chainable, exposes fs as well (includes readdir, walkfiles, glob, abspath, copy, rm - focused on fs rather than generic
This link may also be of use: The Tale of Harry - An explanation of how a mythical programmer moves from traditional programming to callback-based & the patterns that he ends up using. Also a useful insight into the patterns presented in the async library.