I currently have a folder that has two subfolders and about 500 files, totalling ~3GB. In my Electron application I perform a backup of that folder by essentially copy and pasting it to a different drive through NodeJS. However, this process takes about 3 mins when I copy and paste via Windows but around 15 mins when I use NodeJS.
I am using fs-extra to copy the whole folder over:
const path = require('path');
const fs = require('fs-extra');
var source = 'some/source/directory'
var destination = path.join('my/backup/directory','name_of_folder_after_backup')
// copy source folder to destination
await fs.copy(source, destination);
I did consider making this an asynchronous operation with just fs.writeFile() but considering this is running on an HDD at worst, I figured firing 500 write operations and waiting for it to complete might not be the best idea.
My questions are:
Is the the proper way to use NodeJS to do a folder copy operation like this?
Why is the Windows copy operation so much faster than the NodeJS copy operation?
How to reduce the copy time of the folder? Is there some native Electron API I might be missing?
I'm not an expert in this area so I'd appreciate if your answer had some sources I could look at or a bit of a detailed explanation.
Thanks to the comments above I got a solution. I used robocopy on windows to do the copy. Node spawns a child process, launches Robocopy, wraps it in a promise and then awaits it.
const path = require("path");
const util = require("util");
const exec = util.promisify(require("child_process").exec);
const source = 'your/source/dir/here';
const destination = path.join('temp', "PDES-Backup");
// copy source folder to destination
//await fs.copySync(source, destination);
const copyCommand = `Robocopy "${source}" "${destination}" /MIR /LOG:backuplog-${new Date()
.toISOString()
.substr(0, 10)}-${new Date().getTime()}.txt`;
const promise = exec(copyCommand, { stdio: "inherit" });
const child = promise.child;
child.stdout.on("data", (data) => {
console.log("stdout: " + data);
});
child.stderr.on("data", (data) => {
console.log("stderr: " + data);
});
child.on("error", (error) => {
console.error(error);
});
child.on("close", (code) => {
console.warn("closing code: " + code);
});
// i.e. can then await for promisified exec call to complete
const { stdout, stderr } = await promise;
For some reason the first copy gives the wrong closing code but the copy gets completed just fine
Related
I am trying the following code (from sample of parquetjs-lite and stackoverflow) to read a parquet file in nodejs :
const readParquetFile = async () => {
try {
// create new ParquetReader that reads from test.parquet
let reader = await parquet.ParquetReader.openFile('test.parquet');
}
catch (e){
console.log(e);
throw e;
}
// create a new cursor
let cursor = reader.getCursor();
// read all records from the file and print them
let record = null;
while (record = await cursor.next()) {
console.log(record);
}
await reader.close();
};
When I run this code nothing happens . There is nothing written to the console, for testing purpose I have only used a small csv file which I converted using python to parquet.
Is it because I have converted from csv to parquet using python (I couldn't find any JS equivalent for large files on which I have to ultimately be able to use).
I want my application to be able to take in any parquet file and read it. Is there any limitation for parquetjs-lite in this regard.
There are NaN values in my CSV could that be a problem ?
Any pointers would be helpful.
Thanks
Possible failure cases are
you are calling this function in some file without a webserver running.
In this case the file will run in async mode and as async function goes in callback stack and your main stack is empty the program will end and even is you have code in your call stack it will never run or log anything.
To solve this try running a webserver or better use sync calls
//app.js (without webserver)
const readParquetFile = async () => {
console.log("running")
}
readParquetFile()
console.log("exit")
when you run the above code the output will be
exit
//syncApp.js
const readParquetFile = () => {
console.log("running")
// all function should be sync
}
readParquetFile()
console.log("exit")
here the console log will be
running
exit
I have a director data and it contains 10 files and I'll be adding some files to it. I'd like to automatically bring in these files when the node server boots up using something like
let Test = require("./data/Test");
I wrote the following, when I loop over it, I see the filenames.
fs.readdir(__dirname + "/data", (err, files) => {
files.forEach((file) => {
console.log("file", file);
let fileName = file.split(".")[0];
global[fileName] = require("./data/" + fileName);
console.log(global[fileName]);
});
});
If I put a console log on the page and wait 5 seconds, then the data is indeed there. How can I get the contents of this file BEFORE the server finishes starting?
I have a local JSON file which I intent to read/write from a NodeJS electron app. I am not sure, but I believe that instead of using readFile() and writeFile(), I should get a FileHandle to avoid multiple open and close actions.
So I've tried to grab a FileHandle from fs.promises.open(), but the problem seems to be that I am unable to get a FileHandle from an existing file without truncate it and clear it to 0.
const { resolve } = require('path');
const fsPromises = require('fs').promises;
function init() {
// Save table name
this.path = resolve(__dirname, '..', 'data', `test.json`);
// Create/Open the json file
fsPromises
.open(this.path, 'wx+')
.then(fileHandle => {
// Grab file handle if the file don't exists
// because of the flag 'wx+'
this.fh = fileHandle;
})
.catch(err => {
if (err.code === 'EEXIST') {
// File exists
}
});
}
Am I doing something wrong? Are there better ways to do it?
Links:
https://nodejs.org/api/fs.html#fs_fspromises_open_path_flags_mode
https://nodejs.org/api/fs.html#fs_file_system_flags
Because JSON is a text format that has to be read or written all at once and can't be easily modified or added onto in place, you're going to have to read the whole file or write the whole file at once.
So, your simplest option will be to just use fs.promises.readFile() and fs.promises.writeFile() and let the library open the file, read/write it and close the file. Opening and closing a file in a modern OS takes advantage of disk caching so if you're reopening a file you just previously opened not long ago, it's not going to be a slow operation. Further, since nodejs performs these operations in secondary threads in libuv, it doesn't block the main thread of nodejs either so its generally not a performance issue for your server.
If you really wanted to open the file once and hold it open, you would open it for reading and writing using the r+ flag as in:
const fileHandle = await fsPromises.open(this.path, 'r+');
Reading the whole file would be simple as the new fileHandle object has a .readFile() method.
const text = await fileHandle.readFile({encoding 'utf8'});
For writing the whole file from an open filehandle, you would have to truncate the file, then write your bytes, then flush the write buffer to ensure the last bit of the data got to the disk and isn't sitting in a buffer.
await fileHandle.truncate(0); // clear previous contents
let {bytesWritten} = await fileHandle.write(mybuffer, 0, someLength, 0); // write new data
assert(bytesWritten === someLength);
await fileHandle.sync(); // flush buffering to disk
First, I split the video into frames with ffmpeg (on the Converter server, Windows), after that I create an archive that I need to transfer to another server (Generator server, Linux), where the archive with video and/or image, frames, and the data.json file should be unzipped, and the contents of the archive should be used further.
For unzipping, I took the extract-zip library (based on yauzl) and used promisify from the util module.
The problem is that when I send an image to the Converter for processing, everything works as it should; however, when I send a video, I get this error - 'end of central directory record signature not found'.
It seems to me that the problem is related to asynchrony or the event-loop.
I tried not to use promisify, but then I have the following problem - at the moment when I need to use data from the archive, this archive does not yet exist ('folder do not exist');
Tried some more libraries for unarchiving adamzip, unzip, decompress, node-zip;
I tried to make a chain of promises;
...
const fs = require('fs-extra')
const path = require('path')
cosnt util = require('util')
const extractZip = util.promisify(require('extract-zip'))
...
...
/**
* Factory that creates a project based on the archive.
* #param {string} archive Path to the source archive
* #returns {Promise.<Project>}
*/
static async create (archive) {
const dir = fs.mkdtempSync(path.join(this.baseDir, '/'))
await extractZip(archive, { dir })
const data = await fs.readJson(path.join(dir, 'data.json'))
return new this(data, dir)
}
...
/* Tried to do this w/ chaining promises:
static async create (archive) {
const dir = fs.mkdtempSync(path.join(this.baseDir, '/'))
await extractZip(archive, { dir })
.then( async () => {
const data = await fs.readJson(path.join(dir, 'data.json'))
return data
})
.then( async (data) => {
return new this(data, dir)
})
}
*/
I solved the problem. It turned out that I had been downloading a large archive from the server asynchronously, so correcting this moment, everything worked as it should.
Wrong:
fs.writeFile(...)
Correct:
fs.writeFileSync(...)
Conclusion: faced with a similar problem - you should go through the entire chain of data transfer and functions/call stack that is involved in it.
I am using lockfile.locksync to lock a file in node.js. But I'd like to know the complete mechanism of this utility. So far every website says that its a "very polite lock file utility" but none explains the internal mechanism of this utility.
Any suggestions?
NODE FS
To understand file options in Node.js, take a look here #fs_file_system_flags and Linux open() syscall which covers options like
O_TRUNC - truncate existing file
O_CREAT - create if not exists
O_WRONLY - access mode, write-only
O_EXCL - ensure that this call creates the file. In pair with O_CREAT triggers EEXIST error if file exists.
But it doesn't offer file lock alternatives similar to "flock"
NPM LOCKFILE INVESTIGATING
I tries to use npm library called "lockfile" as there are too much copy/pasted examples with it.
Verdict. "lockfile" npm library in current implementation "v1.0.4" is incomplete/useless!
From the source:
exports.lockSync = function (path, opts) {
opts = opts || {}
opts.req = opts.req || req++
debug('lockSync', path, opts)
if (opts.wait || opts.retryWait) {
throw new Error('opts.wait not supported sync for obvious reasons')
}
try {
var fd = fs.openSync(path, wx)
locks[path] = fd
try { fs.closeSync(fd) } catch (er) {}
debug('locked sync!', path, fd)
return
So that 5 lines of actual code in "try{}" block just "openSync" in "wx" mode, save "path" to "locks[path]" and "closeSync".
If file exists it fails! due to "wx"!
YOU CAN'T LOCK THE EXISTING FILE with "lockfile" library!
Finally it tries to call "unlockSync" with onExit signal with this code:
exports.unlockSync = function (path) {
debug('unlockSync', path)
// best-effort. unlocking an already-unlocked lock is a noop
try { fs.unlinkSync(path) } catch (er) {}
delete locks[path]
}
And yes, it will DELETE your file after process exited!
This is not the file locking mechanics!
ANY RELIABLE SOLUTION FOR NODEJS FILE LOCK?
I've found fs-ext works just perfect! Open 2 tabs with node process and check what locking file means:
tab1
const fs = require("fs");
const {flockSync} = require('fs-ext');
const fd = fs.openSync('1.txt', 'w');
flockSync(fd, 'ex');
tab2
const fs = require("fs");
const {flockSync} = require('fs-ext');
const fd = fs.openSync('1.txt', 'r');
flockSync(fd, 'sh'); // PENDING!
In tab2 flockSync(fd, 'sh'); will pending until in tab1 flockSync(fd, 'un'); is called!
It really works!