I'm trying to write a little commander script that proxies to other scripts in my project and I'm trying to use node to pipe the stdout from the spawned process to the current process's stdout:
function runCommand(command, arguments) {
var commandProcess = childProcess.spawn(command, arguments);
commandProcess.stdout.pipe(process.stdout);
commandProcess.on("exit", process.exit);
}
this works fine until I start getting large output from my sub processes (for example one of them is a maven command). What I'm seeing is that it only prints out the first 8192 bytes of the stdout and then stores the rest until the next "data" event. Then it prints out the next 8192 etc. That means there's a lag in the output and at times when we're running a server process sometimes it stops printing things out until you trigger something on the server that triggers another "data" event.
Is there a way to increase the size of this buffer or avoid this behavior? Ideally this commander script just proxies to our other scripts and should print out everything exactly as is.
You are using node process spawn which is asynchronously asynchronous so it will give output as an when the child process gives stdout.
Ref: [https://nodejs.org/api/child_process.html#child_process_child_process_spawn_command_args_options][1]
I recommend to use child process exec to run your process where you can control the size of output buffer, which will give output after the child process is finished.This is how you pass buffer size
var execute = function(command, callback){
exec(command, {maxBuffer: 1024 * 500}, function(error, stdout, stderr){ callback(error, stdout); });
};
Related
tell me please how i can slice audio file with using node.js? Now i read the documentation for ffmpeg module, but don't understand how to slice audio file with using this module.
I found this code, but it gives an error error: NaN
ffmpeg('music/ant.mp3')
.setStartTime('00:00:03')
.setDuration('10')
.output('music/ant.mp3')
.on('end', function(err) {
if(!err)
{
console.log('conversion Done');
}
})
.on('error', function(err){
console.log('error: ', +err);
}).run();
You could probably use ffmpeg. It is command line based, which is great since it is accessible, but subprocesses can sometimes die. Here is a node interface that abstracts ffmpeg usage out of command line calls: https://npmjs.org/package/ffmpeg
Your final commands, in the command line would probably look like this:
ffmpeg -i long.mp3 -acodec copy -ss 00:00:00 -t 00:30:00 half1.mp3
ffmpeg -i long.mp3 -acodec copy -ss 00:30:00 -t 00:60:00 half2.mp3
This command states:
- -i: the input file is long.mp3
- -acodec: use the audio codec
- copy: we are making a copy
- -ss: start time
- -t: length
- and finally the output file name
To handle potentially timeouts/hung processes you should 'retry' and supply timeouts. Not sure how well the error callbacks work. That is do they fail appropriately on a process that hangs.
I did it using child_process
https://nodejs.org/api/child_process.html
const spawn = require('child_process').spawnSync;
var ffmpeg = spawn('ffmpeg', [
'-i',
filename,
'-acodec',
'copy',
'-ss',
ss,
'-t',
t,
output.mp3
], { shell: true });
A few things to note:
Ensure you have ffmpeg installed then spawn with the desired command. Put the arguments in an array. spawn does weird things with quotes but {shell:true} should take care of that.
Create the desired output directory before hand or with another spawn because ffmpeg won't create it for you if it doesn't already exist.
Check the process with this:
process.on('exit', (code, signal) => {
console.log(`ffmpeg exited with code ${code} and signal ${signal}`);
});
Not a lot of info here, but from the looks of it you're passing strings into the .setStartTime and .setDuration methods. NaN means not a number, if you didn't know
'00:00:03' is a String, as is '10' because you put quotes around them. Even if you remove the quotes from the '00:00:03', it's not going to be able to parse that as a number. You probably need to pass it the value as some sort of integer, but it may not be number of seconds, it might be milliseconds or such. If it were seconds, you'd put '3' (without quotes), or milliseconds, you'd put '3000' without quotes. Also, if the unit of time to be passed is not seconds but milliseconds or somesuch, you will probably have to change the value '10' (again without quotes) to '10000' (no quotes) for milliseconds, etc.
I want to create Node.js module which should be able to parse huge binary files (some larger than 200GB). Each file is divided into chunks and each chunk can be larger than 10GB. I tried using flowing and non-flowing methods to read file, but the problem is because the end of the readed buffer is reached while parsing chunk, so parsing of that chunk must be terminated before the next onData event occurs. This is what I've tried:
var s = getStream();
s.on('data', function(a){
parseChunk(a);
});
function parseChunk(a){
/*
There are a lot of codes and functions.
One chunk is larger than buffer passed to this function,
so when the end of this buffer is reached, parseChunk
function must be terminated before parsing process is finished.
Also, when the next buffer is passed, it is not the start of
a new chunk because the previous chunk is not parsed to the end.
*/
}
Loading whole chunk into process memory isn't prossible because I have only 8GB of RAM. How can I synchronously read data from the stream or how can I pause parseChunk function when the end of the buffer is reached and wait until new data is available?
Maybe I'm missing something, but as far as I can tell, I don't see a reason why this couldn't be implemented using streams with a different syntax. I'd use
let chunk;
let Nbytes; // # of bytes to read into a chunk
stream.on('readable', ()=>{
while(chunk = stream.read(Nbytes)!==null) {
// call whatever you like on the chunk of data of size Nbytes
}
})
Note that if you specify the size of the chunk yourself, like done here, null will be returned if the amount of bytes requested are not available at the end of the stream. This doesn't mean there is no data anymore to stream. So just be aware that you should expect back a 'trimmed' buffer object of size < Nbytes at the end of the file.
I just want to call an external exe from a nodejs-App. This external exe makes some calculations and returns an output the nodejs-App needs. But I have no idea how to make the connection between nodejs and an external exe. So my questions:
How do I call an external exe-file with specific arguments from within nodejs properly?
And how do I have to transmit the output of the exe to nodejs efficiently?
Nodejs shall wait for the output of the external exe. But how does nodejs know when the exe has finished its processing? And then how do I have to deliver the result of the exe? I don't want to create a temporary text-file where I write the output to and nodejs simply reads this text-file. Is there any way I can directly return the output of the exe to nodejs? I don't know how an external exe can directly deliver its output to nodejs. BTW: The exe is my own program. So I have full access to that app and can make any necessary changes. Any help is welcome...
With child_process module.
With stdout.
Code will look like this
var exec = require('child_process').exec;
var result = '';
var child = exec('ping google.com');
child.stdout.on('data', function(data) {
result += data;
});
child.on('close', function() {
console.log('done');
console.log(result);
});
You want to use child_process, you can use exec or spawn, depending on your needs. Exec will return a buffer (it's not live), spawn will return a stream (it is live). There are also some occasional quirks between the two, which is why I do the funny thing I do to start npm.
Here's a modified example from a tool I wrote that was trying to run npm install for you:
var spawn = require('child_process').spawn;
var isWin = /^win/.test(process.platform);
var child = spawn(isWin ? 'cmd' : 'sh', [isWin?'/c':'-c', 'npm', 'install']);
child.stdout.pipe(process.stdout); // I'm logging the output to stdout, but you can pipe it into a text file or an in-memory variable
child.stderr.pipe(process.stderr);
child.on('error', function(err) {
logger.error('run-install', err);
process.exit(1); //Or whatever you do on error, such as calling your callback or resolving a promise with an error
});
child.on('exit', function(code) {
if(code != 0) return throw new Error('npm install failed, see npm-debug.log for more details')
process.exit(0); //Or whatever you do on completion, such as calling your callback or resolving a promise with the data
});
I'm trying to implement a routine for Node.js that would allow one to open a file, that is being appended to by some other process at this very time, and then return chunks of data immediately as they are appended to file. It can be thought as similar to tail -f UNIX command, however acting immediately as chunks are available, instead of polling for changes over time. Alternatively, one can think of it as of working with a file as you do with socket — expecting on('data') to trigger from time to time until a file is closed explicitly.
In C land, if I were to implement this, I would just open the file, feed its file descriptor to select() (or any alternative function with similar designation), and then just read chunks as file descriptor is marked "readable". So, when there is nothing to be read, it won't be readable, and when something is appended to file, it's readable again.
I somewhat expected this kind of behavior for following code sample in Javascript:
function readThatFile(filename) {
const stream = fs.createReadStream(filename, {
flags: 'r',
encoding: 'utf8',
autoClose: false // I thought this would prevent file closing on EOF too
});
stream.on('error', function(err) {
// handle error
});
stream.on('open', function(fd) {
// save fd, so I can close it later
});
stream.on('data', function(chunk) {
// process chunk
// fs.close() if I no longer need this file
});
}
However, this code sample just bails out when EOF is encountered, so I can't wait for new chunk to arrive. Of course, I could reimplement this using fs.open and fs.read, but that somewhat defeats Node.js purpose. Alternatively, I could fs.watch() file for changes, but it won't work over network, and I don't like an idea of reopening file all the time instead of just keeping it open.
I've tried to do this:
const fd = fs.openSync(filename, 'r'); // sync for readability' sake
const stream = net.Socket({ fd: fd, readable: true, writable: false });
But had no luck — net.Socket isn't happy and throws TypeError: Unsupported fd type: FILE.
So, any solutions?
UPD: this isn't possible, my answer explains why.
I haven't looked into the internals of the read streams for files, but it's possible that they don't support waiting for a file to have more data written to it. However, the fs package definitely supports this with its most basic functionality.
To explain how tailing would work, I've written a somewhat hacky tail function which will read an entire file and invoke a callback for every line (separated by \n only) and then wait for the file to have more lines written to it. Note that a more efficient way of doing this would be to have a fixed size line buffer and just shuffle bytes into it (with a special case for extremely long lines), rather than modifying JavaScript strings.
var fs = require('fs');
function tail(path, callback) {
var descriptor, bytes = 0, buffer = new Buffer(256), line = '';
function parse(err, bytesRead, buffer) {
if (err) {
callback(err, null);
return;
}
// Keep track of the bytes we have consumed already.
bytes += bytesRead;
// Combine the buffered line with the new string data.
line += buffer.toString('utf-8', 0, bytesRead);
var i = 0, j;
while ((j = line.indexOf('\n', i)) != -1) {
// Callback with a single line at a time.
callback(null, line.substring(i, j));
// Skip the newline character.
i = j + 1;
}
// Only keep the unparsed string contents for next iteration.
line = line.substr(i);
// Keep reading in the next tick (avoids CPU hogging).
process.nextTick(read);
}
function read() {
var stat = fs.fstatSync(descriptor);
if (stat.size <= bytes) {
// We're currently at the end of the file. Check again in 500 ms.
setTimeout(read, 500);
return;
}
fs.read(descriptor, buffer, 0, buffer.length, bytes, parse);
}
fs.open(path, 'r', function (err, fd) {
if (err) {
callback(err, null);
} else {
descriptor = fd;
read();
}
});
return {close: function close(callback) {
fs.close(descriptor, callback);
}};
}
// This will tail the system log on a Mac.
var t = tail('/var/log/system.log', function (err, line) {
console.log(err, line);
});
// Unceremoniously close the file handle after one minute.
setTimeout(t.close, 60000);
All that said, you should also try to leverage the NPM community. With some searching, I found the tail-stream package which might do what you want, with streams.
Previous answers have mentioned tail-stream's approach which uses fs.watch, fs.read and fs.stat together to create the effect of streaming the contents of the file. You can see that code in action here.
Another, perhaps hackier, approach might be to just use tail by spawning a child process with it. This of course comes with the limitation that tail must exist on the target platform, but one of node's strengths is using it to do asynchronous systems development via spawn and even on windows, you can execute node in an alternate shell like msysgit or cygwin to get access to the tail utility.
The code for this:
var spawn = require('child_process').spawn;
var child = spawn('tail',
['-f', 'my.log']);
child.stdout.on('data',
function (data) {
console.log('tail output: ' + data);
}
);
child.stderr.on('data',
function (data) {
console.log('err data: ' + data);
}
);
So, it seems people are still looking for an answer to this question for five years already, and there is yet no answer on topic.
In short: you can't. Not in Node.js particularly, you can't at all.
Long answer: there are few reasons for this.
First, POSIX standard clarifies select() behavior in this regard as follows:
File descriptors associated with regular files shall always select true for ready to read, ready to write, and error conditions.
So, select() can't help with detecting a write beyond the file end.
With poll() it's similar:
Regular files shall always poll TRUE for reading and writing.
I can't tell for sure with epoll(), since it's not standartized and you have to read quite lengthy implementation, but I would assume it's similar.
Since libuv, which is in core of Node.js implementation, uses read(), pread() and preadv() in its uv__fs_read(), neither of which would block when invoked at the end of file, it would always return empty buffer when EOF is encountered. So, no luck here too.
So, summarizing, if such functionality is desired, something must be wrong with your design, and you should revise it.
What you're trying to do is a FIFO file (acronym for First In First Out), which as you said works like a socket.
There's a node.js module that allows you to work with fifo files.
I don't know what do you want that for, but there are better ways to work with sockets on node.js. Try socket.io instead.
You could also have a look at this previous question:
Reading a file in real-time using Node.js
Update 1
I'm not familiar with any module that would do what you want with a regular file, instead of with a socket type one. But as you said, you could use tail -f to do the trick:
// filename must exist at the time of running the script
var filename = 'somefile.txt';
var spawn = require('child_process').spawn;
var tail = spawn('tail', ['-f', filename]);
tail.stdout.on('data', function (data) {
data = data.toString().replace(/^[\s]+/i,'').replace(/[\s]+$/i,'');
console.log(data);
});
Then from the command line try echo someline > somefile.txt and watch at the console.
You might also would like to have a look at this: https://github.com/layerssss/node-tailer
I have a file, with size of 108 bytes.
I want to add to this file some text (buffer), let say "Hello world".
So I wrote the next:
fs.open("./tryit.txt", 'w+', function (err, fd1) {
var buffer = new Buffer("hello world");
fs.write(fd1, buffer, 0, 11, 109, function (err, bytesWrite, buffer) {
})
})
In order to write the file from position of 109.
I see that it write it, but before the hello world, all the text of the file was replaced by the NUL character.
How can I do it? append is not an option, because in some cases I want to write to the middle of the file.
What you want is random access IO (read or write at a specific point in a file).
It's not provided in the default API but you may use an additional package like https://www.npmjs.org/package/random-access-file
From docs:
'w+' - Open file for reading and writing. The file is created (if it does not exist) or truncated (if it exists)
"truncated" means that file becomes empty once opened.
You need a different mode, r+ for instance. a also might work, but not on Linux, according to docs.