I've created:
var access = fs.createWriteStream('/var/log/node/api.access.log', { flags: 'w' });
Then piped:
process.stdout.pipe(access);
Then tried:
console.log("test");
And nothing has appeared in /var/log/node/api.access.log. However this way is working:
process.stdout.pipe(access).write('test');
Could someone explain what am I doing wrong ?
I solved this problem the following way:
var access = fs.createWriteStream('/var/log/node/api.access.log');
process.stdout.write = process.stderr.write = access.write.bind(access);
Of course you can also separate stdout and stderr if you want.
I also would strongly recommend to handle uncaught exceptions:
process.on('uncaughtException', function(err) {
console.error((err && err.stack) ? err.stack : err);
});
This will cover the following situations:
process.stdout.write
process.stderr.write
console.log
console.dir
console.error
someStream.pipe(process.stdout);
throw new Error('Crash');
throw 'never do this';
throw undefined;
Checkout console.Console, the parent class of the normal console.
var myLogFileStream = fs.createWriteStream(pathToMyLogFile);
var myConsole = new console.Console(myLogFileStream, myLogFileStream);
You can then you use myConsole.log, myConsole.error, myConsole.dir, etc. and write directly to your file.
You can also monkey patch process.stdout.write as follows:
var fn = process.stdout.write;
function write() {
fn.apply(process.stdout, arguments);
myLogFileStream.write.apply(myLogFileStream, arguments);
}
process.stdout.write = write;
there are also other options for overwriting console._stdout depending on the motivation for logging the stdout to a file.
process.stdout is a Writable. pipe is a method of Readable(Cf StreamAPI documentation : https://nodejs.org/api/stream.html
You can see the documentation of process.stdout here : https://nodejs.org/api/process.html#process_process_stdout
It's surprising that you can do process.stdout.pipe(...); without any error. But i suppose this call just do nothing. Except returning a new Writable stream binded to stdout (or maybe it returns process.stdout itself. There's no specification for that in the documentation).
If you want to redirect stdout to a file, you have many solutions :
Just use your command line to do that. Windows style : node myfile.js > api.access.log.
Replace the console object by your own object. And you can rewrite console methods.
I'm not sure, but it may be possible to replace process.stdout with your own stream (and you can do whatever you want with this)
#user3173842
for the reply on
I solved this problem the following way:
var access = fs.createWriteStream('/var/log/node/api.access.log');
process.stdout.write = process.stderr.write = access.write.bind(access);
you do understand that process.stdout continues after process.on('exit') and therefore the fs.WriteStream closes after with process.stdout, according to
https://github.com/nodejs/node/issues/7606
so now the question remains, if the developer desired to have the fs.Writestream.write() return to its normal functionality and when fs.Writestream.end is called the writestream closes. How would the developer go about doing this I did
a_l = asyncify_listener
p_std_stream_m is a process stream manager object
p_std_stream_m.std_info.p_stdout_write = process.stdout.write
process.stdout.write = w_stream.write.bind(w_stream)
process.once('beforeExit', a_l( p_std_stream_m.handler,process.stdout,w_stream ) )
where in the 'beforeExit' event listener I did
process.stdout.write = p_std_stream_m.std_info.p_stdout_write
w_stream.end()
It works and you use the once method because the process.stdout seems to do a lot of work
at this time.
Is this good practice, would you do this or what would you do in this situation
anyone can feel free to reply.
Originally based on #Anatol-user3173842 answer
But in my case I needed to hook the stdout & stderr and also write into a file.
So for those who need to keep the normal stdout behaviour in addition to writing into the file. Use the following.
For non-errors:
// stdout logging hook
const stdoutWrite0 = process.stdout.write;
process.stdout.write = (args) => { // On stdout write
CustomLogger.writeToLogFile('log', args); // Write to local log file
args = Array.isArray(args) ? args : [args]; // Pass only as array to prevent internal TypeError for arguments
return stdoutWrite0.apply(process.stdout, args);
};
For errors:
// stderr logging hook
const stderrWrite0 = process.stderr.write;
process.stderr.write = (args) => { // On stderr write
CustomLogger.writeToLogFile('error', args); // Write to local error file
args = Array.isArray(args) ? args : [args]; // Pass only as array to prevent internal TypeError for arguments
return stderrWrite0.apply(process.stderr, args);
};
// uncaught exceptions
process.on('uncaughtException', (err) => {
CustomLogger.writeToLogFile('error', ((err && err.stack) ? err.stack : err));
});
Here is the CustomLogger code, where I also separate the log files by date:
export class CustomLogger {
static LOGS_DIR = 'location-of-my-log-files';
private static logDailyName(prefix: string): string {
const date = new Date().toLocaleDateString().replace(/\//g, '_');
return `${CustomLogger.LOGS_DIR}/${prefix}_${date}.log`;
}
private static writeToLogFile(prefix, originalMsg) {
const timestamp = Date.now();
const fileName = this.logDailyName(prefix);
const logMsg = prepareForLogFile(originalMsg);
fs.appendFileSync(fileName, `${timestamp}\t${logMsg}\n\n`);
return originalMsg;
}
}
Here's a quick example of a logger class that redirects stdout, stderr and exceptions to a file, while still writting everything to the console:
class Logger {
#log_stream
#stdout_write
#stderr_write
constructor(path) {
this.#log_stream = fs.createWriteStream(path, { flags: 'a' })
this.#stdout_write = process.stdout.write.bind(process.stdout)
this.#stderr_write = process.stderr.write.bind(process.stderr)
process.stdout.write = this.stdout_write.bind(this)
process.stderr.write = this.stderr_write.bind(this)
process.on('uncaughtException', function(err) {
console.error((err && err.stack) ? err.stack : err)
})
}
stdout_write(buffer) {
this.#log_stream.write(buffer)
this.#stdout_write(buffer)
}
stderr_write(buffer) {
this.#log_stream.write(buffer)
this.#stderr_write(buffer)
}
}
const logger = new Logger('example.log')
I am trying to append a string to a log file. However writeFile will erase the content each time before writing the string.
fs.writeFile('log.txt', 'Hello Node', function (err) {
if (err) throw err;
console.log('It\'s saved!');
}); // => message.txt erased, contains only 'Hello Node'
Any idea how to do this the easy way?
For occasional appends, you can use appendFile, which creates a new file handle each time it's called:
Asynchronously:
const fs = require('fs');
fs.appendFile('message.txt', 'data to append', function (err) {
if (err) throw err;
console.log('Saved!');
});
Synchronously:
const fs = require('fs');
fs.appendFileSync('message.txt', 'data to append');
But if you append repeatedly to the same file, it's much better to reuse the file handle.
When you want to write in a log file, i.e. appending data to the end of a file, never use appendFile. appendFile opens a file handle for each piece of data you add to your file, after a while you get a beautiful EMFILE error.
I can add that appendFile is not easier to use than a WriteStream.
Example with appendFile:
console.log(new Date().toISOString());
[...Array(10000)].forEach( function (item,index) {
fs.appendFile("append.txt", index+ "\n", function (err) {
if (err) console.log(err);
});
});
console.log(new Date().toISOString());
Up to 8000 on my computer, you can append data to the file, then you obtain this:
{ Error: EMFILE: too many open files, open 'C:\mypath\append.txt'
at Error (native)
errno: -4066,
code: 'EMFILE',
syscall: 'open',
path: 'C:\\mypath\\append.txt' }
Moreover, appendFile will write when it is enabled, so your logs will not be written by timestamp. You can test with example, set 1000 in place of 100000, order will be random, depends on access to file.
If you want to append to a file, you must use a writable stream like this:
var stream = fs.createWriteStream("append.txt", {flags:'a'});
console.log(new Date().toISOString());
[...Array(10000)].forEach( function (item,index) {
stream.write(index + "\n");
});
console.log(new Date().toISOString());
stream.end();
You end it when you want. You are not even required to use stream.end(), default option is AutoClose:true, so your file will end when your process ends and you avoid opening too many files.
Your code using createWriteStream creates a file descriptor for every write. log.end is better because it asks node to close immediately after the write.
var fs = require('fs');
var logStream = fs.createWriteStream('log.txt', {flags: 'a'});
// use {flags: 'a'} to append and {flags: 'w'} to erase and write a new file
logStream.write('Initial line...');
logStream.end('this is the end line');
Besides appendFile, you can also pass a flag in writeFile to append data to an existing file.
fs.writeFile('log.txt', 'Hello Node', {'flag':'a'}, function(err) {
if (err) {
return console.error(err);
}
});
By passing flag 'a', data will be appended at the end of the file.
Use a+ flag to append and create a file (if doesn't exist):
fs.writeFile('log.txt', 'Hello Node', { flag: "a+" }, (err) => {
if (err) throw err;
console.log('The file is created if not existing!!');
});
Docs: https://nodejs.org/api/fs.html#fs_file_system_flags
You need to open it, then write to it.
var fs = require('fs'), str = 'string to append to file';
fs.open('filepath', 'a', 666, function( e, id ) {
fs.write( id, 'string to append to file', null, 'utf8', function(){
fs.close(id, function(){
console.log('file closed');
});
});
});
Here's a few links that will help explain the parameters
open
write
close
EDIT: This answer is no longer valid, look into the new fs.appendFile method for appending.
My approach is rather special. I basically use the WriteStream solution but without actually 'closing' the fd by using stream.end(). Instead I use cork/uncork. This got the benefit of low RAM usage (if that matters to anyone) and I believe it's more safe to use for logging/recording (my original use case).
Following is a pretty simple example. Notice I just added a pseudo for loop for showcase -- in production code I am waiting for websocket messages.
var stream = fs.createWriteStream("log.txt", {flags:'a'});
for(true) {
stream.cork();
stream.write("some content to log");
process.nextTick(() => stream.uncork());
}
uncork will flush the data to the file in the next tick.
In my scenario there are peaks of up to ~200 writes per second in various sizes. During night time however only a handful writes per minute are needed. The code is working super reliable even during peak times.
Node.js 0.8 has fs.appendFile:
fs.appendFile('message.txt', 'data to append', (err) => {
if (err) throw err;
console.log('The "data to append" was appended to file!');
});
Documentation
Using fs.appendFile or fsPromises.appendFile are the fastest and the most robust options when you need to append something to a file.
In contrast to some of the answers suggested, if the file path is supplied to the appendFile function, It actually closes by itself. Only when you pass in a filehandle that you get by something like fs.open() you have to take care of closing it.
I tried it with over 50,000 lines in a file.
Examples :
(async () => {
// using appendFile.
const fsp = require('fs').promises;
await fsp.appendFile(
'/path/to/file', '\r\nHello world.'
);
// using apickfs; handles error and edge cases better.
const apickFileStorage = require('apickfs');
await apickFileStorage.writeLines(
'/path/to/directory/', 'filename', 'Hello world.'
);
})();
Ref: https://github.com/nodejs/node/issues/7560
If you want an easy and stress-free way to write logs line by line in a file, then I recommend fs-extra:
const os = require('os');
const fs = require('fs-extra');
const file = 'logfile.txt';
const options = {flag: 'a'};
async function writeToFile(text) {
await fs.outputFile(file, `${text}${os.EOL}`, options);
}
writeToFile('First line');
writeToFile('Second line');
writeToFile('Third line');
writeToFile('Fourth line');
writeToFile('Fifth line');
Tested with Node v8.9.4.
fd = fs.openSync(path.join(process.cwd(), 'log.txt'), 'a')
fs.writeSync(fd, 'contents to append')
fs.closeSync(fd)
I offer this suggestion only because control over open flags is sometimes useful, for example, you may want to truncate it an existing file first and then append a series of writes to it - in which case use the 'w' flag when opening the file and don't close it until all the writes are done. Of course appendFile may be what you're after :-)
fs.open('log.txt', 'a', function(err, log) {
if (err) throw err;
fs.writeFile(log, 'Hello Node', function (err) {
if (err) throw err;
fs.close(log, function(err) {
if (err) throw err;
console.log('It\'s saved!');
});
});
});
Using jfile package :
myFile.text+='\nThis is new line to be appended'; //myFile=new JFile(path);
Try to use flags: 'a' to append data to a file
var stream = fs.createWriteStream("udp-stream.log", {'flags': 'a'});
stream.once('open', function(fd) {
stream.write(msg+"\r\n");
});
Here's a full script. Fill in your file names and run it and it should work!
Here's a video tutorial on the logic behind the script.
var fs = require('fs');
function ReadAppend(file, appendFile){
fs.readFile(appendFile, function (err, data) {
if (err) throw err;
console.log('File was read');
fs.appendFile(file, data, function (err) {
if (err) throw err;
console.log('The "data to append" was appended to file!');
});
});
}
// edit this with your file names
file = 'name_of_main_file.csv';
appendFile = 'name_of_second_file_to_combine.csv';
ReadAppend(file, appendFile);
const inovioLogger = (logger = "") => {
const log_file = fs.createWriteStream(__dirname + `/../../inoviopay-${new Date().toISOString().slice(0, 10)}.log`, { flags: 'a' });
const log_stdout = process.stdout;
log_file.write(logger + '\n');
}
In addition to denysonique's answer, sometimes asynchronous type of appendFile and other async methods in NodeJS are used where promise returns instead of callback passing. To do it you need to wrap the function with promisify HOF or import async functions from promises namespace:
const { appendFile } = require('fs').promises;
await appendFile('path/to/file/to/append', dataToAppend, optionalOptions);
I hope it'll help 😉
I wrapped the async fs.appendFile into a Promise-based function. Hope it helps others to see how this would work.
append (path, name, data) {
return new Promise(async (resolve, reject) => {
try {
fs.appendFile((path + name), data, async (err) => {
if (!err) {
return resolve((path + name));
} else {
return reject(err);
}
});
} catch (err) {
return reject(err);
}
});
}
I am currently trying to create a util to parse annotations from a PDF. I can load the PDF file just fine, the annotation objects just fine, but I need to obtain the text that is related to those annotations (underlined, highlighted, etc.).
This gets hairy when I try to use the getTextContent() method which fails. Below is the method where this happens:
/**
* #param pdf The PDF document obtained upon `pdfjs.getDocument(pdf).promise` success.
*/
function getAllPages(pdf) {
return new Promise((resolve, reject) => {
let allPromises = [];
for (let i = 0; i < numPages; i++) {
const pageNumber = i + 1; // note: pages are 1-based
const page = pdf.getPage(pageNumber)
.then((pageContent) => {
// testing with just one page to see what's up
if (pageNumber === 1) {
try {
pageContent.getTextContent()
.then((txt) => {
// THIS NEVER OCCURS
console.log('got text');
})
.catch((error) => {
// THIS IS WHERE THE ERROR SHOULD BE CAUGHT
console.error('in-promise error', error)
});
} catch (error) {
// AT LEAST IT SHOULD BE CAUGHT HERE
console.log('try/catch error:', error);
}
}
})
.catch(reject);
allPromises.push(page);
}
Promise.all(allPromises)
.then(() => {
allPagesData.sort(sortByPageNumber);
resolve(allPagesData);
})
.catch(reject);
});
}
When calling pageContent.getTextContent(), which should return a promise, the error "ReferenceError: ReadableStream is not defined" is thrown in the catch() part of the try.
This is weird because I would have expected the pageContent.getTextContent().catch() to be able to, well, catch that. Also, I don't know what to do to resolve this.
Any help is appreciated.
I have noticed that using pdfjs-dist causes the error.
Use pdfjs-dist/es5/build/pdf.js instead.
const pdfjs = require('pdfjs-dist/es5/build/pdf.js');
Update:
const pdfJs = require('pdfjs-dist/legacy/build/pdf')
Example usage
There was a new change, the only way it worked here was to use this path:
const pdfJs = require('pdfjs-dist/legacy/build/pdf')
I started a new project with pdfjs-dist and got the same ReadableStream error at getTextContent. Also i have an older project with the same lib that works fine. So, when I downgraded to an older version (2.0.943 to be precise) the error was gone. I don't realy know why. Hope that helps.
The HTML file where form is in doesn't have any client side javascript code. That I cannot change directly because my task is nodejs and express.
app.post('/',function(request,response){
const htmlCode = fs.readFileSync(__dirname + '/loggain.html');
const loggaInDom = new jsDOM.JSDOM(htmlCode);
const input = request.body.nickname;
try{
if(input.length<3){
throw new Error("nickname must be at least 3 characters");
}
else{
response.cookie('nickName',input);
response.redirect('index.html');
console.log(request.cookies.nickName);
}
}
catch(error){
console.log(error);
}
});
This is part of my nodejs code.
I would like to block form submit when input.length is smaller than 3. Like event.preventDefault() in client javascript code.
Now it throws error in console, which is correct, but browser keeps loading page permanently.
I cannot directly change HTML file but probably can insert client javascript file to HTML with nodejs, but I would like to know if it is possible to do with only nodeJS
Instead of throwing the error you need to send a response to the client with an error status:
if(input.length<3){
response.status(400).send("nickname must be at least 3 characters");
}
Choose the appropriate error code depending on your error: https://en.wikipedia.org/wiki/List_of_HTTP_status_codes
You can also send json instead of text if you want some structure that your frontend can use:
if(input.length<3){
response.status(400).json({
type: "validation error",
message: "nickname must be at least 3 characters"
});
}
Centralized error handling
Alternatively you may not want standardize error handling and do the response.status() thing at one location. Express has a way to catch errors but you cannot use the throw keyword. Instead you pass your error to the next function:
app.post('/',function(request, response, next){ // NOTE: next
const htmlCode = fs.readFileSync(__dirname + '/loggain.html');
const loggaInDom = new jsDOM.JSDOM(htmlCode);
const input = request.body.nickname;
try{
if(input.length<3){
throw new Error("nickname must be at least 3 characters");
}
else{
response.cookie('nickName',input);
response.redirect('index.html');
console.log(request.cookies.nickName);
}
}
catch(error){
next(error); // This is how errors are handled in Express
}
});
Now all you need to do is write a default error handler which is a special middleware that accepts four arguments instead of three or two. Make sure this middleware is loaded last after all your routes:
app.use((error, request, response, next) => {
response.status(500).send(error.message);
});
When used with error types you can send different responses to the browser depending on types of error:
class ValidationError extends Error {
constructor(message) {
super(message);
this.name = "ValidationError"; // (2)
}
}
Then you can throw:
next(new ValidationError("nickname must be at least 3 characters"))
Which you can handle with:
app.use((error, request, response, next) => {
switch(error.name) {
case "ValidationError":
response.status(400);
break;
default:
response.status(500);
}
response.send(error.message);
});
I'm working on a program that reads a file line by line with the readline module. First I get the file name by command line, but I want to check if the file actually exists. I have read about fs.stat() but I want to know if there is a way to catch the error directly with readline. So far I've tried this
try{
var line_reader = read_line.createInterface({
input: file_stream.createReadStream(file_name)
});
}catch(err){
console.log('Please insert a valid file name');
}
But I still get the message
Error: ENOENT: no such file or directory
The exception is thrown by createReadStream.
You need to add 'on error' case to createReadStream:
var fs = file_stream.createReadStream(file_name)
fs.on('error', function (err) {
// handle error here
});
var line_reader = read_line.createInterface({
input: fs
});
Miss read your question at the beginning and updated my answer.
A solution you could use fs.stat
Edit
// fs.stat is async
fs.stat(file_name, function(err,stat){
if (stat && stat.isFile() ) {
var line_reader = read_line.createInterface({
input: file_stream.createReadStream(file_name)
});
}
});