We Keep Coding

AutobahnJS: Remote call to asyncronous function

I'm trying to make a call over WAMP to a remote function. But I don't know how to write the called function if it has asynchronous behavior. In every example I've seen the remote function returns the result. How can this be done in an asynchronous manner where I would normally use a callback?
Example:
This is the registration of a function that would get the contents of a file asynchronously.
session.register('com.example.getFileContents', getFileContents).then(
function (reg) {
console.log("procedure getFileContents() registered");
},
function (err) {
console.log("failed to register procedure: " + err);
}
);
Here is how I would call that function remotely.
session.call('com.example.getFileContents', ["someFile.txt"]).then(
function (res) {
console.log("File Contents:", res);
},
function (err) {
console.log("Error getting file contents:", err);
}
);
But here is the actual function that was registered.
function getFileContents(file) {
fs.readFile(file, 'utf8', function(err, data) {
// How do I return the data?
});
}
How do I return the data from getFileContents so that it can be sent back over the WAMP connection? I know I could use readFileSync and return what it returns. But I'm specifically asking how to do this in an asynchronous manner.

I figured out how to do this with promises. Here is how the function implemented with promises.
var fs = require('fs');
var when = require('when');
function getFileContents(file) {
var d = when.defer();
fs.readFile(file, 'utf8', function(err, data) {
d.resolve(data);
});
return d.promise;
}

Javascript scoping issue, object has no data after assigning it

I am using NodeJS to create an express endpoint that will retrieve the metadata from my images stored on my server. I have the following code for the logic of the endpoint:
/*
* Gallery Controller
*/
var fs = require('fs'),
_ = require('underscore'),
im = require('imagemagick');
/**
* List paths to images stored in selected gallery
*/
exports.list = function(req, res) {
var dir = 'public/images/' + req.params.id;
fs.readdir(dir, function(err, files) {
if (err) return res.send({error: 'No gallery found with provided id'}, 404);
if (files.length > 0) {
var collection = [],
myData = {};
files.forEach(function(file) {
if(file === '.DS_Store') return;
im.readMetadata( dir + '/' + file, function(err, metadata) {
if (err) throw err;
myData = metadata;
console.log(myData); // logs as object with expected data
});
console.log(myData); // logs as empty {}
collection.push(myData);
});
console.log(collection); // logs as [ {}, {} ]
res.json(collection, 200);
} else {
res.json({error: 'Selected gallery is empty'}, 404);
}
});
};
I've listed what the logs appear as in the terminal, why am I getting this scoping issue? I can't seem to wrap my head around it. If I try to return the metadata obj and assign it to the var, I get the following error: TypeError: Converting circular structure to JSON

Use the async module, it'll improve your life in many ways.
The problem you are having is a common one I see, and it is that your loop is asynchronous, but you are treating it as something serial.
Instead of doing files.forEach, you want to loop them asynchronously and then do some more stuff when the looping is done. You can use async.each for that.
async.each(files, function (file, next) {
if (file === '.DS_Store') return next();
im.readMetadata(path.join(dir, file), function (e, data) {
collection.push(data);
next(err);
});
}, function (err) {
if (err) throw err;
console.log(collection);
});
As an alternative, an even more appropriate solution might be to use async.map.
async.map(files, function (file, next) {
if (file === '.DS_Store') return next();
im.readMetadata(path.join(dir, file), next);
}, function (err, collection) {
if (err) throw err;
console.log(collection);
});

You need to restructure your code:
files.forEach(function(file, i) {
if (file === '.DS_Store') return; // see text
im.readMetadata( dir + '/' + file, function(err, metadata) {
if (err) throw err;
collection.push(metadata);
if (i === files.length - 1) {
res.json(collection); // see text
}
});
});
The reason is that the metadata is only available when the callback function to readMetadata is called; that's how asynchronous I/O works in Node.
In that callback, you add the metadata to the collection. If the iteration of the forEach has reached the final element (i is the index of the current element, when its value is one less than the size of the array, it's the last element), the response is sent.
Two issues:
if .DS_Store is the last/only file in the directory, this code will fail because it will never send back a response; I'll leave it to you to deal with that case ;)
res.json will, by default, return a 200 status so you don't have to specify it; if you do want to specify a status, it needs to be res.json(200, collection) (arguments swapped)

Node.js method does not return any response, pending request

Simple upload method using node.js+express.js:
upload: function(req, res, next){
//go over each uploaded file
async.each(req.files.upload, function(file, cb) {
async.auto({
//create new path and new unique filename
metadata: function(cb){
//some code...
cb(null, {uuid:uuid, newFileName:newFileName, newPath:newPath});
},
//read file
readFile: function(cb, r){
fs.readFile(file.path, cb);
},
//write file to new destination
writeFile: ['readFile', 'metadata', function(cb,r){
fs.writeFile(r.metadata.newPath, r.readFile, function(){
console.log('finished');
});
}]
}, function(err, res){
cb(err, res);
});
}, function(err){
res.json({success:true});
});
return;
}
The method iterates each uploaded file, creates a new filename and writes it to a given location set in metadata.
console.log('finished');
is fired when writing is finished, however the client never receives a response. After 2 minutes the request is cancled, however the file was uploaded.
Any ideas why this method does not return any response?

You're using readFile, which is async, too, and works like this:
fs.readFile('/path/to/file',function(e,data)
{
if (e)
{
throw e;
}
console.log(data);//file contents as Buffer
});
I you could pass a reference to a function here, to take care of this, but IMO, it'd be way easier to simply use readFileSync, which returns the Buffer directly, and can be passed to writeFile without issues:
fs.writeFile(r.metadata.newPath, r.readFileSync, function(err)
{
if (err)
{
throw err;
}
console.log('Finished');
});
Check the docs for readFile and writeFile respectively

Node.js how to read a file and then write the same file with two separate functions?

What I want to do is read a file and then be able to perform other operations with that information as I write the file. For example:
read file
write file and at the same time perform MD5 hash, digital signing etc.
I could use fs.readfile and fs.writefile as one operation and just copy the file from the web server to my computer, but I don't think I could still do these same operations. Anyway, skipping the in between stuff. How do I use fs.readfile and writefile to create two separate functions to copy a file? Here is what I have been working on, and yes I've read these forums extensively in search of an answer.
var fs = require('fs');
function getData(srcPath) {
fs.readFile(srcPath, 'utf8', function (err, data) {
if (err) throw err;
return data;
}
);
}
function writeData(savPath, srcPath) {
fs.writeFile (savPath, (getData(srcPath)), function(err) {
if (err) throw err;
console.log('complete');
}
);
}
//getData ('./test/test.txt');
writeData ('./test/test1.txt','./test/test.txt');
I want to be able to download files of any type and just make raw copies, with md5 hash etc attached to a JSON file. That will probably be a question for later though.

As suggested by dandavis in his comment, readFile does nothing because it is an asynchronous call. Check out this answer for additional information on what that means.
In short, an async call will never wait for the result to return. In your example, getData does not wait for readFile() to return the result you want, but will finish right away. Async calls are usually handled by passing callbacks, which is the last parameter to readFile and writeFile.
In any case, there are two ways to do this:
1.Do it asynchronously (which is the proper way):
function copyData(savPath, srcPath) {
fs.readFile(srcPath, 'utf8', function (err, data) {
if (err) throw err;
//Do your processing, MD5, send a satellite to the moon, etc.
fs.writeFile (savPath, data, function(err) {
if (err) throw err;
console.log('complete');
});
});
}
2.Do it synchronously. Your code won't have to change much, you will just need to replace readFile and writeFile by readFileSync and writeFileSync respectively. Warning: using this method is not only against best practises, but defies the very purpose of using nodejs (unless of course you have a very legitimate reason).
Edit: As per OP's request, here is one possible way to separate the two methods, e.g., using callbacks:
function getFileContent(srcPath, callback) {
fs.readFile(srcPath, 'utf8', function (err, data) {
if (err) throw err;
callback(data);
}
);
}
function copyFileContent(savPath, srcPath) {
getFileContent(srcPath, function(data) {
fs.writeFile (savPath, data, function(err) {
if (err) throw err;
console.log('complete');
});
});
}
This way, you are separating the read part (in getFileContent) from the copy part.

I had to use this recently, so I converted verybadallocs answer to promises.
function readFile (srcPath) {
return new Promise(function (resolve, reject) {
fs.readFile(srcPath, 'utf8', function (err, data) {
if (err) {
reject(err)
} else {
resolve(data)
}
})
})
}
function writeFile (savPath, data) {
return new Promise(function (resolve, reject) {
fs.writeFile(savPath, data, function (err) {
if (err) {
reject(err)
} else {
resolve()
}
})
})
}
Then using them is simple.
readFile('path').then(function (results) {
results += ' test manipulation'
return writeFile('path', results)
}).then(function () {
//done writing file, can do other things
})
Usage for async/await
const results = await readFile('path')
results += ' test manipulation'
await writeFile('path', results)
// done writing file, can do other things

To read and write a file with Non-blocking or Asynchronous way, you can use the advance features of es6 or higher like Promise or Async/await, but you must keep eye on Polyfills (https://javascript.info/polyfills) or if there are only a couple of read/write you can use call back Hell.
function readFiles(){
fs.readFile('./txt/start.txt', 'utf-8', (err, data1)=>{
if(err) return console.log(err);
fs.readFile(`./txt/${data1}.txt`, 'utf-8', (err, data2)=>{
if(err) return console.log(err);
fs.readFile('./txt/append.txt', 'utf-8', (err, data3)=>{
if(err) return console.log(err);
writeFile('./txt/final.txt', `${data2}\n${data3}`);
});
});
});
}
function writeFile(path, data){
fs.writeFile(path,data,'utf-8',err=>{
if(err){
console.log(err);
}
})
}
readFiles();

Get data from fs.readFile [duplicate]

This question already has answers here:
Why is my variable unaltered after I modify it inside of a function? - Asynchronous code reference
(7 answers)
Closed 7 days ago.
var content;
fs.readFile('./Index.html', function read(err, data) {
if (err) {
throw err;
}
content = data;
});
console.log(content);
Logs undefined, why?

To elaborate on what #Raynos said, the function you have defined is an asynchronous callback. It doesn't execute right away, rather it executes when the file loading has completed. When you call readFile, control is returned immediately and the next line of code is executed. So when you call console.log, your callback has not yet been invoked, and this content has not yet been set. Welcome to asynchronous programming.
Example approaches
const fs = require('fs');
// First I want to read the file
fs.readFile('./Index.html', function read(err, data) {
if (err) {
throw err;
}
const content = data;
// Invoke the next step here however you like
console.log(content); // Put all of the code here (not the best solution)
processFile(content); // Or put the next step in a function and invoke it
});
function processFile(content) {
console.log(content);
}
Or better yet, as Raynos example shows, wrap your call in a function and pass in your own callbacks. (Apparently this is better practice) I think getting into the habit of wrapping your async calls in function that takes a callback will save you a lot of trouble and messy code.
function doSomething (callback) {
// any async callback invokes callback with response
}
doSomething (function doSomethingAfter(err, result) {
// process the async result
});

There is actually a Synchronous function for this:
http://nodejs.org/api/fs.html#fs_fs_readfilesync_filename_encoding
Asynchronous
fs.readFile(filename, [encoding], [callback])
Asynchronously reads the entire contents of a file. Example:
fs.readFile('/etc/passwd', function (err, data) {
if (err) throw err;
console.log(data);
});
The callback is passed two arguments (err, data), where data is the contents of the file.
If no encoding is specified, then the raw buffer is returned.
SYNCHRONOUS
fs.readFileSync(filename, [encoding])
Synchronous version of fs.readFile. Returns the contents of the file named filename.
If encoding is specified then this function returns a string. Otherwise it returns a buffer.
var text = fs.readFileSync('test.md','utf8')
console.log (text)

function readContent(callback) {
fs.readFile("./Index.html", function (err, content) {
if (err) return callback(err)
callback(null, content)
})
}
readContent(function (err, content) {
console.log(content)
})

Using Promises with ES7
Asynchronous use with mz/fs
The mz module provides promisified versions of the core node library. Using them is simple. First install the library...
npm install mz
Then...
const fs = require('mz/fs');
fs.readFile('./Index.html').then(contents => console.log(contents))
.catch(err => console.error(err));
Alternatively you can write them in asynchronous functions:
async function myReadfile () {
try {
const file = await fs.readFile('./Index.html');
}
catch (err) { console.error( err ) }
};

This line will work,
const content = fs.readFileSync('./Index.html', 'utf8');
console.log(content);

var data = fs.readFileSync('tmp/reltioconfig.json','utf8');
use this for calling a file synchronously,
without encoding its showing output as a buffer.

As said, fs.readFile is an asynchronous action. It means that when you tell node to read a file, you need to consider that it will take some time, and in the meantime, node continued to run the following code. In your case it's: console.log(content);.
It's like sending some part of your code for a long trip (like reading a big file).
Take a look at the comments that I've written:
var content;
// node, go fetch this file. when you come back, please run this "read" callback function
fs.readFile('./Index.html', function read(err, data) {
if (err) {
throw err;
}
content = data;
});
// in the meantime, please continue and run this console.log
console.log(content);
That's why content is still empty when you log it. node has not yet retrieved the file's content.
This could be resolved by moving console.log(content) inside the callback function, right after content = data;. This way you will see the log when node is done reading the file and after content gets a value.

From Node v8
Use the built in promisify library to make these old callback functions more elegant.
const fs = require('fs');
const util = require('util');
const readFile = util.promisify(fs.readFile);
async function doStuff() {
try {
const content = await readFile(filePath, 'utf8');
console.log(content);
} catch (e) {
console.error(e);
}
}
From Node v10
You can use the promises version of fs API:
import { promises as fs } from 'fs';
async function doStuff() {
try {
const content = await fs.readFile(filePath, 'utf8');
console.log(content);
} catch (e) {
console.error(e);
}
}

const fs = require('fs')
function readDemo1(file1) {
return new Promise(function (resolve, reject) {
fs.readFile(file1, 'utf8', function (err, dataDemo1) {
if (err)
reject(err);
else
resolve(dataDemo1);
});
});
}
async function copyFile() {
try {
let dataDemo1 = await readDemo1('url')
dataDemo1 += '\n' + await readDemo1('url')
await writeDemo2(dataDemo1)
console.log(dataDemo1)
} catch (error) {
console.error(error);
}
}
copyFile();
function writeDemo2(dataDemo1) {
return new Promise(function(resolve, reject) {
fs.writeFile('text.txt', dataDemo1, 'utf8', function(err) {
if (err)
reject(err);
else
resolve("Promise Success!");
});
});
}

sync and async file reading way:
//fs module to read file in sync and async way
var fs = require('fs'),
filePath = './sample_files/sample_css.css';
// this for async way
/*fs.readFile(filePath, 'utf8', function (err, data) {
if (err) throw err;
console.log(data);
});*/
//this is sync way
var css = fs.readFileSync(filePath, 'utf8');
console.log(css);
Node Cheat Available at read_file.

var path = "index.html"
const readFileAsync = fs.readFileSync(path, 'utf8');
// console.log(readFileAsync)
using simple readFileSync works for me.

var fs = require('fs');
var path = (process.cwd()+"\\text.txt");
fs.readFile(path , function(err,data)
{
if(err)
console.log(err)
else
console.log(data.toString());
});

var content;
fs.readFile('./Index.html', function read(err, data) {
if (err) {
throw err;
}
content = data;
});
console.log(content);
This is just because node is asynchronous and it will not wait for the read function and as soon as the program starts it will console the value as undefined, Which is actually true because there is no value assigned to content variable.
To handle we can use promises, generators etc.
We can use promise in this way.
new Promise((resolve,reject)=>{
fs.readFile('./index.html','utf-8',(err, data)=>{
if (err) {
reject(err); // in the case of error, control flow goes to the catch block with the error occured.
}
else{
resolve(data); // in the case of success, control flow goes to the then block with the content of the file.
}
});
})
.then((data)=>{
console.log(data); // use your content of the file here (in this then).
})
.catch((err)=>{
throw err; // handle error here.
})

The following is function would work for async wrap or promise then chains
const readFileAsync = async (path) => fs.readFileSync(path, 'utf8');

you can read file by
var readMyFile = function(path, cb) {
fs.readFile(path, 'utf8', function(err, content) {
if (err) return cb(err, null);
cb(null, content);
});
};
Adding on you can write to file,
var createMyFile = (path, data, cb) => {
fs.writeFile(path, data, function(err) {
if (err) return console.error(err);
cb();
});
};
and even chain it together
var readFileAndConvertToSentence = function(path, callback) {
readMyFile(path, function(err, content) {
if (err) {
callback(err, null);
} else {
var sentence = content.split('\n').join(' ');
callback(null, sentence);
}
});
};

To put it roughly, you're dealing with node.js which is asynchronous in nature.
When we talk about async, we're talking about doing or processing info or data while dealing with something else. It is not synonymous to parallel, please be reminded.
Your code:
var content;
fs.readFile('./Index.html', function read(err, data) {
if (err) {
throw err;
}
content = data;
});
console.log(content);
With your sample, it basically does the console.log part first, thus the variable 'content' being undefined.
If you really want the output, do something like this instead:
var content;
fs.readFile('./Index.html', function read(err, data) {
if (err) {
throw err;
}
content = data;
console.log(content);
});
This is asynchronous. It will be hard to get used to but, it is what it is.
Again, this is a rough but fast explanation of what async is.

I like using fs-extra because all functions are promisified, right out of the box, so you can use await. So your code could look like this:
(async () => {
try {
const content = await fs.readFile('./Index.html');
console.log(content);
} catch (err) {
console.error(err);
}
})();