I am developping a node js project.I have a zip file i want to extract it then i read one of the files inside my extracted zip.
The problem that i has that even i code the function for extraction before the readfile function that i call it in the callback.
I always has no such file or directory error like the readfile is passed before the extraction.Help!!
This is my code
var unzip = require('unzip');
const fs = require('fs');
var stream = fs.createReadStream(zipFilePath).pipe(unzip.Extract({ path: 'files/em' }));
stream.on('finish', function () {
fs.readFileSync('files/em/data.json') ;//read the extracted file but always the extraction passed after this
});
You don't observe the right event. The 'finish' event of createReadStream() is triggered before the unzip process occurs. You should instead listen to the 'close' event of the unzip process to be sure the extraction is done.
const unzip = require('unzip');
const fs = require('fs');
const zipFilePath = 'files/data.zip'
let extract = unzip.Extract({ path: 'files/em' })
let stream = fs.createReadStream(zipFilePath).pipe(extract);
extract.on('close', () => {
let data = fs.readFileSync('files/em/data.json') ;
console.log(data.toString()) // print your unziped json file
})
path should be a directory, 'unzip.Extract({ path: 'files/em' })'
Related
I am generating a zip file using the module archiver. I want to wait for the zip file to be generated before proceeding further. Below is the relevant part of the code.
// create a file to stream archive data to.
const archiver = require('archiver');
const fs = require('fs');
async function zipcsv(zipPath) {
let output = fs.createWriteStream(zipPath);
let archive = archiver('zip', {
zlib: { level: 9 } // Sets the compression level.
});
output.on('close', function() {
return 'zip generating complete';
});
archive.pipe(output);
archive.directory('exports/', false);
archive.finalize();
}
module.exports.zipcsv = zipcsv;
zip = require('./zipcsv')
await zip.zipcsv(path);
console.log('futher processing');
I see futher processing on the console and the process does not wait for the zip to be generated and continues.
How can I ensure that the zip is generated and then further processing continues.
Thanks.
You need to use a Promise Function inside your async function.
Then use resolve and reject instead of return.
enter image description here
I am trying to read a file (which can be anything like video, image , text ) with react .
I want to add data in the IPFS through Node JS. In normal case(which means when i m only writing in js) I would do it like this :
const data = await fs.readFile(file_path)
console.log(data)
const filesAdded = await nodeDialer.add({
path:file_path,
content:data,
})
console.log(filesAdded)
THe files would be added easily(any type of file).
I tried fs and found out it only works in the node side not on the react side.
and whereever i looked they were using some readastext functions for .txt file or .csv file.
So a proper solution, I couldn't find.
I solved it with this:
Created a event handler for file upload :
captureFile(event){
const file = event.target.files[0]
this.setState({ file_path : file})
console.log('--------------', this.state.file_path)
const reader = new window.FileReader()
reader.readAsArrayBuffer(file)
reader.onloadend = () => {
this.setState({ buffer: Buffer(reader.result) })
console.log('buffer --- ', this.state.buffer)
console.log('path --- ', this.state.file_path.name)
}
}
then I add the file in IPFS with :
const filesAdded = await node.add({
path: this.state.file_path.name ,
content:this.state.buffer
})
I want to import data from a data.json file into the neural network (which uses the Brain.js framework).Here is the part which is supposed to bring that data to the network and analyse it:
const result = brain.likely(
require('data.js')
,net);
alert("This is the result: " + result);
And get that data analysed by the neural network and shown to the user.
Here are the contents of the data.json file for reference:
{
'Rating1': 0.12434213,
'Rating2': 0.987653236,
'Rating3': 0.432543654
}
For your information this is on written on node.js enviroment.
Assuming your data.json file is in the same directory:
fetch('data.json')
.then(response => response.json())
.then(json => {
const result = brain.likely(json, net);
});
Alternatively, with async/await:
(async () => {
const json = await (await fetch('data.json')).json();
const result = brain.likely(json, net);
})();
If done through a file upload:
// target input element
const input = document.querySelector('input');
// upload event
input.addEventListener('change', () => {
const file = this.files[0];
const reader = new FileReader();
reader.addEventListener('load', e => {
const json = JSON.parse(e.target.result);
const result = brain.likely(json, net);
});
reader.readAsText(file);
});
If done through Node:
const json = require('./data.json');
brain.likely(json, net);
Useful resources for handling files:
Using files from web apps - practical examples on how to use the FileReader API
Fetch API - how to use files already on your server in the browser
Node's File System readFileSync method - to read file contents synchronously in a Node environment
JSON.parse - native JS method to convert a string to JSON, useful in all environments
Background
I am trying to read a several GB sized file line by line. I want to process each line and after that write it to a file. I don't want to ( nor can I ) put everything into memory.
It is important that the order in which i read a line is the order in which I write it to a file.
Code
To achieve this I tried using Node.js Readline interface
const fs = require( "fs" ),
readline = require( "readline" );
const readStream = fs.createReadStream( "./logs/report.csv" );
const writeStream = fs.createWriteStream( "./logs/out.csv", { encoding: "utf8"} );
const rl = readline.createInterface({
input: readStream,
output: writeStream,
terminal: false,
historySize: 0
});
rl.on( "line", function(line) {
//Do your stuff ...
const transformedLine = line.toUpperCase();
console.log(transformedLine);
//Then write to outstream
rl.write(transformedLine );
});
Problem
As you can see, I am trying to read a line, parse it, and write it into a file called out.csv.
The problem is that the output file is always empty. Nothing is ever written into it.
I have read all the methods, events and options, but clearly I am missing something.
Question
Why is this code not writing into the file?
Answer
With the current code, I am actually feeding Readline with transformedLine again.
This is not what I want. What I should be doing is to write directly to writeStream.
rl.on( "line", function(line) {
console.log(line);
//Do your stuff ...
const transformedLine = line.toUpperCase();
console.log(transformedLine);
//Then write to outstream
writeStream.write( transformedLine );
});
This will produce an output file respecting the order of input.
For a more detailed discussion on the stream mechanics and internal buffers see:
https://github.com/nodejs/help/issues/1292
I'm quite late for the question, but for anyone who reads this:
If you write on every read and your write speed is slower than read speed you'll still bloat memory. Though not as much as by reading entire file to memory.
You should use pipe with stream.Transform instead of readline. Reason being that pipe processes data at the phase of the slowest participant in flow and so won't bloat memory.
const stream = require('stream');
const fs = require('fs');
const readStream = fs.createReadStream("./logs/report.csv");
const writeStream = fs.createWriteStream("./logs/report.csv");
const transformer = new stream.Transform({
// buffer is a chunk of stream, enc is type of chunk, done is a callback when transform is done
transform(buffer, enc, done){
const lines = buffer.toString().split('\n');
const transformedChunkAsString = lines.map(workYourMagicAndReturnFormattedLine).join('\n');
const transformedBuffer = Buffer.from(transformedChunkAsString);
this.push(transformedBuffer);
done();
}
})
readStream.pipe(transformStream).pipe(writeStream);
Can you try this
const fs = require( "fs" ),
readline = require( "readline" );
const readStream = fs.createReadStream("./logs/report.csv");
const writeStream = fs.createWriteStream("./logs/report.csv");
readStream.pipe(writeStream);
I am parsing multiple large JSON files to my mongoDB database. At the moment I am using stream-json npm package. After I load one file I change the filename that I am loading and relaunch the script to load the next file. This is unnecessarily time consuming. So how can I iterate through all the files automatically? At the moment my code looks like this:
const StreamArray = require('stream-json/utils/StreamArray');
const path = require('path');
const fs = require('fs');
const filename = path.join(__dirname, './data/xa0.json'); //The next file is named xa1.json and so on.
const stream = StreamArray.make();
stream.output.on('data', function (object) {
// my function block
});
stream.output.on('end', function () {
console.log('File Complete');
});
fs.createReadStream(filename).pipe(stream.input);
I tried iterating through the names of the files by adding a loop that would add +1 to the filename i.e. xa0 to xa1 at the same point where the script console.log('File Complete') but this didn't work. Any ideas how I might be able to achieve this or something similar.
Just scan your JSON files directory using fs.readdir. It will return a list of file names that you can then iterate, something like this :
fs.readdir("./jsonfiles", async (err, files) => {
for( file in files ){
await saveToMongo("./jsonfiles/" + file)
}
})
So you just launch your script once and wait until full completion.
Of course, in order for it to be awaited, you need to promisify the saveToMongo function, something like :
const saveToMongo = fileName => {
return new Promise( (resolve, reject) => {
// ... logic here
stream.output.on('end', function () {
console.log('File Complete');
resolve() // Will trigger the next await
});
})
}