my task is to read the first line from csv-file.
This code works well for the file under 500Mb size, but it does not work for bigger files. I need to go up to 1Gb. For the bigger then 500Mb files i do not get any content from reader.result. Is there any way to handle it?
const reader = new FileReader()
reader.readAsText(file)
const lineBreakCharacters = '\r\n'
const rows = (reader.result as string).split(lineBreakCharacters)
let headerLine = rows.shift()
console.log('headerLine===>', headerLine)
Related
When user uploads a file that has charset=iso-8859-1 it comes with question marks and gibberish.
I have seen that there online web converts it successfully to utf-8 - so after uploading the file after this conversion the file is getting uploaded properly. This is the web: https://subtitletools.com/convert-text-files-to-utf8-online
This is my code:
const file = document.getElementById('some-id').files[0];
const reader = new FileReader();
reader.onloadend = event => {
let data = event.target.result;
console.log(`[data]:`, data); // question marks / gibberish
}
reader.readAsText(file);
I have also tried to use reader.readAsBinaryString but got gibberish instead of question marks.
I have also tried to use the utf8 library: https://www.npmjs.com/package/utf8 but it didn't work.
How the site that I mentioned above achieves to convert the file to the desired charset so its data is not in question marks or gibberish? BTW also Google Drive does it well.
You can use TextDecoder with you own charset
var data = new TextDecoder('iso-8859-1').decode(await file.arrayBuffer())
guess the harder part is to figure out what charset it's
If you are not using async/await then you can instead do this:
const file = document.getElementById('some-id').files[0];
file.arrayBuffer().then(ab => {
const data = new TextDecoder('iso-8859-1').decode(ab)
})
I have a multer code blog, to set a size limit according to the type of file I want to make (limits-fileSize) i have no idea how to do it.
Example:
const filetypesImg = /jpeg|jpg|png|gif/; - 4mb
const filetypesVideo = /mp4|avi/; 10mb
const filetypesSound = /mp3|mp4|flac|wav|m4a/; 1mb
actually I want to add limitation to the types like above
I've been trying to make a simple copy/paste program in Node JS but can't do it because of a problem. I am able to copy the contents of source file as Buffer and write it to another file but while writing, node js messes with newlines and therefore destination file is not as same as source file. The code I use:
var fs = require('fs')
var lazy = require('lazy')
var readStream = fs.createReadStream("SOURCE_FILE.EXE")
var writeStream = fs.createWriteStream("DESTINATION_FILE.EXE", 'binary', { flags : 'w' })
var write = function(line) {
writeStream.write(line)
}
new lazy(readStream)
.lines
.forEach(function(line) {
console.log(line) // Buffer of each line of file
write(line)
})
I use writing buffer of each line one by one so that it doesn't hang on big files.
All data is same in SOURCE_FILE and DESTINATION_FILE except some newlines.
Help is really appreciated.
Thanks in advance.
I was wondering if it was possible to stream data from javascript to the browser's downloads manager.
Using webrtc, I stream data (from files > 1Gb) from a browser to the other. On the receiver side, I store into memory all this data (as arraybuffer ... so the data is essentially still chunks), and I would like the user to be able to download it.
Problem : Blob objects have a maximum size of about 600 Mb (depending on the browser) so I can't re-create the file from the chunks. Is there a way to stream these chunks so that the browser downloads them directly ?
if you want to fetch a large file blob from an api or url, you can use streamsaver.
npm install streamsaver
then you can do something like this
import { createWriteStream } from 'streamsaver';
export const downloadFile = (url, fileName) => {
return fetch(url).then(res => {
const fileStream = createWriteStream(fileName);
const writer = fileStream.getWriter();
if (res.body.pipeTo) {
writer.releaseLock();
return res.body.pipeTo(fileStream);
}
const reader = res.body.getReader();
const pump = () =>
reader
.read()
.then(({ value, done }) => (done ? writer.close() : writer.write(value).then(pump)));
return pump();
});
};
and you can use it like this:
const url = "http://urltobigfile";
const fileName = "bigfile.zip";
downloadFile(url, fileName).then(() => { alert('done'); });
Following #guest271314's advice, I added StreamSaver.js to my project, and I successfully received files bigger than 1GB on Chrome. According to the documentation, it should work for files up to 15GB but my browser crashed before that (maximum file size was about 4GB for me).
Note I: to avoid the Blob max size limitation, I also tried to manually append data to the href field of a <a></a> but it failed with files of about 600MB ...
Note II: as amazing as it might seem, the basic technique using createObjectURL works perfectly fine on Firefox for files up to 4GB !!
How can i make stream for zip archiving for long files? i making server that will upload video for anything size (1>gb) and it will split that video into parts , and return users that parts in zip archive. Or is great npm for that solutions with examples?
I have been using ADM ZIP (npm module). That works great, until using big files. So i need solutions in streams. I have try something like this:
var gzip = zlib.createGzip();
var fs = require('fs');
var inp = fs.createReadStream('input.txt');
var out = fs.createWriteStream('input.txt.gz');
inp.pipe(gzip).pipe(out);
But how i can add to this arhive more than one file, and how i can realize that with event ?
inp.on('data', function(data) {
// add data to zip and do other things like counthing percent processing
});