How to split a big ODS file without causing memory leaks? - javascript

I'm working with a MYSQL database, and have two types of files to import:
First one is a CSV file that I can use
LOAD DATA INFILE 'path-to-csv_file'
The second type of file is ODS (OpenDocument Spreadsheet) that MYSQL doesn't support for LOAD DATA INFILE.
My solution was to convert ODS to CSV using xlsx package that have a XLSX.readfile command and then using csv-writer. But, when working with large ODS files, my program was crashing cause it was using to much memory. I searched for solutions and found streams but xlsx package doesn't have read streams. After this, I tried to use fs cause it has a fs.createReadStream command, but this module doesn't support ODS files. An example is comparing both returns in fs.readFile and xlsx.readFile.
fs.readFile:
PK♥♦m�IQ�l9�.mimetypeapplication/vnd.oasis.opendocument.spreadsheetPK♥♦m�IQM◄ˋ%�%↑Thumbnails/thumbnail.png�PNG
→
IHDR�♥A�-=♥PLTE►►☼§¶►∟↓*.!/<22/8768:G6AN>AM>BP>MaC:;A?GOE?EFJGJRJQ[TJEQOQ\QJYWYKVeX\dX]p\bkXetaNJgTEe[Wp^Wa_aja\ue\hfgektjqztkeqnpyqlwwvco�jw�j}�v{�q⌂�~�⌂{��t��t��u��z��y��|��{��{��}���o]�od�vj�|v�⌂n�⌂r��{��n��x��~��~������
XLSX.readFile:
J323: { t: 's', v: '79770000', w: '79770000' },
K323: { t: 's', v: '20200115', w: '20200115' },
Working with XLSX module is easy, cause I can pick up only the data that I want in this ODS file. Using a javascript code, I extract three columns and put it in an array:
const xlsx = require('xlsx');
let posts = [];
let post = {};
for(let i = 0; i < 1; i++){
let filePath = `C:\\Users\\me\\Downloads\\file_users.ODS`;
let workbook = xlsx.readFile(filePath);
let worksheet = workbook.Sheets[workbook.SheetNames[0]];
for (let cell in worksheet) {
const cellAsString = cell.toString();
cellAsString[0] === 'A' ? post['ID'] = worksheet[cell].v :
cellAsString[0] === 'C' ? post['USER NAME'] = worksheet[cell].v : null;
if (cellAsString[0] === 'J') {
post['USER EMAIL'] = worksheet[cell].v;
Object.keys(post).length == 3 ? posts.push(post) : null;
post = {}
}
}
}
...returns:
{
ID: '1',
'USER NAME': 'John Paul',
'USER EMAIL': 'Paul.John12#hotmail.com'
},
{
ID: '2',
'USER NAME': 'Julia',
'USER EMAIL': 'lejulie31312#outlook.com'
},
{
ID: '3',
'USER NAME': 'Greg Norton',
'USER EMAIL': 'thenorton31031#hotmail.com'
},
... 44660 more items
So, my problem is when working with large ODS files. The return above is when using this script with 78MB file, and is using 1.600MB of RAM. When I try to use this with 900MB files, my memory reaches the limit (4000MB+) and I got the error: 'ERR_STRING_TOO_LONG'
I tried to use readline package for parse the data, but it needs a stream.
If I have to slice the ODS files into small pieces, how could I read the file for this without crashing my vs code?

Related

Taking javascript input

I am developing a CLI using Enquirer. I want user of the CLI to write javascript on a json.
So, i want something like this :
Create a Rule on the the data
const model = {
reviews: {
'5': [
{
customerId: 'A2OKPZ5S9F78PD',
rating: '5',
asin: 'asin2',
reviewStatus: 'APPROVED',
reviewId: 'R379DKACZQRXME',
},
],
'4': [
{
customerId: 'A2OKPZ5S9F78PD',
rating: '4',
asin: 'asin2',
reviewStatus: 'APPROVED',
reviewId: 'R379DKACZQRXME',
},
],
},
entityType: 'LIVE_EVENT',
entityId: 'event2',
};
Then user writes the rule.
Object.values(model.reviews).forEach(reviews =>
(reviews as any).forEach(review => {
if (parseInt(review.rating) < 3 && attributes.reviewId.Value.includes(review.reviewId)) {
output.push({
exceptionName: `Customer ${review.customerId} left a review ${review.reviewId} with rating ${review.rating}`,
});
}
})
);
While writing this rule, Since it is on the above json model, I want to provide autocomplete options on javascript and validate if it is correct javascript.
Is there a way to do this ?
If I'm understanding your question correctly, it sounds like you want to take the model object and write it to a JSON file.
If this is your goal, simply do the following:
import { writeFileSync } from "fs";
// Define the model
const model: any = { foo: bar };
// Transform the model object to JSON
const modelJSON: string = JSON.stringify(model, null, 4); // Indents the JSON 4-spaces
// Write the modelJSON to `model.json`
writeFileSync("./model.json", modelJSON);
The above is TypeScript, but the standard JavaScript version is basically the same. Make sure you add #types/node to your package.json file if you're using TypeScript - hope this helps!

Problem generating buffer for nodejs csv file creation

Iam able to generate a csv file with the data below. I am using a nodejs library "csv-writer" that generates the file quite well. My problem is that I need a way to return back a buffer instead of the file itself. Reason being I need to upload the file to a remote server via sftp.
How do I go ab bout modifying this piece of code to enable buffer response? Thanks.
...
const csvWriter = createCsvWriter({
path: 'AuthHistoryReport.csv',
header: [
{id: 'NAME', title: 'msg_datetime_date'},
{id: 'AGE', title: 'msg_datetime'}
]
});
var rows = [
{ NAME: "Paul", AGE:21 },
{ NAME: "Charles", AGE:28 },
{ NAME: "Teresa", AGE:27 },
];
csvWriter
.writeRecords(rows)
.then(() => {
console.log('The CSV file was written successfully');
});
...
Read your own file with fs.readFile('AuthHistoryReport.csv', data => ... );. If you don't specify an encoding, then the returned data is a buffer, not a string.
fs.readFile('AuthHistoryReport.csv', 'utf8', data => ... ); Returns a string
fs.readFile('AuthHistoryReport.csv', data => ... ); Returns a buffer
Nodejs file system #fs.readFile
You need to store your created file in a buffer using the native package fs
const fs = require('fs');
const buffer = fs.readFileSync('AuthHistoryReport.csv');

Parsing Excel sheet in Hebrew (.xlsx) to JSON produces question marks

I'm trying to parse Excel (*.xlsx) to a JSON object in Node JS , however all the columns with Hebrew characters are converted with question marks.
For example :
Here's the code :
"use strict";
const excelToJson = require("convert-excel-to-json");
// -> Read Excel File to Json Data
const excelData = excelToJson({
sourceFile: "customers.xlsx",
sheets: [
{
// Excel Sheet Name
name: "Customers",
header: {
rows: 1
}
}
]
});
Any idea how to fix it ?
I believe it's only your console that's showing invalid characters. Try dumping the excel file contents to file like so:
"use strict";
const excelToJson = require("convert-excel-to-json");
// -> Read Excel File to Json Data
const excelData = excelToJson({
sourceFile: "customers.xlsx",
sheets: [
{
// Excel Sheet Name
name: "Customers",
header: {
rows: 1
}
}
]
});
const fs = require("fs");
fs.writeFileSync("customers.json", JSON.stringify(excelData));
Then open in say Notepad++. You should see the Hebrew characters correctly. I'm getting exactly this behaviour. I see invalid characters in the command window, but it's all good when I open the customers.json file.
e.g.
{"Customers":[{"A":"לקוח 1"},{"A":"לקוח 2"}]}

How to make complex Json fit a Javascript object

The backend of my webapp, written in node.js interacts with Json file, with a specific format that I thought not so complex but apparently is.
The structure of my json file is as such :
{
"data": [
{
"somefield": "ioremipsum",
"somedate" : "2018-08-23T11:48:00Z",
"someotherdate" : "2018-08-23T13:43:00Z",
"somethingelse":"ioremipsum",
"files": [
{
"specificfieldinarray": "ioremipsum",
"specificotherfieldinarray": "ioremipsum"
},
{
"specificfieldinarray": "ioremipsum",
"specificotherfieldinarray": "ioremipsum"
},
{
"specificfieldinarray": "ioremipsum",
"specificotherfieldinarray": "ioremipsum"
}
]
}
]
}
I try to make this answer fit a JS object like this :
const file = require('specificJsonFile.json');
let fileList = file;
And I need to loop through my 'files' array, for further treatments, but unfortunately, my JS object looks like this :
{ data:
[ { somefield: "ioremipsum",
somedate : "2018-08-23T11:48:00Z",
someotherdate : "2018-08-23T13:43:00Z",
somethingelse:"ioremipsum",
files: [Array] } ] }
Please forgive me if this is obvious, for I am still a beginner with JS.
That's only how console.log logs deep objects. To get a deeper output, you can use util.inspect
const util = require('util');
console.log(util.inspect(yourObject, {showHidden: false, depth: null}));
To loop each data's files, simply loop data, then its files
yourObject.data.forEach(d => {
d.files.forEach(file => console.log(file));
});
It looks like there is nothing wrong there and the console is abbreviating the log.
Try accessing the files list with the following code:
const filesList = file.data[0].files
and then
console.log(filesList) to check that it's eventually working.
Hope it helps!
let fileList = file.data[0].files;
This will create an array of only your files array.
You can console.log(fileList)
Or whatever you like with the data.
Based on your comment, try the of keyword instead of in keyword to get the behaviour you expected.
for (let file of fileList){
console.log(file);
}
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/for...of
You can use for in
for (item in fileList.data) {
for (file in fileList.data[item].files) {
let data = fileList.data[item].files[file];
// process the data
}
}

Writing text to File in Node.JS

I'm new to Node.js. I have a JSON object which looks like the following:
var results = [
{ key: 'Name 1', value: '1' },
{ key: 'Name 2', value: '25%' },
{ key: 'Name 3', value: 'some string' },
...
];
The above object may or may not have different values. Still, I need to get them into a format that looks exactly like the following:
{"Name 1":"1","Name 2":"25%","Name 3":"some string"}
In other words, I'm looping through each key/value pair in results and adding it to a single line. From my understanding this single line approach (with double quotes) is called "JSON Event" syntax. Regardless, I have to print my JSON object out in that way into a text file. If the text file exists, I need to append to it.
I do not know how to append to a text file in Node.js. How do I append to a text file in Node.js?
Thank you!
You can use JSON.stringify to convert a JavaScript object to JSON and fs.appendFile to append the JSON string to a file.
// write all the data to the file
var fs = require('fs');
var str = JSON.stringify(results);
fs.appendFile('file.json', str, function(err) {
if(err) {
console.log('there was an error: ', err);
return;
}
console.log('data was appended to file');
});
If you want to add just one item at a time, just do
// Just pick the first element
var fs = require('fs');
var str = JSON.stringify(results[0]);

Categories