I am trying to read a XLSX file using sheetjs node-module with a column having dates. After parsing I got data in incorrect format
File data is : 2/17/2020
But after xlsx read it gives me 2/17/20. It is changing the year format. My requirement is to get the data as it is.
Sample Code:
var workbook = XLSX.readFile('a.xlsx', {
sheetRows: 10
});
var data = XLSX.utils.sheet_to_json(workbook.Sheets['Sheet1'], {
header: 1,
defval: '',
blankrows: true,
raw: false
});
There's a solution presented here using streams but can equally work with readFile. The problem that was at play in the issue is that the value for the dateNF option in sheet_to_json needs some escape characters.
E.g.:
const XLSX = require('xlsx');
const filename = './Book4.xlsx';
const readOpts = { // <--- need these settings in readFile options
cellText:false,
cellDates:true
};
const jsonOpts = {
header: 1,
defval: '',
blankrows: true,
raw: false,
dateNF: 'd"/"m"/"yyyy' // <--- need dateNF in sheet_to_json options (note the escape chars)
}
const workbook = XLSX.readFile(filename, readOpts);
const worksheet = workbook.Sheets['Sheet1'];
const json = XLSX.utils.sheet_to_json(worksheet, jsonOpts);
console.log(json);
For this input:
Will output:
[
[ 'v1', 'v2', 'today', 'formatted' ],
[ '1', 'a', '14/8/2021', '14/8/2021' ]
]
Without the dateNF: 'd"/"m","/yyyy' you get same problem as you describe in the question:
[
[ 'v1', 'v2', 'today', 'formatted' ],
[ '1', 'a', '8/14/21', '8/14/21' ]
]
The two potential unwanted side effects are:
use of the cellText and cellDates options in readFile
note my custom format of yyyy^mmm^dd in the input - the dateNF setting overrides any custom setting
Related
I am trying to render an excel file in Table format with Exceljs but I am getting this warning before opening the file:
We found a problem with some content in ’test.xlsx’. Do you want us to try to recover as much as we can? If you trust the source of this workbook, click Yes.
If I click yes it 'recovers' the file and everything is perfect, but I always get that warning before opening.
This only happens when I do more than one tab, for a single one works fine.
import Excel from 'exceljs'
const tabs = {
'FIRST TAB': [
{ URL: 'https://google.com', FOO: 10 },
{ URL: 'https://apple.com', FOO: 12.5 }
],
'SECOND TAB': [
{ URL: 'https://google.com', FOO: 10 },
{ URL: 'https://apple.com', FOO: 22.5 }
]
}
;(async () => {
const workbook = new Excel.Workbook()
let worksheet = {}
for (const [label, tab] of Object.entries(tabs)) {
worksheet = workbook.addWorksheet(label)
worksheet.state = 'visible'
const columns = Object.keys(tab[0]).map((items) => ({
name: items,
filterButton: true
}))
const rows = tab.map((entry) => Object.values(entry))
workbook.getWorksheet(label).addTable({
name: label,
ref: 'A1',
headerRow: true,
columns,
rows
})
}
// Write to excel
await workbook.xlsx.writeFile(`test.xlsx`)
})()
The problem is caused by the space in the table name.
One way to fix it would be to replace the space with an underscore, that's actually what Excel does when it 'fixes' the file.
workbook.getWorksheet(label).addTable({
name: label.replace(' ', '_'),
ref: 'A1',
headerRow: true,
columns,
rows
})
For me, this is caused by having a cell whose content is a string with length >=32768. I have to trim the string down to 32767 bytes.
I am using csvtojson npm library, and i read options, but cannot figure out, how can i make separate json files from CSV columns.
The CSV format is :
invoiceNumber;taxNumber
test-1;00000000
test-2
for options i am using: {columns: false, delimiter: ';'}
and the output is :
{ invoiceNumber: 'test-1', taxNumber: '00000000' }
{ invoiceNumber: 'test-2' }
however the expected output is :
{ invoiceNumber: 'test-1'}
{ taxNumber: '00000000' }
{ invoiceNumber: 'test-2' }
Is it possible with any csv parser npm library?
For this task, I recommend using lodash. After csttojson try this script
const _ = require('lodash')
const arr = [{ invoiceNumber: 'test-1', taxNumber: '00000000' },
{ invoiceNumber: 'test-2' }];
console.log(_.flatMap(arr, item=>_.map(_.toPairs(item), d => _.fromPairs([d]))))
Iam able to generate a csv file with the data below. I am using a nodejs library "csv-writer" that generates the file quite well. My problem is that I need a way to return back a buffer instead of the file itself. Reason being I need to upload the file to a remote server via sftp.
How do I go ab bout modifying this piece of code to enable buffer response? Thanks.
...
const csvWriter = createCsvWriter({
path: 'AuthHistoryReport.csv',
header: [
{id: 'NAME', title: 'msg_datetime_date'},
{id: 'AGE', title: 'msg_datetime'}
]
});
var rows = [
{ NAME: "Paul", AGE:21 },
{ NAME: "Charles", AGE:28 },
{ NAME: "Teresa", AGE:27 },
];
csvWriter
.writeRecords(rows)
.then(() => {
console.log('The CSV file was written successfully');
});
...
Read your own file with fs.readFile('AuthHistoryReport.csv', data => ... );. If you don't specify an encoding, then the returned data is a buffer, not a string.
fs.readFile('AuthHistoryReport.csv', 'utf8', data => ... ); Returns a string
fs.readFile('AuthHistoryReport.csv', data => ... ); Returns a buffer
Nodejs file system #fs.readFile
You need to store your created file in a buffer using the native package fs
const fs = require('fs');
const buffer = fs.readFileSync('AuthHistoryReport.csv');
I'm trying to parse Excel (*.xlsx) to a JSON object in Node JS , however all the columns with Hebrew characters are converted with question marks.
For example :
Here's the code :
"use strict";
const excelToJson = require("convert-excel-to-json");
// -> Read Excel File to Json Data
const excelData = excelToJson({
sourceFile: "customers.xlsx",
sheets: [
{
// Excel Sheet Name
name: "Customers",
header: {
rows: 1
}
}
]
});
Any idea how to fix it ?
I believe it's only your console that's showing invalid characters. Try dumping the excel file contents to file like so:
"use strict";
const excelToJson = require("convert-excel-to-json");
// -> Read Excel File to Json Data
const excelData = excelToJson({
sourceFile: "customers.xlsx",
sheets: [
{
// Excel Sheet Name
name: "Customers",
header: {
rows: 1
}
}
]
});
const fs = require("fs");
fs.writeFileSync("customers.json", JSON.stringify(excelData));
Then open in say Notepad++. You should see the Hebrew characters correctly. I'm getting exactly this behaviour. I see invalid characters in the command window, but it's all good when I open the customers.json file.
e.g.
{"Customers":[{"A":"לקוח 1"},{"A":"לקוח 2"}]}
I am trying to parse some csv data, however, I am not able to parse date with momentjs library.
var csv = require('csv-parser');
var fs = require('fs');
var moment = require('moment');
const bittrexDateFormat = "MM/DD/YYYY hh:mm:ss a";
var count = 0;
fs.createReadStream('orders.csv')
.pipe(csv({
headers: ['OrderUuid', 'Exchange', 'Type', 'Quantity', 'Limit', 'CommissionPaid', 'Price', 'Opened', 'Closed']
}))
.on('data', function(data) {
var createDate = moment(data.Opened, bittrexDateFormat);
console.log(createDate.toDate());
});
And the csv data looks like;
OrderUuid,Exchange,Type,Quantity,Limit,CommissionPaid,Price,Opened,Closed
24245deb-134c-4da7-990e-8d22d8fd728c,BTC-STRAT,LIMIT_SELL,77.12739479,0.00087503,0.00016874,0.06749952,12/24/2017 12:09:20 AM,12/24/2017 12:09:21 AM
And this the output;
0002-01-02T09:00:02.000Z
On the other hand, if I directly hardcode the date string I am able to get Date object.
var createDate = moment("12/24/2017 12:09:20 AM", bittrexDateFormat);
console.log(createDate.toDate());
Another thing I figured out is if I print data in event .on('data') it prints encoded string version
Row {
OrderUuid: 'O\u0000r\u0000d\u0000e\u0000r\u0000U\u0000u\u0000i\u0000d\u0000',
Exchange: '\u0000E\u0000x\u0000c\u0000h\u0000a\u0000n\u0000g\u0000e\u0000',
Type: '\u0000T\u0000y\u0000p\u0000e\u0000',
Quantity: '\u0000Q\u0000u\u0000a\u0000n\u0000t\u0000i\u0000t\u0000y\u0000',
Limit: '\u0000L\u0000i\u0000m\u0000i\u0000t\u0000',
CommissionPaid: '\u0000C\u0000o\u0000m\u0000m\u0000i\u0000s\u0000s\u0000i\u0000o\u0000n\u0000P\u0000a\u0000i\u0000d\u0000',
Price: '\u0000P\u0000r\u0000i\u0000c\u0000e\u0000',
Opened: '\u0000O\u0000p\u0000e\u0000n\u0000e\u0000d\u0000',
Closed: '\u0000C\u0000l\u0000o\u0000s\u0000e\u0000d\u0000\r\u0000' }
Row {
OrderUuid: '\u00002\u00004\u00002\u00004\u00005\u0000d\u0000e\u0000b\u0000-\u00001\u00003\u00004\u0000c\u0000-\u00004\u0000d\u0000a\u00007\u0000-\u00009\u00009\u00000\u0000e\u0000-\u00008\u0000d\u00002\u00002\u0000d\u00008\u0000f\u0000d\u00007\u00002\u00008\u0000c\u0000',
Exchange: '\u0000B\u0000T\u0000C\u0000-\u0000S\u0000T\u0000R\u0000A\u0000T\u0000',
Type: '\u0000L\u0000I\u0000M\u0000I\u0000T\u0000_\u0000S\u0000E\u0000L\u0000L\u0000',
Quantity: '\u00007\u00007\u0000.\u00001\u00002\u00007\u00003\u00009\u00004\u00007\u00009\u0000',
Limit: '\u00000\u0000.\u00000\u00000\u00000\u00008\u00007\u00005\u00000\u00003\u0000',
CommissionPaid: '\u00000\u0000.\u00000\u00000\u00000\u00001\u00006\u00008\u00007\u00004\u0000',
Price: '\u00000\u0000.\u00000\u00006\u00007\u00004\u00009\u00009\u00005\u00002\u0000',
Opened: '\u00001\u00002\u0000/\u00002\u00004\u0000/\u00002\u00000\u00001\u00007\u0000 \u00001\u00002\u0000:\u00000\u00009\u0000:\u00002\u00000\u0000 \u0000A\u0000M\u0000',
Closed: '\u00001\u00002\u0000/\u00002\u00004\u0000/\u00002\u00000\u00001\u00007\u0000 \u00001\u00002\u0000:\u00000\u00009\u0000:\u00002\u00001\u0000 \u0000A\u0000M\u0000' }
I am pretty new to nodejs but I don't think the problem occurs from either momentjs or csv-parser libraries. Instead it should be string format of stream api nodejs. Thanks a lot.
I just ran your code, and it worked just fine.
Try updating your node version or try invoking the function with
fs.createReadStream('orders.csv', 'utf16le')
the encoding should make the difference...
https://nodejs.org/api/fs.html#fs_fs_createreadstream_path_options