I am trying to parse some csv data, however, I am not able to parse date with momentjs library.
var csv = require('csv-parser');
var fs = require('fs');
var moment = require('moment');
const bittrexDateFormat = "MM/DD/YYYY hh:mm:ss a";
var count = 0;
fs.createReadStream('orders.csv')
.pipe(csv({
headers: ['OrderUuid', 'Exchange', 'Type', 'Quantity', 'Limit', 'CommissionPaid', 'Price', 'Opened', 'Closed']
}))
.on('data', function(data) {
var createDate = moment(data.Opened, bittrexDateFormat);
console.log(createDate.toDate());
});
And the csv data looks like;
OrderUuid,Exchange,Type,Quantity,Limit,CommissionPaid,Price,Opened,Closed
24245deb-134c-4da7-990e-8d22d8fd728c,BTC-STRAT,LIMIT_SELL,77.12739479,0.00087503,0.00016874,0.06749952,12/24/2017 12:09:20 AM,12/24/2017 12:09:21 AM
And this the output;
0002-01-02T09:00:02.000Z
On the other hand, if I directly hardcode the date string I am able to get Date object.
var createDate = moment("12/24/2017 12:09:20 AM", bittrexDateFormat);
console.log(createDate.toDate());
Another thing I figured out is if I print data in event .on('data') it prints encoded string version
Row {
OrderUuid: 'O\u0000r\u0000d\u0000e\u0000r\u0000U\u0000u\u0000i\u0000d\u0000',
Exchange: '\u0000E\u0000x\u0000c\u0000h\u0000a\u0000n\u0000g\u0000e\u0000',
Type: '\u0000T\u0000y\u0000p\u0000e\u0000',
Quantity: '\u0000Q\u0000u\u0000a\u0000n\u0000t\u0000i\u0000t\u0000y\u0000',
Limit: '\u0000L\u0000i\u0000m\u0000i\u0000t\u0000',
CommissionPaid: '\u0000C\u0000o\u0000m\u0000m\u0000i\u0000s\u0000s\u0000i\u0000o\u0000n\u0000P\u0000a\u0000i\u0000d\u0000',
Price: '\u0000P\u0000r\u0000i\u0000c\u0000e\u0000',
Opened: '\u0000O\u0000p\u0000e\u0000n\u0000e\u0000d\u0000',
Closed: '\u0000C\u0000l\u0000o\u0000s\u0000e\u0000d\u0000\r\u0000' }
Row {
OrderUuid: '\u00002\u00004\u00002\u00004\u00005\u0000d\u0000e\u0000b\u0000-\u00001\u00003\u00004\u0000c\u0000-\u00004\u0000d\u0000a\u00007\u0000-\u00009\u00009\u00000\u0000e\u0000-\u00008\u0000d\u00002\u00002\u0000d\u00008\u0000f\u0000d\u00007\u00002\u00008\u0000c\u0000',
Exchange: '\u0000B\u0000T\u0000C\u0000-\u0000S\u0000T\u0000R\u0000A\u0000T\u0000',
Type: '\u0000L\u0000I\u0000M\u0000I\u0000T\u0000_\u0000S\u0000E\u0000L\u0000L\u0000',
Quantity: '\u00007\u00007\u0000.\u00001\u00002\u00007\u00003\u00009\u00004\u00007\u00009\u0000',
Limit: '\u00000\u0000.\u00000\u00000\u00000\u00008\u00007\u00005\u00000\u00003\u0000',
CommissionPaid: '\u00000\u0000.\u00000\u00000\u00000\u00001\u00006\u00008\u00007\u00004\u0000',
Price: '\u00000\u0000.\u00000\u00006\u00007\u00004\u00009\u00009\u00005\u00002\u0000',
Opened: '\u00001\u00002\u0000/\u00002\u00004\u0000/\u00002\u00000\u00001\u00007\u0000 \u00001\u00002\u0000:\u00000\u00009\u0000:\u00002\u00000\u0000 \u0000A\u0000M\u0000',
Closed: '\u00001\u00002\u0000/\u00002\u00004\u0000/\u00002\u00000\u00001\u00007\u0000 \u00001\u00002\u0000:\u00000\u00009\u0000:\u00002\u00001\u0000 \u0000A\u0000M\u0000' }
I am pretty new to nodejs but I don't think the problem occurs from either momentjs or csv-parser libraries. Instead it should be string format of stream api nodejs. Thanks a lot.
I just ran your code, and it worked just fine.
Try updating your node version or try invoking the function with
fs.createReadStream('orders.csv', 'utf16le')
the encoding should make the difference...
https://nodejs.org/api/fs.html#fs_fs_createreadstream_path_options
Related
I am trying to read a XLSX file using sheetjs node-module with a column having dates. After parsing I got data in incorrect format
File data is : 2/17/2020
But after xlsx read it gives me 2/17/20. It is changing the year format. My requirement is to get the data as it is.
Sample Code:
var workbook = XLSX.readFile('a.xlsx', {
sheetRows: 10
});
var data = XLSX.utils.sheet_to_json(workbook.Sheets['Sheet1'], {
header: 1,
defval: '',
blankrows: true,
raw: false
});
There's a solution presented here using streams but can equally work with readFile. The problem that was at play in the issue is that the value for the dateNF option in sheet_to_json needs some escape characters.
E.g.:
const XLSX = require('xlsx');
const filename = './Book4.xlsx';
const readOpts = { // <--- need these settings in readFile options
cellText:false,
cellDates:true
};
const jsonOpts = {
header: 1,
defval: '',
blankrows: true,
raw: false,
dateNF: 'd"/"m"/"yyyy' // <--- need dateNF in sheet_to_json options (note the escape chars)
}
const workbook = XLSX.readFile(filename, readOpts);
const worksheet = workbook.Sheets['Sheet1'];
const json = XLSX.utils.sheet_to_json(worksheet, jsonOpts);
console.log(json);
For this input:
Will output:
[
[ 'v1', 'v2', 'today', 'formatted' ],
[ '1', 'a', '14/8/2021', '14/8/2021' ]
]
Without the dateNF: 'd"/"m","/yyyy' you get same problem as you describe in the question:
[
[ 'v1', 'v2', 'today', 'formatted' ],
[ '1', 'a', '8/14/21', '8/14/21' ]
]
The two potential unwanted side effects are:
use of the cellText and cellDates options in readFile
note my custom format of yyyy^mmm^dd in the input - the dateNF setting overrides any custom setting
I need to get the current time, according to the database timezone (not my local timezone, and not default UTC timezone), using Sequelize. Is there a Sequelize method to do this?
My database is in Eastern time, and when I query the db directly SELECT CURRENT_TIMESTAMP; it returns the date/time in Eastern time (which is correct).
But when I query via Sequelize in Node const [[{time}]] = await db.sequelize.query('SELECT CURRENT_TIMESTAMP AS time'); it returns the date/time in UTC.
Two problems:
1 - I would prefer using a Sequelize method instead of a raw query.
2 - This still doesn't get me the result I want. The time needs to be Eastern.
This is my DB setup:
const sequelize = new Sequelize(dbUrl, {
dialectOptions: {
useUTC: false // for reading from database
},
timezone: '-04:00', // for writing to database
define: {
charset: 'utf8'
}
})
As mentioned above, when I query using the above queries, the date is always returned in UTC, which I did not expect, given I said useUTC: false. How do I get it in Eastern time (the database timezone)?
I'm not aware of a sequelize method like getCurrentDate().
The UTC conversion problem seems to bite everyone (myself included). Here are some details. Not sure if dialectOptions: {useUTC: false }, has any function at all - just adding the typeCast method solved the problem for me.
dialectOptions: {
typeCast: function (field, next) { // for reading from database
if (field.type === 'DATETIME') {
return field.string()
}
return next()
},
The result can be used for a new js Date object:
const sql = 'select current_timestamp';
my_app.my_DB.query(sql, {raw: true, type: Sequelize.QueryTypes.SELECT})
.then(data => {
console.log(data[0].current_timestamp);
let d1 = new Date(data[0].current_timestamp);
});
This works fine for me - but make sure to test thoroughly!
Add useUTC property in your dialectOptions like this
dialectOptions: {
encrypt: false ,
options: {
useUTC: false, // for reading from database
},
},
I have variable formValuesChanged that contains the following data:
Then I save this data to storage:
sessionStorage.setItem("filters", JSON.stringify(formValuesChanged));
Then if explore session storage I see this object but with different date:
{ classNumber: 2
classSuffix: 5772436057100
date: "2018-09-23T21:07:47.043Z"
teacher: 25203
type: "teacher"
}
Why does JS save wrong date?
There is no date type in the JSON format, it is a bit of a pain but if you want to store a date in a string format you will need to parse it when you read it back.
let obj = JSON.parse(sessionStorage.getItem("filters"));
let objWithDate = { ...obj, date: new Date(obj.date) };
When I export a table to excel using ag-grid's own exportDataAsExcel() the resulting excel contains dates as General data type instead of Date.
[]
I have used this:
exportDataAsExcel({
processCellCallback: ({col, val}) => {( /*date value formatting here*/ )}
})
to format both Date, string with proper date formatting (DD/MM/YYYY) but I can't make excel properly recognize these cells as Date instead of General
This is reproducible with the excel export examples on their website: https://www.ag-grid.com/javascript-grid-excel/?framework=all#gsc.tab=0
I've used this code to apply value formatter in excell as well:
this.gridOptions.api.exportDataAsExcel({
processCellCallback: (params) => {
const colDef = params.column.getColDef()
// try to reuse valueFormatter from the colDef
if (colDef.valueFormatter) {
const valueFormatterParams: ValueFormatterParams = {
...params,
data: params.node.data,
node: params.node!,
colDef: params.column.getColDef()
};
return colDef.valueFormatter(valueFormatterParams);
}
return params.value;
},
});
You need to do three things:
When instantiating the Ag-Grid, you need to add (thanks to another user's answer elsewhere)
let excelStyles = [
{
id: "ExcelDateTime",
dataType: "dateTime",
numberFormat: { format: "yyyy-mm-dd hh:mm:ss;;;" }
},
{
id: "ExcelDate",
dataType: "dateTime",
numberFormat: { format: "yyyy-mm-dd;;;" }
}
];
...
<AgGridReact
... //such as rowData={rowData}
excelStyles={excelStyles}
...
/>;
For columns that are dates:
colDef.cellClass = "ExcelDateTime";
or
colDef.cellClass = "ExcelDate";
as appropriate, and
in your processCellCallback, format the date as an ISO date: "yyyy-mm-ddTHH:MM:ss"
Assuming that you are in control of the server-side application, I have found it easier to do the export from the server-side for any non-trivial exports / formatting.
By doing it on the server, you are in complete control of the data and the production of the Excel file, and not dependent on the limits of Ag-Grid's implementation.
I am working in a Sencha Touch app, backend is running with SAP, and is sending in the json this odata string format '20140811'(for instance) if I define in the model this type nothing happens..
{
name: "createdOn",
type: "date",
dateFormat: "DD-MM-Y"
}
How should I convert correctly Sap date format to Sencha type?
My solution was...
{
name: 'lastSurvey',
type: 'auto',
convert: function(date){
var sNumber = date.replace(/[^0-9]+/g, ''),
iNumber = sNumber * 1, //trick seventeen
oDate = new Date(iNumber);
return Ext.Date.format(oDate, 'd-m-Y');
}
},
Opinions about this solution are welcome!