We Keep Coding

Date in XLSX file not parsing correctly in SheetJs

I am trying to read a XLSX file using sheetjs node-module with a column having dates. After parsing I got data in incorrect format
File data is : 2/17/2020
But after xlsx read it gives me 2/17/20. It is changing the year format. My requirement is to get the data as it is.
Sample Code:
var workbook = XLSX.readFile('a.xlsx', {
sheetRows: 10
});
var data = XLSX.utils.sheet_to_json(workbook.Sheets['Sheet1'], {
header: 1,
defval: '',
blankrows: true,
raw: false
});

There's a solution presented here using streams but can equally work with readFile. The problem that was at play in the issue is that the value for the dateNF option in sheet_to_json needs some escape characters.
E.g.:
const XLSX = require('xlsx');
const filename = './Book4.xlsx';
const readOpts = { // <--- need these settings in readFile options
cellText:false,
cellDates:true
};
const jsonOpts = {
header: 1,
defval: '',
blankrows: true,
raw: false,
dateNF: 'd"/"m"/"yyyy' // <--- need dateNF in sheet_to_json options (note the escape chars)
}
const workbook = XLSX.readFile(filename, readOpts);
const worksheet = workbook.Sheets['Sheet1'];
const json = XLSX.utils.sheet_to_json(worksheet, jsonOpts);
console.log(json);
For this input:
Will output:
[
[ 'v1', 'v2', 'today', 'formatted' ],
[ '1', 'a', '14/8/2021', '14/8/2021' ]
]
Without the dateNF: 'd"/"m","/yyyy' you get same problem as you describe in the question:
[
[ 'v1', 'v2', 'today', 'formatted' ],
[ '1', 'a', '8/14/21', '8/14/21' ]
]
The two potential unwanted side effects are:
use of the cellText and cellDates options in readFile
note my custom format of yyyy^mmm^dd in the input - the dateNF setting overrides any custom setting

SequelizeJS - How to Map Nested (eager loading) Models for Raw Queries?

Below is an example of a raw query.
const query = `SELECT
links.name, links.type, links.code, links.originalUrl,
domains.id as 'domain.id', domains.host as 'domain.host',
count(
CASE WHEN hits.datetime > "${past}" AND hits.datetime <= "${now}" = true then 1 END
) as hitCount
FROM links
LEFT JOIN hits ON links.id = hits.linkId
LEFT JOIN domains ON links.domainId = domains.id
WHERE links.userId = ${req.user.id}
GROUP BY links.id, hits.linkId
ORDER BY hitCount DESC
LIMIT 5`;
const links = await sequelize.query(query.trim(), {
type: sequelize.QueryTypes.SELECT,
model: Link,
mapToModel: true
});
I am mapping the query result into the model Link with mapToModel: true. It works well but when I try to get some data from a joined table and map that into an object in model it doesn't convert into array.
For example I am trying to get domains like domains.id as 'domain.id', domains.host as 'domain.host', This is how I saw sequlize does the query for eager loaded data.
But when I get the result object the I don't get a nested domain object property.
// desired link object
{
id: 3,
name: 'My test link',
domain: {
id: 23,
host: 'example.com'
}
}
instead what I get is
// current link object
{
id: 3,
name: 'My test link',
'domain.id': 23,
'domain.host': 'example.com'
}
So nested objects aren't mapping correctly.
UPDATE
I have found nest options in query() documentation but setting nest: true doesn't seems to have any effect.
Changed Query
const links = await sequelize.query(query.trim(), {
type: sequelize.QueryTypes.SELECT,
model: Link,
mapToModel: true,
nest: true, // doesn't have any effect
});

For me nest worked only in conjunction with raw:
const links = await sequelize.query(query.trim(), {
type: sequelize.QueryTypes.SELECT,
model: Link,
mapToModel: true,
nest: true,
raw: true // Without this `nest` hasn't effect, IDK why
});

I have run into the same problem. My solution was:
return sequelize.query(/* query */, {
nest: true,
type: sequelize.QueryTypes.SELECT
});
As you see, you should not set mapToModel prop, just nest and type.

You can install dottie yourself, which is what nest: true is supposed to be doing (https://github.com/mickhansen/dottie.js):
import dottie from 'dottie';
// ...
const formattedLinks = links.map(l => dottie.transform(l.toJSON()))

It seems that raw queries with nest:true doesn't works with mapToModel. So the only way I found is to include nest:true, map with Link.build(link) and { include: Domain} option:
const query = `SELECT
links.name, links.type, links.code, links.originalUrl,
domains.id as 'domain.id', domains.host as 'domain.host',
count(
CASE WHEN hits.datetime > "${past}" AND hits.datetime <= "${now}" = true then 1 END
) as hitCount
FROM links
LEFT JOIN hits ON links.id = hits.linkId
LEFT JOIN domains ON links.domainId = domains.id
WHERE links.userId = ${req.user.id}
GROUP BY links.id, hits.linkId
ORDER BY hitCount DESC
LIMIT 5`;
const links = (await sequelize.query(query.trim(), {
nest: true,
type: sequelize.QueryTypes.SELECT,
}));
links.map((link) => Link.build(link, {include: Domain}))

As far as I can see the Link model has one domain with association so you will need to pass
include: <domain model> to map to domain model.
have a look at this : https://github.com/sequelize/sequelize/issues/1830

How to add more metrics on the country_map in Apache-superset?

I am using country_map in apache-superset for visualization purposes. When zooming in on a polygon, information from the columns appears inside of the polygon, like so:
There is only one available metric option to display:
Code for the metric update is found on this path:
superset/assets/src/visualizations/CountryMap/CountryMap.js
Code:
const updateMetrics = function (region) {
if (region.length > 0) {
resultText.text(format(region[0].metric));
}
};
The metrics are defined in controls.jsx:
/superset/static/assets/src/explore/controls.jsx
const metrics = {
type: 'MetricsControl',
multi: true,
label: t('Metrics'),
validators: [v.nonEmpty],
default: (c) => {
const metric = mainMetric(c.savedMetrics);
return metric ? [metric] : null;
},
mapStateToProps: (state) => {
const datasource = state.datasource;
return {
columns: datasource ? datasource.columns : [],
savedMetrics: datasource ? datasource.metrics : [],
datasourceType: datasource && datasource.type,
};
},
description: t('One or many metrics to display'),
};
const metric = {
...metrics,
multi: false,
label: t('Metric'),
default: props => mainMetric(props.savedMetrics),
};
Country map is using metric, which doesn't allow multiple metrics to be selected, Code found here:
superset/assets/src/explore/controlPanels/CountryMap.js
controlPanelSections: [
{
label: t('Query'),
expanded: true,
controlSetRows: [
['entity'],
['metric'],
['adhoc_filters'],
],
},
{
label: t('Options'),
expanded: true,
controlSetRows: [
['select_country', 'number_format'],
['linear_color_scheme'],
],
},
],
The python class of country_map is located at viz.py:
class CountryMapViz(BaseViz):
"""A country centric"""
viz_type = 'country_map'
verbose_name = _('Country Map')
is_timeseries = False
credits = 'From bl.ocks.org By john-guerra'
def query_obj(self):
qry = super(CountryMapViz, self).query_obj()
qry['metrics'] = [
self.form_data['metric']]
qry['groupby'] = [self.form_data['entity']]
return qry
Changing the code in CountryMap.js and viz.py from metric to metrics results in the following error:
Traceback (most recent call last):
File "/Documents/superset/superset/superset/viz.py", line 410, in get_df_payload
df = self.get_df(query_obj)
File "/Documents/superset/superset/superset/viz.py", line 213, in get_df
self.results = self.datasource.query(query_obj)
File "/Documents/superset/superset/superset/connectors/sqla/models.py", line 797, in query
sql = self.get_query_str(query_obj)
File "/Documents/superset/superset/superset/connectors/sqla/models.py", line 471, in get_query_str
qry = self.get_sqla_query(**query_obj)
File "/Documents/superset/superset/superset/connectors/sqla/models.py", line 585, in get_sqla_query
elif m in metrics_dict:
TypeError: unhashable type: 'list'
How can I add more metrics to display inside the polygon?

The direct cause of the error TypeError: unhashable type: 'list' is your modification to file "viz.py":
self.form_data['metric']] to self.form_data['metrics']], in the query_obj(self) method.
As you can see in the source code here, form data metrics is a list object that contains metric, where metric is probably a string or other hashable object. In python language, a list object is not hashable. Because you replace a hashable object (metric) with an unhashable one (metrics), an unhashable type error is then raised.
The correct way to modify CoutryMapViz.query_obj() to accept metrics query can be found in the other Viz classes. The code section here is a very nice example:
class CalHeatmapViz(BaseViz):
"""Calendar heatmap."""
...
def query_obj(self):
d = super(CalHeatmapViz, self).query_obj()
fd = self.form_data
d['metrics'] = fd.get('metrics')
return d
Finally, the CoutryMapViz.query_obj() method should look like this:
class CountryMapViz(BaseViz):
...
def query_obj(self):
qry = super(CountryMapViz, self).query_obj()
qry['metrics'] = fd.get('metrics')
qry['groupby'] = [self.form_data['entity']]
return qry

Clean input before change

I'm using handsontable 0.35.1 with a float column. Aim is to allow users to copy paste from spreadsheets (and csvs opened in excel). Problem is, that it comes with some junk that i need to get rid of. Some examples of inputs which are not correctly validated are:
1,000.00
USD 100.00
'10000.00 ' //note there are trailing spaces
I would like to find a way i can manipulate input right before it's written. The only way i've found so far is with beforeChange, but the problem is validation. The system changes input, but seems to have validated already. If i blur in and blur out again, it works.
Here's the fiddle. Steps to reproduce: Enter number a123 -- which should be corrected to 123 and validated as a correct number.
I've tried using beforeValidation instead, but it doesn't work as i intend.

You can use beforePaste callback to clean your input
options = {
columns: [
{ type: 'date' },
{ type: 'numeric', numericFormat: {pattern: '0,0.00'} }
],
colHeaders: ["Date", "Float"],
beforePaste: (data, coords) => {
data.forEach((col, colIndex) => {
col.forEach((row, rowIndex) => {
let value = row.trim().replace(/[^0-9\.\-\+]/g, '')
data[colIndex][rowIndex] = value
})
})
return true
},
}
$('#hot-sheet').handsontable(options)
here is fiddle https://jsfiddle.net/xpvt214o/348195/
Note: you can't create new data array, you've to update data array instead of creating new.

I updated the example https://jsfiddle.net/fma4uge8/29/ this works all in 1 function.
function trimFloat(value) {
return value.trim().replace(/[^0-9\.\-\+]/g, '');
}
options = {
columns: [
{ type: 'date' },
{ type: 'numeric', numericFormat: {pattern: '0,0.00'}, trimWhitespace: true }
],
colHeaders: ["Date", "Float"],
beforeChange: function(changes, source){
let that = this;
_.each(changes, function(change){
if (_.isString(change[3])) {
let value = trimFloat(change[3]);
//prevent endless loop
if (value !== change[3]) {
change[3] = trimFloat(change[3]);
that.setDataAtCell(change[0], change[1], change[3]);
}
}
})
}
}
$('#hot-sheet').handsontable(options)

pg-promise ColumnSet use Postgres functions with def property

I am using a ColumnSet and the helper.insert function for a multi row insert.
I have a table column where I want to use the Postgres Date/Time now() function.
const cs = new helpers.ColumnSet([
'lastname',
{
name: 'rental_date',
def: 'now()'
}
], { table: { table: 'book_rental', schema: 'public' } })
let rentals = [
{
lastname: 'Mueller'
},
{
lastname: 'Johnson'
}
]
let insert = helpers.insert(rentals, cs)
db.result(insert)
.then(data => res.json({ message: 'Ok!' }))
.catch(err => res.json({ message: 'Not ok!' }))
It seems to be working by using def: 'now()', but I want to make sure that I am using it the right way.
Edit:
Regarding the answer in the comment. I tried to do the insert manually and it looks like Postgres is converting the 'now()' string into the now() function.
INSERT INTO book_rental (lastname, rental_date) VALUES ('Mueller', 'now()');
To involve your answer, am I right that this should be the correct code then?
const cs = new helpers.ColumnSet([
'lastname',
{
name: 'rental_date',
mod: ':raw',
def: 'now()'
}
], { table: { table: 'book_rental', schema: 'public' } })

Your code doesn't look right, for the following reasons:
You want to use now() without any condition, but the def value is only used when the property doesn't exist in the source object (see Column). The init callback is what should be used instead to guarantee the right value override.
You return now() as an escaped string, while the query needs it as a raw-text string.
First, let's declare a reusable Raw Text string, as per Custom Type Formatting:
const rawText = text => ({toPostgres: () => text, rawType: true});
Then you can define the column like this:
{
name: 'rental_date',
init: () => rawText('now()')
}
And make sure you are using the latest version of pg-promise (v7.2.1 as of this writing).
Or alternatively, you can declare it like this:
{
name: 'rental_date',
mod: ':raw', // same as mode: '^'
init: () => 'now()'
}
This syntax however will work in all versions of the library, and perhaps is even simpler to use ;)