I am working on a code optimization where I have several rows (~500-1000) of data with at least 300 columns.
The data is stored as an array of objects:
data = [
{col1: 1, col2: 2, col3: null, col4: 1.3, col5:undefined, col6: 2, ....},
{col1: 3, col2: 4, col3: 3, col4: 1.1, col5:8, col6: 2, ....},
.
.
.
{col1: 1.1, col2: 1.2, col3: 7, col4: 3, col5:4, col6: 2, ....}
]
Then I have another array of objects named calculators, these are user-defined calculations. These calculations can have another calculated variable like 2nd object in the following array.
So I need to maintain the order.
calculators = [
{formula:'col1 + sin(col2)', order:0, colname: 'calc1'}
{formula:'calc1 * col6', order:1, colname: 'calc2'}
.
.
.
{formula: 'any random calculation', order:n, colname:'RandomName'}
]
my current code-flow is,
Extraction of dependent columns from calculators, like col1 and col2
for calc1 and col6 from calc2.
dependentCols = [col1, col2, col6]
Looping through the data array with nested for loops for calculators and columns and setting the value in parser for each
dependent column. parser.set(column, row[column]);
Then formula evaluation and appending that value to the dararow.
The original code I have includes some other stuff to handle sub-columns, units, and other stuff. That's why instead of pasting the original code, here is the rough draft.
const parser = math.parser();
data = _.map(data, row=>{
row = {...row};
_.forEach(calculators, (calculator)=>{
_.forEach(dependentCols, col=>{
// cannot add the whole row as it is because of the sub-column stuff.
parser.set(col, row[col]);
});
try {
const value = parser.eval(calculator.formula);
row[colname] = value;
} catch (exception) {
console.warn(parser, exception, calculator.colname);
}
}
});
return row;
});
}
The original code is definitely working (for a year). Its just very slow, with 450 rows and 20 calculations its taking upto 12 seconds.
So instead of going row by row and effectively cell by cell I am thinking about passing the whole column at the same time to the eval function. But I couldn't find any supporting documentation.
Any help will be appreciated.
Related
I have 2 arrays:
[{ Name: ‘Bart’, col2: +4, col3:4},
{ Name: ‘Marge’, col2: +8, col3:},
{ Name: ‘Bart’, col2: -3 , col3:8},
{ Name: ‘Homer’, col2: +12, col3:4},
{ Name: ‘Bart’, col2: +12, col3:2},
{ Name: ‘Homer’, col2: +2, col3:13}]
and
[{ Name: ‘Bart’, col2:4},
{ Name: ‘Marge’, col2:12},
{ Name: ‘Bart’, col2:4},
{ Name: ‘Bart’, col2:4},
{ Name: ‘Homer’, col2:4}]
I want the update the contents from col3 in array1 with the contents from col2 in array2. Col2 in array2 contains multiple times the correct values (the same ones for each name) and I want to update col3 in array1 with these values.
I know how to do this in Python but I'm completely lost in javascript.
*** edit
I tried this:
array1.forEach((x) => {
const bla = array2.find((y) => y.Name === x.Name);
x.col3 = bla.col2;
});
and it is in the last part that I got stuck ... How to update (replace) the value with another one.
I know, data is a bit weird, I'm trying to make one array (or object) of 3 different ones. And yes, the values in array2/col2 are the same for 'Bart', 'Marge',... but in array1 they are a bit random.
The code below is my current code that works fine. It lets me import all data from one spreadsheet tab to another spreadsheet giving me the full range A:J. But now I do not want all data to be imported but I want to filter out specific stores (in sales data) based on their id number. The ID numbers are located in column C in source spreadsheet. I want to Filter for 8 different values in column C and import all data matching that condition into the targeted spreadsheet.
function importhistorical_sales_ee() {
//geth values to be imported from the source sheet
var values = SpreadsheetApp.openById('1mVECYu27lnOIFf7vHk1kI_55DID3APvWr_toWuZho14').
getSheetByName('Historical sales for SC planning -.csv').getRange('A:J').getValues();
//set values imported
var target = SpreadsheetApp.getActiveSpreadsheet().getSheetByName('test_ee');
target.getRange("A3:J").clearContent();
target.getRange(3,1,values.length,values[0].length).setValues(values);
}
I tested several options like the one below (but it is not working). I am expecting to be able to set the column I want to filter on and then also set which values I want to look for in that column.
Here is a simple example.
I have sample data in a spreadsheet like this.
I then filter the data for an array of 3 values as shown in the script. You can make it 8 or whatever you want. I prefer to use getDataRange() but you could use getRange(1,1,sheet.getLastRow(),9).getValues(). This ensures your not getting a bunch of blank rows.
Code.gs
function filterStores () {
try {
let wanted = [1,3,5];
let spread = SpreadsheetApp.getActiveSpreadsheet();
let sheet = spread.getSheetByName("Sheet1");
let stores = sheet.getRange(1,1,sheet.getLastRow(),9).getValues();
let filtered = stores.filter( store => {
return wanted.some( id => store[2] === id );
}
);
let target = SpreadsheetApp.getActiveSpreadsheet().getSheetByName('test_ee');
target.getRange(3,1,target.getLastRow()-2,9).clearContent();
target.getRange(3,1,filtered.length,filtered[0].length).setValues(filtered);
console.log(filtered);
}
catch(err) {
console.log(err);
}
}
Execution log
9:16:03 AM Notice Execution started
9:16:05 AM Info [ [ 'a1', 'b1', 1, 'd1', 'e1', '', 'g1', '', '', 'j1' ],
[ 'a3', 'b3', 3, 'd3', 'e3', '', 'g3', '', '', 'j3' ],
[ 'a5', 'b5', 5, 'd5', 'e5', '', 'g5', '', '', 'j5' ],
[ 'a3', 'b3', 3, 'd3', 'e3', '', 'g3', '', '', 'j3' ] ]
9:16:05 AM Notice Execution completed
References
Array.filtere()
Array.some()
Arrow function
I am trying to find the best way to store in Redux a 2D array with row and column names.
In terms of priorities, I would rank code readability and maintainability above performance. Hence, accessing a value through table[row][col] would be preferable.
I have thought about two solutions:
First, storing it in three distinct variables colNames, rowNames and table. Accessing values would still be doable. But, for example, removing a column would then require to modify both colNames and rowNames. Incrementing some value from row and column names would require to access all three variables.
var rowNames = ["row1", "row2", "row3"];
var colNames = ["col1", "col2", "col3"];
var table = [[1, 2, 3], [4, 5, 6], [7, 8, 9]];
The other way would be to store a single variable table as an Object. It becomes easy to access values. But it is harder to correctly update it in Redux to respect immutable state.
var table = {
row1: {
col1: 1,
col2: 2,
col3: 3
},
row2: {
col1: 4,
col2: 5,
col3: 6
},
row3: {
col1: 7,
col2: 8,
col3: 9
}
};
Some other idea would be to use Immer JS with the second solution to improve code readability.
What is then the best way to store such a value in Redux ?
Solution I chose
I ended up using the second solution with a single variable. Once the modification of this table is properly made in Redux reducers, it makes accessing and displaying said table easy.
I do think your solution is quite nice and adapted to rendering in an html table. However you will have to iterate over Object.keys(table) to go through the rows.
Another solution could be to do somehing like that so you can iterate over objects directly:
var table = [
{
name: "row1",
col1: 1,
col2: 2,
col3: 3
},
{
name: "row2",
col1: 4,
col2: 5,
col3: 6
},
{
name: "row3",
col1: 7,
col2: 8,
col3: 9
}
];
This solution also has it's problems: you have to specifically take the name property out of the object while iterating for rows. You would also have to iterate over Object.keys of the inner object avoiding the treatment of the "name" column.
Hope this help as an overview of your problem :)
I have a input array of objects
[
{ col1: 'col1value1', col2: 'col2value1', col3: col1value3},
{ col1: 'col1value2', col2: 'col2value2', col3: col1value2}
]
Now I want to convert this into a object two arrays like below
columns:["col1" , "col2","col3"],
data: ["col1value1","col2value1" , "col1value3"] ,
["col1value2","col2value2" , "col1value2"]
is there any good way or faster way . as I am trying to use 2 foreach loop on all the values and creating columns and data array
Use Object.keys() method to get all the columns and use Array.prototype.map method to traverse the array and get the values using Object.values() method.
const input = [
{ col1: 'col1value1', col2: 'col2value1', col3: 'col1value3' },
{ col1: 'col1value2', col2: 'col2value2', col3: 'col1value2' },
];
const columns = Object.keys(input[0]);
const data = input.map((x) => Object.values(x));
console.log(columns);
console.log(data);
I'm new to javascript and need to create a web app where a user will click a button, and the array of data will export into a shape file. After reading this answer I know that it is possible with an ARCGIS server, but I do not have access to this.
The array in question is a stream of data similar to the following
var array = [
[17, 70, "mark", "let", "test", "test"],
[18, 50, "marj", "get", "test", "test"],
ETC...]
I've also read about shp-write but I don't know where to start. Would anyone be able to give me any examples of how to do this, or pointers where to start? Thanks.
You should just convert your data from a simple array to an array of points and an array of features, like:
let points = [
[17, 70],
[18, 50],
...
];
let features = [
{col1: "mark", col2: "let", col3: "test", col4: "test"},
{col1: "marj", col2: "get", col3: "test", col4: "test"},
...
];
And then call the write function provided by scp-write, providing your callback function to write the resulting file (check the examples for the callback function).
let scp = require('scp-write');
scp.write(features, 'POINT', points, callbackFunction);