I am new to node-mysql but I am having a speed issue that is worrysome because of how simple it is.
I am working with a fairly reasonable amount of data (2000 mysql entries) and entering it via connection.query() has become very slow when I use multiple queries. The code is this
var rows = xlsx.rows;
for (var i=0; i<rows.length; ++i) {
var escaped = '';
var values = [];
var row = rows[i];
escaped += 'INSERT INTO Main (name, value, enabled) VALUES ';
var name = row['name'];
var val = row['value'];
var enabled = rows['enabled'];
escaped += '(?,?,?);';
values.push(name);
values.push(val);
values.push(enabled);
connection.query(escaped, values);
}
It takes upwards of one minute to input all the rows. The same issue has arisen when I use a multiple statements inside one query. The only time I am able to enter all the rows quickly, and almost instantly, is if I use one string and one entry, a.k.a.
INSERT INTO Main (name, value, enabled) VALUES (?,?,?), (?,?,?), (?,?,?)...
Am I just using the queries in an inefficient manner? Or is there an actual issue with the speed of the queries here?
As mentioned in the comments, this is just a slow way of inserting mysql data, it is much easier to use
connection.query('INSERT INTO Table (col1, col2) VALUES ?', [insert], callback);
where insert is a variable containing the multiple entry values
Related
I want to import rows from one google sheet to the other, however source sheet imports a number of empty rows. Now I use a filter function to get rid of these rows but they will not disappear, can anyone tell me why?
var a = SpreadsheetApp.openByUrl("url").getSheetByName("Admin Use Only").getRange(4,1,6,21).getValues();
var b = SpreadsheetApp.getActive().getSheetByName('Credit_Detail');
b.getRange(b.getLastRow() +1, 1, a.length,21).setValues(a);
//filter function below:
var otarget=b.getRange(2,1,b.getLastRow()-1, 26).getValues();
var data=otarget.filter(function(r){
return !r.every(function(cell){
return cell === "";});
});
Logger.log(data);
b.getRange("A2:Z").clearContent();
b.getRange(3,1,data.length,data[0].length).setValues(data);
here's how I would do it. First, create an variable to store the array of the source. then run a for loop scanning the first column for empties. something like: for (var i = 0, i < data.length; i++) { if (data[i][0] != '') { XXXX } }
XXXX means that you can either put a code to create a new set of array which can be passed to the target sheet at once or use append row to transfer non blank rows to the target sheet one by one.
Note: Creating a new array to store non-empty rows would speedup the execution time if you are dealing with large data, thousands of rows.
I should start with letting you know that I'm an extreme novice in JS. My background is almost solely in SQL/VBA. So, any tips you could provide would be greatly appreciated, whether it's in coding or in Stack Overflow etiquette.
Problem Background:
I've got a Script that I use for sending outbound e-mails from Google Form responses, hosted within a Google Sheet and set to OnFormResponse(). Typically, I'm only asked to send back specific bits of information from the form responses within a HTML template e-mail. However, the business case I have now is that I need to look up the values from another sheet, where an adjacent column's value matches a form response value. With the value that's matched, I need to set the value of a specific column/row (F:F) within the Form Response sheet with it.
Example:
Here's a simplified version of what the Form Responses sheet looks like, along with the formula that I would typically use:
Here's what the other tab, 'Unique Databases!', looks like:
So, my understanding of JavaScript arrays is that on the Form Responses Sheet, I would load all columns (A:E in this example) into a variable, and get the values. Then, get columns A:B of 'Unique Databases!', which loads those values into another array. If that is accurate, how do you compare the index of 1 array against the index of another, and return an adjacent match?
Thanks!
You can try this function:
function dbmanager(dbname) {
var ss=SpreadsheetApp.getActive();
var sh=ss.getSheetByName('Unique Databases!');
var rg=sh.getDataRange();
var vA=rg.getValues();
var r='';
for(var i=0;i<vA.length;i++){
if(dbname==vA[i][0]){
r=vA[i][1];
break;
}
}
return r;
}
I'm not sure you need to use a apps script to make a comparison like this. Perhaps a much faster way would be to use a query in the sheet. Something like this maybe:
=QUERY(Sheet1!A1:B3,"SELECT B WHERE A ='"&E2&"'",0)
The first part of the query is looking up the unique databases data. The second part selects column B in the unique databases data where column A is equal to the data base name in the form responses data. Note this query goes in column F of the responses data.
Another alternative using code might be something like this. with the code below running.
function getFormData(){
var ss = SpreadsheetApp.getActiveSpreadsheet();
var sheet = ss.getSheetByName('Sheet2');
var getRange = sheet.getRange('E2:E');
var data = getRange.getValues();
var lookup = getLookupData();
data.forEach(function(item,index){
sheet.getRange(index + 2 , 6).setValue(lookup[item])
})
}
function getLookupData() {
var obj = {};
var ss = SpreadsheetApp.getActiveSpreadsheet();
var sheet = ss.getSheetByName('Sheet1');
var range = sheet.getRange('A2:B6');
var data = range.getValues();
data.forEach(function(item){
obj[item[0]] = item[1];
})
Logger.log(obj)
return obj;
}
I feel like I'm going about this in all the wrong way. I'm trying to automate some of my workload here. I'm cleaning up spreadsheets with 4 columns (A-E), 2000+ rows. Column B contains website URLs, column D contains the URL's business name, generated from another source.
Sometimes the tool doesn't grab the name correctly or the name is missing, so it populates the missing entries in column D with "------" (6 hyphens). I've been trying to make a function that takes an input cell, checks if the contents of the cell are "------", and if it is the function changes the contents of the input cell to the contents of the cell two columns to the left (which is generally a website url). This is what I've come up with.
function replaceMissing(input) {
var sheet = SpreadsheetApp.getActiveSheet();
//sets active range to the input cell
var cell = sheet.getRange('"' + input + '"');
//gets cell to fill input cell
var urlCell = sheet.getRange(cell.getRow(), cell.getColumn() - 2);
//gets contents of input cell as String
var data = cell.getValue();
//gets contents of urlCell as String
var data2 = cell.getValue();
//checks if input cell should be replaced
if (data === "------") {
//set current cell's value to the value of the cell 2 columns to the left
cell.setValue(data2);
}
}
When I attempt to use my function in my sheet, the cell is returning the error
Error Range not found (line 4).
I'm assuming, based on similar questions people have asked, that this is how you use the A1 notation of the function with an argument. However, that doesn't seem to be the case, so I'm stuck. I also don't think my solution is very good period.
1) It's somewhat ambiguous in GAS documentation, but custom functions have quite a few limitations. They are better suited for scenarios where you need to perform a simple calculation and return a string or a number type value to the cell. While custom functions can call some GAS services, this practice is strongly discouraged by Google.
If you check the docs for the list of supported services, you'll notice that they support only some 'get' methods for Spreadsheet service, but not 'set' methods https://developers.google.com/apps-script/guides/sheets/functions
That means you can't call cell.setValue() in the context of a custom function. It makes sense if you think about it - your spreadsheet can contain 1000s of rows, each with its own custom function making multiple calls to the server. In JavaScript, every function call creates its own execution context, so things could get ugly very quickly.
2) For better performance, use batch operations and don't alternate between read / write actions. Instead, read all the data you need for processing into variables and leave the spreadsheet alone. After processing your data, perform a single write action to update values in the target range. There's no need to go cell by cell when you can get the entire range using GAS.
Google Apps Script - best practices
https://developers.google.com/apps-script/guides/support/best-practices
Below is a quick code example that runs onOpen and onEdit. If you need more flexibility in terms of when to run the script, look into dynamically-created triggers https://developers.google.com/apps-script/reference/script/script-app
Because your spreadsheets have lots of rows, you may hit the execution quota anyway - by using triggers you can work around the limitation.
Finally, if a cell containing '----' is a rare occurrence, it might be better to create another array variable with new values and row numbers to update than updating the entire range.
Personally, I think the single range update action would still be quicker, but you could try both approaches and see which one works best.
function onOpen(){
test();
}
function onEdit() {
test();
}
function test() {
var ss = SpreadsheetApp.getActiveSpreadsheet();
var sheet = ss.getSheetByName('yourSheetName');
//range to replace values in
var range = sheet.getRange(2, 4, sheet.getLastRow() - 1, 1);
//range to get new values from
var lookupRange = range.offset(0, -2);
//2d array of values from the target range
var values = range.getValues();
//2d array of values from the source range
var lookupValues = lookupRange.getValues();
//looping through the values array and checking if array element meets our condition
for (var i=0; i < values.length; i++) {
values[i][0] = (values[i][0] == '------') ? lookupValues[i][0] : values[i][0];
}
// one method call to update the range
range.setValues(values);
}
What I'm looking for is an individual column searching function (exactly as this datatables spreadsheet example) for the Handsontable spreadsheet plugin.
What's already existing and has been developed by the Handsontable team is :
Multiple filtering Excel-like (but included in the PRO version) - Cons for my case are that it's not free, and it doesn't quite fit well what I'm looking for.
Highlighting the cell(s) or row(s) based on an user input - The Con is that I need to only display the relevant row(s)
Is there such thing as displaying only the relevant row(s) based on multiple inputs from an user with Handsontable ?
Based on the solution of this blog, I managed to code a solution.
See this JS fiddle that answers all my requirements.
The main function I was looking for is this one :
// The function push every row satisfying all the input values into an array that is loaded
function filter() {
var row, r_len, col, c_len;
var data = myData; // Keeping the integrity of the original data
var array = [];
var match = true;
for (row = 0, r_len = data.length; row < r_len; row++) {
for(col = 0, c_len = searchFields.length; col < c_len; col++) {
if(('' + data[row][col]).toLowerCase().indexOf(searchFields[col]) > -1);
else match=false;
}
if(match) array.push(data[row]);
match = true;
}
hot.loadData(array);
}
What I did is keeping synchronized a table of Strings with the input fields (searchFields), compare the data of each row between inputs and their corresponding column, and push into an array the relevant row(s) to finally display the resulting array. This function is called for any change in the input fields which result in a live table filtering.
Note that I tried my solution for ~10k rows and their isn't any performance issue with Chrome, Firefox and IE.
Also note that I managed to find a solution to keep synchronized the current displayed table with the original data when editing the values, but this is IMO out of the scope of this question. Please let me know in the comment if you're interested about this.
Am working on a windows store javascript application. The application uses data from azure mobile services.
Consider the below code:
var itemTable = mobileService.getTable('item');
//item is the table name stored in the azure database
The code fetches the entire table item and saves it to a variable itemTable.
What code will return the no of rows present in itemTable??
What you're looking for is the includeTotalCount method on the table/query object (unfortunately it's missing from the documentation, I'll file a bug to the product team to have it fixed).
When you call read on the query object, it will return by default 50 (IIRC, the number may be different) elements from it, to prevent a naïve call from returning all elements in a very large table (thus either incurring the outbound bandwidth cost for reserved services, or hitting the quota for free ones). So getting all the elements in the table, and getting the length of the results may not be accurate.
If all you want is the number of elements in the table, you can use the code below: returning zero elements, and the total count.
var table = client.getTable('tableName');
table.take(0).includeTotalCount().read().then(function (results) {
var count = results.totalCount;
new Windows.UI.Popups.MessageDialog('Total count: ' + count).showAsync();
});
If you want to query some elements, and also include the total count (i.e., for paging), just add the appropriate take() and skip() calls, and also the includeTotalCount as well.
If anybody comes here and interested in how to get the totalCount only on C# (like me), then this is how you do it:
var table = MobileService.GetTable<T> ();
var query = table.Take(0).IncludeTotalCount();
IList<T> results = await query.ToListAsync ();
long count = ((ITotalCountProvider)results).TotalCount;
Credit goes to this blog post here
You need to execute read() on the table query and then get the length of the results.
var items, numItems;
itemTable.read().then(function(results) { items = results; numItems = items.length; });
If you are only showing a record count and not the entire results - you should just select the ID column to reduce the amount of data transmitted. I don't see a count() method available yet in the JS Query API to fill this need.
var itemTable = mobileService.getTable('item').select('itemID');