I'm making use of google spread sheets and Apps Script to collect some financial data, there is one function that takes some time because it processes a lot of information.
because of this, google shows me that exceeded the maximun execution time, but even having a for loop that writes on the file each loop it wont do it until the function finishes.
can someone give me an idea of how to write the spreadsheet while its executing?
thanks.
Leaving here an example script.
function myFunction() {
var sheet = SpreadsheetApp.getActiveSpreadsheet();
var rangeSheet = activeSheet.getRange(1, 1, activeSheet.getLastRow() - 1, 1).getValues();
// after cleared out i have an array of values so i do:
rangeSheet.forEach(async (el, idx) => {
let result = await anotherFunction(el) // <--- this is the function taking around 2 minutes to complete
sheet.getRange(`$B${idx + 1}`).setValue(`${result}
})
}
This will give you some boost in performance:
function myFunction() {
var sheet = SpreadsheetApp.getActiveSpreadsheet();
var rangeSheet = activeSheet.getRange(1, 1, activeSheet.getLastRow() - 1, 1).getValues();
var data = [];
rangeSheet.forEach(async (el, idx) => {
let result = await anotherFunction(el) // <--- this is the function taking around 2 minutes to complete
data.push([result]);
})
sheet.getRange(1,2,data.length,data[0].length).setValues(data);
}
Read Best Practices for more info.
However, if anotherFunction takes too much time, then I am afraid you need to redesign your logic. I would advice you to use UrlfetchApp.fetchAll() if you want to fetch multiple URLs.
References:
google apps script with UrlfetchApp.fetchAll() or with async/ await for multiple http requests?
Related
I wrote a web app to make reservations for a concert. People can select from certain dates and can choose with how many people they come by selecting an amount of seats.
I use a spreadsheet to gather all the reservations. I wrote down the last 2 functions of the process. If I press the button to order the seats, 2 functions activate: validate() on the frontend and getLastCheck() on the backend. This is a last check whether the asked amounts of seats are still available. If so, the data is written to the spreadsheet.
I tested the script a few times with 4 other colleagues and we simultaneously tried to book 3 seats on the same date. Since there were only 10 seats left, 2 of us should get the message that the "seats are not booked". Sometimes it worked fine, other times only 1 of us received the message "seats are not booked" and the other 4 people (1 too many!) could book their seats. In that case we exceeded the maximum capacity.
I presume that the belated updating from the spreadsheet (which results in a wrong evaluation) is caused by the time of traffic from and to the spreadsheet. Is there a way to solve this wrong evaluation when simultaneously submitting the data?
Frontend function:
function validate() {
var info = {};
info.firstName = document.getElementById("first-name").value;
info.lastName = document.getElementById("last-name").value;
info.mail = document.getElementById("email").value.trim();
info.date = document.getElementById("select-date").value;
info.seats = document.getElementById("select-seats").value;
google.script.run.withSuccessHandler(function(result){
console.log(result);
}).getLastCheck(info);
}
backend function:
function getLastCheck(info) {
var listAll = wsRsrv.getRange(2, 5, lastRowRsrv, 2).getValues();
var dates = listAll.map(function(element){ return element[0]; });
var seats = listAll.map(function(element){ return element[1]; });
var sum = 0;
var diff = maxPerDate - info.seats;
for (var i = 0; i<listAll.length; i++) {
if (info.date == dates[i]) { sum += Number(seats[i]); }
}
if (sum <= diff) {
wsRsrv.appendRow([new Date(), info.lastName, info.firstName, info.mail, info.date, info.seats]);
return "seats are booked";
} else {
return "seats are not booked";
}
}
I tested it out and it seems to work right.
function lockedFunction() {
var active_spreadsheet = SpreadsheetApp.getActiveSpreadsheet();
// BEGIN - start lock here
const lock = LockService.getScriptLock();
try {
lock.waitLock(5000); // wait 5 seconds for others' use of the code section and lock to stop and then proceed
} catch (e) {
console.log('Could not obtain lock after 5 seconds.');
return "Error: Server busy try again later... Sorry :("
}
// note: if return is run in the catch block above the following will not run as the function will be exited
const active_sheet = active_spreadsheet.getActiveSheet()
const new_start_row = active_sheet.getLastRow() + 1;
// Run the function
const data = {
firstName: 'David',
lastName: 'Salomon',
mail: 'mail#example.com',
date: new Date(),
seats: 10
}
getLastCheck(data)
SpreadsheetApp.flush(); // applies all pending spreadsheet changes
lock.releaseLock();
// END - end lock here
return;
}
You can check this other thread to have more ideas on how to implement it but as mentioned in the comments, you can check the Google documentation
So i'm attempting to write a google parser.
The idea of my tool is it takes search queries and searches google for them and returns URLs. It is working good so far but now im trying to set a page configuration and im having troubles, my code is:
const needle = require("needle") //for making get request
const sp = require("serp-parser") //for parsing data from the request
const queryup = "watch movies online free" //my search data
const query = encodeURI(queryup) //my search data so google can read it
var page = 0; //initializing the page counter
let pages = 5; //setting amount of pages to loop through
for (var i = 0; i < pages; i++) { //my loop
needle.get(`https://www.google.com/search?q=${query}&start=${page}`, function(err, response){ //MY MAIN PROBLEM <<<--- The issue is its adding to the page value but its not effecting it here, why?
page += 10 //adding to page value (every 10 page value is 1 extra page)
console.log(`----- Page number: `+ page / 10+" -----") //logging the number of the page to confirm that it is indeed increasing the page value
let results = response.body; //defining the body of my request
parser = new sp.GoogleNojsSERP(results); //initializing the parser
let parsed = parser.serp //parsing the body
let objarray = parsed.organic; //parsed body (returns as an array of json objects)
for (var i = 0; i < objarray.length; i++) { //loop the logging of each url
let url = objarray[i].url //defining url
console.log(url) //logging each url
}
});
}
without a billion comments:
const needle = require("needle")
const sp = require("serp-parser")
const queryup = "watch movies online free"
const query = encodeURI(queryup)
var page = 0;
let pages = 5;
for (var i = 0; i < pages; i++) {
needle.get(`https://www.google.com/search?q=${query}&start=${page}`, function(err, response){
//^^^^^ MY MAIN PROBLEM <<<--- The issue is its adding to the page value but its not effecting it here, why?
page += 10
console.log(`----- Page number: `+ page / 10+" -----")
let results = response.body;
parser = new sp.GoogleNojsSERP(results);
let parsed = parser.serp
let objarray = parsed.organic;
for (var i = 0; i < objarray.length; i++) {
let url = objarray[i].url
console.log(url)
}
});
}
This seems to be an issue with async.
I'm not familiar with needle, but I know that external queries are basically never synchronous.
The problem you're experiencing is basically, the actual web query is happening after the loop first runs and has already incremented page to 50. Then, 5 queries are constructed, each one with page=50, because async is complicated and difficult to manage.
Under the hood, the engine is essentially doing literally everything else it can possibly do first, and THEN doing your web queries.
A trip through the needle npm docs tells me that you can use alternative syntax to get needle to return a promise instead, which can then be wrapped in an asynchronous function and managed through await to force synchronous behavior, which is what you're after:
const needle = require('needle');
const sp = require('serp-parser');
const queryup = 'watch movies online free';
const query = encodeURI(queryup);
let page = 0;
const pages = 5;
const googler = async function () {
for (let i = 0; i < pages; i++) {
try {
const response = await needle('get', `https://www.google.com/search?q=${query}&start=${page}`);// MY MAIN PROBLEM <<<--- The issue is its adding to the page value but its not effecting it here, why?
console.log('----- Page number: ' + page / 10 + ' -----');
const results = await response.body;
const parser = new sp.GoogleNojsSERP(results);
const parsed = parser.serp;
const objarray = parsed.organic;
for (let i = 0; i < objarray.length; i++) {
const url = objarray[i].url;
console.log(url);
}
} catch (err) {
console.error(err);
}
page += 10;
}
};
googler();
The key differences:
Per the needle docs, rather than the request method being a method on the needle object, it's instead the first argument you pass directly to invoking needle itself as a function.
When you manage promises with await, a rejected promise throws an error that should be caught with a traditional try/catch block; I've done that here. Though, if needle is anything like node-fetch it probably basically never throws errors, but it's good practice.
One of my extensions automatically changed your var declarations to let and not-reassigned let declarations to const; you're welcome to change them back.
This is a classic asynchronous problem. Add another console.log() immediately before the needle.get() call (and after the for statement) and you will see what is going wrong: All of the needle.get() calls execute before any of the callbacks where you do the page += 10. Then, after the for loop completes, all of the callbacks are executed. But it is too late for this to have any effect on the start= parameter.
One way to fix this could be to move the body of this for loop (the needle.get() and its callback) into a separate function. Initialize your variables and call this function once. Then at the end of the callback, do your page += 10 and update any other variables you need to, and call this function again from there if there are more pages left that you want to load. If you have completed all of the pages, then don't make that call. The for loop is not needed with this technique.
Or, you could keep your current code but move the page += 10 after the callback but still inside the outer for loop. That way this variable will be incremented as you expect. I don't necessarily recommend this, as Google may get unhappy about receiving the get requests so rapidly and may start blocking your calls or throwing CAPTCHAs at you.
There may be an issue of whether this kind of scraping is allowed by Google's Terms of Service, but I will leave that question to you and your legal advisors.
Also, I would avoid using var anywhere. Use const or let instead, and prefer const over let except when you need to reassign the variable.
One tip: in most cases where you use a numeric for loop to iterate over an array, the code will be cleaner if you use a for..of loop. For example, this bit of code:
let parsed = parser.serp
let objarray = parsed.organic;
for (var i = 0; i < objarray.length; i++) {
let url = objarray[i].url
console.log(url)
}
could be more simply written as:
for (const result of parser.serp.organic) {
console.log(result.url)
}
(I know that is just a bit of debug code, but this is a good habit to get into.)
Finally, watch your indentation and be sure to indent nested blocks or functions. I took the liberty of adding some indentation for you.
I am trying to create a function in js for k6 tool scripts which would enable me to create multiple type of metrics for "Transaction Name" as input and then create another function to populate those metrics. This will help in avoiding writing similar code for different transaction names and also help keep consistent names of the metric.
// line of code to define the metrics
let Search_RT_Trend = new Trend("Search_duration");
let Search_PassRate = new Rate("Search_PassRate");
let Search_PassCount = new Counter("Search_PassCount");
let Search_FailCount = new Counter("Search_FailCount");
// line of code populating the data in metrics
Search_RT_Trend.add(res.timings.duration);
Search_PassRate.add(1);
Search_PassCount.add(1);
Search_FailCount.add(1);
Hoping to create two functions which take input for the transaction name possibly as below:
CreateMetric ("Search")
PopulateMetric ("Search")
how to achieve this?
Something like this?
function MetaMetric(name) {
this.RT_Trend = new Trend(`${name}_duration`);
this.PassRate = new Rate(`${name}_PassRate`);
this.PassCount = new Counter(`${name}_PassCount`);
this.FailCount = new Counter(`${name}_FailCount`);
}
MetaMetric.prototype.track = function (req) {
this.RT_Trend.add(req.timings.duration);
if (req.timings.duration < 200 /* or whatever */) {
this.PassRate.add(1);
this.PassCount.add(1);
} else {
this.PassRate.add(0);
this.FailCount.add(1);
}
};
let myMetaMetric = new MetaMetric("Search")
export default function () {
let resp = http.get("https://httpbin.test.loadimpact.com/");
myMetaMetric.track(resp);
sleep(3 * Math.random());
}
Some things to consider:
You don't need pass and fail Counter metrics when you have a Rate one. Rate is essentially the ratio between passing and failing, so it's basically those two counters combined :)
You might find the k6 checks and thresholds useful.
I am trying to generate a unique number when a form is submitted. I have simplified my script to the following for testing.
function onFormSubmit(event) {
// Get a script lock, because we're about to modify a shared resource.
var lock = LockService.getScriptLock();
// Wait for up to 30 seconds for other processes to finish.
lock.waitLock(30000);
var ticketNumber = Number(ScriptProperties.getProperty('lastTicketNumber')) + 1;
ScriptProperties.setProperty('lastTicketNumber', ticketNumber);
targetCell = event.range.offset(0, event.range.getNumColumns(), 1, 1);
targetCell.setValue(ticketNumber);
SpreadsheetApp.flush();
// Release the lock so that other processes can continue.
lock.releaseLock();
};
I find that if I submit two forms within a second of each other I get the same ticketnumber.
Any help would be appreciated.
Preface: You are using the deprecated ScriptProperties service, this has been replaced by the Properties Service. You probably want to change this first.
In the past I have received the same results when trying to utilize the project properties in rapid succession. It's almost as if the old value hangs around in whatever caching apps script uses for script properties for a few seconds.
I would recommend utilizing the Cache Service to suppliment for scripts that need to reflect changes immediately.
Modified Code:
function onFormSubmit(event) {
// Get a script lock, because we're about to modify a shared resource.
var lock = LockService.getScriptLock();
// Wait for up to 30 seconds for other processes to finish.
lock.waitLock(30000);
var scriptCache = CacheService.getScriptCache();
var scriptProperties = PropertiesService.getScriptProperties();
var cachedTicketNumber = scriptCache.get('lastTicketNumber');
var ticketNumber;
if(cachedTicketNumber !== null){
ticketNumber = Number(cachedTicketNumber) + 1;
} else {
//Cache has expired/does not exist, fall back to properties service
ticketNumber = Number(scriptProperties.getProperty('lastTicketNumber')) + 1;
}
//Set properties service, and cache values of the ticket number
scriptProperties.setProperty('lastTicketNumber', ticketNumber);
scriptCache.put('lastTicketNumber', ticketNumber, 21600); //21600 seconds = 6 hours, if you want it to last that long
targetCell = event.range.offset(0, event.range.getNumColumns(), 1, 1);
targetCell.setValue(ticketNumber);
SpreadsheetApp.flush();
// Release the lock so that other processes can continue.
lock.releaseLock();
};
I'm new to node.js, so before releasing my node.js app, I need to be sure it will work as it should.
Let's say I have an array variable and I intialize it on beginning of my script
myArray = [];
then I pull some data from an external API, store it inside myArray, and use setInterval() method to pull this data again each 30 minutes:
pullData();
setInterval(pullData, 30*60*1000);
pullData() function takes about 2-3 seconds to finish.
Clients will be able to get myArray using this function:
http.createServer(function(request, response){
var path = url.parse(request.url).pathname;
if(path=="/getdata"){
var string = JSON.stringify(myArray);
response.writeHead(200, {'Content-Type': 'text/plain'});
response.end(string);
}
}).listen(8001);
So what I'm asking is, can next situation happen?:
An client tries to get data from this node.js server, and in that same moment, data is being written into myArray by pullData() function, resulting in invalid data being sent to client?
I read some documentation, and what I realized is that when pullData() is running, createServer() will not respond to clients until pullData() finishes its job?
I'm really not good at understanding concurrent programming, so I need your confirmation on this, or if you have some better solution?
EDIT: here is the code of my pullData() function:
var now = new Date();
Date.prototype.addDays = function(days){
var dat = new Date(this.valueOf());
dat.setDate(dat.getDate() + days);
return dat;
}
var endDateTime = now.addDays(noOfDays);
var formattedEnd = endDateTime.toISOString();
var url = "https://api.mindbodyonline.com/0_5/ClassService.asmx?wsdl";
soap.createClient(url, function (err, client) {
if (err) {
throw err;
}
client.setEndpoint('https://api.mindbodyonline.com/0_5/ClassService.asmx');
var params = {
"Request": {
"SourceCredentials": {
"SourceName": sourceName,
"Password": password,
"SiteIDs": {
"int": [siteIDs]
}
},
"EndDateTime" : formattedEnd
}
};
client.Class_x0020_Service.Class_x0020_ServiceSoap.GetClasses(params, function (errs, result) {
if (errs) {
console.log(errs);
} else {
var classes = result.GetClassesResult.Classes.Class;
myArray = [];
for (var i = 0; i < classes.length; i++) {
var name = classes[i].ClassDescription.Name;
var staff = classes[i].Staff.Name;
var locationName = classes[i].Location.Name;
var start = classes[i].StartDateTime.toISOString();
var end = classes[i].EndDateTime.toISOString();
var klasa = new Klasa(name,staff,locationName,start,end);
myArray.push(klasa);
}
myArray.sort(function(a,b){
var c = new Date(a.start);
var d = new Date(b.start);
return c-d;
});
string = JSON.stringify(myArray);
}
})
});
No, NodeJs is not multi-threaded and everything run on a single thread, this means except non-blocking calls (ie. IO) everything else will engage CPU until it returns, and NodeJS absolutely doesn't return half-way populated array to the end user, as long as you only do one HTTP call to populate your array.
Update:
As pointed out by #RyanWilcox any asynchronous (non-blocking syscall) call may hint NodeJS interpreter to leave your function execution half way and return to it later.
In general: No.
JavaScript is single threaded. While one function is running, no other function can be.
The exception is if you have delays between functions that access the value of an array.
e.g.
var index = i;
function getNext() {
async.get(myArray[i], function () {
i++;
if (i < myArray.length) {
getNext()
}
});
}
… in which case the array could be updated between the calls to the asynchronous function.
You can mitigate that by creating a deep copy of the array when you start the first async operation.
Javascript is single threaded language so you don't have to be worried about this kind of concurrency. That means no two parts of code are executed at the same time. Unlike many other programming languages, javascript has different concurrency model based on event loop. To achieve best performance, you should use non-blocking operations handled by callback functions, promises or events. I suppose that your external API provides some asynchronous i/o functions what is well suited for node.js.
If your pullData call doesn't take too long, another solution is to cache the data.
Fetch the data only when needed (so when the client accesses /getdata). If it is fetched you can cache the data with a timestamp. If the /getdata is called again, check if the cached data is older than 30 minutes, if so fetch again.
Also parsing the array to json..
var string = JSON.stringify(myArray);
..might be done outside the /getdata call, so this does not have to be done for each client visiting /getdata. Might make it slightly quicker.