i am trying to run this script in google apps script. It lists all files recursively within a folder along with its size, name, url etc to spreadsheet. There is nothing wrong with the script but the problem is i am running it on a folder which has thousands of files and google script only allows few minutes of maximum runtime so each time just after few mins i get the error saying exceeded maximum execution time in google script.
Is there any workaround for this issue? i am fine even if i have to run this code somewhere outside of google apps script if that's the only way out but then again i have been told its not possible to execute this code outside of google script.
function start() {
var sheet = SpreadsheetApp.getActiveSheet();
sheet.clear();
sheet.appendRow(["Name", "Date", "Size", "URL", "Download", "Description", "Type", "Folder", "Folder Slug"]);
var folders = DriveApp.getFolderById('FOLDER_ID');
var folder = folders.getFolders();
if (folder.hasNext()) {
// 1. Retrieve the file list and put to an array.
// 2. Sort the array by the file size.
var list = processFolder(folder).sort((a, b) => a[2] < b[2] ? 1 : -1);
// 3. Put the array to the Spreadsheet.
sheet.getRange(2, 1, list.length, list[0].length).setValues(list);
} else {
Browser.msgBox('Folder not found!');
}
function processFolder(folder, list = []) {
while (folder.hasNext()) {
var f = folder.next();
var contents = f.getFiles();
addFilesToSheet(contents, f, list);
var subFolder = f.getFolders();
processFolder(subFolder, list);
}
return list;
}
function addFilesToSheet(files, folder, list) {
var folderName = folder.getName();
while (files.hasNext()) {
var file = files.next();
list.push([
file.getName(),
file.getDateCreated(),
Math.round(10 * file.getSize() / 1073741824) / 10, // Modified from file.getSize() / 1073741824,
file.getUrl(),
"https://docs.google.com/uc?export=download&confirm=no_antivirus&id=" + file.getId(),
file.getDescription() || "",
file.getMimeType(),
folderName
]);
}
}
}
I have managed this problem in the past using the Continous Execution Library by Patrick Martinent:
https://gist.github.com/patt0/8395003
The basic idea is to:
Set up your function so it can be terminated and resumed without issue
Set up a time trigger to re-run the function
Run it until it nears the execution timeout and exit gracefully
Allow the trigger to restart the function
Repeat until done, and remove the trigger
Related
I'm horrible at coding so I'm sure this will need some work.
My Problem: My company does work for a lot of other companies (example Drive hierarchy photo attached). Each company we work with gets their own folder that is nested under the folder 'Shirts for other companies". We put all their assets into these folders (logos, mocks, quotes).
We want to build a script that will look into the individual company folders and take any file that starts with the word Mock and automatically copy it and put it into the database folder found in the picture below.
Here is the script have so far. but I'm having trouble getting into the subfolders (ford tshirts and designs). Not only that, but if it runs everyday the script will keep duplicating the files it has duplicated in the past which I want to avoid.
Script:
Function getTheFiles() {
var dApp = DriveApp;
var folderIter = dApp.getFolderByName("Shirts for other companies");
var folder = folderIter.next();
var filesIter = folder.getFiles();
var dataBase = folder.getFoldersByName("database1").next();
var i = 1;
while(filesIter.hasNext()) {
var file = filesIter.next();
var filename = file.getName();
if(filename = "mocking") {
file.makeCopy(dataBase);
}
logger.log(filename);
i++;
}
If I understood you correctly, you want to:
Copy all files from the different subfolders inside a folder called "Shirts for other companies", that start with the word mock, to the Database folder (which is also inside the main folder).
Avoid files getting copied many times.
If that's the case, you could do the following:
Search for all files in each subfolder of the main folder that start with mock, excluding the files inside Database folder. For this, you iterate through all files in each subfolder (after checking the subfolder is not named Database folder. For each file, you check that the name start with mock, using the method substring.
To keep track of which files are copied, and so avoid making duplicates, you can use PropertiesService, which can store key-value pairs. The id of every copied file is copied to the script property, and then, the next time the script runs, it checks whether the id is in the property. If that's the case, the file doesn't get copied again. One thing to notice is that script properties can only store strings, so that arrays have to be converted to a string every time we want to store a new id (via toString, and back to an array, via split.
The function below does all these things (check inline comments for more details):
function copyFiles() {
var mainFolder = DriveApp.getFoldersByName("Shirts for other companies").next();
var folders = mainFolder.getFolders(); // Get folder iterator from main folder
var databaseFolderName = "Database folder"; // You database folder name
var databaseFolder = mainFolder.getFoldersByName(databaseFolderName).next(); // Get databse folder
var idsArray = []; // Array to store the ids of the copied files
while (folders.hasNext()) { // Iterate through each folder in the main folder
var folder = folders.next();
if (folder.getName() !== databaseFolderName) { // Check current folder is not the database folder
var files = folder.getFiles();
while (files.hasNext()) { // Iterate through each file in each subfolder
var file = files.next();
if (file.getName().substring(0, 4).toLowerCase() === "mock") { // Check that file name starts with "mock"
// Get ids of the files that were copied in previous executions:
var key = "copied-ids";
var scriptProperties = PropertiesService.getScriptProperties();
var ids = scriptProperties.getProperty(key);
if (ids) idsArray = ids.split(",");
else ids = "";
// Get current file id:
var id = file.getId();
// Check that current file id is not in properties (not copied before):
if (idsArray.indexOf(id) === -1) {
file.makeCopy(databaseFolder); // Copy file to database folder
idsArray.push(id); // Add file id to the array of copied files
}
ids = idsArray.toString();
scriptProperties.setProperty(key, ids);
}
}
}
}
}
Reference:
PropertiesService
String.prototype.substring()
Array.prototype.toString()
String.prototype.split()
I hope this is of any help.
This function would search your entire Google Drive for files starting the the letter Mock and put Name, url, id, type (folder or file) into the active spreadsheet and tab named MoclList;
function getAllMocks() {
var ss=SpreadsheetApp.getActive();
var sh1=ss.getSheetByName('MocksList');
sh1.clearContents();
sh1.appendRow(['Name','Url','Id','Type']);
getFnF();
SpreadsheetApp.getUi().alert('Process Complete')
}
var level=0;
function getFnF(folder) {
var folder= folder || DriveApp.getRootFolder();
var ss=SpreadsheetApp.getActive();
var sh=ss.getSheetByName('MocksList');
var files=folder.getFiles();
while(files.hasNext()) {
var file=files.next();
if(file.getName().toLowerCase().slice(0,4)=='mock') {
var firg=sh.getRange(sh.getLastRow() + 1,level + 1);
firg.setValue(file.getName());
firg.offset(0,1).setValue(Utilities.formatString('=HYPERLINK("%s","%s")',file.getUrl(),'FILE: ' + file.getName()));
firg.offset(0,2).setValue(file.getId());
firg.offset(0,3).setValue('File');
}
}
var subfolders=folder.getFolders()
while(subfolders.hasNext()) {
var subfolder=subfolders.next();
var forg=sh.getRange(sh.getLastRow() + 1,level + 1);
forg.setValue(subfolder.getName());
forg.offset(0,1).setValue(Utilities.formatString('=HYPERLINK("%s","%s")',subfolder.getUrl(),'FOLDER: ' + subfolder.getName()));
forg.offset(0,2).setValue(subfolder.getId());
forg.offsert(0,3).setValue('Folder');
//level++;
getFnF(subfolder);
}
//level--;
}
This problem has been successfully resolved. I am editing my post to document my experience for posterity and future reference.
The Task
I have 117 PDF files (average size ~238 KB) uploaded to Google Drive. I want to convert them all to Google Docs and keep them in a different Drive folder.
The Problem
I attempted to convert the files using Drive.Files.insert. However, under most circumstances, only 5 files could be converted this way before the function expires prematurely with this error
Limit Exceeded: DriveApp. (line #, file "Code")
where the line referenced above is when the insert function is called. After calling this function for the first time, subsequent calls typically failed immediately with no additional google doc created.
Approach
I used 3 main ways to achieve my goal. One was using the Drive.Files.insert, as mentioned above. The other two involved using Drive.Files.copy and sending a batch of HTTP requests. These last two methods were suggested by Tanaike, and I recommend reading his answer below for more information. The insert and copy functions are from Google Drive REST v2 API, while batching multiple HTTP requests is from Drive REST v3.
With Drive.Files.insert, I experienced issues dealing with execution limitations (explained in the Problem section above). One solution was to run the functions multiple times. And for that, I needed a way to keep track of which files were converted. I had two options for this: using a spreadsheet and a continuation token. Therefore, I had 4 different methods to test: the two mentioned in this paragraph, batching HTTP requests, and calling Drive.Files.copy.
Because team drives behave differently from regular drives, I felt it necessary to try each of those methods twice, one in which the folder containing the PDFs is a regular non-Team Drive folder and one in which that folder is under a Team Drive. In total, this means I had 8 different methods to test.
These are the exact functions I used. Each of these was used twice, with the only variations being the ID of the source and destination folders (for reasons stated above):
Method A: Using Drive.Files.insert and a spreadsheet
function toDocs() {
var sheet = SpreadsheetApp.openById(/* spreadsheet id*/).getSheets()[0];
var range = sheet.getRange("A2:E118");
var table = range.getValues();
var len = table.length;
var resources = {
title: null,
mimeType: MimeType.GOOGLE_DOCS,
parents: [{id: /* destination folder id */}]
};
var count = 0;
var files = DriveApp.getFolderById(/* source folder id */).getFiles();
while (files.hasNext()) {
var blob = files.next().getBlob();
var blobName = blob.getName();
for (var i=0; i<len; i++) {
if (table[i][0] === blobName.slice(5, 18)) {
if (table[i][4])
break;
resources.title = blobName;
Drive.Files.insert(resources, blob); // Limit Exceeded: DriveApp. (line 51, file "Code")
table[i][4] = "yes";
}
}
if (++count === 10) {
range.setValues(table);
Logger.log("time's up");
}
}
}
Method B: Using Drive.Files.insert and a continuation token
function toDocs() {
var folder = DriveApp.getFolderById(/* source folder id */);
var sprop = PropertiesService.getScriptProperties();
var contToken = sprop.getProperty("contToken");
var files = contToken ? DriveApp.continueFileIterator(contToken) : folder.getFiles();
var options = {
ocr: true
};
var resource = {
title: null,
mimeType: null,
parents: [{id: /* destination folder id */}]
};
while (files.hasNext()) {
var blob = files.next().getBlob();
resource.title = blob.getName();
resource.mimeType = blob.getContentType();
Drive.Files.insert(resource, blob, options); // Limit Exceeded: DriveApp. (line 113, file "Code")
sprop.setProperty("contToken", files.getContinuationToken());
}
}
Method C: Using Drive.Files.copy
Credit for this function goes to Tanaike -- see his answer below for more details.
function toDocs() {
var sourceFolderId = /* source folder id */;
var destinationFolderId = /* destination folder id */;
var files = DriveApp.getFolderById(sourceFolderId).getFiles();
while (files.hasNext()) {
var res = Drive.Files.copy({parents: [{id: destinationFolderId}]}, files.next().getId(), {convert: true, ocr: true});
Logger.log(res)
}
}
Method D: Sending batches of HTTP requests
Credit for this function goes to Tanaike -- see his answer below for more details.
function toDocs() {
var sourceFolderId = /* source folder id */;
var destinationFolderId = /* destination folder id */;
var files = DriveApp.getFolderById(sourceFolderId).getFiles();
var rBody = [];
while (files.hasNext()) {
rBody.push({
method: "POST",
endpoint: "https://www.googleapis.com/drive/v3/files/" + files.next().getId() + "/copy",
requestBody: {
mimeType: "application/vnd.google-apps.document",
parents: [destinationFolderId]
}
});
}
var cycle = 20; // Number of API calls at 1 batch request.
for (var i = 0; i < Math.ceil(rBody.length / cycle); i++) {
var offset = i * cycle;
var body = rBody.slice(offset, offset + cycle);
var boundary = "xxxxxxxxxx";
var contentId = 0;
var data = "--" + boundary + "\r\n";
body.forEach(function(e){
data += "Content-Type: application/http\r\n";
data += "Content-ID: " + ++contentId + "\r\n\r\n";
data += e.method + " " + e.endpoint + "\r\n";
data += e.requestBody ? "Content-Type: application/json; charset=utf-8\r\n\r\n" : "\r\n";
data += e.requestBody ? JSON.stringify(e.requestBody) + "\r\n" : "";
data += "--" + boundary + "\r\n";
});
var options = {
method: "post",
contentType: "multipart/mixed; boundary=" + boundary,
payload: Utilities.newBlob(data).getBytes(),
headers: {'Authorization': 'Bearer ' + ScriptApp.getOAuthToken()},
muteHttpExceptions: true,
};
var res = UrlFetchApp.fetch("https://www.googleapis.com/batch", options).getContentText();
// Logger.log(res); // If you use this, please remove the comment.
}
}
What Worked and What Didn't
None of the functions using Drive.Files.insert worked. Every
function using insert for conversion failed with this error
Limit Exceeded: DriveApp. (line #, file "Code")
(line number replaced with generic symbol). No further details or
description of the error could be found. A notable variation was one
in which I used a spreadsheet and the PDFs were in a team drive
folder; while all other methods failed instantly without converting a
single file, this one converted 5 before failing. However, when
considering why this variation did better than the others, I think it
was more of a fluke than any reason related to the use of particular
resources (spreadsheet, team drive, etc.)
Using Drive.Files.copy and batch HTTP requests worked only
when the source folder was a personal (non-Team Drive) folder.
Attempting to use the copy function while reading from a Team Drive
folder fails with this error:
File not found: 1RAGxe9a_-euRpWm3ePrbaGaX5brpmGXu (line #, file "Code")
(line number replaced with generic symbol). The line being referenced
is
var res = Drive.Files.copy({parents: [{id: destinationFolderId}]}, files.next().getId(), {convert: true, ocr: true});
Using batch HTTP requests while reading from a Team Drive folder
does nothing -- no doc files are created and no errors are thrown.
Function silently terminates without having accomplished anything.
Conclusion
If you wish to convert a large number of PDFs to google docs or text files, then use Drive.Files.copy or send batches of HTTP requests and make sure that the PDFs are stored in a personal drive rather than a Team Drive.
Special thanks to #tehhowch for taking such an avid interest in my question and for repeatedly coming back to provide feedback, and to #Tanaike for providing code along with explanations that successfully solved my problem (with a caveat, read above for details).
You want to convert from PDF files in the folder to Google Documents. PDF files are in a folder of team drive. You want to import converted them to a folder of your Google Drive. If my understanding is correct, how about this method?
For the conversion from PDF to Google Document, it can convert using not only Drive.Files.insert(), but also Drive.Files.copy(). The advantage of use of Drive.Files.copy() is
Although Drive.Files.insert() has the size limitation of 5 MB, Drive.Files.copy() can use over the size of 5 MB.
In my envoronment, the process speed was faster than Drive.Files.insert().
For this method, I would like to propose the following 2 patterns.
Pattern 1 : Using Drive API v2
In this case, Drive API v2 of Advanced Google Services is used for converting files.
function myFunction() {
var sourceFolderId = "/* source folder id */";
var destinationFolderId = "/* dest folder id */";
var files = DriveApp.getFolderById(sourceFolderId).getFiles();
while (files.hasNext()) {
var res = Drive.Files.copy({parents: [{id: destinationFolderId}]}, files.next().getId(), {convert: true, ocr: true});
// Logger.log(res) // If you use this, please remove the comment.
}
}
Pattern 2 : Using Drive API v3
In this case, Drive API v3 is used for converting files. And here, I used the batch requests for this situation. Because the batch requests can use 100 API calls by one API call. By this, the issue of API quota can be removed.
function myFunction() {
var sourceFolderId = "/* source folder id */";
var destinationFolderId = "/* dest folder id */";
var files = DriveApp.getFolderById(sourceFolderId).getFiles();
var rBody = [];
while (files.hasNext()) {
rBody.push({
method: "POST",
endpoint: "https://www.googleapis.com/drive/v3/files/" + files.next().getId() + "/copy",
requestBody: {
mimeType: "application/vnd.google-apps.document",
parents: [destinationFolderId]
}
});
}
var cycle = 100; // Number of API calls at 1 batch request.
for (var i = 0; i < Math.ceil(rBody.length / cycle); i++) {
var offset = i * cycle;
var body = rBody.slice(offset, offset + cycle);
var boundary = "xxxxxxxxxx";
var contentId = 0;
var data = "--" + boundary + "\r\n";
body.forEach(function(e){
data += "Content-Type: application/http\r\n";
data += "Content-ID: " + ++contentId + "\r\n\r\n";
data += e.method + " " + e.endpoint + "\r\n";
data += e.requestBody ? "Content-Type: application/json; charset=utf-8\r\n\r\n" : "\r\n";
data += e.requestBody ? JSON.stringify(e.requestBody) + "\r\n" : "";
data += "--" + boundary + "\r\n";
});
var options = {
method: "post",
contentType: "multipart/mixed; boundary=" + boundary,
payload: Utilities.newBlob(data).getBytes(),
headers: {'Authorization': 'Bearer ' + ScriptApp.getOAuthToken()},
muteHttpExceptions: true,
};
var res = UrlFetchApp.fetch("https://www.googleapis.com/batch", options).getContentText();
// Logger.log(res); // If you use this, please remove the comment.
}
}
Note :
If the number of API calls at 1 batch request is large (the current value is 100), please modify var cycle = 100.
If Drive API v3 cannot be used for team drive, please tell me. I can convert it for Drive API v2.
If the team drive is the reason of issue for your situation, can you try this after it copied PDF files to your Google Drive?
Reference :
Batching Requests
If these are not useful for you, I'm sorry.
You can first of all fetch and store id of all files in a google sheet. Then you can proceed with processing each file normally by using it's id. Then after you have processed them mark that file as processed. And before processing a file check if that file is already processed.
If there are several files then you can also store the row number till where you have processed, next time continue after that.
Then at last create a trigger to execute your function every 10 minutes or so.
By this you can overcome execution time limit for single execution. API request quota and all will not be by-passed by this method.
I'm currently writing a script that copies a folder, and all of its sub-folders and files of a very large directory from one location in Google Drive to another. This is being used for archiving purposes for the company which I am employed by.
My issue is that the size and quantity of the file that I am trying to archive is WAY too large to handle in the 5 minute execution time given by Google. I am trying to keep this script as a standalone web app, however I am happy to extend on it as needed.
My second issue is that I need the script to run over and over again until the folders have been copied but once it is finished I need it to stop. I was originally going to use triggers, however time-based triggers are inappropriate for the task I am trying to fulfil.
In short, I need my script to run until the task is completed, automatically restarting itself, and avoiding execution time errors.
The full code is included below.
//Global Variables
var classroomFolderName = "Classroom";
var archiveFolderName = "Archive";
function doGet(){
var classroomFolder = DriveApp.getFoldersByName(classroomFolderName).next();
var archiveFolder = DriveApp.getFoldersByName(archiveFolderName).next();
start(archiveFolder);
}
function getFolder(folderName){
var foldercount = 0;
//Selects all folders named exactally as parameter is given
var folder = DriveApp.getFoldersByName(folderName);
while (folder.hasNext()) {
var folders = folder.next();
foldercount++;
}
//Throws errors if number of folders != 1
if (foldercount < 1){
throw 1;
}
else if (foldercount > 1){
throw 2;
}
else{
return folder;
}
}
function start(archiveFolder) {
var sourceFolder = classroomFolderName;
var targetFolder = "New Archive";
var source = DriveApp.getFoldersByName(sourceFolder);
var target = archiveFolder.createFolder(targetFolder);
if (source.hasNext()) {
copyFolder(source.next(), target);
}
}
function copyFolder(source, target) {
var folders = source.getFolders();
var files = source.getFiles();
while(files.hasNext()) {
var file = files.next();
file.makeCopy(file.getName(), target);
}
while(folders.hasNext()) {
var subFolder = folders.next();
var folderName = subFolder.getName();
var targetFolder = target.createFolder(folderName);
copyFolder(subFolder, targetFolder);
}
}
This is something I came across- you may find some value in it-
https://ctrlq.org/code/20016-maximum-execution-time-limit
I have a javascript which uses the Google Drive API. Sometime it works flawlessly and other times it works only in part:
Basically it retrieves a list of files from google drive, downloads the files in a browser, put the files in another folder in google drive and finally erases the file from its original folder.
here is the portion of the script which carrys out the above process.
function makeRequest() {
console.log("make request");
var request = gapi.client.drive.files.list({
'q' : "'0BxTSfcTBxwlXflNjeVRZeWFQbUIzcDJMeElER1pDVEZla0NNVjhpWUpGTzY1ZDVUTS0yTFE' in Parents"}); //get only the files in a specified folder
request.execute(function(resp) {
var x = []; //array for revised list of file to only include those which have a suffix #FHM#
for (i = 0; i < resp.items.length; i++) {
if (resp.items[i].title.substring(0, 5) == "#FHM#") {
x.push([resp.items[i].title, resp.items[i].webContentLink, resp.items[i].id]);
}
}
if (x.length == 0) {
document.getElementById("downloadButton").value = "There are no files to download";
}
var originalFolder = "0BxTSfcTBxwlXflNjeVRZeWFQbUIzcDJMeElER1pDVEZla0NNVjhpWUpGTzY1ZDVUTS0yTFE" //original google drive folder ID
var processedFolder = "0BxTSfcTBxwlXfkVlYVRCdnBaMFRubWM4eUt5V0FYVEdIcHdjWDd6SFhYb3pPSjBZeFZ2T3M" // google drive folder for processed files
//loop through all files and trigger a download event for each for (i = 0; i < x.length; i++) {
console.log("download loop " + i);
var dlUrl = x[i][1];
var fileIdentity = x[i][2];
var fileTitle = x[i][0];//only used in the console log
downloadUrl(dlUrl);
}
//loop through all of final list of files and trigger the function moveFile (i.e link file to a new folder location)
for (i = 0; i < x.length; i++) {
console.log("moveFile loop " + i);
var dlUrl = x[i][1];
fileIdentity = x[i][2];
fileTitle = x[i][0];//only used in the console log
setTimeout(moveFile(processedFolder, originalFolder, fileIdentity, fileTitle),1000);
}
//loop through all of final list of files and trigger the function moveFile2 (i.e the one that deletes the original parent folder)
for (i = 0; i < x.length; i++) {
console.log("moveFile2 loop " + i);
var dlUrl = x[i][1];
fileIdentity = x[i][2];
fileTitle = x[i][0];//only used in the console log
setTimeout(moveFile2(processedFolder, originalFolder, fileIdentity, fileTitle),1000);
}
});
}
function downloadUrl(url) {
var iframe = document.createElement("iframe");
iframe.src = url;
iframe.style.display = "none";
document.body.appendChild(iframe);
console.log("download triggered");
}
function moveFile(processedFolder, originalFolder, fileIdentity, fileTitle) {
//move the file to the new processed folder
var body = {
'id': processedFolder
};
var request = gapi.client.drive.parents.insert({
'fileId': fileIdentity,
'resource': body
});
request.execute(function(resp) {});
console.log("file should be in new location : " + fileTitle);
}
function moveFile2(processedFolder, originalFolder, fileIdentity, fileTitle) {
//this bit removes the file from the the original folder
var request = gapi.client.drive.parents.delete({
'parentId': originalFolder,
'fileId': fileIdentity
});
request.execute(function(resp) {});
console.log("file should be removed from old location : " + fileTitle);
}
OK, so when I run this with 1 file to process the console.log code sections lists the events
as:
Download loop 0
download triggered
Movefile loop 0
file should be in new location : filename is XXXXX
movefile2 loop 0
file should be removed from original location
so I think that tells me the the order that the events have fired. yet sometimes I don't get all the events completing as expected. I might have some downloads missed, or some will move or some will still be in the original location.
If I run any one of the 3 for loops on its own it works flawlessly every time but as soon as I do all 3 it starts to fall apart and get all sorts of console error messages about being forbidden, about get request denied, deleted post request - all sorts of stuff
So.....
is this to do with the way Javascript handles events?
do I need to somehow delay the firing of events until the previous for loop has finished? (I've tried a setTimeout for the 2nd and 3rd For loop but doesn't seems to help)
or I am missing something fundamental to how javascript works
it's like if I try and add a file to a folder before the download has physically arrived on my hard drive then it falls apart, similary if I remove a parent folder before it has finished it also falls apart. as far as i understand Google Drive is not physically moving the file it is simply adding a label to the file. The actual physical location in googles cloud remains unchanged throughout.
help!!!!!
Too much code for me to read on my mobile, but there are a couple of obvious issues.
1/ The use of settimeout is almost always wrong since it introduces timing errors.
2/ you have an execute call with a null callback function, followed by console.log("done "). The api call is only done within the callback, not the next line of JavaScript.
So I'd say that your guess that you've misunderstood the event v handling is correct.
The documentation of the sitemap node.js module does not explain what cacheTime is. Why is it needed to generate a sitemap? What is its purpose?
The cacheTime is how long the sitemap.js module will wait before regenerating the sitemap.xml file from the list of urls given to it.
ie. on the first request, a sitemap.xml file is generated and placed in the cache. Subsequent requests read the sitemap from the cache, until it expires and is regenerated.
I agree it could be clearer, but the source code makes it pretty clear.
According to the source code at sitemap.js, line 136:
// sitemap cache
this.cacheEnable = false;
this.cache = '';
if (cacheTime > 0) {
this.cacheEnable = true;
this.cacheCleanerId = setInterval(function (self) {
self.clearCache();
}, cacheTime, this);
}
and line 187:
Sitemap.prototype.toString = function () {
var self = this
, xml = [ '<?xml version="1.0" encoding="UTF-8"?>',
'<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">'];
if (this.cacheEnable && this.cache) {
return this.cache;
}
// TODO: if size > limit: create sitemapindex
this.urls.forEach( function (elem, index) {
// SitemapItem
var smi = elem;
Specifically:
if (this.cacheEnable && this.cache) {
return this.cache;
}
And the clear cache operation has a setInterval on it equal to the cacheTime parameter given.
Note the implication that your sitemap could become out of date if your urls change and your cacheTime has not triggered a clearing of the sitemap cache.