We Keep Coding

Multiple Calls For Web Data Connectors Tableau using Java Script Promises

I wrote a Web Data Connector that queries an API and gets data for a symbol that is passed in via URL.
However, I want to extend the API to get more than one symbol worth of data, but the way the API is structured, I would need to make multiple calls to this in the URL which means one getJSON call per symbol. I have tried wrapping the calls into a for loop and running the call for every symbol. This works fine when I run it in Simulator, but when I run this in Tableau Desktop, it returns just one symbol. I found one solution that this kind of issue can be solved using Promises, but I’m new in Java Script and I wasn’t able to figure out how to exactly apply this to my code. I provided that part of the code below, can anyone give me some ideas how can I use promises?
// Download the data
myConnector.getData = function(table, doneCallback) {
var dataObj = JSON.parse(tableau.connectionData);
const base_url = "https://myurl/getData.json?";
var copiedObj = Object.assign({}, dataObj);
console.log(copiedObj)
delete copiedObj.symbol;
delete copiedObj.apikey;
var symbols = dataObj.symbol.split(',');
for ( let i = 0; i < symbols.length; i ++){
dataString = base_url + "apikey="+ dataObj.apikey + "&symbol=" + symbols[i];
const url_test = new URL(dataString);
for (const [key,value] of Object.entries(copiedObj)){
url_test.searchParams.set(key,value);
}
const apiCall = url_test.href;
$.getJSON(apiCall, function(resp) {
var feat = resp.results,
tableData = [];
// Iterate over the JSON object
for (var i = 0, len = feat.length; i < len; i++) {
tableData.push({
"attribute1”: feat[i].attribute1,
“attribute2”: feat[i].attribute2,
“attribute3”: feat[i].attribute3,
“attribute4”: feat[i].attribute4,
});
}
table.appendRows(tableData);
doneCallback();
});
}
};

How to (Properly) add in each execution of a loop in node.js

So i'm attempting to write a google parser.
The idea of my tool is it takes search queries and searches google for them and returns URLs. It is working good so far but now im trying to set a page configuration and im having troubles, my code is:
const needle = require("needle") //for making get request
const sp = require("serp-parser") //for parsing data from the request
const queryup = "watch movies online free" //my search data
const query = encodeURI(queryup) //my search data so google can read it
var page = 0; //initializing the page counter
let pages = 5; //setting amount of pages to loop through
for (var i = 0; i < pages; i++) { //my loop
needle.get(`https://www.google.com/search?q=${query}&start=${page}`, function(err, response){ //MY MAIN PROBLEM <<<--- The issue is its adding to the page value but its not effecting it here, why?
page += 10 //adding to page value (every 10 page value is 1 extra page)
console.log(`----- Page number: `+ page / 10+" -----") //logging the number of the page to confirm that it is indeed increasing the page value
let results = response.body; //defining the body of my request
parser = new sp.GoogleNojsSERP(results); //initializing the parser
let parsed = parser.serp //parsing the body
let objarray = parsed.organic; //parsed body (returns as an array of json objects)
for (var i = 0; i < objarray.length; i++) { //loop the logging of each url
let url = objarray[i].url //defining url
console.log(url) //logging each url
}
});
}
without a billion comments:
const needle = require("needle")
const sp = require("serp-parser")
const queryup = "watch movies online free"
const query = encodeURI(queryup)
var page = 0;
let pages = 5;
for (var i = 0; i < pages; i++) {
needle.get(`https://www.google.com/search?q=${query}&start=${page}`, function(err, response){
//^^^^^ MY MAIN PROBLEM <<<--- The issue is its adding to the page value but its not effecting it here, why?
page += 10
console.log(`----- Page number: `+ page / 10+" -----")
let results = response.body;
parser = new sp.GoogleNojsSERP(results);
let parsed = parser.serp
let objarray = parsed.organic;
for (var i = 0; i < objarray.length; i++) {
let url = objarray[i].url
console.log(url)
}
});
}

This seems to be an issue with async.
I'm not familiar with needle, but I know that external queries are basically never synchronous.
The problem you're experiencing is basically, the actual web query is happening after the loop first runs and has already incremented page to 50. Then, 5 queries are constructed, each one with page=50, because async is complicated and difficult to manage.
Under the hood, the engine is essentially doing literally everything else it can possibly do first, and THEN doing your web queries.
A trip through the needle npm docs tells me that you can use alternative syntax to get needle to return a promise instead, which can then be wrapped in an asynchronous function and managed through await to force synchronous behavior, which is what you're after:
const needle = require('needle');
const sp = require('serp-parser');
const queryup = 'watch movies online free';
const query = encodeURI(queryup);
let page = 0;
const pages = 5;
const googler = async function () {
for (let i = 0; i < pages; i++) {
try {
const response = await needle('get', `https://www.google.com/search?q=${query}&start=${page}`);// MY MAIN PROBLEM <<<--- The issue is its adding to the page value but its not effecting it here, why?
console.log('----- Page number: ' + page / 10 + ' -----');
const results = await response.body;
const parser = new sp.GoogleNojsSERP(results);
const parsed = parser.serp;
const objarray = parsed.organic;
for (let i = 0; i < objarray.length; i++) {
const url = objarray[i].url;
console.log(url);
}
} catch (err) {
console.error(err);
}
page += 10;
}
};
googler();
The key differences:
Per the needle docs, rather than the request method being a method on the needle object, it's instead the first argument you pass directly to invoking needle itself as a function.
When you manage promises with await, a rejected promise throws an error that should be caught with a traditional try/catch block; I've done that here. Though, if needle is anything like node-fetch it probably basically never throws errors, but it's good practice.
One of my extensions automatically changed your var declarations to let and not-reassigned let declarations to const; you're welcome to change them back.

This is a classic asynchronous problem. Add another console.log() immediately before the needle.get() call (and after the for statement) and you will see what is going wrong: All of the needle.get() calls execute before any of the callbacks where you do the page += 10. Then, after the for loop completes, all of the callbacks are executed. But it is too late for this to have any effect on the start= parameter.
One way to fix this could be to move the body of this for loop (the needle.get() and its callback) into a separate function. Initialize your variables and call this function once. Then at the end of the callback, do your page += 10 and update any other variables you need to, and call this function again from there if there are more pages left that you want to load. If you have completed all of the pages, then don't make that call. The for loop is not needed with this technique.
Or, you could keep your current code but move the page += 10 after the callback but still inside the outer for loop. That way this variable will be incremented as you expect. I don't necessarily recommend this, as Google may get unhappy about receiving the get requests so rapidly and may start blocking your calls or throwing CAPTCHAs at you.
There may be an issue of whether this kind of scraping is allowed by Google's Terms of Service, but I will leave that question to you and your legal advisors.
Also, I would avoid using var anywhere. Use const or let instead, and prefer const over let except when you need to reassign the variable.
One tip: in most cases where you use a numeric for loop to iterate over an array, the code will be cleaner if you use a for..of loop. For example, this bit of code:
let parsed = parser.serp
let objarray = parsed.organic;
for (var i = 0; i < objarray.length; i++) {
let url = objarray[i].url
console.log(url)
}
could be more simply written as:
for (const result of parser.serp.organic) {
console.log(result.url)
}
(I know that is just a bit of debug code, but this is a good habit to get into.)
Finally, watch your indentation and be sure to indent nested blocks or functions. I took the liberty of adding some indentation for you.

Javascript: Is there anything I can do on my end to stop a 403 from occurring?

im on javascript and im currently trying to work with pull requests, issues and commits from a repo. I have the following code:
const axios = require('axios');
var gitPullApiLink = "https://api.github.com/repos/elixir-lang/elixir/pulls";
var listOfCommits = [];
var listOfSHAs = [];
var mapOfInfoObjects = new Map();
var mapPullRequestNumberToCommits = new Map();
var mapPRNumbersToCommitObjects = new Map();
var listOfPrObjects = [];
var setOfFileObjects = new Set();
var listOfNumbersOfTargetedIssues = [];
var mapPRnumberToCloseOpenDateObjects = new Map();
class PullRequestParser {
async getListOfPullRequests(pullrequestLink) {
const message = await axios.get(pullrequestLink);
//console.log(message);
listOfPrObjects = message['data'];
}
async getCommitsForEachPullRequestAndPRinformation() {
var listOfPrNumbers = [];
var k;
// this loop will just make a list of Pull Request Numbers
for (k = 0; k < listOfPrObjects.length; k++){
var currPrNumber = listOfPrObjects[k]['number'];
listOfPrNumbers.push(currPrNumber);
}
// I created a separate list just because... I did it this way because on the github API website it seems
// like the pull request has the same number as the issue it affects. I explain how you can see this down below
listOfNumbersOfTargetedIssues = listOfPrNumbers;
// next loop will make objects that contain information about each pull request.
var n;
for (n = 0; n < listOfPrNumbers; n++){
var ApiLinkForEachPullRequest = gitPullApiLink + "/" + listOfPrNumbers[n];
const mes = await axios.get(ApiLinkForEachPullRequest);
var temp = {OpeningDate: mes['data']['created_at'],
ClosingDate: mes['data']['closed_at'],
IssueLink: mes['data']['_links']['issue']['href']};
//mapPRnumberToCloseOpenDateObjects will be a map where the key is the pull request number and the value
// is the object that stores the open date, close date, and issue link for that pull request. The reason
// why I said I think the pull request number is the same as the number of the issue it affects is because
// if you take any object from the map, say you do mapPRnumberToCloseOpenDateObjects.get(10). You'll
// get an object with a pull request number 10. Now if you take this object and look at it's "IssueLink"
// field, the very last part of the link will have the number 10, and if you look at the github API
// it says for a single issue, you do: /repos/:owner/:repo/issues/:issue_number <---- As you can see,
// the IssueLink field will have this structure and in place of the issue_number, the field will be 10
// for our example object.
mapPRnumberToCloseOpenDateObjects.set(listOfPrNumbers[n], temp);
}
//up to this point, we have the pull request numbers. we will now start getting the commits associated with
//each pull request
var j;
for (j = 0; j < listOfPrNumbers.length; j++){
var currentApiLink = "https://api.github.com/repos/elixir-lang/elixir/pulls/" + listOfPrNumbers[j] + "/commits";
const res = await axios.get(currentApiLink);
//here we map a single pull request to the information containing the commits. I'll just warn you in
// advance: there's another object called mapPRNumbersToCommitObjects. THIS MAP IS DIFFERENT! I know it's
// subtle, but I hope the language can make the distinction: mapPullRequestNumberToCommits will just
// map a pull request number to some data about the commits it's linked to. In contrast,
// mapPRNumbersToCommitObjects will be the map that actually maps pull request numbers to objects
// containing information about the commits a pull request is associated with!
mapPullRequestNumberToCommits.set(listOfPrNumbers[j], res['data']);
}
// console.log("hewoihoiewa");
}
async createCommitObjects(){
var x;
// the initial loop using x will loop over all pull requests and get the associated commits
for (x = 0; x < listOfPrObjects.length; x++){
//here we will get the commits
var currCommitObjects = mapPullRequestNumberToCommits.get(listOfPrObjects[x]['number']);
//console.log('dhsiu');
// the loop using y will iterate over all commits that we get from a single pull request
var y;
for (y = 0; y < currCommitObjects.length; y++){
var currentSHA = currCommitObjects[y]['sha'];
listOfSHAs.push(currentSHA);
var currApiLink = "https://api.github.com/repos/elixir-lang/elixir/commits/" + currentSHA;
const response = await axios.get(currApiLink);
//console.log("up to here");
// here we start extracting some information from a single commit
var currentAuthorName = response['data']['commit']['committer']['name'];
var currentDate = response['data']['commit']['committer']['date'];
var currentFiles = response['data']['files'];
// this loop will iterate over all changed files for a single commit. Remember, every commit has a list
// of changed files, so this loop will iterate over all those files, get the necessary information
// from those files.
var z;
// we create this temporary list of file objects because for every file, we want to make an object
// that will store the necessary information for that one file. after we store all the objects for
// each file, we will add this list of file objects as a field for our bigger commit object (see down below)
var tempListOfFileObjects = [];
for (z = 0; z < currentFiles.length; z++){
var fileInConsideration = currentFiles[z];
var nameOfFile = fileInConsideration['filename'];
var numberOfAdditions = fileInConsideration['additions'];
var numberOfDeletions = fileInConsideration['deletions'];
var totalNumberOfChangesToFile = fileInConsideration['changes'];
//console.log("with file");
var tempFileObject = {fileName: nameOfFile, totalAdditions: numberOfAdditions,
totalDeletions: numberOfDeletions, numberOfChanges: totalNumberOfChangesToFile};
// we add the same file objects to both a temporary, local list and a global set. Don't be tripped
// up by this; they're doing the same thing!
setOfFileObjects.add(tempFileObject);
tempListOfFileObjects.push(tempFileObject);
}
// here we make an object that stores information for a single commit. sha, authorName, date are single
// values, but files will be a list of file objects and these file objects will store further information
// for each file.
var tempObj = {sha: currentSHA, authorName: currentAuthorName, date: currentDate, files: tempListOfFileObjects};
var currPrNumber = listOfPrObjects[x]['number'];
console.log(currPrNumber);
// here we will make a single pull request number to an object that will contain all the information for
// every single commit associated with that pull request. So for every pull request, it will map to a list
// of objects where each object stores information about a commit associated with the pull request.
mapPRNumbersToCommitObjects.set(currPrNumber, tempObj);
}
}
return mapPRNumbersToCommitObjects;
}
startParsingPullRequests() {
this.getListOfPullRequests(gitPullApiLink + "?state=all").then(() => {
this.getCommitsForEachPullRequestAndPRinformation().then(() => {
this.createCommitObjects().then((response) => {
console.log("functions were successful");
return mapPRNumbersToCommitObjects;
}).catch((error) => {
console.log("printing first error");
// console.log(error);
})
}).catch((error2) => {
console.log("printing the second error");
console.log(error2);
})
}).catch((error3) => {
console.log("printing the third error");
// console.log(error3);
});
}
//adding some getter methods so they can be used to work with whatever information people may need.
//I start all of them with the this.startParsingPullRequests() method because by calling that method it gets all
// the information for the global variables.
async getSetOfFileObjects(){
var dummyMap = this.startParsingPullRequests();
return setOfFileObjects;
}
async OpenCloseDateObjects(){
var dummyMap = this.startParsingPullRequests();
return mapPRnumberToCloseOpenDateObjects;
}
async getNumbersOfTargetedIssues(){
var dummyMap = this.startParsingPullRequests();
return listOfNumbersOfTargetedIssues;
}
}
I then try to play around and run the function to make sure all the data I need is there by doing:
var dummy = new PullRequestParser();
var dummyMap = dummy.startParsingPullRequests();
And when I run it on webstorm using:
node PullRequestParser.js
It will print out some pull request numbers, then stop about halfway with a 403 error. I know what the 403 error is, but I'm wondering if there's anything on my end to stop it from happening, or is it just a matter of working with another repo that won't throw me this error. Thanks!

The 403 error from the server means that your access is forbidden. You either need to use different credentials (that is, log in as a user with that access), not use that repository, or gracefully handle the error and do something else. Retrying will not be effective, since GitHub wouldn't be very secure if it just let you have access to things you weren't supposed to.

How to create multiple objects in a loop in Parse.com SDK?

I'm trying to create several objects in a loop in Parse's Javascript SDK.
for (var i = 0; i < results.length; i++){
var user = results[i],
newPuzzle = new Puzzle();
newPuzzle.set("userAsked", user);
newPuzzle.save();
}
But it works only for several objects (from 2 to 5) and then falls to response. I found method Parse.Object.saveAll(list, options) but it doesn't work for creating AFAIK - only for updating.
I also used local function written on pure Node.js with Parse's master key - it can save objects in a loop and works perfectly. But I need working with filesystem and local JavaScript for me is a headache.
How can I create multiple objects in one request in Parse's SDK?
Thanks in advance!

You want to use, saveAll function, see example below:
var TimelineEvent = Parse.Object.extend("Timeline");
exports.processNotifications = function (notifications, successCallback, failureCallback) {
var timelineEvents = [];
for (var i = 0; i < notifications.length; i++) {
var notification = notifications[i];
if (notification.hasOwnProperty("timelineEvent")) {
var timelineEvent = new TimelineEvent();
timelineEvent.set("eventNotificationKey", notification.notificationKey);
timelineEvent.set("isDevelopmentAccount", notification.isDebugOnly);
timelineEvent.set("eventName", notification.timelineEvent.name);
timelineEvent.set("eventDescription", notification.timelineEvent.description);
timelineEvent.set("eventValue", notification.timelineEvent.value);
timelineEvent.set("channel", notification.channel);
timelineEvents.push(timelineEvent);
}
}
Parse.Object.saveAll(timelineEvents, {
success:successCallback,
error:failureCallback
});
};

Use saveAll function
https://www.parse.com/docs/js/api/symbols/Parse.Object.html#.saveAll
You will need to create an array of objects you wish to save first and pass as the first parameter.

Multiple file upload - check which of the files exist on server by Ajax-request

I can't seem to find what's wrong with the idea of this script and can't get it to work. Maybe you can help me, it would be highly appreciated.
I'm working on a multiple file upload form.
Prior to uploading (and not before then) I want to check whether some of the files (and if so, which ones) already exist in the upload directory.
I'm using XMLHttpRequests to do that. Since I can't control how long exactly those will take to get a response I run a loop using arrays for all the variables so they (at least that was my idea ;-) ) can do their job indepently from each other.
function NoOverwrite() {
var fields = document.querySelectorAll("input[type=file]");
var existing = new Array(); //files existing on server
var checkFile = new Array();
var file = new Array();
var fileUrl = new Array();
for (var i = 0; i < fields.length; i++) {
file[i] = document.getElementById('file'+i).files[0];
//the input- fields of the form are called file0, file1, file2, and so on...
if(file[i]) {
fileUrl[i] = 'upload_path/' + file[i].name;
//up to here everything works fine - when setting an alert after this I get
//the names of all the names of the files selected in the file fields!
checkFile[i] = new XMLHttpRequest();
checkFile[i].open('HEAD', fileUrl[i], true);
checkFile[i].onreadystatechange = function() {
if (checkFile[i].readyState == 4) {
if (checkFile[i].status == 200) {
existing[i] = true;
alert(existing[i]); //this never came up...
}
}
checkFile[i].send();
}
}
}
if (existing.indexOf(true) == -1) {
//none of the files to be uploaded are already on server
//this _always_ was fired - even when there are files with the same name on the server!!!??
return true;
}
else {
//list filenames and/or upload field numbers of the files that already exist on server
return false;
}
}
Did I make an error in my thinking? Or are there some simple mistakes in my code? Any ideas how I could archive my goal?

I see two potential issues with your code.
The first one is the same origin policy: if upload_path is not in the same domain as your current page, the request might fail.
The second is your index: as i increases, when you do alert(existing[i]) i is already equal to fields.length. You need to create a closure:
for (var i = 0; i < fields.length; i++) {
(function(index){
// use index within your code
})(i); // the loop will execute with index=i
}

Ok, I found the answer - just in case anyone is interested...
In addition to Christophes remark that a closure for the loop has to be created, the checking of the array of course also has to happen inside that loop function, otherwise it's checked immediately (i.e. when there hasn't been a single response of the XMLHttpRequests yet) and since it is only checked once, it will always be negative.
Apart from that we have to make sure that ALL requests are finished (not just being processed) before we check the values in the array. We do that by comparing the number of filled in upload fields against the number of values that have been set into the string (which happens after the readystate-response of the corresponding request came back).
More explanations in the code.
Cheers, Chris
function NoOverwrite() {
var fields = document.querySelectorAll("input[type=file]");
var existing = new Array();
var checkFile = new Array();
var file = new Array();
var fileUrl = new Array();
var counter = 0;
for (var i = 0; i < fields.length; i++) {
(function(index){
file[index] = document.getElementById('file'+i).files[0];
if(file[index]) {
fileUrl[index] = 'upload_path/' + file[index].name;
checkFile[index] = new XMLHttpRequest();
checkFile[index].onreadystatechange = function() {
if (checkFile[index].readyState == 4) {
if (checkFile[index].status == 200) {
existing[index] = true;
counter += 1;
}
else {
existing[index] = false;
counter += 1;
}
if (counter == fileUrl.length) {
//existing.length of the array "true, false,,true" (i.e. with one undefined value) would deliver "4".
//therefore we have to check for the number of set variables in the string rather than the strings length.
//we use a counter for that purpose. everything after this point is only executed when the last file has been checked!
if (existing.indexOf(true) == -1) {
//none of the files to be uploaded are already on server
return true;
}
else {
//list filenames and/or upload field numbers of the files that already exist on server
// ->> inform user...
return false;
}
}
}
}
checkFile[index].open('HEAD', fileUrl[index], true);
checkFile[index].send();
}
})(i);
}
}