We Keep Coding

Multiple Calls For Web Data Connectors Tableau using Java Script Promises

I wrote a Web Data Connector that queries an API and gets data for a symbol that is passed in via URL.
However, I want to extend the API to get more than one symbol worth of data, but the way the API is structured, I would need to make multiple calls to this in the URL which means one getJSON call per symbol. I have tried wrapping the calls into a for loop and running the call for every symbol. This works fine when I run it in Simulator, but when I run this in Tableau Desktop, it returns just one symbol. I found one solution that this kind of issue can be solved using Promises, but I’m new in Java Script and I wasn’t able to figure out how to exactly apply this to my code. I provided that part of the code below, can anyone give me some ideas how can I use promises?
// Download the data
myConnector.getData = function(table, doneCallback) {
var dataObj = JSON.parse(tableau.connectionData);
const base_url = "https://myurl/getData.json?";
var copiedObj = Object.assign({}, dataObj);
console.log(copiedObj)
delete copiedObj.symbol;
delete copiedObj.apikey;
var symbols = dataObj.symbol.split(',');
for ( let i = 0; i < symbols.length; i ++){
dataString = base_url + "apikey="+ dataObj.apikey + "&symbol=" + symbols[i];
const url_test = new URL(dataString);
for (const [key,value] of Object.entries(copiedObj)){
url_test.searchParams.set(key,value);
}
const apiCall = url_test.href;
$.getJSON(apiCall, function(resp) {
var feat = resp.results,
tableData = [];
// Iterate over the JSON object
for (var i = 0, len = feat.length; i < len; i++) {
tableData.push({
"attribute1”: feat[i].attribute1,
“attribute2”: feat[i].attribute2,
“attribute3”: feat[i].attribute3,
“attribute4”: feat[i].attribute4,
});
}
table.appendRows(tableData);
doneCallback();
});
}
};

How to (Properly) add in each execution of a loop in node.js

So i'm attempting to write a google parser.
The idea of my tool is it takes search queries and searches google for them and returns URLs. It is working good so far but now im trying to set a page configuration and im having troubles, my code is:
const needle = require("needle") //for making get request
const sp = require("serp-parser") //for parsing data from the request
const queryup = "watch movies online free" //my search data
const query = encodeURI(queryup) //my search data so google can read it
var page = 0; //initializing the page counter
let pages = 5; //setting amount of pages to loop through
for (var i = 0; i < pages; i++) { //my loop
needle.get(`https://www.google.com/search?q=${query}&start=${page}`, function(err, response){ //MY MAIN PROBLEM <<<--- The issue is its adding to the page value but its not effecting it here, why?
page += 10 //adding to page value (every 10 page value is 1 extra page)
console.log(`----- Page number: `+ page / 10+" -----") //logging the number of the page to confirm that it is indeed increasing the page value
let results = response.body; //defining the body of my request
parser = new sp.GoogleNojsSERP(results); //initializing the parser
let parsed = parser.serp //parsing the body
let objarray = parsed.organic; //parsed body (returns as an array of json objects)
for (var i = 0; i < objarray.length; i++) { //loop the logging of each url
let url = objarray[i].url //defining url
console.log(url) //logging each url
}
});
}
without a billion comments:
const needle = require("needle")
const sp = require("serp-parser")
const queryup = "watch movies online free"
const query = encodeURI(queryup)
var page = 0;
let pages = 5;
for (var i = 0; i < pages; i++) {
needle.get(`https://www.google.com/search?q=${query}&start=${page}`, function(err, response){
//^^^^^ MY MAIN PROBLEM <<<--- The issue is its adding to the page value but its not effecting it here, why?
page += 10
console.log(`----- Page number: `+ page / 10+" -----")
let results = response.body;
parser = new sp.GoogleNojsSERP(results);
let parsed = parser.serp
let objarray = parsed.organic;
for (var i = 0; i < objarray.length; i++) {
let url = objarray[i].url
console.log(url)
}
});
}

This seems to be an issue with async.
I'm not familiar with needle, but I know that external queries are basically never synchronous.
The problem you're experiencing is basically, the actual web query is happening after the loop first runs and has already incremented page to 50. Then, 5 queries are constructed, each one with page=50, because async is complicated and difficult to manage.
Under the hood, the engine is essentially doing literally everything else it can possibly do first, and THEN doing your web queries.
A trip through the needle npm docs tells me that you can use alternative syntax to get needle to return a promise instead, which can then be wrapped in an asynchronous function and managed through await to force synchronous behavior, which is what you're after:
const needle = require('needle');
const sp = require('serp-parser');
const queryup = 'watch movies online free';
const query = encodeURI(queryup);
let page = 0;
const pages = 5;
const googler = async function () {
for (let i = 0; i < pages; i++) {
try {
const response = await needle('get', `https://www.google.com/search?q=${query}&start=${page}`);// MY MAIN PROBLEM <<<--- The issue is its adding to the page value but its not effecting it here, why?
console.log('----- Page number: ' + page / 10 + ' -----');
const results = await response.body;
const parser = new sp.GoogleNojsSERP(results);
const parsed = parser.serp;
const objarray = parsed.organic;
for (let i = 0; i < objarray.length; i++) {
const url = objarray[i].url;
console.log(url);
}
} catch (err) {
console.error(err);
}
page += 10;
}
};
googler();
The key differences:
Per the needle docs, rather than the request method being a method on the needle object, it's instead the first argument you pass directly to invoking needle itself as a function.
When you manage promises with await, a rejected promise throws an error that should be caught with a traditional try/catch block; I've done that here. Though, if needle is anything like node-fetch it probably basically never throws errors, but it's good practice.
One of my extensions automatically changed your var declarations to let and not-reassigned let declarations to const; you're welcome to change them back.

This is a classic asynchronous problem. Add another console.log() immediately before the needle.get() call (and after the for statement) and you will see what is going wrong: All of the needle.get() calls execute before any of the callbacks where you do the page += 10. Then, after the for loop completes, all of the callbacks are executed. But it is too late for this to have any effect on the start= parameter.
One way to fix this could be to move the body of this for loop (the needle.get() and its callback) into a separate function. Initialize your variables and call this function once. Then at the end of the callback, do your page += 10 and update any other variables you need to, and call this function again from there if there are more pages left that you want to load. If you have completed all of the pages, then don't make that call. The for loop is not needed with this technique.
Or, you could keep your current code but move the page += 10 after the callback but still inside the outer for loop. That way this variable will be incremented as you expect. I don't necessarily recommend this, as Google may get unhappy about receiving the get requests so rapidly and may start blocking your calls or throwing CAPTCHAs at you.
There may be an issue of whether this kind of scraping is allowed by Google's Terms of Service, but I will leave that question to you and your legal advisors.
Also, I would avoid using var anywhere. Use const or let instead, and prefer const over let except when you need to reassign the variable.
One tip: in most cases where you use a numeric for loop to iterate over an array, the code will be cleaner if you use a for..of loop. For example, this bit of code:
let parsed = parser.serp
let objarray = parsed.organic;
for (var i = 0; i < objarray.length; i++) {
let url = objarray[i].url
console.log(url)
}
could be more simply written as:
for (const result of parser.serp.organic) {
console.log(result.url)
}
(I know that is just a bit of debug code, but this is a good habit to get into.)
Finally, watch your indentation and be sure to indent nested blocks or functions. I took the liberty of adding some indentation for you.

Javascript: Is there anything I can do on my end to stop a 403 from occurring?

im on javascript and im currently trying to work with pull requests, issues and commits from a repo. I have the following code:
const axios = require('axios');
var gitPullApiLink = "https://api.github.com/repos/elixir-lang/elixir/pulls";
var listOfCommits = [];
var listOfSHAs = [];
var mapOfInfoObjects = new Map();
var mapPullRequestNumberToCommits = new Map();
var mapPRNumbersToCommitObjects = new Map();
var listOfPrObjects = [];
var setOfFileObjects = new Set();
var listOfNumbersOfTargetedIssues = [];
var mapPRnumberToCloseOpenDateObjects = new Map();
class PullRequestParser {
async getListOfPullRequests(pullrequestLink) {
const message = await axios.get(pullrequestLink);
//console.log(message);
listOfPrObjects = message['data'];
}
async getCommitsForEachPullRequestAndPRinformation() {
var listOfPrNumbers = [];
var k;
// this loop will just make a list of Pull Request Numbers
for (k = 0; k < listOfPrObjects.length; k++){
var currPrNumber = listOfPrObjects[k]['number'];
listOfPrNumbers.push(currPrNumber);
}
// I created a separate list just because... I did it this way because on the github API website it seems
// like the pull request has the same number as the issue it affects. I explain how you can see this down below
listOfNumbersOfTargetedIssues = listOfPrNumbers;
// next loop will make objects that contain information about each pull request.
var n;
for (n = 0; n < listOfPrNumbers; n++){
var ApiLinkForEachPullRequest = gitPullApiLink + "/" + listOfPrNumbers[n];
const mes = await axios.get(ApiLinkForEachPullRequest);
var temp = {OpeningDate: mes['data']['created_at'],
ClosingDate: mes['data']['closed_at'],
IssueLink: mes['data']['_links']['issue']['href']};
//mapPRnumberToCloseOpenDateObjects will be a map where the key is the pull request number and the value
// is the object that stores the open date, close date, and issue link for that pull request. The reason
// why I said I think the pull request number is the same as the number of the issue it affects is because
// if you take any object from the map, say you do mapPRnumberToCloseOpenDateObjects.get(10). You'll
// get an object with a pull request number 10. Now if you take this object and look at it's "IssueLink"
// field, the very last part of the link will have the number 10, and if you look at the github API
// it says for a single issue, you do: /repos/:owner/:repo/issues/:issue_number <---- As you can see,
// the IssueLink field will have this structure and in place of the issue_number, the field will be 10
// for our example object.
mapPRnumberToCloseOpenDateObjects.set(listOfPrNumbers[n], temp);
}
//up to this point, we have the pull request numbers. we will now start getting the commits associated with
//each pull request
var j;
for (j = 0; j < listOfPrNumbers.length; j++){
var currentApiLink = "https://api.github.com/repos/elixir-lang/elixir/pulls/" + listOfPrNumbers[j] + "/commits";
const res = await axios.get(currentApiLink);
//here we map a single pull request to the information containing the commits. I'll just warn you in
// advance: there's another object called mapPRNumbersToCommitObjects. THIS MAP IS DIFFERENT! I know it's
// subtle, but I hope the language can make the distinction: mapPullRequestNumberToCommits will just
// map a pull request number to some data about the commits it's linked to. In contrast,
// mapPRNumbersToCommitObjects will be the map that actually maps pull request numbers to objects
// containing information about the commits a pull request is associated with!
mapPullRequestNumberToCommits.set(listOfPrNumbers[j], res['data']);
}
// console.log("hewoihoiewa");
}
async createCommitObjects(){
var x;
// the initial loop using x will loop over all pull requests and get the associated commits
for (x = 0; x < listOfPrObjects.length; x++){
//here we will get the commits
var currCommitObjects = mapPullRequestNumberToCommits.get(listOfPrObjects[x]['number']);
//console.log('dhsiu');
// the loop using y will iterate over all commits that we get from a single pull request
var y;
for (y = 0; y < currCommitObjects.length; y++){
var currentSHA = currCommitObjects[y]['sha'];
listOfSHAs.push(currentSHA);
var currApiLink = "https://api.github.com/repos/elixir-lang/elixir/commits/" + currentSHA;
const response = await axios.get(currApiLink);
//console.log("up to here");
// here we start extracting some information from a single commit
var currentAuthorName = response['data']['commit']['committer']['name'];
var currentDate = response['data']['commit']['committer']['date'];
var currentFiles = response['data']['files'];
// this loop will iterate over all changed files for a single commit. Remember, every commit has a list
// of changed files, so this loop will iterate over all those files, get the necessary information
// from those files.
var z;
// we create this temporary list of file objects because for every file, we want to make an object
// that will store the necessary information for that one file. after we store all the objects for
// each file, we will add this list of file objects as a field for our bigger commit object (see down below)
var tempListOfFileObjects = [];
for (z = 0; z < currentFiles.length; z++){
var fileInConsideration = currentFiles[z];
var nameOfFile = fileInConsideration['filename'];
var numberOfAdditions = fileInConsideration['additions'];
var numberOfDeletions = fileInConsideration['deletions'];
var totalNumberOfChangesToFile = fileInConsideration['changes'];
//console.log("with file");
var tempFileObject = {fileName: nameOfFile, totalAdditions: numberOfAdditions,
totalDeletions: numberOfDeletions, numberOfChanges: totalNumberOfChangesToFile};
// we add the same file objects to both a temporary, local list and a global set. Don't be tripped
// up by this; they're doing the same thing!
setOfFileObjects.add(tempFileObject);
tempListOfFileObjects.push(tempFileObject);
}
// here we make an object that stores information for a single commit. sha, authorName, date are single
// values, but files will be a list of file objects and these file objects will store further information
// for each file.
var tempObj = {sha: currentSHA, authorName: currentAuthorName, date: currentDate, files: tempListOfFileObjects};
var currPrNumber = listOfPrObjects[x]['number'];
console.log(currPrNumber);
// here we will make a single pull request number to an object that will contain all the information for
// every single commit associated with that pull request. So for every pull request, it will map to a list
// of objects where each object stores information about a commit associated with the pull request.
mapPRNumbersToCommitObjects.set(currPrNumber, tempObj);
}
}
return mapPRNumbersToCommitObjects;
}
startParsingPullRequests() {
this.getListOfPullRequests(gitPullApiLink + "?state=all").then(() => {
this.getCommitsForEachPullRequestAndPRinformation().then(() => {
this.createCommitObjects().then((response) => {
console.log("functions were successful");
return mapPRNumbersToCommitObjects;
}).catch((error) => {
console.log("printing first error");
// console.log(error);
})
}).catch((error2) => {
console.log("printing the second error");
console.log(error2);
})
}).catch((error3) => {
console.log("printing the third error");
// console.log(error3);
});
}
//adding some getter methods so they can be used to work with whatever information people may need.
//I start all of them with the this.startParsingPullRequests() method because by calling that method it gets all
// the information for the global variables.
async getSetOfFileObjects(){
var dummyMap = this.startParsingPullRequests();
return setOfFileObjects;
}
async OpenCloseDateObjects(){
var dummyMap = this.startParsingPullRequests();
return mapPRnumberToCloseOpenDateObjects;
}
async getNumbersOfTargetedIssues(){
var dummyMap = this.startParsingPullRequests();
return listOfNumbersOfTargetedIssues;
}
}
I then try to play around and run the function to make sure all the data I need is there by doing:
var dummy = new PullRequestParser();
var dummyMap = dummy.startParsingPullRequests();
And when I run it on webstorm using:
node PullRequestParser.js
It will print out some pull request numbers, then stop about halfway with a 403 error. I know what the 403 error is, but I'm wondering if there's anything on my end to stop it from happening, or is it just a matter of working with another repo that won't throw me this error. Thanks!

The 403 error from the server means that your access is forbidden. You either need to use different credentials (that is, log in as a user with that access), not use that repository, or gracefully handle the error and do something else. Retrying will not be effective, since GitHub wouldn't be very secure if it just let you have access to things you weren't supposed to.

How do I trigger events using Meteor server side events

I am using Meteor, which uses Mongodb as its database. I have code that inserts several documents into a collection when users fill out a form. When these documents are inserted, I would like to fire some JavaScript code within the server side directories that sorts through the collection in question for documents with matching fields as the documents just inserted.
My problem is that I do not know how to fire code on the server when the new documents arrive. Would it make sense to Meteor.call a Meteor.method at the end of the code involved with inserting, with the Meteor.method called preforming the sorting code I need?
Edit:
As you can see, in the below code I'm not calling any Meteor methods as none exist yet. The vast majority of this code is simply lead up for the insert({}) at the end of the page, so I think it can be safely ignored. The only server side code I have is to declare the possibleGames mongo collection.
I am not sure what you mean by call a plain JavaScript function, my problem is getting any code firing at all.
possibleGames = new Mongo.Collection("possibleGames");
Template.meet_form.events({
"submit .meet_form": function(event, template){
event.preventDefault();
var user = Meteor.userId();
var where = event.target.where.value;
var checkedGames = [];
function gameCheck (game) {
if (game.checked === true){
checkedGames.push(game.value);
};
};
var checkedDays = [];
function dayCheck (day) {
if (day.checked === true){
checkedDays.push(day.value);
};
};
console.log(event.target.where.value)
gameCheck(event.target.dnd);
gameCheck(event.target.savageWorlds);
gameCheck(event.target.shadowRun);
console.log(checkedGames);
dayCheck(event.target.monday);
dayCheck(event.target.tuesday);
dayCheck(event.target.wednesday);
dayCheck(event.target.thursday);
dayCheck(event.target.friday);
dayCheck(event.target.saturday);
dayCheck(event.target.sunday);
console.log(checkedDays);
var whereWhat = [];
for (i = 0; i < checkedGames.length; i++) {
var prepareWhereWhat = where.concat(checkedGames[i]);
whereWhat.push(prepareWhereWhat);
};
console.log(whereWhat);
var whereWhatWhen = [];
for (a = 0; a < whereWhat.length; a++) {
var prepareWWW1 = whereWhat[a];
for (b = 0; b < checkedDays.length; b++) {
var prepareWWW2 = prepareWWW1.concat(checkedDays[b]);
whereWhatWhen.push(prepareWWW2);
};
};
console.log(whereWhatWhen);
for (i = 0; i < whereWhatWhen.length; i++) {
possibleGames.insert({
game: whereWhatWhen[i],
user: user,
created_on: new Date().getTime()
})
}
}
});

You don't need to do a meteor.call on the server because you're already on the server.
Just call a plain javascript function.
If what you want to call from your first Meteor.method is already in another Meteor.method, then refactor that function to extract out the common bit.
Some code would also help if this is still confusing.

Variable in for-loop not being reassigned each iteration

I've got a for loop inside a for loop. The first loop should get a username, get their rating, append them to the same list item, then start over with the next username, rating, append, and so on and so forth until it's gone through ever user in the friends list.
//Get usernames
var current = Parse.User.current();
var relation = current.relation("FriendRelations");
relation.query().find({
success: function(results) {
for (var i = 0; i < results.length; i++) {
//This shouldn't increment until the rating has been retrieved in the
// next function.
theuser = results[i].getUsername();
$('ul').prepend('<li id = "frienditems_' + i + '"><div id ="friendname">' +
results[i].getUsername() + '</div></li>');
//Get friend's rating.
//This is a query within a query.
var GameScore = Parse.Object.extend("Rating");
var query = new Parse.Query(GameScore);
query.equalTo("user", results[i].getUsername());
query.find({
success: function(result) {
//The log just outputs the last user's name each time, because I guess the
//other function already looped through completely, so it's forever set to the last one?**
console.log(theuser);
for (var y = 0; y < result.length; y++) {
var object = result[y].get("Rating");
//Logging the rating.
console.log("Rating:" + object);
$('#frienditems_' + y).append('<p class="friendrating">' + object + '</p>');
}
},
error: function(error) {}
});
}
}
});
Here is my console.log:
John
Rating:5
John
Rating:50
John
Rating:43
John
Rating:80
I want it to be this:
George
Rating:5
Smith
Rating:50
Robert
Rating:43
John
Rating:80
Can anyone help? I've looked at other questions, but I can't figure out how to apply them to my situation. I wish I could at least access the first functions results within the second function.
UPDATE: Mihail's answer really helped me out. The console log now shows data being retrieved in the correct order. But it's still not all appending to its respective list item:

Your issue is caused by the asynchronous call to query.find(). You practically order the browser to retrieve your information from the cloud database and tell him what to do with the result if it retrieves anything successfully, while the code continues execution (in your case, the for loop still iterates through the first collection).
By the time the first request ends, for reaches the end and because the second query's success is in the scope of the first, the variable theuser is still instantiated and has the last value available.
To prevent that, you can change the scope of the variable using a function call with your parameter. I've rewritten your code and it looks like this:
var current = Parse.User.current();
var relation = current.relation("FriendRelations");
var $ul = $('ul')
relation.query().find({
success:function(results){
for(var i = 0; i < results.length; i++) {
theuser = results[i].getUsername();
// you can use <<var $friendrating = $("<li id='frienditems_" + i"'/>").html(...... >>
var $friendrating = $(document.createElement("li")).attr("id", "frienditems_" + i).html('<div class="friendname">' + results[i].getUsername() + '</div>')
$ul.prepend($friendrating);
getUserRating(theuser, $friendrating);
}
}
});
function getUserRating(theUser, $node) {
//Get friend's rating.
//This is a query within a query.
var GameScore = Parse.Object.extend("Rating");
var query = new Parse.Query(GameScore);
query.equalTo("user", theUser);
query.find({
success: function(result) {
//The log just outputs the last user's name each time, because I guess the
//other function already looped through completely, so it's forever set to the last one?**
console.log(theUser);
for (var y = 0; y < result.length; y++) {
var object = result[y].get("Rating");
//Logging the rating.
console.log("Rating:" +object);
$node.append('<p class="friendrating">' + object + '</p>');
}
},
error: function(error) {
}
});
}
Please, be advised that i haven't tested it and i'm not 100% sure it will work on the first try.

because the anonymous callback function success: function(result) {} is called after the for loop finishes in this case, query.find() is an asynchronous call, meaning the for loop will continue independently to the next iteration no matter if the query.find() has returned or not.
variable theuser is always the value of theuser in the last for loop because the for loop is able to complete before any of the callback functions are triggered