We Keep Coding

Axios API call in nested loops doesn't return expected results

I'm using Axios to scrape JSON product data from a website. The script below first requests category data from server and loops through three hierarchies of categories, subcategories, and (I guess) sub-subcategories. In the third loop, the sub-subcategories query parameter is used as a parameter in a function call, which calls an asynchronous function which uses this query parameter to concatenate onto the URL to make the new URL, make a GET request for the product data related to the query parameter, and loop this until the the pageCounter iteration variable (which is used as a pagination query parameter to get N pages for each new URL) is equal to the pagination value within the current new url containing the sub-subcategory query parameter.
for (let i=0; i<obj.length; i++) {
let subcats = obj[i].subcategories.length;
for(let n=0; n<subcats; n++) {
if(obj[i].subcategories[n].facetValueData) {
let subsubcats = obj[i].subcategories[n].facetValueData.length
for(let p=0; p<subsubcats; p++) {
let productData = []
obj[i].subcategories[n].facetValueData[p].productData = productData
const scrapedData = await scrapeData(obj[i].subcategories[n].facetValueData.code)
obj[i].subcategories[n].facetValueData[p].productData.push(scrapedData)
}
} else {
console.log(`No facet values present - ${obj[i].subcategories[n]}`)
}
}
}
This is the async function that the third loop calls
async function scrapeData(queryParam) {
var product = []
var pagination;
try {
do {
var nextPageLink = `currentPage=${pageCounter}&query=::cagCollectionPoint:Departure+from+Singapore:cagCategory:${queryParam}`
var nextUrl = url.concat(nextPageLink)
const response = await axios({
method: "GET",
url: nextUrl,
withCredentials: true,
headers: headers
})
product = response.data["products"]
pagination = response.data["pagination"]["totalPages"]
pageCounter++;
//this logs all of the correct current queries
console.log(response.data.currentQuery.query.value)
} while (pageCounter<=pagination)
return product
} catch (error) {
console.error(error)
}
}
At first glance it appears as if it is working, as it populates the first few sub-subcategory objects with an array of products scraped, however some of these are not populated, i.e. productData = []
The console log function in the scrapeData function returns all of the correct current queries per iteration, however when they are returned it only returns the first few reponses.
I'm guessing the product array needs to wait? But the axios request is already awaited, so i dont see why this is happening.

If I'm understanding your intent on this piece of code correctly:
var product = [];
do {
// ...
const response = await axios(/* ... */);
product = response.data["products"]
// ...
} while (pageCounter<=1)
return product
it seems like there are multiple pages, and you want to get all of them into the product array?
But in reality, each page you are replacing the product array. I would assume that on the empty ones your last fetch gets no results, and you are just losing all of the rest of them.
What you probably want to do is just change the product = response.data["products"] to:
product.push(...response.data["products"])

Javascript: Is there anything I can do on my end to stop a 403 from occurring?

im on javascript and im currently trying to work with pull requests, issues and commits from a repo. I have the following code:
const axios = require('axios');
var gitPullApiLink = "https://api.github.com/repos/elixir-lang/elixir/pulls";
var listOfCommits = [];
var listOfSHAs = [];
var mapOfInfoObjects = new Map();
var mapPullRequestNumberToCommits = new Map();
var mapPRNumbersToCommitObjects = new Map();
var listOfPrObjects = [];
var setOfFileObjects = new Set();
var listOfNumbersOfTargetedIssues = [];
var mapPRnumberToCloseOpenDateObjects = new Map();
class PullRequestParser {
async getListOfPullRequests(pullrequestLink) {
const message = await axios.get(pullrequestLink);
//console.log(message);
listOfPrObjects = message['data'];
}
async getCommitsForEachPullRequestAndPRinformation() {
var listOfPrNumbers = [];
var k;
// this loop will just make a list of Pull Request Numbers
for (k = 0; k < listOfPrObjects.length; k++){
var currPrNumber = listOfPrObjects[k]['number'];
listOfPrNumbers.push(currPrNumber);
}
// I created a separate list just because... I did it this way because on the github API website it seems
// like the pull request has the same number as the issue it affects. I explain how you can see this down below
listOfNumbersOfTargetedIssues = listOfPrNumbers;
// next loop will make objects that contain information about each pull request.
var n;
for (n = 0; n < listOfPrNumbers; n++){
var ApiLinkForEachPullRequest = gitPullApiLink + "/" + listOfPrNumbers[n];
const mes = await axios.get(ApiLinkForEachPullRequest);
var temp = {OpeningDate: mes['data']['created_at'],
ClosingDate: mes['data']['closed_at'],
IssueLink: mes['data']['_links']['issue']['href']};
//mapPRnumberToCloseOpenDateObjects will be a map where the key is the pull request number and the value
// is the object that stores the open date, close date, and issue link for that pull request. The reason
// why I said I think the pull request number is the same as the number of the issue it affects is because
// if you take any object from the map, say you do mapPRnumberToCloseOpenDateObjects.get(10). You'll
// get an object with a pull request number 10. Now if you take this object and look at it's "IssueLink"
// field, the very last part of the link will have the number 10, and if you look at the github API
// it says for a single issue, you do: /repos/:owner/:repo/issues/:issue_number <---- As you can see,
// the IssueLink field will have this structure and in place of the issue_number, the field will be 10
// for our example object.
mapPRnumberToCloseOpenDateObjects.set(listOfPrNumbers[n], temp);
}
//up to this point, we have the pull request numbers. we will now start getting the commits associated with
//each pull request
var j;
for (j = 0; j < listOfPrNumbers.length; j++){
var currentApiLink = "https://api.github.com/repos/elixir-lang/elixir/pulls/" + listOfPrNumbers[j] + "/commits";
const res = await axios.get(currentApiLink);
//here we map a single pull request to the information containing the commits. I'll just warn you in
// advance: there's another object called mapPRNumbersToCommitObjects. THIS MAP IS DIFFERENT! I know it's
// subtle, but I hope the language can make the distinction: mapPullRequestNumberToCommits will just
// map a pull request number to some data about the commits it's linked to. In contrast,
// mapPRNumbersToCommitObjects will be the map that actually maps pull request numbers to objects
// containing information about the commits a pull request is associated with!
mapPullRequestNumberToCommits.set(listOfPrNumbers[j], res['data']);
}
// console.log("hewoihoiewa");
}
async createCommitObjects(){
var x;
// the initial loop using x will loop over all pull requests and get the associated commits
for (x = 0; x < listOfPrObjects.length; x++){
//here we will get the commits
var currCommitObjects = mapPullRequestNumberToCommits.get(listOfPrObjects[x]['number']);
//console.log('dhsiu');
// the loop using y will iterate over all commits that we get from a single pull request
var y;
for (y = 0; y < currCommitObjects.length; y++){
var currentSHA = currCommitObjects[y]['sha'];
listOfSHAs.push(currentSHA);
var currApiLink = "https://api.github.com/repos/elixir-lang/elixir/commits/" + currentSHA;
const response = await axios.get(currApiLink);
//console.log("up to here");
// here we start extracting some information from a single commit
var currentAuthorName = response['data']['commit']['committer']['name'];
var currentDate = response['data']['commit']['committer']['date'];
var currentFiles = response['data']['files'];
// this loop will iterate over all changed files for a single commit. Remember, every commit has a list
// of changed files, so this loop will iterate over all those files, get the necessary information
// from those files.
var z;
// we create this temporary list of file objects because for every file, we want to make an object
// that will store the necessary information for that one file. after we store all the objects for
// each file, we will add this list of file objects as a field for our bigger commit object (see down below)
var tempListOfFileObjects = [];
for (z = 0; z < currentFiles.length; z++){
var fileInConsideration = currentFiles[z];
var nameOfFile = fileInConsideration['filename'];
var numberOfAdditions = fileInConsideration['additions'];
var numberOfDeletions = fileInConsideration['deletions'];
var totalNumberOfChangesToFile = fileInConsideration['changes'];
//console.log("with file");
var tempFileObject = {fileName: nameOfFile, totalAdditions: numberOfAdditions,
totalDeletions: numberOfDeletions, numberOfChanges: totalNumberOfChangesToFile};
// we add the same file objects to both a temporary, local list and a global set. Don't be tripped
// up by this; they're doing the same thing!
setOfFileObjects.add(tempFileObject);
tempListOfFileObjects.push(tempFileObject);
}
// here we make an object that stores information for a single commit. sha, authorName, date are single
// values, but files will be a list of file objects and these file objects will store further information
// for each file.
var tempObj = {sha: currentSHA, authorName: currentAuthorName, date: currentDate, files: tempListOfFileObjects};
var currPrNumber = listOfPrObjects[x]['number'];
console.log(currPrNumber);
// here we will make a single pull request number to an object that will contain all the information for
// every single commit associated with that pull request. So for every pull request, it will map to a list
// of objects where each object stores information about a commit associated with the pull request.
mapPRNumbersToCommitObjects.set(currPrNumber, tempObj);
}
}
return mapPRNumbersToCommitObjects;
}
startParsingPullRequests() {
this.getListOfPullRequests(gitPullApiLink + "?state=all").then(() => {
this.getCommitsForEachPullRequestAndPRinformation().then(() => {
this.createCommitObjects().then((response) => {
console.log("functions were successful");
return mapPRNumbersToCommitObjects;
}).catch((error) => {
console.log("printing first error");
// console.log(error);
})
}).catch((error2) => {
console.log("printing the second error");
console.log(error2);
})
}).catch((error3) => {
console.log("printing the third error");
// console.log(error3);
});
}
//adding some getter methods so they can be used to work with whatever information people may need.
//I start all of them with the this.startParsingPullRequests() method because by calling that method it gets all
// the information for the global variables.
async getSetOfFileObjects(){
var dummyMap = this.startParsingPullRequests();
return setOfFileObjects;
}
async OpenCloseDateObjects(){
var dummyMap = this.startParsingPullRequests();
return mapPRnumberToCloseOpenDateObjects;
}
async getNumbersOfTargetedIssues(){
var dummyMap = this.startParsingPullRequests();
return listOfNumbersOfTargetedIssues;
}
}
I then try to play around and run the function to make sure all the data I need is there by doing:
var dummy = new PullRequestParser();
var dummyMap = dummy.startParsingPullRequests();
And when I run it on webstorm using:
node PullRequestParser.js
It will print out some pull request numbers, then stop about halfway with a 403 error. I know what the 403 error is, but I'm wondering if there's anything on my end to stop it from happening, or is it just a matter of working with another repo that won't throw me this error. Thanks!

The 403 error from the server means that your access is forbidden. You either need to use different credentials (that is, log in as a user with that access), not use that repository, or gracefully handle the error and do something else. Retrying will not be effective, since GitHub wouldn't be very secure if it just let you have access to things you weren't supposed to.

Not able to retrieve object properties when looping through objects in indexedDB

I am not able to retrieve properties when looping through objects stored in indexedDB. When i attempt to get objects trough tasksStore.get(i);, i get the error Cannot read property 'property' of undefined at IDBRequest.getTasks.onsuccess, however, if I change it to tasksStore.get(1);, it works fine, and gets object with id=1 x index's length.
I've tried checking the typeof of both ways, and they both return number.
//success handler on connection
request.onsuccess = function(e) {
db = request.result;
//define store index
tasksStore = tasksTx.objectStore("tasksStore");
//error handler on result of the request
db.onerror = function(e) {
console.log("ERROR " + e.target.errorCode);
}
//variable for counting objects in the index
let amountOfTasks = tasksIndex.count();
//error handler
amountOfTasks.onerror = function() {
console.log("There was an error finding the amount of tasks")
}
//success handler
amountOfTasks.onsuccess = function() {
for (var i = 1; i < amountOfTasks.result; i++) {
let getTasks = tasksStore.get(i);
let getTasksElementContainer = document.getElementById("list-tasks");
let createTasksList = document.createElement("li");
createTasksList.id = "task-" + i;
getTasks.onerror = function() {
console.log("There was an error looping through the tasks")
}
getTasks.onsuccess = function() {
console.log(getTasks.result.title); //getTasks.result works, getTasks.result.title does not.
getTasksElementContainer.appendChild(createTasksList);
//JSON stringify to return object in string format, and not [Object object]
createTasksList.innerHTML = JSON.stringify(getTasks.result.title);
}
}
}
}

When you call IDBObjectStore.get with a key that doesn't exist in your database, the resulting value is undefined. That likely explains why sometimes getTasks.result is undefined.
If you're still having trouble, you'd likely be well served by making a self-contained reproducible example. You'll probably find your own bug in the process of doing that. If not, it's easier to get more specific help on Stack Overflow if you have some code that other people can run to directly observe the problem (so including database creation and inserting data).

How to check if key is not found in IndexedDB?

On the indexeddb i want to look if there is a key permanent and do some actions. But if not, i want to make some other actions. I can do the actions if the permanent is there, however when it is not I can get the onerror to work. Is the onerror suppose to do this thing? How can I check if there is not value in it?
var hashtype = 'permanent';
var getPermanent = store.get(hashtype);
getPermanent.onsuccess = function() {
var ivrame = getPermanent.result.value;
};
getPermanent.onerror = function() {
console.log('onerror')
};

See the note under https://w3c.github.io/IndexedDB/#dom-idbobjectstore-get - the get method yields success with undefined if there is no matching record.
So you have a few options:
Use get(key) and test the result for undefined. This works unless undefined is a value you expect to store (it's a valid value)
Use count(key) - the result will be 1 if present, 0 if absent. Easy if you're just testing for existence, but doesn't get you the record.
Use openCursor(key) and test to see if the request's result is a cursor (record present as request.result.value) or undefined (no record in range)
For your code:
var hashtype='permanent';
// #1: Use get
var getPermanent = store.get(hashtype);
getPermanent.onsuccess = function() {
if (getPermanent.result === undefined) {
// no record with that key
} else {
var value = getPermanent.result;
}
};
// #2: Use count
var getPermanent = store.count(hashtype);
getPermanent.onsuccess = function() {
if (getPermanent.result === 0) {
// no record with that key
} else {
...
}
};
// #3: Use cursor
var getPermanent = store.openCursor(hashtype);
getPermanent.onsuccess = function() {
var cursor = getPermanent.result;
if (!cursor) {
// no record with that key
} else {
var value = cursor.value;
}
};

The function assigned to request.onsuccess is a callback function that is always called, regardless of whether the value is present in the store. When there is no corresponding object in the store, the result object will be undefined. When there is a corresponding object in the store, the result object will be defined. So you simply need to check if the object is defined from within the onsuccess callback function.
request.onerror is a separate callback from request.onsuccess. onerror gets called when there is some type of failure in indexedDB (e.g. something like you tried to get a value from a store that doesn't exist, or you tried to put a duplicate object into a store that doesn't permit duplicates). request.onerror does not get called when no value is found as a result of calling store.get, because that is not considered an 'error' in the failure sense.
So, what you want to do is something like this:
var hashtype='permanent';
var getPermanent = store.get(hashtype);
getPermanent.onsuccess = function(event) {
//var ivrame=getPermanent.result.value;
var result = getPermanent.result;
if(result) {
console.log('Got a result!', result);
var ivrame = result;
} else {
console.log('Result was undefined! No matching object found');
}
};
getPermanent.onerror = function() {
console.log('Something went wrong trying to perform the get request');
};
Do not try and access request.result.value. There is no such thing in the case of a get request. When using store.get, request.result contains the matching object you want, or is undefined. When using store.openCursor, request.result contains the cursor, which is defined if there is at least one matching object and you have not already iterated past it. To get the matching object at the cursor's current position, you would use cursor.value. Here, cursor.value will always be defined, because cursor would otherwise be undefined, and you would obviously check for that beforehand.

Instead of using getPermanent.result to access data provided by 'get' request it is better to use event.target.result. It also can be compared with undefined to check absence of requested key:
db = this.result;
var tr = db.transaction("data");
var objstore = tr.objectStore("data");
var getres = objstore.get(0);
getres.onsuccess = function(event)
{
if(event.target.result.data === undefined)
console.log("key not found");
}

Check for duplicate record in Chrome Storage extension before saving

I'm developing a small Chrome extension that would allow me to save some records to chrome.storage and then display them.
I've managed to make the set and get process work as I wanted (kinda), but now I'd like to add a duplicate check before saving any record, and I'm quite stuck trying to find a nice and clean solution.
That's what I came up for now:
var storage = chrome.storage.sync;
function saveRecord(record) {
var duplicate = false;
var recordName = record.name;
storage.get('records', function(data) {
var records = data.records;
console.log('im here');
for (var i = 0; i < records.length; i++) {
var Record = records[i];
if (Record.name === recordName) {
duplicate = true;
break;
} else {
console.log(record);
}
}
if (duplicate) {
console.log('this record is already there!');
} else {
arrayWithRecords.push(record);
storage.set({ bands: arrayWithRecords }, function() {
console.log('saved ' + record.name);
});
}
});
}
I'm basically iterating on the array containing the records and checking if the name property already exists. The problem is it breaks basic set and get functionality -- in fact, when saving it correctly logs 'im here' and the relative record object, but it doesn't set the value. Plus, after a while (generally after trying to list the bands with a basic storage.get function) it returns this error:
Error in response to storage.get: TypeError: Cannot read property
'name' of null
I'm guessing this is due to the async nature of the set and get and my incompetence working with it, but I can't get my head around it in order to find a better alternative. Ideas?
Thanks in advance.