Context
I'm using an XMLHttpRequest to request a resource from the GitHub API. Since the response is paginated, I want to use the Link header that is provided.
When logging request.getResponseHeader("link") in the response callback, I get either null (if not paginated) or something like:
<https://api.github.com/repositories/14173222/forks?sort=stargazers&per_page=100&page=2>; rel="next", <https://api.github.com/repositories/14173222/forks?sort=stargazers&per_page=100&page=4>; rel="last"
Question
Is there an easy way to navigate this returned String without having to parse it?
Something along the lines of request.getResponseHeader("link").get("next") and which would return https://api.github.com/repositories/14173222/forks?sort=stargazers&per_page=100&page=2
.
My actual goal is to extract the next page's number (here, for example, I'd like a function which returns 2 since it is the next page to be requested).
I won't mark this as an accepted answer since it doesn't completely answer the question, but this is nonetheless what I ended up doing:
/** Paginated request. Pages index start at 1. */
function request_fork_page(page_number, user, repo, token) {
// ...
/* Pagination (beyond 100 forks). */
const link_header = request.getResponseHeader("link");
if (link_header) {
let contains_next_page = link_header.indexOf('>; rel="next"');
if (contains_next_page !== -1) {
request_fork_page(++page_number, user, repo);
}
}
}
Related
I am making a get request with additional params options, since I am using that request on a filter, so the params are filters for what to get back:
const res = await axios.get("http://localhots:3000/getsomedata", {
params: {
firstFilter: someObject,
secondFilter: [someOtherObject, someOtherObject]
}
});
The request goes through just fine, on the other end, when I console.log(req.query); I see the following:
{
firstFilter: 'someObject',
'secondFilter[]': ['{someOtherObject}', '{someOtherObject}'],
}
If I do req.query.firstFilter that works just fine, but req.query.secondFilter does not work and in order for me to get the data, I have to do it with req.query["secondFilter[]"], is there a way to avoid this and be able to get my array of data with req.query.secondFilter?
My workaround for now is to do:
const filter = {
firstFilter: req.query.firstFilter,
secondFilter: req.query["secondFilter[]"]
}
And it works of course, but I don't like it, I am for sure missing something.
Some tools for parsing query strings expect arrays of data to be encoded as array_name=1&array_name=2.
This could be a problem if you have one or more items because it might be an array or might be a string.
To avoid that problem PHP required arrays of data to be encoded as array_name[]=1&array_name[]=2 and would discard all but the last item if you left the [] out (so you'd always get a string).
A lot of client libraries that generated data for submission over HTTP decided to do so in a way that was compatible with PHP (largely because PHP was and is very common).
So you need to either:
Change the backend to be able to parse PHP style
Change your call to axios so it doesn't generate PHP style
Backend
The specifics depend what backend you are using, but it looks like you might be using Express.js.
See the settings.
You can turn on Extended (PHP-style) query parsing by setting it to "extended" (although that is the default)
const app = express()
app.set("query parser", "extended");
Frontend
The axios documentation says:
// `paramsSerializer` is an optional function in charge of serializing `params`
// (e.g. https://www.npmjs.com/package/qs, http://api.jquery.com/jquery.param/)
paramsSerializer: function (params) {
return Qs.stringify(params, {arrayFormat: 'brackets'})
},
So you can override that
const res = await axios.get("http://localhots:3000/getsomedata", {
params: {
firstFilter: someObject,
secondFilter: [someOtherObject, someOtherObject]
},
paramsSerializer: (params) => Qs.stringify(params, {arrayFormat: 'repeat'})
});
My example requires the qs module
This has to do with params not being serialized correctly for HTTP GET method. Remember that GET has no "body" params similar to POST, it is a text URL.
For more information I refer to this answer, which provides more detailed info with code snippets.
I'm fetching data from a PS4 Games API but it's split into 400+ pages. I wanted to get the data from all pages, but the solution I came up with did not work very well. It gives me an error 'JSON Value of type NSNull cannot be converted to a valid URL'. Also, I don't think the for loop works well either, it shows me it going through all the pages when it displays the results in my list.
Additionally, this API is dynamic because new games keep getting released. So how could I get data up to latest page without manually changing my last page number everytime? I looked at some questions here but I couldn't fit it into my code
My code is rather long so I'm just going to post the part that matter:
componentDidMount() {
var i;
for (i = 0; i < 400; i++) {
fetch(`https://api.rawg.io/api/games?page=${i+1}&platforms=18`, {
"method": "GET",
"headers": {
"x-rapidapi-host": "rawg-video-games-database.p.rapidapi.com",
"x-rapidapi-key": "495a18eab9msh50938d62f12fc40p1a3b83jsnac8ffeb4469f"
}
})
.then(res => res.json())
.then(json => {
const { results: games } = json;
this.setState({ games });
//setting the data in the games state
});
}
}
The API also has an item that gives me the link of the next page, I think there is a way to use 'next' and fetch data from that URL
If anyone could help, that would be AWESOME. Thank you in advance
its my first answer on this forum, so... I hope be helpfull.
For me you have 2 options:
Each request send 2 variables in the answer count and next, or you make a for loop with count/20 as limit (20 is the number of items gaves in the answer), or you make a while loop until the next variable give null as answer (currently at page 249).
What is currently happening is you are making 400 requests and when each comes in it is overwriting the component with the response it received. It does not care about what is already there or any of the other requests.
An approach you could try instead is as the responses come in append the results to the ongoing list and update the state with the running list.
Going forward and for your other question about handling new releases. Instead of running 400 queries every time the application is used, try looking into caching the results. When the app loads you can see if cache exists and load or query if it does not. The rawg.io /games endpoint has a parameter for ordering by release. When the application loads in future you can conditionally loop until you reach a game that is already in cache at which point terminate.
I'm trying to fix one mistake which one of the previous developer has did. Currently in project we have half-bookmarkable pages. Why half? Because if token has expired user will be redirect to resourse where he has to provide his credentials and on success he will be redirect to previous resource which he has used before. Sounds good for now. The problem here is next, some of resources have server side filtering and highly complicated logic which comes in the end to the one object like this:
param = {
limit: 10,
offset: 0,
orderBy: 'value',
query: 'string query can include 10 more filters'
};
then thru the service it sends a request to the end point
var custom = {};
function data(param) {
custom.params = param;
return $http.get('/data_endpoint', custom)
.then(function (response) {
return response.data;
});
From this part request works fine and response is correct but url isn't. User cannot store current url with search params because url doesn't have it and if user will try to get back to previous page with applied filters he won't be able to do that.
Also there is interceptor in this application, which add one more query param to every api request, because every api request requires some specific validation. This param is always the same.
I have tried to add $location.search(param) exactly to this data service but it breaks the app with infinity loop of login / logout looping, because interceptor stops working and all other request are sending without special query param
Then I thought to add this params inside interceptor like this:
config.params.hasOwnProperty('query') ? $location.search(config.params) : $location.search();
But it still breaks app to infinity loop.
This method adds query to url but doesn't allow to modify it, so if user applied additional filtering, it doesn't send new modified request but sends old one.
if (config.params.hasOwnProperty('query')){
for (var key in config.params){
$location.search(key, config.params[key])
}
}
That's why I decided to ask question here, probably somebody gives an advice how to solve this problem. Thanks in advance.
This is one of them Youtube internal decisions that is highly counter-productive. Correct me if I'm wrong, but they currently don't want to expose the channel name of private / deleted videos. They even call it a 'bug' if it happens ( http://code.google.com/p/gdata-issues/issues/detail?id=5893 ). So what happens if I solely request video IDs that were public beforehand but then later became private or deleted? I will get, via usage of Youube Data API v3, a "response is undefined" error message in console. WHat happens when that message happens? My code breaks!
This is the code I currently use:
function DisplayThemVideos(yeah) {
var yeah = $("#ThoseMissingIDs").text();
var vidrequestOptions = {
id: yeah,
part: 'snippet',
fields: 'items(id),items(snippet(channelId)),items(snippet(channelTitle)),
items(snippet(title)),items(snippet(thumbnails(default)))'
};
var vidrequest = gapi.client.youtube.videos.list(vidrequestOptions);
vidrequest.execute(function(response) {
var videoIdItems = response.result.items;
if (videoIdItems) { // If results
displayResults(videoIdItems);
} else { // if NO results
alert('Sawwy, YouLose, thx to Youtube!'); // This alert never fires!
}
});
}
Now the undefined "response" is actually an empty request sent back to my app from Youtube server. Youtube answers back with an empty "item" tag which obviously doesn't help much for the displayResults(videoIdItems) function which gets fired without any item to display! They should at least let the channel name and channel ID filter through so a User could click on a link in order to access the remaining public videos of that channel (wouldn't that be productive Jeff P?).
SO my dilemma is to get the else section working like so:
displayResults(videoIdItems);
} else { // if NO results or if results return EMPTY or response "undefined"
alert('Sorry pal but these IDs ___________ are currently missing.
Click the Channel link to access the public videos of that channel.');
}
The else part works as it should for similar API calls as demonstrated in Youtube Data API v.3 sample code, but I guess it currently isn't able to handle empty requests.
So what am I to do? WIll I have to use an ajax call with success fail error handling? Like I said beforehand, the API returns an empty request so the response is legit but the response content comes empty for private/deleted videos, hence, "undefined" with code breaking along the wway.
Any hints leading to a working solution would help! Thx for guidance.
I can only partly answer my question. Easiest is to use Catch of Try - Catch method which detects everything except syntax errors, helpful in detecting an empty response (undefined) and activating your function that can access the Youtube player for displaying current availability status of the missing ID:
...
var vidrequest = gapi.client.youtube.videos.list(vidrequestOptions);
vidrequest.execute(function(response) {
try {
var videoIdItems = response.result.items;
if (videoIdItems) { // If results
displayResults(videoIdItems);
return false;
}
} catch (e) {
// put code here that handles errors such as empty or undefined response
// which otherwise breaks your code
} finally {
// optionally, put code here that will always trigger
}
});
Hope this helps others.
I am trying to read the response headers 'name' and 'value'. The end goal is to compare them to some pre-set name and a value to see if they match.
Here is what I have so far, it's the function that run every time I get a response header.
var observer = require("observer-service");
observer.add("http-on-examine-response", onHttpRequest);
function onHttpRequest(subject, data)
{
console.log("request subject...." + subject);
console.log("request data...." + data);
}
The output is as follows:
request subject....[xpconnect wrapped nsISupports]
request data....null
I was hoping to know how to get the rest of the data out of the response.
Any help would be great, thanks.
The subject for http-on-examime-response implements nsIHttpChannel, among some other things. You may use .QueryInterface() or instanceof (which internally kinda uses QueryInteface, so that this works as well) to get to that interface.
const {Ci} = require("chrome");
if (subject instanceof Ci.nsIHttpChannel) {
console.log("content-type", subject.getResponseHeader("content-type"));
subject.visitResponseHeaders(function(header, value) {
console.log(header, value);
});
}
There are a couple of other questions around here going into more detail on how to use these notifications... Also, mxr can help a lot checkout out what interfaces there are, how it fits together and how one could use it (in particular the existing tests are great to see some uses for all kinds of stuff).
There is also the "nsITraceableChannel, Intercept HTTP Traffic" article going into more details, e.g. on how to use nsITraceableChannel to get the payload data from such a channel.