We Keep Coding

How to use the Pagination in Microsoft Graph API

I would like to be able to retrieve all users in my Azure Active Directory and cache it so I could filter my users on demande. I know about the nextLink used for Pagination. I'm fairly new in the React world and Javascript so I need some help understanding how to cycle through all pages to get all my users. Here's the code I have right now :
export async function searchADUser(searchQuery?: string, filter: string, orderBy: string = 'displayName')
{
const searchedUser = await graphClient!
.api('/users')
.header('ConsistencyLevel', 'eventual')
.filter(`${filter}`)
.search(`"${searchQuery}"`)
.orderby(`${orderBy}`)
.get();
const nextLink = searchedUser["#odata.nextLink"]
return searchedUser;
I was able to access the nextLink url using the ["#odata.nextLink"]. So my question is how to get all users ? Do I just loop until nextLink is null or there is a better way.
Thank you

The default and maximum page sizes are 100 and 999 user objects
respectively
Here's the api description for page size. If user account in your tenant is more than 999, then you have no other choice but to loop the query until there's no nextLink.
If user account is less than 1000, then you may try .orderby().top(999).get();

Here's the code to be able to get all pages of data from a Microsoft Graph :
export async function getAllAADUsers(
authProvider: AuthCodeMSALBrowserAuthenticationProvider
) {
ensureClient(authProvider);
try {
let response: PageCollection = await graphClient!.api('/users').get();
// let allUsers: User[] = [];
let callback: PageIteratorCallback = (data) => {
console.log('file: GraphService.tsx ~ line 129 ~ data', data);
return true;
};
let pageIterator = new PageIterator(graphClient!, response, callback);
await pageIterator.iterate();
} catch (e) {
throw e;
}
That works wonders!
Thank you all for the help

Deserialize DeepObject Querystring Keys in Azure Functions / Javascript

We are constructing an API with Azure Functions, and the spec calls for DeepObject references in the GET request's querystring. So, the structure looks like https://example.com/api/persons?name[first]=Todd. The expectation is that some of the query keys may be DeepObject references, others will be flat.
This is a pattern that apparently Express can handle, but Azure Functions uses an ASP.NET router. The expectation is that the reference above should deserialize into req.params: { name: { first: "Todd" } }. However, instead the output looks like req.params: { "name[first]": "Todd" }.
I would really love to not have to regex/search each key, so does anyone know if:
There's a config/flag in ASP.NET to support this structure?
There's a good pattern in Javascript to deserialize this in some functional -- or at least non-idiosyncratic -- way?
Note: Anyone who suggest some use of the eval() method will not be selected. But for playing you will take home a comment with a Nineties reference, because that was the last decade the use of the that method was considered acceptable. :stuck_out_tongue_winking_eye:

For this problem, I don't think we can change some configuration to support this structure. What we can do is to implement in code by ourselves.
Here is my function code:
module.exports = async function (context, req) {
console.log("======query url is:" + req.url);
const result = queryStringToJSON(req.url);
console.log(result);
context.res = {
body: "success"
};
}
function queryStringToJSON(queryUrl) {
if(queryUrl.indexOf('?') > -1){
var queryString = queryUrl.split('?')[1];
}
var pairs = queryString.split('&');
var result = {};
pairs.forEach(function(pair) {
if (pair.indexOf('[') > -1) {
var nameObj = {};
var firstObj = {};
var nameStr = pair.substring(0, pair.indexOf('['));
var firstStr = pair.substring(pair.indexOf('[')+1, pair.indexOf(']'));
firstObj[firstStr] = pair.split('=')[1];
nameObj[nameStr] = firstObj;
Object.assign(result, nameObj);
}else {
pair = pair.split('=');
result[pair[0]] = decodeURIComponent(pair[1] || '');
}
});
return result;
}
Start the function project, I request it with http://localhost:7071/api/HttpTrigger1?name[first]=Todd&email=test#mail.com. The result shows:

After much searching, I wasn't able to find any way to natively implement this in the ASP.NET router. Though there is a great deal of suggestions on how to deserialize this structure directly in your ASP.NET controller functions, I am working in Javascript.
What was helpful was the qs library, available in NPM, which supports a number of nuances related to this query string structure.
const { query } = req;
// => { "name[first]": "Todd" };
const deserializedQuery = qs.parse(query);
// => { name: { first: "Todd" }
Equally helpful to me is that I need a way to restructure my outbound query string in this same format. Qs works with the paramsSerializer attribute in Axios.
const params = { name: { first: "Todd" };
const paramsSerializer = (params) => { return Qs.stringify(params); };
const reqOptions = { params, paramsSerializer };
axios.get("https://example.com/api/persons", reqOptions);
// => GET https://example.com/api/persons?name[first]=Todd
Thanks to #hury-shen for a completely workable solution. It just wasn't turnkey solution I was looking for.

Firefox Extension - Filter & modify HTTP POST data using onBeforeRequest

Using browser.webRequest.onBeforeRequest.addListener() I am able to filter HTTP requests and modify the source of pages before they are returned. I need to intercept HTTP POST requests and modify the data submitted by a form before sending it to the server. Check the code:
browser.webRequest.onBeforeRequest.addListener(
function(details) {
// The code in this if is just some experimentation.
if(details.method == "POST") {
let filter = browser.webRequest.filterResponseData(details.requestId)
let decoder = new TextDecoder("utf-8")
let encoder = new TextEncoder()
filter.ondata = event => {
let str = decoder.decode(event.data, {
stream: true
});
console.log(str)
}
// This is a where I attempt to modify POST data.
let formData = details.requestBody.formData;
//console.log(formData)
if(formData) {
Object.keys(formData).forEach(key => {
formData[key].forEach(value => {
//console.log(key)
if(key == 'testField'){
//console.log("found", formData[key])
details.requestBody.formData[key] = "INJECTED"
console.log('injected', details.requestBody.formData[key])
}
})
})
}
return {details: details};
}
},
{urls: ["https://example.com/*"]},
["blocking", "requestBody"]
);
The last console.log prints the modified value but returning details does not forward the modified value to the server. How can I use onBeforeRequest (Or any other method) in a Firefox extension to modify POST variables before they are sent to the server?
I do not want to inject code into the web page. I want to filter the traffic.

This seems to be a missing feature.
Relevant bugs:
https://bugzilla.mozilla.org/show_bug.cgi?id=1376155
https://bugs.chromium.org/p/chromium/issues/detail?id=91191

How to pass data between pageFunction executions in Apify web

I'm scraping website with Apify. I want to scrape different types of pages and then combine the data into one data set. Now i have different sets of data for each kind of pages (users, shots). How to transfer data between pageFunction executions, ex. to calculate followers number for each shot author.
async function pageFunction(context) {
const { request, log, jQuery } = context;
const $ = jQuery;
if (request.url.indexOf('/shots/') > 0) {
const title = $('.shot-title').text();
return {
url: request.url,
title
};
} else if (request.userData.label === "USER") {
var followers_count = $('.followers .count').first().text();
return {
url: request.url,
followers_count
};
}
}

If I understand the question correctly, you can pass the data through crawled pages and save only one item in the end. For this use case, you can work with userData, which you can pass with every request.
For example, if you would like to pass the data from /shots site to the USER, you could do it like this. (but it requires you to enqueue pages manually to control the flow of the data, also this approach except that the /shots type of the page is the first one you visit and then continue)
async function pageFunction(context) {
const { request, log, jQuery } = context;
const $ = jQuery;
if (request.url.indexOf('/shots/') > 0) {
const title = $('.shot-title').text();
const userLink = 'some valid url to user page'
//add to the queue your request with the title in the userData
await context.enqueueRequest({
url: userLink,
userData:{
label:'USER',
shotsTitle: title
}
})
} else if (request.userData.label === "USER") {
var followers_count = $('.followers .count').first().text();
//here you need to get the shotsTitle and return it
return {
url: request.url,
followers_count,
shotsTitle: request.userData.shotsTitle
};
}
}
If you would need to share the between runs of the actors, that is other topic, let me know if it helped.
Also would recommend going through the getting started guide which is here.

is it possible to pass data in fetch options?

I have a react client-side app.
I want to wrap it with express so I won't have CORS issues when I'm fetching data from API's.
right now all my fetches happens on the client-side with a some endpoints urls.
correct me please if I'm wrong.
In order to re-write correctly my fetch requests, I fetch from a route endpoint which I'll define in my server.js right ?
and in my server.js I'll make the actual fetch from the API, right ?
If so, then I have another question:
in my client-side I have some logics will determine how the fetch's url will be.
My Question: is there's a way for me to pass the correct url as some kind of a data into my fetch request, so I could use it in my server-side to make the actual API fetching?
here is an example of my code and what i try to understand if possible:
const { current, amount, page} = this.pagination;
const { client_id, search } = this.API;
const limit = `_limit=${amount}`;
let currentPage;
let url;
if(this.isFetching) return;
if(direction == 'next'){
this.pagination.current = current + amount;
currentPage = page + 1
let offset = `_offset=${this.pagination.current}`;
} else {
if(page !== 1) {
this.pagination.current = current - amount;
currentPage = page - 1
let offset = `_offset=${this.pagination.current}`;
}
url = `${search}${client_id}&${limit}&${offset}`; // Here is the final url
fetch('/api', {data: url}) // PSUEDO use of fetch, is it possible?
.then(response => response.json())
.then(data => {
this.fetchedProperties = [...data.search_results];
this.pagination.page = currentPage;
this.isFetching = false;
});
}

The normal way to pass data from a client to an HTTP endpoint is to encode it in a query string.
http://example.com/fetch?url=http%3A%2F%2Fwww.example.net%2Ffoo%2Fbar%3Fquery%3Dsomething%26etc%3Detc