Good day all,
I'm working on extracting some data out of PipeDrive's API using Axios for Node.js. The way that PipeDrive developed their API pagination is a bit different. Here is how their pagination indicator looks:
"additional_data": {
"pagination": {
"start": 0,
"limit": 100,
"more_items_in_collection": true,
"next_start": 100
}
}
I need to interate through all pages to extract the data, and my code has successfully done that, but I cannot get my promise to resolve for some reason.
My logic in the code is as follows:
if(more_items_in_collection){Add current page's data then re-run the same function with the next_start as a parameter}
else{Add current page's data then RESOLVE the promise to complete the function}
But this resolution never happens, even though my code works (strange).
Gurus, can you please take a look at my code and let me know why you think it won't resolve (function().then((result) => {OUTPUT}) never returns happens)?
Thanks as always!
const queryPipeDrive = (start) =>{
// Query is a full then-able Async function
return new Promise((resolve, reject) => {
// API CALL
axios({
method: 'GET',
url: pipeDriveURI,
params: {
api_token: apiKey,
start: start
}
})
// THEN DO THIS
.then((response) => {
// IF there are more items in collection on additional pages, iterate through the pages
if(response.data.additional_data.pagination.more_items_in_collection){
// Add all deals on page to the container
for (const deal of response.data.data) {
db.get('deals').push(deal) // Push deal data to the StormDB File
}
console.log(chalk.cyan(`${response.data.additional_data.pagination.next_start}`))
// Function loop created. We will loop UNTIL the 'more_items_in_collection' prop is false, then we'll resolve the promise.
queryPipeDrive(response.data.additional_data.pagination.next_start)
}else{
// Add all deals on this page to the reponse container
for (const deal of response.data.data) {
db.get('deals').push(deal)
}
db.save() // Save changes to temp DB
resolve(response.data.data) // Resolve Promise with the data from the successful call
}
})
.catch((err) => {
console.log(chalk.red(err))
reject(err)
})
})
}
Your more_items_in_collection case never resolves the promise. It just creates a new one, then does nothing with it.
Additionally, you're making your code more complicated than it needs to be by using new Promise. Axios already returns a promise, so there's no need to explicitly create a new one. Calling .then will create a new promise automatically, which resolves to whatever value you return in the callback.
const queryPipeDrive = (start) => {
// API CALL
return axios({
method: "GET",
url: pipeDriveURI,
params: {
api_token: apiKey,
start: start,
},
})
// THEN DO THIS
.then((response) => {
// IF there are more items in collection on additional pages, iterate through the pages
if (response.data.additional_data.pagination.more_items_in_collection) {
// Add all deals on page to the container
for (const deal of response.data.data) {
db.get("deals").push(deal); // Push deal data to the StormDB File
}
console.log(
chalk.cyan(`${response.data.additional_data.pagination.next_start}`)
);
// Function loop created. We will loop UNTIL the 'more_items_in_collection' prop is false, then we'll resolve the promise.
return queryPipeDrive(
response.data.additional_data.pagination.next_start
);
} else {
// Add all deals on this page to the reponse container
for (const deal of response.data.data) {
db.get("deals").push(deal);
}
db.save(); // Save changes to temp DB
return response.data.data;
}
})
.catch((err) => {
console.log(chalk.red(err));
throw err;
});
};
Besides the accepted answer.
Would you consider using this async function in await? In this way, you call the main().
const main = async start => {
const res = await queryPipeDrive(start);
if (res.isMoreItems === true) {
await main(res.nextStart);
}
};
async function queryPipeDrive(start) {
const response = await axios({
method: "GET",
url: pipeDriveURI,
params: {
api_token: apiKey,
start: start,
},
});
for (const deal of response.data.data) {
db.get("deals").push(deal);
}
if (response.data.additional_data.pagination.more_items_in_collection) {
console.log(
chalk.cyan(`${response.data.additional_data.pagination.next_start}`)
);
return {
isMoreItems: true,
nextStart: response.data.additional_data.pagination.next_start,
};
} else {
db.save(); // Save changes to temp DB
return {
isMoreItems: false,
};
}
}
Related
Using fetch API and async/await, is it possible to continue polling indefinitely, regardless of availability of a URL? I anticipate that a URL might become available eventually, so I want to keep trying until a condition is met. Tried to come up with a minimum viable code sample and I'm not sure I pulled it off:
// this is just a placeholder. It will eventually be a function
// that evaluates something real.
// Assume validContinue gets updated elsewhere.
function shouldContinue() {
return validContinue;
}
async function wonderPoll(someUrl) {
// just a delay mechanism
function wait(ms = 1000) {
return new Promise(resolve => {
setTimeout(resolve, ms);
});
}
// the actual individual poll
async function pollingFunction(url) {
const response = await fetch(url, {
cache: 'no-store'
});
if (response.ok) {
return response;
} else {
Promise.reject(response);
}
}
// allegedly keep polling until condition is met.
// But the rejected Promise is breaking out!
while (shouldContinue()) {
await wait();
result = await pollingFunction(someUrl);
}
// when the fetch hits a rejected state, we never get here!
console.log('done with the while loop, returning last successful result')
return result;
}
const sampleUrl = 'https://get.geojs.io/v1/ip/country.json?ip=8.8.8.8';
const sampleUrl2 = 'http://totallybroken_fo_sho';
// swap the URL to test
wonderPoll(sampleUrl)
.then((result) => {
console.log('got a result', result)
})
.catch((err) => {
console.log('got an error', err)
});
I see what's happening (I think). The parent call ultimately executes the polling function, which rejects on the Promise. The condition to continue is still theoretically met, but the rejection breaks out of the While loop and sends to rejection directly up. This propagates all the way up to the catch method of the original/initial Promise. It doesn't even hit any code that would have come after the While loop in the case of resolved Promises.
What I don't know is how to prevent that from happening. I think I don't understand the syntax for intercepting and resolving the promise. When I replace Promise.reject in the response parser with Promise.resolve(response), it still ends up rejecting up to the top.
If the URL I provide is valid, it will continue until the condition is no longer met.
Here's a fiddle: https://jsfiddle.net/gregpettit/qf495bjm/5/
To use the fiddle, the "stop" button simulates the condition being met, and I've provided two different URLs that have to be manually swapped (by passing someUrl or someUrl2) to test.
Expected results:
with good URL, continuous polling (will have to dig into network in dev tools) until condition is met (by pressing Stop!) and then the calling function's 'then' can show the result.
with bad URL, continuous polling until condition is met, and then calling function's 'catch' shows the error
Actual results:
positive test case is OK
negative test case goes directly to the catch
You can try…catch it to prevent breaking out of loop.
while (shouldContinue()) {
try {
await wait();
result = await pollingFunction(someUrl);
} catch (e) {}
}
Change the code in while loop to try/catch so you can catch the error
result can hold a value when there's no error, or a reason when there is an error
Once the loop is stopped, you either return the value, or throw with the reason
As below
async function wonderPoll(someUrl) {
// just a delay mechanism
function wait(ms = 1000) {
return new Promise(resolve => {
setTimeout(resolve, ms);
});
}
// the actual individual poll
async function pollingFunction(url) {
const response = await fetch(url, {
cache: 'no-store'
});
if (response.ok) {
return response;
} else {
Promise.reject(response);
}
}
// allegedly keep polling until condition is met. But the rejected Promise is breaking out!
while (shouldContinue()) {
try {
await wait();
const value = await pollingFunction(someUrl);
result = {value};
} catch (reason) {
result = {reason};
}
}
// when the fetch hits a rejected state, we never get here!
console.log('done with the while loop, returning last successful result')
if (result.reason) {
throw result.reason;
}
return result.value;
}
Running example https://jsfiddle.net/twkbo9pg/
the example includes status in the result, but that is unnecessary (I borrowed code from my Promise.allSettled polyfill and forgot to remove that property)
you might want to check out observable streams! If you're going to have a lot of data coming in over time, that's rxjs's whole thing.
There's actually a few ways to do this if this feels janky (it kinda does haha).
import { ajax } from "rxjs/ajax";
import { duration } from "moment-timezone"; // I copied this from some old code... whatever.
import { catchError, map, share, switchMap } from "rxjs/operators";
const baseUrl = "http://foo.bar"
const base = (method, headers = {}) => ({
method,
headers: {
Accept: "application/json",
...headers,
},
crossDomain: true,
withCredentials: true,
})
const ajaxGet = url => ajax({ ...base("GET"), url })
export const userEpic = timer(0, duration(5, "minutes").asMilliseconds()).pipe(
switchMap(() =>
ajaxGet(`${baseUrl}/users`).pipe(
map(({ response }) => getUsersSuccess(response)),
catchError(e => of(getUsersError(e))),
)
),
share()
)
Two things
} else {
Promise.reject(response);
}
should return that. It's working "by accident" right now.
} else {
return Promise.reject(response);
}
Secondly, result = await pollingFunction(someUrl); might want to add .catch to it:
result = await pollingFunction(someUrl).catch(_=>null); or whatever can be tested for in the enclosing while
But I think you can simplify the whole thing thus:
export async function wonderPoll(someUrl) {
while (shouldContinue()) {
await wait();
const response = await fetch(someUrl, { cache: 'no-store' });
if (response.ok)
return response;
}
return Promise.reject(); // only if !shouldContinue()
}
I'm writing a React application that fetches image data from a server for an array of URLs. I am storing the camera images as large strings that are placed into the image's src attribute. I am using useReducer to store my dictionary of camera objects.
I am having a couple of problems getting the reducer to work, and one of them has to do with some confusion I'm having with asynchronous values and why the async function returns correct output but the completion handler (.then()) receives undefined as a result.
Here is the code for useEffect() and the asynchronous fetching function.
useEffect()
//Why is cameras undefined?
useEffect(() => {
if (phase === 0) {
let cameras = {}
getCameraInformation().then((cameras) => {
debugger;
dispatch({
type: 'loadedCameraInformation',
payload: {cameras: cameras}
});
}).finally(() => setPhase(1))
}
});
My function signature and variables:
export default function Main() {
const [state, dispatch] = useReducer(cameraReducer, initialState);
let [phase, setPhase] = useState(0);
My function for getCameraInformation:
This returns a dictionary full of correct information!
async function getCameraInformation() {
//returns a json with the following: url, cam_name, cam_pass, cam_user, channel, chunk, group, path, port,
// uptime, username.
let cam_json = await axios
.get(getCamerasURL, { headers: { auth: get_cookie("token") } })
.then(response => {
let tempCameraArray = response.data.body;
let tempCameraDictionary = {};
for (var camera in tempCameraArray) {
tempCameraDictionary[tempCameraArray[camera].sid] = {
cameraInformation: tempCameraArray[camera],
cameraImage: null
};
}
return tempCameraDictionary;
})
.catch(error => console.log(error));
}
Your async function getCameraInformation doesn't have a return statement, so its promise will not resolve any value. There is a return in the then callback, but that's a different function entirely.
You are also using await and then() on the same promise, which isn't ideal. Use one or the other, because it's very easy to get confused when you mix and match here.
You already have an async, so don't use then at all in side that function.
async function getCameraInformation() {
//returns a json with the following: url, cam_name, cam_pass, cam_user, channel, chunk, group, path, port,
// uptime, username.
let response = await axios.get(getCamerasURL, { headers: { auth: get_cookie('token') } })
let tempCameraArray = response.data.body
let tempCameraDictionary = {}
for (var camera in tempCameraArray) {
tempCameraDictionary[tempCameraArray[camera].sid] = {
cameraInformation: tempCameraArray[camera],
cameraImage: null,
}
}
return tempCameraDictionary
}
I'm quite a newbie in JavaScript and in Promises.
I'm trying to build an array of objects that I get from an API.
To do so, I've build two functions in a file MyFile.js.
The first one returns a promise when an axios promise is resolved. It's
function get_items (url) {
return new Promise((resolve, reject) => {
let options = {
baseURL: url,
method: 'get'
}
axios(options)
.then(response => {
resolve(response.data)
})
.catch(error => {
reject(error.stack)
})
})
}
The second one looks like this:
let output = []
let next_url = 'https://some_url.com/api/data'
async function get_data () {
try {
let promise = new Promise((resolve, reject) => {
if (next_url) {
get_items(next_url)
.then(response => {
output.push(...response.results)
if (response.next) {
next_url = response.next
console.log('NEXT_URL HERE', next_url)
get_data()
} else {
console.log('else')
next_url = false
get_data()
}
})
.catch(error => {
reject(error.stack)
})
} else {
console.log('before resolve')
resolve(output)
}
})
return await promise
} catch(e) {
console.log(e)
}
}
It's where I'm grinding my teeth.
What I think I understand of this function, is that:
it's returning the value of a promise (that's what I understand return await promise is doing)
it's a recursive function. So, if there is a next_url, the function continues on. But if there is not, it gets called one last time to go into the else part where it resolves the array output which contains the results (values not state) of all the promises. At least, when I execute it, and check for my sanity checks with the console.log I wrote, it works.
So, output is filled with data and that's great.
But, when I call this function from another file MyOtherFile.js, like this:
final_output = []
MyFile.get_data()
.then(result => {
console.log('getting data')
final_output.push(...result)
})
it never gets into the then part. And when I console.log MyFile.get_data(), it's a pending promise.
So, what I would like to do, is be able to make get_data() wait for all the promises result (without using Promise.all(), to have calls in serie, not in parallel, that would be great for performances, I guess?) and then be able to retrieve that response in the then part when calling this function from anywhere else.
Keep in mind that I'm really a newbie in promises and JavaScript in general (I'm more of a Python guy).
Let me know if my question isn't clear enough.
I've been scratching my head for two days now and it feels like I'm running in circle.
Thanks for being an awesome community!
This is a bit untested
const api_url = 'https://some_url.com/api/data';
get_data(api_url).then((results) => {
console.log(results);
}).catch((error) => {
// console.error(error);
});
function get_items (url) {
const options = {
baseURL: url,
method: 'get'
};
return axios(options).then((response) => response.data);
}
async function get_data(next_url) {
const output = [];
while (next_url) {
const { results, next } = await get_items(next_url);
output.push(...results);
next_url = next;
}
return output;
}
Basically it makes things a bit neater. I suggest to look at more examples with Promises and the advantage and when to ease await/async. One thing to keep in mind, if you return a Promise, it will follow the entire then chain, and it will always return a Promise with a value of the last then.. if that makes sense :)
There are a few problems. One is that you never resolve the initial Promise unless the else block is entered. Another is that you should return the recursive get_data call every time, so that it can be properly chained with the initial Promise. You may also consider avoiding the explicit promise construction antipattern - get_items already returns a Promise, so there's no need to construct another one (same for the inside of get_items, axios calls return Promises too).
You might consider a plain while loop, reassigning the next_url string until it's falsey:
function get_items (baseURL) {
const options = {
baseURL: url,
method: 'get'
}
// return the axios call, handle errors in the consumer instead:
return axios(options)
.then(res => res.data)
}
async function get_data() {
const output = []
let next_url = 'https://some_url.com/api/data'
try {
while (next_url) {
const response = await get_items(next_url);
output.push(...response.results)
next_url = response.next;
}
} catch (e) {
// handle errors *here*, perhaps
console.log(e)
}
return output;
}
Note that .catch will result in a Promise being converted from a rejected Promise to a resolved one - you don't want to .catch everywhere, because that will make it difficult for the caller to detect errors.
Another way of doing it is to not use async at all and just recursively return a promise:
const getItems = (url) =>
axios({
baseURL: url,
method: 'get',
}).then((response) => response.data);
const getData = (initialUrl) => {
const recur = (result, nextUrl) =>
!nextUrl
? Promise.resolve(result)
: getItems(nextUrl).then((data) =>
recur(result.concat([data.results]), data.next),
);
return recur([],initialUrl)
.catch(e=>Promise.reject(e.stack));//reject with error stack
};
As CertainPerformance noted; you don't need to catch at every level, if you want getData to reject with error.stack you only need to catch it once.
However; if you had 100 next urls and 99 of them were fine but only the last one failed would you like to reject in a way that keeps the results so far so you can try again?
If you do then the code could look something like this:
const getData = (initialUrl) => {
const recur = (result, nextUrl) =>
!nextUrl
? Promise.resolve(result)
: getItems(nextUrl)
.catch(e=>Promise.reject([e,result]))//reject with error and result so far
.then((data) =>
recur(result.concat([data.results]), data.next),
);
return recur([],initialUrl);//do not catch here, just let it reject with error and result
};
I've been trying to figure out what the proper way would be to write a promise for this function. I have an asynchronous function that makes an HTTP request to the server to retrieve a response, "documents_fileUploader." I am mapping through the "url" of each item within the response, and each url will go in to a function that makes another HTTP request and then sets the state. I want to fire the "upload()" function only after everything within the "documents_fileUploader()" function is complete. I tried doing this without a promise and it went straight to my "upload()" function because request was still pending. Any suggestions on how to go about this?
documents_fileUploader(formData).then(resp => {
resp.data.items.map(url => {
const key = url.split("/")[4];
this.setState({
urls: [...this.state.urls, url],
keys: [...this.state.keys, key]
});
this.getFileObject(key);
})
}).then(() => {
this.upload();
})
getFileObject = file => {
file_view(file).then(resp => {
this.setState({
mimeTypes: [...this.state.mimeTypes, resp.data.item.headers.contentType]
})
}).catch(err => {
console.log(err);
})
}
To your main question, you can wait for every promise that your .map call returns by using the Promise.all method.
Second, in order for that to work, your getFileObject function must return the promise it creates.
So incorporating those two changes, your snippet might look like:
documents_fileUploader(formData).then(resp => {
return Promise.all(resp.data.items.map(url => { // Wrap in Promise.all and return it
const key = url.split("/")[4];
this.setState({
urls: [...this.state.urls, url],
keys: [...this.state.keys, key]
});
return this.getFileObject(key); // Make sure to return this promise as well.
}));
}).then(() => {
// Now this won't happen until every `getFileObject` promise has resolved...
this.upload();
})
getFileObject = file => {
return file_view(file).then(resp => { // And return the promise here.
this.setState({
mimeTypes: [...this.state.mimeTypes, resp.data.item.headers.contentType]
})
}).catch(err => {
console.log(err);
})
}
I am trying to refactor my AngularJS code by having my service resolve a promise call that will automatically bind an object to to my scope.
// converting this:
imageService.getImages(userId)
.then(images => {
$scope.images = images;
});
// into this:
$scope.images = imageService.getImages(userId);
My current solution I am trying to refactor is:
function getImages(userId) {
const config = {
method: 'get',
url: '/a/images',
params: { userId: userId }
};
return $http(config).then(onGetImagesSuccess, onGetImagesError);
}
function onGetImagesSuccess(response) {
$log.info('Get image success', response);
return response.data;
}
function onGetImagesError(response) {
$log.info('Get image error', response);
}
I started playing around with $q.defer() and I got my getImages() to this following state. The problem is that I imageService.getImages() returns a Promise instead of the images. It is interesting that the list of images do get logged tho.
function getImages(userId) {
const deferred = $q.defer();
const config = {
method: 'get',
url: '/a/images',
params: { userId: userId }
};
let call = $http(config).then(onGetImagesSuccess, onGetImagesError);
deferred.resolve(call);
return deferred.promise.then(images => {
$log.info(images); // List of images does get logged
return images; // Promise gets returned instead of the actual images
});
}
I am wondering what do I need to do to refactor my code to get my imageService.getImages() to return a list of images instead of a Promise.
This can be done by same recipe that is used in $resource. Empty object (array) is returned which is filled with data asynchronously. A promise should be returned as well, in case there's a need to know when and how a request finished:
var result = {
data: [],
promise: $http(...).then(response => {
return Object.assign(result.data, response.data);
})
};
return result;
Then imageService.data array is bound in view.
This pattern has its pros and cons. It is natural to have then in controller to unwrap promises.