I have spreadsheets with data that needs to be uploaded via an API. After extracting the data from the spreadsheet, I create an object and put it in an array.
My idea was to iterate over this array and make a request for every single object in there and use the status code to display a success or error message.
The code seems to work fine with small quantities (~ 200 elements) but when I try to stress it (lets say 3500 elements), the line response.statusCode breaks because response is undefined. I thought I wouldn't have any problems with the async part because the callback would be called only when the server responded, but it seems not to be the case here.
Why am I getting response undefined in a callback?
Here's a piece of code i'm using:
array.forEach(function (element) {
//Configure Request
var options = {
url: 'http://example',
method: 'POST',
headers: headers,
body: element,
json: true
}
request(options, function (error, response, body) {
if (!error && response.statusCode == 200) {
console.log(response.statusCode);
console.log(body)
} else {
console.log(response.statusCode);
console.log(error)
}
});
});
Thanks!
Please run this code in async.mapLimit()
https://caolan.github.io/async/docs.html#mapLimit
Related
I am sending a fetch request with node-fetch to the following url: http://fantasy.premierleague.com/api/bootstrap-static/ in order to get back some JSON-data. Accessing the URL in the browser, or sending a get-request with postman both returns the expected JSON data.
However, when i send the request from node, I get back an object that I do not know how to extract the data from (pics below).
I am not very experienced with node but I have made successful API calls before. Usually parsing the response with response.json() or JSON.parse(response) or response.body or response.toString() or some combinations of those have worked for me. I am half familiar with buffers and streams, but not confident and the solution might be related to those, however I cannot seem to figure it out.
I get som different errors and objects depending on what I try. I have tried using fetch and just plain http requests from node.
This call:
Returns this:
If I do JSON.parse(response) i get the following error:
Response.body looks like this:
Fetch returns a response stream as mentioned here in the answer to a similar question
You can read data in chunks and add the chunk to array and then do whatever you need to do with that data. A simpler approach would be to use npm request package. Here's an example.
const request = require('request');
let options = {json: true};
const url = 'http://fantasy.premierleague.com/api/bootstrap-static/'
request(url, options, (error, res, body) => {
if (error) {
return console.log(error)
};
if (!error && res.statusCode == 200) {
console.log(body);
// do something with JSON, using the 'body' variable
};
});
I have created a get web api that i want to hit at my server startup for some specific task. I have used this method
var request = require('request');
request('http://localhost:3000/api/jobs/asd', function (error, response, body) {
if (!error && response.statusCode == 200) {
console.log(body) // Print the google web page.
}
}
)
But i am getting status code as 403. So i am not able to access my API at serve startup. Please suggest me the proper solution for the same.
thanks in adv.
You can place your code inside http callback like so :
http.createServer(app).listen(3000, () => {
//your code here
});
I am making a POST request in one of my routes add-users. I have created an array called success. In each loop provided the API POST runs successfully I am adding a string 'user added' to the array. Once the array has completed I want to send the response to the browser with the success array.
I have noticed something strange. When I type in the url the start of add-users it runs the loop before I hit enter to navigate to the page. This seems strange? Is node listening and predicting which url I am going to hit?
Here is my current attempt but its not working for some reason.
app.get('/add-users', function (req, res) {
var success = [];
var count = 0;
users.forEach(function(user, i){
request({
url: url,
method: 'POST',
json: true
}, function(err, resp, body){
if (!err && resp.statusCode === 200) {
success.push('user added');
}
});
if(count === users.length) {
res.json(success);
}
});
});
Regarding browser fetching the response before hitting enter key on the url, it is very unusual behaviour. Maybe you should check your machine if it is infected by any malware!
Regarding the code used by you, count is not incremented anywhere in the forEach loop. So it remains 0 forever and never equals users.length. So the loop will end but it will never send a response.
Also, you are testing for equality between count and users.length at the wrong place in the code.
This code should work:
app.get('/add-users', function (req, res) {
var success = [];
var count = 0;
users.forEach(function(user){
request({
url: url,
method: 'POST',
json: true
}, function(err, resp, body){
if (!err && resp.statusCode === 200) {
success.push('user added');
}
count++; // <<=== increment count
//
if(count === users.length) { // and then test if all done
res.json(success);
}
});
});
});
The problem here is you are mixing synchronous and asynchronous code together in a wrong way. Please note that forEach is synchrounous and request is asynchronous. So, looping over users finishes faster than the first result you get from request method.
#SantanuBiswas has one way you can solve your problem with the response returning before your requests are all complete, though depending on how many users you've got in your array and how slow the upstream service is, this is a potentially disastrous user experience as it will wait until all the requests are complete and only then fire back a response.
A better solution (in my opinion) would be to respond immediately with a 202 Accepted status code, and then update your DB with information about each user's status in the request handler for later reporting or debugging. Something like this (assuming you're using mongoose for your local user storage):
app.get('/add-users', function (req, res) {
res.setStatus(202).send(); // you can put more info in the body if desired
users.forEach(function(user){
request({
url: url,
method: 'POST',
json: true
}, function(err, resp, body){
const status = err ? 'error' : 'complete'; // obviously you might want to put more info somewhere, eg. error log, if there is an error
User.update({_id: user.id}, {status:status}, function(e) { if (e) console.error(err);});
});
});
});
Still another way, though it adds complexity to your solution, is to implement websockets to have your client get updated each time a request is complete. That example is a bit longer than I have time to post here, but the docs on the site I linked are excellent.
I am trying to access Firebase Database using REST API. Using cURL i am able to retrive data. I am unable to do the same in javascript.
cURL
curl 'https://testproject123.firebaseio.com/subscription.json?auth=aabbccddeeff123'
Javascript
var request = require('request');
var options = {
url: 'https://testproject123.firebaseio.com/subscription.json?auth=aabbccddeeff123' };
function callback(error, response, body) {
if (!error && response.statusCode == 200) {
console.log(body);
} }
request(options, callback);
Note: The above code i am trying to run on Bluemix/OpenWhisk.
Kindly let me know how to fix this.
The structure of your action is a bit off for OpenWhisk. Javascript actions need to have a main method and you'll have to use whisk.async() and whisk.done() to make your (asynchronous) REST API call work as you'd expect it.
An example of how to call an external API using a Javascript action in OpenWhisk can be found here.
I am using the request module in node to make an http request. How do I read the response content of that request so that I can make a programmatic decision based on the response?
gulp.task("run-server-tests", function(){
var responseJson = request("http://myurl/run-server-tests")
plugins.util.log(responseJson); // not sure how to get at the response json
});
request allows you to use callback function, see more detail
request('http://www.google.com', function (error, response, body) {
if (!error && response.statusCode == 200) {
console.log(body) // Show the HTML for the Google homepage.
}
})