This question already has answers here:
Why am I not able to access the result of multiple networking calls?
(2 answers)
Closed 5 years ago.
I'm trying to get some information from a web page using request to get the page and then cheerio to traverse the DOM to the specific part I need I'm repeating this process for multiple elements in an array using array.forEach using this code:
const cheerio = require('cheerio');
const request = require('request');
var i = 0;
var rates = [];
['AUD', 'CAD'].forEach(function(currancy){
var url = "https://www.google.com/finance/converter?a=1&from=USD&to=" + currancy
request(url , function(error, res , body){
const $ = cheerio.load(body);
var temp = $('.bld');
var rate = temp.text();
console.log(rate);
rates[i]= rate;
i++;
})
});
console.log('done');
the result I'm expecting is something like this:
1.31 AUD
1.28 CAD
done
but I'm getting this insted:
done
1.31 AUD
1.28 CAD
can you tell me why is array.forEach not blocking my code?
You are printing 'done' before any of your http requests to google return.
You loop though the currencies, making a call to google for each one, and print 'done'. Then the calls start to return (in random order, btw) and you print the results.
So, the forEach isn't asynchronous, but the http requests are.
Related
I have been trying to complete this past paper but Im not sure if im on the right lines as to my solution and so wanted some help to see if I am on the right track or whether I need some guidance on getting there!
the question is as follows:
You have been contracted to write part of a RESTful web API for products on an e-commerce platform using Node.js. Another developer had previously been working on the project and had written the following tests using Mocha and Chai.
const expect = require('chai').expect;
const request = require('superagent');
const status = {OK:200,BADREQUEST:400};
const endpointURL =
'http://localhost:3000/api/v1/products/';
describe('Products APIT,function(){
/* Details of below data unnecessary to answer question */
const validId = /* ID of product already in database */;
const invalidId = /* Invalid product ID */;
const validProduct = {/* Valid product object*/};
it('valid GET requests',function(done){
request.get(apiRoot + validId).end(function(err,res){
expect(res.statusCode).to.equal(status.OK);
expect(res.body).to.have.property('price');
expect(res.body).to.have.property('quantity');
done();
});
});
it('invalid GET requests',function(done){
request.get(apiRoot + invalidId).end(function(err,res){
expect(res.statusCode).to.equal(status.BAD_REQUEST);
done();
});
});
it('valid PUT requests',function(done){
request.put(endpointURL + validId).send(validProduct)
.end(function(err,res){
expect(res.statusCode).to.equal(status.OK);
expect(res.body).to.deep.equal(validProduct);
request.get(apiRoot + validID).end(function(err,res){
expect(res.body.price).to.equal(validProduct.price);
expect(res.body.quantity)
.to.equal(validProduct.quantity);
done();
});
});
});
The tests also partially specify the behaviour of the API when responding to a PUT request. It should be possible to update an existing product to take a new (valid) value by making a PUT request to the path /api/vl/products/ followed by the product ID (e.g. /api/vl/products/42 for the product with ID 42). The response to a successful update should have status code 400 (OK) and the body of the response should contain the updated product. Updating the product should not change its ID.
Below is a partial implementation of the API using the express framework
var express = require('express');
var app = express();
var products = require('./lib/products');
app.getC/api/v1/products/:idP,function(req,res,next){
const id = req.params._id;
/*i) Complete for question i) here */
} ) ;
app.putC/api/v1/products/:idP,function(req,res,next){
const id = req.params._id;
/*ii) Complete for question ii) here */
} ) ;
var port = 3000;
app.listen(port,function(){
console.log('Listening on port ' + port);
});
where the products variable contains a data layer with the following methods:
Method Signature
Description
Result
read(id,callback)
reads product with
Product
specified id.
with
specified id
---------------------------
--------------------
------------
update(id,product,callback)
updates product with
updated
specified id to have
product.
value of parameter
product.
In both of these cases, callback will be a function to be called after the database operation has completed (successfully or unsuccessfully) with two arguments. The first argument will be an error object (null if and only if the operation has completed without errors). The second argument will be a result object containing the result of a successfully completed operation as described in the table above.
i) Complete the code above such that the API responds to GET requests as specified in the tests.
ii) Complete the code above such that the API responds to PUT requests as specified in the tests.
I have placed my attempt at question i) here below:
product.read(id,function(err,res)){
if(res.status(status.OK)
return res.sendStatus(status.OK);
return res.body('price' && 'quantity');
if(res.status(status.BAD_REQUEST))
return res.sendStatus(status.BAD_REQUEST);
}
but for ii) im afraid my brain just literally CRASHES, im new to node.js and js so please dumb down the explanation as much as you can for me!! Thank you i very much will appreciate all help since i am a beginner and im finding it hard to wrap my head around this question! Lecture notes do not seem to help much :((
I'm building an app which i need to stream data to client, my data is simply an array of objects .
this is the for loop which makes the array
for(let i =0;i<files.length;i++){
try {
let file = files[i]
var musicPath = `${baseDir}/${file}`
let meta = await getMusicMeta(musicPath)
musics.push(meta)
}
right now I wait for the loop to finish it's works then I send the whole musics array to client, I want to use stream to send musics array one by one to client instead of waiting for the loop to finish
Use scramjet and send the stream straight to the response:
const {DataStream} = require("scramjet");
// ...
response.writeHead(200);
DataStream.fromArray(files)
// all the magic happens below - flow control
.map(file => getMusicMeta(`${baseDir}/${file}`))
.toJSONArray()
.pipe(response);
Scramjet will make use of your flow control and most importantly - it'll get the result out faster than any other streaming framework.
Edit: I wrote a couple lines of code to make this use case easier in scramjet. :)
This question might be a little vague, but I'll try my best to explain.
I'm trying to create an array of all of the tweets that I can retrieve from Twitter's API, but it limits each request to 200 returned tweets. How can I request to Twitter asynchronously up to the maximum limit of 3200 returned tweets? What do I mean is, is it possible to asynchronously call Twitter's API but build the array sequentially, making sure that the tweets are correctly sorted with regard to date?
So, I have an array:
var results = [];
and I'm using node's request module:
var request = require('request');
what I have right now (for just the limit of 200) is
request(options, function(err, response, body) {
body = JSON.parse(body);
for (var i = 0; i < body.length; i++) {
results.push(body[i].text);
}
return res.json(results);
});
I've looked into maybe using the 'promise' module, but it was confusing to understand. I tried using a while loop, but it got complicated because I couldn't follow the path that the server was taking.
Let me know if this didn't explain things well.
In the end, I want results to be an array populated with all of the tweets that the requests send.
I would suggest using request-promise instead of request. Here is my solution.
var rp = require('request-promise');
var tweets = [];
var promises = [];
for (var i =1; i< 10; i++){
var promise = rp(options);
promises.push(promise);
}
Promise.all(promises).then(function(data){
data.forEach(function(item){
// handle tweets here
});
return res.json(tweets);
});
I am new to BigQuery and need more functions in BiogQuery + Javascript so i can get total execution time and how much GB of data processed by that query.
How can i know total exe time and processed data size in javascript api.
Eg. Query complete (1.6s elapsed, 35.7 GB processed)
the above example result from javascript api.
The total processed bytes i can get from response. But query exeution time from where i will get this. I dont want to run timer (to calculate time) before and after query exe.
Also need a intel on how to see executed query history from JavaScript api.
Thanks in advance.
To determine how long a job took, you can compare statistics.creationTime, statistics.startTime, and statistics.endTime, depending on your needs. These can be accessed from the jobs.list or jobs.get API. These responses will also contain the bytes processed by a query in the statistics.query.totalBytesProcessed field.
To retrieve a history of jobs (including queries, and any other load, copy, or extract jobs you may have run) you can call the jobs.list API.
Specifically in JS, if you have a query response containing a jobReference, you can run something like the following to retrieve the full job details using the jobs.get method and log them to the console. The logged response should contain the fields linked above.
var projectId = response['jobReference']['projectId'];
var jobId = response['jobReference']['jobId'];
var path = 'https://clients6.google.com/bigquery/v2/projects/' + projectId + '/jobs/' + jobId;
var request = {'path': path, 'method': 'GET'};
gapi.client.request(request).execute(function(response) { console.log(response) });
So I have an array of URLs
I want to pull the html from each (for which I am using restler node.js library)
Then select some of that data to act on via jquery (for which I am using cheerio node.js library)
The code I have works, but duplicates the pulled data by however many URLS there are.
I am doing this in Node but suspect it's a generalized Javascript matter that I don't understand too well.
url.forEach(function(ugh){
rest.get(ugh).on('complete', function(data) {
$ = cheerio.load(data);
prices.push($(".priceclass").text());
//i only want this code to happen once per item in url array
//but it happens url.length times per item
//probably because i don't get events or async very well
});
});
So if there are 3 items in the 'url' array, the 'prices' array with the data I want will have 9 items. Which I don't want
--EDIT:
Added a counter to verify that the 'complete' callback was executing array-length times per array item.
x=0;
url.forEach(function(ugh){
rest.get(ugh).on('complete', function(data) {
var $ = cheerio.load(data);
prices.push($(".priceclass").text());
console.log(x=x+1);
});
});
Console outputs 1 2 3 4 5 6 7 8 9
I was thinking that I might be going about this wrong. I've been trying to push some numbers onto an array, and then outside the callbacks do something with that array.
Anyways it seems clear that >1 restler eventlisteners aren't gonna work together at all.
Maybe rephrasing the question would help:
How would I scrape a number of URLs, then act on that data?
Currently looking into request & async libraries, via code from the extinguished node.io library
To answer the rephrased question, scramjet is great for this if you use ES6+ and node which I assume you do:
How would I scrape a number of URLs, then act on that data?
Install the packages:
npm install scramjet node-fetch --save
Scramjet works on streams - it will read your list of url's and make each url a stream that you can work with as simple as with Array's. node-fetch is a simple node module that follows the standard Fetch Web API.
A simple example that also reads the url's from a file, assuming you store them one per line:
const {StringStream} = require("scramjet");
const fs = require("fs")
const fetch = require("node-fetch");
fs.createReadStream(process.argv[2]) // open the file for reading
.pipe(new StringStream()) // redirect it to scramjet stream
.split("\n") // split line by line
.map((url) => fetch(url)) // get the URL from the endpoint
.map((resp) => JSON.parse(resp)) // parse the response
.toArray() // accumulate the data into an Array
.then(
(data) => doYourStuff(data), // do the calculations
(err) => showErrorMessage(data)
)
Thanks to the way scramjet works, you needn't worry about error handling (all errors are caught automatically) and managing simultaneous requests. If you can parse the files url by url, then you can also make this very memory and resources efficient - as it won't ready and try to fetch all the items at once, but it will do some work in parallel.
There are more examples and the full API description in the scramjet docs.