I am having trouble with a Node.js http request. (A bigger question I will ask about later if I can't figure it out).
I have code that I modified from and an example and I don't understand what the response.on means. As read more about http in Node.js (Anatomy of an HTTP Transaction) I don't see people using examples with 'response.on'. I think I know but I want to clarify. Oh I am using Express.js too.
Thanks
Below is the code I am using to try and call the BART api I need a response from.
...
// Real Time Departure from a given station
router.route('/departTimeStation')
.get(function(req, res) {
vCmd = 'etd';
vOrig = req.query.vOriginStation;
vDir = 'n'; // [NOTE] - 'n' or 's', north or south, OPTIONAL
vPlat = 1; // [NOTE] - 1 to 4, number of platform, OPTIONAL
var xoptions = {
host: 'api.bart.gov',
path: '/api/etd.aspx?cmd=' + vCmd + '&orig=' + vOrig + '&key=' + config.bart.client
};
var xcallback = function(response) {
response.on('data', function(chunk) {
vParsed += chunk;
});
response.on('end', function() {
parseString(vParsed, function(err, result) {
vShow = JSON.stringify(result);
vTemp = result;
});
});
};
http.request(xoptions, xcallback).end();
return res.send (vTemp)
});
...
response.on can take a few 'events' that happen during the response life cycle,
The two you have above are response.on('data'...) which is when you get the data
and response.on('end'...) is at the end of the response, simple as that!
Node Docs on response: https://nodejs.org/api/http.html#http_class_http_incomingmessage
response.on is a function, or rather method, since it is a property.
Related
Background: Been trying for the last 2 day to resolve this myself by looking at various examples from both this website and others and I'm still not getting it. Whenever I try adding callbacks or async/await I'm getting no where. I know this is where my problem is but I can't resolve it myself.
I'm not from a programming background :( Im sure its a quick fix for the average programmer, I am well below that level.
When I console.log(final) within the 'ready' block it works as it should, when I escape that block the output is 'undefined' if console.log(final) -or- Get req/server info, if I use console.log(ready)
const request = require('request');
const ready =
// I know 'request' is deprecated, but given my struggle with async/await (+ callbacks) in general, when I tried switching to axios I found it more confusing.
request({url: 'https://www.website.com', json: true}, function(err, res, returnedData) {
if (err) {
throw err;
}
var filter = returnedData.result.map(entry => entry.instrument_name);
var str = filter.toString();
var addToStr = str.split(",").map(function(a) { return `"trades.` + a + `.raw", `; }).join("");
var neater = addToStr.substr(0, addToStr.length-2);
var final = "[" + neater + "]";
// * * * Below works here but not outside this block* * *
// console.log(final);
});
// console.log(final);
// returns 'final is not defined'
console.log(ready);
// returns server info of GET req endpoint. This is as it is returning before actually returning the data. Not done as async.
module.exports = ready;
Below is an short example of the JSON that is returned by website.com. The actual call has 200+ 'result' objects.
What Im ultimately trying to achieve is
1) return all values of "instrument_name"
2) perform some manipulations (adding 'trades.' to the beginning of each value and '.raw' to the end of each value.
3) place these manipulations into an array.
["trades.BTC-26JUN20-8000-C.raw","trades.BTC-25SEP20-8000-C.raw"]
4) export/send this array to another file.
5) The array will be used as part of another request used in a websocket connection. The array cannot be hardcoded into this new request as the values of the array change daily.
{
"jsonrpc": "2.0",
"result": [
{
"kind": "option",
"is_active": true,
"instrument_name": "26JUN20-8000-C",
"expiration_timestamp": 1593158400000,
"creation_timestamp": 1575305837000,
"contract_size": 1,
},
{
"kind": "option",
"is_active": true,
"instrument_name": "25SEP20-8000-C",
"expiration_timestamp": 1601020800000,
"creation_timestamp": 1569484801000,
"contract_size": 1,
}
],
"usIn": 1591185090022084,
"usOut": 1591185090025382,
"usDiff": 3298,
"testnet": true
}
Looking your code we find two problems related to final and ready variables. The first one is that you're trying to console.log(final) out of its scope.
The second problem is that request doesn't immediately return the result of your API request. The reason is pretty simple, you're doing an asynchronous operation, and the result will only be returned by your callback. Your ready variable is just the reference to your request object.
I'm not sure about what is the context of your code and why you want to module.exports ready variable, but I suppose you want to export the result. If that's the case, I suggest you to return an async function which returns the response data instead of your request variable. This way you can control how to handle your response outside the module.
You can use the integrated fetch api instead of the deprecated request. I changed your code so that your component exports an asynchronous function called fetchData, which you can import somewhere and execute. It will return the result, updated with your logic:
module.exports = {
fetchData: async function fetchData() {
try {
const returnedData = await fetch({
url: "https://www.website.com/",
json: true
});
var ready = returnedData.result.map(entry => entry.instrument_name);
var str = filter.toString();
var addToStr = str
.split(",")
.map(function(a) {
return `"trades.` + a + `.raw", `;
})
.join("");
var neater = addToStr.substr(0, addToStr.length - 2);
return "[" + neater + "]";
} catch (error) {
console.error(error);
}
}
}
I hope this helps, otherwise please share more of your code. Much depends on where you want to display the fetched data. Also, how you take care of the loading and error states.
EDIT:
I can't get responses from this website, because you need an account as well as credentials for the api. Judging your code and your questions:
1) return all values of "instrument_name"
Your map function works:
var filter = returnedData.result.map(entry => entry.instrument_name);
2)perform some manipulations (adding 'trades.' to the beginning of each value and '.raw' to the end of each value.
3) place these manipulations into an array. ["trades.BTC-26JUN20-8000-C.raw","trades.BTC-25SEP20-8000-C.raw"]
This can be done using this function
const manipulatedData = filter.map(val => `trades.${val}.raw`);
You can now use manipulatedData in your next request. Being able to export this variable, depends on the component you use it in. To be honest, it sounds easier to me not to split this logic into two separate components - regarding the websocket -.
I'm building an application using Node/Express/MongoDB (first time with all these) that will let me pull data from the DB and display it on an Express page. This is what the GET request looks like:
var str = "";
app.route('/view-reports').get(function(req, res) {
var cursor = collections.find({});
cursor.each(function(err, item) {
if (item != null) {
console.log(str);
str = str + "Substance: " + item.substance + "<br>";
}
if (err) {
console.log(err);
}
});
console.log(str);
res.send(str);
str = "";
});
I would expect this to return something like this:
Substance: a
Substance: b
Substance: c
However, the initial request does not return anything at all. The second request will return the above. If I enclose res.send(str) in an if conditional it simply will not load until a second request is made.
cursor.each() is asynchronous. That means it runs sometimes LATER, after your res.send(str), thus you get the previous version of str. You need to collect all the data first and then send your response only when you have all the data.
If you want all the data, then you could use promises and .toArray() like this:
app.route('/view-reports').get(function(req, res) {
collections.find({}).toArray().then(data => {
let result = data.map(item => {
return "Substance: " + item.substance + "<br>";
}).join("");
res.send(result);
}).catch(err => {
// database error
console.log(err);
res.sendStatus(500);
});
});
Note: This also wisely gets rid of the str variable which was outside the scope of the request and thus could easily lead to a concurrency bug when multiple requests were in flight at the same time (from different users).
Create a router specifically for substances and use it in app. Instead of breaks, you can create a ul, also, that processing should happen on the front end. Separate your concerns. The server shouldn't have to worry about any rendering and etc. One purpose per process.
The routers can be created per resource. Create a router for substances, for cats, for dogs. Each individual router has it's own get post delete and puts that allow you to modify that resource. app can use all the routers at once.
app.use(catRouter);
app.use(mooseRouter);
app.use(platypusRouter);
const { Router } = require('express');
const createError = require('http-errors');
let substanceRouter = new Router();
function buildElement(arr)
{
let start = '';
arr.forEach(val => {
if(!val) return;
start += `Substance : ${val}<br>`;
});
return start;
}
subtanceRouter.get('/endpoint/whatever', function(req, res, next) {
collectios.find({})
.then(results => {
if(!results) throw new Error('Resource Not Found');
let output = buildElement(results);
res.json(output);
next();
})
.catch(err => next(createError(404, err.message)));
})
app.use(substanceRouter);
Alternately we can write :
let output = results
.filter(sub => !!sub)
.join('<br>');
res.json(output);
But be advised this will add memory overhead, generates a completely new array to house the results, consuming at worst, O(n) memory.
I'd like to write a feature like this:
Scenario: new Singleton create
When a new, unmatchable identity is received
Then a new tin record should be created
And a new bronze record should be created
And a new gold record should be created
which would tie to steps like this:
defineSupportCode(function ({ Before, Given, Then, When }) {
var expect = require('chai').expect;
var chanceGenerator = require('./helpers/chanceGenerator')
var request = require('./helpers/requestGenerator')
let identMap;
// reset identMap before each scenario
Before(function () {
identMap = [];
});
// should generate a valid identity
// persist it in a local variable so it can be tested in later steps
// and persist to the db via public endpoint
When('a new, unmatchable identity is received', function (callback) {
identMap.push(chanceGenerator.identity());
request.pubPostIdentity(identMap[identMap.length-1], callback);
});
// use the local variable to retrieve Tin that was persisted
// validate the tin persisted all the props that it should have
Then('a new tin record should be created', function (callback) {
request.pubGetIdentity(identMap[identMap.length-1], callback);
// var self = this;
// request.pubGetIdentity(identMap[identMap.length-1], callback, () => {
// console.log('never gets here...');
// self.callback();
// callback();
// });
// request.pubGetIdentity(identMap[identMap.length-1], (callback) => {
// console.log('never gets here...');
// self.callback();
// callback();
// });
});
The issue that I'm having is that I can't do anything in the Then callback. That is where I'd like to be able to verify the response has the right data.
Here are relevant excerpts from the helper files:
var pubPostIdentity = function (ident, callback) {
console.log('pubIdentity');
var options = {
method: 'POST',
url: 'http://cucumber.utu.ai:4020/identity/' + ident.platform + '/' + ident.platformId,
headers: {
'X-Consumer-Custom-Id': ident.botId + '_' + ident.botId
},
body: JSON.stringify(ident)
};
console.log('ident: ', ident);
request(options, (err, response, body) => {
if (err) {
console.log('pubPostIdentity: ', err);
callback(err);
}
console.log('pubPostIdentity: ', response.statusCode);
callback();
});
}
// accept an identity and retrieve from staging via identity public endpoint
var pubGetIdentity = function (ident, callback) {
console.log('pubGetIdentity');
var options = {
method: 'GET',
url: 'http://cucumber.utu.ai:4020/identity/' + ident.platform + '/' + ident.platformId,
headers: {
'X-Consumer-Custom-Id': ident.botId + '_' + ident.botId
}
};
request(options, (err, response) => {
if (err) {
console.log('pubGetIdentity: ', err);
callback(err);
}
console.log('pubGetIdentity: ', response.body);
callback();
});
}
Something that we are considering as an option is to re-write the feature to fit a different step definition structure. If we re-wrote the feature like this:
Scenario: new Singleton create
When a new, unmatchable 'TIN_RECORD' is received
Then the Identity Record should be created successfully
When the Identity Record is retreived for 'tin'
Then a new 'tin' should be created
When the Identity Record is retreived for 'bronze'
Then a new 'bronze' should be created
When the Identity Record is retreived for 'gold'
Then a new 'gold' should be created
I believe it bypasses the instep callback issue we are wrestling with, but I really hate the breakdown of the feature. It makes the feature less readable and comprehensible to the business.
So... my question, the summary feature presented first, is it written wrong? Am I trying to get step definitions to do something that they shouldn't? Or is my lack of Js skills shining bright, and this should be very doable, I'm just screwing up the callbacks?
Firstly, I'd say your rewritten feature is wrong. You should never go back in the progression Given, When, Then. You are going back from the Then to the When, which is wrong.
Given is used for setting up preconditions. When is used for the actual test. Then is used for the assertions. Each scenario should be a single test, so should have very few When clauses. If you want, you can use Scenario Outlines to mix several very similar tests together.
In this case, is recommend to take it back to first principles and see if that works. Then build up slowly to get out working.
I suspect in this case that the problem is in some exception being thrown that isn't handled. You could try rewriting it to use promises instead, which will then be rejected on error. That gives better error reporting.
im very very new to Node.js, javascript in general, and also functional programming (which node is if im not mistaken?)
Im currently on stage of doing learnyounode tutorials.
I know i can find all the solutions and work it out just fine, but im a little curious why wouldnt my code work...
If anyone is familiar with the learnyounode im stuck at "Juggling async".
The code that i wrote:
var http = require("http");
var addriee = [process.argv[2], process.argv[3], process.argv[4]];
function getStuffFromNet(address, callback) {
http.get(address, function getShitDone(response) {
var dataToCallback = "";
response.on("error", function(data) {
callback(data, null);
});
response.on("data", function(data) {
dataToCallback+=data;
});
response.on("end", function(data) {
callback(null, dataToCallback);
});
});
};
function printToConsole(data) {
console.log(data);
}
printToConsole(getStuffFromNet(addriee[0]));
My goal was to reuse function that would get "stuff from net", the error i get is:
learnyounode run http-get3.js
undefined
/home/ubuntu/workspace/learnyounode/http-get3.js:17
callback(null, dataToCallback);
^
TypeError: undefined is not a function
at IncomingMessage.<anonymous> (/home/ubuntu/workspace/learnyounode/http-get3.js:17:7)
at IncomingMessage.emit (events.js:117:20)
at _stream_readable.js:944:16
at process._tickCallback (node.js:442:13)
Why is the last callback null and not data ?
Also it might be handier to not initialize
var dataToCallback = "";
to
var dataToCallback;
because else you can't use data
typeof dataToCallback !== 'undefined'
Not sure about 's atm.
Also try to comment you're code a lot more. Especially when you're learning it.
Example of some debugging level I have (noob or not I quickly find errors this way)
/**
* Divest the desired amount
*/
socket.on("divest", function (amount) {
error.debug(classname + "Divest is called [" + amount + "]");
invest.divest(hash, amount, function (err, callback) {
if (!err) {
error.debug(uid, name + " />divesting [CBACK]" + callback);
} else {
error.debug(uid, name + " />divesting [ERROR]" + err);
}
socket.emit("done", true);
});
});
Hope I helped.
To explain your situation, the data was read to the end and "callback" is invoked,
but the "callback" was not defined at the last line of your script.
If you wonder why the data.on("error" .....) wasn't triggered, It will only be triggered by data error of the http.get(), it means you are "ABLE TO READ DATA" from the URLs, so the http.get() will trigger data.on("data" ....) and data.on("end" .....) only.
Hi all so I've totally reworded this and included some code I produced. Normally I'd have tests for this first however I'm at a lost as to how to approach this.
I want to start off with unit tests for this function.
exports.getMatches = function(callback) {
var url = "http://football-api.comapi/?Action=today&APIKey=" + secrets.APIKey + "&comp_id=1204";
async.waterfall([
function(callback) {
request(url, function (error, response, body) {
if (error) return (error);
var parsedJSON = JSON.parse(body);
var todaysMatches = parsedJSON.matches;
var schedule = new Schedule({
date: dates.today // external module
matches: []
});
_.each(todaysMatches, function (match) {
schedule.matches.push({
match_id: match.match_id,
match_time: match.match_time,
match_localteam_id: match.match_localteam_id,
match_localteam_name: match.match_localteam_name,
match_visitorteam_id: match.match_visitorteam_id,
match_visitorteam_name: match.match_visitorteam_name
});
});
callback(schedule);
});
}
], function(schedule) {
schedule.save(function (err) {
if (err) return (err);
});
done();
});
};
I'm really sorry for not providing any sort of headway because I'm just having a mental block here. There's only a few lines of code but there is so much going on and so much to test that I'm not sure where to start / what to mock / how to mock it.
It sounds like you are wanting to test request() and not your actual controller method.
Your unit test should evaluate whatever is passed to the list() callback, and determine if it has been transformed accordingly.
For example, if list() turns your JSON array into an HTML list <ul><li /><li />...</ul>, you should test the response/body output to determine if the transformation has been successful.
var mockJSON = [{id: 1, name: 'spinach'}];
var expected = '<ul><li data-id="1">spinach</li></ul>';
//...
matchController.list(function (error, response, body) {
// something like this...
expect(body).to.equal(expected);
});