So I had been stuck on this exercise in Treehouse some time ago and just moved on. I came back to it now that I understand things better and I'm still fighting with the wunderground api. I've read through the json data and documentation, updated a few things from when the class was first recorded (and the API updated since then), and still am getting errors I can't field. I've got three js files- app.js, weather.js, and api.json (which is just my api key so not shared here.)
After my corrections, I'm still getting the error "TypeError: Cannot read property 'temp_f' of undefined" which doesn't make sense as I keep reading over the JSON to check that it's pointing to the right place.
Can anyone put an end to my misery trying to fix this?
App.js:
const weather = require('./weather');
//Join multiple values passed as arguments and replace all spaces with underscores
const query = process.argv.slice(2).join("_").replace(' ', '_');
//query: 90201
//query: Cleveland_OH
//query: London_England
weather.get(query);
Weather.js
const https = require('https');
const http = require('http');
const api = require('./api.json');
// Print out temp details
function printWeather(weather) {
const message = `Current temp in ${weather.location} is ${weather.current_observation.temp_f}F`;
console.log(message);
}
// Print out error message
function get(query) {
const request = http.get(`http://api.wunderground.com/api/${api.key}/conditions/q/${query}.json`, response => {
let body = "";
// Read the data
response.on('data', chunk => {
body += chunk;
});
response.on('end', () => {
//Parse data
const weather = JSON.parse(body);
//Print the data
printWeather(weather);
});
});
}
module.exports.get = get;
//TODO: Handle any errors
Related
I have an application developing using Nodejs. This application is making a request to GitLab API and obtaining the raw file data from it.
I would like to read the particular string which is present after another string and get all similar data from it. I am just a bit confused on this part and unable to proceed further can someone please explain to me how to achieve this?
Following is the sample file data:
I would like to read all the numbers if present after the keyword Scenario: i.e in this case I would like to get A001-D002 & K002-M002. These numbers can be anything random and can appear anywhere within the file content. I would like to read them and store them within an array for that particular file.
FileName: File Data
Background:
This is some random background
Scenario: A001-D002 My first scenario
Given I am sitting on a plane
When I am offered drinks
Scenario: K002-M002 My second scenario
Given when I book the uber taxi
When I get the notifications
I am not understanding how to iterate over the file content and read every word and match and accordingly obtain the ids.
Following is the code that I have which makes the request to GitLab and obtains the raw file content:
./index.js:
const express = require('express');
const http = require("http");
const bodyParser = require('body-parser');
const app = express();
const port = process.env.PORT || 9000;
const gitlabDump = require("./controller/GitLabDump");
app.use(bodyParser.json());
app.use(bodyParser.urlencoded({ extended: true }));
//Make NodeJS to Listen to a particular Port in Localhost
app.listen(port, function(){
gitlabDump.gitlabDump(type, function(data){
console.log("Completed Execution for GitLab")
process.exit();
})
}
My ./controller/GitLabDump.js:
const request = require('request');
const https = require('https');
const axios = require('axios');
exports.gitlabDump = function(callback){
var gitlabAPI = "https://gitlab.com/api/v4/projects/<project_id>/repository/files/tree/<subfolders>/<fileName>/raw?ref=master&private_token=<privateToken>";
//Make the request to the each file and read its raw contents
request(gitlabAPI, function(error, response, body) {
const featureFileData = JSON.parse(JSON.stringify(body)).toString();
console.log(featureFileData)
for(const match of featureFileData.matchAll("Scenario:")){
console.log(match);
}
callback("Completed");
})
}
I am able to print the file contents. Can someone please explain me how can I iterate over the raw file contents and get all the required ids?
I suggest you to use a method by analyzing each part of your string by iterating over each lines (i assume that your string is compose like in your exemple). It is easier to understand and coding it than using a regex.
The exemple below represent your request callback function.
I split the code in 3 logics :
search the filename
search the line we are interesting with ("Scenario" word)
extract the ID by filter function
You can after that, easily change you ID filter (txt.substr(0, txt.indexOf(' ')) to use a more proper expression to extract your sentence.
The result is sent to a callback function with as first argument the filename, and as second all ids. Like you did in your exemple.
((callback) => {
const featureFileData = `FileName: File Data
Background:
This is some random background
Scenario: A001-D002 My first scenario
Given I am sitting on a plane
When I am offered drinks
Scenario: K002-M002 My second scenario
Given when I book the uber taxi
When I get the notifications`;
// find "filename"
const filenames = featureFileData.split('\n')
.filter(line => line.trim().substr(0,8) === 'FileName')
.map((raw) => {
if(!raw) return 'unknown';
const sentences = raw.trim().split(':');
if(sentences[1] && sentences[1].length) {
return sentences[1].trim();
}
});
// filter the "Scenario" lines
const scenarioLines = featureFileData.split('\n')
.map((line) => {
if(line.trim().substr(0,8) === 'Scenario') {
const sentences = line.trim().split(':');
if(sentences[1] && sentences[1].length) {
return sentences[1].trim();
}
}
return false;
})
.filter(r => r !== false);
// search ids
const ids = scenarioLines.map(txt => txt.substr(0, txt.indexOf(' ')));
callback(filenames[0], ids);
})(console.log)
I need to request data from my REST server to populate my UI (frontend). In doing so, I need to request some data from my and other servers. One such request is to get a list of states (provinces), process each one and add them to a select HTML component. I use fetch() and .json() amongst other tools to do this.
Problem:
In calling my REST server for json data, I receive the following data (taken from Chrome console):
{provinces:[Eastern Cape,Mpumalanga,Western Cape,Gauteng,KwaZulu Natal,North West,Northern Cape,Free
State,Limpopo]}
I intend to add each of these as an option to a select. While attempting to acquire the value for the provinces key, I get undefined.
I am making this call using:
fetch("http://localhost:3443/app/location/provinces").then(e => e.json()).then(e => console.log(e.provinces));
Further, since I can directly refer to json keys using the [] operator, I attempt this using
fetch("http://localhost:3443/app/location/provinces").then(e => e.json()).then(e => console.log(e['provinces']));
which as you may have guessed aswel, also returns undefined.
For the record, the full Chrome Console output is
Promise {<pending>}
undefined
Looking over some SO examples, I believe my call(s) may be correct, this one, and this one, and this one all which confirm its validity.
What else have I tried:
This SO post and this one suggested to use the json data response inside of the same then() call e.g.
fetch("http://localhost:3443/app/location/provinces").then(e => {
e.json().then(s => {
console.log(s['provinces']);
});
});
and
fetch("http://localhost:3443/app/location/provinces").then(e => {
e.json().then(s => {
console.log(s.provinces);
});
});
both which return:
Promise {<pending>}
undefined
What am I missing / doing wrong?
Update
Screenshot of Chrome console in order of commands listed above:
Resource file za-province-city.json
NodeJS express code:
const express = require('express');
const router = express.Router();
const fs = require('fs');
const raw = fs.readFileSync("./res/za-province-city.json");
const map = JSON.parse(raw);
const mapProvinceCity = {};
map.forEach(item => {
if (!mapProvinceCity.hasOwnProperty(item.ProvinceName)) {
mapProvinceCity[item.ProvinceName] = [];
}
mapProvinceCity[item.ProvinceName].push(item.City);
});
for (let key in mapProvinceCity) {
mapProvinceCity[key].sort((a, b) => a.toLocaleString().localeCompare(b.toLowerCase()));
}
router.get('/location/provinces', function (req, res, next) {
let strings = Object.keys(mapProvinceCity);
let json = JSON.stringify({provinces: strings}).replace(/"/g, '');
return res.json(json);
});
router.get('/location/:province/cities', function (req, res, next) {
let province = req.param('province');
let cities = mapProvinceCity[province];
let json = JSON.stringify({cities: cities}).replace(/"/g, '');
return res.json(json);
});
module.exports = router;
Note: if you are wondering about the replace(), each time I requested data in postman, I got
I think your issues all stem from a misunderstanding of Express' res.json().
This is basically a shortcut for
res.set("Content-type: application/json")
res.status(200).send(JSON.stringify(data))
I imagine your problems started when you thought you needed to stringify your data. What happens then is that your data is double-encoded / double stringified, hence the extra quotes. Removing the quotes though mangles your data.
console.log() is not a particularly good debugging tool as it obfuscates a lot of information. In your code, s is actually a string
"{provinces:[Eastern Cape,Mpumalanga,...]}"
I suggest you use the actual debugger instead.
The simple solution is to use res.json() as intended
router.get('/location/provinces', function (req, res, next) {
return res.json({ provinces: Object.keys(mapProvinceCity) });
});
with your client-side code looking like
fetch("http://localhost:3443/app/location/provinces")
.then(res => {
if (!res.ok) {
throw res
}
return res.json()
})
.then(data => {
console.log('Provinces:', data.provinces)
})
This goes for all your Express routes. Do not use JSON.stringify().
I have over 20k documents in my Realtime Database. I need to stream them, but I'm not even sure how to get started with it. This is kind of what I was trying to go for
sendEmail.get('/:types/:message', cors(), async (req, res, next) => {
console.log(5);
const types = JSON.parse(req.params.types);
console.log('types', types);
let recipients = [];
let mails = [];
if (types.includes('students')) {
console.log(1);
const tmpUsers = await admin.database().ref('Users').orderByChild('student').equalTo(true).once('value').then(r => r.val()).catch(e => console.log(e));
recipients = recipients.concat(tmpUsers);
}
if (types.includes('solvers')) {
console.log(2);
let tmpUsers = await admin.database().ref('Users').orderByChild('userType').equalTo('person').once('value').then(r => r.val()).catch(e => console.log(e));
tmpUsers = tmpUsers.concat(arrayFromObject(await admin.database().ref('Users').orderByChild('userType').equalTo('company').once('value').then(r => r.val()).catch(e => console.log(e))));
recipients = recipients.concat(tmpUsers);
}
});
But this code causes my server to run out of memory. Someone suggested streams in my previous question, but as much as I like the idea, I have no idea how to actually do the streaming. I know it should be something like:
const fs = require('fs');
const readStream = fs.createReadStream('path goes here');
const data = [];
readStream.on('data', chunk => {
data.push(chunk);
})
readStream.on('end', () => {
console.log(data);
res.end(data);
});
But how on earth do I pass a firebase query into the path?
I tried this, but it said Argument type Promise is not assignable to parameter type PathLike, which makes sense, but how do I get around it?
const users = fs.createReadStream(admin.database().ref('News').once('value').then(r => r.val()))
To summarize: How do I stream data from a Firebase realtime database?
Edit:
How is that a duplicate? These are 2 completely different questions. The starting code is the same but the ways of solving I'm asking about are completely different
Realtime Database doesn't have any streaming APIs. When you perform a query, the entire result set is always loaded into memory in one shot.
A possible alternative is to page through the results using limitToFirst() along with some query ordering that allows you to perform a subsequent query after the first batch of items is processed, picking up where the last query left off.
File called testing.js
I can do whatever I like with the data in saveWeatherData but cannot call this function and return the data without getting 'undefined'
For example if i tried the below code in saveWeatherData it will print out the summary as expected...
console.log(The summary of the weather today is: ${dataArray[0]});
However I want to use these values within another file such as a server file that when connected to will display weather summary temperature etc.
So I need to return an array with these values in it so that I can call this function and get my data stored in an array for further use.
I know that the reason the array --dataArray is returning undefined is because asynchronous code.
The array is returned before we have gotten the data using the callback.
My question, is there anyway to do what I am trying to do?
I tried my best to explain the problem and what I want to do, hopefully its understandable.
Would I have to use a callback inside of a callback? To callback here to return the data when its been fetched?
I just cant get my head about it and have tried multiple things to try and get the result I am looking for.
My last idea and something i would prefer not to do is the use the 'fs' module to save the data to a text or json file for use in my other files through reading the data from the saved file...
I feel im close but cant get over the last hurdle, so ive decided to ask for a little help, even just point me on the right track and Ill continue to try and figure it out.
Phew...
Thank you for your time!
const request = require("request");
let dataArray = [];
let saveWeatherData = function(weatherData) {
dataArray = weatherData;
return dataArray;
};
let getWeatherData = function(callback) {
request({
url: `https://api.forecast.io/forecast/someexamplekey/1,-1`,
json: true
}, (error, response, body) => {
//Creating array to hold weather data until we can save it using callback...
let array = [];
if (error) {
console.log("Unable to connect with Dark Sky API servers.")
}
else {
console.log(`Successfully connected to Dark Sky API servers!\n`);
array.push(body.currently.summary, body.currently.temperature, body.currently.apparentTemperature, body.currently.windSpeed, body.currently.windBearing);
callback(array);
}
});
};
getWeatherData(saveWeatherData);
module.exports = {
saveWeatherData
};
My Other File...
File called server.js
const http = require("http");
const testing = require("./testing");
function onRequest(request, response){
let data = testing.saveWeatherData();
console.log(`A user made a request: ${request.url}`);
response.writeHead(200, {"context-type": "text/plain"});
response.write("<!DOCTYPE html>");
response.write("<html>");
response.write("<head>");
response.write("<title>Weather</title>");
response.write("</head>");
response.write("<body>");
response.write("Weather summary for today: " + data[0]);
response.write("</body>");
response.write("</html>");
response.end();
}
http.createServer(onRequest).listen(8888);
console.log("Server is now running on port 8888...");
I'm still not sure about what are you trying to do. However, I think you're not exporting what you suppose to be exporting. To avoid the use of so many callbacks you may use async/await.
Change this part of your server.js
async function onRequest(request, response) {
let data = await testing.getWeatherData();
console.log(`A user made a request: ${request.url}`);
response.writeHead(200, { 'context-type': 'text/plain' });
response.write('<!DOCTYPE html>');
response.write('<html>');
response.write('<head>');
response.write('<title>Weather</title>');
response.write('</head>');
response.write('<body>');
response.write('Weather summary for today: ' + data[0]);
response.write('</body>');
response.write('</html>');
response.end();
}
And this of your testing.
let getWeatherData = function() {
return new Promise(resolve =>
request(
{
url: `https://api.darksky.net/forecast/someexamplekey/1,-1`,
json: true
},
(error, response, body) => {
//Creating array to hold weather data until we can save it using callback...
let array = [];
if (error) {
console.log('Unable to connect with Dark Sky API servers.');
} else {
console.log(`Successfully connected to Dark Sky API servers!\n`);
array.push(
body.currently.summary,
body.currently.temperature,
body.currently.apparentTemperature,
body.currently.windSpeed,
body.currently.windBearing
);
resolve(array);
}
}
)
);
};
module.exports = {
getWeatherData
};
It will check for new Weather in each request. If you want to save the result to avoid checking every single time you might need to do something else. But I think for a weather app the important is to keep it updated.
I'm trying to make a .js file that will constantly have the price of bitcoin updated (every five minutes or so). I've tried tons of different ways to web scrape but they always output with either null or nothing. Here is my latest code, any ideas?
var express = require('express');
var path = require('path');
var request = require('request');
var cheerio = require('cheerio');
var fs = require('fs');
var app = express();
var url = 'https://blockchain.info/charts/';
var port = 9945;
function BTC() {
request(url, function (err, res, body) {
var $ = cheerio.load(body);
var a = $(".market-price");
var b = a.text();
console.log(b);
})
setInterval(BTC, 300000)
}
BTC();
app.listen(port);
console.log('server is running on '+port);
It successfully says what port it's running on, that's not the problem. This example (when outputting) just makes a line break every time the function happens.
UPDATE:
I changed the new code I got from Wartoshika and it stopped working, but im not sure why. Here it is:
function BTCPrice() {
request('https://blockchain.info/de/ticker', (error, response, body) => {
const data = JSON.parse(body);
var value = (parseInt(data.USD.buy, 10) + parseInt(data.USD.sell, 10)) / 2;
return value;
});
};
console.log(BTCPrice());
If I have it console.log directly from inside the function it works, but when I have it console.log the output of the function it outputs undefined. Any ideas?
I would rather use a JSON api to get the current bitcoin value instead of an HTML parser. With the JSON api you get a strait forward result set that is parsable by your browser.
Checkout Exchange Rates API
Url will look like https://blockchain.info/de/ticker
Working script:
const request = require('request');
function BTC() {
// send a request to blockchain
request('https://blockchain.info/de/ticker', (error, response, body) => {
// parse the json answer and get the current bitcoin value
const data = JSON.parse(body);
value = (parseInt(data.THB.buy, 10) + parseInt(data.THB.sell, 10)) / 2;
console.log(value);
});
}
BTC();
Using the value as callback:
const request = require('request');
function BTC() {
return new Promise((resolve) => {
// send a request to blockchain
request('https://blockchain.info/de/ticker', (error, response, body) => {
// parse the json answer and get the current bitcoin value
const data = JSON.parse(body);
value = (parseInt(data.THB.buy, 10) + parseInt(data.THB.sell, 10)) / 2;
resolve(value);
});
});
}
BTC().then(val => console.log(val));
As the other answer stated, you should really use an API. You should also think about what type of price you want to request. If you just want a sort of index price that aggregates prices from multiple exchanges, use something like the CoinGecko API. Also if you need real-time data you need a websocket-based API, not a REST API.
If you need prices for a particular exchange, for example you're building a trading bot for one or more exchanges, you;ll need to communicate with each exchange's websoceket API directly. For that I would recommend something like the Coygo API, a node.js package that connects you directly to each exchange's real-time data feeds. You want something that doesn't add a middleman since that would add latency to your data.