So I've been trying to call this one website for a json of event data and I can't quite figure out how to get the first chunk of data in the json. This is the website https://api.tftech.de/Event/ I'm only trying to pull this data and then make each one of the values into a variable. [{"name": "Spooky Event","start": "2020-07-27T02:35:00-05:00","end": "2020-07-27T03:35:00-05:00","priority": "HIGH","color": "ff9800","intervalInMinutes": 7440}] Can anyone help?
You can use axios to make requests,
Here is the code block with the results put into variables
var axios = require("axios")
axios.get("https://api.tftech.de/Event/").then(data => {
var name = data.data[0].name;
var start = data.data[0].start;
var end = data.data[0].end;
var priority = data.data[0].priority;
var color = data.data[0].color;
var intervalInMinutes = data.data[0].intervalInMinutes;
})
Related
I have an application developing using Nodejs. This application is making a request to GitLab API and obtaining the raw file data from it.
I would like to read the particular string which is present after another string and get all similar data from it. I am just a bit confused on this part and unable to proceed further can someone please explain to me how to achieve this?
Following is the sample file data:
I would like to read all the numbers if present after the keyword Scenario: i.e in this case I would like to get A001-D002 & K002-M002. These numbers can be anything random and can appear anywhere within the file content. I would like to read them and store them within an array for that particular file.
FileName: File Data
Background:
This is some random background
Scenario: A001-D002 My first scenario
Given I am sitting on a plane
When I am offered drinks
Scenario: K002-M002 My second scenario
Given when I book the uber taxi
When I get the notifications
I am not understanding how to iterate over the file content and read every word and match and accordingly obtain the ids.
Following is the code that I have which makes the request to GitLab and obtains the raw file content:
./index.js:
const express = require('express');
const http = require("http");
const bodyParser = require('body-parser');
const app = express();
const port = process.env.PORT || 9000;
const gitlabDump = require("./controller/GitLabDump");
app.use(bodyParser.json());
app.use(bodyParser.urlencoded({ extended: true }));
//Make NodeJS to Listen to a particular Port in Localhost
app.listen(port, function(){
gitlabDump.gitlabDump(type, function(data){
console.log("Completed Execution for GitLab")
process.exit();
})
}
My ./controller/GitLabDump.js:
const request = require('request');
const https = require('https');
const axios = require('axios');
exports.gitlabDump = function(callback){
var gitlabAPI = "https://gitlab.com/api/v4/projects/<project_id>/repository/files/tree/<subfolders>/<fileName>/raw?ref=master&private_token=<privateToken>";
//Make the request to the each file and read its raw contents
request(gitlabAPI, function(error, response, body) {
const featureFileData = JSON.parse(JSON.stringify(body)).toString();
console.log(featureFileData)
for(const match of featureFileData.matchAll("Scenario:")){
console.log(match);
}
callback("Completed");
})
}
I am able to print the file contents. Can someone please explain me how can I iterate over the raw file contents and get all the required ids?
I suggest you to use a method by analyzing each part of your string by iterating over each lines (i assume that your string is compose like in your exemple). It is easier to understand and coding it than using a regex.
The exemple below represent your request callback function.
I split the code in 3 logics :
search the filename
search the line we are interesting with ("Scenario" word)
extract the ID by filter function
You can after that, easily change you ID filter (txt.substr(0, txt.indexOf(' ')) to use a more proper expression to extract your sentence.
The result is sent to a callback function with as first argument the filename, and as second all ids. Like you did in your exemple.
((callback) => {
const featureFileData = `FileName: File Data
Background:
This is some random background
Scenario: A001-D002 My first scenario
Given I am sitting on a plane
When I am offered drinks
Scenario: K002-M002 My second scenario
Given when I book the uber taxi
When I get the notifications`;
// find "filename"
const filenames = featureFileData.split('\n')
.filter(line => line.trim().substr(0,8) === 'FileName')
.map((raw) => {
if(!raw) return 'unknown';
const sentences = raw.trim().split(':');
if(sentences[1] && sentences[1].length) {
return sentences[1].trim();
}
});
// filter the "Scenario" lines
const scenarioLines = featureFileData.split('\n')
.map((line) => {
if(line.trim().substr(0,8) === 'Scenario') {
const sentences = line.trim().split(':');
if(sentences[1] && sentences[1].length) {
return sentences[1].trim();
}
}
return false;
})
.filter(r => r !== false);
// search ids
const ids = scenarioLines.map(txt => txt.substr(0, txt.indexOf(' ')));
callback(filenames[0], ids);
})(console.log)
So I had been stuck on this exercise in Treehouse some time ago and just moved on. I came back to it now that I understand things better and I'm still fighting with the wunderground api. I've read through the json data and documentation, updated a few things from when the class was first recorded (and the API updated since then), and still am getting errors I can't field. I've got three js files- app.js, weather.js, and api.json (which is just my api key so not shared here.)
After my corrections, I'm still getting the error "TypeError: Cannot read property 'temp_f' of undefined" which doesn't make sense as I keep reading over the JSON to check that it's pointing to the right place.
Can anyone put an end to my misery trying to fix this?
App.js:
const weather = require('./weather');
//Join multiple values passed as arguments and replace all spaces with underscores
const query = process.argv.slice(2).join("_").replace(' ', '_');
//query: 90201
//query: Cleveland_OH
//query: London_England
weather.get(query);
Weather.js
const https = require('https');
const http = require('http');
const api = require('./api.json');
// Print out temp details
function printWeather(weather) {
const message = `Current temp in ${weather.location} is ${weather.current_observation.temp_f}F`;
console.log(message);
}
// Print out error message
function get(query) {
const request = http.get(`http://api.wunderground.com/api/${api.key}/conditions/q/${query}.json`, response => {
let body = "";
// Read the data
response.on('data', chunk => {
body += chunk;
});
response.on('end', () => {
//Parse data
const weather = JSON.parse(body);
//Print the data
printWeather(weather);
});
});
}
module.exports.get = get;
//TODO: Handle any errors
I have followed a tutorial and expanded on it to make this twitter bot. It takes the latest 100 tweets from the given twitter account and gives me the dates they were posted when I run the program in terminal. This bit works perfectly.
The 3 lines at the bottom are used to write the data to a spreadsheet although it only gives me one result. I'm wanting to know if there's anything I have done wrong or if there is anything I can add that will give me all the results in the spreadsheet?
This is the code I have written
//API
var Twit = require('twit');
// Getting the API keys from the config file.
var config = require('./config');
var T = new Twit(config);
//getting 5 tweets from the username entered in the top username.
//can get up to 3200 tweets
var params = {
screen_name: 'mascott10',
count: 100
}
T.get('statuses/user_timeline', params, gotData);
//function to be triggered when the data is collected.
function gotData(err, data, response) {
var tweets = data;
for (var i = 0; i < tweets.length; i++) {
console.log(tweets[i].created_at);
var date_posted = tweets[i].created_at.toString();
}
//console.log(data);
//write to file
var fs = require('fs');
fs.writeFile("/Users/mikey/Desktop/node/test.xls", date_posted);
};
This is some of the results in terminal
terminal view of bot result
This is the result in the spreadsheet
spreadsheet result
Your code is over-writing date_posted on each iteration of the for loop. Define your date_posted variable outside of the loop:
var date_posted;
And then inside of the loop concatenate each successive date:
date_posted += tweets[i].created_at.toString();
I'm trying to make a .js file that will constantly have the price of bitcoin updated (every five minutes or so). I've tried tons of different ways to web scrape but they always output with either null or nothing. Here is my latest code, any ideas?
var express = require('express');
var path = require('path');
var request = require('request');
var cheerio = require('cheerio');
var fs = require('fs');
var app = express();
var url = 'https://blockchain.info/charts/';
var port = 9945;
function BTC() {
request(url, function (err, res, body) {
var $ = cheerio.load(body);
var a = $(".market-price");
var b = a.text();
console.log(b);
})
setInterval(BTC, 300000)
}
BTC();
app.listen(port);
console.log('server is running on '+port);
It successfully says what port it's running on, that's not the problem. This example (when outputting) just makes a line break every time the function happens.
UPDATE:
I changed the new code I got from Wartoshika and it stopped working, but im not sure why. Here it is:
function BTCPrice() {
request('https://blockchain.info/de/ticker', (error, response, body) => {
const data = JSON.parse(body);
var value = (parseInt(data.USD.buy, 10) + parseInt(data.USD.sell, 10)) / 2;
return value;
});
};
console.log(BTCPrice());
If I have it console.log directly from inside the function it works, but when I have it console.log the output of the function it outputs undefined. Any ideas?
I would rather use a JSON api to get the current bitcoin value instead of an HTML parser. With the JSON api you get a strait forward result set that is parsable by your browser.
Checkout Exchange Rates API
Url will look like https://blockchain.info/de/ticker
Working script:
const request = require('request');
function BTC() {
// send a request to blockchain
request('https://blockchain.info/de/ticker', (error, response, body) => {
// parse the json answer and get the current bitcoin value
const data = JSON.parse(body);
value = (parseInt(data.THB.buy, 10) + parseInt(data.THB.sell, 10)) / 2;
console.log(value);
});
}
BTC();
Using the value as callback:
const request = require('request');
function BTC() {
return new Promise((resolve) => {
// send a request to blockchain
request('https://blockchain.info/de/ticker', (error, response, body) => {
// parse the json answer and get the current bitcoin value
const data = JSON.parse(body);
value = (parseInt(data.THB.buy, 10) + parseInt(data.THB.sell, 10)) / 2;
resolve(value);
});
});
}
BTC().then(val => console.log(val));
As the other answer stated, you should really use an API. You should also think about what type of price you want to request. If you just want a sort of index price that aggregates prices from multiple exchanges, use something like the CoinGecko API. Also if you need real-time data you need a websocket-based API, not a REST API.
If you need prices for a particular exchange, for example you're building a trading bot for one or more exchanges, you;ll need to communicate with each exchange's websoceket API directly. For that I would recommend something like the Coygo API, a node.js package that connects you directly to each exchange's real-time data feeds. You want something that doesn't add a middleman since that would add latency to your data.
Here is the situation:
I have a Rails app that scrapes a website and outputs valid JSON to a restful API endpoint.
So far I can get the JSON with my Node.js script, but I cannot store the values I need to local variables.
Here is the json:
[{"cohort":"1507","teacher":"Josh M."}]
I would like to store "1507" in a variable as well as "Josh M."
My current script is:
var http = require('http');
var url = 'http://localhost:8080/api/v1/classroom_bs';
http.get(url, function(res){
var body = '';
res.on('data', function(chunk){
body += chunk;
});
res.on('end', function(){
var responseB = JSON.parse(body);
var responseBStr = JSON.stringify(body);
var jsonB = JSON.parse(responseBStr);
console.log(responseB);
console.log(responseBStr);
console.log(jsonB);
});
}).on('error', function(e){
console.log("Error: ", e);
});
Any help would be greatly appreciated!
I have tried some functions I found on SO, but for some reason all my console.log(Object.keys) return numbers like "1", "2", "3", etc..
The reason I need to store these as variables is that I am going to output those values to an LCD screen using the johnny5 library via an Arduino.
The rails app that you are using to send to node is an array of length 1, which is an object. To access that object you have to say which array element you are accessing. In this case array[0]. It appears your rails scraper is populating an array.
To access the variables you mentioned you would simply
//Access that 1507
var variable1 = payloadFromRails[0].cohort;
//Access that teach, Josh M.
var variable2 = payloadFromRails[0].teacher;
Try selecting object at index 0 of parsed json string
var response = '[{"cohort":"1507","teacher":"Josh M."}]';
var responseB = JSON.parse(response);
var keys = Object.keys(responseB[0]);
var cohort = responseB[0][keys[0]];
var teacher = responseB[0][keys[1]];
console.log(cohort, teacher)