We Keep Coding

Using Node.js to find the value of Bitcoin on a webpage at real time

I'm trying to make a .js file that will constantly have the price of bitcoin updated (every five minutes or so). I've tried tons of different ways to web scrape but they always output with either null or nothing. Here is my latest code, any ideas?
var express = require('express');
var path = require('path');
var request = require('request');
var cheerio = require('cheerio');
var fs = require('fs');
var app = express();
var url = 'https://blockchain.info/charts/';
var port = 9945;
function BTC() {
request(url, function (err, res, body) {
var $ = cheerio.load(body);
var a = $(".market-price");
var b = a.text();
console.log(b);
})
setInterval(BTC, 300000)
}
BTC();
app.listen(port);
console.log('server is running on '+port);
It successfully says what port it's running on, that's not the problem. This example (when outputting) just makes a line break every time the function happens.
UPDATE:
I changed the new code I got from Wartoshika and it stopped working, but im not sure why. Here it is:
function BTCPrice() {
request('https://blockchain.info/de/ticker', (error, response, body) => {
const data = JSON.parse(body);
var value = (parseInt(data.USD.buy, 10) + parseInt(data.USD.sell, 10)) / 2;
return value;
});
};
console.log(BTCPrice());
If I have it console.log directly from inside the function it works, but when I have it console.log the output of the function it outputs undefined. Any ideas?

I would rather use a JSON api to get the current bitcoin value instead of an HTML parser. With the JSON api you get a strait forward result set that is parsable by your browser.
Checkout Exchange Rates API
Url will look like https://blockchain.info/de/ticker
Working script:
const request = require('request');
function BTC() {
// send a request to blockchain
request('https://blockchain.info/de/ticker', (error, response, body) => {
// parse the json answer and get the current bitcoin value
const data = JSON.parse(body);
value = (parseInt(data.THB.buy, 10) + parseInt(data.THB.sell, 10)) / 2;
console.log(value);
});
}
BTC();
Using the value as callback:
const request = require('request');
function BTC() {
return new Promise((resolve) => {
// send a request to blockchain
request('https://blockchain.info/de/ticker', (error, response, body) => {
// parse the json answer and get the current bitcoin value
const data = JSON.parse(body);
value = (parseInt(data.THB.buy, 10) + parseInt(data.THB.sell, 10)) / 2;
resolve(value);
});
});
}
BTC().then(val => console.log(val));

As the other answer stated, you should really use an API. You should also think about what type of price you want to request. If you just want a sort of index price that aggregates prices from multiple exchanges, use something like the CoinGecko API. Also if you need real-time data you need a websocket-based API, not a REST API.
If you need prices for a particular exchange, for example you're building a trading bot for one or more exchanges, you;ll need to communicate with each exchange's websoceket API directly. For that I would recommend something like the Coygo API, a node.js package that connects you directly to each exchange's real-time data feeds. You want something that doesn't add a middleman since that would add latency to your data.

Unable to get information from <div> Node spider with Cheerio

I'm trying to download the lat/long locations of CCTV locations from the City of Baltimore website (project on the surveillance state) but not getting the console to log anything.
Here's the site:
and my code is:
const request = require('request');
const cheerio = require('cheerio');
let URL = 'https://data.baltimorecity.gov/Public-Safety/CCTV-Locations/hdyb-27ak/data'
let cameras = [];
request(URL, function(err, res, body) {
if(!err && res.statusCode == 200) {
let $ = cheerio.load(body);
$('div.blist-t1-c140113793').each(function() {
let camera = $(this);
let location = camera.text();
console.log(location);
cameras.push(location);
});
console.log(cameras);
}
});
I've tried setting the to blist-t1-c140113793 and blist-td blist-t1-c140113793 but neither has worked.

That's because data for those divs are loaded asynchronously, after the page was rendered. JavaScript is not executed by Cherrio, or any other such library. You'll need either to analyze network traffic and understand which HTTP call loads this data, or use something like Selenium, that actually executes JavaScript inside the browser.

Request for api not working properly

I'm making a request but it doesn't seem to work. If I copy code into my browser it works good, but in my console it shows up this :
{
"status" : "success",
"data" : {
"error_message" : "API access enabled, but unable to verify two-factor authentication code. If you need help with this, please contact support#bitskins.com."
}
}
What am I doing wrong? It's based on two-factor authentication that as I said works good while printing the url itself and when i'm copying it into my browser.
var url = 'https://bitskins.com/api/v1/get_item_price/?api_key='+bitskins.apikey+'&code='+bitskins.code+'&names='+encodeURIComponent(items[i].market_hash_name)+'&delimiter=!END!';
console.log(url);
request(url, function (error, response, body) {
if (!error) {
console.log(body)
}
});
In case you want, here is my api key module to generating it (api key deleted for security)
var TOTP = require('onceler').TOTP;
//Create a TOTP object with your secret
var totp = new TOTP('deleted');
// print out a code that's valid right now
// console.log(totp.now());
var code = totp.now();
module.exports = {
code: code,
apikey: 'deleted'
}

Founder of BitSkins, Inc. here. You need to have the following:
1) Your API Key
2) Your Secure Access Secret
You see the Secret when you enable Secure Access. If you do not have this, just disable/re-enable Secure Access and note the Secret down. The TOTP code you generate is with that Secret. Generate the TOTP code right before every API call and you'll be fine.

I think it should work. For me it works fine.
var API_KEY = ''; //It is very important
var SECRET_KEY = ''; //It is very important
var totp = new TOTP(SECRET_KEY);
var code = totp.now();
var options = {
url: 'https://bitskins.com/api/v1/get_item_price',
form: {
'api_key': API_KEY,
'names': 'Tec-9%20%7C%20Sandstorm%20(Minimal%20Wear)',
'delimiter': '!END!',
'code': code
}
};
function callback(error, response, body) {
if (!error) {
var info = JSON.parse(body);
console.log(info);
}
}
request.post(options, callback);

What npm package do you use to create 2FA code? I'm using "onceler" from example but I think it creates wrond codes. Here is my code:
var API_KEY = ''; //correct key from settings page
var SECRET_KEY = ''; // correct key which I copied from form with QR code.
var totp = new TOTP("SECRET_KEY");
var code = totp.now();
This code doesn't equal code which I can see in my mobile device and with this code I get error message like in author's question. But if I put code from my mobile in programm code - it works fine. So what package should I use to get correct codes?

How to use promises to return the final result of an array?

I am currently trying to return a request of all the file names (in each existing folder) on a particular website. My web application is using NodeJS, Express, Cheerio, and Request to web scrape. My code is first getting a list of all the folder names. After retrieving a list of folder names, it then goes inside each folder name to get a list of file names and store them in the 'files' array. Finally, the 'files' array is what will be sent to the client-side.
Right now I am having a big issue with asynchronous stuff because my request would always return an empty list of 'files'. I have the Q node module installed and have tried using promises, but have had no luck getting the results I want. I am still new to nodeJS and would love it if someone can help me out.. :)
exports.getAllImages = function(req, res) {
var folders = [];
var files = [];
//Step 1: Get folder names and store all of them in the 'folders' array
var foldersUrl = 'http://students.washington.edu/jmzhwng/Images/';
request(foldersUrl, function(error, response, html){
if(!error){
var $ = cheerio.load(html);
$("a:contains('-')").filter(function(){
var data = $(this)[0].attribs.href;
folders.push(data);
})
//Step 2: Using the 'folders' array, get file names in each folder and store all of them in the 'files' array
for (var i=0; i < folders.length; i++) {
var imagesUrl = 'http://students.washington.edu/jmzhwng/Images/' + folders[i];
request(imagesUrl, function(error, response, html){
if(!error){
var $ = cheerio.load(html);
$("a:contains('.')").filter(function(){
var data = $(this)[0].attribs.href;
files.push(data);
})
}
})
}
//Step 3: Return all file names to client-side
res.json({
images: files
}, 200);
console.log('GET ALL IMAGES - ' + JSON.stringify(files));
}
})
For better readability or support, you can view the JSFiddle I created here: http://jsfiddle.net/fKGrm/

You don’t necessarily need promises for this—you’re 95% of the way there already without them. The main issue, as I think you’re aware, is that your response is being sent before the image requests come back. You just need to wait for those to finish before you send the response.
The most basic way is to count the number of callbacks you receive in your Step 2. When it equals the folders.length, then send your response.
Here’s a simplified version of that:
var request = require('request'),
cheerio = require('cheerio');
var baseUrl = 'http://students.washington.edu/jmzhwng/Images/';
var files = [];
request(baseUrl, function (error, res, body) {
var folders = folderLinks(cheerio.load(body));
count = 0;
folders.forEach(function (folder) {
request(baseUrl + folder, function (error, res, body) {
files.push.apply(files, fileLinks(cheerio.load(body)));
if (++count == folders.length) {
console.log(files);
}
});
});
});
function folderLinks ($) {
return $('a:contains(-)').get().map(function (a) {
return a.attribs.href;
});
}
function fileLinks ($) {
return $('a:contains(.)').get().map(function (a) {
return a.attribs.href;
});
}

JSONP support with node.js and mongo

I'm attempting to create a simple web service using node.js, express, monk, and mongodb which returns results from mongodb based on the params in the URL. I want to add jsonp support to the calls. The service will be called as such:
localhost:3000/database/collection/get?param1=Steve&param2=Frank&callback=foo
app.js
var mongo_address = 'localhost:27017/database';
var db = monk(mongo_address);
app.get('/:coll/get', routes.handle(db);
routes/index.js
exports.handle = function(db) {
return function(req, res) {
// Send request
db.get(req.params.coll).find(req.query, {fields:{_id:0}}, function(e,docs) {
if (e) throw e;
res.jsonp(docs)
});
};
}
When I use the built in JSONP support with res.jsonp, it sends the callback param to mongo and returns an empty list. I've tried stripping out the callback param during the query and then manually adding it back to the results without much luck. I feel like I'm missing something simple. Any help would be appreciated.

After some messing around with JS, I found a workable solution with minimal additional code. AFter stripping the callback from the query and storing the function value, I had to explicitly build the return string for JSONP requests.
exports.handle = function(db) {
return function(req, res) {
//Determine if URL implements JSONP
var foundCallback = req.query.callback;
var callbackVar;
//If asking for JSONP, determine the callback function name and delete it from the map
if (foundCallback){
callbackVar = req.query.callback;
delete req.query.callback
}
// Send request
db.get(req.params.coll).find(req.query, {fields:{_id:0}}, function(e,docs) {
if (e) throw e;
//If callback, send function name and query results back, else use express JSON built in
if (foundCallback)
res.send('typeof ' + callbackVar + ' === \'function\' && ' + callbackVar + '(' + JSON.stringify(docs) + ');');
else
res.json(docs);
});
};
}

Try
app.set("jsonp callback", true);