My overall goal is the following:
-> Get data from fabric.io (=crashlytics) into a geckoboard dashboard.
To my knowledge there is no API on crashlytics/fabric side, so I had the following idea:
Write a script in nodejs (because I know node js a bit, I'm no expert though) that would:
Open the html page where the data I want is
Find my data using htmlparser
Save that data into a google sheet
Make geckoboard read the google sheet and display the data
Step 4 is done already and I'm working on step 1 now.
Unfortunately I'm having a problem as the page is not publicly accessible, I need to authenticate using my user account.
I've reused some code that works just fine when doing GET/POST/PUT on some other websites' REST api, but it doesn't seem to work here as fabric is redirecting me to the login page.
However when I search the web for node auth, I find modules for people to create a server that will handle authentication, whereas I'm trying to use node to login to a website.
It's well possible that what I'm trying to achieve or the way I'm trying to do it don't make sense at all. But as I'm not skilled enough to realise that, I'd be happy if someone could confirm it to me. At least I'd know I'm looking into the wrong direction ;-)
Thanks for reading me
Here is my code:
var fs = require('fs');
var https = require('https');
var creds = require('./config/credentials.js');
var date = new Date();
var htmlpage = "";
var authorizationHeader = "Basic " + new Buffer(creds.login + ":" + creds.pwd).toString("base64");
var get_options = {
hostname: "fabric.io",
path: "/my-company-account/ios/apps/app.identifier/answers/stability",
method: "GET",
port: 443,
headers: {
"content-type": "application/json",
"Authorization": authorizationHeader
}
};
var get_req = https.request(get_options, function(res){
res.setEncoding('utf8');
res.on('data', function(chunk){
htmlpage += chunk;
}); // end res.on 'data'
res.on('end', function(){
console.log(htmlpage);
//tmp debug
fs.writeFile('./logs/ac-ios_' + date.getTime() + '.html', htmlpage, function(err){
if (err) {
console.log(err);
}
}); // end write html file
/*var parser = new htmlparser.Parser(handler);
parser.parseComplete(htmlpage);*/
}); // end res.on 'end'
}); // end https.request
get_req.on("error", function(err){
console.log(err);
});
get_req.end();
Which gives me the following html:
<html>
<body>You are being redirected.</body>
</html>
Related
this is my first time posting a question to stack overflow since I have an issue I cant get past.
What I am trying to do:
I am trying to extract data of my car from the "Mercedes me" platform. It's a pure hobby project.
"Mercedes me" has an API with oAuth2 verification.
I am trying to realize this in Google Sheets since I want to do calculations with the data retrieved.
Hence I started coding in Apps Script.
I am able to exchange an Authentification Code with a Token and then get data from the platform using the following code:
var options = {
headers : { Authorization: 'Basic '+ Utilities.base64Encode('Client ID:Client Secret')},
method : 'post',
'content-type' : 'application/x-www-form-urlencoded',
muteHttpExceptions : false,
}
var authorization_code = 'TGVWD9pNif1wuDBYa6fFPn4QHNW9h6_f2-kxL34V'
var authUrl = 'https://id.mercedes-benz.com/as/token.oauth2?grant_type=authorization_code&code='+authorization_code+'&redirect_uri=https://localhost/';
var response = UrlFetchApp.fetch(authUrl, options);
var data = JSON.parse(response);
var token = data.access_token
'Logger.log(token);'
var options = {
headers : { Authorization: `Bearer ${token}`},
accept : 'application/json'
}
var response = UrlFetchApp.fetch('https://api.mercedes-benz.com/vehicledata/v2/vehicles/<vehicle identification number>/resources/', options);
'var jsonObject = JSON.parse(response.getContentText()); '
'var jsonObject = JSON.stringify(response);'
Logger.log(response);
So this is not the issue. The issue is that currently I need to manually insert the Authorization Code (as you can see in the code snippet above) which will then be exchanged with the Token by my script. The goal would be to generate the Authorization Code automatically.
I have created an authorization URL which will give me the Authorization Code when entered and opened in the Browser.
However, when I try to achieve the same in Apps Script via UrlFetchApp it will send me back some unusable html code instead of an Authorization Code.
function Authentification() {
var url = 'https://id.mercedes-benz.com/as/authorization.oauth2?response_type=code&client_id=a9f67f5f-cb2b-413b-9ea5-78df02b668a1&redirect_uri=https://localhost/&scope=mb:vehicle:mbdata:payasyoudrive&state=1000';
var options = {
muteHttpExceptions : false, //theoretisch nicht notwendig
followRedirects : true, //theoretisch nicht notwendig
method : 'get', //theoretisch nicht notwendig
}
var response = UrlFetchApp.fetch(url, options);
var json = response.getHeaders;
var data = JSON.stringify(json);
Logger.log(response)
}
Result is some html like this:
<!DOCTYPE html><html lang=en xmlns=http://www.w3.org/1999/xhtml><head><meta charset=utf-8><m
But what I am trying is to get a redirect Url like this (including the Authorization Code):
https://localhost/?code=DXPzKdV58t3rWlpuVmD3XT35BbL__n0auuAxL34V&state=1000
My assumption is, the problem is that when the link is opened by Apps Script's URLFetchApp it is never asked to enter my "Mercedes me" login credentials. Most likely I am not seing a basic OAuth logic here. Would really appreciate your help. Thanks a lot in advance.
Best Regards,
Chris
The solution came to my mind in my sleep :-D
I deployed the whole thing as a webapp and used the webapp-link as the redirect uri.
After this I was able to extract the authorization code from the url with the following code:
function doGet(e) {
var param = e.queryString;
param = e.parameter;
var code = param.code;
I'm writing an AWS Lambda in node.js 6.10 for a school project with Amazon's Alexa software, and I don't have much experience with Javascript and none with JSON. My school has a transportation API for finding if it is up at: https://prtstatus.wvu.edu/api/[TENDIGITTIMESTAMP]/?format=json
If I go there with the stamp, I get "{"status":"7","message":"The PRT is closed.","timestamp":"1494028926","stations":[],"bussesDispatched":"0","duration":[]}"
What I am trying to get is the message and relay it to something else (I've got that part covered). What I don't know is how to break up the JSON response from the URL or write a request in the first place. Can someone help me figure out what to write to use the "message" string in my project?
So far I have:
'getPRTStatus': function() {
var date = Math.round(new Date().getTime()/1000);
//this is the spot where I need help filling in
//var object = JSON.parse('http://prtstatus.wvu.edu/api/'+date+'/?format=json');
this.attributes.speechOutput = this.t(object.message);
this.attributes.repromptSpeech = this.t(object.message);
this.emit(':ask', this.attributes.speechOutput, this.attributes.repromptSpeech);
},
Thanks for your help!
Is it possible for you to post the JSON response here from the URL because that would help a lot to narrow down the issue.
Update
You need to make an http get request to the API endpoint. You won't get a JSON response with,
var url = "http://prtstatus.wvu.edu/api/"+date+"/?format=json"
You can use a package like https://www.npmjs.com/package/request Check out their documentation on how you can make it work.
Something like this,
var options = {
"method": "get",
"url": "http://prtstatus.wvu.edu/api/1501906657/?format=json",
}
request(options, function(err, response, body) {
if (err) {
console.log(err)
} else {
console.log(body);
}
Another Update
You can try something like,
var request = require('request'); //Import the NPM package
var object; //global variable to be used later on to store the response
Then in your function,
'getPRTStatus': function() {
var date = Math.round(new Date().getTime()/1000);
var options = {
'method' : 'get',
'url' : 'http://prtstatus.wvu.edu/api/' + date + '/?format=json'
};
request(options, function(err, response, body){
if(err) {
console.log(err);
}
else {
object = JSON.parse(body); //You got the response parsed & stored in the global variable named object
}
});
this.attributes.speechOutput = this.t(object.message);
this.attributes.repromptSpeech = this.t(object.message);
this.emit(':ask', this.attributes.speechOutput,
this.attributes.repromptSpeech);
}
Just updated my answer according to your question. Hope that helps. For any future API related issues, you should try Postman in chrome. I'll post a link on how to get started with that. You will also get the direct code of your API call in postman.
Link to postman app: https://chrome.google.com/webstore/detail/postman/fhbjgbiflinjbdggehcddcbncdddomop?utm_source=gmail
I'm new to node.js and I'm trying out a few easy examples using localhost:XXXX.
I want to read my request object from node. I have a book and in the book they use cURL(some program) to comunicate with node instead of the browser. Is it possible to write something in the browser adress field and send it to localhost and have a request object sent to node that looks exactly like if I had a typed in a url to a server somewhere? OIf so, how do I write? Do I have to use cURL or something like it if i use localhost?
I'm very new to node and javascript so I dont't know if I'm using the right words. I have tried to search but I dont't think I know the right terms to search for.
This is my server code:
var port = 3000;
http.createServer(function (req, res) {
var url = parse(req.url);
res.writeHead(200, { 'Content-Type': 'text/plain' });
res.end('Hello World\n' + url );
}).listen(port);
When i write http://localhost:3000/hello.com in the address field i was hoping i would get Hello world hello.com in the browser but i get hello world [object object]
Please help.
You can use your regular browser by testing it. In your URL address enter URL address that you have in your cURL address. For instance:
localhost:3000/index.html
If you would like to have more sophisticated tool that gives you more information about request/response you can use tool like Postman for that
In your code use:
res.end('Hello World\n' + url.parse(req.url, true));
url is an object, you need to specify property or method that you are calling on it.
Here is an example on how to parse URL. Easy URL Parsing With Isomorphic JavaScript:
Above answer given by #Vlad Beden looks good but you may want to play with following code
var http = require("http");
var port = 3000;
http.createServer(function (req, res) {
console.log('Requested method: ', req.method);
var params = parseUrl(req.url);
res.writeHead(200, { 'Content-Type': 'text/plain' });
var data = 'Hello World'
for(var i=0; i<params.length; i++)
data += '\n'+params[i]
res.end(data);
}).listen(port);
var parseUrl = function(url) {
var params = [];
if(url && url != '' && url != '/') {
url = url.replace(/^\/|\/$/g, '');
params = url.split('/');
}
return params;
}
You may try http://localhost:3000/hello.com or http://localhost:3000/hello.com/google.com/more.com/etc . I would like to recommend you print request object console.log(req) and have a look to understand url, method, headers, etc.
I'm using Node + Express. On page load, the app calls a remote database, gets data and sends it to a handlebars template. All this is done server side. But I'd like to be able to have this same JSON data be available for the client to interact with. How do I do that?
Example, server displays a table of ten records. I want the client to be able to click on one record and get a details view of just that one record. Thanks.
Here's the code:
app.get('/', function(req, res) {
getDataFromDatabase(function(data) {
data = JSON.parse(data);
res.render('index', {
stuff: data
});
});
});
function getDataFromDatabase(callback) {
var options = {
hostname: this.hostname,
path: this.path,
port: 80,
method: 'GET'
}
http.request(url, function(res) {
var data = '';
res.on('data', function(chunk) {
data += chunk;
}).on('end', function() {
callback(data);
}).on('error', function() {
console.log("error");
})
}).end()
}
How do you get the data as a variable accessible to the client, on
page load, when the server is doing all of the work? Do you have to
make a redundant call from the client and get a "copy" of the data?
- June March
If you don't want to make an AJAX call after the page loads to get at the data, you can send it along with the rest of the page inside of a script tag. Not sure which template engine you're using so the following is sudo code (might be Jade, I don't know). Do something like this in your template:
script var data = JSON.stringify(stuff); // <- horrible variable name btw
If you can successfully create a script tag in your template, and initialize a variable with the data you want to pass to the client, you shouldn't have to make another call to the server.
I am looking to work with the twitter stream api in javascript and have a script that run successfully pulling from the streaming API. My question is what would be the best way for me to set this up so it can run constantly. Or would it be better to switch to the search API instead? I am just trying to collect tweets based on a few keywords, but I want to collect a load of them and store them into Mongolab. Would a Cron job be best for this? I am going to use openshift to handle the streaming and processing.
I think I am looking for guidance on the best route so I don't have to constantly monitor and check that it is collecting tweets.
Thank you!
var Twit = require('twit');
var MongoClient = require('mongodb').MongoClient;
var T = new Twit({
consumer_key: '***',
consumer_secret: '***',
access_token: '***',
access_token_secret: '***'
});
var url = "***";
MongoClient.connect(url, function (err, db) {
var col= db.collection('test');
// filter public stream on keywords
var stream = T.stream('statuses/filter', {track: ['#food', 'drinks']
});
stream.on('tweet', function (data) {
console.log("tweet: " + data);
col.insert(data, function (err, result) {
if (!err) {
console.log("insert successful on tweet: " + data.id);
} else {
console.log(err);
}
});
});
});
Someone else might be able to provide a better answer but, I think using the cronjob add-on cart would be the way to go. You could set it up to execute jobs on your cartridge in timed intervals and thus would keep you from having to pull things manually. Here's a OpenShift blog article that can help you get started https://blog.openshift.com/getting-started-with-cron-jobs-on-openshift/.