I'm trying to request this website but I keep getting a 400 Bad Request error. This code works for just about any other site I've tried that isn't built with squarespace so I'm guessing that's where the problem is.
var request = require('request');
var cheerio = require('cheerio');
var url = 'http://www.pond-mag.com/';
request(url, function(error, resp, body){
if(!error){
var $ = cheerio.load(body);
console.log(body);
}
});
Figured it out just had to manually set the headers object.
Heres the code that fixed it in case anyone else has the problem:
var options = {
url : 'http://www.pond-mag.com/',
headers: {
'User-Agent': 'request'
}
};
Then, just pass the options var to the request instead of the url.
I use Squarespace for a lot of different apps and thought it was worth mentioning that Squarespace has native support for getting the JSON of any Squarespace page. If you append ?format=json to the URL you can pull make the request and get JSON back.
Related
I'm writing an AWS Lambda in node.js 6.10 for a school project with Amazon's Alexa software, and I don't have much experience with Javascript and none with JSON. My school has a transportation API for finding if it is up at: https://prtstatus.wvu.edu/api/[TENDIGITTIMESTAMP]/?format=json
If I go there with the stamp, I get "{"status":"7","message":"The PRT is closed.","timestamp":"1494028926","stations":[],"bussesDispatched":"0","duration":[]}"
What I am trying to get is the message and relay it to something else (I've got that part covered). What I don't know is how to break up the JSON response from the URL or write a request in the first place. Can someone help me figure out what to write to use the "message" string in my project?
So far I have:
'getPRTStatus': function() {
var date = Math.round(new Date().getTime()/1000);
//this is the spot where I need help filling in
//var object = JSON.parse('http://prtstatus.wvu.edu/api/'+date+'/?format=json');
this.attributes.speechOutput = this.t(object.message);
this.attributes.repromptSpeech = this.t(object.message);
this.emit(':ask', this.attributes.speechOutput, this.attributes.repromptSpeech);
},
Thanks for your help!
Is it possible for you to post the JSON response here from the URL because that would help a lot to narrow down the issue.
Update
You need to make an http get request to the API endpoint. You won't get a JSON response with,
var url = "http://prtstatus.wvu.edu/api/"+date+"/?format=json"
You can use a package like https://www.npmjs.com/package/request Check out their documentation on how you can make it work.
Something like this,
var options = {
"method": "get",
"url": "http://prtstatus.wvu.edu/api/1501906657/?format=json",
}
request(options, function(err, response, body) {
if (err) {
console.log(err)
} else {
console.log(body);
}
Another Update
You can try something like,
var request = require('request'); //Import the NPM package
var object; //global variable to be used later on to store the response
Then in your function,
'getPRTStatus': function() {
var date = Math.round(new Date().getTime()/1000);
var options = {
'method' : 'get',
'url' : 'http://prtstatus.wvu.edu/api/' + date + '/?format=json'
};
request(options, function(err, response, body){
if(err) {
console.log(err);
}
else {
object = JSON.parse(body); //You got the response parsed & stored in the global variable named object
}
});
this.attributes.speechOutput = this.t(object.message);
this.attributes.repromptSpeech = this.t(object.message);
this.emit(':ask', this.attributes.speechOutput,
this.attributes.repromptSpeech);
}
Just updated my answer according to your question. Hope that helps. For any future API related issues, you should try Postman in chrome. I'll post a link on how to get started with that. You will also get the direct code of your API call in postman.
Link to postman app: https://chrome.google.com/webstore/detail/postman/fhbjgbiflinjbdggehcddcbncdddomop?utm_source=gmail
I'm trying to scrape a simple website with Cheerio and Request
Here is my code:
import request from 'request';
request('http://michaelhyatt.com/page/2', function(err, res, html) {
console.log(html);
});
But the HTML that I get back is gibberish, some kind of weird encoded content:
���r� �lE�?��iSZb�,�DI�<��[k��-yy��v(#H�U������nE��y��y��9;��D����S֗�����M�duϲ�M�
H$�D"3��x����gg?�{����:�z���v�����4��7�c |���&����V��ڇ␌��3⎼�┌["�:��
What am I doing wrong? Other websites I have tried to scrape do not experience this issue.
I solve the same issue with axios, by just disable encoding on headers by:
const response = await axios.get(baseUrl, {
headers: {
"Accept-Encoding" : null
}
});
console.log(response)
The answer was found from the comments above from here
I am trying to use meteor together with iron-router for a simple RESTful API and I have verified that when I do a POST to my api if the header type is set to 'Content-Type: application/json' then I can parse the body of the http request using var thisVar = this.request.body.myKey, however if I don't specify the Content-Type of course the default is application/x-www-form-urlencoded and the body doesn't get parsed as json.
My question is then, how can I 'convert' my content to JSON as it really is JSON. Now, most of you might think "why not just change the content-type to json in the POST request". Well, the answer is that I do not control the POST, this is a 3rd party software. I could add an enhancement request, but that will take months and may not ever happen.
So, therefore my question again is: Within my iron-router Can I 'convert' the content-type of my body to json so that I can parse it easily using the format var thisVar = this.request.body.myKey
EDIT
Let me clarify, I already have a basic working RESTful API working with iron-router. When I use curl and POST with some json data and specify the content-type is json like so curl -XPOST -H "Content-Type:application/json" -d"{\"myKey\":\"myValue\"}" http://localhost:3000/api/insert then in my route for handling POSTs I can parse the body with var thisVar = this.request.body.myKey and then insert into my mongodb, etc...no problem.
However, the client is posting without specifying the content-type, so the default is application/x-www-form-urlencoded and so when I use the following code var thisVar = this.request.body.myKey the value thisVar is undefined so I cannot easily parse the body of the http request.
Make sense?
Yes, I am answering my own question but hopefully it will help someone out there.
After a series of just printing out the http response, I found out that my raw data had the following characteristics that prevented it from being parsed into a JSON object:
Backslash escapes of double-quotes
The entire raw string after 'JSON.stringified' was a key with surrounded double-quotes, and then basically an empty value.
Example: {"{ \"myKey\" : \"ValueA\"}":""}
I used the string.replace() function to fix #1 and #2. (see sample code below)
Example: { "myKey" : "ValueA"}
After applying fix #1 and #2 I could then do a JSON.parse() on the JSON string.
Here's my code with fixes #1 and #2:
Router.route('/api/insert/test', function(){
this.response.statusCode = 200;
this.response.setHeader("Content-Type", "application/json");
this.response.setHeader("Access-Control-Allow-Origin", "*");
this.response.setHeader("Access-Control-Allow-Headers", "Origin, X-Requested-With, Content-Type, Accept");
var re = /\\/g;
var re2 = /\{\"/g;
var re3 = /\":\"\"\}/g;
var json_str = JSON.stringify(this.request.body);
var json_str_after_re = json_str.replace(re, '').replace(re2, '').replace(re3, '');
var json = JSON.parse(json_str_after_re);
this.response.end('this is ' + json_str_after_re + ' and value is ' + json.myKey);
}, {where: 'server'});
Making a RESTful API with iron-router is pretty easy:
Router.route('/webhooks/stripe', { where: 'server' })
.get(function () {
// GET /webhooks/stripe
})
.post(function () {
// POST /webhooks/stripe
})
.put(function () {
// PUT /webhooks/stripe
})
Source
I have some code that is trying to get a JSON result from the Soundcloud API.
I registered an app, got the client id and such, and I'm trying to make a call like this:
var querystring = require('querystring');
var http = require('http');
var addr = 'http://api.soundcloud.com/resolve.json?url=http://soundcloud.com/matas/hobnotropic&client_id=XXXXX';
var options = {
hostname: "api.soundcloud.com",
path: "/resolve.json?url=http://soundcloud.com/matas/hobnotropic&client_id=XXXXXx",
method: "GET",
headers: {
"Content-Type": "application/json"
}
}
var request = http.get(options, function(response) {
response.setEncoding('utf8');
response.on('data', function(chunk) {
console.log(chunk);
});
});
This produces a result that looks like this:
{"status":"302 - Found","location":"https://api.soundcloud.com/tracks/49931.json?client_id=xxxx"}
When I use the same URL in Chrome, I get the proper JSON info. How do I properly make this call from server side script?
The built-in http client does not handle redirects. However request does and has many other features the built-in client does not support out of the box.
Today I updated my own NodeJS Api-Wrapper Package for Soundcloud, which can be found here: https://www.npmjs.com/package/soundcloud-nodejs-api-wrapper
It does server side API communication, which includes data modification. No user popup window and redirect url is needed.
I did not found yet any other package having support for this in NodeJS.
This is an annoying problem, and I don't suppose that it's only IE that has this problem. Basically I have a Node.js server, from which I am making cross-domain calls to get some JSON data for display.
This needs to be a JSONP call and I give a callback in the URL. What I am not sure is, how to do this?
So the website (domainA.com) has an HTML page with a JS script like this (all works fine in Firefox 3):
<script type="text/javascript">
var jsonName = 'ABC'
var url = 'http://domainB.com:8080/stream/aires/' //The JSON data to get
jQuery.getJSON(url+jsonName, function(json){
// parse the JSON data
var data = [], header, comment = /^#/, x;
jQuery.each(json.RESULT.ROWS,function(i,tweet){ ..... }
}
......
</script>
Now my Node.js server is very simple (I'm using express):
var app = require('express').createServer();
var express = require('express');
app.listen(3000);
app.get('/stream/aires/:id', function(req, res){
request('http://'+options.host+':'+options.port+options.path, function (error, response, body) {
if (!error && response.statusCode == 200) {
console.log(body); // Print the google web page.
res.writeHead(200, {
'Content-Type': 'application/json',
'Cache-Control': 'no-cache',
'Connection': 'keep-alive',
'Access-Control-Allow-Origin': '*',
'Access-Control-Allow-Credentials': 'true'
});
res.end(JSON.stringify(JSON.parse(body)));
}
})
});
How can I change these two so they will work with cross-domain GET in IE? I have been searching the internet and there seem to be a few different things like jQuery.support.cors = true; which does not work. There also seem to be a lot of lengthy workarounds.
There is no real 'ideal' design pattern which I have been able to find for this type of thing.
Seeing as I have control over both the web page and the cross domain web service I'm sending to what is the best change to make to ensure compatability across all IE versions along with FireFox, Opera, Chrome etc?
Cheers!
Say we have two servers, myServer.com and crossDomainServer.com, both of which we control.
Assuming we want a client of myServer.com to pull some data from crossDomainServer.com, first that client needs to make a JSONP request to crossDomainServer.com:
// client-side JS from myServer.com
// script tag gets around cross-domain security issues
var script = document.createElement('script');
script.src = 'http://crossDomainServer.com/getJSONPResponse';
document.body.appendChild(script); // triggers a GET request
On the cross-domain server we need to handle this GET request:
// in the express app for crossDomainServer.com
app.get('/getJSONPResponse', function(req, res) {
res.writeHead(200, {'Content-Type': 'application/javascript'});
res.end("__parseJSONPResponse(" + JSON.stringify('some data') + ");");
});
Then in our client-side JS we need a global function to parse the JSONP response:
// gets called when cross-domain server responds
function __parseJSONPResponse(data) {
// now you have access to your data
}
Works well across a wide variety of browsers, IE 6 included.
The following code shows how to handle the GET request (using express) and how to wrap the JSON response using the callback given:
app.get('/foo', function(req, res){
res.header('Content-Type', 'application/json');
res.header('Charset', 'utf-8')
res.send(req.query.callback + '({"something": "rather", "more": "pork", "tua": "tara"});');
});