I am using request module and i am getting empty body in my response,here is the code
request.get(data_url, function (error, response,body) {
console.log('----######----');
console.log(response);
console.log('----######----');
console.log(body);
actually when i manually hit the url contained in my data_url variable, a csv file gets downloaded automatically, is the problem due to this nature, if not please suggest a suitable solution.
Also if i replace the data_url with the actual url contained in the variable then i am getting the body part in response.
Related
I have this endpoint that will write a static file with json data in to it.
fs.writeFile("name.json", JSON.stringify(JSON.parse(data)), function (err) {
if (err) {
return console.log(err);
}
console.log("The file was saved!");
});
And then i have another endpoint that will send the data inside this json file.
const fileData = require("./name.json")
res.json(fileData)
However after i trigger a change inside the file and then try to get that new data its not getting the new data instead sends the old one. But if i refresh the server and then try to get it it will send me the new data. I can see inside the file that changes are there after i write data but it still doesn't send the new data. It feels like some kind of caching. Ive tried to disable etag but still no success.
app.set('etag', false)
app.use(express.static("*", {
etag: false,
}))
When you start the server, filedata is read once and it is then available in memory, regardless of file changes, which is why you can get fresh data only after restarting the server.
It's caching, but it's not traffic caching, see: What is require?
What you actually need is to read the file every time you access it.
So, instead of require, you could just read the file every time you access it, and get fresh data (and since it's JSON, you'd need to parse it before sending it):
const fileData = fs.readFileSync("./name.json");
res.json(JSON.parse(fileData));
I am struggling to understand when does the server return XML data vs HTML text data, and how come responseXML can return both? Is XML data returned only when making a POST request?
It's hard to actually test these, because I couldn't properly setup a PHP server, and making a POST request keeps returning 404 bad request, but when I make a GET request, I always get the HTML document in the responseText property, but when I try to use responseXML, I get null. So, if responseXML can return either HTML or XML, why does it not return the HTML document then?
Note: Before you accuse me of not doing any research. Let me tell you that I have been doing research for the past 3 days, and the book I'm reading just doesn't clarify these differences, and does not explain what exactly is XML in the first place. It says that XML data needs to be parsed to be displayed as a text, but doesn't explain why. It's all very ambiguous. So, I would appreciate if someone could clarify things for me.
POST requests return XML data, if the backend server is configured to return XML data. Completely depends on the server you're talking to, there's no way of predicting the behavior otherwise. Also, it's worth noting that every input can change the behavior of the server. E.g. if you provide a query with a specific value, the server could also return a CSS file, instead of an HTML or XML one.
This is an example of getting HTML from server response using fetch API.
fetch('https://someweb.com/api/list').then(function (response) {
// The API call was successful!
return response.text();
}).then(function (html) {
// Convert the HTML string into a document object
var parser = new DOMParser();
var doc = parser.parseFromString(html, 'text/html');
}).catch(function (err) {
// There was an error
console.warn('Something went wrong.', err);
});
I'm trying to use instagrams api, without having to authenticate.
if I go to: https://www.instagram.com/explore/tags/chicago/?__a=1
I get a json response.
I'm trying to build an angular 4 provider that returns that data.
getInsta(term){
return this.jsonp.request(this.url+term+'/?__a=1&callback=__ng_jsonp__.__req0.finished')
.map(res => {
console.log(res.json());
});
}
I've tried "callback" and "c". I've tried JSONP_CALLBACK.
I get the error of:
"SyntaxError: Unexpected token ':'. Parse error. - In chicago:1"
and
"JSONP injected script did not invoke callback"
If I click on the unexpected token error, it brings me to the response with all the json data in it. So that means it's coming through, I just cant access it!
Any idea how i can get around this?
Here is a screenshot of the error and when clicking on the error
JSONP is a method to fetch data cross origin, it works because it injects a script tag to the body when the src of the file contains the callback function name.
<script src="http://ani.site.com/api/?callback=myGlobalFunc">
And it assumes that there is a global func on the window with the name myGlobalFunc.
The response of that call gonna look like: myGlobalFunc({data: 'data1'}).
Invocation of the function with one argument which is the data.
In your case, Instagram API return JSON not JSONP.
There for you can't use that mechanism.
I want to get the content of a webpage by running javascript code on NodeJs . I want the content to be exactly the same as what I see in the browser.
This is the URL :
https://www.realtor.ca/Residential/Single-Family/17219235/2103-1185-THE-HIGH-STREET-Coquitlam-British-Columbia-V3B0A9
I use the following code but I get 405 in response.
var fs = require('fs');
var link = 'https://www.realtor.ca/Residential/Single-Family/17219235/2103-1185-THE-HIGH-STREET-Coquitlam-British-Columbia-V3B0A9';
var request = require('request');
request(link, function (error, response, body) {
fs.writeFile("realestatedata.html", body, function(err) {
if(err) {
console.log('error in saving the file');
return console.log(err);
}
console.log("The file was saved!");
});
})
The file which is saved is not related to what I can see in the browser.
I think a real answer will be easier to understand since my comment was truncated.
It seems the method of the request you send is not supported by the server (405 Method Not Allowed - The method specified in the Request-Line is not allowed for the resource identified by the Request-URI. The response MUST include an Allow header containing a list of valid methods for the requested resource.). Do you have more information about the HTTP response.
Have you tried the following code instead of yours ?
request('https://www.realtor.ca/Residential/Single-Family/17219235/2103-1185-THE-HIGH-STREET-Coquitlam-British-Columbia-V3B0A9').pipe(fs.createWriteStream('realestatedata.html'))
You could also have a look at In Node.js / Express, how do I "download" a page and gets its HTML?.
Note that anyway the page will not render the same way when you only open the html since it also requires many other resources (110 requests are done when display the page).
I think the following answer can help you to download the whole page.
https://stackoverflow.com/a/34935427/1630604
I'm writing a test using SlimerJS for a website and need to check the response body coming from the server. I'm using the following piece of code to get the response:
page.onResourceReceived = function (response) {
console.log(JSON.stringify(response));
};
I do receive the response but since by default to prevent too much memory usage SlimerJS keeps the response body empty I too receive an empty body, unless I tell it not to keep the body empty for certain formats using something like this:
webpage.captureContent = [ /css/, /image\/.*/ ]
I understand this works well for files with extensions like css,jpg and avi, but what about an AJAX response coming from the server. The response is in JSON format and the response body is left empty.
By looking at the response header you can tell that the response type is in text/html so by using the following code you can get the body.
page.captureContent = [/text/, /html/]