I have the following code using fetch
var url1 = 'http://jsonplaceholder.typicode.com/users';
var url2 = 'http://teaconcepts.net/temp/api_test.php';
fetch(url1)
.then(function (result){
console.log("Result received");
return result.json();
}).then(function(data){
// console.log("data received",data[1]['name']);
console.log("data received",data);
}).catch(function(error){
console.log("Error in Fetch",error.message);
});
I'm fetching valid JSON data from both URLs, but with slightly different structure
var url1 = 'http://jsonplaceholder.typicode.com/users';
var url2 = 'http://teaconcepts.net/temp/api_test.php';
If I load data from URL1, things are fine. But if I load url2 i get an error "Failed to fetch"
What gives???
See demo: http://jsbin.com/yaxecewopu/1/edit?js,console
The issue here is the same-origin policy. If you run this JavaScript on http://teaconcepts.net it will work. Or, you can get teaconcepts.net to respond with an Access-Control-Allow-Origin header to allow any origin or specific origins (such as jsbin.com and stackoverflow.com)
Here issue with second url is encoding problem, check second url here
Paste your url in
https://jsonformatter.curiousconcept.com/ and check what is issue with return type of json. or else see below image result of second url json
if you author for that php file see the below thread to solve issue php returns invalid json
The second URL refers to a site that doesn't allow cross-origin requests. The browser therefore disallows access via fetch() or XMLHttpRequest.
Related
when i debug this code in to chrome console then its not show any output or alert! please help me to complete this code! i need to get read my read.txt file text in to console.log....
the code was i try one is shows below.
function loadText() {
fetch('C:\Windows\Temp\read.txt')
.then(function(response){
return response.text();
})
.then(function(data){
console.log(data);
alert(data)
})
.catch(function(error){
console.log(error);
alert(data)
})
}
Try below code and indicate your directory like below
async function fetchText() {
let response = await fetch('../demo.txt');
console.log(response.status); // 200
console.log(response.statusText); // OK
if (response.status === 200) {
let data = await response.text();
console.log(data);
// handle data
}
}
fetchText();
This seems like a duplicate of this issue - AJAX request to local file system not working in Chrome?
The problems are the same, however you are using the fetch API (https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API) not an XMLHttp request (https://developer.mozilla.org/en-US/docs/Web/API/XMLHttpRequest)
When i run loadText i get the following error:-
Fetch API cannot load . URL scheme must be "http" or "https" for CORS request.
You cannot make requests to the filesystem in Chrome. However you can disable Chrome security using flags (--allow-file-access-from-files)
see - Allow Google Chrome to use XMLHttpRequest to load a URL from a local file however this is not advised.
You will also need to update your path in the fetch function by prefixing with file:/// this tells it to look in the file system, and changed the protocol from http or https.
I get two different errors depending on if I GET or POST, but my autodesk rep assures me that using postman (not javascript) the resource does exist and he can get an authentication token.
If I do:
var url = "https://developer.api.autodesk.com/authentication/v1/authenticate";
var options = {
"method": "GET",
"headers":{"Content-Type": "application/x-www-form-urlencoded",},
"body": {
"client_id" : "Z---F",
"client_secret" : "m---8",
"grant_type": "client_credentials",
"scope": "data:read"
}
}
;
console.log(url);
console.log("Options:\n"+JSON.stringify(options));
var res = UrlFetchApp.fetch(url, options).getContentText();
Logger.log(res);
}
I get
Exception: Request failed for https://developer.api.autodesk.com returned code 404. Truncated server >response: { "developerMessage":"The requested resource does not exist.", "moreInfo": >https://forge.autodesk.com/en/docs/oauth/v2/developers_guide/error_hand... (use muteHttpExceptions >option to examine full response)
if I do
var url = "https://developer.api.autodesk.com/authentication/v1/authenticate";
var options = {
"method": "POST",
"headers":{"Content-Type": "application/x-www-form-urlencoded",},
"body": {
"client_id" : "Z---F",
"client_secret" : "m---8",
"grant_type": "client_credentials",
"scope": "data:read"
}
}
;
console.log(url);
console.log("Options:\n"+JSON.stringify(options));
var res = UrlFetchApp.fetch(url, options).getContentText();
Logger.log(res);
}
I get
Exception: Request failed for https://developer.api.autodesk.com returned code 400. Truncated server >response: {"developerMessage":"The required parameter(s) client_id,client_secret,grant_type not present >in the request","errorCode":"AUTH-008","more info":"h... (use muteHttpExceptions option to examine full >response)
Any thoughts why it would say a resource doesn't exist that does exist? There is no personal info on here so the fact that he is doing it versus me doing it shouldn't matter. All the posts I could find on this issue were dealing with the curl version of this call rather than a javascript with options bundle
The URL is definitely valid (here's the API reference for this endpoint), so I would recommend debugging the actual code that's making the request, for example:
debug the UrlFetchApp class (where is it coming from btw?) and its fetch method, making sure that it's no modifying the url you're passing in
if the code is running in a browser, try looking at the DevTools Network tab and see if the request actually goes out to https://developer.api.autodesk.com/authentication/v1/authenticate
if the code is running in a browser, try using the built-in Fetch API instead
Also, I'm not sure if this is something handled automatically by the UrlFetchApp but specifying the content type of your request as application/x-www-form-urlencoded and then passing in a JSON object as the body doesn't seem right. As you can see in the documentation the request body should look more like this:
client_id=<your client id>&
client_secret=<your client secret>&
grant_type=client_credentials&
scope=data:read
I want to get the content of a webpage by running javascript code on NodeJs . I want the content to be exactly the same as what I see in the browser.
This is the URL :
https://www.realtor.ca/Residential/Single-Family/17219235/2103-1185-THE-HIGH-STREET-Coquitlam-British-Columbia-V3B0A9
I use the following code but I get 405 in response.
var fs = require('fs');
var link = 'https://www.realtor.ca/Residential/Single-Family/17219235/2103-1185-THE-HIGH-STREET-Coquitlam-British-Columbia-V3B0A9';
var request = require('request');
request(link, function (error, response, body) {
fs.writeFile("realestatedata.html", body, function(err) {
if(err) {
console.log('error in saving the file');
return console.log(err);
}
console.log("The file was saved!");
});
})
The file which is saved is not related to what I can see in the browser.
I think a real answer will be easier to understand since my comment was truncated.
It seems the method of the request you send is not supported by the server (405 Method Not Allowed - The method specified in the Request-Line is not allowed for the resource identified by the Request-URI. The response MUST include an Allow header containing a list of valid methods for the requested resource.). Do you have more information about the HTTP response.
Have you tried the following code instead of yours ?
request('https://www.realtor.ca/Residential/Single-Family/17219235/2103-1185-THE-HIGH-STREET-Coquitlam-British-Columbia-V3B0A9').pipe(fs.createWriteStream('realestatedata.html'))
You could also have a look at In Node.js / Express, how do I "download" a page and gets its HTML?.
Note that anyway the page will not render the same way when you only open the html since it also requires many other resources (110 requests are done when display the page).
I think the following answer can help you to download the whole page.
https://stackoverflow.com/a/34935427/1630604
I would like to know if there exist any way to check if a url exists and to check information on it.
Here you have an example about what I mean:
imagine you are in "https://de.yahoo.com/?p=us" webpage.
Then using Javascript check if "https://www.google.com" exists and if it contains both words "google" and "yahoo"
if it exists:
Check if the word "google" is in the page:
if it exists return "true"
if it doesn't return "false"
Check if the word "yahoo" is in the page:
if it exists return "true"
if it doesn't return "false"
if it doesn't exist:
return "false"
Writing a full code solution for you in this answer may not be possible for me, But I will set a pathway for you. By following it you can get the desired result.
First of all, getting contents of a webpage which is not under your domain nor in your control will be not possible through javaScript as browser will prevent it unless CORS is enabled on that domain.
To overcome this shortage, you need to get the contents of the page through a server side scripting, most probably you will use php. So create a file on your server to which you will make ajax request with the desired url as data. In jquery you can achieve this as follow:
$.ajax('getPage.php', {
data: 'url=your desired url'
}).success(function (data) {
//Process the data
}).error(function () {
//Error Handler
});
Now in php file, use curl or get_file_contents function to get the data from the page
You can do something like this in getPage.php
$page = file_get_contents($_GET['url']);
echo $page;
Now you can either use the JavaScript and parse the contents received via ajax and check for your keyword or you can do the parsing right in your getPage.php file and return either true or false.
If you want to do this on the same domain, subdomain, port, or protocol, you can set a JSON response to you AJAX request and lets say that the response will contain all of the required info with the keys: "checkResults" the Ajax would look like this:
$.ajax({
type: "GET",
url: "YourURL",
dataType: "json",
success: function (data.checkResults) {
//do whatever
}
});
this can't be done on a different domain, sub-domain, port, or protocol. unless ofc the CORS is enabled which is not likely the case.
Notes
Due to browser security restrictions, most Ajax requests are subject
to the same origin policy; the request can not successfully retrieve
data from a different domain, subdomain, port, or protocol. Script and
JSONP requests are not subject to the same origin policy restrictions.
SEE: Breaking-cross-domain it is really helpful
Plugins for cross-domain cross-domain-plugin
CORS Anywhere
CORS Anywhere is a node.js proxy which adds CORS headers to the proxied request.
To use the API, just prefix the URL with the API URL. (Supports https: see [github repository][3])
If you want to automatically enable cross-domain requests when needed, use the following snippet:
$.ajaxPrefilter( function (options) {
if (options.crossDomain && jQuery.support.cors) {
var http = (window.location.protocol === 'http:' ? 'http:' : 'https:');
options.url = http + '//cors-anywhere.herokuapp.com/' + options.url;
//options.url = "http://cors.corsproxy.io/url=" + options.url;
}
});
$.get(
'http://en.wikipedia.org/wiki/Cross-origin_resource_sharing',
function (response) {
console.log("> ", response);
$("#viewer").html(response);
});
credit: #jherax
How can we get the source code of a webpage from a webpage in php and/or javascript?
In Javascript without using unnecessary frameworks (in the example api.codetabs.com is a proxy to bypass Cross-Origin Resource Sharing):
fetch('https://api.codetabs.com/v1/proxy?quest=google.com').then((response) => response.text()).then((text) => console.log(text));
Thanks to:
#PLB
#Shadow Wizard
Getting the source code of an iframe
http://www.frihost.com/forums/vt-32602.html
#Matt Coughlin.
First, you must know that you will never be able to get the source code of a page that is not on the same domain as your page in javascript. (See http://en.wikipedia.org/wiki/Same_origin_policy).
In PHP, this is how you do it :
file_get_contents($theUrl);
In javascript, there is three ways :
Firstly, by XMLHttpRequest : http://jsfiddle.net/635YY/1/
var url="../635YY",xmlhttp;//Remember, same domain
if("XMLHttpRequest" in window)xmlhttp=new XMLHttpRequest();
if("ActiveXObject" in window)xmlhttp=new ActiveXObject("Msxml2.XMLHTTP");
xmlhttp.open('GET',url,true);
xmlhttp.onreadystatechange=function()
{
if(xmlhttp.readyState==4)alert(xmlhttp.responseText);
};
xmlhttp.send(null);
Secondly, by iFrames : http://jsfiddle.net/XYjuX/1/
var url="../XYjuX";//Remember, same domain
var iframe=document.createElement("iframe");
iframe.onload=function()
{
alert(iframe.contentWindow.document.body.innerHTML);
}
iframe.src=url;
iframe.style.display="none";
document.body.appendChild(iframe);
Thirdly, by jQuery : http://jsfiddle.net/edggD/2/
$.get('../edggD',function(data)//Remember, same domain
{
alert(data);
});
Following Google's guide on fetch() and using the D.Snap answer, you would have something like this:
fetch('https://api.codetabs.com/v1/proxy?quest=URL_you_want_to_fetch')
.then(
function(response) {
if (response.status !== 200) {
console.log('Looks like there was a problem. Status Code: ' +
response.status);
return;
}
// Examine the text in the response
response.text().then(function(data) {
// data contains all the plain html of the url you previously set,
// you can use it as you want, it is typeof string
console.log(data)
});
}
)
.catch(function(err) {
console.log('Fetch Error :-S', err);
});
This way you are using a CORS Proxy, in this example it is Codetabs CORS Proxy.
A CORS Proxy allows you to fetch resources that are not in your same domain, thus avoiding the Same-Origin policies blocking your requests.
You can take a look at other CORS Proxys:
https://nordicapis.com/10-free-to-use-cors-proxies/
Ajax example using jQuery:
// Display the source code of a web page in a pre tag (escaping the HTML).
// Only works if the page is on the same domain.
$.get('page.html', function(data) {
$('pre').text(data);
});
If you just want access to the source code, the data parameter in the above code contains the raw HTML source code.