I use http://instagram.com/username/ this URL for image scraping from Instagram public profiles.
Everything works fine in localhost and works as I expect.
However, in the server(CentOS 7) scenario is different. It does not fetch the same response when I compared it with the localhost (I used "curl -v http://instagram.com/username/").
The only difference that I could understand in response headers is csrf_token is missing from the server and in localhost, with git bash, it gets attached immediately.
Any hints or answers are appreciated.
PS: I checked in different servers and it gives the same response so it cannot be that Instagram blocked the IP. ^_^
More importantly I use https://www.instagram.com/graphql/query/?query_hash=f92f56d47dc7a55b606908374b43a314&variables={"tag_name":"rose","first":25} this URL to scrape hashtag images and it works fine on both server and localhost.
If you can provide something like above is also fine ^_^
This is expected behaviour, Instagram isn't just going to let you scrape their whole website. They have access-control headers that prevent incorrect embedding and when you do cURL the page will be checking for certain headers, specifically an API key, which you haven't provided and so you are denied.
You need to use the Instagram API : https://www.instagram.com/developer
Related
I am having an issue with 'ERR_CERT_COMMON_NAME_INVALID' when making a request from a website.
First of all, am really a bit of a novice when it comes to setting up and using certs... so this is probably something stupid that I have ether done or not done!
I have a ssl certificate www.my-dummy-url.abc and api.my-dummy-url.abc (obviously my-dummy-url is for example only). Where api.my-dummy-url.abc is the certificate Subject Alternative Name and www.my-dummy-url.abc is the certificate Subject.
The Apache Web sever, is hosted at www.my-dummy-url.abc, whilst a reverse proxy points 'api.my-dummy-url.abc to a Raspberry Pi running a little node.js API endpoint (/weather).
I have set node.js up to use express: https.createServer(ssl_options, ..., where ssl_options is my cert.pem and key.pem. CORS is enabled (app.use(cors());) and 'Access-Control-Allow-Origin', '*' is present in the response.
The Website (apart from the API call) renders fine at https://www.my-dummy-url.abc (Chrome).
The API, (https://api.my-dummy-url.abc/weather) if requested within the address bar of a Chrome browser or through Postman receives the correct JSON response.
However, from my website as a jQuery $.get function I get the following error:
GET https://api.my-dummy-url.abc/weather net::ERR_CERT_COMMON_NAME_INVALID
Any ideas, helpers, pointer or solutions will be gratefully received...
As an addition, if I use Edge rather then Chrome, it works...!
Kind Regards,
Harold Clements
This issue was that Chrome was caching the certificate somehow. After shutting down and coming back to it the next day, it was working fine.
I am working on ionic app which is supposed to make http request to an api. This means I cannot change server side settings. I am using the angular http module. My first attempt gave rise to a CORS error, I managed to fix by using a proxy like below:
"proxies": [
{
"path": "/api",
"proxyUrl": "https://api.com/api/"
}
]
This works perfectly fine in the browser when using ionic serve, however when deploying to a device the request does not seem to work.
When using the browser it give a status code 200 with a proper response.
Response code
When deployed it seems looks like it returns the same response, however in the console the following error is displayed (this is retrieved using the remote devices in google chrome), also the response tab for the request is empty.
Edit:
I checked the AndroidManifest.xml file and it contains the internet permission. However I tried to build it to android with the full URL instead of the proxy URL, but then the app unexpectedly closes.
So you're working directly with 3rd party API? Usually it's considered as a bad practice, because you have to store or retrieve your API keys on clientside app. I suggest you to implement your own backend and proxy requests to the API.
If there is no keys, or you simply don't care about security you could try other solutions, here is a great article about CORS:
https://ionicframework.com/docs/faq/cors
I need to get some data from external API, from other domain.
I use test app, runned on localhost and I get error:
Origin http://localhost is not allowed by Access-Control-Allow-Origin.
I try to get access to: https://api.coinmarketcap.com/v1/ticker/ by get method
I did a lot of search and:
I know that I can ask that API by using node.js, CURL, and also by running my frontend code in jest tests
By using chrome extension - https://chrome.google.com/webstore/detail/allow-control-allow-origi/nlfbmbojpeacfghkpbjhddihlkkiljbi?hl=en also everything works correctly
But unfortunatelly by simply AJAX request on my page load I can't reach that API and get an error.
And my prediction is that user agent (chrome) somehow block my request.
I can't use chrome extension because I want also access my website from my mobile phone which can't get that desktop extension.
For now my solution is "proxy", node.js server which can access that API and flow looks like:
FE
-- ASK BE -->
BE
-- ASK API -->
-- RESPONSE TO BE -->
BE
-- RESPONSE TO FE -->
FE SHOWS EVERYTHING
FE - Frontend, BE - Backend (node.js proxy), API - external API
But I want to avoid unnecessary code on the BE. Maybe there is some solution?
Like some magical header which will tell chrome to let me pass to external domain without any additional extension?
You are correct you cannot bypass the CORS rules unless you are using jsonp and the server in question supports it. When working with that ticker in the past I had to build a backend exactly as you describe with express.
FE => BE => Ticker => BE => FE
Well you can do that one clean way as described here - https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS
Set the header value
Access-Control-Allow-Origin: *
with your response to the first page load from server.
If all you pages uses HTML5,you can try using Window.postMessage.
Another common solution is creating a proxy-server.
I have a site that is running in Node and using express for the server. I also have an API on a different server I need to hit. Everything works fine except in IE9. The issue is when ever i try to hit my API once i have the page loaded it breaks.
For example on the home page i have a search form where each step makes an API fetch and once I try to make my first fetch i get this error
"Unhanded promise rejectionError: Access Denied"
It seems like it has to do with a CORS issue since IE9 didnt have CORS and the era of the error message. Also when I make the fetched server side I have no issue, its just when the client tries to perform them.
Has anyone had an issue like this and if so how did you fix it?
Currently I am trying this but it hasnt worked yet:
First I changed my api host address to the current sites address with /api at the end.
apiHost: process.env.API_HOST || 'http://127.0.0.1:3000/api'
Then in my express server is run this:
app.use(modRewrite([`^/api/(.*)$ http://api.mysite.dev:8080/$1 [P]`]));
I was hoping to avoid making a fetch cross address with this but no luck as of yet.
Any help would be much appreciated
thanks!
To solve this problem I changed my host file for my machine to use mysite.dev for the 127.0.0.1 IP. Then went to mysite.dev rather than 127.0.0.1 when testing my site. Then did what I did above to get it to work
I have protected my Couchdb database for access with a username. The access via curl https://user:password#url:port/dbname works fine. If I put this in the browser via url or using Javascript with CouchJS it works not (Error 401). I have enabled CORS in the CouchDB-config.
Can anyone help me, please.
If pasting the same URL you used with curl into the address bar of your browser doesn't work, then you may have a different problem. However, if you're just having trouble making AJAX requests against a CouchDB (which has CORS enabled) you probably need to set the Authorization header. See the answers to this question: How to use Basic Auth with jQuery and AJAX?