I was just trying to make covid vaccine alert using Cowin Setu API (India) in nodejs. But I am facing some strange thing, whenever I hit get request I got 403 response code from cloudfront says 'Request Blocked' but the same is working from postman as well as from browser. Please help me in this
Getting this error:-
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
<HTML><HEAD><META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=iso-8859-1">
<TITLE>ERROR: The request could not be satisfied</TITLE>
</HEAD><BODY>
<H1>403 ERROR</H1>
<H2>The request could not be satisfied.</H2>
<HR noshade size="1px">
Request blocked.
We can't connect to the server for this app or website at this time. There might be too much traffic or a configuration error. Try again later, or contact the app or website owner.
<BR clear="all">
If you provide content to customers through CloudFront, you can find steps to troubleshoot and help prevent this error by reviewing the CloudFront documentation.
<BR clear="all">
<HR noshade size="1px">
<PRE>
Generated by cloudfront (CloudFront)
Request ID: Q1RZ94qgFp6AjUUKE4e9urMB85VejcqMbaJO6Y8Xq5Qp4kNjDBre9A==
</PRE>
<ADDRESS>
</ADDRESS>
</BODY></HTML>
Here's my nodejs code:
var express = require("express");
var app = express();
var bodyParser = require("body-parser");
const axios = require("axios");
const { Telegram } = require("telegraf");
const fetch = require("node-fetch");
var cors = require('cors');
var request=require('request');
const tg = new Telegram(process.env.BOT_TOKEN);
const bot = new Telegram(process.env.BOT_TOKEN, {
polling: true
});
//bot.start((ctx) => ctx.reply('Welcom to Covid Vaccine Finder'))
/*bot.hears("about", ctx => {
ctx.reply("Hey, I am CoviBot!");
});
bot.launch();*/
app.use(bodyParser.json());
app.use(cors());
app.use(
bodyParser.urlencoded({
extended: true
})
);
app.get("/", function(req, res) {
res.send("Welcom to Covid Vaccine Finder");
});
app.get("/test", function(req, res, next) {
var d = new Date();
var options = {
year: "numeric",
month: "2-digit",
day: "2-digit"
};
var date = String(d.toLocaleDateString("en", options));
date = date.replace(/\//g, "-");
console.log(date);
const URL =
"https://cdn-api.co-vin.in/api/v2/appointment/sessions/public/findByPinpincode=110088&date=13-05-2021";
var options = {
url: URL,
method: 'GET',
headers: {
'Accept-Encoding': 'gzip, deflate',
'Accept-Language': 'en-GB,en;q=0.8,en-US;q=0.6,hu;q=0.4',
'Cache-Control': 'max-age=0',
Connection: 'keep-alive',
Host: 'cdn-api.co-vin.in',
'User-Agent': 'request',
}
};
request(options,function(err,res,body){
let json = body;
console.log(json);
});
const txt = "Finding vaccine centres for you....";
//tg.sendMessage(process.env.GROUP_ID, txt);
res.send(txt);
});
// Finally, start our server
app.listen(process.env.PORT, function() {
console.log("Covid app listening on port 3000!");
});
I hope this problem will solve
Thanks
I added a user-agent header to the request so that the API would recognize that my request is coming from a browser, rather than a script.
headers = {
'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36',
}
url = "https://cdn-api.co-vin.in/api/v2/appointment/sessions/public/calendarByDistrict?district_id=303&date="+date
response = requests.get(url, headers=headers)
Use following
var options = {
url: URL,
method: 'GET',
headers: {
Host: 'cdn-api.co-vin.in',
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.93 Safari/537.36'
}
};
Try These Headers They worked for me on local server (not production)
let options = {
headers: {
"user-agent":
"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.93 Safari/537.36",
},
};
These will not work in production because Cowin APIs are geofenced and can't be accessed from IP address other than Indian. In most free hosting sites like Heroku, Indian IP is not an option. So alternative solution might be to use AWS, GCP, Azure with an Indian server (not tried yet).
Reference - https://github.com/cowinapi/developer.cowin/issues/228
It seems the api is blocked from using outside India. Try to combine some Indian proxy/use in Indian server
You have to use User Agent Identifier API
Please refer this
https://devcenter.heroku.com/articles/useragentidentifier#using-with-python
You have to make your request in the following format, I am attaching sample format for states metadata API:
curl --location --request GET 'https://cdn-api.co-vin.in/api/v2/admin/location/states' --header 'Accept-Language: hi_IN' --header 'Accept: application/json' --header 'User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.93 Safari/537.36'
Its not about the request user-agent or format. I faced the same issue and further testing proved cloudFront is blocking the IP if multiple requests are coming from same IP back to back. Its also unblocking after couple minutes.
Basically they don't want these alerting this, probably its overloading their server.
Ok if you want to work local you can use
let headers = {
'accept': 'application/json',
'Accept-Language': 'hi_IN',
'X-Requested-With': 'XMLHttpRequest',
'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.97 Safari/537.36',
}
Now if you want to deploy to Heroku or firebase, then it will return 403, I think it's mostly that they are blocking any IP hit outside from Indian server.
Github link: https://github.com/manojkumar3692/reactjs_nodejs_cowin
I Will keep you posted here
Related
I'm making an api that communicates with a website to pull player statistics. I've made multiple POST/GET HTTP/1 requests to the server to get a session token and player ID. I then use those values(valid values which I have tested before passing to my function) in my last function to fetch player statistics. The last request is a HTTP/2 GET request. I'm using the got library and vanilla Node. Here is my request:
//THESE ALL HAVE SOME VALUE AFTER I USE SOME OF MY FUNCTIONS; THE FUNCTION I'M
//HAVING TROUBLE WITH IS THE LAST FUNCTION AND IS PASSED VERIFIED NON-NULL VALUES
var session = {
app_id: '3587dcbb-7f81-457c-9781-0e3f29f6f56a',
space_id: '5172a557-50b5-4665-b7db-e3f2e8c5041d',
session_id: null,
ticket: null,
};
var player = {
name: null,
id: null,
platform: 'uplay',
kills: null,
deaths: null,
rank: null,
};
async function get_player_stats(session, player) {
var platform = 'PC';
if (player.platform === 'uplay') {
platform = 'PC';
}
var options = {
':authority': 'r6s-stats.ubisoft.com',
':method': 'GET',
':path': `/v1/current/operators/${player.id}?gameMode=all,ranked,casual,unranked&platform=${platform}&teamRole=attacker,defender&startDate=20200723&endDate=20201120`,
':scheme': 'https',
'authorization': `ubi_v1 t=${session.ticket}`,
'ubi-appid': session.app_id,
'ubi-sessionid': session.session_id,
'content-type': 'application/json',
'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.193 Safari/537.36',
}
const url = `https://r6s-stats.ubisoft.com/v1/current/operators/${player.id}?gameMode=all,ranked,casual,unranked&platform=${platform}&teamRole=attacker,defender&startDate=20200723&endDate=20201120`;
try {
const response = got(url, {headers: options, http2: true});
console.log(response);
}
catch (err) {
console.log(err);
}
}
//FUNCTION CALL
async function fetch(user) {
var stats_string = await get_player_stats(session, player);
console.log(stats_string);
}
fetch(username);
Chrome's request header from network log:
:authority: r6s-stats.ubisoft.com
:method: GET
:path: /v1/current/operators/e96ae749-8939-43ed-895f-bf1817e849d9?gameMode=all,ranked,casual,unranked&platform=PC&teamRole=attacker,defender&startDate=20200723&endDate=20201120
:scheme: https
accept: */
accept-encoding: gzip, deflate, br
accept-language: en-US,en;q=0.9
authorization: ubi_v1 t= LONG TOKEN
dnt: 1
expiration: 2020-11-21T09:13:54.804Z
origin: https://www.ubisoft.com
referer: https://www.ubisoft.com/
sec-fetch-dest: empty
sec-fetch-mode: cors
sec-fetch-site: same-site
ubi-appid: 3587dcbb-7f81-457c-9781-0e3f29f6f56a
ubi-sessionid: d78f3306-0e5c-4ac8-ad63-5a711b816f76
user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.193 Safari/537.36
Chrome's Response header from network tools:
access-control-allow-credentials: true
access-control-allow-origin: https://www.ubisoft.com
content-encoding: gzip
content-length: 16969
content-type: application/json
date: Sat, 21 Nov 2020 06:14:47 GMT
status: 200
vary: Origin
What I've tried:
I've tried just about everything. I've googled what causes 400 errors, which apparently are mostly user error, and I've looked through my code for days and also looked at Chrome's network activity. I've tried matching Chrome's request header with mine to no avail(my header variable is one of many iterations I've tried--pretty sure I've tried every combination of possible headers). However, sometimes I'll get 400 bad error, or an invalid header response from the server. I've tried using the vanilla Node http2Client.request and that gives me an invalid header/400 as well.
Okay, finally figured out why this wasn't working. I missed one tiny line in what I thought I already tried millions of times.
In the request header on the Chrome Network activity there is a field for expiration.
I needed to set the expiration value in the header to get the data.
So the value I needed to add to my header in my code was:
expiration: 2020-11-21T09:13:54.804Z
Future edit: The expiration is the date in ISO format. You can make a date Object and convert to ISO:
var time = new Date();
var expiration = time.toISOString();
function someRequest() {
var options = {
'expiration': expiration,
}
}
When I am trying to receive a message from Socket.IO using the native socket module, instead of receiving the message, I receive this:
GET /socket.io/?EIO=3&transport=polling&t=MIlsTQ_ HTTP/1.1
Host: localhost:5000
Connection: keep-alive
Accept: */*
Origin: http://localhost:8080
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36
DNT: 1
Referer: http://localhost:8080/
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.9
How do I fix this, and receive the message rather than the data?
Client code:
socketsSend: function() {
const socket = io.connect('http://localhost:5000');
/*var socket = io.Socket('http://localhost', {
port: 5000
});*/
socket.connect();
socket._connectTimer = setTimeout(function() {
socket.close();
}, 500);
socket.on('connect', function() {
// socket connected successfully, clear the timer
clearTimeout(socket._connectTimer);
});
Server code:
import socket
import json
addr = 'localhost',5000
sock=socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.bind(addr)
sock.listen(3)
while True:
connection, client_address = sock.accept()
data = connection.recv(100000)
print(data.decode('utf-8'))
From this how to guide:
In general, they [socket's recv and send functions] return when the associated network buffers have been filled (send) or
emptied (recv). They then tell you how many bytes they handled. It is your
responsibility to call them again until your message has been completely dealt with.
So, from my understanding, connection.recv(100000) won't return any data until its buffer fills up, so to fix this you would need to set buffer to something low, (maybe 2048?).
control:no-cache` field into my request when requesting RSS feed
I cant quit figure what values should i put in
In case of Content-type it works well but it refuses to add correctly Cachce-Control
code :
options = {uri :SUPPORT_FEED_URI,
headers : {
'Content-Type': 'application/x-www-form-urlencoded',
'Cache-Control': 'no-cache'
},
}
request.get(options)
.on('error', (err) => { reject(err); })
.pipe(feedparser)
.on('end', () => { return resolve(items); });
What i get in request headers :
Accept:*/*
Accept-Encoding:gzip, deflate, sdch
Accept-Language:en-US,en;q=0.8
Access-Control-Request-Headers:cache-control <-- doesnt seems to be right Want something like Cache-Control : no-cache
Access-Control-Request-Method:GET
Connection:keep-alive
Host: xxxx.yyyy.zz
Origin:http://127.0.0.1:8888
Referer:http://127.0.0.1:8888/webconsole/
User-Agent:Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.133 Safari/537.36
content-type:application/x-www-form-urlencoded
Your capture is a CORS pre-flight (OPTIONS) request as the URL is on a different domain or considered to be different-origin.
Such a request will not include custom headers, they are added to Access-Control-Request-Headers instead to see if the destination server will allow them.
If the destination server responds with an acceptable allow- response the subsequent GET will include your header.
Depends what you are trying to achieve.
If you are trying to force a non-cached response and dont have control over the server, one thing you can do is to add a fake query param like this.
options = {
uri :`${SUPPORT_FEED_URI}?${new Date().getTime()}`,
headers : {
'Content-Type': 'application/x-www-form-urlencoded'
},
}
For more information on the 'Cache-Control' header see the top answer here.
What's the difference between Cache-Control: max-age=0 and no-cache?
I have been trying to get into a website and fetch some data using python. But I am facing an error when I run my script. Here I am trying to just log in to the website and print the entire website text. Script and error are as below:
Script:
import requests
with requests.session() as s:
proxy_url = "http://{0}:{1}#proxy.blah.blah.com:8099".format('user_id', 'Password')
s.proxies = {'http': proxy_url, 'https': proxy_url}
user_id_url = "https://example.ex.com/my.policy"
Headers = {'Host': 'example.ex.com', 'Connection': 'keep-alive','Cache-Control': 'max-age=0', 'Accept-Language': 'en-US,en;q=0.8', 'Accept-Encoding': 'gzip, deflate, sdch','Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8', 'Cookie': '_ga=GA1.2.1822238803.1429212674; LastMRH_Session=0a0d8c67; MRHSession=ded054e0afe1bb151c3d35cb0a0d8c67; TIN=273000', 'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.101 Safari/537.36'}
data = data = {'username': 'website_uid', 'password': 'website_password'}
r= s.post(user_id_url, data = data, headers = Headers)
print r.status_code
print r.text
Error:
<script language="javascript">
setDefaultLang();
</script><noscript>
<div id="noscript_warning_red">JavaScript is not enabled. Please enable JavaScript in your browser or contact your system administrator for assistance.</div>
<div id="noscript_warning_newsession">To open a new session, please click here.</div>
</noscript>
PS: I am able to print the html text of the page, but I am not able to login correctly and hence displaying the error output.
Javascript is enabled in my browser, I double checked it even while posting this question
Any help is really appreciated
I have been trying to log in to a .aspx site (https://web.iress.com.au/html/LogonForm.aspx - For source / initial cookie reference) which uses a javascript function __doPostBack(eventTarget, eventArgument) to submit the form (very limited knowledge of javascript- so best guess).
My current understanding of HTTP requests is that, in the context of forms, they are mainly of a POST type request. I used Chrome to sniff out the request Headers and form data used when my credentials weren't typed in (For security sake) and they are as follows:
Remote Address:##BLANKEDOUT##
Request URL:https://web.iress.com.au/html/logon.aspx
Request Method:POST
Status Code:302 Found
**Request Headers**
Accept:text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Accept-Encoding:gzip,deflate
Accept-Language:en-US,en;q=0.8
Cache-Control:no-cache
Connection:keep-alive
Content-Length:585
Content-Type:application/x-www-form-urlencoded
Cookie:ASP.NET_SessionId=##SESSION ID STRING##
Host:web.iress.com.au
Origin:https://web.iress.com.au
Pragma:no-cache
Referer:https://web.iress.com.au/html/LogonForm.aspx
User-Agent:Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/##ADDRESS## Safari/537.36
**Form Data**
__EVENTTARGET:
__EVENTARGUMENT:
__VIEWSTATE: ##VIEWSTATE STRING##
__VIEWSTATEGENERATOR:##VIEWSTATEGENERATOR KEY##
__PREVIOUSPAGE: ##PREVIOSUPAGE STRING##
__EVENTVALIDATION: ##STRING##
fu:LogonForm.aspx
su:Default.aspx
un: # Would be my username if i had typed it in
pw: # Would be password
ImageButton1.x:45 # These two values change depending on where i click the submit button
ImageButton1.y:13
and this is the code I'm using to attempt a login:
from requests import session
payload = {
'__EVENTTARGET' : '',
'__EVENTARGUMENT' : '',
'__VIEWSTATE' : '##STRING FOUND FROM CHROME SNIFF##',
'__VIEWSTATEGENERATOR' : '##STRING FOUND FROM CHROME SNIFF##',
'__PREVIOUSPAGE' : '##STRING FOUND FROM CHROME SNIFF##',
'__EVENTVALIDATION' : '##STRING FOUND FROM CHROME SNIFF##',
'fu' : 'LogonForm.aspx',
'su' : 'Default.aspx',
'un' : 'myuser#company',
'pw' : 'mypassword',
'ImageButton1.x' : '0',
'ImageButton1.y' : '0'
}
requestheaders = {
'Accept' : 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8',
'Accept-Encoding' : 'gzip,deflate',
'Accept-Language' : 'en-US,en;q=0.8',
'Cache-Control' : 'no-cache',
'Connection' : 'keep-alive',
'Content-Type' : 'application/x-www-form-urlencoded',
'Host' : 'web.iress.com.au',
'Origin' : 'https://web.iress.com.au',
'Cookie' : '',
'Pragma' : 'no-cache',
'Referer' : 'https://web.iress.com.au/html/LogonForm.aspx',
'User-Agent' : 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/##ADRESSS AS ABOVE## Safari/537.36'
}
with session() as sesh:
LOGINURL = 'https://web.iress.com.au/html/LogonForm.aspx'
sesh.get(LOGINURL) #Get request to get the session ID cookie
sessionID = sesh.cookies['ASP.NET_SessionId'] #Grab session ID value
sessionIDname = 'ASP.NET_SessionId='
sessionIDheader = str(sessionIDname + sessionID) #Prepare session ID header
requestheaders['Cookie'] = sessionIDheader # Add session ID header to requestheaders dictionary
response = sesh.post('https://web.iress.com.au/html/LogonForm.aspx', data=payload, headers=requestheaders)
print(response.headers)
print(response.content)
All I seem to get is the source of the page (https://web.iress.com.au/html/LogonForm.aspx) for the content and its headers as a response. I am not sure if it has anything to do with the __VARIABLES either but they don't seem to change, previouspage being the exception. Would I possibly have to extract these __VARIABLES to use them in my request headers?
You are posting to the wrong URL; your own data shows the form posts to https://web.iress.com.au/html/logon.aspx but you are posting to /LogonForm.aspx instead.
Note that the session object will take care of the cookie for you, do not set the Cookie header yourself. You should avoid setting the Host, Origin and Content-Type headers, and the Cache-Control, Accept* headers and Pragma are not going to have any influence how this works.