I'm doing a project where my parser steals gets data about every video on the specific site and save it to my database. I have accomplished everything except full link to the video which is hidden.
There is a player, which automaticaly starts on page load. I have found the JavaScript code which starts the player:
function getVidData(resolution, init) {
<< some code here >>
jQuery.ajax({type: 'POST', url: '/ajaxdata.php', dataType: 'json', data: 'mod=videodata&vid=48902&res=' + resolution, success: function (response) {
if (response.error != '' && response.error != undefined) {
<< error handling code here >>
} else {
StartPlayer(response.width, response.height, response.filename);
}
} });
}
So after a call if no error found it starts a player using filename from response. That is what I need.
I rechecked a call in Live HTTP Headers:
http://<< SITE_URL >>/ajaxdata.php
POST /ajaxdata.php HTTP/1.1
Host: << SITE_URL >>
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:28.0) Gecko/20100101 Firefox/28.0
Accept: application/json, text/javascript, */*; q=0.01
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
Content-Type: application/x-www-form-urlencoded; charset=UTF-8
X-Requested-With: XMLHttpRequest
Referer: << VIDEO_PAGE >>
Content-Length: 31
Cookie: << COOKIE VALUES >>
DNT: 1
Connection: keep-alive
Pragma: no-cache
Cache-Control: no-cache
mod=videodata&vid=48901&res=640
HTTP/1.1 200 OK
Server: nginx/1.5.9
Date: Tue, 22 Apr 2014 16:30:06 GMT
Content-Type: text/html
Transfer-Encoding: chunked
Connection: keep-alive
Expires: Tue, 22 Apr 2014 16:30:05 GMT
Cache-Control: no-cache
Pragma: no-cache
Content-Encoding: gzip
So it calls ajaxdata.php with specific params and in response i should find the filename.
However this Python code returns absolutely nothing to me (neither content nor errors)
import requests
url = "http://LALLALAA/ajaxdata.php"
data_video = {"mod": "videodata", "vid": "48901", 'res': '640'}
s = requests.Session()
s.post(login_url, data=login_data) # Authentication
content = s.post(url, data=data_video)
print content.content
Variable content prints only "Response [200]"
Now I'm completely stuck and would be grateful if anyone could point to errors I done or solutions i could try.
Thanks
As
Martijn Pieters
suggested, I tried headers one by one and found that this combination is working now:
import requests
headers = {
'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; WOW64; rv:28.0) Gecko/20100101 Firefox/28.0',
'Accept': 'application/json, text/javascript, */*; q=0.01',
'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8',
'X-Requested-With': 'XMLHttpRequest'
}
s = requests.Session()
s.post(login_url, data=login_data)
content = s.post(url, data=data_video, headers=headers)
I thank everyone and especially Martijn Pieters.
Related
I have a button in HTML.
<button class="btn btn-success" id="invoicePrint" disabled="disabled">
<i class="fa fa-print"></i> Print Invoice
</button>
On click of button, ajax is called
Ajax End Code looks like this:
$('#invoicePrint').click(function (e) {
e.preventDefault();
documentCommon.ajax({
dataType: 'text',
type: 'GET',
url: '/download/invoice/' + $("#invoiceId").val() ,
data: {
'_CONV_ID': $('input[name="_CONV_ID"]').val()
},
success: function (data) {
},
error:function (xhr, ajaxOptions, thrownError) {
alert("Server Side issues. Kindly retry or contact system administrator");
}
});// End ajax
});
When i click on button; Nothing happens, no file download
but when i chrome inspect i can see following details:
Request URL: http://localhost:8080/download/invoice/6
Request Method: GET
Status Code: 200
Remote Address: [::1]:8080
Referrer Policy: strict-origin-when-cross-origin
Access-Control-Allow-Headers: Content-Type
Access-Control-Allow-Methods: GET
Access-Control-Allow-Origin: *
Cache-Control: no-cache, no-store, max-age=0, must-revalidate
Connection: keep-alive
Content-Disposition: attachment; filename=Invoice_99_6.pdf
Content-Length: 430385
Content-Type: application/pdf
Date: Sun, 17 Jan 2021 06:21:50 GMT
Expires: 0
Keep-Alive: timeout=60
Pragma: no-cache
X-Content-Type-Options: nosniff
X-Frame-Options: DENY
X-XSS-Protection: 1; mode=block
Accept: text/plain, */*; q=0.01
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.9
Connection: keep-alive
Cookie: JSESSIONID=A8A8ACC5AD3A925394EDB729684624FA
Host: localhost:8080
Referer: http://localhost:8080/welcome
sec-ch-ua: "Google Chrome";v="87", "\"Not;A\\Brand";v="99", "Chromium";v="87"
sec-ch-ua-mobile: ?0
Sec-Fetch-Dest: empty
Sec-Fetch-Mode: cors
Sec-Fetch-Site: same-origin
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.66 Safari/537.36
X-Requested-With: XMLHttpRequest
Also a response binary data:
From internet i can see guys using href to download. but is it possible without using href in above code or any other alternative as browser has .pdf plugin which will be used to open this.
Give it a try "$.ajax()"？
I'm trying to fetch data from my Jira Cloud instance through the Jira and Jira Agile REST APIs using JavaScript in a browser. Queries to Jira REST API work fine but identical queries to Jira Agile REST API keep failing with the response
Response for preflight has invalid HTTP status code 401.
I'm using Basic Authentication with user ID and an API token obtained from Jira. With cURL and ARC, I'm able to successfully retrieve data both from the Jira REST API and the Jira Agile REST API, so the authentication against both APIs seems to work. In JS I have tried with both fetch() and jquery ajax() and the result was the same.
function fetchFromJira(url, id, token) {
const authorizationString = 'Basic ' + btoa(id + ':' + token);
const options = {
method: 'GET',
headers: {
Authorization: authorizationString,
'Content-Type': 'application/json',
},
};
fetch(url, options)
.then(response => {
if (response.ok) {
return response.json();
} else {
throw new Error(response.status);
}
})
.then(json => {
console.log(json);
})
.catch(error => {
console.log(error);
});
}
fetchFromJira(
'https://fredrikastrom.atlassian.net/rest/api/latest/issue/10000',
'<user id>',
'<API token>'
); // successful
fetchFromJira(
'https://fredrikastrom.atlassian.net/rest/agile/1.0/board',
'<user id>',
'<API token>'
); // fails
The output onto the console looks as follows:
test.js:11 OPTIONS https://fredrikastrom.atlassian.net/rest/agile/1.0/board 401 ()
fetchFromJira # test.js:11
(anonymous) # test.js:33
index.html:1 Failed to load https://fredrikastrom.atlassian.net/rest/agile/1.0/board: Response for preflight has invalid HTTP status code 401.
test.js:23 TypeError: Failed to fetch
test.js:20 {expand: "renderedFields,names,schema,operations,editmeta,changelog,versionedRepresentations", id: "10000", self: "https://fredrikastrom.atlassian.net/rest/api/latest/issue/10000", key: "FAT-1", fields: {…}}
Here are the preflight request and response headers of the successful query to the Jira REST API:
t=3241 [st= 89] HTTP_TRANSACTION_HTTP2_SEND_REQUEST_HEADERS
--> :authority: fredrikastrom.atlassian.net
:method: OPTIONS
:path: /rest/api/latest/issue/10000
:scheme: https
accept: */*
accept-encoding: gzip, deflate, br
accept-language: sv-SE,sv;q=0.9,en-US;q=0.8,en;q=0.7,fi;q=0.6
access-control-request-headers: authorization,content-type
access-control-request-method: GET
cache-control: no-cache
origin: http://127.0.0.1:8080
pragma: no-cache
referer: http://127.0.0.1:8080/test/index.html
user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.62 Safari/537.36
t=3241 [st= 89] -HTTP_TRANSACTION_SEND_REQUEST
t=3241 [st= 89] +HTTP_TRANSACTION_READ_HEADERS [dt=68]
t=3278 [st=126] HTTP2_STREAM_UPDATE_SEND_WINDOW
--> delta = 0
--> stream_id = 1
--> window_size = 65535
t=3309 [st=157] HTTP_TRANSACTION_READ_RESPONSE_HEADERS
--> HTTP/1.1 200
status: 200
server: AtlassianProxy/1.15.8.1
vary: Accept-Encoding
cache-control: no-cache, no-store, no-transform
content-type: text/html;charset=UTF-8
content-encoding: gzip
strict-transport-security: max-age=315360000; includeSubDomains; preload
date: Sat, 12 Oct 2019 06:33:50 GMT
atl-traceid: 519aa518a8e8e5ea
x-arequestid: c68d7b95-3635-49e1-a2fd-971e0502adf5
x-xss-protection: 1; mode=block
timing-allow-origin: *
x-content-type-options: nosniff
set-cookie: atlassian.xsrf.token=7a27221d-39bc-4555-9569-b26a0beb9689_b9e038120f5696c0bac7202f986ee24d3752c6fa_lout; Path=/; Secure
and here are the correspinding headers from the failing request to Jira Agile REST API:
t=5918 [st= 5] HTTP_TRANSACTION_HTTP2_SEND_REQUEST_HEADERS
--> :authority: fredrikastrom.atlassian.net
:method: OPTIONS
:path: /rest/agile/latest/board
:scheme: https
accept: */*
accept-encoding: gzip, deflate, br
accept-language: en-GB,en-US;q=0.9,en;q=0.8
access-control-request-headers: authorization,content-type
access-control-request-method: GET
origin: http://127.0.0.1:8080
referer: http://127.0.0.1:8080/test/index.html
user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.62 Safari/537.36
t=5919 [st= 6] -HTTP_TRANSACTION_SEND_REQUEST
t=5919 [st= 6] +HTTP_TRANSACTION_READ_HEADERS [dt=65]
t=5984 [st=71] HTTP_TRANSACTION_READ_RESPONSE_HEADERS
--> HTTP/1.1 401
status: 401
server: AtlassianProxy/1.15.8.1
vary: Accept
www-authenticate: OAuth realm="https%3A%2F%2Ffredrikastrom.atlassian.net"
cache-control: no-transform
content-type: application/xml;charset=UTF-8
strict-transport-security: max-age=315360000; includeSubDomains; preload
date: Sat, 12 Oct 2019 07:05:10 GMT
atl-traceid: 2caf28fb1cce9a77
x-arequestid: 817e2b89-e3d1-431b-b892-781fc78c9669
x-xss-protection: 1; mode=block
timing-allow-origin: *
x-content-type-options: nosniff
set-cookie: atlassian.xsrf.token=7a27221d-39bc-4555-9569-b26a0beb9689_dafc86c05dbdc472c9b99300b351fe0dd62b305d_lout; Path=/; Secure
content-length: 174
Interestingly, the request headers look slightly different even if the requests are made with the same function where only the requested URLs differ. The succeeding request includes the cache-control and pragma headers, and the accept-language header includes one more language. But none of these should reasonably have any impact on whether the server will accept the preflight request?
Any clue why the one request succeeds and the other one fails?
Below is the call that I make to the Rest API.It is just the part of the bigger script.
SomeServiceService.addNewCall = function(data)
{
deferred = $q.defer();
addNewPaymentMethodsServiceCall = $http({
url: rootUrl + 'user/v1/something',
method: 'POST',
data: data,
headers: {
'Content-Type': 'application/json',
'Accept-Language': 'en-us'
}
}).success(function(response){
deferred.resolve(response);
}).error(function(response,status){
response.status= status;
deferred.reject(response);
});
return deferred.promise;
};
The issue I am having it the difference in Content-Type in chrome and firefox.
In chrome
Accept-Language:en-US
Cache-Control:no-cache
Content-Type:application/json
If-Modified-Since:Mon, 26 Jul 1997 05:00:00 GMT
Origin:https://mydomain
Pragma:no-cache
Referer:mydomain/users/app/
User-Agent:Mozilla/5.0(Macintosh; Intel Mac OS X 10_12_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.143
Safari/537.36
In Firefox
Accept text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Encoding gzip, deflate
Accept-Language en-US
Cache-Control no-cache
Connection keep-alive
Content-Length 157
Content-Type application/json; charset=UTF-8
Host mydomain
If-Modified-Since Mon, 26 Jul 1997 05:00:00 GMT
Pragma no-cache
Referer https://mydomain/users/app/
User-Agent Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:43.0) Gecko/20100101 Firefox/43.0
Even though in the ajax call I am explicitly declaring the Content-type as application/json firefox adds the charset-UTF-8 which breaks my API call since the API only supports application/json.
How can I make firefox not change the Content-Type? This only happens for POST,PUT.
Any suggestions or help is appreciated.
Quentin's answer above is correct:
You should really fix the API. In 2016 it should be able to cope with having a parameter on the content-type.
We had the same issue today and solved it by changing the API: instead of checking whether the content-type equals 'application/json', we check whether the content type contains that string.
Specific code is language-specific, but for Python (using the Falcon REST framework) it was a matter of replacing
if req.content_type == 'application/json':
with
if 'application/json' in req.content_type:
I am trying to make an ajax request like below:
function updateLastSeen() {
var url = 'http://my.url.com/conversation/' + $('#conversationId').val() + '/seen/' + $('#senderId').val();
$.ajax({
type: 'PUT',
url: url,
contentType: "application/json",
success: function(result) {
alert('Logged out')},
error: function(result) {
alert('error')
}
});
}
The request in the preflight looks like this:
Request headers
OPTIONS http://my.url.com/conversation/3/seen/3 HTTP/1.1
Host: m.url.com
User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:37.0) Gecko/20100101 Firefox/37.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
Origin: null
Access-Control-Request-Method: PUT
Connection: keep-alive
And this is the response
HTTP/1.1 200 OK
Date: Wed, 15 Apr 2015 06:16:54 GMT
Server: Apache
Access-Control-Allow-Origin: *
Access-Control-Allow-Methods: *
Access-Control-Allow-Headers: Origin, X-Requested-With, Content-Type, Accept
Keep-Alive: timeout=5, max=98
Connection: Keep-Alive
Transfer-Encoding: chunked
Content-Type: application/json
So according to me it should work. However I get an error that Method PUT is not allowed by Access-Control-Allow-Methods.
I also tried via Postman to see if there is a problem with the backend (which is totally obscure to me) but via Postman it works! So what am I doing wrong?
The request headers with postman:
PUT /conversation/4/seen/3 HTTP/1.1
Host: my.url.com
Connection: keep-alive
Content-Length: 0
Pragma: no-cache
Cache-Control: no-cache
Origin: chrome-extension://fdmmgilgnpjigdojojpjoooidkmcomcm
User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.118 Safari/537.36
Content-Type: text/plain;charset=UTF-8
Accept: */*
Accept-Encoding: gzip, deflate, sdch
Accept-Language: en-US,en;q=0.8,nl;q=0.6
Cookie: _ga=GA1.2.1794721550.1428851890
The code below is throwing error without going to .fail or .always when it receives "401 - Unauthorized" error. Is there a way to trigger fail and retrieve the error [myErrorMessage] that comes in response from server?
JavaScript
$(document).ready(function() {
var jqXHR = $.ajax({
url: "http://[myDomain]/[myPage]",
dataType: "jsonp",
jsonpCallback: "myFunction"
})
.done(function() { alert("success"); })
.fail(function() { alert("error"); })
.always(function() { alert("complete"); });
function myFunction(data) {
alert("callback");
}
});
Request Result
GET http://[myDomain]/[myPage]?callback=myJsFunction&_=1361463044315 HTTP/1.1
Host: [myDomain]
Connection: keep-alive
Accept: */*
User-Agent: Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.17 (KHTML, like Gecko) Chrome/24.0.1312.57 Safari/537.17
Referer: http://[myTotallyDifferentDomain]/Default.aspx
Accept-Encoding: gzip,deflate,sdch
Accept-Language: en-US,en;q=0.8
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3
Response Result
HTTP/1.1 401 Unauthorized
Cache-Control: no-cache
Pragma: no-cache
Content-Type: application/json; charset=utf-8
Expires: -1
Server: Microsoft-IIS/8.0
X-AspNet-Version: 4.0.30319
X-Powered-By: ASP.NET
Date: Thu, 21 Feb 2013 16:10:42 GMT
Content-Length: 67
{"Message":"[myErrorMessage]"}
Chrome Result
GET http://[myDomain]/[myPage]?callback=myJsFunction&_=1361463044315 401 (Unauthorized) jquery.min.js:5
JSONP requests do not go to the fail handler on error as of jQuery 1.9, this is the documented behavior.
There isn't a way around it as far as i know. (other than using a proxy, of course)
http://api.jquery.com/jQuery.ajax/