Post to AmazonAWS Kinesis Stream Using Query String Authentication - javascript

I've got the interesting (and theoretically impossible) task of getting AmazonAWS Kinesis analytics from IE 8 and 9. According to Amazon's own SDK, this is not possible since XDomainRequest does not allow custom headers. Contrary to this statement, however, AmazonAWS allows you to authenticate using query string parameters. My goal was to write a shim for XMLHttpRequest which utilized the XDomainRequest object and converted all Amazon headers into query string parameters.
The actual implementation turned out to be much more difficult than I would have liked. Since Amazon's query string authentication only uses the "host" for SignedHeaders (whereas the AmazonAWS SDK was attempting to use host, date, and target) I had to re-compute the signature. This meant CryptoJS and lots of experimentation to get everything working.
After 4 hours of receiving "Computed signature did not match", I finally started getting a different error code: Unable to determine service/operation name to be authorized
Googling this error was not very helpful: anything from a typo to an extra new-line character to using a datestamp instead of a version number. However I tried everything and nothing helped.
Below is an example cURL request and the return value:
curl -H "Content-Type:text/plain" --data "{\"Data\":\"VALID BASE64 DATA\",\"PartitionKey\":\"PARTITION\",\"StreamName\":\"STREAM\"}" "https://kinesis.us-east-1.amazonaws.com/?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAJMAGAYBGGRNZQI4A/20140723/us-east-1/kinesis/aws4_request&X-Amz-Date=20140723T153144Z&X-Amz-SignedHeaders=host&X-Amz-Target=Kinesis_20131202.PutRecord&X-Amz-User-Agent=aws-sdk-js/2.0.0&X-Amz-Signature=VALID_SIGNATURE"
Return:
<AccessDeniedException>
<Message>Unable to determine service/operation name to be authorized</Message>
</AccessDeniedException>
I've tried appending Action and Version parameters (noting that the Version should be in YYYY-MM-DD format as opposed to YYYYMMDD) and this didn't help. I also tried escaping all of my / characters or escaping all of my . characters (or both).
For comparison, here's the same request through Google Chrome using headers instead of a query string:
Remote Address:176.32.102.203:443
Request URL:https://kinesis.us-east-1.amazonaws.com/
Request Method:POST
Status Code:200 OK
Request Headers
Accept:* / *
Accept-Encoding:gzip,deflate,sdch
Accept-Language:en-US,en;q=0.8
Authorization:AWS4-HMAC-SHA256 Credential=AKIAJMAGAYBGGRNZQI4A/20140723/us-east-1/kinesis/aws4_request, SignedHeaders=host;x-amz-date;x-amz-target, Signature=OMITTED
Cache-Control:no-cache
Connection:keep-alive
Content-Length:3236
Content-Type:application/x-amz-json-1.1
Host:kinesis.us-east-1.amazonaws.com
Origin:OMITTED
Pragma:no-cache
Referer:OMITTED
User-Agent:Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.125 Safari/537.36
X-Amz-Date:20140723T145554Z
X-Amz-Target:Kinesis_20131202.PutRecord
X-Amz-User-Agent:aws-sdk-js/2.0.0
Request Payload
OMITTED (because it's long)
Response:
{"SequenceNumber":"49540780386103606919741841581837328106424971136629473281","ShardId":"shardId-000000000000"}
Does anyone know what I'm doing wrong, and why I can't communicate with Kinesis?

Going through and cleaning up some old questions that never got answers.
Per Michael - sqlbot:
Plan B is to send your request to your application server and proxy it to kinesis
This turned out to be the only solution for communicating with Kinesis. Set up a proxy which allowed me to pass custom headers as a query string, then it recombobulated it and sent the request onward

Related

Why am I able to send cross-domain ajax requests from the javascript console?

How come when I go to almost any website, for example SO, if I open up the console, inject jQuery and send a cross-domain ajax request to a server I have running on my localhost, I don't get any errors as I would have expected? However, if I open up one of the webpages that I have written myself, and which is also running on my localhost (but on a different port from the one used by the server), if I try to send an ajax request from the console I get this message:
XMLHttpRequest cannot load https://localhost:10000/. Request header field My-First-Header is not allowed by Access-Control-Allow-Headers in preflight response.
The ajax request looks like:
$.ajax({
type: 'POST',
url: 'https://localhost:10000',
headers: {
"My-First-Header":"first value",
"My-Second-Header":"second value"
}
})
To be clear, my question is not about how to fix this, but rather why I am even able to make cross-domain requests from most other websites (shouldn't they be not allowed?). Do these sites have some sort of mechanism set up that automatically bypasses the restrictions?
Request headers:
Accept:*/*
Accept-Encoding:gzip, deflate, sdch
Accept-Language:en-US,en;q=0.8
Connection:keep-alive
Cookie:csrftoken=lxe5MaAlb9GC5lPGQpXtSj9HvCP0QhCz; PHPSESSID=uta0nlhlh8r1uimdklmt3v3ho1
Host:localhost:10000
Referer:http://stackoverflow.com/
User-Agent:Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.94 Safari/537.36
Response headers:
Content-Length:3
Content-Type:text/html
Date:Mon, 16 May 2016 06:29:03 GMT
Server:TwistedWeb/16.0.0
It seems obvious what's going on here: the browser simply handles code typed in the console differently than code run by the page itself. It makes philosophical sense that it would work that way. After all, the point of the Same-Origin Policy is to prevent XSS and CSRF attacks, and if a user opens up their console and sends cross-domain requests, they're just attacking themselves.
On the other hand, it is possible to trick users into performing XSS on themselves. If you go to Facebook and open up the console, Facebook has code that logs a warning message telling ordinary users not to paste unknown code into the console because it could be malicious. Apparently that's a problem they've seen.
This is called as CORS request.
By default all Cross Domain requests are blocked by most of the major browsers.
Most of the portals services that you are able to request cross domain does special settings in Response. If you don't provide those settings at api level then your api will be blocked for cross domain requests.
These Response settings are as follows:
Response Header Need to have access-control-allow-origin attribute.
access-control-allow-origin can specify * for every api service at global level.
access-control-allow-origin can specify specific method name for every api service separately..

MixedContent when I'm loading https page through ajax, but browser still thinks it's http

After Installing SSL Cert on a web page, I had the problem where the page served with https would require http endpoint with ajax.
I'm using restangular, and I changed the base url to have https.
var uri = location.protocol + "//" + location.host;
RestangularProvider.setBaseUrl(uri);
The interesting part is that, when I see the request in the chrome developer tools i see
Request URL:https://theaddress.com/api/endpoint
Request Headers
Provisional headers are shown
Accept:application/json, text/plain, */*
Referer:https://theadress.com/somepage
User-Agent:Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.93 Safari/537.36
X-Requested-With:XMLHttpRequest
So the request should be an https one, yet I still get:
Mixed Content: The page at 'https://theaddress.com/somepage' was loaded over HTTPS, but requested an insecure XMLHttpRequest endpoint 'http://theadress.com/api/endpoint'. This request has been blocked; the content must be served over HTTPS.
Also I should mention, this happens on prod server, but on my local test it works fine ( I have self signed ssl cert ) after I have made it use base url that includes https.
What could be the problem?
I just spent a good 4 hours trying to fix a similar problem. Here's what solved mine:
Summary: add a trailing '/' to your request
I found this post useful in fixing my problem. Basically, the server doesn't care if you send your request with a trailing '/' or not, because it internally routes to '/' if you don't add it. However, if the routing is happening internally (e.g. nginx passing the request to a local process), you get an http redirect which will make your request fail.
I tried #Kadi's fix of adding a slash, and it worked, but a more elegant solution for me was to change the request from GET to POST, which also fixed the issue.
Still not sure what the root cause was.

Yahoo - OAuth2 - SocialAPI : Not returning "Access-Control-Allow-Origin" in initial response [duplicate]

This question already has answers here:
Ways to circumvent the same-origin policy
(8 answers)
Closed 7 years ago.
I am using Yahoo Social API for Contacts using OAuth2 via Javascript (as given here https://developer.yahoo.com/oauth2/guide/#implicit-grant-flow-for-client-side-apps)
However, after successful authentication and correct Access Token, I am unable to complete the call via JS. The Browser says:
XMLHttpRequest cannot load https://social.yahooapis.com/v1/user/me/contacts
No 'Access-Control-Allow-Origin' header is present on the requested resource. Origin 'http://..' is therefore not allowed access.
However, when I try PHP's Curl with the same call, it works (hence proves that its not an issue with the token).
Anyone knows how to solve this?
Thanks.
So the issue is with https://social.yahooapis.com/ API and not your code, however there are ways around it.
Someone needs to contact the Yahoo Social API Developers and tell them to implement the following solution:
Say for example your aouth2 access token is "XXXXXXXX" and you make the following request to get the userid from your javascript code.
$.ajax({
url: 'https://social.yahooapis.com/v1/me/guid?format=json',
beforeSend: function (xhr) {
xhr.setRequestHeader ("authorization", "Bearer " + "XXXXXXXX");
},
success:function(guuid) {
console.log(guuid);
}
});
Before actually sending this HTTP GET request to the social.yahooapis.com domain, your browser recognizes this is a CORS request (making a request which is not the same as the origin domain) and does a "preflight check" with HTTP OPTIONS to see if this is a valid call.
This is what the preflight request looks like:
Request Header
:host:social.yahooapis.com
:method:OPTIONS
:path:/v1/me/guid?format=json
:scheme:https
:version:HTTP/1.1
accept:*/*
accept-encoding:gzip, deflate, sdch
accept-language:en-US,en;q=0.8
access-control-request-headers:accept, authorization
access-control-request-method:GET
cache-control:no-cache
origin:http://yourorigin.io
pragma:no-cache
referer:http://yourorigin.io
user-agent:Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.135 Safari/537.3
AND the yahoo API responds back with
Response Header
age:0
allow:OPTIONS,HEAD,GET
content-length:0
content-type:application/vnd.sun.wadl+xml
date:Sun, 10 May 2015 19:21:40 GMT
last-modified:Thu, 30 Apr 2015 13:20:58 PDT
server:ATS
servletwebservicefilter-enabled:true
status:200 OK
vary:Accept
version:HTTP/1.1
via:http/1.1 r18.ycpi.ne1.yahoo.net (ApacheTrafficServer [c sSf ]), https/1.1 r26.ycpi.sjb.yahoo.net (ApacheTrafficServer [c sSf ])
x-yahoo-social-data-source:default_source
x-yahoo-social-host:ws127.progrss.ne1.yahoo.com
y-rid:er2nai1akvbu4
Even though the Reponse comes back with 200OK status it is missing the following Response Header:
Access-Control-Allow-Origin:*
Chrome and other modern web browsers have a safety feature builtin that if you are making a GET CORS request and the Response does not have the Header Access-Control-Allow-Origin then the following message is displayed in the logs REGARDLESS of what is actually returned by the social.yahooapis.com
XMLHttpRequest cannot load https://social.yahooapis.com/v1/me/guid?format=json.
No 'Access-Control-Allow-Origin' header is present on the requested resource.
Origin 'http://yourorigin.com' is therefore not allowed access.
So what you actually get back is :
{
"guid" :
{
"uri": "XXXX", // URI value
"value": "XXXX"
}
}
However because of your browsers security features it shows:
No 'Access-Control-Allow-Origin' header is present on the requested resource.
It works fine for people who are using YDN SDK (non browser ways to access the API), server side implementation and old web browser. However doesnt work for new modern browers. Which also explains the inconsistent user experience of it working for some people but not other people in this forum https://developer.yahoo.com/forum/OAuth-General-Discussion-YDN-SDKs/http-social-yahooapis-com-Will-be-right-back/1395509802423-89faffa2-1503-486d-bc29-6505719bd774/
The only way to actually use this right now is to make the GET HTTP request from your server code instead of the client javascript code. And since your server does not have security features it will receive the actual result.
However to fix the issue yahoo developers to add Access-Control-Allow-Origin:*
to allow client javascript requests to their api.
I wrote this up because I know many people will be going to the same issue. If this was helpful to you and the issue still hasnt been fixed please contact the yahoo developers and get them to implement this solution.

Obtain JSON response from URL with AJAX using Javascript

I'm relatively new to Javascript and I'm stuck trying to obtain a JSON data from an URL using AJAX.
The url returns an array on characters that I want to request/obtain and then handle the data to show it in html. That url is:
http://stark-tundra-9859.herokuapp.com/locations
The code that I'm using is the following, and the problem is that it appears as if I received nothing for response. Besides I dont know what the request info variable should be:
function ajax_request() {
requestInfo='';
var params = JSON.stringify(requestInfo);
$.ajax({
type: "GET",
url: 'http://stark-tundra-9859.herokuapp.com/locations',
data: params,
contentType: "application/json",
dataType: "json",
converters: {
'text json': true
},
success: function(response) {
$("#responseParagraph").html(response);
},
error: function(error) {
$("#responseParagraph").html(error.responseText);
}
});
}
#agam360, I also have done a version of this code using JQUERY and I do receive a message in the console which goes as follows:
GET http://stark-tundra-9859.herokuapp.com/locations 200 OK 198ms
Response header
Connection keep-alive
Content-Length 154
Content-Type application/json;charset=utf-8
Server thin 1.5.1 codename Straight Razor
X-Content-Type-Options nosniff
Request header
Accept application/json, text/javascript, /; q=0.01
Accept-Encoding gzip, deflate
Accept-Language es-ES,es;q=0.8,en-US;q=0.5,en;q=0.3
Connection keep-alive
Host stark-tundra-9859.herokuapp.com
Origin null
User-Agent Mozilla/5.0 (Windows NT 6.1; rv:16.0) Gecko/20100101 Firefox/16.0
The code used to receive that answer is the following:
function json_request() {
$.getJSON(url,
function(data) {
alert(data);
$('#responseParagraph').append("<p>"+data.responseMessage+"</p>");
});
}
In this JQUERY very It seems as if I dont receive the DATA from the JSON request correctly. Maybe I am handling it erronously?
I would greatly appreciate any help in advance!
To answer this question I will need to ask you two things:
1) On what domain are you running this script? (on the same server as "http://stark-tundra-9859.herokuapp.com/" ?)
Because that URL does not allow CORS (Cross-Origin Resource Sharing)
2) Open up your console (in chrome: Ctrl+shift+j) and tell us what error message you get, if you get any.
If your answer to the first question is no (meaning you are not running the script on the same host) then, if you have control over that page, enable CORS by sending the following header (please read some info about this, related to security before implementing):
Access-Control-Allow-Origin: *
More language specific implementations can be found on enable-cors.org
(You can also read about 'JSONP' implementations regarding cross domain requests - if your not willing to use CORS).
On the other hand, (your answer is yes), we will need to see your server-side related code, and any error messages thrown on the client side.
Note: From what I see (as to now - 5 minutes after posting my answer), the http://stark-tundra-9859.herokuapp.com/locations URL returns a "500 - Internal server error" - Your problem seems to be related to server side of things.
Update,
You need to acess your data as a JSON object after you got it,
like so:
data[0].lat // will hold the lat value of the first object in the JSON wrapper object
As to implementing JSONP with jQuery, I ran a quick search in SO, here is something you will probably want to take a look at: jQuery AJAX cross domain

Difference between AJAX request and a regular browser request

Is there a difference between an AJAX request and a direct browser request (in terms of how a web page is called and loaded)?
In other words, I mean: is a direct server-side request handled in any way differently than a client-side request (initiated by the browser)?
There may be some header differences, but the main behavior difference is on the client.
When the browser makes a regular request as in window.location.href = "index.html", it clears the current window and loads the server response into the window.
With an ajax request, the current window/document is unaffected and javascript code can examine the results of the request and do what it wants to with those results (insert HTML dynamically into the page, parse JSON and use it the page logic, parse XML, etc...).
The server doesn't do anything different - it's just in how the client treats the response from the two requests.
An AJAX request is identical to a "normal" browser request as far as the server is concerned other than potentially slightly different HTTP headers. e.g. chrome sends:
X-Requested-With:XMLHttpRequest
I'm not sure if that header is standardized or not, or if it's different in every browser or even included at all in every browser.
edit: I take that back, that header is sent by jQuery (and likely other JS libraries), not the browser as is evidenced by:
var xhr = new XMLHttpRequest();
xhr.open('GET', '/');
xhr.send();
which sends:
Accept:*/*
Accept-Charset:ISO-8859-1,utf-8;q=0.7,*;q=0.3
Accept-Encoding:gzip,deflate,sdch
Accept-Language:en-US,en;q=0.8
Connection:keep-alive
Cookie: ....
Host:stackoverflow.com
If-Modified-Since:Sat, 31 Dec 2011 01:57:24 GMT
Referer:http://stackoverflow.com/questions/8685750/how-does-an-ajax-request-differ-from-a-normal-browser-request/8685758
User-Agent:Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_7) AppleWebKit/535.11 (KHTML, like Gecko) Chrome/17.0.963.12 Safari/535.11
which leads me to the conclusion that by default there is absolutely no difference.
Some popular client-side libraries like jQuery include the X-Requested-With header in their requests and set it to XMLHttpRequest to mark them as AJAX.
This seems to have been considered standard enough a few years ago (probably due to the huge popularity of jQuery and its presence in almost every website) that many server-side frameworks even have helpers that take care of checking for this header in the received request for you:
ASP.NET MVC 5:
HttpRequestBase.IsAjaxRequest()
Django:
HttpRequest.is_ajax()
Flask:
flask.Request.is_xhr
However, it seems that with the end of jQuery's reign in the front end world and the standardization of the fetch API and the rise of other modern client-side libraries that don't add any header for this purpose by default, the pattern has fallen into obsolescence also in the backend; with ASP.NET MVC not including the helper in newer versions and Flask marking it as deprecated.
Not really. Except that most Ajax clients send a X-Requested-With=XMLHttpRequest HTTP header
I always check if "text/html" is the request's "best" Accept mimetype, because browsers always send that as the first.
Firefox example:
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Of course, this may still be a ajax request with text/html as the Accept mimetype, but I found this to be reliable when you know what client will consume your backend api.
An AJAX request in Blink and Gecko-powered browsers (Firefox, Chrome, Edge etc) will send this header with AJAX requests:
Sec-Fetch-Dest: empty
That means that:
The destination is the empty string. This is used for destinations that do not have their own value. For example fetch(), navigator.sendBeacon(), EventSource, XMLHttpRequest, WebSocket, etc.
Safari does not send this header at the time of writing (and IE never has and never will, but it's only a few months before IE should be completely irrelevant).
your user-agent, aka browser, sends an XHR header which you can catch from php like this:
$_SERVER['HTTP_X_REQUESTED_WITH']

Categories