Now as example, I'm getting an response which has partially the key/values as an javascript object:
status: '200 OK',
'content-encoding': 'gzip'
I can easily read out and log the status message by: headers.status but when I try to log the content-encoding (which I need in this particular situation) it errors on:
headers.'content-encoding' <- obviously the quotes it doesn't like
headers.content-encoding <- obviously the '-' it doesn't like
How am I suppose to get/read/log it's content-encoding value?
Greets,
m0rph3v5
Javascript also supports square bracket notation for referring to properties so if headers is an appropriate object, you can use headers['content-encoding'].
JavaScript properties have names as you know. When the name is a legal identifier and you know the literal name you want when you're writing the code, you can use it with dotted notation.
var foo = headers.foo;
When the name isn't a legal identifier, or if you want to determine the name you're looking up at runtime, you can use a string:
var encoding = headers['content-encoding'];
or
var name = 'content-encoding';
var encoding = headers[name];
or even
var x = 'encoding';
var encoding = headers['content-' + x];
As you can see, it doesn't have to be a literal string. This is very handy for general-purpose functions that have to accept a property name as a function argument or similar.
Note that property names are case sensitive.
I think you should install the very good express framework. I really simplifies node.js webdevelopment.
You could install it using npm
npm install express
This snippet shows you how to set headers and read headers
var express = require('express');
var app = express.createServer();
app.get('/', function(req, res){
console.log(req.header('a'));
res.header('time', 12345);
res.send('Hello World');
});
app.listen(3000);
Curl from command line
$curl http://localhost:3000/ -H "a:3434" -v
* About to connect() to localhost port 3000 (#0)
* Trying ::1... Connection refused
* Trying 127.0.0.1... connected
* Connected to localhost (127.0.0.1) port 3000 (#0)
> GET / HTTP/1.1
> User-Agent: curl/7.21.2 (i686-pc-linux-gnu) libcurl/7.21.2 OpenSSL/0.9.8o zlib/1.2.3.4 libidn/1.18
> Host: localhost:3000
> Accept: */*
> a:3434
>
< HTTP/1.1 200 OK
< X-Powered-By: Express
< time: 12345
< Content-Type: text/html; charset=utf-8
< Content-Length: 11
< Date: Tue, 28 Dec 2010 13:58:41 GMT
< X-Response-Time: 1ms
< Connection: keep-alive
<
* Connection #0 to host localhost left intact
* Closing connection #0
Hello World
The log outputting the header send via curl to node server:
$ node mo.js
3434
Related
Is it possible to know the nature of a URL before GETting it ?
I have one URL in particular that ends with m3u but is not a simple file I can download. This is actually a radio stream. As I expect a (finite) file, the GET method never ends. The timeout options does not works in this case (normal).
const options = {timeout: 5000};
return HTTP.call('GET', "http://av.rasset.ie/av/live/radio/junior.m3u", options);
The safe solution should be to ask for the type of the response before actually getting the file.
How can I do that?
Thanks,
Mickael.
I guess you can run a HEAD request first (instead of GET) and verify the headers. Then after making GET you will know how to react.
Unfortunately in this particular case HEAD works for the first request (which returns a redirect):
curl -v -X HEAD http://icecast1.rte.ie/junior http://av.rasset.ie/av/live/radio/junior.m3u
Warning: Setting custom HTTP method to HEAD with -X/--request may not work the
Warning: way you want. Consider using -I/--head instead.
* Trying 104.16.107.29...
* TCP_NODELAY set
* Connected to av.rasset.ie (104.16.107.29) port 80 (#0)
> HEAD /av/live/radio/junior.m3u HTTP/1.1
> Host: av.rasset.ie
> User-Agent: curl/7.51.0
> Accept: */*
>
< HTTP/1.1 302 FOUND
< Date: Mon, 13 Nov 2017 11:07:33 GMT
< Content-Type: text/html; charset=utf-8
< Connection: keep-alive
< Set-Cookie: __cfduid=d89353ae357a0452208835b3092f0fbee1510571253; expires=Tue, 13-Nov-18 11:07:33 GMT; path=/; domain=.rasset.ie; HttpOnly
< Location: http://icecast1.rte.ie/junior
< X-Server-Name: djd
< Cache-Control: max-age=0
< Accept-Ranges: bytes
< X-Varnish: 2802121867
< X-Served-By: MISS: mt-www2.rte.ie
< CF-Cache-Status: MISS
< Server: cloudflare-nginx
< CF-RAY: 3bd1449d2764410c-HAM
* no chunk, no close, no size. Assume close to signal end
<
^C
But fails for second (probably HEAD is not supported, should return 405):
curl -v -X HEAD http://icecast1.rte.ie/junior
Warning: Setting custom HTTP method to HEAD with -X/--request may not work the
Warning: way you want. Consider using -I/--head instead.
* Trying 89.207.56.171...
* TCP_NODELAY set
* Connected to icecast1.rte.ie (89.207.56.171) port 80 (#0)
> HEAD /junior HTTP/1.1
> Host: icecast1.rte.ie
> User-Agent: curl/7.51.0
> Accept: */*
>
* HTTP 1.0, assume close after body
< HTTP/1.0 400 Bad Request
< Server: Icecast 2.4.2
< Date: Mon, 13 Nov 2017 11:07:42 GMT
< Content-Type: text/html; charset=utf-8
< Cache-Control: no-cache
< Expires: Mon, 26 Jul 1997 05:00:00 GMT
< Pragma: no-cache
<
<html><head><title>Error 400</title></head><body><b>400 - unknown request</b></body></html>
* Curl_http_done: called premature == 0
* Closing connection 0
Even if in this particular case, running HEAD runs into an HTTP Parse Error, while I know the stream is OK (if someone can still explains me why, I would be grateful), I think Opal gave the general solution:
You can run a HEAD request first
I'm getting CORS errors in Firefox and Chrome, but not in cURL. Here's my cURL:
curl -H "Origin: http://mymachine:8080" https://wamoyo.cloudant.com/simpsons -v
That's my command, here's the output:
* Hostname was NOT found in DNS cache
* Trying 184.173.163.133...
* Connected to wamoyo.cloudant.com (184.173.163.133) port 443 (#0)
* successfully set certificate verify locations:
* CAfile: none
CApath: /etc/ssl/certs
* SSLv3, TLS handshake, Client hello (1):
* SSLv3, TLS handshake, Server hello (2):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS handshake, Server finished (14):
* SSLv3, TLS handshake, Client key exchange (16):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSL connection using AES256-SHA
* Server certificate:
* subject: C=US; ST=Massachusetts; L=Boston; O=Cloudant, Inc.; OU=Engineering; CN=*.cloudant.com
* start date: 2013-01-29 00:00:00 GMT
* expire date: 2016-02-19 12:00:00 GMT
* subjectAltName: wamoyo.cloudant.com matched
* issuer: C=US; O=DigiCert Inc; OU=www.digicert.com; CN=DigiCert High Assurance CA-3
* SSL certificate verify ok.
> GET /simpsons HTTP/1.1
> User-Agent: curl/7.35.0
> Host: wamoyo.cloudant.com
> Accept: */*
> Origin: http://mymachine:8080
>
< HTTP/1.1 200 OK
< X-Couch-Request-ID: 1efb92f7dd
* Server CouchDB/1.0.2 (Erlang OTP/R14B) is not blacklisted
< Server: CouchDB/1.0.2 (Erlang OTP/R14B)
< Date: Wed, 09 Jul 2014 18:25:37 GMT
< Content-Type: text/plain;charset=utf-8
< Content-Length: 362
< Cache-Control: must-revalidate
< Access-Control-Expose-Headers: content-type, accept-ranges, etag, server, x-couch-request-id, x-couch-update-newrev
Here's the important bit:
< Access-Control-Allow-Origin: http://mymachine:8080
< Access-Control-Allow-Credentials: true
<
{"update_seq":"34-g1AAAADreJzLYWBgYMlgTmFQTElKzi9KdUhJMtPLzc_PK87IzEvVS87JL01JzCvRy0styQEqZUpkSLL___9_ViI_qiZjfJqSHIBkUj1YH5plRvj05bEASYYGIAXUuj8rkQtVrylhvQcgeoH2smYBAApoT3A","db_name":"simpsons","purge_seq":0,"other":{"data_size":593},"doc_del_count":0,"doc_count":6,"disk_size":750276,"disk_format_version":5,"compact_running":false,"instance_start_time":"0"}
* Connection #0 to host wamoyo.cloudant.com left intact
Okay, now the browser, still returns this error:
Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at https://wamoyo.cloudant.com/simpsons/_changes?timeout=25000&style=all_docs&feed=longpoll&since=35-g1AAAAEjeJzLYWBgYMlgTmFQTElKzi9KdUhJMtPLzc_PK87IzEvVS87JL01JzCvRy0styQEqZUpkSLL___9_VgZTIn8uUIA9zcQ81cgkCdUIY3xGJDkAyaR6qCmsEFOMkxMTUy1RTTHCZ0oeC5BkaABSQIP2g0ziApuUYpJmZmZqjmqSKWGTDkBMQnKTuVFymoFxUhYAK3pbEA&limit=25&_nonce=7YzIfmsBHKTHaGPq. This can be fixed by moving the resource to the same domain or enabling CORS.
When I run PouchDB's sync or replicate functions.
PouchDB.sync('https://wamoyo.cloudant.com/simpsons/', 'simpsons', {live: true})
.on('change', onChange)
.on('complete', onComplete)
.on('error', onError);
function onChange (info) {
alert('onChange running');
}
function onComplete (info) {
alert('onComplete running');
}
function onError (err) {
alert('onError ' + err);
}
CORS is applicable only in a browser context. So, for it to work with PouchDB, you have to set the CouchDB CORS Headers to allow whatever domain you are accessing it from.
CORS is a security feature meant only for browsers. Browsers try to protect the users from the website which otherwise may make AJAX requests to other domains.
For example: when you are on stackoverflow.com and if it tries to make a AJAX request to mail.google.com, then your browser has a reason to believe that this may not be allowable by mail.google.com. Hence, it asks mail.google.com through a OPTIONS request to tell if stackoverflow.com is white-listed to make that particular request. If it does, then browser allows the actual request. Otherwise, it will block it as an error.
Now, as far as curl or any other non-browser request tool is concerned, they work differently. They are a representative of you and hence it is assumed that you would not be doing anything wrong to yourself.
Say I got:
app.get('/json', function(req, res) {
res.set({
'content-type': 'application/json'
}).send('{"status": "0"}');
});
I'm trying to send the response as UTF-8 with the following with no success:
app.get('/json', function(req, res) {
// From Node.js Official Doc
// http://nodejs.org/api/http.html#http_http_request_options_callback
res.setEncoding('utf8');
res.set({
'content-type': 'application/json'
}).send('{"status": "0"}');
});
What is the correct way to set character encoding in Express?
You will probably want to explicitly add a charset to the end of your content-type string if you find it's not being set already by Express:
res.set({ 'content-type': 'application/json; charset=utf-8' });
The charset is not always set automagically and does need to be set to work correctly everywhere (i.e. with all browsers and all ajax libraries) or you can run into encoding bugs.
In Express 4.x specifically I've found that depending on the object you trying to return, it normally automatically returns with content-type: application/json; charset=utf-8 when you call res.json(someObject), however not always.
When calling res.json() on some objects it can return content-type: application/json (i.e. without the charset encoding!). I'm not actually sure what triggers this, other than it's something about the specific object being returned.
I've only noticed it because of automated tests which explicitly checked the headers and found it was missing the charset declaration on some responses (even though the content-type was still application/json).
Use res.charset: http://expressjs.com/api.html#res.charset
res.charset = 'value';
res.send('some html');
// => Content-Type: text/html; charset=value
However, JSON is UTF-8 by default so you don't need to set anything.
This worked for me
res.writeHead(200, {'Content-Type': 'text/html; charset=utf-8'});
Having similar issues I'm collecting Swedish characters from a database and outputting them as JSON object, node doesn't really care if json must be UTF-8 or not when the chars from the database isn't in UTF-8.. So assuming "you don't need to set anything" is false. Depending on what charsets you are working with.
Before you go to the trouble of manually setting header parameters, check what your server is already sending by default. In my case, I'm using a "serverless" cloud provided Node.js instance. Apparently, these are usually front-ended w/ NGINX which I assume is what sets some of this stuff based on default settings. ...I didn't need to res.set anything at all. Granted, I'm serving back HTML, ...just sayin - before you go fixin, make sure it's broke.
accept-ranges: bytes
accept-ranges: bytes
cache-control: private
content-encoding: gzip
content-type: text/html; charset=utf-8
date: Fri, 21 Dec 2018 21:40:37 GMT
etag: W/"83-xwilN/BBLLLAAAHHH/0NBLAH0U"
function-execution-id: 5thvkjd4wwru
server: nginx
status: 200
vary: accept-encoding, cookie, authorization
via: 1.1 varnish
x-cache: MISS
x-cache-hits: 0
x-cloud-trace-context: 18c611BBBBLLLLAAAHHH9594d9;o=1
x-powered-by: Express
x-served-by: cache-dfw18631-DFW
x-timer: S15BBLLLAAHHH.913934,VS0,VE3404
I am trying to create a simple WebSocket connection in JavaScript against my Rails app. I get the following:
WebSocket connection to 'ws://localhost:4000/' failed: Error during WebSocket handshake: 'Sec-WebSocket-Accept' header is missing
What am I doing wrong? Here is my code:
JavaScript:
var socket = new WebSocket('ws://localhost:4000');
socket.onopen = function() {
var handshake =
"GET / HTTP/1.1\n" +
"Host: localhost\n" +
"Upgrade: websocket\n" +
"Connection: Upgrade\n" +
"Sec-WebSocket-Key: x3JJHMbDL1EzLkh9GBhXDw==\n" +
"Sec-WebSocket-Protocol: quote\n" +
"Sec-WebSocket-Version: 13\n" +
"Origin: http://localhost\n";
socket.send(handshake);
};
socket.onmessage = function(data) {
console.log(data);
};
Ruby:
require 'rubygems'
require 'em-websocket-server'
module QuoteService
class WebSocket < EventMachine::WebSocket::Server
def on_connect
handshake_response = "HTTP/1.1 101 Switching Protocols\n"
handshake_response << "Upgrade: websocket\n"
handshake_response << "Connection: Upgrade\n"
handshake_response << "Sec-WebSocket-Accept: HSmrc0sMlYUkAGmm5OPpG2HaGWk=\n"
handshake_response << "Sec-WebSocket-Protocol: quote\n"
send_message(handshake_response)
end
def on_receive(data)
puts 'RECEIVED: ' + data
end
end
end
EventMachine.run do
print 'Starting WebSocket server...'
EventMachine.start_server '0.0.0.0', 4000, QuoteService::WebSocket
puts 'running'
end
The handshake headers are per Wikipedia.
I think that once the connection is open the request and response have already occurred, so sending headers at that point is too late. In addition, headers have to end with a blank line, which you omitted.
According to the demos, you don't even have to set headers in the client or the server--the ruby module automatically takes care of the headers on the server side, and html5 automatically takes care of the headers on the client side. I think this should work:
require "em-websocket-server"
class EchoServer < EM::WebSocket::Server
def on_connect
EM::WebSocket::Log.debug "Connected"
puts "I felt a connection."
end
def on_receive msg
puts "RECEIVED: #{msg}"
send_message msg
end
end
EM.run do
myhost = "0.0.0.0"
myport = 8000
puts "Starting WebSocket server. Listening on port #{myport}..."
EM.start_server myhost, myport, EchoServer
end
html file:
<!DOCTYPE html> <html> <head><title>Test</title>
<script type="text/javascript">
var myWebSocket = new WebSocket("ws://localhost:8000");
myWebSocket.onopen = function(evt) {
console.log("Connection open. Sending message...");
myWebSocket.send("Hello WebSockets!"); };
myWebSocket.onmessage = function(evt) {
console.log(evt.data);
myWebSocket.close(); };
myWebSocket.onclose = function(evt) {
console.log("Connection closed."); };
myWebSocket.onerror = function(err) {
alert(err.name + " => " + err.message); } </script>
</head> <body> <div>Hello</div> </body> </html>
And it does work in Safari 5.1.9 (which is an older browser): I see the expected output on both the server and the client. However, the code does not work in Firefox 21: I get the error message...
Firefox can't establish a connection to the server at ws://localhost:8000/.
var myWebSocket = new WebSocket("ws://localhost:8000");
I notice that in both Firebug and Safari Developer Tools, the server does not send a Sec-WebSocket-Accept header:
Response Headers
Connection Upgrade
Upgrade WebSocket
WebSocket-Location ws://localhost:8000/
WebSocket-Origin null
Request Headers
Accept text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Encoding gzip, deflate
Accept-Language en-US,en;q=0.5
Cache-Control no-cache
Connection keep-alive, Upgrade
DNT 1
Host localhost:8000
Origin null
Pragma no-cache
Sec-WebSocket-Key r9xT+ywe533EHF09wxelkg==
Sec-WebSocket-Version 13
Upgrade websocket
User-Agent Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:21.0) Gecko/20100101 Firefox/21.0
Nothing I tried would make the code work in Firefox 21.0. To check whether Firefox 21.0 even supports websockets, I went to:
http://www.websocket.org/echo.html
and it said my browser does support websockets.
Is there any reason you have to use the em-websocket-server module? The last modification for that module on github was three years ago. And whenever you see require rubygems in ruby code, that should alert you that the code is old. I tried the newer em-websocket module, and I was able to successfully transfer data back and forth using websockets on both Firefox 21.0 and Safari 5.1.9:
require 'em-websocket'
myhost = "0.0.0.0"
myport = 8000
EM.run {
puts "Listening on port #{myport}..."
EM::WebSocket.run(:host => myhost, :port => myport, :debug => false) do |ws|
ws.onopen do |handshake|
path = handshake.path
query_str = handshake.query
origin = handshake.origin
puts "WebSocket opened:"
puts "\t path \t\t -> #{path}"
puts "\t query_str \t -> #{query_str}"
puts "\t origin \t -> #{origin}"
end
ws.onmessage { |msg|
ws.send "Pong: #{msg}"
}
ws.onclose {
puts "WebSocket closed"
}
ws.onerror { |e|
puts "Error: #{e.message}"
}
end
}
Same client side code. Now the response headers include Sec-WebSocket-Accept:
Response Headers
Connection Upgrade
Sec-WebSocket-Accept LyIm6d+kAAqkcTR744tVK9HMepY=
Upgrade websocket
Request Headers
Accept text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Encoding gzip, deflate
Accept-Language en-US,en;q=0.5
Cache-Control no-cache
Connection keep-alive, Upgrade
DNT 1
Host localhost:8000
Origin null
Pragma no-cache
Sec-WebSocket-Key pbK8lFHQAF+arl9tFvHn/Q==
Sec-WebSocket-Version 13
Upgrade websocket
User-Agent Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:21.0) Gecko/20100101 Firefox/21.0
In your code, I don't think you are setting any headers. Instead, you are just sending messages back and forth that happen to contain characters that look like headers. Apparently, your browser requires the Sec-WebSocket-Accept header in the response before it will allow the connection, and when the em-websocket-server module fails to set that header in the response, your browser refuses the connection.
The relevant source code for em-websockets-server looks like this:
module EM
module WebSocket
module Protocol
module Version76
# generate protocol 76 compatible response headers
def response
response = "HTTP/1.1 101 Web Socket Protocol Handshake\r\n"
response << "Upgrade: WebSocket\r\n"
response << "Connection: Upgrade\r\n"
response << "Sec-WebSocket-Origin: #{origin}\r\n"
response << "Sec-WebSocket-Location: #{scheme}://#{host}#{path}\r\n"
if protocol
response << "Sec-WebSocket-Protocol: #{protocol}\r\n"
end
response << "\r\n"
response << Digest::MD5.digest(keyset)
response
end
As you can see, it doesn't set the Sec-WebSocket-Accept header. That code is in a module called Version76, and searching google for websockets version 76 yields an obsolete protocol(which contains an example of a request and response):
https://datatracker.ietf.org/doc/html/draft-hixie-thewebsocketprotocol-76
Here is the current websockets protocol(which also contains an example of a request and response):
https://www.rfc-editor.org/rfc/rfc6455
Conclusion: em-websockets-server is obsolete.
I am parsing in about 4000 URLs with a generic Node.js HTTP request script:
(function (i){
http.get(options, function(res) {
var obj = {};
obj.url = hostNames[i];
obj.statusCode = res.statusCode;
obj.headers = res.headers;
db.scrape.save(obj);
}).on('error',function(e){
console.log("Error: " + hostNames[i] + "\n" + e.stack);
})
})(i);
Around 1300 URLs in, I get this error, which stops the entire script. I don't know what page.ly is, as I do not have that in my list of URLs. I've done a lot of research, but I could not pin-point what's causing this error.
If someone is familiar with HTTP requests on Node.js - could you help me out?
Error: key page.ly must not contain '.'
at Error (unknown source)
at Function.checkKey (/Users/loop/node_modules/mongojs/node_modules/mongodb/node_modules/bson/lib/bson/bson.js:1421:11)
at serializeObject (/Users/loop/node_modules/mongojs/node_modules/mongodb/node_modules/bson/lib/bson/bson.js:355:14)
at packElement (/Users/loop/node_modules/mongojs/node_modules/mongodb/node_modules/bson/lib/bson/bson.js:854:23)
at serializeObject (/Users/loop/node_modules/mongojs/node_modules/mongodb/node_modules/bson/lib/bson/bson.js:359:15)
at Function.serializeWithBufferAndIndex (/Users/loop/node_modules/mongojs/node_modules/mongodb/node_modules/bson/lib/bson/bson.js:332:10)
at BSON.serializeWithBufferAndIndex (/Users/loop/node_modules/mongojs/node_modules/mongodb/node_modules/bson/lib/bson/bson.js:1502:15)
at InsertCommand.toBinary (/Users/loop/node_modules/mongojs/node_modules/mongodb/lib/mongodb/commands/insert_command.js:132:37)
at Connection.write (/Users/loop/node_modules/mongojs/node_modules/mongodb/lib/mongodb/connection/connection.js:198:35)
at __executeInsertCommand (/Users/loop/node_modules/mongojs/node_modules/mongodb/lib/mongodb/db.js:1745:14)
at Db._executeInsertCommand (/Users/loop/node_modules/mongojs/node_modules/mongodb/lib/mongodb/db.js:1801:5)
Loops-MacBook-Air:JS loop$
What could prevent this? It seems my script does not scale very well.
EDIT: From the answers I am getting - there exists a key somewhere that has a ".", which isn't allowed in MongoDB, and I am supposed to escape it. But the question remains - if my keys are only url, statusCode, and headers, what is causing the key with a . in it to show up?
EDIT: Bug is found. Answer below.
This error is caused when you attempt persist an Object in MongoDB and one (or more) of the keys contain the character '.', e.g:
{
"name": "bob",
"url": "http://example.com",
"some.field": "value"
}
would raise the error Error: key some.field must not contain '.'.
Scrub your object keys of '.'s before saving to MongoDB!
The site "divensurf.com" has a header which is called page.ly: v4.0
I have no idea what the is, but that broke my import into MongoDB, since keys cannot symbols. I found the culprit by printing the output onto a .txt file, did a search on the header page.ly, found the site, and deleted it.
I will be sanitizing the headers before importing.
Thanks for the help guys.
HTTP/1.1 304 Not Modified
X-Varnish: 2236761436 2236710300
Vary: Accept-Encoding,Cookie,X-UA-Device
Cache-Control: max-age=7200, must-revalidate
X-Cache: V1HIT 5
Content-Type: text/html; charset=UTF-8
Page.ly: v4.0
Content-Encoding: gzip
X-Pingback: http://divensurf.com/xmlrpc.php
Date: Thu, 21 Mar 2013 19:45:35 GMT
Accept-Ranges: bytes
Via: 1.1 varnish
Connection: keep-alive
Last-Modified: Thu, 21 Mar 2013 19:40:57 GMT
Age: 278