We Keep Coding

Server-Sent Events with Ruby Grape

I am trying to create Server-Sent events on my Ruby Grape API.
The problem is that the connection seems to be closed really fast all the time, as I get Connection closed event all the time on test webpage.
The client connects to the server as I can see the method being called, but I would like to know why is the connection not constant and why I don't receive the data I send using the Thread.
Here is my Ruby code:
$connections = []
class EventsAPI < Sinantra::Base
def connections
$connections
end
get "/" do
content_type "text/event-stream"
stream(:keep_open) { |out|
puts "New connection"
out << "data: {}\n\n"
connections << out
}
end
post "/" do
data = "data\n\n"
connections.each { |out| out << data }
puts "sent\n"
end
end
Here is my Javascript:
var source = new EventSource('http://localhost:9292/events');
source.onmessage = function(e) {
console.log("New message: ", e.data);
showMessage(e.data);
};
source.onopen = function(e) {
// Connection was opened.
};
source.onerror = function(e) {
console.log("Source Error", e)
if (e.eventPhase == EventSource.CLOSED) {
console.log("Connection was closed");
// Connection was closed.
}
};
var showMessage = function(msg) {
var out = document.getElementById('stream');
var d = document.createElement('div')
var b = document.createElement('strong')
var now = new Date;
b.innerHTML = msg;
d.innerHTML = now.getHours() + ":" + now.getMinutes() + ":" +now.getSeconds() + " ";
d.appendChild(b);
out.appendChild(d);
};
EDIT: I got it working with the GET method (I changed the Grape::API to Sinatra::Base as Grape does not implement stream). I now receive data, but the connection is not kept alive and when I use the post method the data never reaches the browser.
Thank you in advance for your answers.

The JS code looks correct. My guess is that you should not start a new thread for your infinite loop. What will be happening is that the main thread will carry on executing, reach the end of its block, and close the http request. Your detached thread is then left writing to a non-existent out stream.
UPDATE in response to your EDIT: POST is not supported in SSE. Data can only be passed to an SSE process by using GET data or cookies.

If Websocket connection can't be established do something else

I'm currently running into a problem you guys might be able to help me with..
I'm using websockets to connect to a custom server. Now i want to integrate a second Server IP if the first one isn't available.
How is it possible to detect, that the connection couldn't be made because the server isn't reachable? When I enter a wrong ws://url in my script, Chrome for example gives me the following error:
WebSocket connection to 'wss://1234/' failed: Error in connection establishment: net::ERR_NAME_NOT_RESOLVED
in Firefox it's something complete different. Do you guys of any method to catch this error with Javascript?
Basically when the ws:// url can't be reached, i want to change a variable with a different Server-IP and try it with this one again...
Thanks for your help!

It seems there's no way to catch the problem on instantiation, even though the magical JavaScript black box somehow seems to know the problem occurs on the new WebSocket.
To detect this error, use the following:
ws = new WebSocket(server);
ws.onerror = function (evt) {
if (ws.readyState == 3) {
//Connection closed.
}
}

thanks #orbitbot,
I'm using a framework called jwebsocket (jwebsocket.org). My Code is basically this:
serverstate = "0";
console.log(serverstate);
function logon() {
if(serverstate == "0") {
lURL = "wss://testurl-one:9797";
} else if (serverstate == "1") {
lURL = "wss://testurl-two:9797";
}
var gUsername = "user";
var lPassword = "pass";
console.log( "Connecting to " + lURL + " and console.logging in as '" + gUsername + "'..." );
var lRes = lWSC.logon( lURL, gUsername, lPassword, {
// OnOpen callback
OnOpen: function( aEvent ) {
console.log( "jWebSocket connection established." );
},
// OnMessage callback
OnMessage: function( aEvent, aToken ) {
var lDate = "";
if( aToken.date_val ) {
lDate = jws.tools.ISO2Date( aToken.date_val );
}
console.log( "jWebSocket '" + aToken.type + "' token received, full message: '" + aEvent.data + "' " + lDate + "" );
console.log(aToken);
}
},
// OnClose callback
OnClose: function( aEvent ) {
console.log( "Disconnected from Server" );
console.log("Using next server..");
serverstate = "1";
console.log(serverstate);
console.log("Trying to connect to next server");
logon();
},
// OnClose callback
OnError: function( aEvent ) {
console.log ("Some error appeared");
}
});
console.log( lWSC.resultToString( lRes ) );
}
Of course this would work so far. My Problem is that im using websockets to open a connection, get some information, and after that close the connection again.
since this code will always be fired if the server connection is closed (which in many cases i want to..) i can't use it like that...
any other ideas on this problem ?

I got it.. for everyone else who's interested:
When connection is made you receive a message from there server. So if the server is not available.. there'll be no message. So i just added a variable to the "OnMessage" Part.. and in the disconnect i check if a message was received. If not the server isn't there.. if yes, everything works fine..

Assuming your code is something like this,
var host = 'ws://a.real.websocket.url';
var socket = new WebSocket(host);
... you just need to surround the call to new WebSocket with a try/catch block, something like this
try {
var socket = new WebSocket(host);
} catch (e) {
// inspect e to understand why there was an error
// and connect to another url if necessary
}
That being said, it might be easier to work with a websockets library such as Socket.IO or SockJS (https://github.com/sockjs/sockjs-client), which then would change your reconnection code logic to whatever the libraries provide.

Optimize crawler script on cronjob

i have about 66Million domains in a MySQL table, i need to run crawler on all the domains and update the row count = 1 when the crawler completed.
the crawler script is in php using php crawler library
here is the script.
set_time_limit(10000);
try{
$strWebURL = $_POST['url'];
$crawler = new MyCrawler();
$crawler->setURL($strWebURL);
$crawler->addContentTypeReceiveRule("#text/html#");
$crawler->addURLFilterRule("#\.(jpg|jpeg|gif|png)$# i");
$crawler->enableCookieHandling(true);
$crawler->setTrafficLimit(1000 * 1024);
$crawler->setConnectionTimeout(10);
//start of the table
echo '<table border="1" style="margin-bottom:10px;width:100% !important;">';
echo '<tr>';
echo '<th>URL</th>';
echo '<th>Status</th>';
echo '<th>Size (bytes)</th>';
echo '<th>Page</th>';
echo '</tr>';
$crawler->go();
echo '</table>';
$this->load->model('urls');
$this->urls->incrementCount($_POST['id'],'urls');
}catch(Exception $e){
}
$this->urls->incrementCount(); only update the row and to mark the count column = 1
and because i have 66M domains i needed to run a cronjob on my server
and as cronjob runs on command line i needed a headless browser so i choose phanjomjs
because the crawler doesnt work the way i wanted it to work without the headless browser (phantomjs)
first problem i faced was to load domains from mysql db and run crawler script from a js script
i tried this:
create a php script that returns domains in json form and load it from js file and foreach the domains and run the crawler, but it didnt work very well and get stuck after sometime
next thing i tried, which im still using is create a python script to load the domains directly from mysql db and run the phantom js script on each domains from python script.
here is the code
import MySQLdb
import httplib
import sys
import subprocess
import json
args = sys.argv;
db = MySQLdb.connect("HOST","USER","PW","DB")
cursor = db.cursor()
#tablecount = args[1]
frm = args[1]
limit = args[2]
try:
sql = "SELECT * FROM urls WHERE count = 0 LIMIT %s,%s" % (frm,limit)
cursor.execute(sql)
print "TOTAL RECORDS: "+str(cursor.rowcount)
results = cursor.fetchall()
count = 0;
for row in results:
try:
domain = row[1].lower()
idd = row[0]
command = "/home/wasif/public_html/phantomjs /home/wasif/public_html/crawler2.js %s %s" % (domain,idd)
print command
proc = subprocess.Popen(command, shell=True, stdout=subprocess.PIPE)
script_response = proc.stdout.read()
print script_response
except:
print "error running crawler: "+domain
except:
print "Error: unable to fetch data"
db.close()
it takes 2 arguments to set the limit to select domain from database.
foreach domains and run this command using subproces
command = "/home/wasif/public_html/phantomjs /home/wasif/public_html/crawler2.js %s %s" % (domain,idd)
command
proc = subprocess.Popen(command, shell=True, stdout=subprocess.PIPE)
script_response = proc.stdout.read()
print script_response
crawler2.js file also takes 2 args 1 is domain and 2nd is the id to update the count = 1 when crawler completed
this is the crawler2.js
var args = require('system').args;
var address = '';
var id = '';
args.forEach(function(arg, i) {
if(i == 1){
address = arg;
}
if(i == 2){
id = arg;
}
});
address = "http://www."+address;
var page = require('webpage').create(),
server = 'http://www.EXAMPLE.net/main/crawler',
data = 'url='+address+'&id='+id;
console.log(data);
page.open(server, 'post', data, function (status) {
if (status !== 'success') {
console.log(address+' Unable to post!');
} else {
console.log(address+' : done');
}
phantom.exit();
});
it works well but my script get stuck after sometime n need to restart after sometime and log shows nothing wrong
i need to optimize this process and run crawler as fast as i can, any help would be appreciated

Web crawler programmer is in here. :)
Your python execute the phantom serially. You should do it in parallel. To do it, execute the phantom then leave it, don't wait it.
In PHP, would be like this:
exec("/your_executable_path > /dev/null &");
Don't use phantom if you don't need to. It render everything. > 50MB memory will be needed.

Websocket and Perl CGI

I am bit new in CGI programming, and I trying to make an online chat API but face not few troubles:
I was looking online for solution and found Websocket for client (js) and HTTP::Daemon for perl, but I have no idea where to start to make the server listen for the connections from the browser.
Here is my JavaScript code:
ws = new WebSocket('ws://www.crazygao.com:3000'); // test
ws.onopen = function() {
alert('Connection is established!'); // test
};
ws.onclose = function() {
alert('Connection is closed');
};
ws.onmessage = function(e) {
var message = e.data;
//alert('Got new message: ' + message);
};
ws.onerror = function(e) {
//var message = e.data;
alert('Error: ' + e);
};
Here is my Perl script test code:
use HTTP::Daemon;
use HTTP::Status;
my $d = HTTP::Daemon->new(
LocalAddr => 'www.crazygao.com',
LocalPort => 3000
) || die; print "Please contact me at: <URL:", $d->url, ">\n";
while(my $c = $d->accept) {
$c->send_response("1"); # test
while (my $r = $c->get_request) {
if ($r->method eq 'GET') {
$c->send_response("...");
}
}
$c->close;
undef($c);
}
When the page loads, the connection closing immediately, and in Chrome console window I see the following error:
WebSocket connection to 'ws://198.38.89.14:3000/' failed: Error in connection establishment: net::ERR_CONNECTION_REFUSED
I run the perl script manually (using simple call to http://example.com/cgi-bin/xxx.cgi) and then when I refresh the page I get:
WebSocket connection to 'ws://198.38.89.14:3000/' failed: Error during WebSocket handshake: Unexpected response code: 200
I understand that the server normally returns 200 when OK, but Websocket is waiting for 101 code as "OK".
My question is, if so, how can I achieve this?

I know this is old and I got here because I was looking for an answer myself. I ended up finding the answer myself by using Net::WebSocket::Server.
http://search.cpan.org/~topaz/Net-WebSocket-Server-0.003004/lib/Net/WebSocket/Server.pm for more details on how to use the module and example.
Basically, you'll have this perl code to match your javascript (copied and modified from the CPAN page of Net::WebSocket::Server):
use Net::WebSocket::Server;
my $origin = 'http://www.crazygao.com';
Net::WebSocket::Server->new(
listen => 3000,
on_connect => sub {
my ($serv, $conn) = #_;
$conn->on(
handshake => sub {
my ($conn, $handshake) = #_;
$conn->disconnect() unless $handshake->req->origin eq $origin;
},
utf8 => sub {
my ($conn, $msg) = #_;
$_->send_utf8($msg) for $conn->server->connections;
},
binary => sub {
my ($conn, $msg) = #_;
$_->send_binary($msg) for $conn->server->connections;
},
);
},
)->start;

Winsock.SendData equivalent with Javascript?

Can the following VB Script to open an IP cash drawer be done in Javascript instead?
Private Sub CashDrawerConnect_Click()
Winsock1.Close
ipaddr = "192.168.2.5"
Winsock1.RemoteHost = ipaddr
Winsock1.RemotePort = 30998
Winsock1.Connect
Sleep 250
TxtOpStatus = "Connection to the cash drawer at " & ipaddr & " is established..."
TxtOpStatus.Refresh
End Sub
Private Sub CashDrawerOpen_Click()
If Winsock1.State = sckConnected Then
Winsock1.SendData "opendrawer\0a"
Else
TxtOpStatus = "Not connected to the device"
TxtOpStatus.Refresh
End If
End Sub

You could do it on javascript, but not while running on a browser.
You would need to install nodejs and run your js file directly from the console.
This is a small example that would connect you the the drawer and send the "opendrawer" command on your example:
var net = require('net');
var client = net.connect({port: 30998, host: "yourip"}, function() {
client.write("opendrawer\0a");
});
If however the server has access to the drawer the javascript code could just make a request to the server which would be on charge of opening the connection to the drawer and sending the payload (opendrawer).
If you use php you can take a look at the sockets documentation.

Using VB and JavaScript the calls are mostly the same, you just jhave to adapt it to the language. http://www.ostrosoft.com/oswinsck/oswinsck_javascript.asp
The following is a snippet that uses WinSock from JScript
var oWinsock;
var sServer = "192.168.2.5";
var nPort = 3098;
var bClose = false;
oWinsock = new ActiveXObject("OSWINSCK.Winsock");
// Hooking up handlers
WScript.ConnectObject(oWinsock, "oWinsock_");
oWinsock.Connect(sServer, nPort);
WScript.Echo("Invalid URL");
bClose = true;
function oWinsock_OnConnect() {
oWinsock.SendData('Your data');
}
function oWinsock_OnDataArrival(bytesTotal) {
var sBuffer = oWinsock.GetDataBuffer();
sSource = sSource + sBuffer;
}
function oWinsock_OnError(Number, Description, Scode, Source,
HelpFile, HelpContext, CancelDisplay) {
WScript.Echo(Number + ': ' + Description);
}
function oWinsock_OnClose() {
oWinsock.CloseWinsock();
WScript.Echo(sSource);
oWinsock = null;
bClose = true;
}
while (!bClose) {
WScript.Sleep(1);
}
In the browser? Not really, but you can use WebSockets http://en.wikipedia.org/wiki/WebSocket
You'll need to implement a WebSocket server, so if you need to talk directly to a socket, you can't do it from a browser. But you could implement a proxy server that relays information between the socket server and the WebSocket server.
If you don't need two way communication, the best thing would be for your server to provide a webservice that wraps that socket request. Then your client can just make an AJAX call.