We Keep Coding

Real time audio streaming from ffmpeg to browser (am I missing something?)

I have tried a couple of solutions already, but nothing works for me.
I want to stream audio from my PC to another computer with almost zero latency. Things are working fine so far in a sense of lagging and everything, sound is clear and not choppy at all, but there is something like a delay between the moment when audio starts playing on my PC and remote PC. For example when I click on Youtube 'play' button audio starts playing only after 3-4 seconds on the remote machine. The same when I click 'Pause', the sound on the remote PC stops after a couple of seconds.
I've tried to use websockets\plain audio tag, but no luck so far.
For example this is my solution by using websockets and pipes:
def create_pipe():
return win32pipe.CreateNamedPipe(r'\\.\pipe\__audio_ffmpeg', win32pipe.PIPE_ACCESS_INBOUND,
win32pipe.PIPE_TYPE_MESSAGE |
win32pipe.PIPE_READMODE_MESSAGE |
win32pipe.PIPE_WAIT, 1, 1024 * 8, 1024 * 8, 0, None)
async def echo(websocket):
pipe = create_pipe()
win32pipe.ConnectNamedPipe(pipe, None)
while True:
data = win32file.ReadFile(pipe, 1024 * 2)
await websocket.send(data[1])
async def main():
async with websockets.serve(echo, "0.0.0.0", 7777):
await asyncio.Future() # run forever
if __name__ == '__main__':
asyncio.run(main())
The way I start ffmpeg
.\ffmpeg.exe -f dshow -i audio="Stereo Mix (Realtek High Definition Audio)" -acodec libmp3lame -ab 320k -f mp3 -probesize 32 -muxdelay 0.01 -y \\.\pipe\__audio_ffmpeg
On the JS side the code is a little bit long, but essentially I am just reading a web socket and appending to buffer
this.buffer = this.mediaSource.addSourceBuffer('audio/mpeg')
Also as you see I tried to use -probesize 32 -muxdelay 0.01 flags, but no luck as well
I tried to use plain tag as well, but still - this couple-of-seconds delay exists
What can I do? Am I missing something? Maybe I have to disable buffering somewhere?

I have some code, but all I learned was from https://webrtc.github.io/samples/ website and some from MDN. It's pretty simple.
The idea is to connect 2 peers using a negotiating server just for the initial connection. Afterwards they can share streams (audio, video, data). When I say peers I mean client computers like browsers.
So here's an example for connecting, and broadcasting and of course receiving.
Now for some of my code.
a sketch of the process
note: the same code is used for connecting to and connecting from. this is how my app works bcz it's kind of like a chat. ClientOutgoingMessages and ClientIncomingMessages are just my wrapper around sending messages to server (I use websockets, but it's possible also ajax).
Start: peer initiates RTCPeerConnection and sends an offer via server. also setup events for receiving. The other peer is notified of the offer by the server, then sends answer the same way (should he choose to) and finally the original peer accepts the answer and starts streaming. Among this there is another event about candidate I didn't even bothered to know what it is. It works without knowing it.
function create_pc(peer_id) {
var pc = new RTCPeerConnection(configuration);
var sender
var localStream = MyStreamer.get_dummy_stream();
for (var track of localStream.getTracks()) {
sender = pc.addTrack(track, localStream);
}
// when a remote user adds stream to the peer connection, we display it
pc.ontrack = function (e) {
console.log("got a remote stream")
remoteVideo.style.visibility = 'visible'
remoteVideo.srcObject = e.streams[0]
};
// Setup ice handling
pc.onicecandidate = function (ev) {
if (ev.candidate) {
ClientOutgoingMessages.candidate(peer_id, ev.candidate);
}
};
// status
pc.oniceconnectionstatechange = function (ev) {
var state = pc.iceConnectionState;
console.log("oniceconnectionstatechange: " + state)
};
MyRTC.set_pc(peer_id, {
pc: pc,
sender: sender
});
return pc;
}
function offer_someone(peer_id, peer_name) {
var pc = MyRTC.create_pc(peer_id)
pc.createOffer().then(function (offer) {
ClientOutgoingMessages.offer(peer_id, offer);
pc.setLocalDescription(offer);
});
}
function answer_offer(peer_id) {
var pc = MyRTC.create_pc(peer_id)
var offer = MyOpponents.get_offer(peer_id)
pc.setRemoteDescription(new RTCSessionDescription(offer));
pc.createAnswer().then(function (answer) {
pc.setLocalDescription(answer);
ClientOutgoingMessages.answer(peer_id, answer);
// alert ("rtc established!")
MyStreamer.stream_current();
});
}
handling messages from server
offer: function offer(data) {
if (MyRTC.get_pc(data.connectedUser)) {
// alert("Not accepting offers already have a conn to " + data.connectedUser)
// return;
}
MyOpponents.set_offer(data.connectedUser, data.offer)
},
answer: function answer(data) {
var opc = MyRTC.get_pc(data.connectedUser)
opc && opc.pc.setRemoteDescription(new RTCSessionDescription(data.answer)).catch(function (err) {
console.error(err)
// alert (err)
});
// alert ("rtc established!")
MyStreamer.stream_current();
},
candidate: function candidate(data) {
var opc = MyRTC.get_pc(data.connectedUser)
opc && opc.pc.addIceCandidate(new RTCIceCandidate(data.candidate));
},
leave: function leave(data) {
MyRTC.close_pc(data.connectedUser);
},

Slow performance on Mac OS X

I am currently learning how to use NightmareJS. I found the performance is so slow while I was running the code below. It took up to 30 seconds to get the output. Did I do anything wrong?
Moreover, I have tried to use wait() with a selector but that does not help so much.
I am not sure whether this is related to my Internet connection, however, open the same site using Google Chrome and perform the same task is faster than using Nightmare.
Soruce Code
var Nightmare = require('nightmare');
var after;
var before = Date.now();
new Nightmare({
loadImages: false
}).goto('https://www.wikipedia.org/')
.type('#searchInput', process.argv[2])
.click('input[name="go"]')
.wait()
.url(function(url) {
after = Date.now();
console.log('>>> [' + (after - before) / 1000.0 + 's] ' + url);
})
.run(function(err, nightmare) {
if (err) console.log(err);
});
Output
node n02_extract_wiki_link.js "node.js"
>>> [31.227s] https://en.wikipedia.org/wiki/Node.js
My current environment is listed below.
Mac OS X 10.10.4
node v0.12.5
PhantomJS 2.0.0
nightmare#1.8.2

This worked for me:
https://github.com/segmentio/nightmare/issues/126#issuecomment-75944613
It's the socket connection between the phantomjs module and it's dependency, shoe.
You can manually edit shoe yourself. Go into node_modules/phantom/shoe/index.js and change line 8 to read
var server = sockjs.createServer({
heartbeat_delay : 200
});

Node.js / Server.js socket implementation problems

Having a hard time implementing a node.js/server.js setup
I'm a bit stuck right now, and hoping someone can shed some light. I'm relatively new to sockets in general, but have been programming in javascript on and off for several years, although only about as deep as is necessary to accomplish the task at hand. As a result, my understanding of some of the concepts surrounding the javascript stack heap, and sockets in general are somewhat limited.
Ok Here's the situation:
I've created an application intended to simply increment a counter, on several machines. Several users can click the "next" button and it will update instantly on all machines.
When you first connect, it retrieves the current number, and spits it out locally.
I've created the server here:
var io = require("socket.io");
var sockets = io.listen(8000);
var currentlyServing=0;
sockets.on("connection", function (socket)
{
console.log("client connected");
socket.emit("receive", currentlyServing);
socket.on("update", function(serving)
{
currentlyServing=serving;
if(currentlyServing>100)
currentlyServing=0;
if(currentlyServing<0)
currentlyServing=99;
socket.broadcast.emit("receive", currentlyServing);
console.log("update received: "+currentlyServing);
});
});
console.log("Server Started");
Here is the relevant (I hope) excerpt from the client side:
var socket = io.connect("http://www.sampledomain.com:8000");
//function to update the page when a new update is received
socket.on("receive", function(receivedServing)
{
document.getElementById('msgs').value=""+String("00" + receivedServing).slice(-2);
document.getElementById('nowServing').value=receivedServing;
});
//this is called in an onClick event in the HTML source
//sends the new number to all other stations except this one (handled by server side)
function nextServing()
{
var sendServing = parseInt(document.getElementById('nowServing').value)+1;
socket.emit("update", sendServing);
document.getElementById('nowServing').value=sendServing;
document.getElementById('msgs').value=""+String("00" + sendServing).slice(-2);
}
Ok so here's my problem. This runs absolutely fine in every system I've put it in, smoothly and beautifully - except for IE8. If left alone for more than 2-3 minutes (with no activity at all), I eventually receive a "stack overflow" error. The line number it appears on fluctuates (haven't determined the factors involved yet), but it always happens at that interval. On some workstations it takes longer, which I'm beginning to think has a direct correlation to the amount of phsyical RAM the machine has, or at least how much is being allocated to the web browser.
I found an online function to determine "max stack size", which I realize is not an exact science, however I did consistently get a number in the area of 3000. On my IE11 machine with considerable more resources, I found it to be in the area of 20,000. This may not be relevant, but I figured the more info the better :)
To avoid this problem for now so that the end users don't see this error message, I've take the entire client script, and put it into an iFrame which reloads itself every 60 seconds,essentially resetting the stack, which feels so dirty sitting so close to a web socket, but has bought me the time to post here. I've googled until I can't google any more, but when you search "node.js" or "socket.io" along with "stack overflow" on google, you just get a lot of posts about the two topics that are hosted on the stackoverflow dot com website. ARG lol
Anyone?
EDIT ON NOVEMBER 18TH 2014 AS PER COMMENTS BELOW:
the error message is most often claiming stack overflow at line 1056. IE Developer tools points towards the file socket.io.js. Line 1056 is:
return fn.apply(obj, args.concat(slice.call(arguments)));
which is insdie this section of the file:
var slice = [].slice;
/**
* Bind `obj` to `fn`.
*
* #param {Object} obj
* #param {Function|String} fn or string
* #return {Function}
* #api public
*/
module.exports = function(obj, fn){
if ('string' == typeof fn) fn = obj[fn];
if ('function' != typeof fn) throw new Error('bind() requires a function');
var args = slice.call(arguments, 2);
return function(){
return fn.apply(obj, args.concat(slice.call(arguments)));
}
};

From what I've read it seems that the problem on IE8 might be related to flash. It IE8 uses flashsocket as the default configuration. I suggest to try the following on the client side:
if(navigator.appName.indexOf("Internet Explorer")!=-1 && navigator.appVersion.indexOf("MSIE 8")==-1 ){
socket = io.connect("http://www.sampledomain.com:8000", {
transports: ['xhr-polling']
});
}
else
{
socket = io.connect("http://www.sampledomain.com:8000" );
}
This should make IE8 use long polling while all other machines use the best method they can.
On a side note: You might also want to consider incrementing the "serving" variable on the server.

Find existing issue Causes a "Stack Overflow" in IE8 when using xhr-polling #385.
This was fixed by disabling Flash.
Also find Safari over windows client use xhr-polling instead of websocket - performance are severely harm #1147. While this is Safari it may apply to IE8 because it is using similar mechanism.
I did a small test using your socket.io but in IE 10 and emulated IE8
so that I could debug well. Started capturing Network in the tab and
noticed the requests logging every few seconds.Left alone for few
minutes and I see a lot of requests logged in. You will not see this
in Chrome because it has true WebSockets. While IE8 does not support
WebSockets socket.io emulate that using plain HTTP GET/POST using some
mechanism. So my theory is that even if socket.io works with IE8 it
does not reliably emulate web sockets
My advice is to rule out IE 8 for long running client application. IE8 is no longer supported by Microsoft.

maybe try to replace
""+String("00" + receivedServing).slice(-2)
with
('00' + receivedServing).slice(-2)

Worklight - Paho MQTT Javascript: Always timeout on Android device and works in simulator

I have a Worklight project that uses Paho MQTT Javascript 'mqttws31.js'.
In the simulator everything works fine, but when connecting from a device it always gives me a timeout.
When I connect to the broker through the app called 'MyMQTT' on the device it connects fine. I can't get it to work with Javascript.
I tested if websockets are supported with this code - Link
The webview in the application supports websockets, so that could not be the problem.
I tried several connection options, but it always times out.
Are there some Android permissions I have to add?
I added one of the examples I tried below.
The broker that I use is HiveMQ and I enabled websocket support on port 8000.
I also tried to connect to a public broker - test.mosquitto.org", 80
But that gives me the same results (works in simulator, not on device).
Please help me!
I know that the Cordova plugin would be better than the Javascript version, but I don't find the Cordove MQTT plugin for Cordova 3.1. That could help me out to.
var client = new Messaging.Client("192.168.137.2", 8000, "prototype2");
$(function() {
$("#publishBtn").click(function() {
publish("test", "prototype2/testpublish", 1);
});
var options = {
// connection attempt timeout in seconds
timeout : 5,
// Gets Called if the connection has successfully been established
onSuccess : function() {
$("#console").append(
'<br/>' + "Connected to MQTT Broker over Websockets.");
// client.subscribe("testtopic", {qos: 2});
client.subscribe("prototype3/testpublish");
},
// Gets Called if the connection could not be established
onFailure : function(message) {
$("#console").append(
'<br/>' + "Connection failed: " + message.errorMessage);
}
};
// Gets called whenever you receive a message for your subscriptions
client.onMessageArrived = function(message) {
var topic = message.destinationName;
var message = message.payloadString;
$("#console").append(
'<br/>' + 'Message arrived: Topic: ' + topic + ', Message: '
+ message);
};
// Attempt to connect
client.connect(options);
});

For Android 4.3 and lower, you need a cordova plug-in to provide WebSockets. There are a number of these around, but most don't implement the features required by MQTT (support for sub-protocols and for binary messages)
One that does is: https://github.com/mkuklis/phonegap-websocket. With that plug in, mqttws31.js works fine on Android 4.3.
There is currently an issue with installing standard Cordova 3 plugins in Worklight 6.1, which means that when you add the plug-in to your project, you will ned to have to edit the JavaScript provided on Github. The edit is simple, just change: require() to: cordova.require()

I would suggest that you use the develop branch of Paho JS client.
The code below is what I've developed for the Eclipse IoT website demo ; it works just fine on my Android phone (4.4 though)
var client = new Messaging.Client("ws://iot.eclipse.org/ws", "clientId");
client.onConnectionLost = onConnectionLost;
client.onMessageArrived = onMessageArrived;
client.connect({
onSuccess: onConnect
});
function onConnect() {
// Once a connection has been made, make a subscription and send a message.
console.log("onConnect");
client.subscribe("/fosdem/#");
};
function onConnectionLost(responseObject) {
if (responseObject.errorCode !== 0)
console.log("onConnectionLost:" + responseObject.errorMessage);
};
function onMessageArrived(message) {
console.log("onMessageArrived: "+message.destinationName +": " +message.payloadString);
// my stuff ...
};

Poor network performance with Websockets running on apple device

I am working on an HTML/Javascript running on mobile devices that is communicating with a Qt/C++ application running on a PC. Both the mobile device and the PC are on a local network. The communication between the HTML page (client) and the C++ app (server) is done using Websockets.
The HTML page is a remote control for the C++ application, so it is needed to have a low latency connection between the mobile device and the PC.
When using any non-Apple device as a client, data is sent to a rate between 60 to 120 frames/sec, which is totally acceptable. When using an Apple device, this rate falls to 3-4 frames/sec.
I also checked ping times (Websocket implementation, not a ping command from command line). They are acceptable (1-5 ms) for Apple devices as long as the device is not transmitting data. Whenever it transmits data, this ping time raises to 200ms.
Looking from the Javascript side, the Apple devices always send data at a consistent rate of 60 frames/sec, as any other devices do. However, on the server side, only 3 to 4 of these 60 frames are received when the client is an Apple device.
Does anyone have any idea on what can be happening?
Here is my Javascript code :
<script language="javascript" type="text/javascript">
var wsUri = document.URL.replace("http", "ws");
var output;
var websocket;
function init()
{
output = document.getElementById("output");
wsConnect();
}
function wsConnect()
{
console.log("Trying connection to " + wsUri);
try
{
output = document.getElementById("output");
websocket = new WebSocket(wsUri);
websocket.onopen = function(evt)
{
onOpen(evt)
};
websocket.onclose = function(evt)
{
onClose(evt)
};
websocket.onmessage = function(evt)
{
onMessage(evt)
};
websocket.onerror = function(evt)
{
onError(evt)
};
}
catch (e)
{
console.log("Exception " + e.toString());
}
}
function onOpen(evt)
{
alert("Connected to " + wsUri);
}
function onClose(evt)
{
alert("Disconnected");
}
function onMessage(evt)
{
alert('Received message : ' + evt.data);
}
function onError(evt)
{
alert("Error : " + evt.toString());
}
function doSend(message)
{
websocket.send(message);
}
window.addEventListener("load", init, false);
</script>
Data is sent from Javascript side using dosend() function.

Few ideas and suggestions.
Check if the client's WebSocket protocol is supported by the server. This question and answer discuss a case where different protocol versions were an issue.
The WebSocket standard permits implementations to arbitrarily delay transmissions and perform fragmentation. Additionally, control frames, such as Ping, do not support fragmentation, but are permitted to be interjected. These permitted behavioral difference may be contributing to the difference in times.
Check if the bufferedAmount attribute on the WebSocket to determine if the WebSocket is buffering the data. If the bufferedAmount attribute is often zero, then data has been passed to the OS, which may be buffering it based on OS or socket configurations, such as Nagle.
This question and answer mentions resolving delays by having the server send acknowledgements for each message.
To get a deeper view into the interactions, it may be useful to perform a packet trace. This technical Q&A in the Mac Developer Library may provide some resources as to how to accomplish this.

The best way to get some more insight is to use the AutobahnTestsuite. You can test both clients and servers with that suite and find out where problems are situated.
I have created QWebSockets, a Qt based websockets implementation, and used that on several occasions to create servers. Performance from Apple devices is excellent.
However, there seems to be a severe problem with Safari when it comes to large messages (see https://github.com/KurtPattyn/QWebSockets/wiki/Performance-Tests). Maybe that is the problem.