How to create a live media stream with Javascript - javascript

I am wanting to create a live audio stream from one device to a node server which can then broadcast that live feed to several front ends.
I have searched extensively for this and have really hit a wall so hoping somebody out there can help.
I am able to get my audio input from the window.navigator.getUserMedia API.
getAudioInput(){
const constraints = {
video: false,
audio: {deviceId: this.state.deviceId ? {exact: this.state.deviceId} : undefined},
};
window.navigator.getUserMedia(
constraints,
this.initializeRecorder,
this.handleError
);
}
This then passes the stream to the initializeRecorder function which utilises the AudioContext API to create a createMediaStreamSource`
initializeRecorder = (stream) => {
const audioContext = window.AudioContext;
const context = new audioContext();
const audioInput = context.createMediaStreamSource(stream);
const bufferSize = 2048;
// create a javascript node
const recorder = context.createScriptProcessor(bufferSize, 1, 1);
// specify the processing function
recorder.onaudioprocess = this.recorderProcess;
// connect stream to our recorder
audioInput.connect(recorder);
// connect our recorder to the previous destination
recorder.connect(context.destination);
}
In my recorderProcess function, I now have an AudioProcessingEvent object which I can stream.
Currently I am emitting the audio event as as a stream via a socket connection like so:
recorderProcess = (e) => {
const left = e.inputBuffer.getChannelData(0);
this.socket.emit('stream', this.convertFloat32ToInt16(left))
}
Is this the best or only way to do this? Is there a better way by using fs.createReadStream and then posting the an endpoint via Axios? As far as I can tell this will only work with a file as opposed to a continuous live stream?
Server
I have a very simple socket server running ontop of express. Currently I listen for the stream event and then emit that same input back out:
io.on('connection', (client) => {
client.on('stream', (stream) => {
client.emit('stream', stream)
});
});
Not sure how scalable this is but if you have a better suggestion, I'm very open to it.
Client
Now this is where I am really stuck:
On my client I am listening for the stream event and want to listen to the stream as audio output in my browser. I have a function that receives the event but am stuck as to how I can use the arrayBuffer object that is being returned.
retrieveAudioStream = () => {
this.socket.on('stream', (buffer) => {
// ... how can I listen to the buffer as audio
})
}
Is the way I am streaming audio the best / only way I can upload to the node server?
How can I listen to the arrayBuffer object that is being returned on my client side?

Is the way I am streaming audio the best / only way I can upload to the node server?
Not really the best but i have seen worse, its not the only way either using websockets its considered ok from point of view since you want things to be "live" and not keep sending http post request every 5sec.
How can I listen to the arrayBuffer object that is being returned on my client side?
You can try this BaseAudioContext.decodeAudioData to listen to data streamed, the example is pretty simple.
From the code snippets you provide i assume you want to build something from scratch to learn how things work.
In that case, you can try MediaStream Recording API along with an websocket server that sends the chunks to X clients so they can reproduce the audio, etc.
It would make sense to invest time into WebRTC API, to learn how to stream from client to another client.
Also take a look at the links below for some useful information.
(stackoverflow) Get live streaming audio from NodeJS server to clients
(github) video-conference-webrtc
twitch.tv tech stack article
rtc.io

Related

Web Audio API -- Capture Stream from Analyser

The Web Audio API examples show a source stream going directly to an audio context destination node, like the speakers or canvas. I want to collect data from the analyser, and then render a React component with it. The best I came up with is to interval poll:
const stream = await navigator.mediaDevices.getUserMedia({ audio: true });
mediaRecorder = new MediaRecorder(stream);
// start button click
mediaRecorder.start();
analyserPoll = setInterval(() => {
analyser.getFloatTimeDomainData(buffer);
collect = [...collect, buffer];
}, 500);
// stop button click
mediaRecorder.stop();
clearInterval(analyserPoll);
Is there a more official way of doing this inside the API without the setTimeout? For instance, saving to a Blob or file, and then running my analyser code on that? It's a McLeod pitch detector.
AudioWorklets are not required. What's required is proper design of state and effects. Create an effectful action that set states for what you don't want to render, like a buffer:
useEffect(() => {
// getFloatTimeDomainData
// setBufferState
}, //run effect when these deps change, buffer);
Then, after the audio api collection is over, like when the user hits a "Stop button" set the buffer to the data you want to render.
setData(buffer)
The effect won't render as you leave buffer alone. It's helpful when for expensive components and collecting data.
There are a ton of edge cases. User needs to gesture before audio api can be used. Audio Worklets are streams, so there's no real way to persist data. Sending messages on a port result in the same thing but more complicated.

Recording and uploading audio from javascript

I am trying to record and upload audio from javascript. I can successfullly record audio Blobs from a MediaRecorder. My understanding is that after recording several chunks into blobs, I would concatenate them as a new Blob(audioBlobs) and upload that. Unfortunately, the result on the server-side keeps being more or less gibberish. I'm currently running a localhost connection, so converting to uncompressed WAV isn't a problem (might be come one later, but that's a separate issue). Here is what I have so far
navigator.mediaDevices.getUserMedia({audio: true, video: false})
.then(stream => {
const mediaRecorder = new MediaRecorder(stream);
mediaRecorder.start(1000);
const audioChunks = [];
mediaRecorder.addEventListener("dataavailable", event => {
audioChunks.push(event.data);
});
function sendData () {
const audioBlob = new Blob(audioChunks);
session.call('my.app.method', [XXXXXX see below XXXXXX])
}
})
The session object here is an autobahn.js websockets connection to a python server (using soundfile. I tried a number of arguments in the place that was labelled by XXXXX in the code.
Just pass the audioBlob. In that case, the python side just receives an empty dictionary.
Pass audioBlob.text() in that case, I get something that looks somewhat binary (starts with OggS), but it can't be decoded.
Pass audioBlob.arrayBuffer(). In that case the python side receives an empty dictionary.
A possible solution could be to convert the data to WAV on the serverside (just changing the mime-type on the blob doesn't work) or to find a way to interpret the .text() output on the server side.
The solution was to use recorder.js and then use the getBuffer method in there to get the wave data as a Float32Array.

JavaScript MediaSource and MediaRecorder lag in playing live-stream video

I have working on streaming live video using WebRTC based on RTCConnection with library called simple-peer, but I have faced with some lag between live stream video (with MediaRecorder) and that was played on using MediaSource
Here is recorder:
var mediaRecorder = new MediaRecorder(stream, options);
mediaRecorder.ondataavailable = handleDataAvailable;
function handleDataAvailable(event) {
if (connected && event.data.size > 0) {
peer.send(event.data);
}
}
...
peer.on('connect', () => {
// wait for 'connect' event before using the data channel
mediaRecorder.start(1);
});
Here is source that is played:
var mediaSource = new MediaSource();
var sourceBuffer;
mediaSource.addEventListener('sourceopen', args => {
sourceBuffer = mediaSource.addSourceBuffer(mimeCodec);
});
...
peer.on('data', data => {
// got a data channel message
sourceBuffer.appendBuffer(data);
});
I open two tabs and connect to myself and I see delay in playing video ...
Seems like I configured badly MediaRecorder or MediaSource
Any help will be appreciated ;)
You've combined two completely unrelated techniques for streaming the video, and are getting the worst tradeoffs of both. :-)
WebRTC has media stream handling built into it. If you expect realtime video, the WebRTC stack is what you want to use. It handles codec negotiation, auto-scales bandwidth, frame size, frame rate, and encoding parameters to match network conditions, and will outright drop chunks of time to keep playback as realtime as possible.
On the other hand, if retaining quality is more desirable than being realtime, MediaRecorder is what you would use. It makes no adjustments based on network conditions because it is unaware of those conditions. MediaRecorder doesn't know or care where you put the data after it gives you the buffers.
If you try to play back video as it's being recorded, will inevitably lag further and further behind because there is no built-in catch-up method. The only thing that can happen is a buffer underrun, where the playback side waits until there is enough data to begin playback again. Even if it becomes minutes behind, it isn't going to automatically skip ahead.
The solution is to use the right tool. It sounds like from your question that you want realtime video. Therefore, you need to use WebRTC. Fortunately simple-peer makes this... simple.
On the recording side:
const peer = new Peer({
initiator: true,
stream
});
Then on the playback side:
peer.on('stream', (stream) => {
videoEl.srcObject = stream;
});
Much simpler. The WebRTC stack handles everything for you.

write to a wav file while streaming node js

Is there a way to force .pipe on a stream to write to a file every certain time/size?
Basically, I am using socket io stream, from a browser I am sending a buffer with audio and I send with emit:
Browser
c.onaudioprocess = function(o)
{
var input = o.inputBuffer.getChannelData(0);
stream1.write( new ss.Buffer( convertFloat32ToInt16( input ) ));
}
Server (nodejs)
var fileWriter = new wav.FileWriter('/tmp/demo.wav', {
channels: 1,
sampleRate: 44100,
bitDepth: 16
});
ss(socket).on('client-stream-request', function(stream)
{
stream.pipe(fileWriter);
}
The problem I have is that the file demo.wav will be only wrote when I finish the stream, so when I stop the microphone. But I would like it to write always, as I will be doing speech recognition using google, any ideas? If I call the speech recognition from google using pipe, the chuncks are too small and google is not able to recognize it.
Looking over the Node stream API it looks like you should be able to add an options parameter to the pipe function.
Try
stream.pipe(fileWriter, { end: false });

Live streaming audio with WebRTC and WebAudio (WebSockets)

I'm trying to set up a live audio streaming system where a client will broadcast the audio from his microphone (accessed with getUserMedia) to one or more peers.
To do so, chunks of the audio stream are sent through a WebSocket to a server, which will then relay this information to all the peers connected to the WebSocket.
My main problem comes from how to play chunks of data recieved by the peers on a website.
First, that's how I send the chunks of audio data on my client broadcasting JS script :
var context = new AudioContext();
var audioStream = context.createMediaStreamSource(stream);
// Create a processor node of buffer size, with one input channel, and one output channel
var node = context.createScriptProcessor(2048, 1, 1);
// listen to the audio data, and record into the buffer
node.onaudioprocess = function(e){
var inputData = e.inputBuffer.getChannelData(0);
ws.send(JSON.stringify({sound: _arrayBufferToBase64(convertoFloat32ToInt16(inputData))}));
}
audioStream.connect(node);
node.connect(context.destination);
arrayBufferToBase64 and convertoFloat32ToInt16 are methods that I use to send respectively the stream in base64 format, and to convert the inputData to Int16, instead of that fancy Float32 representation (I used methods found on SO, supposed to work).
Then, after the data has gone through the WebSocket, I collect the data in another script, which will be executed on the website of each peer :
var audioCtx = new AudioContext();
var arrayBuffer = _base64ToArrayBuffer(mediaJSON.sound);
audioCtx.decodeAudioData(arrayBuffer, function(buffer) {
playSound(buffer);
});
I also need to convert the base64 data recieved to an ArrayBuffer, which will then be decoded by decodedAudioData to produce an audioBuffer of type AudioBuffer. The playSound function is as simple as this :
function playSound(arrBuff) {
var src = audioCtx.createBufferSource();
src.buffer = arrBuff;
src.looping = false;
src.connect(audioCtx.destination);
src.start();
}
But for some reasons, I can't get any sound to play on this script. I'm pretty sure the broadcasting script is correct, but not the "listener" script. Can anyone help me on this ?
Thanks !

Categories