We Keep Coding

Stream audio over websocket with low latency and no interruption

I'm working on a project which requires the ability to stream audio from a webpage to other clients. I'm already using websocket and would like to channel the data there.
My current approach uses Media Recorder, but there is a problem with sampling which causes interrupts. It registers 1s audio and then send's it to the server which relays it to other clients. Is there a way to capture a continuous audio stream and transform it to base64?
Maybe if there is a way to create a base64 audio from MediaStream without delay it would solve the problem. What do you think?
I would like to keep using websockets, I know there is webrtc.
Have you ever done something like this, is this doable?
--> Device 1
MediaStream -> MediaRecorder -> base64 -> WebSocket -> Server --> Device ..
--> Device 18
Here a demo of the current approach... you can try it here: https://jsfiddle.net/8qhvrcbz/
var sendAudio = function(b64) {
var message = 'var audio = document.createElement(\'audio\');';
message += 'audio.src = "' + b64 + '";';
message += 'audio.play().catch(console.error);';
eval(message);
console.log(b64);
}
navigator.mediaDevices.getUserMedia({
audio: true
}).then(function(stream) {
setInterval(function() {
var chunks = [];
var recorder = new MediaRecorder(stream);
recorder.ondataavailable = function(e) {
chunks.push(e.data);
};
recorder.onstop = function(e) {
var audioBlob = new Blob(chunks);
var reader = new FileReader();
reader.readAsDataURL(audioBlob);
reader.onloadend = function() {
var b64 = reader.result
b64 = b64.replace('application/octet-stream', 'audio/mpeg');
sendAudio(b64);
}
}
recorder.start();
setTimeout(function() {
recorder.stop();
}, 1050);
}, 1000);
});

Websocket is not the best. I solved by using WebRTC instead of websocket.
The solution with websocket was obtained while recording 1050ms instead of 1000, it causes a bit of overlay but still better than hearing blanks.

Although you have solved this through WebRTC, which is the industry recommended approach, I'd like to share my answer on this.
The problem here is not websockets in general but rather the MediaRecorder API. Instead of using it one can use PCM audio capture and then submit the captured array buffers into a web worker or WASM for encoding to MP3 chunks or similar.
const context = new AudioContext();
let leftChannel = [];
let rightChannel = [];
let recordingLength = null;
let bufferSize = 512;
let sampleRate = context.sampleRate;
const audioSource = context.createMediaStreamSource(audioStream);
const scriptNode = context.createScriptProcessor(bufferSize, 1, 1);
audioSource.connect(scriptNode);
scriptNode.connect(context.destination);
scriptNode.onaudioprocess = function(e) {
// Do something with the data, e.g. convert it to WAV or MP3
};
Based on my experiments this would give you "real-time" audio. My theory with the MediaRecorder API is that it does some buffering first before emitting out anything that causes the observable delay.

AnalyserNode.getFloatFrequencyData always returns -Infinity

Alright, so I'm trying to determine the intensity (in dB) on samples of an audio file which is recorded by the user's browser.
I have been able to record it and play it through an HTML element.
But when I try to use this element as a source and connect it to an AnalyserNode, AnalyserNode.getFloatFrequencyData always returns an array full of -Infinity, getByteFrequencyData always returns zeroes, getByteTimeDomainData is full of 128.
Here's my code:
var audioCtx = new (window.AudioContext || window.webkitAudioContext)();
var source;
var analyser = audioCtx.createAnalyser();
var bufferLength = analyser.frequencyBinCount;
var data = new Float32Array(bufferLength);
mediaRecorder.onstop = function(e) {
var blob = new Blob(chunks, { 'type' : 'audio/ogg; codecs=opus' });
chunks = [];
var audioURL = window.URL.createObjectURL(blob);
// audio is an HTML audio element
audio.src = audioURL;
audio.addEventListener("canplaythrough", function() {
source = audioCtx.createMediaElementSource(audio);
source.connect(analyser);
analyser.connect(audioCtx.destination);
analyser.getFloatFrequencyData(data);
console.log(data);
});
}
Any idea why the AnalyserNode behaves like the source is empty/mute? I also tried to put the stream as source while recording, with the same result.

I ran into the same issue, thanks to some of your code snippets, I made it work on my end (the code bellow is typescript and will not work in the browser at the moment of writing):
audioCtx.decodeAudioData(this.result as ArrayBuffer).then(function (buffer: AudioBuffer) {
soundSource = audioCtx.createBufferSource();
soundSource.buffer = buffer;
//soundSource.connect(audioCtx.destination); //I do not need to play the sound
soundSource.connect(analyser);
soundSource.start(0);
setInterval(() => {
calc(); //In here, I will get the analyzed data with analyser.getFloatFrequencyData
}, 300); //This can be changed to 0.
// The interval helps with making sure the buffer has the data
Some explanation (I'm still a beginner when it comes to the Web Audio API, so my explanation might be wrong or incomplete):
An analyzer needs to be able to analyze a specific part of your sound file. In this case I create a AudioBufferSoundNode that contains the buffer that I got from decoding the audio data. I feed the buffer to the source, which eventually will be able to be copied inside the Analyzer. However, without the interval callback, the buffer never seems to be ready and the analysed data contains -Inifinity (which I assume is the absence of any sound, as it has nothing to read) at every index of the array. Which is why the interval is there. It analyses the data every 300ms.
Hope this helps someone!

You need to fetch the audio file and decode the audio buffer.
The url to the audio source must also be on the same domain or have have the correct CORS headers as well (as mentioned by Anthony).
Note: Replace <FILE-URI> with the path to your file in the example below.
var audioCtx = new (window.AudioContext || window.webkitAudioContext)();
var source;
var analyser = audioCtx.createAnalyser();
var button = document.querySelector('button');
var freqs;
var times;
button.addEventListener('click', (e) => {
fetch("<FILE-URI>", {
headers: new Headers({
"Content-Type" : "audio/mpeg"
})
}).then(function(response){
return response.arrayBuffer()
}).then((ab) => {
audioCtx.decodeAudioData(ab, (buffer) => {
source = audioCtx.createBufferSource();
source.connect(audioCtx.destination)
source.connect(analyser);
source.buffer = buffer;
source.start(0);
viewBufferData();
});
});
});
// Watch the changes in the audio buffer
function viewBufferData() {
setInterval(function(){
freqs = new Uint8Array(analyser.frequencyBinCount);
times = new Uint8Array(analyser.frequencyBinCount);
analyser.smoothingTimeConstant = 0.8;
analyser.fftSize = 2048;
analyser.getByteFrequencyData(freqs);
analyser.getByteTimeDomainData(times);
console.log(freqs)
console.log(times)
}, 1000)
}

If the source file from a different domain? That would fail in createMediaElementSource.

How can I play an arrayBuffer as an audio file?

I am receiving an arrayBuffer via a socket.io event and want to be able to process and play the stream as an audio file.
I am receiving the buffer like so:
retrieveAudioStream = () => {
this.socket.on('stream', (arrayBuffer) => {
console.log('arrayBuffer', arrayBuffer)
})
}
Is it possible to set the src attribute of an <audio/> element to a buffer? If not how can I play the the incoming buffer stream?
edit:
To show how I am getting my audio input and streaming it:
window.navigator.getUserMedia(constraints, this.initializeRecorder, this.handleError);
initializeRecorder = (stream) => {
const audioContext = window.AudioContext;
const context = new audioContext();
const audioInput = context.createMediaStreamSource(stream);
const bufferSize = 2048;
// create a javascript node
const recorder = context.createScriptProcessor(bufferSize, 1, 1);
// specify the processing function
recorder.onaudioprocess = this.recorderProcess;
// connect stream to our recorder
audioInput.connect(recorder);
// connect our recorder to the previous destination
recorder.connect(context.destination);
}
This is where I receive the inputBuffer event and stream via a socket.io event
recorderProcess = (e) => {
const left = e.inputBuffer.getChannelData(0);
this.socket.emit('stream', this.convertFloat32ToInt16(left))
}
EDIT 2:
Adding Raymonds suggestion:
retrieveAudioStream = () => {
const audioContext = new window.AudioContext();
this.socket.on('stream', (buffer) => {
const b = audioContext.createBuffer(1, buffer.length, audioContext.sampleRate);
b.copyToChannel(buffer, 0, 0)
const s = audioContext.createBufferSource();
s.buffer = b
})
}
Getting error: NotSupportedError: Failed to execute 'createBuffer' on 'BaseAudioContext': The number of frames provided (0) is less than or equal to the minimum bound (0).

Based on a quick read of what initializeRecorder and recorderProcess do, it looks like you're converting the float32 samples to int16 in some say and that gets sent to retrieveAudioStream in some way.
If this is correct, then the arrayBuffer is an array of int16 values. Convert them to float32 (most likely by dividing each value by 32768) and save them in a Float32Array. Then create an AudioBuffer of the same lenght and copyToChannel(float32Array, 0, 0) to write the values to the AudioBuffer. Use an AudioBufferSourceNode with this buffer to play out the audio.

Read samples from wav-file

I'm trying to make a webpage in html5 which stores sample-data from a wav-file in an array. Is there any way to get the sample-data with javascript?
I'm using a file-input to select the wav-file.
In the javascript I already added:
document.getElementById('fileinput').addEventListener('change', readFile, false);
but I have no idea what to do in readFile.
EDIT:
I tried to get the file in an ArrayBuffer, pass it to the decodeAudioData method and get a typedArraybuffer out of it.
This is my code:
var openFile = function(event) {
var input = event.target;
var audioContext = new AudioContext();
var reader = new FileReader();
reader.onload = function(){
var arrayBuffer = reader.result;
console.log("arrayBuffer:");
console.log(arrayBuffer);
audioContext.decodeAudioData(arrayBuffer, decodedDone);
};
reader.readAsArrayBuffer(input.files[0]);
};
function decodedDone(decoded) {
var typedArray = new Uint32Array(decoded, 1, decoded.length);
console.log("decoded");
console.log(decoded);
console.log("typedArray");
console.log(typedArray);
for (i=0; i<10; i++)
{
console.log(typedArray[i]);
}
}
The elements of typedArray are all 0. Is my way of creating the typedArray wrong or did I do something else wrong on?
EDIT:
I finally got it. This is my code:
var openFile = function(event) {
var input = event.target;
var audioContext = new AudioContext();
var reader = new FileReader();
reader.onload = function(){
var arrayBuffer = reader.result;
console.log("arrayBuffer:");
console.log(arrayBuffer);
audioContext.decodeAudioData(arrayBuffer, decodedDone);
};
reader.readAsArrayBuffer(input.files[0]);
};
function decodedDone(decoded) {
var typedArray = new Float32Array(decoded.length);
typedArray=decoded.getChannelData(0);
console.log("typedArray:");
console.log(typedArray);
}
Thanks for the answers!

You'll need to learn a lot about Web APIs to accomplish that, but in the end it's quite simple.
Get the file contents in an ArrayBuffer with the File API
Pass it to the Web Audio API's decodeAudioData method.
Get a typed ArrayBuffer with the raw samples you wanted.
Edit: If you want to implement an equalizer, you're wasting your time, there's a native equalizer node in the Audio API. Depending on the length of your wave file it might be better not to load it all in memory and instead to just create a source that reads from it and connect that source to an equalizer node.

Here's a simple code example to get a Float32Array from a wav audio file in JavaScript:
let audioData = await fetch("https://example.com/test.wav").then(r => r.arrayBuffer());
let audioCtx = new AudioContext({sampleRate:44100});
let decodedData = await audioCtx.decodeAudioData(audioData); // audio is resampled to the AudioContext's sampling rate
console.log(decodedData.length, decodedData.duration, decodedData.sampleRate, decodedData.numberOfChannels);
let float32Data = decodedData.getChannelData(0); // Float32Array for channel 0

How can i play raw samples PCM_16 audio data record from Android in Web (using Web-Audio or other)?

In my app on Android, i use AudioRecord and send continuouslly an bytes array PCM_16 to Node.js server.
byte[] audioBuffer = new byte[mAudioBufferSampleSize];
mAudioRecord.startRecording();
int audioRecordingState = mAudioRecord.getRecordingState();
if (audioRecordingState != AudioRecord.RECORDSTATE_RECORDING) {
Log.e(TAG, "AudioRecord is not recording");
return;
} else {
Log.v(TAG, "AudioRecord has started recording...");
}
while (inRecordMode) {
int samplesRead = mAudioRecord.read(audioBuffer, 0,
mAudioBufferSampleSize);
Log.v(TAG, "Got samples: " + samplesRead);
if (WebSocketManager.roomSocket.isConnected()) {
WebSocketManager.roomSocket.send(audioBuffer);
}
}
After that, i can stream it to the web browser in ArrayBuffer type and try to convert it to an Float32Array to be buffer for an instance of AudioContext. But i can't hear any thing or with loud noise.
function onMessage(evt) {
var context = new AudioContext();
var source = context.createBufferSource();
source.connect(context.destination);
array = new Int8Array(evt.data);
abs = new Float32Array(evt.data);
arrays = new Float32Array(abs.length);
ab = context.createBuffer(1, array.length, 44100);
ab.getChannelData(0).set(arrays);
source.buffer = ab;
source.start(0);
// then do it
}
So anyone can give me an advance, please?
P/s: use decodeAudioData just give an null error
Sorry for my poor English

The array you're getting is each sample 0-32k (range of a uint16). Each sample in an AudioBuffer's channel data is a float32 - nominal range of -1 to +1.
You need to convert the data in each sample, not just assign the value and rely on conversion.