We Keep Coding

Stream audio over websocket with low latency and no interruption

I'm working on a project which requires the ability to stream audio from a webpage to other clients. I'm already using websocket and would like to channel the data there.
My current approach uses Media Recorder, but there is a problem with sampling which causes interrupts. It registers 1s audio and then send's it to the server which relays it to other clients. Is there a way to capture a continuous audio stream and transform it to base64?
Maybe if there is a way to create a base64 audio from MediaStream without delay it would solve the problem. What do you think?
I would like to keep using websockets, I know there is webrtc.
Have you ever done something like this, is this doable?
--> Device 1
MediaStream -> MediaRecorder -> base64 -> WebSocket -> Server --> Device ..
--> Device 18
Here a demo of the current approach... you can try it here: https://jsfiddle.net/8qhvrcbz/
var sendAudio = function(b64) {
var message = 'var audio = document.createElement(\'audio\');';
message += 'audio.src = "' + b64 + '";';
message += 'audio.play().catch(console.error);';
eval(message);
console.log(b64);
}
navigator.mediaDevices.getUserMedia({
audio: true
}).then(function(stream) {
setInterval(function() {
var chunks = [];
var recorder = new MediaRecorder(stream);
recorder.ondataavailable = function(e) {
chunks.push(e.data);
};
recorder.onstop = function(e) {
var audioBlob = new Blob(chunks);
var reader = new FileReader();
reader.readAsDataURL(audioBlob);
reader.onloadend = function() {
var b64 = reader.result
b64 = b64.replace('application/octet-stream', 'audio/mpeg');
sendAudio(b64);
}
}
recorder.start();
setTimeout(function() {
recorder.stop();
}, 1050);
}, 1000);
});

Websocket is not the best. I solved by using WebRTC instead of websocket.
The solution with websocket was obtained while recording 1050ms instead of 1000, it causes a bit of overlay but still better than hearing blanks.

Although you have solved this through WebRTC, which is the industry recommended approach, I'd like to share my answer on this.
The problem here is not websockets in general but rather the MediaRecorder API. Instead of using it one can use PCM audio capture and then submit the captured array buffers into a web worker or WASM for encoding to MP3 chunks or similar.
const context = new AudioContext();
let leftChannel = [];
let rightChannel = [];
let recordingLength = null;
let bufferSize = 512;
let sampleRate = context.sampleRate;
const audioSource = context.createMediaStreamSource(audioStream);
const scriptNode = context.createScriptProcessor(bufferSize, 1, 1);
audioSource.connect(scriptNode);
scriptNode.connect(context.destination);
scriptNode.onaudioprocess = function(e) {
// Do something with the data, e.g. convert it to WAV or MP3
};
Based on my experiments this would give you "real-time" audio. My theory with the MediaRecorder API is that it does some buffering first before emitting out anything that causes the observable delay.

Recording browser audio using navigator.mediaDevices.getUserMedia

I am recording browser audio input from the microphone, and sending it via websocket to a nodeJs service that writes the stream to a .wav file.
My problem is that the first recording comes out fine, but any subsequent recordings come out sounding very slow, about half the speed and are therefore unusable.
If I refresh the browser the first recording works again, and subsequent recordings are slowed down which is why I am sure the problem is not in the nodeJs service.
My project is an Angular 5 project.
I have pasted the code I am trying below.
I am using binary.js ->
https://cdn.jsdelivr.net/binaryjs/0.2.1/binary.min.js
this.client = BinaryClient(`ws://localhost:9001`)
createStream() {
window.Stream = this.client.createStream();
window.navigator.mediaDevices.getUserMedia({ audio: true }).then(stream => {
this.success(stream);
})
}
stopRecording() {
this.recording = false;
this.win.Stream.end();
}
success(e) {
var audioContext = window.AudioContext || window.webkitAudioContext;
var context = new audioContext();
// the sample rate is in context.sampleRate
var audioInput = context.createMediaStreamSource(e);
var bufferSize = 2048;
var recorder = context.createScriptProcessor(bufferSize, 1, 1);
}
recorder.onaudioprocess = (e) => {
if (!this.recording) return;
console.log('recording');
var left = e.inputBuffer.getChannelData(0);
this.win.Stream.write(this.convertoFloat32ToInt16(left));
}
audioInput.connect(recorder)
recorder.connect(context.destination);
}
convertoFloat32ToInt16(buffer) {
var l = buffer.length;
var buf = new Int16Array(l)
while (l--) {
buf[l] = buffer[l] * 0xFFFF; //convert to 16 bit
}
return buf.buffer
}
I am stumped as to what can be going wrong so if anyone has experience using this browser tech I would appreciate any help.
Thanks.

I've had this exact problem - your problem is the sample rate you are writing your WAV file with is incorrect.
You need to pass the sample rate used by the browser and the microphone to the node.js which writes the binary WAV file.
Client side:
After a successfull navigator.mediaDevices.getUserMedia (in your case, success function), get the sampleRate variable from the AudioContext element:
var _smapleRate = context.sampleRate;
Then pass it to the node.js listener as a parameter. In my case I used:
binaryClient.createStream({ SampleRate: _smapleRate });
Server (Node.js) side:
Use the passed SampleRate to set the WAV file's sample rate. In my case this is the code:
fileWriter = new wav.FileWriter(wavPath, {
channels: 1,
sampleRate: meta.SampleRate,
bitDepth: 16
});
This will prevent broken sounds, low pitch sounds, low or fast WAV files.
Hope this helps.

Access microphone from a browser - Javascript

Is it possible to access the microphone (built-in or auxiliary) from a browser using client-side JavaScript?
Ideally, it would store the recorded audio in the browser. Thanks!

Here we capture microphone audio as a Web Audio API event loop buffer using getUserMedia() ... time domain and frequency domain snippets of each audio event loop buffer are printed (viewable in browser console just hit key F12 or ctrl+shift+i )
<html><head><meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1">
<title>capture microphone audio into buffer</title>
<script type="text/javascript">
var webaudio_tooling_obj = function () {
var audioContext = new AudioContext();
console.log("audio is starting up ...");
var BUFF_SIZE = 16384;
var audioInput = null,
microphone_stream = null,
gain_node = null,
script_processor_node = null,
script_processor_fft_node = null,
analyserNode = null;
if (!navigator.getUserMedia)
navigator.getUserMedia = navigator.getUserMedia || navigator.webkitGetUserMedia ||
navigator.mozGetUserMedia || navigator.msGetUserMedia;
if (navigator.getUserMedia){
navigator.getUserMedia({audio:true},
function(stream) {
start_microphone(stream);
},
function(e) {
alert('Error capturing audio.');
}
);
} else { alert('getUserMedia not supported in this browser.'); }
// ---
function show_some_data(given_typed_array, num_row_to_display, label) {
var size_buffer = given_typed_array.length;
var index = 0;
var max_index = num_row_to_display;
console.log("__________ " + label);
for (; index < max_index && index < size_buffer; index += 1) {
console.log(given_typed_array[index]);
}
}
function process_microphone_buffer(event) { // invoked by event loop
var i, N, inp, microphone_output_buffer;
microphone_output_buffer = event.inputBuffer.getChannelData(0); // just mono - 1 channel for now
// microphone_output_buffer <-- this buffer contains current gulp of data size BUFF_SIZE
show_some_data(microphone_output_buffer, 5, "from getChannelData");
}
function start_microphone(stream){
gain_node = audioContext.createGain();
gain_node.connect( audioContext.destination );
microphone_stream = audioContext.createMediaStreamSource(stream);
microphone_stream.connect(gain_node);
script_processor_node = audioContext.createScriptProcessor(BUFF_SIZE, 1, 1);
script_processor_node.onaudioprocess = process_microphone_buffer;
microphone_stream.connect(script_processor_node);
// --- enable volume control for output speakers
document.getElementById('volume').addEventListener('change', function() {
var curr_volume = this.value;
gain_node.gain.value = curr_volume;
console.log("curr_volume ", curr_volume);
});
// --- setup FFT
script_processor_fft_node = audioContext.createScriptProcessor(2048, 1, 1);
script_processor_fft_node.connect(gain_node);
analyserNode = audioContext.createAnalyser();
analyserNode.smoothingTimeConstant = 0;
analyserNode.fftSize = 2048;
microphone_stream.connect(analyserNode);
analyserNode.connect(script_processor_fft_node);
script_processor_fft_node.onaudioprocess = function() {
// get the average for the first channel
var array = new Uint8Array(analyserNode.frequencyBinCount);
analyserNode.getByteFrequencyData(array);
// draw the spectrogram
if (microphone_stream.playbackState == microphone_stream.PLAYING_STATE) {
show_some_data(array, 5, "from fft");
}
};
}
}(); // webaudio_tooling_obj = function()
</script>
</head>
<body>
<p>Volume</p>
<input id="volume" type="range" min="0" max="1" step="0.1" value="0.5"/>
</body>
</html>
Since this code exposes microphone data as a buffer you could add ability to stream using websockets or simply aggregate each event loop buffer into a monster buffer then download the monster to a file
Notice the call to
var audioContext = new AudioContext();
which indicates its using the Web Audio API which is baked into all modern browsers (including mobile browsers) to provide an extremely powerful audio platform of which tapping into the mic is but a tiny fragment ... NOTE the CPU usage jumps up due to this demo writing each event loop buffer into browser console log which is for testing only so actual use is far less resource intensive even when you mod this to stream audio to elsewhere
Links to some Web Audio API documentation
Basic concepts behind Web Audio API
SO wiki on Web Audio API
nice Web Audio API demos ... some with github links

Yes you can.
Using the getUserMedia() API, you can capture raw audio input from your microphone.

In a secure context, to query the devices.
getUserMedia() is a powerful feature which can only be used in secure
contexts; in insecure contexts, navigator.mediaDevices is undefined,
preventing access to getUserMedia(). A secure context is, in short, a
page loaded using HTTPS or the file:/// URL scheme, or a page loaded
from localhost.
async function getMedia(constraints) {
let stream = null;
try {
stream = await navigator.mediaDevices.getUserMedia(constraints);
console.log(stream)
} catch(err) {
document.write(err)
}
}
getMedia({ audio: true, video: true })
https://developer.mozilla.org/en-US/docs/Web/API/MediaDevices/getUserMedia

This is a simple way:
//event:
const micButtonClicked = () => {
//check the access:
isMicrophoneAllowed(isAllowed => {
if(isAllowed)
record();
else
navigator.mediaDevices.getUserMedia({audio: true})
.then(stream => record())
.catch(err => alert('need permission to use microphone'));
});
}
//isMicrophoneAllowed:
const isMicrophoneAllowed = callback => {
navigator.permissions.query({name: 'microphone'})
.then(permissionStatus => Strings.runCB(callback, permissionStatus.state === 'granted'));
}
//record:
const record = () => {
// start recording...
}

Web Audio API multiple scriptprocessor nodes

I've been searching a solution about nearly two days now for this problem.
I have a web audio api app that catches the microphone input. In one script processor i'm windowing the signal with a hanning window, which works fine when the audio chain looks like this:
source -> windowScriptProcessorNode -> audioContext.destination
Then i wanted to add another script processor to the chain like this:
source -> windowScriptProcessorNode -> otherScriptProcessorNode -> audioContext.destination
but at the inputBuffer of the otherScriptProcessorNode there are just zeros instead of the signal of windowScriptProcessorNode.
Here is some code:
var audioContext = new AudioContext();
//get microphone input via getUserMedia
navigator.getUserMedia({audio: true}, function(stream) {
//set up source
var audioSource = audioContext.createMediaStreamSource(stream);
audioSource.buffer = stream;
//set up hanning window script processor node
var windowScriptProcessorNode = audioContext.createScriptProcessor(BLOCKLENGTH,1,1);
windowScriptProcessorNode.onaudioprocess = function(e){
var windowNodeInput = e.inputBuffer.getChannelData(0);
var windowNodeOutput = e.outputBuffer.getChannelData(0);
if (windowfunction==true) {
windowNodeOutput.set(calc.applyDspWindowFunction(windowNodeInput));
}else{
windowNodeOutput.set(windowNodeInput);
}
}
//some other script processor node, just passing through the signal
var otherScriptProcessorNode = audioContext.createScriptProcessor(BLOCKLENGTH,1,1);
otherScriptProcessorNode.onaudioprocess = function(e){
var otherNodeInput = e.inputBuffer.getChannelData(0);
var otherNodeOutput = e.outputBuffer.getChannelData(0);
otherNodeOutput.set(otherNodeInput);
}
// this connnection works fine!
audioSource.connect(windowScriptProcessorNode);
windowScriptProcessorNode.connect(audioContext.destination);
/* // this connnection does NOT work
audioSource.connect(windowScriptProcessorNode);
windowScriptProcessorNode.connect(otherScriptProcessorNode);
otherScriptProcessorNode.connect(audioContext.destination);
*/
}

How can i publish audio by web socket

I am developing an application which publish audio stream from mic through web sockets i am not able to play web socket response in audio control or can anyone tell how to play audio buffer in audio control please help me out?

I use the following code to play the sounds created with a software-synth.
The samples need to be in the range [-1.0 .. 1.0]. You should initialize context in the page init function.
var context = new webkitAudioContext();
function playSound(buffer, freq, vol) // buffer, sampleRate, 0-100
{
var mBuffer = context.createBuffer(1, buffer.length, freq);
var dataBuffer = mBuffer.getChannelData(0);
var soundBuffer = buffer;
var i, n = buffer.length;
for (i=0;i<n;i++)
dataBuffer[i] = soundBuffer[i];
var node = context.createBufferSource();
node.buffer = mBuffer;
node.gain.value = 0.5 * vol/100.0;
node.connect(context.destination);
node.noteOn(0);
}