We Keep Coding

Firefox navigator.mediaDevices.getUserMedia stream is correct on the first run, on the second run stream is broken and I cannot display video

I'm having trouble accessing the microphone and camera while using Firefox on windows after running this script on the second time. Chrome/edge is fine
let stream;
try {
document.getElementById('record').onclick = async () => {
stream = await navigator.mediaDevices.getUserMedia({ video: true, audio: true});
document.getElementById('video').srcObject = stream;
};
document.getElementById('stop').onclick = () => {
stream.getTracks().forEach(track => track.stop());
document.getElementById('video').srcObject = undefined;
stream = null;
};
} catch (e) {
console.error(e);
}
On the second go stream seams to be legit, it contains video and audio track, but it won't display video correctly whereas chrome and safari deals with it without any issues. Should I treat firefox in a specific way? What could be wrong? I'll add that my camera & microphone is fine + I've granted the permissions
fiddle link to the example code
Closing and reopening browser seam to make the issue go away, until I run that script again. Thanks in advance

Your code is correct. It's just that webcams tend to take a little extra time between when they're closed, and when they're re-opened. It's a big issue for webcams that don't support multiple clients simultaneously.
I've experienced this problem occasionally on Chrome, as well as Firefox.
The best thing to do is handle errors and try again.

iPhone 14 won't record using MediaRecorder

Our website records audio and plays it back for a user. It has worked for years with many different devices, but it started failing on the iPhone 14. I created a test app at https://nmp-recording-test.netlify.app/ so I can see what is going on. It works perfectly on all devices but it only works the first time on an iPhone 14. It works on other iPhones and it works on iPad and MacBooks using Safari or any other browser.
It looks like it will record if that is the first audio you ever do. If I get an AudioContext somewhere else the audio playback will work for that, but then the recording won't.
The only symptom I can see is that it doesn't call MediaRecorder.ondataavailable when it is not working, but I assume that is because it isn't recording.
Here is the pattern that I'm seeing with my test site:
Click "new recording". (the level indicator moves, the data available callback is triggered)
Click "listen" I hear what I just did
Click "new recording". (no levels move, no data is reported)
Click "listen" nothing is played.
But if I do anything, like click the metronome on and off then it won't record the FIRST time, either.
The "O.G. Recording" is the original way I was doing the recording, using deprecated method createMediaStreamSource() and createScriptProcessor()/createJavaScriptNode(). I thought maybe iPhone finally got rid of that, so I created the MediaRecorder version.
What I'm doing, basically, is (truncated to show the important part):
const chunks = []
function onSuccess(stream: MediaStream) {
mediaRecorder = new MediaRecorder(stream);
mediaRecorder.ondataavailable = function (e) {
chunks.push(e.data);
}
mediaRecorder.start(1000);
}
navigator.mediaDevices.getUserMedia({ audio: true }).then(onSuccess, onError);
Has anyone else seen anything different in the way the iPhone 14 handles recording?
Does anyone have a suggestion about how to debug this?
If you have an iPhone 14, would you try my test program above and let me know if you get the same results? We only have one iPhone 14 to test with, and maybe there is something weird about that device.
If it works you should see a number of lines something like data {"len":6784} appear every second when you are recording.
--- EDIT ---
I reworked the code similar to Frank zeng's suggestion and I am getting it to record, but it is still not right. The volume is really low, it looks like there are some dropouts, and there is a really long pause when resuming the AudioContext.
The new code seems to work perfectly in the other devices and browsers I have access to.
--- EDIT 2 ---
There were two problems - one is that the deprecated use of createScriptProcessor stopped working but the second one was an iOS bug that was fixed in version 16.2. So rewriting to use the AudioWorklet was needed, but keeping the recording going once it is started is not needed.

I have the same problem as you,I think the API of AudioContent.createScriptProcessor is Invalid in Iphone14, I used new API About AudioWorkletNode to replace it. And don't closed the stream, Because the second recording session of iPhone 14 is too laggy, Remember to destroy the data after recording. After testing, I have solved this problem，Here's my code，
// get stream
window.navigator.mediaDevices.getUserMedia(options).then(async (stream) => {
// that.stream = stream
that.context = new AudioContext()
await that.context.resume()
const rate = that.context.sampleRate || 44100
that.mp3Encoder = new lamejs.Mp3Encoder(1, rate, 128)
that.mediaSource = that.context.createMediaStreamSource(stream)
// API开始逐步淘汰了，如果可用则继续用，如果不可用则采用worklet方案写入音频数据
if (that.context.createScriptProcessor && typeof that.context.createScriptProcessor === 'function') {
that.mediaProcessor = that.context.createScriptProcessor(0, 1, 1)
that.mediaProcessor.onaudioprocess = event => {
window.postMessage({ cmd: 'encode', buf: event.inputBuffer.getChannelData(0) }, '*')
that._decode(event.inputBuffer.getChannelData(0))
}
} else { // 采用新方案
that.mediaProcessor = await that.initWorklet()
}
resolve()
})
// content of audioworklet function
async initWorklet() {
try {
/*音频流数据分析节点*/
let audioWorkletNode;
/*---------------加载AudioWorkletProcessor模块并将其添加到当前的Worklet----------------------------*/
await this.context.audioWorklet.addModule('/get-voice-node.js');
/*---------------AudioWorkletNode绑定加载后的AudioWorkletProcessor---------------------------------*/
audioWorkletNode = new AudioWorkletNode(this.context, "get-voice-node");
/*-------------AudioWorkletNode和AudioWorkletProcessor通信使用MessagePort--------------------------*/
console.log('audioWorkletNode', audioWorkletNode)
const messagePort = audioWorkletNode.port;
messagePort.onmessage = (e) => {
let channelData = e.data[0];
window.postMessage({ cmd: 'encode', buf: channelData }, '*')
this._decode(channelData)
}
return audioWorkletNode;
} catch (e) {
console.log(e)
}
}
// content of get-voice-node.js, Remember to put it in the static resource directory
class GetVoiceNode extends AudioWorkletProcessor {
/*
* options由new AudioWorkletNode()时传递
* */
constructor() {
super()
}
/*
* `inputList`和outputList`都是输入或输出的数组
* 比较坑的是只有128个样本？？？如何设置
* */
process (inputList, outputList, parameters) {
// console.log(inputList)
if(inputList.length>0&&inputList[0].length>0){
this.port.postMessage(inputList[0]);
}
return true //回来让系统知道我们仍处于活动状态并准备处理音频。
}
}
registerProcessor('get-voice-node', GetVoiceNode)
Destroy the recording instance and free the memory,if want use it the nextTime,you have better create new instance
this.recorder.stop()
this.audioDurationTimer && window.clearInterval(this.audioDurationTimer)
const audioBlob = this.recorder.getMp3Blob()
// Destroy the recording instance and free the memory
this.recorder = null

Web Audio API: how can I detect speech and record until silence, with or without a Push-To-Talk button

I'm running succesfully a client web page that act as a voice message sender, using MediaRecorder APIs:
when the user press any key, start an audio recording,
when the key is released, the audio recording is sent, via soketio, to a server for further processing.
This is a sort of PTT (Push To Talk) user experience, where the user has just to press a key (push) to activate the voice recording. And afterward he must release the key to stop the recording, triggering the message send to the server.
Here a javascript code chunk I used:
navigator.mediaDevices
.getUserMedia({ audio: true })
.then(stream => {
const mediaRecorder = new MediaRecorder(stream)
var audioChunks = []
//
// start and stop recording:
// keyboard (any key) events
//
document
.addEventListener('keydown', () => mediaRecorder.start())
document
.addEventListener('keyup', () => mediaRecorder.stop())
//
// add data chunk to mediarecorder
//
mediaRecorder
.addEventListener('dataavailable', event => {
audioChunks.push(event.data)
})
//
// mediarecorder event stop
// trigger socketio audio message emission.
//
mediaRecorder
.addEventListener('stop', () => {
socket.emit('audioMessage', audioChunks)
audioChunks = []
})
})
Now, What I want is to activate/deactivate the audio(speech) recording not only from a web page button/key/touch, but from an external hardware microphone (with a Push-To-Talk button). More precisely, I want to interface an industrial headset with PTT button on the ear dome, see the photo:
BTW, the PTT button is just a physical button that act as short-circuit toggle switch, as in the photo, just as an example:
By default the microphone is grounded and input signal == 0
When the PTT button is pressed, the micro is activated and input signal != 0.
Now my question is: how can I use Web Audio API to possibly detect when the PTT button is pressed (so audio signal is > 0) to do a mediaRecorder.start() ?
reading here: I guess I have to use the stream returned by mediaDevices.getUserMedia and create an AudioContext() processor:
navigator.mediaDevices.getUserMedia({ audio: true, video: false })
.then(handleSuccess);
const handleSuccess = function(stream) {
const context = new AudioContext();
const source = context.createMediaStreamSource(stream);
const processor = context.createScriptProcessor(1024, 1, 1);
source.connect(processor);
processor.connect(context.destination);
processor.onaudioprocess = function(e) {
// Do something with the data,
console.log(e.inputBuffer);
};
};
But what the processor.onaudioprocess function must do to start (volume > DELTA) and stop (volume < DELTA) the MediaRecorder?
I guess the volume detection could be useful for two situation:
With PTT button, where the user explicitly decide the duration of the speech, pressing and releasing the button
Without the PTT button, in this case the voice message is created with the so called VOX mode (continous audio processing)
Any idea?

I answer to my question just to share a solution I found.
The #cwilso old project: volume-meter seems to be the precise implementation of what #scott-stensland stated in comment above. See the demo: https://webaudiodemos.appspot.com/volume-meter/
UPDATE
BTW, using #cwilso project and #scott-stensland suggestion, I implemented a WeBAD opensource project to solve also my original question:
https://github.com/solyarisoftware/WeBAD

webrtc audio device disconnection and reconnection

I have a video call application based on WebRTC. It is working as expected. However when call is going on, if I disconnect and connect back audio device (mic + speaker), only speaker part is working. The mic part seems to be not working - the other side can't hear anymore.
Is there any way to inform WebRTC to take audio input again once audio device is connected back?

Is there any way to inform WebRTC to take audio input again once audio device is connected back?
Your question appears simple—the symmetry with speakers is alluring—but once we're dealing with users who have multiple cameras and microphones, it's not that simple: If your user disconnects their bluetooth headset they were using, should you wait for them to reconnect it, or immediately switch to their laptop microphone? If the latter, do you switch back if they reconnect it later? These are application decisions.
The APIs to handle these things are: primarily the ended and devicechange events, and the replaceTrack() method. You may also need the deviceId constraint, and the enumerateDevices() method to a handle multiple devices.
However, to keep things simple, let's take the assumptions in your question at face value to explore the APIs:
When the user unplugs their sole microphone (not their camera) mid-call, our job is to resume conversation with it when they reinsert it, without dropping video:
First, we listen to the ended event to learn when our local audio track drops.
When that happens, we listen for a devicechange event to detect re-insertion (of anything).
When that happens, we could check what changed using enumerateDevices(), or simply try getUserMedia again (microphone only this time).
If that succeeds, use await sender.replaceTrack(newAudioTrack) to send our new audio.
This might look like this:
let sender;
(async () => {
try {
const stream = await navigator.mediaDevices.getUserMedia({video: true, audio: true});
pc.addTrack(stream.getVideoTracks()[0], stream);
sender = pc.addTrack(stream.getAudioTracks()[0], stream);
sender.track.onended = () => navigator.mediaDevices.ondevicechange = tryAgain;
} catch (e) {
console.log(e);
}
})();
async function tryAgain() {
try {
const stream = await navigator.mediaDevices.getUserMedia({audio: true});
await sender.replaceTrack(stream.getAudioTracks()[0]);
navigator.mediaDevices.ondevicechange = null;
sender.track.onended = () => navigator.mediaDevices.ondevicechange = tryAgain;
} catch (e) {
if (e.name == "NotFoundError") return;
console.log(e);
}
}
// Your usual WebRTC negotiation code goes here
The above is for illustration only. I'm sure there are lots of corner cases to consider.

How to check which is the active user's microphone (audio input device) with JavaScript?

I've created a web app that allows users to do a voice recording and have noticed that there are problems with picking the correct audio input device. FireFox works great but Chrome and Safari don't always record if I use the default way for initializing the audio recording: navigator.mediaDevices.getUserMedia({audio: true}). Because of this, I have to specify which microphone to use like so:
let dD = [];
navigator.mediaDevices.enumerateDevices().then((devices) => {
dD = devices.filter((d) => d.kind === 'audioinput');
try {
// checking if there is a second audio input and select it
// it turns out that it works in most cases for Chrome :/
let audioD = dD[1] === undefined ? dD[0] : dD[1];
navigator.mediaDevices.getUserMedia({audio: { deviceId: audioD.deviceId }})
.then(function(stream){
startUserMedia(stream);
})
.catch(function(err) {
console.log(`${err.name}: ${err.message}`);
});
} catch (err) {
console.log(`${err.name}: ${err.message}`);
}
});
The problem with this code is that it only works sometimes. I still get reports from users complaining that the recording is not working for them or the recording is empty (which might mean that I'm using the wrong audio input).
I assume that my code is not the correct way to get the active (or let's say the working) audio input devices. How I can check which audio input is the correct one?