We Keep Coding

iPhone 14 won't record using MediaRecorder

Our website records audio and plays it back for a user. It has worked for years with many different devices, but it started failing on the iPhone 14. I created a test app at https://nmp-recording-test.netlify.app/ so I can see what is going on. It works perfectly on all devices but it only works the first time on an iPhone 14. It works on other iPhones and it works on iPad and MacBooks using Safari or any other browser.
It looks like it will record if that is the first audio you ever do. If I get an AudioContext somewhere else the audio playback will work for that, but then the recording won't.
The only symptom I can see is that it doesn't call MediaRecorder.ondataavailable when it is not working, but I assume that is because it isn't recording.
Here is the pattern that I'm seeing with my test site:
Click "new recording". (the level indicator moves, the data available callback is triggered)
Click "listen" I hear what I just did
Click "new recording". (no levels move, no data is reported)
Click "listen" nothing is played.
But if I do anything, like click the metronome on and off then it won't record the FIRST time, either.
The "O.G. Recording" is the original way I was doing the recording, using deprecated method createMediaStreamSource() and createScriptProcessor()/createJavaScriptNode(). I thought maybe iPhone finally got rid of that, so I created the MediaRecorder version.
What I'm doing, basically, is (truncated to show the important part):
const chunks = []
function onSuccess(stream: MediaStream) {
mediaRecorder = new MediaRecorder(stream);
mediaRecorder.ondataavailable = function (e) {
chunks.push(e.data);
}
mediaRecorder.start(1000);
}
navigator.mediaDevices.getUserMedia({ audio: true }).then(onSuccess, onError);
Has anyone else seen anything different in the way the iPhone 14 handles recording?
Does anyone have a suggestion about how to debug this?
If you have an iPhone 14, would you try my test program above and let me know if you get the same results? We only have one iPhone 14 to test with, and maybe there is something weird about that device.
If it works you should see a number of lines something like data {"len":6784} appear every second when you are recording.
--- EDIT ---
I reworked the code similar to Frank zeng's suggestion and I am getting it to record, but it is still not right. The volume is really low, it looks like there are some dropouts, and there is a really long pause when resuming the AudioContext.
The new code seems to work perfectly in the other devices and browsers I have access to.
--- EDIT 2 ---
There were two problems - one is that the deprecated use of createScriptProcessor stopped working but the second one was an iOS bug that was fixed in version 16.2. So rewriting to use the AudioWorklet was needed, but keeping the recording going once it is started is not needed.

I have the same problem as you,I think the API of AudioContent.createScriptProcessor is Invalid in Iphone14, I used new API About AudioWorkletNode to replace it. And don't closed the stream, Because the second recording session of iPhone 14 is too laggy, Remember to destroy the data after recording. After testing, I have solved this problem，Here's my code，
// get stream
window.navigator.mediaDevices.getUserMedia(options).then(async (stream) => {
// that.stream = stream
that.context = new AudioContext()
await that.context.resume()
const rate = that.context.sampleRate || 44100
that.mp3Encoder = new lamejs.Mp3Encoder(1, rate, 128)
that.mediaSource = that.context.createMediaStreamSource(stream)
// API开始逐步淘汰了，如果可用则继续用，如果不可用则采用worklet方案写入音频数据
if (that.context.createScriptProcessor && typeof that.context.createScriptProcessor === 'function') {
that.mediaProcessor = that.context.createScriptProcessor(0, 1, 1)
that.mediaProcessor.onaudioprocess = event => {
window.postMessage({ cmd: 'encode', buf: event.inputBuffer.getChannelData(0) }, '*')
that._decode(event.inputBuffer.getChannelData(0))
}
} else { // 采用新方案
that.mediaProcessor = await that.initWorklet()
}
resolve()
})
// content of audioworklet function
async initWorklet() {
try {
/*音频流数据分析节点*/
let audioWorkletNode;
/*---------------加载AudioWorkletProcessor模块并将其添加到当前的Worklet----------------------------*/
await this.context.audioWorklet.addModule('/get-voice-node.js');
/*---------------AudioWorkletNode绑定加载后的AudioWorkletProcessor---------------------------------*/
audioWorkletNode = new AudioWorkletNode(this.context, "get-voice-node");
/*-------------AudioWorkletNode和AudioWorkletProcessor通信使用MessagePort--------------------------*/
console.log('audioWorkletNode', audioWorkletNode)
const messagePort = audioWorkletNode.port;
messagePort.onmessage = (e) => {
let channelData = e.data[0];
window.postMessage({ cmd: 'encode', buf: channelData }, '*')
this._decode(channelData)
}
return audioWorkletNode;
} catch (e) {
console.log(e)
}
}
// content of get-voice-node.js, Remember to put it in the static resource directory
class GetVoiceNode extends AudioWorkletProcessor {
/*
* options由new AudioWorkletNode()时传递
* */
constructor() {
super()
}
/*
* `inputList`和outputList`都是输入或输出的数组
* 比较坑的是只有128个样本？？？如何设置
* */
process (inputList, outputList, parameters) {
// console.log(inputList)
if(inputList.length>0&&inputList[0].length>0){
this.port.postMessage(inputList[0]);
}
return true //回来让系统知道我们仍处于活动状态并准备处理音频。
}
}
registerProcessor('get-voice-node', GetVoiceNode)
Destroy the recording instance and free the memory,if want use it the nextTime,you have better create new instance
this.recorder.stop()
this.audioDurationTimer && window.clearInterval(this.audioDurationTimer)
const audioBlob = this.recorder.getMp3Blob()
// Destroy the recording instance and free the memory
this.recorder = null

Is there a way to get Chrome to “forget” a device to test navigator.usb.requestDevice, navigator.serial.requestPort?

I'm hoping to migrate from using WebUSB to SerialAPI (which is explained nicely here).
Current Code:
try {
let device = await navigator.usb.requestDevice({
filters: [{
usbVendorId: RECEIVER_VENDOR_ID
}]
})
this.connect(device)
} catch (error) {
console.log(DEVICE_NAME + ': Permission Denied')
}
New Code:
try {
let device = await navigator.serial.requestPort({
filters: [{
usbVendorId: RECEIVER_VENDOR_ID
}]
})
this.connect(device)
} catch (error) {
console.log(DEVICE_NAME + ': Permission Denied')
}
The new code appears to work, but I think it's because the browser has already requested the device via the old code.
I've tried restarting Chrome as well as clearing all of the browsing history. Even closed the USB-claiming page and claimed the device with another app (during which it returns the DOMException: Unable to claim interface error), but Chrome doesn't seem to want to ask again. It just happily streams the data with the previous connection.
My hope was that using SerialAPI would be a way to avoid fighting over the USB with other processes, or at least losing to them.
Update
I had forgotten about:
Failed to execute 'requestPort' on 'Serial': "Must be handling a user gesture to show a permission request"
Does this mean that the user will need to use a button to connect to the device via SerialUSB? I think with WebUSB I was able to make the connect window automatically pop up.

For both APIs, as is noted in the update, a user gesture is required in order to call the requestDevice() or requestPort() method. It is not possible to automatically pop up this prompt. (If there is that's a bug so please let the Chrome team know so we can fix it.)
Permissions granted to a site through the WebUSB API and Web Serial API are currently tracked separately so permission to access a device through one will not automatically translate into the other.
There is not currently a way to programatically forget a device permission. That would require the navigator.permissions.revoke() method which has been abandoned. You can however manually revoke permission to access the device by clicking on the "lock" icon in the address bar while visiting the site or going to chrome://settings/content/usbDevices (for USB devices) and chrome://settings/content/serialPorts (for serial ports).

To get Chrome to "forget" the WebUSB device previously paired via navigator.usb.requestDevice API:
Open the page paired to the device you want to forget
Click on the icon in the address bar
Click x next to device. If nothing is listed, then there are no paired devices for this web page.

The new code was NOT working. It just appeared to be because Chrome was already paired with the port via the old code. There is no way the "new code" could have worked because, as noted in Kalido's comment, the SerialAPI (due to its power) requires a user gesture to connect.
The code I'm using to actually connect and get data is basically built up from a few pieces from the above links in the OP:
navigator.serial.addEventListener('connect', e => {
// Add |e.target| to the UI or automatically connect.
console.log("connected");
});
navigator.serial.addEventListener('disconnect', e => {
// Remove |e.target| from the UI. If the device was open the disconnection can
// also be observed as a stream error.
console.log("disconnected");
});
console.log(navigator.serial);
document.addEventListener('DOMContentLoaded', async () => {
const connectButton = document.querySelector('#connect') as HTMLInputElement;
if (connectButton) {
connectButton.addEventListener('click', async () => {
try {
// Request Keiser Receiver from the user.
const port = await navigator.serial.requestPort({
filters: [{ usbVendorId: 0x2341, usbProductId: not_required }]
});
try {
// Open and begin reading.
await port.open({ baudRate: 115200 });
} catch (e) {
console.log(e);
}
while (port.readable) {
const reader = port.readable.getReader();
try {
while (true) {
const { value, done } = await reader.read();
if (done) {
// Allow the serial port to be closed later.
reader.releaseLock();
break;
}
if (value) {
console.log(value);
}
}
} catch (error) {
// TODO: Handle non-fatal read error.
console.log(error);
}
}
} catch (e) {
console.log("Permission to access a device was denied implicitly or explicitly by the user.");
console.log(e);
console.log(port);
}
}
}
The device-specific Vendor and Product IDs would obviously change from device to device. In the above example I have inserted an Arduino vendor ID.
It doesn't answer the question of how to get Chrome to "forget", but I'm not sure if that's relevant when using SerialAPI because of the required gesture.
Hopefully someone with more experience will be able to post a more informative answer.

webrtc audio device disconnection and reconnection

I have a video call application based on WebRTC. It is working as expected. However when call is going on, if I disconnect and connect back audio device (mic + speaker), only speaker part is working. The mic part seems to be not working - the other side can't hear anymore.
Is there any way to inform WebRTC to take audio input again once audio device is connected back?

Is there any way to inform WebRTC to take audio input again once audio device is connected back?
Your question appears simple—the symmetry with speakers is alluring—but once we're dealing with users who have multiple cameras and microphones, it's not that simple: If your user disconnects their bluetooth headset they were using, should you wait for them to reconnect it, or immediately switch to their laptop microphone? If the latter, do you switch back if they reconnect it later? These are application decisions.
The APIs to handle these things are: primarily the ended and devicechange events, and the replaceTrack() method. You may also need the deviceId constraint, and the enumerateDevices() method to a handle multiple devices.
However, to keep things simple, let's take the assumptions in your question at face value to explore the APIs:
When the user unplugs their sole microphone (not their camera) mid-call, our job is to resume conversation with it when they reinsert it, without dropping video:
First, we listen to the ended event to learn when our local audio track drops.
When that happens, we listen for a devicechange event to detect re-insertion (of anything).
When that happens, we could check what changed using enumerateDevices(), or simply try getUserMedia again (microphone only this time).
If that succeeds, use await sender.replaceTrack(newAudioTrack) to send our new audio.
This might look like this:
let sender;
(async () => {
try {
const stream = await navigator.mediaDevices.getUserMedia({video: true, audio: true});
pc.addTrack(stream.getVideoTracks()[0], stream);
sender = pc.addTrack(stream.getAudioTracks()[0], stream);
sender.track.onended = () => navigator.mediaDevices.ondevicechange = tryAgain;
} catch (e) {
console.log(e);
}
})();
async function tryAgain() {
try {
const stream = await navigator.mediaDevices.getUserMedia({audio: true});
await sender.replaceTrack(stream.getAudioTracks()[0]);
navigator.mediaDevices.ondevicechange = null;
sender.track.onended = () => navigator.mediaDevices.ondevicechange = tryAgain;
} catch (e) {
if (e.name == "NotFoundError") return;
console.log(e);
}
}
// Your usual WebRTC negotiation code goes here
The above is for illustration only. I'm sure there are lots of corner cases to consider.

How to check which is the active user's microphone (audio input device) with JavaScript?

I've created a web app that allows users to do a voice recording and have noticed that there are problems with picking the correct audio input device. FireFox works great but Chrome and Safari don't always record if I use the default way for initializing the audio recording: navigator.mediaDevices.getUserMedia({audio: true}). Because of this, I have to specify which microphone to use like so:
let dD = [];
navigator.mediaDevices.enumerateDevices().then((devices) => {
dD = devices.filter((d) => d.kind === 'audioinput');
try {
// checking if there is a second audio input and select it
// it turns out that it works in most cases for Chrome :/
let audioD = dD[1] === undefined ? dD[0] : dD[1];
navigator.mediaDevices.getUserMedia({audio: { deviceId: audioD.deviceId }})
.then(function(stream){
startUserMedia(stream);
})
.catch(function(err) {
console.log(`${err.name}: ${err.message}`);
});
} catch (err) {
console.log(`${err.name}: ${err.message}`);
}
});
The problem with this code is that it only works sometimes. I still get reports from users complaining that the recording is not working for them or the recording is empty (which might mean that I'm using the wrong audio input).
I assume that my code is not the correct way to get the active (or let's say the working) audio input devices. How I can check which audio input is the correct one?

Force getUserMedia re-prompt when using HTTPS on Firefox

I am making a simple app with two webcams that needs to work only on latest Firefox. Locally it works fine:
the user is prompted for the access to the camera
the user selects one camera
the user is prompted again
the user selects the second camera
both streams work fine
However, when I upload it to the server which serves the page through HTTPS, the access from the first camera is remembered and I just get two of the same streams.
Is there a way to force re-prompting on HTTPS so that the user can select the other camera, as well?
This is my code:
function handleSuccess1(stream) {
video1.srcObject = stream;
navigator.mediaDevices.getUserMedia(constraints).
then(handleSuccess2).catch(handleError);
}
function handleSuccess2(stream) {
// this gets called automatically with the first stream
// without re-prompting the user
video2.srcObject = stream;
}
const constraints = {
video: true
};
function handleError(error) {
console.error(error);
}
navigator.mediaDevices.getUserMedia(constraints).
then(handleSuccess1).catch(handleError);

Use navigator.mediaDevices.enumerateDevices() to list the available cameras and/or microphones.
You can read about it in more detail here: https://developer.mozilla.org/en-US/docs/Web/API/MediaDevices/enumerateDevices