We Keep Coding

iPhone 14 won't record using MediaRecorder

Our website records audio and plays it back for a user. It has worked for years with many different devices, but it started failing on the iPhone 14. I created a test app at https://nmp-recording-test.netlify.app/ so I can see what is going on. It works perfectly on all devices but it only works the first time on an iPhone 14. It works on other iPhones and it works on iPad and MacBooks using Safari or any other browser.
It looks like it will record if that is the first audio you ever do. If I get an AudioContext somewhere else the audio playback will work for that, but then the recording won't.
The only symptom I can see is that it doesn't call MediaRecorder.ondataavailable when it is not working, but I assume that is because it isn't recording.
Here is the pattern that I'm seeing with my test site:
Click "new recording". (the level indicator moves, the data available callback is triggered)
Click "listen" I hear what I just did
Click "new recording". (no levels move, no data is reported)
Click "listen" nothing is played.
But if I do anything, like click the metronome on and off then it won't record the FIRST time, either.
The "O.G. Recording" is the original way I was doing the recording, using deprecated method createMediaStreamSource() and createScriptProcessor()/createJavaScriptNode(). I thought maybe iPhone finally got rid of that, so I created the MediaRecorder version.
What I'm doing, basically, is (truncated to show the important part):
const chunks = []
function onSuccess(stream: MediaStream) {
mediaRecorder = new MediaRecorder(stream);
mediaRecorder.ondataavailable = function (e) {
chunks.push(e.data);
}
mediaRecorder.start(1000);
}
navigator.mediaDevices.getUserMedia({ audio: true }).then(onSuccess, onError);
Has anyone else seen anything different in the way the iPhone 14 handles recording?
Does anyone have a suggestion about how to debug this?
If you have an iPhone 14, would you try my test program above and let me know if you get the same results? We only have one iPhone 14 to test with, and maybe there is something weird about that device.
If it works you should see a number of lines something like data {"len":6784} appear every second when you are recording.
--- EDIT ---
I reworked the code similar to Frank zeng's suggestion and I am getting it to record, but it is still not right. The volume is really low, it looks like there are some dropouts, and there is a really long pause when resuming the AudioContext.
The new code seems to work perfectly in the other devices and browsers I have access to.
--- EDIT 2 ---
There were two problems - one is that the deprecated use of createScriptProcessor stopped working but the second one was an iOS bug that was fixed in version 16.2. So rewriting to use the AudioWorklet was needed, but keeping the recording going once it is started is not needed.

I have the same problem as you,I think the API of AudioContent.createScriptProcessor is Invalid in Iphone14, I used new API About AudioWorkletNode to replace it. And don't closed the stream, Because the second recording session of iPhone 14 is too laggy, Remember to destroy the data after recording. After testing, I have solved this problem，Here's my code，
// get stream
window.navigator.mediaDevices.getUserMedia(options).then(async (stream) => {
// that.stream = stream
that.context = new AudioContext()
await that.context.resume()
const rate = that.context.sampleRate || 44100
that.mp3Encoder = new lamejs.Mp3Encoder(1, rate, 128)
that.mediaSource = that.context.createMediaStreamSource(stream)
// API开始逐步淘汰了，如果可用则继续用，如果不可用则采用worklet方案写入音频数据
if (that.context.createScriptProcessor && typeof that.context.createScriptProcessor === 'function') {
that.mediaProcessor = that.context.createScriptProcessor(0, 1, 1)
that.mediaProcessor.onaudioprocess = event => {
window.postMessage({ cmd: 'encode', buf: event.inputBuffer.getChannelData(0) }, '*')
that._decode(event.inputBuffer.getChannelData(0))
}
} else { // 采用新方案
that.mediaProcessor = await that.initWorklet()
}
resolve()
})
// content of audioworklet function
async initWorklet() {
try {
/*音频流数据分析节点*/
let audioWorkletNode;
/*---------------加载AudioWorkletProcessor模块并将其添加到当前的Worklet----------------------------*/
await this.context.audioWorklet.addModule('/get-voice-node.js');
/*---------------AudioWorkletNode绑定加载后的AudioWorkletProcessor---------------------------------*/
audioWorkletNode = new AudioWorkletNode(this.context, "get-voice-node");
/*-------------AudioWorkletNode和AudioWorkletProcessor通信使用MessagePort--------------------------*/
console.log('audioWorkletNode', audioWorkletNode)
const messagePort = audioWorkletNode.port;
messagePort.onmessage = (e) => {
let channelData = e.data[0];
window.postMessage({ cmd: 'encode', buf: channelData }, '*')
this._decode(channelData)
}
return audioWorkletNode;
} catch (e) {
console.log(e)
}
}
// content of get-voice-node.js, Remember to put it in the static resource directory
class GetVoiceNode extends AudioWorkletProcessor {
/*
* options由new AudioWorkletNode()时传递
* */
constructor() {
super()
}
/*
* `inputList`和outputList`都是输入或输出的数组
* 比较坑的是只有128个样本？？？如何设置
* */
process (inputList, outputList, parameters) {
// console.log(inputList)
if(inputList.length>0&&inputList[0].length>0){
this.port.postMessage(inputList[0]);
}
return true //回来让系统知道我们仍处于活动状态并准备处理音频。
}
}
registerProcessor('get-voice-node', GetVoiceNode)
Destroy the recording instance and free the memory,if want use it the nextTime,you have better create new instance
this.recorder.stop()
this.audioDurationTimer && window.clearInterval(this.audioDurationTimer)
const audioBlob = this.recorder.getMp3Blob()
// Destroy the recording instance and free the memory
this.recorder = null

Unable to load a specific URL with Cypress

I’m unable to load the following URL with Cypress. Getting timeout error. I have set the page load time to 2 mins, still same issue. General URLs eg. (https://www.google.co.nz/) works fine.
it(‘First Test’, () => {
cy.visit(‘https://shop.countdown.co.nz/‘)
})

Here's a way, not the best, could be improved...
The Countdown site has an aversion to being run in an iframe, but it can be tested in a child window, see custom command here Cypress using child window
Cypress.Commands.add('openWindow', (url, features) => {
const w = Cypress.config('viewportWidth')
const h = Cypress.config('viewportHeight')
if (!features) {
features = `width=${w}, height=${h}`
}
console.log('openWindow %s "%s"', url, features)
return new Promise(resolve => {
if (window.top.aut) {
console.log('window exists already')
window.top.aut.close()
}
// https://developer.mozilla.org/en-US/docs/Web/API/Window/open
window.top.aut = window.top.open(url, 'aut', features)
// letting page enough time to load and set "document.domain = localhost"
// so we can access it
setTimeout(() => {
cy.state('document', window.top.aut.document)
cy.state('window', window.top.aut)
resolve()
}, 10000)
})
})
Can test with that like this
cy.openWindow('https://shop.countdown.co.nz/').then(() => {
cy.contains('Recipes').click()
cy.contains('Saved Recipes', {timeout:10000}) // if this is there, have navigated
})
I bumped the setTimeout() in custom command to 10 seconds, cause this site drags it's feet a bit.
Configuration:
// cypress.json
{
"baseUrl": "https://shop.countdown.co.nz/",
"chromeWebSecurity": false,
"defaultCommandTimeout": 20000 // see below for better way
}
Command timeout error
Using Gleb's child window command, there's a timeout error that I can't track the source of.
To avoid it I set "defaultCommandTimeout": 20000 in config, but since it's only needed for the openWindow call it's better to remove the global setting and use this instead
cy.then({timeout:20000}, () => {
cy.openWindow('https://shop.countdown.co.nz/', {}).then(() => {
cy.contains('Recipes').click()
cy.contains('Saved Recipes', {timeout:10000}) // if this is there, have navigated
})
})
To check if the long command timeout only applies once, break one of the inner test commands and check that that it times out in the standard 4000 ms.
cy.then({timeout:20000}, () => {
cy.openWindow('https://shop.countdown.co.nz/', {}).then(() => {
cy.contains('Will not find this').click() // Timed out retrying after 4000ms

The quotes are wrong. Try the below code:
it('First Test', ()=>{ cy.visit('https://shop.countdown.co.nz/') })

On trying to visit the URL I am getting the error:
cy.visit() failed trying to load:
https://shop.countdown.co.nz/
We attempted to make an http request to this URL but the request
failed without a response.
We received this error at the network level:
Error: ESOCKETTIMEDOUT
Common situations why this would fail:
you don't have internet access
you forgot to run / boot your web server
your web server isn't accessible
you have weird network configuration settings on your computer
Error Screenshot:
Lets look into the common situations where this might happen:
you don't have internet access: I have a internet access, so this can be ruled out.
you forgot to run / boot your web server - Your site is accessible from a normal browser, this can be ruled out as well.
your web server isn't accessible - This is a possibility where may be there are firewall settings at the server end because of which cypress is not getting any response when accessing the site.
you have weird network configuration settings on your computer - This can be ruled out as well.

I had a similar issue, so what I observed in my case was that the URL was not getting added to the iframe src property and hence cy.visit() was getting timed out each time.
So, I added the URL manually to the src property of the iframe.
Here's my custom command for reference:
Cypress.Commands.add('goto', url => {
return new Promise(res => {
setTimeout(() => {
const frame = window.top.document.getElementsByClassName('aut-iframe')[0];
frame.src = url;
var evt = window.top.document.createEvent('Event');
evt.initEvent('load', false, false);
window.dispatchEvent(evt);
res();
}, 300);
});
});
Now use cy.goto('https://yoururl.com') and you are good to go.

Is there a way to get Chrome to “forget” a device to test navigator.usb.requestDevice, navigator.serial.requestPort?

I'm hoping to migrate from using WebUSB to SerialAPI (which is explained nicely here).
Current Code:
try {
let device = await navigator.usb.requestDevice({
filters: [{
usbVendorId: RECEIVER_VENDOR_ID
}]
})
this.connect(device)
} catch (error) {
console.log(DEVICE_NAME + ': Permission Denied')
}
New Code:
try {
let device = await navigator.serial.requestPort({
filters: [{
usbVendorId: RECEIVER_VENDOR_ID
}]
})
this.connect(device)
} catch (error) {
console.log(DEVICE_NAME + ': Permission Denied')
}
The new code appears to work, but I think it's because the browser has already requested the device via the old code.
I've tried restarting Chrome as well as clearing all of the browsing history. Even closed the USB-claiming page and claimed the device with another app (during which it returns the DOMException: Unable to claim interface error), but Chrome doesn't seem to want to ask again. It just happily streams the data with the previous connection.
My hope was that using SerialAPI would be a way to avoid fighting over the USB with other processes, or at least losing to them.
Update
I had forgotten about:
Failed to execute 'requestPort' on 'Serial': "Must be handling a user gesture to show a permission request"
Does this mean that the user will need to use a button to connect to the device via SerialUSB? I think with WebUSB I was able to make the connect window automatically pop up.

For both APIs, as is noted in the update, a user gesture is required in order to call the requestDevice() or requestPort() method. It is not possible to automatically pop up this prompt. (If there is that's a bug so please let the Chrome team know so we can fix it.)
Permissions granted to a site through the WebUSB API and Web Serial API are currently tracked separately so permission to access a device through one will not automatically translate into the other.
There is not currently a way to programatically forget a device permission. That would require the navigator.permissions.revoke() method which has been abandoned. You can however manually revoke permission to access the device by clicking on the "lock" icon in the address bar while visiting the site or going to chrome://settings/content/usbDevices (for USB devices) and chrome://settings/content/serialPorts (for serial ports).

To get Chrome to "forget" the WebUSB device previously paired via navigator.usb.requestDevice API:
Open the page paired to the device you want to forget
Click on the icon in the address bar
Click x next to device. If nothing is listed, then there are no paired devices for this web page.

The new code was NOT working. It just appeared to be because Chrome was already paired with the port via the old code. There is no way the "new code" could have worked because, as noted in Kalido's comment, the SerialAPI (due to its power) requires a user gesture to connect.
The code I'm using to actually connect and get data is basically built up from a few pieces from the above links in the OP:
navigator.serial.addEventListener('connect', e => {
// Add |e.target| to the UI or automatically connect.
console.log("connected");
});
navigator.serial.addEventListener('disconnect', e => {
// Remove |e.target| from the UI. If the device was open the disconnection can
// also be observed as a stream error.
console.log("disconnected");
});
console.log(navigator.serial);
document.addEventListener('DOMContentLoaded', async () => {
const connectButton = document.querySelector('#connect') as HTMLInputElement;
if (connectButton) {
connectButton.addEventListener('click', async () => {
try {
// Request Keiser Receiver from the user.
const port = await navigator.serial.requestPort({
filters: [{ usbVendorId: 0x2341, usbProductId: not_required }]
});
try {
// Open and begin reading.
await port.open({ baudRate: 115200 });
} catch (e) {
console.log(e);
}
while (port.readable) {
const reader = port.readable.getReader();
try {
while (true) {
const { value, done } = await reader.read();
if (done) {
// Allow the serial port to be closed later.
reader.releaseLock();
break;
}
if (value) {
console.log(value);
}
}
} catch (error) {
// TODO: Handle non-fatal read error.
console.log(error);
}
}
} catch (e) {
console.log("Permission to access a device was denied implicitly or explicitly by the user.");
console.log(e);
console.log(port);
}
}
}
The device-specific Vendor and Product IDs would obviously change from device to device. In the above example I have inserted an Arduino vendor ID.
It doesn't answer the question of how to get Chrome to "forget", but I'm not sure if that's relevant when using SerialAPI because of the required gesture.
Hopefully someone with more experience will be able to post a more informative answer.

webrtc audio device disconnection and reconnection

I have a video call application based on WebRTC. It is working as expected. However when call is going on, if I disconnect and connect back audio device (mic + speaker), only speaker part is working. The mic part seems to be not working - the other side can't hear anymore.
Is there any way to inform WebRTC to take audio input again once audio device is connected back?

Is there any way to inform WebRTC to take audio input again once audio device is connected back?
Your question appears simple—the symmetry with speakers is alluring—but once we're dealing with users who have multiple cameras and microphones, it's not that simple: If your user disconnects their bluetooth headset they were using, should you wait for them to reconnect it, or immediately switch to their laptop microphone? If the latter, do you switch back if they reconnect it later? These are application decisions.
The APIs to handle these things are: primarily the ended and devicechange events, and the replaceTrack() method. You may also need the deviceId constraint, and the enumerateDevices() method to a handle multiple devices.
However, to keep things simple, let's take the assumptions in your question at face value to explore the APIs:
When the user unplugs their sole microphone (not their camera) mid-call, our job is to resume conversation with it when they reinsert it, without dropping video:
First, we listen to the ended event to learn when our local audio track drops.
When that happens, we listen for a devicechange event to detect re-insertion (of anything).
When that happens, we could check what changed using enumerateDevices(), or simply try getUserMedia again (microphone only this time).
If that succeeds, use await sender.replaceTrack(newAudioTrack) to send our new audio.
This might look like this:
let sender;
(async () => {
try {
const stream = await navigator.mediaDevices.getUserMedia({video: true, audio: true});
pc.addTrack(stream.getVideoTracks()[0], stream);
sender = pc.addTrack(stream.getAudioTracks()[0], stream);
sender.track.onended = () => navigator.mediaDevices.ondevicechange = tryAgain;
} catch (e) {
console.log(e);
}
})();
async function tryAgain() {
try {
const stream = await navigator.mediaDevices.getUserMedia({audio: true});
await sender.replaceTrack(stream.getAudioTracks()[0]);
navigator.mediaDevices.ondevicechange = null;
sender.track.onended = () => navigator.mediaDevices.ondevicechange = tryAgain;
} catch (e) {
if (e.name == "NotFoundError") return;
console.log(e);
}
}
// Your usual WebRTC negotiation code goes here
The above is for illustration only. I'm sure there are lots of corner cases to consider.

Issue when recording sound through browser

I'm currently working on audio record through web browser (.wav format) and currently we use the following code to start record, and emit data via socket :
async start () {
this.stream = await navigator.mediaDevices.getUserMedia({
audio: true,
video: false
})
this.audioCtx = new AudioContext()
const source = this.audioCtx.createMediaStreamSource(this.stream)
const scriptProcessor = this.audioCtx.createScriptProcessor(0, 1, 1)
source.connect(scriptProcessor)
scriptProcessor.connect(this.audioCtx.destination)
scriptProcessor.onaudioprocess = event => {
this.emit('record', event.inputBuffer.getChannelData(0))
}
return {
device: this.stream.getAudioTracks().length && this.stream.getAudioTracks()[0].label || 'Unknown',
sampleRate: this.audioCtx.sampleRate
}
Although it seems to work well, it seems that randomly, silent sequences are inserted in the data emitted (~8 consecutive frames, sample). This doesn't seems to be hardware related as we have the same issue regardless of the microphone used.
I would like to know if it's the way we collect the data or how we send them that causes this issue. (and possibly how to fix it)
Thanks,