Implementing an AudioWorklet in Safari - javascript

I have been exploring some of the samples which are listed here: Google Chrome Labs - Audio Worklet. They have helped me get most of the way to implementing my own WASM based AudioWorklet. It soon came to my attention that Safari doesn't support Worklet.addModule()? I can't find any alternative ways online that demonstrate how to implement a Audio Worklet without addModule, can anyone help me understand if it is possible use an Audio Worklet in Safari without this method?

I found that in Chrome, you have to:
// wait for user interaction
const context = new AudioContext()
await context.audioWorklet.addModule('./worklet.js')
const worklet = new AudioWorkletNode(context, 'workletName')
worklet.connect(context.destination)
However in Safari, if I enable "Console > Media Logging" I can see this fails with
BaseAudioContext::willBeginPlayback returning false, not processing user gesture or capturing
This seems to be due to the await between the user interaction and the new AudioWorkletNode.
If I change the order in Safari to:
const context = new AudioContext()
await context.audioWorklet.addModule('./worklet.js')
// wait for user interaction
const worklet = new AudioWorkletNode(context, 'workletName')
worklet.connect(context.destination)
This works, but it now doesn't work in Chrome (!) which now gives the error:
The AudioContext was not allowed to start. It must be resumed (or created) after a user gesture on the page
Perhaps enable "Console > Media Logging" is Safari dev tools and see if this is the same issue you are getting?
To make it work in both Safari and Chrome, you can create the AudioContext on page load and then call resume after user interaction:
const context = new AudioContext()
await context.audioWorklet.addModule('./worklet.js')
// wait for user interaction
await context.resume();
const worklet = new AudioWorkletNode(context, 'workletName')
worklet.connect(context.destination)

I'm not sure why it didn't work for you but the latest versions of Safari support the AudioWorklet (with some bugs). It doesn't work on pages served over http. But it should work on pages served over https though. Maybe that was the issue.
If you want to support older browsers (that don't support the AudioWorklet yet) you could try standardized-audio-context or jariseon/audioworklet-polyfill or GoogleChromeLabs/audioworklet-polyfill or audioworkletpolyfill.js from the javascriptmusic project.

Related

how to launch Google Chrome with authenticated Proxies

I want to launch Google Chrome with authenticated proxies so I can connect a puppeteer instance with it. I'm using this cmd line to launch a new instance:
chrome --remote-debugging-port=9222 --user-data-dir="C:\Users\USER\AppData\Local\Google\Chrome\User Data
I managed to use authenticated proxies with Chrome, but it was a bit complicated, especially in my case, as I want to launch multiple Chrome browser, each with its own proxies.
I used this: proxy-login-automator
It worked fine, but as I said, it's bit complicated and it needs a bit of work, so I can integrate it as I want to use it. This is how am connecting to the Chrome instance:
const browserURL = 'http://127.0.0.1:9222';
const browser = await puppeteer.connect({browserURL});
const page = await browser.newPage();

web worker not communicating with react component

I use audio-recorder-polyfill in my React project, to make possible audio recording for Safari. It seems to work in getting the recording to take place, however, no audio data gets available. The event "dataavailable" never gets fired, and no data seems to be "compiled" after stopping recording either.
recordFunc() {
navigator.mediaDevices.getUserMedia({ audio: true }).then(stream => {
recorder = new MediaRecorder(stream);
// Set record to <audio> when recording will be finished
recorder.addEventListener('dataavailable', e => {
this.audio.current.src = URL.createObjectURL(e.data);
})
// Start recording
recorder.start();
})
}
stopFunc() {
// Stop recording
recorder.stop();
// Remove “recording” icon from browser tab
recorder.stream.getTracks().forEach(i => i.stop());
}
There have been a number of similar issues posted on audio-recorder-polyfill's issue tracker.
a
b
c
d
e
Root cause
One of those issues, #4 (not listed above), is still open. Several comments on that issue tracker hint that the root issue is that Safari cancels the AudioContext if it was not created in a handler for a user interaction (e.g. a click).
Possible solutions
You may be able to get it to work if you:
Do the initialisation inside a handler for user interaction (i.e. <button onclick="recordFunc()">)
Do not attempt to reuse the MediaStream returned from getUserMedia() for multiple recordings
Do not attempt more than 4 (or 6?) audio recordings on the same page (sources [1], [2] mention that Safari will block this)
Alternative libraries
You might also be able to try the StereoAudioRecorder class from the RecordRTC package, which has more users (3K) but appears less maintained, and might work
Upcoming support
If you'd prefer to stick to the MediaRecorder API and the tips above don't work for you, the good news is that there is experimental support for MediaRecorder in Safari 12.4 and up (iOS and macOS), and it appears to be supported in the latest Safari Technology Preview.
See also
The comments in this issue may also be useful

Cannot record video+audio on iPhone Safari

I have used RecordRTC for capturing the video+audio from the browser.
For Android devices, it's working perfectly as expected. But in iPhone devices especially on the safari browser, it's not recording as expected.
Browser console produces the following error.
Your browser does not support Media Recorder API. Please try other modules e.g. WhammyRecorder or StereoAudioRecorder.
Could someone please help me out like:
Does Safari support basic video capturing?
It is better to use StereoAudioRecorder which is made in RecordRTC.js as Recorder for Safari.
This is the document of StereoAudioRecorder.
You're going to have to read this first :)
https://recordrtc.org/StereoAudioRecorder.html
const option = {
type: 'video',
recorderType: StereoAudioRecorder
};
const recorder = new RecordRTC(mic, options);
recorder.startRecording();
I hope this will help you.
https://developer.mozilla.org/en-US/docs/Web/API/MediaRecorder
MediaRecorder was introduced in iOS14 (Safari 14), if you have an older version, this can be your problem.

Web Audio API filter not working in Safari

I am in the process of developing an HTML5 canvas interactive piece that uses Createjs and the Web Audio API. I've managed to get audio working in Chrome/Firefox/Safari despite the deprecation of webkitAudioContext by Chrome and FF but not Safari. However, filters for some reason are not working in Safari, but sound still plays. Filters DO work in Chrome/FF.
I have my filters set up like this:
var sound = new Audio();
sound.src = './sounds/sound.mp3';
sound.autoplay = false;
sound.loop = true;
soundSource = context.createMediaElementSource(sound);
var soundFilter = context.createBiquadFilter();
soundFilter.type = "lowpass";
soundFilter.frequency.value = 500;
soundSource.connect(soundFilter);
soundFilter.connect(context.destination);
Am I unknowingly using a deprecated term or something? Live project can be found here. Cheers.
UPDATE: This has been recognised as a genuine bug by the WebKit team, and will be patched. Full details here
Apparently Safari doesn't implement createMediaElementSource correctly. So instead of redirecting the sound through your Web Audio nodes, it still just plays the sound directly to the audio device.
Is there any particular reason why you can't use a BufferSourceNode? It makes you jump through extra hoops to get the sound file and decode it, but it should work.

Why aren't Safari or Firefox able to process audio data from MediaElementSource?

Neither Safari or Firefox are able to process audio data from a MediaElementSource using the Web Audio API.
var audioContext, audioProcess, audioSource,
result = document.createElement('h3'),
output = document.createElement('span'),
mp3 = '//www.jonathancoulton.com/wp-content/uploads/encodes/Smoking_Monkey/mp3/09_First_of_May_mp3_3a69021.mp3',
ogg = '//upload.wikimedia.org/wikipedia/en/4/45/ACDC_-_Back_In_Black-sample.ogg',
gotData = false, data, audio = new Audio();
function connect() {
audioContext = window.AudioContext ? new AudioContext() : new webkitAudioContext(),
audioSource = audioContext.createMediaElementSource( audio ),
audioScript = audioContext.createScriptProcessor( 2048 );
audioSource.connect( audioScript );
audioSource.connect( audioContext.destination );
audioScript.connect( audioContext.destination );
audioScript.addEventListener('audioprocess', function(e){
if ((data = e.inputBuffer.getChannelData(0)[0]*3)) {
output.innerHTML = Math.abs(data).toFixed(3);
if (!gotData) gotData = true;
}
}, false);
}
(function setup(){
audio.volume = 1/3;
audio.controls = true;
audio.autoplay = true;
audio.src = audio.canPlayType('audio/mpeg') ? mp3 : ogg;
audio.addEventListener('canplay', connect);
result.innerHTML = 'Channel Data: ';
output.innerHTML = '0.000';
document.body.appendChild(result).appendChild(output);
document.body.appendChild(audio);
})();
Are there any plans to patch this in the near future? Or is there some work-around that would still provide the audio controls to the user?
For Apple, this something that could be fixed in the WebKit Nightlies or will we have to wait until Safari 8.0 release to get HTML5 <audio> playing nicely with the Web Audio API? The Web Audio API has existed in Safari since at least version 6.0 and I initially posted this question long before Safari 7.0 was released. Is there a reason this wasn't fixed already? Will it ever be fixed?
For Mozilla, I know you're still in the process of switching over from the old Audio Data API, but is this a known issue with your Web Audio implementation and is it going to be fixed before the next release of Firefox?
This answer is quoted almost exactly from my answer to a related question: Firefox 25 and AudioContext createJavaScriptNote not a function
Firefox does support MediaElementSource if the media adheres to the Same-Origin Policy, however there is no error produced by Firefox when attempting to use media from a remote origin.
The specification is not really specific about it (pun intended), but I've been told that this is an intended behavior, and the issue is actually with Chrome… It's the Blink implementations (Chrome, Opera) that need to be updated to require CORS.
MediaElementSource Node and Cross-Origin Media Resources:
From: Robert O'Callahan <robert#ocallahan.org>
Date: Tue, 23 Jul 2013 16:30:00 +1200
To: "public-audio#w3.org" <public-audio#w3.org>
HTML media elements can play media resources from any origin. When an
element plays a media resource from an origin different from the page's
origin, we must prevent page script from being able to read the contents of
the media (e.g. extract video frames or audio samples). In particular we
should prevent ScriptProcessorNodes from getting access to the media's
audio samples. We should also information about samples leaking in other
ways (e.g. timing channel attacks). Currently the Web Audio spec says
nothing about this.
I think we should solve this by preventing any non-same-origin data from
entering Web Audio. That will minimize the attack surface and the impact on
Web Audio.
My proposal is to make MediaElementAudioSourceNode convert data coming from
a non-same origin stream to silence.
If this proposal makes it into spec it will be nearly impossible for a developer to even realize why his MediaElementSource is not working. As it stands right now, calling createMediaElementSource() on an <audio> element in Firefox 26 actually stops the <audio> controls from working at all and throws no errors.
What dangerous things can you do with the audio/video data from a remote origin? The general idea is that without applying the Same-Origin Policy to a MediaElementSource node, some malicious javascript could access media that only the user should have access to (session, vpn, local server, network drives) and send its contents—or some representation of it—to an attacker.
The HTML5 media elements don't have these restrictions by default. You can include remote media across all browsers by using the <audio>, <img>, or <video> elements. It's only when you want to manipulate or extract the data from these remote resources that the Same-Origin Policy comes into play.
[It's] for the same reason that you cannot dump image data cross-origin via <canvas>: media may contain sensitive information and therefore allowing rogue sites to dump and re-route content is a security issue. - #nmaier
createMediaElementSource() does not work properly in Safari 8.0.5 (and possibly earlier) but is fixed in Webkit Nightly as of 10600.5.17, r183978

Categories