Since some days I`m trying to visualize an audiostream which is coming over webrtc.
We already wrote some visuals which are working fine for the normal local stream (webaudio microphone usage).
Then I found some really interesting things on https://github.com/muaz-khan/WebRTC-Experiment/tree/master/ for streaming the microphone input between different browsers.
We need this to have the same audio data from one backend for all clients in the frontend.
Everything works fine and some tests showed that we can hear each other. So I thought that it is also not a problem to visualize the incoming stream.
But: all frequency data are empty (zero), even if we can hear each other.
Does anybody has a solution or hint for this? Thanks in advance!
This is my test for analysing the remote frequence data:
include these files first:
webrtc-experiment.com/firebase.js
webrtc-experiment.com/one-to-many-audio-broadcasting/meeting.js
var meeting = new Meeting('test');
var audioContext = new window.webkitAudioContext();
var analyser = audioContext.createAnalyser();
// on getting local or remote streams
meeting.onaddstream = function(e) {
console.log(e.type);
console.log(e.audio);
console.log(e.stream);
if(e.type === 'local')
{
//try it with the local stream, it works!
}
else
{
var source = audioContext.createMediaStreamSource(e.stream);
source.connect(analyser);
analyser.connect(audioContext.destination);
console.log(analyser.fftSize);
console.log(analyser.frequencyBinCount);
analyser.fftSize = 64;
console.log(analyser.frequencyBinCount);
var frequencyData = new Uint8Array(analyser.frequencyBinCount);
analyser.getByteFrequencyData(frequencyData);
function update() {
requestAnimationFrame(update);
analyser.getByteFrequencyData(frequencyData);
console.log(frequencyData);
};
update();
}
};
meeting.check();
meeting.setup('test');
Note that analysing remote streams should work for Firefox and it is known to not to work in Chrome, see http://code.google.com/p/chromium/issues/detail?id=241543
A possible workaround could be taking the remote audio level value by using WebRTC statistics API.
If you go to chrome://webrtc-internals/ then select your page playing remote stream then one of the ssrc_XXXX_recv will contain dynamically changing audioOutputLevel value which you can use.
You can access the value using Chrome PeerConnection's statistics API, specifically, getStats() method.
A possible downside may be that this is the value of actual sound the user hears from the video/audio element, so if the user mutes or changes volume of the media element, it will affect the audioOutputLevel value.
Good luck! :-)
I found a simple "solution" at least for the next time:
I have plugged a male / male audio cable from microphone to headphone in all my clients. Then I read the local microphone stream and, what a miracle, i can visualize what Im hearing.
Not a good solution, but it`s doing the job..
One Question: Is it possible to re-grab the destination as a stream in javascript? Then I would not need the audio cables..
Related
I'm completely new to the Web Audio API, and not too terribly proficient in javascript. However, I had a specific function that I want to implement into a website I'm working on that requires Google's TTS API, which returns Base64 audio, to go through a reverb filter and then (preferably) autoplay the resulting audio.
So here's how the workflow looks.
TTS request to Google => Base64 response from Google => Base64 converted & sent through Convolver (reverb) node => Output sent to user's output device.
So what I'm struggling on first and foremost is getting ANY sort of response from an audio file going through the nodes. After that, I can deal with the Base64 conversions.
Any help would be appreciated. My IDE's are no help whatsoever. They all basically tell me "Congrats, this code looks fantastic!". Meanwhile, I'm over here pulling my hairs out and 2 lines of code away from jumping out my window.
Here's the code I've been working with. This obviously wouldn't be the entirety of it, but I thought I should first get some sound coming out of it before moving on.
let context;
let compressor;
let reverb;
let source1
let lowpassFilter;
let waveShaper;
let panner;
let wet;
let dry;
let masterDry;
let masterWet;
function effectsBoard () {
context = new (window.AudioContext || window.webkitAudioContext)();
// Effects Setup
lowpassFilter = context.createBiquadFilter();
waveShaper = context.createWaveShaper();
panner = context.createPanner();
compressor = context.createDynamicsCompressor();
reverb = context.createConvolver();
//Master Gains for Wet and Dry
masterDry = context.createGain();
masterWet = context.createGain();
//Connect the compressor (the last effect) to the final destination (audio output)
compressor.connect(context.destination);
//Connect the Master Wet and Dry signals to the compressor for mixing before the output.
masterDry.connect(compressor);
masterWet.connect(compressor);
//Connect Reverb to the Wet Master Gain
reverb.connect(masterWet);
//Connect source1 to the effectt - first the dry signal and then the wet
source1.connect(lowpassFilter);
lowpassFilter.connect(masterDry);
lowpassFilter.connect(reverb);
//Create a Source Buffer
fetch("voice.mp3")
.then(data => data.arrayBuffer())
.then(arrayBuffer => context.decodeAudioData(arrayBuffer))
.then(decodedAudio => {
avaAudio = decodedAudio;
});
//Then start the sources on run event
function playback() {
source1 = context.createBufferSource();
source1.buffer = avaAudio;
source1.start(context.currentTime);
}
window.addEventListener("mousedown", playback);
In skimming through your code, it looks okay. I think you're getting bit by autoplay policy.
When you create an audio context, it usually starts out as paused. You need to call context.resume(), but you can only do that on a trusted event.
mousedown isn't a trusted event. You actually need a full click event for that.
Also, at least in the code you show here, it seems effectsBoard() is never called, but I assume that there's more code.
Use your browser's developer tools to see what errors you need to see.
I'm analysing an audio file in order to use the channelData to drive another part of my webapp (basically draw graphics based on the audio file). The callback function for the playback looks something like this:
successCallback(mediaStream) {
var audioContext = new (window.AudioContext ||
window.webkitAudioContext)();
source = audioContext.createMediaStreamSource(mediaStream);
node = audioContext.createScriptProcessor(256, 1, 1);
node.onaudioprocess = function(data) {
var monoChannel = data.inputBuffer.getChannelData(0);
..
};
Somehow I thought if I run the above code with the same file it would yield the same results all the time. But that's not the case. The same audio file would trigger the onaudioprocess function sometimes 70, sometimes 72 times for instance, yielding different data all the time.
Is there a way to get consistent data of that sort in the browser?
EDIT: I'm getting the audio from a recording function on the same page. When the recording is finished the resulting file gets set as the src of an <audio> element. recorder is my MediaRecorder.
recorder.addEventListener("dataavailable", function(e) {
fileurl = URL.createObjectURL(e.data);
document.querySelector("#localaudio").src = fileurl;
..
To answer your original question: getChannelData is deterministic, i.e. it will yield the same Float32Array from the same AudioBuffer for the same channel (unless you happen to transfer the backing ArrayBuffer to another thread, in which case it will return an empty Float32Array with a detached backing buffer from then on).
I presume the problem you are encountering here is a threading issue (my guess is that the MediaStream is already playing before you start processing the audio stream from it), but it's hard to tell exactly without debugging your complete app (there are at least 3 threads at work here: an audio processing thread for the MediaStream, an audio processing thread for the AudioContext you are using, and the main thread that runs your code).
Is there a way to get consistent data of that sort in the browser?
Yes.
Instead of processing through a real-time audio stream for real-time analysis, you could just take the recording result (e.data), read it as an ArrayBuffer, and then decode it as an AudioBuffer, something like:
recorder.addEventListener("dataavailable", function (e) {
let reader = new FileReader();
reader.onload = function (e) {
audioContext.decodeAudioData(e.target.result).then(function (audioBuffer) {
var monoChannel = audioBuffer.getChannelData(0);
// monoChannel contains the entire first channel of your recording as a Float32Array
// ...
});
};
reader.readAsArrayBuffer(e.data);
}
Note: this code would become a lot simpler with async functions and Promises, but it should give a general idea of how to read the entire completed recording.
Also note: the ScriptProcessorNode is deprecated due to performance issues inherent in cross-thread data copy, especially involving the JS main thread. The preferred alternative is the much more advanced AudioWorklet, but this is a fairly new way to do things on the web and requires a solid understanding of worklets in general.
I have working on streaming live video using WebRTC based on RTCConnection with library called simple-peer, but I have faced with some lag between live stream video (with MediaRecorder) and that was played on using MediaSource
Here is recorder:
var mediaRecorder = new MediaRecorder(stream, options);
mediaRecorder.ondataavailable = handleDataAvailable;
function handleDataAvailable(event) {
if (connected && event.data.size > 0) {
peer.send(event.data);
}
}
...
peer.on('connect', () => {
// wait for 'connect' event before using the data channel
mediaRecorder.start(1);
});
Here is source that is played:
var mediaSource = new MediaSource();
var sourceBuffer;
mediaSource.addEventListener('sourceopen', args => {
sourceBuffer = mediaSource.addSourceBuffer(mimeCodec);
});
...
peer.on('data', data => {
// got a data channel message
sourceBuffer.appendBuffer(data);
});
I open two tabs and connect to myself and I see delay in playing video ...
Seems like I configured badly MediaRecorder or MediaSource
Any help will be appreciated ;)
You've combined two completely unrelated techniques for streaming the video, and are getting the worst tradeoffs of both. :-)
WebRTC has media stream handling built into it. If you expect realtime video, the WebRTC stack is what you want to use. It handles codec negotiation, auto-scales bandwidth, frame size, frame rate, and encoding parameters to match network conditions, and will outright drop chunks of time to keep playback as realtime as possible.
On the other hand, if retaining quality is more desirable than being realtime, MediaRecorder is what you would use. It makes no adjustments based on network conditions because it is unaware of those conditions. MediaRecorder doesn't know or care where you put the data after it gives you the buffers.
If you try to play back video as it's being recorded, will inevitably lag further and further behind because there is no built-in catch-up method. The only thing that can happen is a buffer underrun, where the playback side waits until there is enough data to begin playback again. Even if it becomes minutes behind, it isn't going to automatically skip ahead.
The solution is to use the right tool. It sounds like from your question that you want realtime video. Therefore, you need to use WebRTC. Fortunately simple-peer makes this... simple.
On the recording side:
const peer = new Peer({
initiator: true,
stream
});
Then on the playback side:
peer.on('stream', (stream) => {
videoEl.srcObject = stream;
});
Much simpler. The WebRTC stack handles everything for you.
I have a simple synth that plays a note for some length of time:
// Creating audio graph
var audioCtx = new (window.AudioContext || window.webkitAudioContext)();
var oscillator = audioCtx.createOscillator();
var gainNode = audioCtx.createGain();
oscillator.connect(gainNode);
gainNode.connect(audioCtx.destination);
// Setting parameters
oscillator.type = "sine";
oscillator.frequency.value = 2500;
// Run audio graph
var currentTime = offlineCtx.currentTime;
oscillator.start(currentTime);
oscillator.stop(currentTime + 1);
How can I get the PCM data of the sound the synthesiser makes? I've managed to do this with audio samples by using decodeAudioData, but I can't find an equivalent for an audio graph that isn't based on loading a sample.
I specifically want to render the audio graph with the OfflineAudioContext since I only care about retrieving the PCM data as fast as possible.
Thanks!
You say you want to use an offline context and then you don't actually use an offline context. So you should do
var offlineCtx = new OfflineAudioContext(nc, length, rate)
where nc = number of channels, length is the number of samples, and rate is the sample rate you want to use.
Create your graph, start everything and then do
offlineCtx.startRendering().then(function (buffer) {
// buffer has the PCM data you want. Save it somewhere,
// or whatever
})
(I'm not sure all browsers support promises from an offline context. If not, use offlineCtx.oncomplete to get the data. See the spec.)
Eventually I found an answer here: http://www.pp4s.co.uk/main/tu-sms-audio-recording.html#co-tu-sms-audio-recording__js but you will not like it. Apparently, the Audio API is not yes standardized enough for this to work on all browsers. So I have been able to run the code above in Firefox, but not Chrome.
Basic ideas:
use dest = ac.createMediaStreamDestination(); to get a destination
of the sound
use new MediaRecorder(dest.stream); to get a recorder
use the MediaRecorder ondataavailable and stop events to get the data and combine it into a Blob
Im trying to record a 48000Hz recording via getUserMedia. But without luck. The returned audio MediaStream returns 44100Hz. How can i set this to 48000Hz?
Here are snippets of my code:
var startUsermedia = this.startUsermedia;
navigator.getUserMedia({
audio: true,
//sampleRate: 48000
}, startUsermedia, function (e) {
console.log('No live audio input: ' + e);
});
The startUsermedia function:
startUsermedia: function (stream) {
var input = audio_context.createMediaStreamSource(stream);
console.log('Media stream created.');
// Uncomment if you want the audio to feedback directly
//input.connect(audio_context.destination);
//__log('Input connected to audio context destination.');
recorder = new Recorder(input);
console.log('Recorder initialised.');
},
I tried changing the property sampleRate of the AudioContext, but no luck.
How can i change the sampleRate to 48000Hz?
EDIT : We are also now okay with a flash solution that can record and export wav files at 48000Hz
As far as I know, there is no way to change the sample rate within an audio context. The sample rate will usually be the sample rate of your recording device and will stay that way. So you will not be able to write something like this:
var input = audio_context.createMediaStreamSource(stream);
var resampler = new Resampler(44100, 48000);
input.connect(resampler);
resampler.connect(audio_context.destination);
However, if you want to take your audio stream, resample it and then send it to the backend (or do sth. else with it outside of the Web Audio API), you can use an external sample rate converter (e.g. https://github.com/taisel/XAudioJS/blob/master/resampler.js).
var resampler = new Resampler(44100, 48000, 1, 2229);
function startUsermedia(stream) {
var input = audio_context.createMediaStreamSource(stream);
console.log('Media stream created.');
recorder = audio_context.createScriptProcessor(2048);
recorder.onaudioprocess = recorderProcess;
recorder.connect(audio_context.destination);
}
function recorderProcess(e) {
var buffer = e.inputBuffer.getChannelData(0);
var resampled = resampler.resampler(buffer);
//--> do sth with the resampled data for instance send to server
}
It looks like there is an open bug about the inability to set the sampling rate:
https://github.com/WebAudio/web-audio-api/issues/300
There's also a Chrome issue:
https://bugs.chromium.org/p/chromium/issues/detail?id=432248
I checked the latest Chromium code and there is nothing in there that lets you set the sampling rate.
Edit: Seems like it has been implemented in Chrome, but is broken currently - see the comments in the Chromium issue.
it's been added to chrome:
var ctx = new (window.AudioContext || window.webkitAudioContext)({ sampleRate:16000});
https://developer.mozilla.org/en-US/docs/Web/API/AudioContext/AudioContext
audioContext = new AudioContext({sampleRate: 48000})
Simply Set sample rate when created AudioContext object, This worked for me
NOTE: This answer is outdated.
You can't. The sample rate of the AudioContext is set by the browser/device and there is nothing you can do to change it. In fact, you will find that 44.1kHz on your machine might be 48kHz on mine. It varies to whatever the OS picks by default.
Also remember that not all hardware is capable of all sample rates.
You can use an OfflineAudioContext to essentially render your audio buffer to a different sample rate (but this is batch operation).
So you would record your recording using the normal audio context, and then use an OfflineAudioContext with a different sample rate to render your buffer. There is an example on the Mozilla page.
It is now in the spec but not yet implemented in Chromium.
Also in bugs.chromium.org, "Status: Available" does not mean it is implemented. It just means that nobody is working on it and that it is available for anyone who wants to work on it. So "Available" means "Not assigned".