I'm completely new to the Web Audio API, and not too terribly proficient in javascript. However, I had a specific function that I want to implement into a website I'm working on that requires Google's TTS API, which returns Base64 audio, to go through a reverb filter and then (preferably) autoplay the resulting audio.
So here's how the workflow looks.
TTS request to Google => Base64 response from Google => Base64 converted & sent through Convolver (reverb) node => Output sent to user's output device.
So what I'm struggling on first and foremost is getting ANY sort of response from an audio file going through the nodes. After that, I can deal with the Base64 conversions.
Any help would be appreciated. My IDE's are no help whatsoever. They all basically tell me "Congrats, this code looks fantastic!". Meanwhile, I'm over here pulling my hairs out and 2 lines of code away from jumping out my window.
Here's the code I've been working with. This obviously wouldn't be the entirety of it, but I thought I should first get some sound coming out of it before moving on.
let context;
let compressor;
let reverb;
let source1
let lowpassFilter;
let waveShaper;
let panner;
let wet;
let dry;
let masterDry;
let masterWet;
function effectsBoard () {
context = new (window.AudioContext || window.webkitAudioContext)();
// Effects Setup
lowpassFilter = context.createBiquadFilter();
waveShaper = context.createWaveShaper();
panner = context.createPanner();
compressor = context.createDynamicsCompressor();
reverb = context.createConvolver();
//Master Gains for Wet and Dry
masterDry = context.createGain();
masterWet = context.createGain();
//Connect the compressor (the last effect) to the final destination (audio output)
compressor.connect(context.destination);
//Connect the Master Wet and Dry signals to the compressor for mixing before the output.
masterDry.connect(compressor);
masterWet.connect(compressor);
//Connect Reverb to the Wet Master Gain
reverb.connect(masterWet);
//Connect source1 to the effectt - first the dry signal and then the wet
source1.connect(lowpassFilter);
lowpassFilter.connect(masterDry);
lowpassFilter.connect(reverb);
//Create a Source Buffer
fetch("voice.mp3")
.then(data => data.arrayBuffer())
.then(arrayBuffer => context.decodeAudioData(arrayBuffer))
.then(decodedAudio => {
avaAudio = decodedAudio;
});
//Then start the sources on run event
function playback() {
source1 = context.createBufferSource();
source1.buffer = avaAudio;
source1.start(context.currentTime);
}
window.addEventListener("mousedown", playback);
In skimming through your code, it looks okay. I think you're getting bit by autoplay policy.
When you create an audio context, it usually starts out as paused. You need to call context.resume(), but you can only do that on a trusted event.
mousedown isn't a trusted event. You actually need a full click event for that.
Also, at least in the code you show here, it seems effectsBoard() is never called, but I assume that there's more code.
Use your browser's developer tools to see what errors you need to see.
Related
I'm developing a game using javascript and other web technologies. In it, there's a game mode that is basically a tower defense, in which multiple objects may need to make use of the same audio file(.ogg) at the same time. Loading a file and creating a new webaudio for each one of those lags it too much, even if I attempt to stream it instead of a simple sync read, and if I create and save a webaudio in a variable to use multiple times, each time its playing and there is a new request to play said audio, the one that was playing will stop to allow for the new one to play(so, with enough of those, nothing plays at all).
With those issues, I decided to make copies of said webaudio object each time it was gonna be played, but its not only slow to do so, but also creates a minor memory leak(at least the way I did it).
How can I properly cache a webaudio for re-use? Consider that I'm pretty sure I'll need a new one each time because each audio has a position, and thus each of them will play differently, based on player position in relation to object that is playing the audio
You tagged your question with web-audio-api, but from the body of this question, it seems you are using an HTMLMediaElement <audio> instead of the Web Audio API.
So I'll invite you to do the transition to that Web Audio API.
From there you'll be able to decode once your audio file, keep only once the decoded data as an AudioBuffer, and create many readers that will all hook to that one and only AudioBuffer, without eating any more memory.
const btn = document.querySelector("button")
const context = new AudioContext();
// a GainNode to control the output volume of our audio
const volumeNode = context.createGain();
volumeNode.gain.value = 0.5; // from 0 to 1
volumeNode.connect(context.destination);
fetch("https://dl.dropboxusercontent.com/s/agepbh2agnduknz/camera.mp3")
// get the resource as an ArrayBuffer
.then((resp) => resp.arrayBuffer())
// decode the Audio data from this resource
.then((buffer) => context.decodeAudioData(buffer))
// now we have our AudioBuffer object, ready to be played
.then((audioBuffer) => {
btn.onclick = (evt) => {
// allowing an AudioContext to make noise
// must be required from an user-gesture
if (context.status === "suspended") {
context.resume();
}
// a very light player object
const source = context.createBufferSource();
// a simple pointer to the big AudioBuffer (no copy)
source.buffer = audioBuffer;
// connect to our volume node, itself connected to audio output
source.connect(volumeNode);
// start playing now
source.start(0);
};
// now you can spam the button!
btn.disabled = false;
})
.catch(console.error);
<button disabled>play</button>
I made changes to an audio buffer like gain and panning, connected them to an audio context.
Now I want to save to a file with all the implemented changes.
Saving the buffer as is would give me the original audio without the changes.
Any idea of a method or a procedure existed to do that?
On way is to use a MediaRecorder to save the modified audio.
So, in addition to connecting to the destination, connect to a MediaStreamDestinationNode. This node has a stream object that you can use to initialize a MediaRecorder. Set up the recorder to save the data when data is available. When you're down recording, you have a blob that you can then download.
Many details are missing here, but you can find out how to use a MediaRecorder using the MDN example.
I found a solution, with OfflineAudioContext.
Here is an example with adding a gain change to my audio and saving it.
On the last line of the code I get the array buffer with the changes I made.
From there, I can go on saving the file.
let offlineCtx = new OfflineAudioContext(this.bufferNode.buffer.numberOfChannels, this.bufferNode.buffer.length, this.bufferNode.buffer.sampleRate);
let obs = offlineCtx.createBufferSource();
obs.buffer = this.buffer;
let gain = offlineCtx.createGain();
gain.gain.value = this.gain.gain.value;
obs.connect(gain).connect(offlineCtx.destination);
obs.start();
let obsRES = this.ctx.createBufferSource();
await offlineCtx.startRendering().then(r => {
obsRES.buffer = r;
});
I'm analysing an audio file in order to use the channelData to drive another part of my webapp (basically draw graphics based on the audio file). The callback function for the playback looks something like this:
successCallback(mediaStream) {
var audioContext = new (window.AudioContext ||
window.webkitAudioContext)();
source = audioContext.createMediaStreamSource(mediaStream);
node = audioContext.createScriptProcessor(256, 1, 1);
node.onaudioprocess = function(data) {
var monoChannel = data.inputBuffer.getChannelData(0);
..
};
Somehow I thought if I run the above code with the same file it would yield the same results all the time. But that's not the case. The same audio file would trigger the onaudioprocess function sometimes 70, sometimes 72 times for instance, yielding different data all the time.
Is there a way to get consistent data of that sort in the browser?
EDIT: I'm getting the audio from a recording function on the same page. When the recording is finished the resulting file gets set as the src of an <audio> element. recorder is my MediaRecorder.
recorder.addEventListener("dataavailable", function(e) {
fileurl = URL.createObjectURL(e.data);
document.querySelector("#localaudio").src = fileurl;
..
To answer your original question: getChannelData is deterministic, i.e. it will yield the same Float32Array from the same AudioBuffer for the same channel (unless you happen to transfer the backing ArrayBuffer to another thread, in which case it will return an empty Float32Array with a detached backing buffer from then on).
I presume the problem you are encountering here is a threading issue (my guess is that the MediaStream is already playing before you start processing the audio stream from it), but it's hard to tell exactly without debugging your complete app (there are at least 3 threads at work here: an audio processing thread for the MediaStream, an audio processing thread for the AudioContext you are using, and the main thread that runs your code).
Is there a way to get consistent data of that sort in the browser?
Yes.
Instead of processing through a real-time audio stream for real-time analysis, you could just take the recording result (e.data), read it as an ArrayBuffer, and then decode it as an AudioBuffer, something like:
recorder.addEventListener("dataavailable", function (e) {
let reader = new FileReader();
reader.onload = function (e) {
audioContext.decodeAudioData(e.target.result).then(function (audioBuffer) {
var monoChannel = audioBuffer.getChannelData(0);
// monoChannel contains the entire first channel of your recording as a Float32Array
// ...
});
};
reader.readAsArrayBuffer(e.data);
}
Note: this code would become a lot simpler with async functions and Promises, but it should give a general idea of how to read the entire completed recording.
Also note: the ScriptProcessorNode is deprecated due to performance issues inherent in cross-thread data copy, especially involving the JS main thread. The preferred alternative is the much more advanced AudioWorklet, but this is a fairly new way to do things on the web and requires a solid understanding of worklets in general.
I've been building a music app and today I finally got around to the point where I started trying to work playing the music into it.
As an outline of how my environment is set up, I am storing the music files as MP3s which I have uploaded into a MongoDB database using GridFS. I then use a socket.io server to download the chunks from the MongoDB database and send them as individual emits to the front end where the are processed by the Web Audio API and scheduled to play.
When they play, they are all in the correct order but there is this very tiny glitch or skip at the same spots every time (presumably between chunks) that I can't seem to get rid of. As far as I can tell, they are all scheduled right up next to each other so I can't find a reason why there should be any sort of gap or overlap between them. Any help would be appreciated. Here's the code:
Socket Route
socket.on('stream-audio', () => {
db.client.db("dev").collection('music.files').findOne({"metadata.songId": "3"}).then((result) =>{
const bucket = new GridFSBucket(db.client.db("dev"), {
bucketName: "music"
});
bucket.openDownloadStream(result._id).on('data',(chunk) => {
socket.emit('audio-chunk',chunk)
});
});
});
Front end
//These variable are declared as object variables, hence all of the "this" keywords
context: new (window.AudioContext || window.webkitAudioContext)(),
freeTime: null,
numChunks: 0,
chunkTracker: [],
...
this.socket.on('audio-chunk', (chunk) => {
//Keeping track of chunk decoding status so that they don't get scheduled out of order
const chunkId = this.numChunks
this.chunkTracker.push({
id: chunkId,
complete: false,
});
this.numChunks += 1;
//Callback to the decodeAudioData function
const decodeCallback = (buffer) => {
var shouldExecute = false;
const trackIndex = this.chunkTracker.map((e) => e.id).indexOf(chunkId);
//Checking if either it's the first chunk or the previous chunk has completed
if(trackIndex !== 0){
const prevChunk = this.chunkTracker.filter((e) => e.id === (chunkId-1))
if (prevChunk[0].complete) {
shouldExecute = true;
}
} else {
shouldExecute = true;
}
//THIS IS THE ACTUAL WEB AUDIO API STUFF
if (shouldExecute) {
if (this.freeTime === null) {
this.freeTime = this.context.currentTime
}
const source = this.context.createBufferSource();
source.buffer = buffer
source.connect(this.context.destination)
if (this.context.currentTime >= this.freeTime){
source.start()
this.freeTime = this.context.currentTime + buffer.duration
} else {
source.start(this.freeTime)
this.freeTime += buffer.duration
}
//Update the tracker of the chunks that this one is complete
this.chunkTracker[trackIndex] = {id: chunkId, complete: true}
} else {
//If the previous chunk hasn't processed yet, check again in 50ms
setTimeout((passBuffer) => {
decodeCallback(passBuffer)
},50,buffer);
}
}
decodeCallback.bind(this);
this.context.decodeAudioData(chunk,decodeCallback);
});
Any help would be appreciated, thanks!
As an outline of how my environment is set up, I am storing the music files as MP3s which I have uploaded into a MongoDB database using GridFS.
You can do this if you want, but these days we have tools like Minio, which can make this easier using more common APIs.
I then use a socket.io server to download the chunks from the MongoDB database and send them as individual emits to the front end
Don't go this route. There's no reason for the overhead of web sockets, or Socket.IO. A normal HTTP request would be fine.
where the are processed by the Web Audio API and scheduled to play.
You can't stream this way. The Web Audio API doesn't support useful streaming, unless you happened to have raw PCM chunks, which you don't.
As far as I can tell, they are all scheduled right up next to each other so I can't find a reason why there should be any sort of gap or overlap between them.
Lossy codecs aren't going to give you sample-accurate output. Especially with MP3, if you give it some arbitrary number of samples, you're going to end up with at least one full MP3 frame (~576 samples) output. The reality is that you need data ahead of the first audio frame for it to work properly. If you want to decode a stream, you need a stream to start with. You can't independently decode MP3 this way.
Fortunately, the solution also simplifies what you're doing. Simply return an HTTP stream from your server, and use an HTML audio element <audio> or new Audio(url). The browser will handle all the buffering. Just make sure your server handles range requests, and you're good to go.
Since some days I`m trying to visualize an audiostream which is coming over webrtc.
We already wrote some visuals which are working fine for the normal local stream (webaudio microphone usage).
Then I found some really interesting things on https://github.com/muaz-khan/WebRTC-Experiment/tree/master/ for streaming the microphone input between different browsers.
We need this to have the same audio data from one backend for all clients in the frontend.
Everything works fine and some tests showed that we can hear each other. So I thought that it is also not a problem to visualize the incoming stream.
But: all frequency data are empty (zero), even if we can hear each other.
Does anybody has a solution or hint for this? Thanks in advance!
This is my test for analysing the remote frequence data:
include these files first:
webrtc-experiment.com/firebase.js
webrtc-experiment.com/one-to-many-audio-broadcasting/meeting.js
var meeting = new Meeting('test');
var audioContext = new window.webkitAudioContext();
var analyser = audioContext.createAnalyser();
// on getting local or remote streams
meeting.onaddstream = function(e) {
console.log(e.type);
console.log(e.audio);
console.log(e.stream);
if(e.type === 'local')
{
//try it with the local stream, it works!
}
else
{
var source = audioContext.createMediaStreamSource(e.stream);
source.connect(analyser);
analyser.connect(audioContext.destination);
console.log(analyser.fftSize);
console.log(analyser.frequencyBinCount);
analyser.fftSize = 64;
console.log(analyser.frequencyBinCount);
var frequencyData = new Uint8Array(analyser.frequencyBinCount);
analyser.getByteFrequencyData(frequencyData);
function update() {
requestAnimationFrame(update);
analyser.getByteFrequencyData(frequencyData);
console.log(frequencyData);
};
update();
}
};
meeting.check();
meeting.setup('test');
Note that analysing remote streams should work for Firefox and it is known to not to work in Chrome, see http://code.google.com/p/chromium/issues/detail?id=241543
A possible workaround could be taking the remote audio level value by using WebRTC statistics API.
If you go to chrome://webrtc-internals/ then select your page playing remote stream then one of the ssrc_XXXX_recv will contain dynamically changing audioOutputLevel value which you can use.
You can access the value using Chrome PeerConnection's statistics API, specifically, getStats() method.
A possible downside may be that this is the value of actual sound the user hears from the video/audio element, so if the user mutes or changes volume of the media element, it will affect the audioOutputLevel value.
Good luck! :-)
I found a simple "solution" at least for the next time:
I have plugged a male / male audio cable from microphone to headphone in all my clients. Then I read the local microphone stream and, what a miracle, i can visualize what Im hearing.
Not a good solution, but it`s doing the job..
One Question: Is it possible to re-grab the destination as a stream in javascript? Then I would not need the audio cables..