We Keep Coding

How can i add incoming audio stream into existing audio stream while recording [duplicate]

I want to make a recording where, I get multiple audio tracks from different mediaStream objects (some of them, remote). Use the getAudioTracks () method and add them to a mediaStream object using addTrack (). At the moment of passing this last object as a parameter for mediaRecorder I realize that it only records the audio track located in position [0]. That gives me to understand that mediaRecorder is capable of recording a track by type, is there any way to join these tracks into one to record them correctly using mediaRecorder? I would be grateful for any page that explains this if possible and if it exists

I was battling with this for a while and took me ages to realise that the MediaStream only ever recorded the first track I added. My solution was to get the Web Audio API involved. This example uses two UserMedia (e.g. a mic & Stereo Mix) and merges them. The UserMedia are identified by their deviceId as shown when you use await navigator.mediaDevices.enumerateDevices().
In summary:
Create an AudioContext()
Get your media using navigator.mediaDevices.getUserMedia()
Add these as a stream source to the AudioContext
Create an AudioContext stream destination object
Connect your sources to this single destination
And your new MediaRecorder() takes this destination as its MediaStream
Now you can record yourself singing along to your favourite song as it streams ;)
const audioContext = new AudioContext();
audioParams_01 = {
deviceId: "default",
}
audioParams_02 = {
deviceId: "7079081697e1bb3596fad96a1410ef3de71d8ccffa419f4a5f75534c73dd16b5",
}
mediaStream_01 = await navigator.mediaDevices.getUserMedia({ audio: audioParams_01 });
mediaStream_02 = await navigator.mediaDevices.getUserMedia({ audio: audioParams_02 });
audioIn_01 = audioContext.createMediaStreamSource(mediaStream_01);
audioIn_02 = audioContext.createMediaStreamSource(mediaStream_02);
dest = audioContext.createMediaStreamDestination();
audioIn_01.connect(dest);
audioIn_02.connect(dest);
const recorder = new MediaRecorder(dest.stream);
chunks = [];
recorder.onstart = async (event) => {
// your code here
}
recorder.ondataavailable = (event) => {
chunks.push(event.data);
}
recorder.onstop = async (event) => {
// your code here
}

Finish using a library built by muazKhan, which allows you to merge the streams and return them in one!
It's incredibly easy!
https://github.com/muaz-khan/MultiStreamsMixer

Use live media stream as participant voice instead of getUserMedia: Twilio Video

I want to add a hidden participant in a group video call to play song stream as participant's voice(without video), with some control like whenever we want to stop or start, we can. I'm trying to pass media stream from a URL as tracks while making a connect request to join room. I'm using quickstart example for this task:
try {
// Fetch an AccessToken to join the Room.
const response = await fetch(`/token?identity=${identity}`);
// Extract the AccessToken from the Response.
const token = await response.text();
// Add the specified Room name to ConnectOptions.
connectOptions.name = roomName;
const audio = new Audio("http://mediaserv30.live-streams.nl:8086/live");
const ctx = new (window.AudioContext || window.webkitAudioContext)();
const stream_dest = ctx.createMediaStreamDestination();
const source = ctx.createMediaElementSource(audio);
source.connect(stream_dest);
const stream = stream_dest.stream;
console.log("==================", stream.getAudioTracks());
const tracks = stream.getTracks().map(track => track.kind === 'audio' ? new LocalAudioTrack(track) : new LocalVideoTrack(track));
connectOptions.tracks = tracks;
await joinRoom(token, connectOptions);
}
Here is what I'm getting after running this:
Any help is really appreciated. I'm stuck on this problem from few days.

Twilio developer evangelist here.
Your code looks correct to me. And the error message in the screenshot doesn't have anything to do with what you have written.
The suggestion in the screenshot is that you are perhaps not running the application in a secure context. In order to use WebRTC you need to do so from either localhost or a site served over HTTPS. Are you testing this on a development domain over HTTP or using a local IP address?
If you are testing this on localhost then perhaps there is another issue. Please share any error logs from the developer tools that may be relevant too.

Web Audio API - How do I save the audio buffer to a file including all changes?

I made changes to an audio buffer like gain and panning, connected them to an audio context.
Now I want to save to a file with all the implemented changes.
Saving the buffer as is would give me the original audio without the changes.
Any idea of a method or a procedure existed to do that?

On way is to use a MediaRecorder to save the modified audio.
So, in addition to connecting to the destination, connect to a MediaStreamDestinationNode. This node has a stream object that you can use to initialize a MediaRecorder. Set up the recorder to save the data when data is available. When you're down recording, you have a blob that you can then download.
Many details are missing here, but you can find out how to use a MediaRecorder using the MDN example.

I found a solution, with OfflineAudioContext.
Here is an example with adding a gain change to my audio and saving it.
On the last line of the code I get the array buffer with the changes I made.
From there, I can go on saving the file.
let offlineCtx = new OfflineAudioContext(this.bufferNode.buffer.numberOfChannels, this.bufferNode.buffer.length, this.bufferNode.buffer.sampleRate);
let obs = offlineCtx.createBufferSource();
obs.buffer = this.buffer;
let gain = offlineCtx.createGain();
gain.gain.value = this.gain.gain.value;
obs.connect(gain).connect(offlineCtx.destination);
obs.start();
let obsRES = this.ctx.createBufferSource();
await offlineCtx.startRendering().then(r => {
obsRES.buffer = r;
});

Javascript + Node - Possible to edit audio from mediastream on the server and send it back to the client?

So I'm trying to manipulate the audio which comes from the client on the server and then return the newly manipulated audio back to the client. Of course this runs into the error that Node does not support the web-audio-api so I'm using the Node version of web-audio-api along with a WebRTC library.
As I'm new to the WebRTC world I've been building on from these examples which use the WebRTC library. Using the audio-video-loopback example as the starting point I've utilised some of the libraries none standard APIs to create an audio sink that allows me to directly access the samples from the client. For now I just want to change the volume so I'm just changing the values and pushing them into a new track which is how the doc (scroll down to Programmatic Audio) says to do this. At the end I just want to return to the newly created track which is done using the .replaceTrack method (which I believe retriggers a renegotiation).
Here's what I got so far for the server code (client is the same as the original example found in the link above):
const { RTCAudioSink, RTCAudioSource } = require("wrtc").nonstandard;
function beforeOffer(peerConnection) {
const audioTransceiver = peerConnection.addTransceiver("audio");
const videoTransceiver = peerConnection.addTransceiver("video");
let { track } = audioTransceiver.receiver;
const source = new RTCAudioSource();
const newTrack = source.createTrack();
const sink = new RTCAudioSink(track);
const sampleRate = 48000;
const samples = new Int16Array(sampleRate / 100); // 10 ms of 16-bit mono audio
const dataObj = {
samples,
sampleRate,
};
const interval = setInterval(() => {
// Update audioData in some way before sending.
source.onData(dataObj);
});
sink.ondata = (data) => {
// Do something with the received audio samples.
const newArr = data.samples.map((el) => el * 0.5);
dataObj[samples] = newArr;
};
return Promise.all([
audioTransceiver.sender.replaceTrack(newTrack),
videoTransceiver.sender.replaceTrack(videoTransceiver.receiver.track),
]);
}
Not surprisingly this doesn't work, I just get silence back even though the dataObj contains the correctly manipulated samples which is then passed to the newTrack when source.onData is called.
Is what I'm trying to do even possible server side? Any suggestions are welcome, like I said I'm very green with WebRTC.

Getting number of audio channels for an AudioTrack

I have a video element, with data being added via MSE. I'm trying to determine how many audio channels there are in each track.
The AudioTrack objects themselves don't have a property with this information. The only way I know to go about it is to use the Web Audio API:
const v = document.querySelector('video');
const ctx = new OfflineAudioContext(32, 48000, 48000);
console.log(Array.from(v.audioTracks).map((track) => {
return ctx.createBufferSource(track.sourceBuffer).channelCount;
}));
For a video with a single mono track, I expect to get [1]. For a video with a single stereo track, I expect to get [2]. Yet, every time I get [2] no matter what the channel count is in the original source.
Questions:
Is there a proper direct way to get the number of channels in an AudioTrack?
Is there something else I could be doing with the Web Audio API to get the correct number of channels?

I stumbled upon an answer for this that seems to be working. It looks like by using decodeAudioData we can grab some buffer data about a file. I built a little function that returns a Promise with the buffer data that should return the correct number of channels of an audio file:
function loadBuffer(path) {
return fetch(path)
.then(response => response.arrayBuffer())
.then(
buffer =>
new Promise((resolve, reject) =>
audioContext.decodeAudioData(
buffer,
data => resolve(data),
err => reject(err)
)
)
)
}
Then you can use it like this:
loadBuffer(audioSource).then(data => console.log(data.numberOfChannels))
Might be best to store and reuse the data if it can be called multiple times.