I have an html canvas that was created in p5, and I would like to add an audio track to it so that I can stream it with a webrtc connection. I currently can stream the visuals but not the audio.
I am adding the audio stream to my canvas as follows:
let canvasSource = document.getElementById('canvas-viz');
navigator.mediaDevices.getUserMedia({
audio: true
}).then(audioStream => {
audioStream.getAudioTracks().forEach(
track => {
canvasSource.captureStream().addTrack(track)
})
console.log("canv source: ",canvasSource.captureStream().getAudioTracks()); // prints []
})
So my main problem here is that when I call canvasSource.captureStream().getAudioTracks() I get []. So it seems that addTrack isn't properly working. I tried calling canvasSource.captureStream().getAudioTracks() in the dev tools in case there was some asynchronous tomfoolery happening and also go []. I also tried the following in the dev tools:
audioTracks = audioStream.getAudioTracks();
canvasSource.captureStream().addTrack(audioTracks[0]);
But this also didn't work, returning [] when looking at getAudioTracks(). When calling audioStream.getAudioTracks(), I get an array of size 1 with my microphone input stream.
I was following methods shown in:
how to add a audio stream on canvas stream in webrtc
I am developing this in Chrome. For my purposes, at the time being, it doesn't need to be cross compatible in Firefox.
canvasSource.captureStream() returns a new MediaStream at each call. You have added your audiotrack to a MediaStream you can't access anymore.
Store the canvas MediaStream in a variable accessible there and add the track to that MediaStream.
Kaiido's solution should work. As an alternative, what I do is simply create a new MediaStream with the tracks from the other two streams...
new MediaStream([
canvasVideoTrack,
audioTrack
]);
Related
the title is indicating what I want, I just want to record audio as buffer and play that byte with only javascript at the same time (not node js) .
I researched about that for a while, and finally, I tried this method, and works for recording
const handleSuccess = function(stream) {
const context = new AudioContext();
const source = context.createMediaStreamSource(stream);
const processor = context.createScriptProcessor(1024*4, 1, 1);
source.connect(processor);
processor.connect(context.destination);
processor.onaudioprocess = function(e) {
console.log(e);
};
};
navigator.mediaDevices.getUserMedia({ audio: true, video: false })
.then(handleSuccess);
now the problem is playing this e
I tried decoding it with context.decodeAudioData(e.inputBuffer.getChannelData(0).buffer) but it throws an error :
DOMException: Failed to execute 'decodeAudioData' on 'BaseAudioContext': Unable to decode audio data
why it's not working? e.inputBuffer.getChannelData(0).buffer returns an object of type ArrayBuffer which is exactly what decodeAudioData wants, I also ensured that the output array is not empty.
Please help me to solve this problem.
Thank you
The audioprocess event of a ScriptProcessorNode gives you access to the current audio as an AudioBuffer. If you want to play that AudioBuffer you can use it as is and don't need to decode it anymore. It's already decoded. It can be played with an AudioBufferSourceNode.
The ScriptProcessorNode is officially deprecated but I guess it will never go away. Its successor - the AudioWorkletProcessor - gives you access to the raw channel data.
I made changes to an audio buffer like gain and panning, connected them to an audio context.
Now I want to save to a file with all the implemented changes.
Saving the buffer as is would give me the original audio without the changes.
Any idea of a method or a procedure existed to do that?
On way is to use a MediaRecorder to save the modified audio.
So, in addition to connecting to the destination, connect to a MediaStreamDestinationNode. This node has a stream object that you can use to initialize a MediaRecorder. Set up the recorder to save the data when data is available. When you're down recording, you have a blob that you can then download.
Many details are missing here, but you can find out how to use a MediaRecorder using the MDN example.
I found a solution, with OfflineAudioContext.
Here is an example with adding a gain change to my audio and saving it.
On the last line of the code I get the array buffer with the changes I made.
From there, I can go on saving the file.
let offlineCtx = new OfflineAudioContext(this.bufferNode.buffer.numberOfChannels, this.bufferNode.buffer.length, this.bufferNode.buffer.sampleRate);
let obs = offlineCtx.createBufferSource();
obs.buffer = this.buffer;
let gain = offlineCtx.createGain();
gain.gain.value = this.gain.gain.value;
obs.connect(gain).connect(offlineCtx.destination);
obs.start();
let obsRES = this.ctx.createBufferSource();
await offlineCtx.startRendering().then(r => {
obsRES.buffer = r;
});
I have working on streaming live video using WebRTC based on RTCConnection with library called simple-peer, but I have faced with some lag between live stream video (with MediaRecorder) and that was played on using MediaSource
Here is recorder:
var mediaRecorder = new MediaRecorder(stream, options);
mediaRecorder.ondataavailable = handleDataAvailable;
function handleDataAvailable(event) {
if (connected && event.data.size > 0) {
peer.send(event.data);
}
}
...
peer.on('connect', () => {
// wait for 'connect' event before using the data channel
mediaRecorder.start(1);
});
Here is source that is played:
var mediaSource = new MediaSource();
var sourceBuffer;
mediaSource.addEventListener('sourceopen', args => {
sourceBuffer = mediaSource.addSourceBuffer(mimeCodec);
});
...
peer.on('data', data => {
// got a data channel message
sourceBuffer.appendBuffer(data);
});
I open two tabs and connect to myself and I see delay in playing video ...
Seems like I configured badly MediaRecorder or MediaSource
Any help will be appreciated ;)
You've combined two completely unrelated techniques for streaming the video, and are getting the worst tradeoffs of both. :-)
WebRTC has media stream handling built into it. If you expect realtime video, the WebRTC stack is what you want to use. It handles codec negotiation, auto-scales bandwidth, frame size, frame rate, and encoding parameters to match network conditions, and will outright drop chunks of time to keep playback as realtime as possible.
On the other hand, if retaining quality is more desirable than being realtime, MediaRecorder is what you would use. It makes no adjustments based on network conditions because it is unaware of those conditions. MediaRecorder doesn't know or care where you put the data after it gives you the buffers.
If you try to play back video as it's being recorded, will inevitably lag further and further behind because there is no built-in catch-up method. The only thing that can happen is a buffer underrun, where the playback side waits until there is enough data to begin playback again. Even if it becomes minutes behind, it isn't going to automatically skip ahead.
The solution is to use the right tool. It sounds like from your question that you want realtime video. Therefore, you need to use WebRTC. Fortunately simple-peer makes this... simple.
On the recording side:
const peer = new Peer({
initiator: true,
stream
});
Then on the playback side:
peer.on('stream', (stream) => {
videoEl.srcObject = stream;
});
Much simpler. The WebRTC stack handles everything for you.
So, I am working on a small electron desktop app that captures desktop screen and records video and audio.
When I am trying to add audio to the stream it starts echoing really badly and I am not sure why.
I am using:
Windows 10 PRO 18362.778
Chrome 81.0.4044.113
Electron 8.2.3
Here is some code.
I create these constraints when I want to capture and record video only:
const constraints = {
audio: false,
video: {
mandatory: {
chromeMediaSource: 'desktop',
chromeMediaSourceId: source.id
}
}
}
Then I pass is to the stream like that:
const stream = await navigator.mediaDevices.getUserMedia(constraints)
It works like a charm. However when I start adding audio it gives me echo:
const constraints = {
audio: {
mandatory: {
chromeMediaSource: 'desktop',
}
},
video: {
mandatory: {
chromeMediaSource: 'desktop',
}
}
}
Also, I can't just set the audio to true. It then gives me this error:
Uncaught (in promise) DOMException: Error starting screen capture
An interesting fact. When I go to Mozilla documentation page on audio constraints and use the demo button it gives me echo too. I tried doing it on Edge and the result was better, but still had echo. So can it be the audio codec?
Here it says that echoCancellation constraint is supported and on by default starting Chrome version 62.
Here is the branch on the Github where I tried to find solution, but failed.
Here is my git repo if you want to look at it more closely.
PS: this is my first post here. Let me know if I did something wrong here and can improve the post. Thank you!
The problem comes from attempting to initiate a single stream of your microphone audio and computer screen video at the same time. To fix this issue, you will need to create an audio stream first, then separately create a video stream that captures your computer screen, and finally, combine the streams into one.
// create audio and video constraints
const constraintsVideo = {
audio: false,
video: {
mandatory: {
chromeMediaSource: 'desktop',
chromeMediaSourceId: source.id
}
}
}
const constraintsAudio = {audio: true}
// create audio and video streams separately
const audioStream = await navigator.mediaDevices.getUserMedia(constraintsAudio)
const videoStream = await navigator.mediaDevices.getUserMedia(constraintsVideo)
// combine the streams
const combinedStream = new MediaStream([...videoStream.getVideoTracks(), ...audioStream.getAudioTracks()])
After you combine the streams, you can use combinedStream as if it originated as a single stream.
I think the easy fix here would be to add a muted element to the playback on your page.
// Preview the source in a video element
videoElement.srcObject = stream
videoElement.muted = true
videoElement.play()
That will work in all browsers and you'll still record the audio.
A very late answer I know, but I was trying to understand your issue, because I'm currently dealing with echo cancelling issues via WebRTC in regards to desktop capturing and I was wondering why your initial approach (adding chromeMediaSource: 'desktop' as a mandatory constraint to video and audio) causes echo, because this is the correct usage, as described in the Electron docs. I don't think that post which is marked as the correct answer, really solves the problem. Instead the second answer is correct in this case. The problem is not related to the usage of the constraints / streams. The reason you are hearing echo is, that you are adding the captured audio to your speaker output implicitly again, by assigning it to the <video> element. This creates the echoing loop. To avoid this, the correct solution is to mute the <video> element like you did in your repo. I tried your code and recording works fine (also with the muted <video> element), when re-enabling the code section which captures video/audio at once (via the constraint mentioned above).
I am trying to do some audio analysis for a visualizer running on my computer.
Is is possible to access the output audio data stream directly from the browser?
Currently running JavaScript with the three.js and meyda libraries.
I've figured out how to use the webAudio API to analyze input from the microphone, but can't seem to gain access to the audio output on my computer.
I've tried to connect source to the destination using
source.connect(audioContext.destination)
but this doesn't seem to do anything.
This is our current listener config:
// // Listener
const bufferSize = 256;
let analyzer;
// The navigator object contains information about the browser.
// this async call initializes audio input from the user
navigator.mediaDevices.getUserMedia({ audio: true, video: false }).then(stream => {
if (!analyzer) initAnalyzer(stream)
})
function initAnalyzer(stream) {
const audioContext = new AudioContext();
// set audio source to input stream from microphone (Web Audio API https://developer.mozilla.org/en-US/docs/Web/API/MediaStreamAudioSourceNode)
const source = audioContext.createMediaStreamSource(stream);
analyzer = Meyda.createMeydaAnalyzer({
audioContext: audioContext,
source: source,
bufferSize: bufferSize,
featureExtractors: [ 'amplitudeSpectrum', 'spectralFlatness' ], // ["rms", "energy"],
callback: features => null
});
analyzer.start();
}
It's not possible to grab the audio from the computer without external software like Audio Hijack. Sorry!
audiooutputs cannot be accessed for privacy, only audioinputs (and only after confirming with the user). On Windows you can enable "stereo mix" that routes all outputs to a virtual input, and you can use that, but it requires all users to have stereo mix enabled...
The visualizers you see are using the buffer or source that they created so of course they have access to it.