I'm looking to get the microphone activity level of a WebRTC MediaStream. However, I need to get this information without playing back the microphone to the user (otherwise there will be the loopback effect).
The answer in Microphone activity level of WebRTC MediaStream relies on the audio being played back to the user. How can I do this, without playing back the microphone?
Take a look at createGain method. It allows you to set stream's volume.
Here is my (simplified) example that I use in my project:
navigator.getUserMedia({audio: true, video: true}, function(stream) {
var audioContext = new AudioContext; //or webkitAudioContext
var source = audioContext.createMediaStreamSource(stream);
var volume = audioContext.createGain();
source.connect(volume);
volume.connect(audioContext.destination);
volume.gain.value = 0; //turn off the speakers
//further manipulations with source
}, function(err) {
console.log('error', err);
});
Related
I have working on streaming live video using WebRTC based on RTCConnection with library called simple-peer, but I have faced with some lag between live stream video (with MediaRecorder) and that was played on using MediaSource
Here is recorder:
var mediaRecorder = new MediaRecorder(stream, options);
mediaRecorder.ondataavailable = handleDataAvailable;
function handleDataAvailable(event) {
if (connected && event.data.size > 0) {
peer.send(event.data);
}
}
...
peer.on('connect', () => {
// wait for 'connect' event before using the data channel
mediaRecorder.start(1);
});
Here is source that is played:
var mediaSource = new MediaSource();
var sourceBuffer;
mediaSource.addEventListener('sourceopen', args => {
sourceBuffer = mediaSource.addSourceBuffer(mimeCodec);
});
...
peer.on('data', data => {
// got a data channel message
sourceBuffer.appendBuffer(data);
});
I open two tabs and connect to myself and I see delay in playing video ...
Seems like I configured badly MediaRecorder or MediaSource
Any help will be appreciated ;)
You've combined two completely unrelated techniques for streaming the video, and are getting the worst tradeoffs of both. :-)
WebRTC has media stream handling built into it. If you expect realtime video, the WebRTC stack is what you want to use. It handles codec negotiation, auto-scales bandwidth, frame size, frame rate, and encoding parameters to match network conditions, and will outright drop chunks of time to keep playback as realtime as possible.
On the other hand, if retaining quality is more desirable than being realtime, MediaRecorder is what you would use. It makes no adjustments based on network conditions because it is unaware of those conditions. MediaRecorder doesn't know or care where you put the data after it gives you the buffers.
If you try to play back video as it's being recorded, will inevitably lag further and further behind because there is no built-in catch-up method. The only thing that can happen is a buffer underrun, where the playback side waits until there is enough data to begin playback again. Even if it becomes minutes behind, it isn't going to automatically skip ahead.
The solution is to use the right tool. It sounds like from your question that you want realtime video. Therefore, you need to use WebRTC. Fortunately simple-peer makes this... simple.
On the recording side:
const peer = new Peer({
initiator: true,
stream
});
Then on the playback side:
peer.on('stream', (stream) => {
videoEl.srcObject = stream;
});
Much simpler. The WebRTC stack handles everything for you.
I'm developing a website where the user can send audio commands which are captured with getUserMedia (only audio) and interpreted in the backend with a Speech-to-Text service. In order to keep the latency as low as possible, I'm sending small audio chunks to my server. This is working just fine on Chrome/Firefox and even Edge. However, I'm struggling with iOS Safari. I know that Safari is my only choice on Apple devices because of the missing WebRTC support on iOS Chrome/Firefox.
The problem is that I normally get the user's voice a couple of times (for some commands). But without any pattern the stream then suddenly contains only empty bytes. I tried a lot of different strategies but in general I stuck to the following plan:
After user clicks a button, call getUserMedia (with audio constraint) and save stream to a variable
Create AudioContext (incl. Gain, MediaStreamSource, ScriptProcess) and connect the audio stream to the MediaStreamSource
Register an event listener to the ScriptProcessor and send audio chunks in callback to the server
When a result is returned from the server close AudioContext and audio's MediaStream
The interesting part is now what happens after a subsequent user command. I tried various things: Call getUserMedia again for each call and close the MediaStream track each time, use the initially created MediaStream and reconnect the EventHandler every time, close the AudioContext after every call, use only one initially created AudioContext... All my attempts failed so far, because I either got empty bytes from the Stream or the AudioContext was created in a "suspended" state. Only closing MediaStream/AudioContext and creating it every time again seems to be more stable, but fetching the MediaStream with getUserMedia takes quite a while on iOS (~1,5-2s), which gives a bad user experience.
I'll show you my latest attempt where I tried to mute/disable the stream in between user commands and keep the AudioContext open:
var audioStream: MediaStream;
var audioContext: AudioContext;
var startButton = document.getElementById("startButton");
startButton.onclick = () => {
if (!audioStream) {
getUserAudioStream();
} else {
// mute/disable stream
audioStream.getAudioTracks()[0].enabled = true;
}
}
var stopButton = document.getElementById("stopButton");
stopButton.onclick = () => {
// unmute/enable stream
audioStream.getAudioTracks()[0].enabled = false;
}
function getUserAudioStream(): Promise<any> {
return navigator.mediaDevices.getUserMedia({
audio: true
} as MediaTrackConstraints,
}).then((stream: MediaStream) => {
audioStream = stream;
startRecording();
}).catch((e) => { ... });
}
const startRecording = () => {
const ctx = (window as any).AudioContext || (window as any).webkitAudioContext;
if (!ctx) {
console.error("No Audio Context available in browser.");
return;
} else {
audioContext = new ctx();
}
const inputPoint = audioContext.createGain();
const microphone = audioContext.createMediaStreamSource(audioStream);
scriptProcessor = inputPoint.context.createScriptProcessor(4096, 1, 1);
microphone.connect(inputPoint);
inputPoint.connect(scriptProcessor);
scriptProcessor.connect(inputPoint.context.destination);
scriptProcessor.addEventListener("audioprocess", streamCallback);
};
const streamCallback = (e) => {
const samples = e.inputBuffer.getChannelData(0);
// Here I stream audio chunks to the server and
// observe that buffer sometimes only contains empty bytes...
}
I hope the snippet makes sense to you, because I let some stuff out to keep it readable. I think I made clear that this is only one of many attempts and actually my question is: Is there some kind of special characteristic in WebRTC/getUserMedia on iOS that I missed so far? Why does iOS treat MediaStream differently than Chrome/Firefox on Windows? As a last comment: I know that the ScriptProcessorNode is no longer recommended. Actually, I'd like to use MediaRecorder for that but this is also not yet supported on iOS. Also, the polyfill I know is not really suitable because it only support ogg for streaming audio and which also leads to problems because I would need to set the sample rate for that to a fixed value.
I am trying to do some audio analysis for a visualizer running on my computer.
Is is possible to access the output audio data stream directly from the browser?
Currently running JavaScript with the three.js and meyda libraries.
I've figured out how to use the webAudio API to analyze input from the microphone, but can't seem to gain access to the audio output on my computer.
I've tried to connect source to the destination using
source.connect(audioContext.destination)
but this doesn't seem to do anything.
This is our current listener config:
// // Listener
const bufferSize = 256;
let analyzer;
// The navigator object contains information about the browser.
// this async call initializes audio input from the user
navigator.mediaDevices.getUserMedia({ audio: true, video: false }).then(stream => {
if (!analyzer) initAnalyzer(stream)
})
function initAnalyzer(stream) {
const audioContext = new AudioContext();
// set audio source to input stream from microphone (Web Audio API https://developer.mozilla.org/en-US/docs/Web/API/MediaStreamAudioSourceNode)
const source = audioContext.createMediaStreamSource(stream);
analyzer = Meyda.createMeydaAnalyzer({
audioContext: audioContext,
source: source,
bufferSize: bufferSize,
featureExtractors: [ 'amplitudeSpectrum', 'spectralFlatness' ], // ["rms", "energy"],
callback: features => null
});
analyzer.start();
}
It's not possible to grab the audio from the computer without external software like Audio Hijack. Sorry!
audiooutputs cannot be accessed for privacy, only audioinputs (and only after confirming with the user). On Windows you can enable "stereo mix" that routes all outputs to a virtual input, and you can use that, but it requires all users to have stereo mix enabled...
The visualizers you see are using the buffer or source that they created so of course they have access to it.
I see a lot of questions for how to record audio then stop recording, then play audio or save it to a file, but none of this is what I want.
tl;dr Here's my question in a nutshell: "How can I immediately play audio recorded from the user's microphone?" That is, I don't want to save a recording and play it when the user hits a "Play" button, I don't want to save a recording to a file on the user's computer and I don't want to use WebRTC to stream audio anywhere. I just want to talk into my microphone and hear my voice come out the speakers.
All I'm trying to do is make a very simple "echo" page that just immediately plays back audio recorded from the mic. I started using a mediaRecorder object, but that wasn't working and from what I can tell that's meant for recording full audio files, so I switched to an AudioContext-based approach.
A very simple page would just look like this:
<!DOCTYPE html>
<head>
<script type="text/javascript" src="mcve.js"></script>
</head>
<body>
<audio id="speaker" volume="1.0"></audio>
</body>
and the script looks like this:
if (navigator.mediaDevices) {
var constrains = {audio: true};
navigator.mediaDevices.getUserMedia(constrains).then(
function (stream) {
var context = new AudioContext();
var source = context.createMediaStreamSource(stream);
var proc = context.createScriptProcessor(2048, 2, 2);
source.connect(proc);
proc.onaudioprocess = function(e) {
console.log("audio data collected");
let audioData = new Blob(e.inputBuffer.getChannelData(0), {type: 'audio/ogg' } )
|| new Blob(new Float32Array(2048), {type: 'audio/ogg'});
var speaker = document.getElementById('speaker');
let url = URL.createObjectURL(audioData);
speaker.src = url;
speaker.load();
speaker.play().then(
() => { console.log("Playback success!"); },
(error) => { console.log("Playback failure... ", error); }
);
};
}
).catch( (error) => {
console.error("couldn't get user media.");
});
}
It can record non-trivial audio data (i.e. not every collection winds up as a Blob made from the new Float32Array(2048) call), but it can't play it back. It never hits the "could not get user media" catch, but it always hits the "Playback Failure..." catch. The error prints like this:
DOMException [NotSupportedError: "The media resource indicated by the src attribute or assigned media provider object was not suitable."
code: 9
nsresult: 0x806e0003]
Additionally, the message Media resource blob:null/<long uuid> could not be decoded. is printed to the console repeatedly.
There are two things that could be going on here, near as I can tell (maybe both):
I'm not encoding the audio. I'm not sure if this is a problem, since I thought that data collected from the mic came with 'ogg' encoding automagically, and I've tried leaving the type property of my Blobs blank to no avail. If this is what's wrong, I don't know how to encode a chunk of audio given to me by the audioprocess event, and that's what I need to know.
An <audio> element is fundamentally incapable of playing audio fragments, even if properly encoded. Maybe by not having a full file, there's some missing or extraneous metadata that violates encoding standards and is preventing the browser from understanding me. If this is the case, maybe I need a different element, or even an entirely scripted solution. Or perhaps I'm supposed to construct a file-like object in-place for each chunk of audio data?
I've built this code on examples from MDN and SO answers, and I should mention I've tested my mic at this example demo and it appears to work perfectly.
The ultimate goal here is to stream this audio through a websocket to a server and relay it to other users. I DON'T want to use WebRTC if at all possible, because I don't want to limit myself to only web clients - once it's working okay, I'll make a desktop client as well.
Check example https://jsfiddle.net/greggman/g88v7p8c/ from https://stackoverflow.com/a/38280110/351900
Required to be run from HTTPS
navigator.getUserMedia = navigator.getUserMedia ||navigator.webkitGetUserMedia || navigator.mozGetUserMedia;
var aCtx;
var analyser;
var microphone;
if (navigator.getUserMedia) {
navigator.getUserMedia(
{audio: true},
function(stream) {
aCtx = new AudioContext();
microphone = aCtx.createMediaStreamSource(stream);
var destination=aCtx.destination;
microphone.connect(destination);
},
function(){ console.log("Error 003.")}
);
}
In recent days, I tried to use javascript to record audio stream.
I found that there is no example code which works.
Is there any browser supporting?
Here is my code
navigator.getUserMedia = navigator.getUserMedia || navigator.webkitGetUserMedia ||
navigator.mozGetUserMedia || navigator.msGetUserMedia;
navigator.getUserMedia({ audio: true }, gotStream, null);
function gotStream(stream) {
msgStream = stream;
msgStreamRecorder = stream.record(); // no method record :(
}
getUserMedia gives you access to the device, but it is up to you to record the audio. To do that, you'll want to 'listen' to the device, building a buffer of the data. Then when you stop listening to the device, you can format that data as a WAV file (or any other format). Once formatted you can upload it to your server, S3, or play it directly in the browser.
To listen to the data in a way that is useful for building your buffer, you will need a ScriptProcessorNode. A ScriptProcessorNode basically sits between the input (microphone) and the output (speakers), and gives you a chance to manipulate the audio data as it streams. Unfortunately the implementation is not straightforward.
You'll need:
getUserMedia to access the device
AudioContext to create a MediaStreamAudioSourceNode and a ScriptProcessorNode
MediaStreamAudioSourceNode to represent the audio stream
ScriptProcessorNode to get access to the streaming audio data via an onaudioprocessevent. The event exposes the channel data that you'll build your buffer with.
Putting it all together:
navigator.getUserMedia({audio: true},
function(stream) {
// create the MediaStreamAudioSourceNode
var context = new AudioContext();
var source = context.createMediaStreamSource(stream);
var recLength = 0,
recBuffersL = [],
recBuffersR = [];
// create a ScriptProcessorNode
if(!context.createScriptProcessor){
node = context.createJavaScriptNode(4096, 2, 2);
} else {
node = context.createScriptProcessor(4096, 2, 2);
}
// listen to the audio data, and record into the buffer
node.onaudioprocess = function(e){
recBuffersL.push(e.inputBuffer.getChannelData(0));
recBuffersR.push(e.inputBuffer.getChannelData(1));
recLength += e.inputBuffer.getChannelData(0).length;
}
// connect the ScriptProcessorNode with the input audio
source.connect(node);
// if the ScriptProcessorNode is not connected to an output the "onaudioprocess" event is not triggered in chrome
node.connect(context.destination);
},
function(e) {
// do something about errors
});
Rather than building all of this yourself I suggest you use the AudioRecorder code, which is awesome. It also handles writing the buffer to a WAV file. Here is a demo.
Here's another great resource.
for browsers that support MediaRecorder API, use it.
for older browsers that does not support MediaRecorder API, there are three ways to do it
as wav
all code client-side.
uncompressed recording.
source code --> http://github.com/mattdiamond/Recorderjs
as mp3
all code client-side.
compressed recording.
source code --> http://github.com/Mido22/mp3Recorder
as opus packets (can get output as wav, mp3 or ogg)
client and server(node.js) code.
compressed recording.
source code --> http://github.com/Mido22/recordOpus
You could check this site:
https://webaudiodemos.appspot.com/AudioRecorder/index.html
It stores the audio into a file (.wav) on the client side.
There is a bug that currently does not allow audio only. Please see http://code.google.com/p/chromium/issues/detail?id=112367
Currently, this is not possible without sending the data over to the server side. However, this would soon become possible in the browser if they start supporting the MediaRecorder working draft.