We Keep Coding

Request/add webcam after calling navigator.mediaDevices.getUserMedia() or removing video track

I've created a basic MediaStream which gets a video & audio track in my react-app like this:
const getLocalStream: MediaStream = await navigator.mediaDevices.getUserMedia({
video: true,
audio: true,
});
setLocalStream(getLocalStream);
const handleShowCamClick = async () => {
if (!callContext.localStream) return;
callContext.localStream.getVideoTracks().forEach((track: MediaStreamTrack) => track.enabled = true);
callContext.setShowCam(true);
};
const handleHideCamClick = () => {
if (!callContext.localStream) return;
// callContext.localStream.getVideoTracks().forEach((track: MediaStreamTrack) => track.enabled = false);
callContext.localStream.getVideoTracks().forEach((track: MediaStreamTrack) => track.stop());
callContext.setShowCam(false);
};
So now I want the user to be able to disable its webcam. Setting track.enabled = false will result in the webcam still being used by the webapp but turning the video to black, which is not the behaviour I want.
Instead I want the webcam to no longer be used by the webapp.
I have a webcam with a blue light shining every time to show that the cam is recording. With track.enabled = false my webcam shows me that its still technically recording.
If I remove the video, track.stop() will result in the behaviour I want. The webcam is no longer used, but I how do I add the video track of the webcam back to the localStream?
track.stop() removes the track from localStream and frees the webcam from the MediaStream, but since the video track is not there anymore, how can I request a new video track of the webcam and attach it to localStream without re-initializing a MediaStream?

The following solution uses Vanilla Javascript because this is a problem that I'm sure a lot of folks will be interested in solving in the future.
Treat this as a proof of concept that can be adapted to ANY Javascript framework to support WebRTC tasks.
Final Note - This example only deals with the fetching, displaying, and merging of Media Streams. All other WebRTC stuff like sending the stream across via RTCPeerConnection, etc. has to be performed using the streams created here - and replacing/updating those is beyond the scope of this example.
Core Idea:
Fetch Stream via getUserMedia()
Assign Stream to HTMLMediaElement
Use getVideoTracks() to stop the video track only.
Use getUserMedia() again to fetch a new stream without audio.
Use MediaStream constructor to create a new stream using - the video from the new stream + audio from existing stream as follows -
new MediaStream([...newStream.getVideoTracks(), ...existingStream.getAudioTracks()]);
Use newly generated MediaStream as required (i.e. replace in RTCPeerConnection, etc.).
CodePen Demo
let localStream = null;
let mediaWrapperDiv = document.getElementById('mediaWrapper');
let videoFeedElem = document.createElement('video');
videoFeedElem.id = 'videoFeed';
videoFeedElem.width = 640;
videoFeedElem.height = 360;
videoFeedElem.autoplay = true;
videoFeedElem.setAttribute('playsinline', true);
mediaWrapperDiv.appendChild(videoFeedElem);
let fetchStreamBtn = document.getElementById('fetchStream');
let killEverythingBtn = document.getElementById('killSwitch');
let killOnlyVideoBtn = document.getElementById('killOnlyVideo');
let reattachVideoBtn = document.getElementById('reattachVideo');
async function fetchStreamFn() {
localStream = await navigator.mediaDevices.getUserMedia({
video: true,
audio: true
});
if (localStream) {
await attachToDOM(localStream);
}
}
async function killEverythingFn() {
localStream.getTracks().map(track => track.stop());
localStream = null;
}
async function killOnlyVideoFn() {
localStream.getVideoTracks().map(track => track.stop());
}
async function reAttachVideoFn() {
let existingStream = localStream;
let newStream = await navigator.mediaDevices.getUserMedia({
video: true,
audio: false
});
localStream = new MediaStream([...newStream.getVideoTracks(), ...existingStream.getAudioTracks()]);
if (localStream) {
await attachToDOM(localStream);
}
}
async function attachToDOM(stream) {
videoFeedElem.srcObject = new MediaStream(stream.getTracks());
}
fetchStreamBtn.addEventListener('click', fetchStreamFn);
killOnlyVideoBtn.addEventListener('click', killOnlyVideoFn);
reattachVideoBtn.addEventListener('click', reAttachVideoFn);
killEverythingBtn.addEventListener('click', killEverythingFn);
div#mediaWrapper {
margin: 0 auto;
text-align: center;
}
div#mediaWrapper video {
object-fit: cover;
}
div#mediaWrapper video#videoFeed {
border: 2px solid blue;
}
div#btnWrapper {
text-align: center;
margin-top: 10px;
}
button {
border-radius: 0.25rem;
color: #ffffff;
display: inline-block;
font-size: 1rem;
font-weight: 400;
line-height: 1.6;
padding: 0.375rem 0.75rem;
text-align: center;
cursor: pointer;
}
button.btn-blue {
background-color: #007bff;
border: 1px solid #007bff;
}
button.btn-red {
background-color: #dc3545;
border: 1px solid #dc3545;
}
button.btn-green {
background-color: #28a745;
border: 1px solid #28a745;
}
<h3>How to check if this actually works?
<h3>
<h4>Just keep speaking in an audibly loud volume, you'll hear your own audio being played from your device's speakers.<br> You should be able to hear yourself even after you "Kill Only Video" (i.e. Webcam light goes off)
</h4>
<div id="mediaWrapper"></div>
<div id="btnWrapper">
<button id="fetchStream" class="btn-blue" type="button" title="Fetch Stream (Allow Access)">Fetch Stream</button>
<button id="killOnlyVideo" class="btn-red" type="button" title="Kill Only Video">Kill Only Video</button>
<button id="reattachVideo" class="btn-green" type="button" title="Re-attach Video">Re-attach Video</button>
<button id="killSwitch" class="btn-red" type="button" title="Kill Everything">Kill Everything</button>
</div>

How can I stack a canvas exactly on top of the webcam stream?

Here is the code:
<!DOCTYPE html>
<html lang="en">
<head>
<script src="https://cdn.jsdelivr.net/npm/#tensorflow/tfjs-core"></script>
<script src="https://cdn.jsdelivr.net/npm/#tensorflow/tfjs-converter"></script>
<script src="https://cdn.jsdelivr.net/npm/#tensorflow-models/facemesh"></script>
<script src="https://cdn.jsdelivr.net/npm/#tensorflow/tfjs-backend-webgl"></script>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>Document</title>
<script>
async function get_facemesh() {
var canvas = document.getElementById("facemesh");
var draw = canvas.getContext("2d");
const stream = document.getElementById("movie");
const model = await facemesh.load((maxFaces = 1));
while (1) {
const faces = await model.estimateFaces(stream);
if (faces.length != 0) {
canvas.width = canvas.width;
var mesh = faces[0].scaledMesh;
var widthAcross = mesh[454][0] - mesh[234][0];
var heightVertical = mesh[152][1] - mesh[10][1];
console.log(heightVertical);
draw.fillStyle = "red";
for (var i = 0; i < mesh.length; i++) {
draw.fillRect(mesh[i][0], mesh[i][1], 2, 2);
}
} else {
console.log(`No faces have been detected`);
}
await tf.nextFrame();
}
}
</script>
</head>
<body>
<div class="webcam">
<video width="600" height="450" autoplay id="movie"></video>
<canvas width="600" height="450" id="facemesh"> </canvas>
</div>
<script>
video = document.getElementById("movie");
if (navigator.mediaDevices.getUserMedia) {
navigator.mediaDevices.getUserMedia({ video: true }).then(function (stream) {
video.srcObject = stream;
});
}
main();
function main() {
if (video.readyState == 4) {
console.log("video is ready for processing..");
get_facemesh();
} else {
console.log("nope, not loaded yet..");
setTimeout(main, 1000 / 30);
}
}
</script>
</body>
<style>
body {
background-color: limegreen;
margin: 0;
padding: 0;
}
video,
canvas {
-webkit-transform: scaleX(-1);
transform: scaleX(-1);
position: fixed;
margin: 0 auto;
}
.webcam {
width: 600px;
height: 450px;
background-color: limegreen;
margin: auto;
}
</style>
</html>
Here is the codepen.io link for those who like that: https://codepen.io/mdsiam8/pen/gOrPayp.
I tried using the position fixed trick, but it did not work, even though they are exactly on top of one another. It feels like the canvas is a bit stretched out or maybe is transformed to the left slightly. I tried fiddling with this but I couldn't find a fix, so I would really appreciate if any of you guys can help with this. By the way, I know that the little facemesh being offset on top of a webcam is not normal, because I have another file and used the position fixed trick, and that one is perfect, so I am not sure why it isn't the case with this one as well.

The <video>'s width and height attributes set "the dimensions of the visual content of the element" ref, this means, only how it will be presented.
Your face detector is working on the video data, at the video's intrinsic dimensions, that is, the ones defined by the media itself and not by the element, which is either stretching or shrinking the visual content when displaying it.
On the other hand, the canvas' width and height attributes do set its intrinsic dimensions.
So both media's intrinsic dimensions are actually different, and the coords that are found in the video do not match the ones of the canvas.
To fix that, you can simply set your canvas' width and height to the video.videoWidth and video.videoHeight respectively for it to match the intrinsic dimensions of your video content, and then through CSS, resize it so it matches the display dimensions of the <video> element.
However note that MediaStreams can change their dimensions while playing, so you may want to listen for the resize event of your <video>.
Link to updated codepen since StackSnippets don't allow gUM.
And the most notable changes:
const canvas = document.getElementById("facemesh");
const video = document.getElementById("movie");
if (navigator.mediaDevices.getUserMedia) {
navigator.mediaDevices.getUserMedia({ video: true })
.then( stream => {
video.srcObject = stream;
video.addEventListener( "resize", (evt) => {
canvas.width = video.videoWidth;
canvas.height = video.videoHeight;
} );
// don't poll, use events
video.addEventListener( "loadedmetadata", (evt) => {
get_facemesh();
} );
} );
}
video,
canvas {
-webkit-transform: scaleX(-1);
transform: scaleX(-1);
position: fixed;
margin: 0 auto;
/* here set the same display dimensions */
width: 600px;
height: 450px;
/* force same object fit too */
object-fit: contain;
}
<div class="webcam">
<video autoplay id="movie"></video>
<canvas id="facemesh"> </canvas>
</div>

RTCPeerConnection.ontrack event not firing

Getting my feet wet with WebRTC and running into a problem with the RTCPeerConnection.ontrack event not firing whenever a new MediaStreamTrack object has been created (by the RTCPeerConnection.addTrack() function).
Using the textbook WebRTC getUserMedia example, I grab a single stream from my laptop's camera which I set as the srcObject for one element (local) when the Start button is clicked. When the Call button is clicked, I use the addTrack method on the grabbed stream, which I hold in the global localStream variable. At this point the globally defined ontrack event handler should fire and give me the second video, right? No dice, though.
I was able to get this to work with addStream and onaddstream - but both have been removed from the latest WebRTC spec and are not supported by the most recent versions of Chrome and FireFox.
Pic and script attached - any guidance would be greatly appreciated!
Pic with Web Console output
'use strict';
var localStream;
var yourVideo = document.querySelector('#yours');
var theirVideo = document.querySelector('#theirs');
var callBtn = document.querySelector('#callBtn');
var startBtn = document.querySelector('#startBtn');
startBtn.onclick = hasUserMedia;
callBtn.onclick = call;
var cfg = null;
var pc1 = new RTCPeerConnection(cfg);
var pc2 = new RTCPeerConnection(cfg);
pc1.ontrack = function(e){
console.log("ontrack fired!");
theirVideo.srcObject = e.streams[0];
}
function hasUserMedia(){
console.log("entering hasUserMedia()...");
navigator.mediaDevices.getUserMedia({video: true, audio: false}).then(function(stream){
localStream = stream;
console.log("stream val: " + localStream);
yourVideo.srcObject = stream;
});
}
function call(){
console.log("stream val # call(): " + localStream);
localStream.getTracks().forEach(track => pc1.addTrack(track, localStream));
}
index.html
<html lang="en">
<head>
<meta charset="utf-8" />
<title>Learning WebRTC - Chapter 4: Creating a
RTCPeerConnection</title>
<style>
body {
background-color: #3D6DF2;
margin-top: 15px;
}
video {
background: black;
border: 1px solid gray;
}
#container {
position: relative;
display: block;
margin: 0 auto;
width: 500px;
height: 500px;
}
#yours {
width: 150px;
height: 150px;
position: absolute;
top: 15px;
right: 15px;
}
#theirs {
width: 500px;
height: 500px;
}
</style>
</head>
<body>
<div id="container">
<video id="yours" autoplay></video>
<video id="theirs" autoplay></video>
<button id="startBtn">Start</button>
<button id="callBtn">Call</button>
</div>
<script src="https://webrtc.github.io/adapter/adapter-latest.js"></script>
<script type="text/javascript" src="main.js"></script>
</body>
</html>

Both Firefox and Chrome still support onaddstream. Chrome does not yet support addTrack, you can still use addStream (despite Firefox complaining).
Your script is missing the exchange of SDP and ICE candidates for ontrack to fire as well.
Check https://webrtc.github.io/samples/src/content/peerconnection/pc1/ for a complete and working example.

addTrack on Chromium Issue Tracker
addTrack is blocked by RTCRtpSender and RTCRtpReceiver extensions to RTCPeerConnection

I got into the same problem while trying to implement a video chat, but I found a fix. The fix is simple; make sure to add the ontrack event, onicecandidate and the video stream object at the creation of the rtcpeerconnection. See the code below
var configuration = {
offerToReceiveAudio: true,
offerToReceiveVideo: true
}
var peerConn = new RTCPeerConnection(cofiguration);
peerConn.ontrack = gotStream;
peerConn.onicecandidate = gotIceCandidate;
peerConn.addStream(localStream);
//and do the rest your style... Hope this helps.

youtube iframe player doesn't automatically switch quality in low bandwidth on mobile

Using iOS 9.3, link conditioning set to 3g speed, loading a video in the youtube iframe api in safari.
I would expect the iframe api to realize that it has been buffering a bunch and try to get a lower quality stream to keep the video playback smooth, like it does in the native youtube app.
Am I missing something obvious? I basically copied and pasted out of the youtube ios helper wrapper but it still tries to play in a quality that is too high for the connection speed.
<!DOCTYPE html>
<html>
<head>
<style>
body { margin: 0; width:100%; height:100%; background-color:#000000; }
html { width:100%; height:100%; background-color:#000000; }
.embed-container iframe,
.embed-container object,
.embed-container embed {
position: absolute;
top: 0;
left: 0;
width: 100% !important;
height: 100% !important;
}
</style>
</head>
<body>
<div class="embed-container">
<div id="player"></div>
</div>
<script src="https://www.youtube.com/iframe_api" onerror="window.location.href='ytplayer://onYouTubeIframeAPIFailedToLoad'"></script>
<script>
var player;
var error = false;
YT.ready(function() {
player = new YT.Player('player', {
"events" : {
"onPlaybackQualityChange" : "onPlaybackQualityChange",
"onReady" : "onReady",
"onError" : "onPlayerError",
"onStateChange" : "onStateChange"
},
"width" : "100%",
"height" : "100%",
"videoId" : 'NP7nK2zPirc',
"playerVars" : {
"showinfo" : 0,
"modestbranding" : 1,
"autohide" : 1,
"playsinline" : 1,
"controls" : 0
}
});
player.setSize(window.innerWidth, window.innerHeight);
window.location.href = 'ytplayer://onYouTubeIframeAPIReady';
// this will transmit playTime frequently while playng
function getCurrentTime() {
var state = player.getPlayerState();
if (state == YT.PlayerState.PLAYING) {
time = player.getCurrentTime()
// window.location.href = 'ytplayer://onPlayTime?data=' + time;
}
}
window.setInterval(getCurrentTime, 500);
});
function onReady(event) {
// window.location.href = 'ytplayer://onReady?data=' + event.data;
}
function onStateChange(event) {
if (!error) {
// window.location.href = 'ytplayer://onStateChange?data=' + event.data;
}
else {
error = false;
}
}
function onPlaybackQualityChange(event) {
// window.location.href = 'ytplayer://onPlaybackQualityChange?data=' + event.data;
}
function onPlayerError(event) {
if (event.data == 100) {
error = true;
}
// window.location.href = 'ytplayer://onError?data=' + event.data;
}
window.onresize = function() {
player.setSize(window.innerWidth, window.innerHeight);
}
</script>
</body>
</html>

Based from this documentation, if the playback quality changes, the onPlaybackQualityChange event will fire, and your code should respond to the event rather than the fact that it called the setPlaybackQuality function.
onPlaybackQualityChange event fires whenever the video playback quality changes. For example, if you call the setPlaybackQuality(suggestedQuality) function, this event will fire if the playback quality actually changes. Your application should respond to the event and should not assume that the quality will automatically change when the setPlaybackQuality(suggestedQuality) function is called. Similarly, your code should not assume that playback quality will only change as a result of an explicit call to setPlaybackQuality or any other function that allows you to set a suggested playback quality.
Then I recommend calling the getAvailableQualityLevels() function to determine which quality levels are available for a video. For example, if your page displays a 1280px by 720px video player, a hd720 quality video will actually look better than an hd1080 quality video.
Check these related SO threads:
YouTube iFrame API "setPlaybackQuality" or "suggestedQuality" not working
If you use event 3 (buffering) instead of event 5 (playing) there's no
stutter for the user. Quality is changed as soon as it starts loading.
Only weird thing is you need to set it in onPlayerReady as well or it
doesn't work.
function onPlayerReady(event) {
event.target.setPlaybackQuality('hd720');
}
function onPlayerStateChange(event) {
if (event.data == YT.PlayerState.BUFFERING) {
event.target.setPlaybackQuality('hd720');
}
}
Set jw player youtube video playback quality
Hope this helps!

Web Audio API, MediaElementAudioSourceNode, Safari

I am trying to build a sound player based on Web Audio API. I have an issue with Safari that to me looks like a grave Safari bug, so I don't know if anything can be done about it.
Basically I need to stream an mp3 file into the Web Audio API graph. This goes via createMediaElementSource. The problem in Safari (9.1.1) is that everytime one calls play() on the underlying audio element, the initial audio buffer is corrupted, giving a brief "hiccup". Depending on the complexity of the actual Web Audio graph, these drop outs can last up to several hundred milliseconds. Firefox and Chrome don't have this problem.
The following illustrates the problem. Click on "Load" to set the audio element's src. Wait. Then Play, Stop, RTZ (return-to-zero) repeatedly -- you should hear an undistorted sine tone, but you actually hear that the initial buffer upon play is corrupted.
JSFiddle: http://jsfiddle.net/4zgf33os/6/
HTML:
<div id="load" class="button">Load</div>
<div id="play" class="button">Play</div>
<div id="stop" class="button">Stop</div>
<div id="rtz" class="button">RTZ</div>
CSS:
div.button {
position: relative;
display: inline;
width: 100px;
height: 24px;
margin: 4px;
padding: 4px;
background: black;
color: white;
cursor: pointer;
}
JavaScript:
window.AudioContext = window.AudioContext || window.webkitAudioContext;
var elem = document.createElement("audio");
elem.setAttribute("crossorigin", "anonymous");
var context = new window.AudioContext;
var node = context.createMediaElementSource(elem);
node.connect(context.destination);
elem.preload = "auto";
elem.addEventListener("error", function(e) {
alert("Error!");
});
var load = function() {
elem.src = "https://ia800209.us.archive.org/12/items/SineWaveStudiosSineWaveTestFile/sinetest.mp3";
};
var rtz = function() {
elem.pause();
elem.currentTime = 0.0;
};
var play = function() {
elem.play();
};
var stop = function() {
elem.pause();
};
$("#load").click(load);
$("#rtz" ).click(rtz );
$("#play").click(play);
$("#stop").click(stop);