We Keep Coding

How to use RTCPeerConnection.removeTrack() to remove video or audio or both?

I'm studying WebRTC and try to figure how it works.
I modified this sample on WebRTC.github.io to make getUserMedia source of leftVideo and streaming to rightVideo.It works.
And I want to add some feature, like when I press pause on leftVideo(My browser is Chrome 69)
I change apart of Call()
...
stream.getTracks().forEach(track => {
pc1Senders.push(pc1.addTrack(track, stream));
});
...
And add function on leftVideo
leftVideo.onpause = () => {
pc1Senders.map(sender => pc1.removeTrack(sender));
}
I don't want to close the connection, I just want to turn off only video or audio.
But after I pause leftVideo, the rightVideo still gets track.
Am I doing wrong here, or maybe other place?
Thanks for your helping.

First, you need to get the stream of the peer. You can mute/hide the stream using the enabled attribute of the MediaStreamTrack. Use the below code snippet toggle media.
/* stream: MediaStream, type:trackType('audio'/'video') */
toggleTrack(stream,type) {
stream.getTracks().forEach((track) => {
if (track.kind === type) {
track.enabled = !track.enabled;
}
});
}

const senders = pc.getSenders();
senders.forEach((sender) => pc.removeTrack(sender));
newTracks.forEach((tr) => pc.addTrack(tr));
Get all the senders;
Loop Through and remove each sending track;
Add new tracks (if so desired);
Edit: or, if you won't need renegotiation (conditions listed below), use replaceTrack (https://developer.mozilla.org/en-US/docs/Web/API/RTCRtpSender/replaceTrack).
Not all track replacements require renegotiation. In fact, even
changes that seem huge can be done without requiring negotation. Here
are the changes that can trigger negotiaton:
The new track has a resolution which is outside the bounds of the
bounds of the current track; that is, the new track is either wider or
taller than the current one.
The new track's frame rate is high enough
to cause the codec's block rate to be exceeded. The new track is a
video track and its raw or pre-encoded state differs from that of the
original track.
The new track is an audio track with a different
number of channels from the original.
Media sources that have built-in
encoders — such as hardware encoders — may not be able to provide the
negotiated codec. Software sources may not implement the negotiated
codec.

async switchMicrophone(on) {
if (on) {
console.log("Turning on microphone");
const stream = await navigator.mediaDevices.getUserMedia({audio: true});
this.localAudioTrack = stream.getAudioTracks()[0];
const audioSender = this.peerConnection.getSenders().find(e => e.track?.kind === 'audio');
if (audioSender == null) {
console.log("Initiating audio sender");
this.peerConnection.addTrack(this.localAudioTrack); // will create sender, streamless track must be handled on another side here
} else {
console.log("Updating audio sender");
await audioSender.replaceTrack(this.localAudioTrack); // replaceTrack will do it gently, no new negotiation will be triggered
}
} else {
console.log("Turning off microphone");
this.localAudioTrack.stop(); // this will turn off mic and make sure you don't have active air-on indicator
}
}
This is simplified code. Solves most of the issues described in this topic.

Tracking JS object tracker eyes

I'm using Tracking JS to help detect if a users eyes are open and then only playing audio if it detects less than 3 objects, face, mouth, and one eye. I have it mostly convinced and working but the event is tracking and updating the data way too frequently, resulting in a very choppy and jarring experience.
Currently, I have the tracker checking constantly with the tracker on as follows:
tracker.on('track', function(event) {
if (event.data.length > 2 ) {
document.body.classList.add("hide");
pauseAudio();
} else {
document.body.classList.remove("hide");
playAudio();
}
Is there a way to run this function less frequently or specify how many times it should run per second?

You can use some sort of throttling mechanisem, lodash has it build it.
_.throttle, gets a function & time of throttling, and returns a throttled function, so you can use it as:
const handler = function(event) {
if (event.data.length > 2 ) {
document.body.classList.add("hide");
pauseAudio();
} else {
document.body.classList.remove("hide");
playAudio();
}
};
const throttledHandler = _.throttle(handler, 100); // 100ms
tracker.on('track', throttledHandler);

How to listen volume changes of youtube iframe?

Here I found an example how I can listen Play\Pause button of youtube iframe.
player.addEventListener('onStateChange', function(e) {
console.log('State is:', e.data);
});
Now I need to listen the volume changes.
In the youtube documentation and here I found a method player.getVolume(), but I have no idea how this method can be implemented if I want to be informed about volume changes from iframe side, instead of ask iframe from my side.
On YouTube Player Demo page such functionality exists (when I change the volume of a player, I see appropriate changes in the row Volume, (0-100) [current level: **]), but neither in the doc nor in internet I can not find how to implement it.
I also tried to use the above mentioned code with onApiChange event (it is not clear for me what this event actually does), like:
player.addEventListener('onApiChange', function(e) {
console.log('onApiChange is:', e.data);
});
but console shows nothing new.
player.getOptions(); shows Promise {<resolved>: Array(0)}.
Could anyone show an example?

See this question.
You can listen to postMessage events emitted by the IFrame and react only to the volume change ones:
// Instantiate the Player.
function onYouTubeIframeAPIReady() {
var player = new YT.Player("player", {
height: "390",
width: "640",
videoId: "dQw4w9WgXcQ"
});
// This is the source "window" that will emit the events.
var iframeWindow = player.getIframe().contentWindow;
// Listen to events triggered by postMessage.
window.addEventListener("message", function(event) {
// Check that the event was sent from the YouTube IFrame.
if (event.source === iframeWindow) {
var data = JSON.parse(event.data);
// The "infoDelivery" event is used by YT to transmit any
// kind of information change in the player,
// such as the current time or a volume change.
if (
data.event === "infoDelivery" &&
data.info &&
data.info.volume
) {
console.log(data.info.volume); // there's also data.info.muted (a boolean)
}
}
});
}
See it live.
Note that this relies on a private API that may change at anytime without previous notice.

I inspected the code of YouTube Player Demo page and found that the html line which shows the current YouTube volume (<span id="volume">**</span>) constantly blinking (~ 2 times per 1 sec), so I can assume this demo page uses something like this:
// YouTube returns Promise, but we need actual data
self = this
setInterval(function () { self.player.getVolume().then(data => { self.volumeLv = data }) }, 250)
Possibly not the best method, but it seems there is no other option (I also tried to listen changes in the appropriate style of the volume bar, but no luck due to the cross-origin problem).
So, this let us 'listen' volume changes of youtube.
Just in case, if someone wants to set youtube volume, you need to use [this.]player.setVolume(volume_from_0_to_100)

Set videoElement.currentTime in Hulu video player doesn't work, and breaks the player

I have a Chrome extension in which I'm trying to jump forward or backward (based on a user command) to a specific time in the video by setting the currentTime property of the video object. Before trying to set currentTime, a variety of operations work just fine. For example:
document.getElementsByTagName("video")[1].play(); // works fine
document.getElementsByTagName("video")[1].pause(); // works fine
document.getElementsByTagName("video")[1].muted = true; // works fine
document.getElementsByTagName("video")[1].muted = false; // works fine
BUT as soon as I try to jump to a specific point in the video by doing something like this:
document.getElementsByTagName("video")[1].currentTime = 500; // doesn't work
No errors are thrown, the video pauses, and any attempted actions after this point do nothing. So the items shown above (play/pause/mute/unmute) no longer work after attempting to set currentTime. If I read the value of currentTime after setting it, it correctly displays the new time that I just set it to. Yet nothing I do will make it play, and in fact even trying to make the video play by clicking the built-in toolbar no longer works. So, apparently setting currentTime wreaks all kinds of havoc in the video player. Yet if I reload the video, all works as before as long as I don't try to set currentTime.
I can easily jump to various times (backward or forward) by sliding the slider on the toolbar, so there must be some way internally to do that. Is there some way I can discover what code does a successful time jump? Because it's a Chrome extension I can inject custom js into the executing Hulu js, but I don't know what command I would send.
Any ideas?

Okay I fiddled around with it for a little while to see how I could reproduce the click event on the player and came up with the following solution:
handleViewer = function(){
var thumbnailMarker = $('.thumbnail-marker'),
progressBarTotal = thumbnailMarker.parent(),
controlsBar = $('.controls-bar'),
videoPlayer = $('#content-video-player');
var init = function(){
thumbnailMarker = $('.thumbnail-marker');
progressBarTotal = thumbnailMarker.parent();
controlsBar = $('.controls-bar');
videoPlayer = $('#content-video-player');
},
check = function(){
if(!thumbnailMarker || !thumbnailMarker.length){
init();
}
},
show = function(){
thumbnailMarker.show();
progressBarTotal.show();
controlsBar.show();
},
hide = function(){
controlsBar.hide();
},
getProgressBarWidth = function(){
return progressBarTotal[0].offsetWidth;
};
return {
goToTime: function(time){
var seekPercentage,
duration;
check();
duration = videoPlayer[0].duration;
if(time > 0 && time < duration){
seekPercentage = time/duration;
this.jumpToPercentage(seekPercentage);
}
},
jumpToPercentage: function(percentage){
check();
if(percentage >= 1 && percentage <= 100){
percentage = percentage/100;
}
if(percentage >= 0 && percentage < 1){
show();
thumbnailMarker[0].style.left = (getProgressBarWidth()*percentage)+"px";
thumbnailMarker[0].click();
hide();
}
}
}
}();
Once that code is initialized you can do the following:
handleViewer.goToTime(500);
Alternatively
handleViewer.jumpToPercentage(50);
I've tested this in chrome on a MacBook pro. Let me know if you run into any issues.

Rather than try to find the javascript responsible for changing the time, why not try to simulate the user events that cause the time to change?
Figure out the exact sequence of mouse events that trigger the time change.
This is probably some combination of mouseover, mousedown, mouseup, and click.
Then recreate those events synthetically and dispatch them to the appropriate elements.
This is the approach taken by extensions like Stream Keys and Vimium.

The video should be ready to play before setting the currentTime.
Try adding this line before setting currentTime?
document.getElementsByTagName("video")[1].play();
document.getElementsByTagName("video")[1].currentTime = 500;

Looks like it works if you first pause, then set currentTime, then play again.
document.getElementsByTagName("video")[1].pause()
document.getElementsByTagName("video")[1].currentTime = 800.000000
document.getElementsByTagName("video")[1].play()
Probably would need to hook into some event like onseeked to put in the play command to make it more robust.

Changing source in videos-js eating memory until leak

i'm using video-js media player with flash fallback to dynamically play and change mp4-videos (h.264 - only source available) on a site i'm working on.
My problem is, every time i'm changing the video/source, the browser eats more and more memory until it is out of memory and crashes.
the problem occurs on every browser, flash player hardware acceleration enabled and disabled.
Player is initialized like this:
_V_.options.flash.swf = "../Scripts/ThirdParty/video-js.swf";
_V_.options.flash.iFrameMode = true; //false didn't help
_V_.players = {};
_V_("main_video", { "techOrder": ["flash", "html5"] }).ready(function () {
$.b4dvideo.videoPlayer = this;
if (!$.b4dvideo.contentInitialised) {
$.b4dvideo.contentInitialised = true;
$.b4dvideo._loadContent();
}
this.on("pause", function () {
this.posterImage.show()
});
this.on("ended", function () {
$.b4dvideo.videoPlayer.pause();
$.b4dvideo.videoPlayer.currentTime(0);
$.b4dvideo.videoPlayer.pause();
this.posterImage.show()
});
});
And changing the source of the player
if (!$.b4dvideo.videoPlayer.paused()) {
$.b4dvideo.videoPlayer.pause();
}
$.b4dvideo.videoPlayer.currentTime(0);
$.b4dvideo.videoPlayer.pause();
$.b4dvideo.videoPlayer.src(videoPath);
$.b4dvideo.videoPlayer.play();
It looks like the flash player is keeping the whole video in memory and never releasing it.
Any ideas on that?
I have even tried using jwplayer - same problem :-(
Update 1:
I also created a js-fiddle demonstrating this issue... just press play several times and watch your memory
http://jsfiddle.net/fwcJh/2/