We Keep Coding

I get a blank (dark) screen when I try to launch a picture in picture video based on canvas

Im pretty new to web dev and Svelte, so apologies if this is basic. I can't seem to get picture in picture (PiP) working. It will show up but the screen is a very dark grey, the content is not being displayed. The canvas in question does display elsewhere on the page and I can see it being updated properly.
My browser allows PiP from youtube, etc. so I don't think that is the issue. The relevant code is as follows, its written using Svelte.
<script>
....
const buildPopout = () => {
writeToCanvas();
createVideo();
}
const writeToCanvas = () => {
const canvasEl = document.getElementById("canvas");
const canvasCtx = canvasEl.getContext('2d');
//canvasCtx.font = '52px serif';
canvasCtx.clearRect(0, 0, canvas.width, canvas.height);
canvasCtx.textAlign = 'center';
//Note: timeString is set elsewhere and is working, as is this update
canvasCtx.fillText(timeString, canvas.width / 2, canvas.height / 2);
}
//
async function createVideo() {
const canvasEl = document.getElementById("canvas");
console.log('canvasEL: ' + canvasEl); //TODO: Remove
const video = document.createElement('video');
video.muted = true;
video.autoplay = false;
video.controls = true;
video.srcObject = canvasEl.captureStream();
video.play();
video.addEventListener('loadedmetadata', () => {
video.requestPictureInPicture();
});
}
<div>
...
<button on:click={buildPopout}>Pop Out</button>
<canvas id="canvas" width="200" height="100"></canvas>
</div>
The expectation is this should mirror the content of the canvas in the picture in picture container and follow me between sites. It pops the container open and that does follow me. However, it doesn't contain the canvas content.
I can see the canvas and the content is changing, it just isn't showing up in the right place.

MediaPipe Selfie Segmentation results are not invoking when moved to other Chrome tabs

const videoElement =document.getElementsByClassName('input_video')[0];
selfieSegmentationObj = new SelfieSegmentation({locateFile: (file) => {
return `https://cdn.jsdelivr.net/npm/#mediapipe/selfie_segmentation/${file}`;
}});
selfieSegmentationObj.setOptions({
modelSelection: 1,
selfieMode: false,
effect: 'mask',
});
selfieSegmentationObj.onResults(selfieSegmentationResults);
cameraObj = new Camera(videoElement, {
onFrame: async () => {
await selfieSegmentationObj.send({image: videoElement});
},
width: 1280,
height: 720
});
cameraObj.start();
function selfieSegmentationResults(results) {
const canvasElement = document.getElementsByClassName('output_canvas')[0];
var canvasCtx = canvasElement.getContext('2d');
canvasCtx.clearRect(0, 0, canvasElement.width, canvasElement.height);
canvasCtx.save();
canvasCtx.drawImage(results.segmentationMask, 0, 0, canvasElement.width, canvasElement.height);
}
When we moved from one tab to another tab then callback function "selfieSegmentationResults" is not getting called from the SelfieSegmentation object.
Please, someone, help if there is any other solution that can get the segmented image data even if we moved to another Chrome tab.

Selfisegmentation might be using requestAnimationFrame under the hood which only works in the active tab.
The same thing was happening to me. I was building a background removing app with selfisegmentation and whenever I switched the tab, it stopped on the last frame. I solved it by using web workers.

How to play Multiple Videos in requestPictureInPicture?

requestPictureInPicture is so amazing, but it looks like it only works with 1 video.
How can I get requestPictureInPicture to play multiple videos, so I can watch two videos at the same time?
Basically this only displays one video:
video
.requestPictureInPicture()
.catch(error => {
console.log(error) // Error handling
});
video2
.requestPictureInPicture()
.catch(error => {
console.log(error) // Error handling
});
https://codepen.io/zecheesy/pen/YzwBJMR
Thoughts: Maybe we could put two videos in a canvas? And have the pictureInPicture play both videos at the same time? https://googlechrome.github.io/samples/picture-in-picture/audio-playlist
I'm not sure if this is possible. Would love your help so much!

Regarding opening two PictureInPitcure windows simultaneously, the specs have a paragraph just for it, where they explain they actually leave it as an implementation detail:
Operating systems with a Picture-in-Picture API usually restrict Picture-in-Picture mode to only one window. Whether only one window is allowed in Picture-in-Picture mode will be left to the implementation and the platform. However, because of the one Picture-in-Picture window limitation, the specification assumes that a given Document can only have one Picture-in-Picture window.
What happens when there is a Picture-in-Picture request while a window is already in Picture-in-Picture will be left as an implementation detail: the current Picture-in-Picture window could be closed, the Picture-in-Picture request could be rejected or even two Picture-in-Picture windows could be created. Regardless, the User Agent will have to fire the appropriate events in order to notify the website of the Picture-in-Picture status changes.
So the best we can say it that you should not expect it to open two windows simultaneously.
Now, if you really wish, you can indeed draw both videos on a canvas and pass this canvas to a PiP window, after piping its captureStream() to a third <video>, though this require that both videos are served with the proper Access-Control-Allow-Origin headers, and moreover, it requires your browser to actually support the PiP API (current Firefox has a PiP feature which is not the PiP API).
Here is a proof of concept:
const vids = document.querySelectorAll( "video" );
const btn = document.querySelector( "button" );
// wait for both video has their metadata
Promise.all( [ ...vids ].map( (vid) => {
return new Promise( (res) => vid.onloadedmetadata = () => res() );
} ) )
.then( () => {
if( !HTMLVideoElement.prototype.requestPictureInPicture ) {
return console.error( "Your browser doesn't support the PiP API" );
}
btn.onclick = async (evt) => {
const canvas = document.createElement( "canvas" );
// both videos share the same 16/9 ratio
// so in this case it's really easy to draw both on the same canvas
// to make it dynamic would require more maths
// but I'll let it to the readers
const height = 720;
const width = 1280;
canvas.height = height * 2; // vertical disposition
canvas.width = width;
const ctx = canvas.getContext( "2d" );
const video = document.createElement( "video" );
video.srcObject = canvas.captureStream();
let began = false; // rPiP needs video's metadata
anim();
await video.play();
began = true;
video.requestPictureInPicture();
function anim() {
ctx.drawImage( vids[ 0 ], 0, 0, width, height );
ctx.drawImage( vids[ 1 ], 0, height, width, height );
// iff we are still in PiP mode
if( !began || document.pictureInPictureElement === video ) {
requestAnimationFrame( anim );
}
else {
// kill the stream
video.srcObject.getTracks().forEach( track => track.stop() );
}
}
}
} );
video { width: 300px }
<button>enter Picture in Picture</button><br>
<video crossorigin muted controls autoplay loop
src="https://upload.wikimedia.org/wikipedia/commons/2/22/Volcano_Lava_Sample.webm"></video>
<video crossorigin muted controls autoplay loop
src="https://upload.wikimedia.org/wikipedia/commons/a/a4/BBH_gravitational_lensing_of_gw150914.webm"></video>
And beware, since I did mute the videos fo SO, scrolling in a way the original videos are out of sight will pause them.

Create thumbnail from video file via file input

I am attempting to create a thumbnail preview from a video file (mp4,3gp) from a form input type='file'. Many have said that this can be done server side only. I find this hard to believe since I just recently came across this Fiddle using HTML5 Canvas and Javascript.
Thumbnail Fiddle
The only problem is this requires the video to be present and the user to click play before they click a button to capture the thumbnail. I am wondering if there is a way to get the same results without the player being present and user clicking the button. For example: User click on file upload and selects video file and then thumbnail is generated. Any help/thoughts are welcome!

Canvas.drawImage must be based on html content.
source
here is a simplier jsfiddle
//and code
function capture(){
var canvas = document.getElementById('canvas');
var video = document.getElementById('video');
canvas.getContext('2d').drawImage(video, 0, 0, video.videoWidth, video.videoHeight);
}
The advantage of this solution is that you can select the thumbnail you want based on the time of the video.

Recently needed this so I wrote a function, to take in a video file and a desired timestamp, and return an image blob at that time of the video.
Sample Usage:
try {
// get the frame at 1.5 seconds of the video file
const cover = await getVideoCover(file, 1.5);
// print out the result image blob
console.log(cover);
} catch (ex) {
console.log("ERROR: ", ex);
}
Function:
function getVideoCover(file, seekTo = 0.0) {
console.log("getting video cover for file: ", file);
return new Promise((resolve, reject) => {
// load the file to a video player
const videoPlayer = document.createElement('video');
videoPlayer.setAttribute('src', URL.createObjectURL(file));
videoPlayer.load();
videoPlayer.addEventListener('error', (ex) => {
reject("error when loading video file", ex);
});
// load metadata of the video to get video duration and dimensions
videoPlayer.addEventListener('loadedmetadata', () => {
// seek to user defined timestamp (in seconds) if possible
if (videoPlayer.duration < seekTo) {
reject("video is too short.");
return;
}
// delay seeking or else 'seeked' event won't fire on Safari
setTimeout(() => {
videoPlayer.currentTime = seekTo;
}, 200);
// extract video thumbnail once seeking is complete
videoPlayer.addEventListener('seeked', () => {
console.log('video is now paused at %ss.', seekTo);
// define a canvas to have the same dimension as the video
const canvas = document.createElement("canvas");
canvas.width = videoPlayer.videoWidth;
canvas.height = videoPlayer.videoHeight;
// draw the video frame to canvas
const ctx = canvas.getContext("2d");
ctx.drawImage(videoPlayer, 0, 0, canvas.width, canvas.height);
// return the canvas image as a blob
ctx.canvas.toBlob(
blob => {
resolve(blob);
},
"image/jpeg",
0.75 /* quality */
);
});
});
});
}

Recently needed this and did quite some testing and boiling it down to the bare minimum, see https://codepen.io/aertmann/pen/mAVaPx
There are some limitations where it works, but fairly good browser support currently: Chrome, Firefox, Safari, Opera, IE10, IE11, Android (Chrome), iOS Safari (10+).
video.preload = 'metadata';
video.src = url;
// Load video in Safari / IE11
video.muted = true;
video.playsInline = true;
video.play();

You can use this function that I've written. You just need to pass the video file to it as an argument. It will return the dataURL of the thumbnail(i.e image preview) of that video. You can modify the return type according to your need.
const generateVideoThumbnail = (file: File) => {
return new Promise((resolve) => {
const canvas = document.createElement("canvas");
const video = document.createElement("video");
// this is important
video.autoplay = true;
video.muted = true;
video.src = URL.createObjectURL(file);
video.onloadeddata = () => {
let ctx = canvas.getContext("2d");
canvas.width = video.videoWidth;
canvas.height = video.videoHeight;
ctx.drawImage(video, 0, 0, video.videoWidth, video.videoHeight);
video.pause();
return resolve(canvas.toDataURL("image/png"));
};
});
};
Please keep in mind that this is a async function. So make sure to use it accordingly.
For instance:
const handleFileUpload = async (e) => {
const thumbnail = await generateVideoThumbnail(e.target.files[0]);
console.log(thumbnail)
}

The easiest way to display a thumbnail is using the <video> tag itself.
<video src="http://www.w3schools.com/html/mov_bbb.mp4"></video>
Use #t in the URL, if you want the thumbnail of x seconds.
E.g.:
<video src="http://www.w3schools.com/html/mov_bbb.mp4#t=5"></video>
Make sure that it does not include any attributes like autoplay or controls and it should not have a source tag as a child element.
With a little bit of JavaScript, you may also be able to play the video, when the thumbnail has been clicked.
document.querySelector('video').addEventListener('click', (e) => {
if (!e.target.controls) { // Proceed, if there are no controls
e.target.src = e.target.src.replace(/#t=\d+/g, ''); // Remove the time, which is set in the URL
e.target.play(); // Play the video
e.target.controls = true; // Enable controls
}
});
<video src="http://www.w3schools.com/html/mov_bbb.mp4#t=5"></video>

With jQuery Lib you can use my code here. $video is a Video element.This function will return a string
function createPoster($video) {
//here you can set anytime you want
$video.currentTime = 5;
var canvas = document.createElement("canvas");
canvas.width = 350;
canvas.height = 200;
canvas.getContext("2d").drawImage($video, 0, 0, canvas.width, canvas.height);
return canvas.toDataURL("image/jpeg");;
}
Example usage:
$video.setAttribute("poster", createPoster($video));

I recently stumbled on the same issue and here is how I got around it.
firstly it will be easier if you have the video as an HTML element, so you either have it in the HTML like this
<video src="http://www.w3schools.com/html/mov_bbb.mp4"></video>
or you take from the input and create an HTML element with it.
The trick is to set the start time in the video tag to the part you want to seek and have as your thumbnail, you can do this by adding #t=1.5 to the end of the video source.
<video src="http://www.w3schools.com/html/mov_bbb.mp4#t=1.5"></video>
where 1.5 is the time you want to seek and get a thumbnail of.
This, however, makes the video start playing from that section of the video so to avoid that we add an event listener on the video's play button(s) and have the video start from the beginning by setting video.currentTime = 0
const video = document.querySelector('video');
video.addEventListener('click', (e)=> {
video.currentTime = 0 ;
video.play();
})

Capture frames from video with HTML5 and JavaScript

I want to capture a frame from video every 5 seconds.
This is my JavaScript code:
video.addEventListener('loadeddata', function() {
var duration = video.duration;
var i = 0;
var interval = setInterval(function() {
video.currentTime = i;
generateThumbnail(i);
i = i+5;
if (i > duration) clearInterval(interval);
}, 300);
});
function generateThumbnail(i) {
//generate thumbnail URL data
var context = thecanvas.getContext('2d');
context.drawImage(video, 0, 0, 220, 150);
var dataURL = thecanvas.toDataURL();
//create img
var img = document.createElement('img');
img.setAttribute('src', dataURL);
//append img in container div
document.getElementById('thumbnailContainer').appendChild(img);
}
The problem I have is the 1st two images generated are the same and the duration-5 second image is not generated. I found out that the thumbnail is generated before the video frame of the specific time is displayed in < video> tag.
For example, when video.currentTime = 5, image of frame 0s is generated. Then the video frame jump to time 5s. So when video.currentTime = 10, image of frame 5s is generated.

Cause
The problem is that seeking video (by setting it's currentTime) is asynchronous.
You need to listen to the seeked event or else it will risk take the actual current frame which is likely your old value.
As it is asynchronous you must not use the setInterval() as it is asynchronous too and you will not be able to properly synchronize when the next frame is seeked to. There is no need to use setInterval() as we will utilize the seeked event instead which will keep everything is sync.
Solution
By re-writing the code a little you can use the seeked event to go through the video to capture the correct frame as this event ensures us that we are actually at the frame we requested by setting the currentTime property.
Example
// global or parent scope of handlers
var video = document.getElementById("video"); // added for clarity: this is needed
var i = 0;
video.addEventListener('loadeddata', function() {
this.currentTime = i;
});
Add this event handler to the party:
video.addEventListener('seeked', function() {
// now video has seeked and current frames will show
// at the time as we expect
generateThumbnail(i);
// when frame is captured, increase here by 5 seconds
i += 5;
// if we are not past end, seek to next interval
if (i <= this.duration) {
// this will trigger another seeked event
this.currentTime = i;
}
else {
// Done!, next action
}
});

If you'd like to extract all frames from a video, see this answer. The example below assumes that you want to extract a frame every 5 seconds, as OP requested.
This answer requires WebCodecs which is supported in Chrome and Edge as of writing.
<canvas id="canvasEl"></canvas>
<script type="module">
import getVideoFrames from "https://deno.land/x/get_video_frames#v0.0.8/mod.js"
let ctx = canvasEl.getContext("2d");
// `getVideoFrames` requires a video URL as input.
// If you have a file/blob instead of a videoUrl, turn it into a URL like this:
let videoUrl = URL.createObjectURL(fileOrBlob);
const saveFrameEverySeconds = 5;
let elapsedSinceLastSavedFrame = 0;
await getVideoFrames({
videoUrl,
onFrame(frame) { // `frame` is a VideoFrame object:
elapsedSinceLastSavedFrame += frame.duration / 1e6; // frame.duration is in microseconds, so we convert to seconds
if(elapsedSinceLastSavedFrame > saveFrameEverySeconds) {
ctx.drawImage(frame, 0, 0, canvasEl.width, canvasEl.height);
elapsedSinceLastSavedFrame = 0;
}
frame.close();
},
onConfig(config) {
canvasEl.width = config.codedWidth;
canvasEl.height = config.codedHeight;
},
});
URL.revokeObjectURL(fileOrBlob); // revoke URL to prevent memory leak
</script>
Demo: https://jsbin.com/qovapeziqi/edit?html,output
Github: https://github.com/josephrocca/getVideoFrames.js