We're working on a project where people can be in a chat room with their webcams, and they can grab a snapshot of someone's cam at that moment, do some annotations on top of it, and then share that modified picture as if it was their own webcam (like sharing a whiteboard).
Capturing the webcam stream into a canvas element where it can be edited was relatively easy. Finding the canvas element on our page and doing a .getContext('2d') on it,
Used an open library to add editing tools to it. Grabbing a stream from that canvas was done like so:
var canvasToSend = document.querySelector('canvas');
var stream = canvasToSend.captureStream(60);
var room = osTwilioVideoWeb.getConnectedRoom();
var mytrack = null;
room.localParticipant.publishTrack(stream.getTracks()[0]).then((publication) => {
mytrack = publication.track;
var videoElement = mytrack.attach();
});
This publishes the stream alright, but the first frame will not get sent unless you draw something else on the canvas. Let's say you drew 2 circles and then hit Share, the stream will start but will not be shown on the recipients' side unless you draw a line, or another circle, or anything. It seems like it needs a frame change for it to send data over.
I was able to force this with developer tools by doing something like context.fill();, but when I tried adding this after the publishing function, even in a then()... no luck.
Any ideas on how to force this "refresh" to happen?
So it seems it is expected behavior (and thus would make my FF buggy).
From the specs about the frame request algorithm:
A new frame is requested from the canvas when frameCaptureRequested is true and the canvas is painted.
Let's put some emphasis on the "and the canvas as been painted". This means that we need both these conditions, and while captureStream itself, or its frameRate argument ellapsing or a method like requestFrame would all set the frameCaptureRequested flag to true, we still need the new painting...
The specs even have a note stating
This algorithm results in a captured track not starting until something changes in the canvas.
And Chrome indeed seems to generate an empty CanvasCaptureMediaStreamTrack if the call to captureStream has been made after the canvas has been painted.
const ctx = document.createElement('canvas')
.getContext('2d');
ctx.fillRect(0,0,20,20);
// let's request a stream from before it gets painted
// (in the same frame)
const stream1 = ctx.canvas.captureStream();
vid1.srcObject = stream1;
// now let's wait that a frame ellapsed
// (rAF fires before next painting, so we need 2 of them)
requestAnimationFrame(()=>
requestAnimationFrame(()=> {
const stream2 = ctx.canvas.captureStream();
vid2.srcObject = stream1;
})
);
<p>stream initialised in the same frame as the drawings (i.e before paiting).</p>
<video id="vid1" controls autoplay></video>
<p>stream initialised after paiting.</p>
<video id="vid2" controls autoplay></video>
So to workaround this, you should be able to get a stream with a frame by requesting the stream from the same operation as a first drawing on the canvas, like stream1 in above example.
Or, you could redraw the canvas context over itself (assuming it is a 2d context) by calling ctx.drawImage(ctx.canvas,0,0) after having set its globalCompositeOperation to 'copy' to avoid transparency issues.
const ctx = document.createElement('canvas')
.getContext('2d');
ctx.font = '15px sans-serif';
ctx.fillText('if forced to redraw it should work', 20, 20);
// produce a silent stream again
requestAnimationFrame(() =>
requestAnimationFrame(() => {
const stream = ctx.canvas.captureStream();
forcePainting(stream);
vid.srcObject = stream;
})
);
// beware will work only for canvas intialised with a 2D context
function forcePainting(stream) {
const ctx = (stream.getVideoTracks()[0].canvas ||
stream.canvas) // FF has it wrong...
.getContext('2d');
const gCO = ctx.globalCompositeOperation;
ctx.globalCompositeOperation = 'copy';
ctx.drawImage(ctx.canvas, 0, 0);
ctx.globalCompositeOperation = gCO;
}
<video id="vid" controls autoplay></video>
Related
I'm new to React. I'm trying to create a <canvas> that will render a video in React so that I can provide a restriction on video-downloading as mentioned here(the paint on canvas part). (I know I can't prevent users from downloading it, just a preventive measure from my side for newbie users).
I was using this for HTML-JS :
var canvas = document.getElementById("canV");
var ctx = canvas.getContext("2d");
var video = document.createElement("video");
video.src = "http://techslides.com/demos/sample-videos/small.mp4";
video.addEventListener('loadeddata', function() {
video.play(); // start playing
update(); //Start rendering
});
function update(){
ctx.drawImage(video,0,0,256,256);
requestAnimationFrame(update); // wait for the browser to be ready to present another animation fram.
}
and
<canvas id="canV" width='256' height='256'></canvas>
Now, How can I do this in React? Also, any other simple preventive measures to prevent the videos from getting downloaded?
Also, there is this answer, Can I do something like this in react/gatsby? If yes, then please guide.
Thanks.
I was trying to make a basic media recorder with the MediaRecorder API which is fairly straight forward: get the stream from getDisplayMedia, then record it.
The problem: This only records the maximum screen size, but no more. So if my screen is 1280/720, it will not record 1920/1080.
This may seem quite obvious, but my intent is that it should record the smaller resolution inside of the bigger one. For example:
With the red rectangle representing what my actual screen is recording, and the surrounding black rectangle is simply black space, but the entire video is now a higher resolution, 1920/1080, which is useful for youtube, since youtube scales down anything that is in between 720 and 1080 resolution, which is a problem.
Anyway I tried simply adding the stream from getDisplayMedia to a video element video vid.srcObject = stream, then made a new canvas with the resolution 1920/1080, and in the animate loop just did ctx.drawImage(vid, offsetX, offsetY), and outside of the loop, where the MediaRecorder was made, simply did newStream = myCanvas.captureStream() as per the documentation of the API, and passed that to the MediaRecorder; however, the problem is that because of the huge canvas overhead, everything is really slow and the framerate is absolutely terrible (don't have video example, but just test it yourself).
So is there some way to optimize the canvas to not affect the framerate (tried looking into OffscreenCanvas but I couldn't find a way to get the stream from it itself to use with MediaRecorder, so it didn't really help), or is there a better way to capture and record the canvas, or is there a better way to record the screen within a larger resolution, in client-size JavaScript? If not with client-size JavaScript, is there some kind of real-time video encoder (ffmpeg is too slow) that could be run on the server, and each frame of the canvas could be sent to the server and saved there? Is there some better way to make a video recorder with any kind of JavaScript -- client or server or both?
Don't know what your code looks like, but I managed to get a smooth experience with this piece of code:
(You will also find very good example here: https://mozdevs.github.io/MediaRecorder-examples/)
<!doctype html>
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<script src="script.js"></script>
</head>
<body>
<canvas id="canvas" style="background: black"></canvas>
</body>
// DISCLAIMER: The structure of this code is largely based on examples
// given here: https://mozdevs.github.io/MediaRecorder-examples/.
window.onload = function () {
navigator.mediaDevices.getDisplayMedia({
video: true
})
.then(function (stream) {
var video = document.createElement('video');
// Use "video.srcObject = stream;" instead of "video.src = URL.createObjectURL(stream);" to avoid
// errors in the examples of https://mozdevs.github.io/MediaRecorder-examples/
// credits to https://stackoverflow.com/a/53821674/5203275
video.srcObject = stream;
video.addEventListener('loadedmetadata', function () {
initCanvas(video);
});
video.play();
});
};
function initCanvas(video) {
var canvas = document.getElementById('canvas');
// Margins around the video inside the canvas.
var xMargin = 100;
var yMargin = 100;
var videoWidth = video.videoWidth;
var videoHeight = video.videoHeight;
canvas.width = videoWidth + 2 * xMargin;
canvas.height = videoHeight + 2 * yMargin;
var context = canvas.getContext('2d');
var draw = function () {
// requestAnimationFrame(draw) will render the canvas as fast as possible
// if you want to limit the framerate a particular value take a look at
// https://stackoverflow.com/questions/19764018/controlling-fps-with-requestanimationframe
requestAnimationFrame(draw);
context.drawImage(video, xMargin, yMargin, videoWidth, videoHeight);
};
requestAnimationFrame(draw);
}
I wanted to display same video in two area of the application. So using canvas its working fine but the quality of original video is getting dropped but canvas video quality is fine.
var canvas = document.getElementById('shrinkVideo');
var context = canvas.getContext('2d');
var video = document.getElementById('mainVideo');
video.addEventListener('play', () => {
// canvas.width = 270;
// canvas.height = 480;
this.draw(video, context, canvas.width,canvas.height);
}, false);
draw(v, c, w, h) {
if (v.paused || v.ended) return false;
c.drawImage(v, 0, 0, w, h);
setTimeout(this.draw, 20, v, c, w, h);
}
This is my code to sync two video's and it is working fine but 'mainVideo' quality gets dropped.
But if I remove all the canvas code and just play 'mainVideo' the quality is maintained but using canvas its quality get dropped.
Expected Result This is output of the video when canvas code is not added
Actual Result This is output I am getting after adding the canvas code
Thanks In Advance
I came to this answer because I thought I was experiencing the same issue.
I have 1080p source on a element (HD content from a HDMI capture device, which registers as a webcam in the browser)
I had a 1920x1080 canvas and I was using ctx.drawImage(video, 0, 0, 1920, 1080) - as mentioned by a commenter above, I think I've found it crucial that you draw only in good multiples of the original height/width values.
I tried with-and-without imageSmoothingEnabled and various imageSmoothingQuality settings in Chrome/Brave; ultimately, I saw no vast difference with these settings.
My canvas on my webapp was still coming out extremely blurry -- unable to read even
~24pt font on the screen, basically couldn't use the video at all
I was frustrated by my blurry video so I recreated a full test in a "clean suite" here and now I experience no scaling issues anymore -- I don't know what my main application is doing differently yet, but in this example, you can attach any 1080p/720p device and see it scaled quite nicely to 1080p (change the resolution in the JS file if you want to scale to 720p)
https://playcode.io/543520
const WIDTH = 1920;
const HEIGHT = 1080;
const video = document.getElementById('video1'); // Video element
const broadcast = document.getElementById('broadcast'); // Canvas element
broadcast.width = video.width = WIDTH;
broadcast.height = video.height = HEIGHT;
let ctx = broadcast.getContext('2d')
onFrame = function() {
ctx.drawImage(video, 0, 0, broadcast.width, broadcast.height)
window.requestAnimationFrame(onFrame);
}
navigator.mediaDevices
.getUserMedia({ video: true })
.then(stream => {
window.stream = stream
console.log('got UM')
video.srcObject = stream;
window.onFrame();
})
Below you can see my viewport, with a Video and Canvas element (both 1080p, scaled to ~45% so they fit), using requestAnimationFrame to draw my content. The content is zoomed out, so you can see anti-aliasing, but if you load the example and click on the Canvas, it goes to Fullscreen, and the quality is pretty good - I played a 1080p Youtube video on my source machine, and couldn't see any difference on my full screen 1080p canvas element.
I'm creating a PDF output tool using jsPDF but need to add multiple pages, each holding a canvas image of a video frame.
I am stuck on the logic as to the best way to achieve this as I can't reconcile how to queue the operations and wait on events to achieve the best result.
To start I have a video loaded into a video tag and can get or set its seek point simply with:
video.currentTime
I also have an array of video seconds like the following:
var vidSecs = [1,9,13,25,63];
What I need to do is loop through this array, seek in the video to the seconds defined in the array, create a canvas at these seconds and then add each canvas to a PDF page.
I have a create canvas from video frame function as follows:
function capture_frame(video_ctrl, width, height){
if(width == null){
width = video_ctrl.videoWidth;
}
if(height == null){
height = video_ctrl.videoHeight;
}
canvas = document.createElement('canvas');
canvas.width = width;
canvas.height = height;
var ctx = canvas.getContext('2d');
ctx.drawImage(video_ctrl, 0, 0, width, height);
return canvas;
}
This function works fine in conjunction with the following to add an image to the PDF:
function addPdfImage(pdfObj, videoObj){
pdfObj.addPage();
pdfObj.text("Image at time point X:", 10, 20);
var vidImage = capture_frame(videoObj, null, null);
var dataURLWidth = 0;
var dataURLHeight = 0;
if(videoObj.videoWidth > pdfObj.internal.pageSize.width){
dataURLWidth = pdfObj.internal.pageSize.width;
dataURLHeight = (pdfObj.internal.pageSize.width/videoObj.videoWidth) * videoObj.videoHeight;
}else{
dataURLWidth = videoObj.videoWidth;
dataURLHeight = videoObj.videoHeight;
}
pdfObj.addImage(vidImage.toDataURL('image/jpg'), 'JPEG', 10, 50, dataURLWidth, dataURLHeight);
}
My logic confusion is how best to call these bits of code while looping through the vidSecs array as the problem is that setting the video.currentTime needs the loop to wait for the video.onseeked event to fire before code to capture the frame and add it to the PDF can be run.
I've tried the following but only get the last image as the loop has completed before the onseeked event fires and calls the frame capture code.
for(var i = 0; i < vidSecs.length; i++){
video.currentTime = vidSecs[i];
video.onseeked = function() {
addPdfImage(jsPDF_Variable, video);
};
}
Any thoughts much appreciated.
This is not a real answer but a comment, since I develop alike application and got no solution.
I am trying to extract viddeo frames from webcam live video stream and save as canvas/context, updated every 1 - 5 sec.
How to loop HTML5 webcam video + snap photo with delay and photo refresh?
I have created 2 canvases to be populated by setTimeout (5000) event and on test run I don't get 5 sec delay between canvas/contextes, sometimes, 2 5 sec. delayed contextes get populated with image at the same time.
So I am trying to implement
Draw HTML5 Video onto Canvas - Google Chrome Crash, Aw Snap
var toggle = true;
function loop() {
toggle = !toggle;
if (toggle) {
if (!v.paused) requestAnimationFrame(loop);
return;
}
/// draw video frame every 1/30 frame
ctx.drawImage(v, 0, 0);
/// loop if video is playing
if (!v.paused) requestAnimationFrame(loop);
}
to replace setInterval/setTimeout to get video and video frames properly synced
"
Use requestAnimationFrame (rAF) instead. The 20ms makes no sense. Most video runs at 30 FPS in the US (NTSC system) and at 25 FPS in Europe (PAL system). That would be 33.3ms and 40ms respectively.
"
I am afraid HTML5 provided no quality support for synced real time live video processing via canvas/ context, since HTML5 offers no proper timing since was intended to be event/s controlled and not run as real time run app code ( C, C++ ...).
My 100+ queries via search engine resulted in not a single HTML5 app I intend to develop.
What worked for me was Snap Photo from webcam video input, Click button event controlled.
If I am wrong, please correct me.
Two approaches:
create a new video element for every seek event, code provided by Chris West
reuse the video element via async/await, code provided by Adrian Wong
Requirement:
Now: Draw on a Canvas, and hit Save (store Canvas state/drawing offline - but NOT as image).
Later: Open up the Canvas with previously saved drawing showing, and continue to draw again.
For drawing we normally use code as follows:
canvas = document.getElementById('can');
ctx = canvas.getContext("2d");
...
ctx.beginPath();
ctx.moveTo(prevX, prevY);
ctx.lineTo(currX, currY);
....
In order to restore Canvas state later - exporting to Image does not help.
I want to restore the Canvas to it's original state to continue editing the drawing at a later date.
I guess, the Canvas context has to be exported and stored offline - how?
Your best shot here is to use a Proxy that will both store the draw commands and perform the drawings.
Since the browser support for Proxy is very bad (only FF as of today), you'll have to build the Proxy yourself, either by using nosuchmethod, or by building a new brand new WatchedContext Class out of the Context2D.
I took the last solution (WatchedContext Class) for this short demo :
function WatchedContext(hostedCtx) {
this.commands= [];
Context2dPrototype = CanvasRenderingContext2D.prototype;
for (var p in Context2dPrototype ) {
this[p] = function(methodName) {
return function() {
this.commands.push(methodName, arguments);
return Context2dPrototype[methodName].apply(hostedCtx, arguments);
}
}(p);
}
this.replay=function() {
for (var i=0; i<this.commands.length; i+=2) {
var com = this.commands[i];
var args = this.commands[i+1];
Context2dPrototype[com].apply(hostedCtx, args);
}
}
}
Obviously you might need some other method (start/stop recording, clear, ...)
Just a small example of use :
var cv = document.getElementById('cv');
var ctx=cv.getContext('2d');
var watchedContext=new WatchedContext(ctx);
// do some drawings on the watched context
// --> they are performed also on the real context
watchedContext.beginPath();
watchedContext.moveTo(10, 10);
watchedContext.lineTo(100, 100);
watchedContext.stroke();
// clear context (not using the watched context to avoid recording)
ctx.clearRect(0,0,100,1000);
// replay what was recorded
watchedContext.replay();
You can see here :
http://jsbin.com/gehixavebe/2/edit?js,output
That the replay does work, and the line is re-drawn as a result of replaying the stored commands.
For storing offline you can either store the commands locally using localStorage or store them remotely on a server an use AJAX calls or similar.