I am writing an html5 app which manipulates video on a canvas.
I am also showing a custom (self-drawn) mouse cursor over it.
To update my app I am calling the setInterval function to draw stuff to the canvas.
As I am manipulating video I need: Very high refresh rates + Lot's of processor time to draw. (~50ms per frame).
My draw function causes my on-mouse function to starve. (this is acceptable by me).
But... After it is finished starving it responds to old events. It can take up to 3 frames for the mouse to catch so that I can render it in the right position. Meaning you can see the cursor "crawling" after you've stopped moving the mouse.
Is there a way to give the onmousemove events a higher priority then my setInterval(drawFunction)?
When in the draw function, Can I "ask" if there are pending mouse events, and revoke my current call to draw?
Is there some-other hack I can use? (I can draw to back-buffer in a webWorker, but as I understand from reading up on html5 this is only thread abstraction [threads are not concurrent] )
You can't prioritize event handling, at least not directly.
What you might consider would be having your own timer-driven code check for pending mouse events. The mouse event handler would just put a work request into a queue. Your video manipulating could could just check that queue and handle operations as it sees fit. Thus, the real mouse work would also be done in the timer code.
edit Here's what I'm thinking. You'd still have handlers for your mouse events:
var eventQueue = [];
canvasElement.onmousemove = function(evt) {
evt = evt || event;
eventQueue.push({ type: evt.type, x: evt.clientX, y: evt.clientY, when: new Date() });
};
Thus the mouse handlers would not do any actual work. They just record the details in a list of events and then return.
The main, timer-driven loop can then just check the queue of events and decide what to do. For example if it sees a whole string of "mousemove" events, it could compute the net change in position over all of them and update the cursor position just once.
You certainly can use a Web Worker to manipulate your pixels in the background. Web Workers do run concurrently. It doesn't seem to be part of the spec, but every implementation runs workers in a separate process. So your main script will be responsible for updating your custom cursor, but you'd pass the canvas ImageData off to the background worker for the video processing. Note that you can't send an actual Canvas or Context to the worker, as workers have no DOM access.
There is no way to give events higher priority, they seem to be serviced on a first come, first serve basic. Assuming you are using setTimeout, one thing that might help though is to break up your tasks into smaller restartable chunks, e.g. if you are processing an image like this:
for(y=0; y<height; y++) {
// Deal with rows of pixels
}
// Show image
You could do this instead:
var INTERVAL = height/4;
for(y = old_y; y<height && y<old_y+INTERVAL ; y++) {
// Deal with row of pixels
}
if (y == height) {
// show image
}
old_y = (y == height) ? 0 : y;
Then the other events will have 4x (or whatever, depending on the INTERVAL) greater chance of being dealt with.
Related
I've built this HTML5 video player that I am loading into a canvas to manipulate and back onto a canvas to display it. The video starts out quite slow and the frame rate only gets worse each time it is played. All I am currently manipulating in the video now is the color value when the video is paused, but will eventually be using real time manipulation throughout videos that will be posted in the future.
I used the below tutorial to learn this trick https://www.youtube.com/watch?v=zjQzP3mOXdc
Here is the relevant code, but there may possibly be interference coming from elsewhere so feel free to check the source code at the link at the bottom
var v = document.getElementById('video');
var color = "#DA7AC1";
var processes={
timerCallback:function() {
if (this.v2.paused || this.v2.ended) {
return;
}
this.ctxIn.drawImage(this.v2,0,0,this.width,this.height);
this.pixelScan();
var self=this;
setTimeout(function() {
self.timerCallback();
}, 0);
},
doLoad:function(){
this.v2=document.getElementById("video");
this.cIn=document.getElementById("cIn");
this.ctxIn=this.cIn.getContext("2d");
this.cOut=document.getElementById("cOut");
this.ctxOut=this.cOut.getContext("2d");
var self=this;
this.v2.addEventListener("playing", function() {
self.width=self.v2.videoWidth;
self.height=self.v2.videoHeight;
cIn.width=self.v2.videoWidth;
cIn.height=self.v2.videoHeight;
cOut.width=self.v2.videoWidth;
cOut.height=self.v2.videoHeight;
self.timerCallback();
}, false);
},
pixelScan: function() {
var frame = this.ctxIn.getImageData(0,0,this.width,this.height);
for(var i=0; i<frame.data.length;i+=4) {
var grayscale=frame.data[i]*.3+frame.data[i+1]*.59+frame.data[i+2]*.11;
frame.data[i]=grayscale;
frame.data[i+1]=grayscale;
frame.data[i+2]=grayscale;
}
this.ctxOut.putImageData(frame,0,0);
return;
}
}
http://coreytegeler.com/ethan/
Any and all help would be greatly appreciated! Thanks!
Reason 1
Try to adjust your timer avoiding 0 as timeout value:
setTimeout(function() {
self.timerCallback();
}, 34);
34ms is plenty as video frame rate is typically never more than 30 FPS (NTSC) or 25 FPS (PAL), ie 1000 / 30. If you use 0 you risk stacking up your calls which means the browser will be busy trying to empty the event queue.
If you use anything lower than 33-34ms you end up having the same frame processed twice or more which of course is unnecessary (your video is actually 29.97 FPS/NTSC so you might want to consider keeping 34ms).
Reason 2
The video resolution is also full HD (1920x1080) which is a bit too much for canvas and JS to process in real-time (for a typcial consumer computer). Try to reduce the video size so a normal spec'ed computer will be able to process the data.
Reason 3 (in part)
You don't need two on-screen canvases or even an on-screen video. Try to create these tags dynamically and not inserting them into the DOM. Use a single canvas on-screen and draw the result to that (you can putImageData from one canvas to another).
Reason 4 (in part)
Ideally, replace setTimeout with a requestAnimationFrame approach as this improves the synchronization and efficiency considerably. You can implement a toggle to reduce the FPS to for example 30 as you don't need to process each frame twice (ref. 30 FPS video frame rate).
Update
To create these elements dynamically (ref reason 3) you can do something like this:
var canvas = document.createElement('canvas'),
video = document.createElement('video'),
ctx = canvas.getContext('2d');
video.preload = 'auto';
video.addEventListener('canplay', start, false);
if (video.canPlayType('video/mp4')) {
video.src = 'videoUrl.mp4';
} else if ...etc.
Then when the video has loaded enough data (on metadata or canplay) you set the off-screen (and on-screen) canvas element to the size of the video:
canvas.width = video.videoWidth;
canvas.height = video.videoHeight;
Then when playing process its buffer and copy to the on-screen canvas you defined before.
You don't have have an off-screen canvas - I merely mention this as you in your original code used and in and out canvas IIRC. You can simply use a single on-screen canvas and the off-screen video and draw to the video frame to the canvas, process it and put back the processed data. Should work fine too in this case.
I ran a profile in chrome and it points to line 46 as taking up the most CPU.
setTimeout(function() {
self.timerCallback();
}, 0);
Perhaps increasing the timeout will stop it from lagging.
I had the same issues and tried a number of fixes. I was using Premier Elements which didn't export to mp4 and using HandBrake to convert the format. I also Tried FFMpeg to do the conversion, but neither worked.
What I did was switch to Kdenlive as my video editor, it exported directly to MP4, and that video worked perfectly.
So, if you are have this slow render issue, it is probably an issues with the video encoding. Easiest fix is to get a high quality video editor like Premier Pro, Final Cut, or Kdenlive. Kdenlive is free but it has a huge learning curve and poor public documentation.
So, I'm making a relatively trivial HTML5 Canvas drawing web app. Basically, you can select your color, and then draw on a 500x500 canvas. It's going to be themed as a "graffiti" wall, so I am attempting to create a graffiti effect to the drawing, much like the Spray tool in MS Paint of yore.
Feel free to take a look at it here.
In order to facilitate this effect, I'm making use of web workers to callback on mouse events and asynchronously draw to the canvas. The naive implementation I have now is that on any mouse event, 5 pixels are randomly drawn around the coords of the event.
What I would like to do though, is to have those pixels drawn continuously from the mousedown event until the mouseup event, while updating the coords on mousemove events. From my limited knowledge of JavaScript, I imagine that this could involve a setTimeout(), but I'm not sure how to manipulate this to achieve what I want.
DISCLAIMER: This is part of a school project, and as such I am trying to avoid JQuery, Ajax, and other such frameworks; my goal here is to make an as-pure-as-possible JavaScript/HTML5 web app.
Thanks in advance.
Using a timer (no worker required):
var mouseX = 0,
mouseY = 0,
mouseDown = false;
function ev_canvas( ev ) {
if (ev.offsetX || ev.offsetX == 0) { //opera
mouseX = ev.offsetX;
mouxeY = ev.offsetY;
} else if (ev.layerX || ev.layerX == 0) { //firefox
var canvasOffset = document.getElementById("graffiti_wall").getBoundingClientRect();
mouseX = ev.layerX - canvasOffset.left;
mouseY = ev.layerY - canvasOffset.top;
}
if ( ev.type == 'mousedown' ) {
mouseDown = true;
}
else if ( ev.type == 'mouseup' ) {
mouseDown = false;
}
}
function draw_spray() {
if( !mouseDown ) {
//Don't do anything since the mouse is not pressed down
return;
}
//Draw something at the last known location
context.strokeRect( mouseX, mouseY, 1, 1 );
}
//Call draw_spray function continuously every 16 milliseconds
window.setInterval( draw_spray, 16 );
If you want to use a WebWorker (for example for more complex drawing algorithms), I could think of the following setup:
onmousedown, spawn a new worker and register a handler on it that paints objects on the canvas
onmouseup (and -leave etc), terminate the worker
onmousemove, if a worker exists determine mouse coordinates and send them into the worker
In the worker
listen to new mouse coordinates
start an timeout interval which constantly fires draw-events (depending on current coordinates and a clever algorithm)
Yet I think that a Worker is too much overhead for a simple graffiti tool. Use the simple solution without a Worker like #Esailija demonstrated.
If you had a more complex application which could make good use of Workers, you wouldn't really spawn them onmousedown and terminate them. Instead, you maybe instantiated single Workers for single kinds of tools, and fired start-processing and end-processing events to them.
I have most of a module written to handle multitouch pinch, pan and zoom on an HTML 5 canvas element. I will share it below. I've been developing in JavaScript for some time now, and this one continues to boggle me. If anybody has any insights, I will post the final version up on stack for everybody to share once it is confirmed working on my iPad.
Here is what I am doing:
touchmove events trigger the changing of variables. I use these variables to change the way in which my image is painted onto the canvas. I have eight variables, each corresponding to the options that can be put into the drawImage() function. These eight variables get updated through functions that increment/decrement their values and keep them within a certain range. The variables are closure variables, so they are global throughout my module. To prevent over-processing, I make a call to this drawImage() function once every 40ms while the user has their finger pressed to the screen using a setInterval().
Here is the problem:
touchmove events seem to be causing a race condition where my variables get updated by many different instances of that same event. I can somewhat confirm this through my console output, that tracks one variable that is bounded to never reach below 20. When I swipe in one direction quickly, that variable dips down far below 20. Then when I release my finger, swipe slowly, it returns to 20. Another thing that points me in this direction, when I look at these variables while stepping through my program, they differ from what my console.log() pumps out.
Note: The code successfully draws the image the first time, but not anytime thereafter. A basic rendition of my code is below... The full version is on GitHub inside the Scripts folder. It is a Sencha Touch v1.1 app at heart
function PinchPanZoomFile(config)
{
/*
* Closure variable declaration here...
* Canvas Declaration here...
*/
function handleTouchStart(e) {
whatDown.oneDown = (e.originalEvent.targetTouches.length == 1) ? true : false;
whatDown.twoDown = (e.originalEvent.targetTouches.length >= 2) ? true : false;
drawInterval = setInterval(draw, 100);
}
function handleTouchEnd(e) {
whatDown.oneDown = (e.originalEvent.targetTouches.length == 1) ? true : false;
whatDown.twoDown = (e.originalEvent.targetTouches.length >= 2) ? true : false;
clearInterval(drawInterval);
}
function handleTouchMove(e) {
if(whatDown.twoDown) {
/*
* Do Panning & Zooming
*/
changeWindowXBy(deltaDistance); //deltaDistance
changeWindowYBy(deltaDistance); //deltaDistance
changeCanvasXBy(deltaX); //Pan
changeCanvasYBy(deltaY); //Pan
changeWindowDimsBy(deltaDistance*-1,deltaDistance*-1); //(deltaDistance)*-1 -- get smaller when zooming in.
changeCanvasWindowDimsBy(deltaDistance,deltaDistance); //deltaDistance -- get bigger when zooming in
} else if(whatDown.oneDown) {
/*
* Do Panning
*/
changeWindowXBy(0);
changeWindowYBy(0);
changeCanvasXBy(deltaX);
changeCanvasYBy(deltaY);
changeWindowDimsBy(0,0);
changeCanvasWindowDimsBy(0,0);
}
}
function draw() {
//Draw Image Off Screen
var offScreenCtx = offScreenCanvas[0].getContext('2d');
offScreenCtx.save();
offScreenCtx.clearRect(0, 0, canvasWidth, canvasHeight);
offScreenCtx.restore();
offScreenCtx.drawImage(base64Image,
parseInt(windowX),
parseInt(windowY),
parseInt(windowWidth),
parseInt(windowHeight),
parseInt(canvasX),
parseInt(canvasY),
parseInt(canvasWindowWidth),
parseInt(canvasWindowHeight)
);
//Draw Image On Screen
var offScreenImageData = offScreenCtx.getImageData(0, 0, canvasWidth, canvasHeight);
var onScreenCtx = canvas[0].getContext('2d');
onScreenCtx.putImageData(offScreenImageData, 0, 0);
}
}
I strongly recommend using Sencha Touch 2.0.1, as it supports a lot of the touch events you need. See http://dev.sencha.com/deploy/touch/examples/production/kitchensink/#demo/touchevents for examples.
We're developing arcade (a lot of action and speed) browser 2d-game using canvas.
Sometimes our testing players report us that there is a delay: player still moving 5-10 pixels away after keyup.
I've digged this issue, you can see yourself delay http://jsfiddle.net/C4ev3/7/ (try keydown/up any key as fast as you can). My results is from 70 to 120ms. And i think that's a lot. (FYI, our network latency is 10-20ms).
Any ideas how to reduce this delay?
upd i've noticed that on good hardware this delay is under 30-40ms. But i'm testing on core2duo, winxp, chrome 19 - it's not a P4 with IE6 :)
Hi one thing you could do is instead of using an anonymous function try using defined functions,
http://jsfiddle.net/C4ev3/10/ - for me this reported at 50-100 MS
However i would not recommend jQuery for Canvas Applications it's very big for the very little you using, you should try using native Javascript
http://jsfiddle.net/C4ev3/11/ - for me this reported 30-70 MS
Javascript Threading
One thing i noticed in the comments Javascript is not Multi-Threaded Well Urm-Arr,
it sort of is setInterval is Async not Sync, however affecting the window is a single thread E.G if you have a Class that has some number is it using a setInteval will use another thread and not have a problem altering the math however in the Task then requires a Draw on the page it will enter the bottom of the JS handle Que,
Certain parts of Javascript are on a different thread how ever any thing changing the page has to run on the Main Thread same as any Windows application if your thread want to change the Form your have to invoke the main thread to do it for you
however it is not multi-threaded like any thing else you cant just handle or abort at a given Wim like windows,
Other ASync Tasks include AJAX has the option to be both Async and Sync
Updated to show my comment about FPS limiting:
Please bear with me. This is linking to a project that is allready built to show the example:
so my Game is Completely OOP
var elem = document.getElementById('myCanvas');
var context = elem.getContext("2d");
context.fillStyle = '#888';
context.lineWidth = 4;
// Draw some rectangles.
context.fillRect(0, 0, 800, 600);
context.fillStyle = '#f00';
var ball = new Ball();
var leftPadel = new Padel(10, 60, 40, 120);
var rightPadel = new Padel(750, 520, 40, 120);
pong = new Pong();
pong.draw();
setTimeout("ball.move()", pong.redrawTime());
Inside my pong class is where all the main workings of the game goes but here are the FPS bit you need to see
this.fps = 30;
this.maxFPS = 60;
this.redrawTime = function(){
return (1000 / this.fps)
}
this.lastDraw = (new Date)*1 - 1;
Then as you can see my Interval is on ball.move this calls the main pong class again on redraw at the End of the redraw i have the FPS checking and limiting code
this.fps = ((now=new Date) - this.lastDraw);
if(this.fps > this.maxFPS){
this.fps = this.maxFPS;
}
this.lastDraw = (new Date)*1 - 1;
if(this.reporting = true){
console.clear();
console.log("FPS: "+this.fps.toFixed(1))
}
setTimeout("ball.move()", pong.redrawTime());
This then forces you to get the Best Possible FPS without queuing the Main Thread
Try this:
e.stopPropagation()
Stops the bubbling of an event to parent elements, preventing any
parent handlers from being notified of the event.
e.preventDefault()
Prevents the browser from executing the default action. Use the method
isDefaultPrevented to know whether this method was ever called (on
that event object).
My min. results in Google chrome: 7ms
I am coding a script in which the user selects a range of data, and then I fetch a bunch of images (over 150) from the server and then I loop trough them to make something like a movie. What I want to know is the most efficient way to load prevent lag when moving trough the images.
Currently I am fetching the images from the server using Ajax and store them in a array of Image objects on the JavaScript. In the HTML I have a div tag in which I wish to put the images. After I finished creating all the Image object (and setting their proper src) in the array, I do the following:
imgElem = document.createElement('img');
document.getElementById('loopLocation').appendChild(imgElem);
imgElem.src = images[0].src;
After this I just make that last call but changing the loop index. I do that every 400ms. The loop works, but sometimes it lags and it freezes on an image for longer that it is supposed to be. I want to know if I am able to improve this anymore from the client side or I just need a server that responds faster.
you might wanna consider spriting which is putting all images into one big image. with this, you only need to load one big image, and then just reposition for every scene.
or, you might also want to pre-load those 150 images, before actually using them. you can use JS array to store Image objects and then loop through that array to get your images.
var images = [];
var expectLoaded = 150;
for(var i = 0; i<expectLoaded;i++){
(function(i){
//new image object
var img = new Image();
//onload hander
img.onload = function(){
//after load, push it into the array
images.push[img];
//and check if the all has loaded
if(images.length === expectLoaded){
//preload done
}
}
//start loading this image
img.src = //path to image[i];
},(i));
}
loops block the UI thread. JS is single-threaded, meaning code gets executed in a linear fashion, one after the other. anything that comes after that loop statement will wait until the loop finishes. if that loop takes long... grab some coffee. plus, since you are manipulating the DOM, you don't see the changes since the UI thread is blocked.
but there are ways to bypass this, and one of them is using timeouts to delay and queue the code for later execution, when JS is not busy.
function animate(frameNo){
//animate frame[frameNo];
if(frameNo < total_frames){ //make the UI breate in between frames
setTimeout(function(){ //so that changes reflect on the screen
animate(++frameNo); //then animate next frame
},200);
}
}
//start
animate(0);
I agree with Joseph on his second point. Here's a nice link to accomplish image preloading before you start the loop: http://www.javascriptkit.com/javatutors/preloadimagesplus.shtml