I want to play a HTML5 video in segments of x seconds. For example, start at t=0 and pause at t=x, then start at t=x and pause at t=2x. The problem is that the updates I receive with ontimeupdate don't match my intervals. There is one update short before the intended stop time and one update shortly after. I could just stop the video whenever currentTime >= x, but the problem here is that this stop point would fall into the new interval. The requirement for this task is to stop at the end of a given interval.
If stopping exactly at a given time is not possible, is there any way to determine the closest possible stop time before that time? That would still be better than stopping too late. I checked the deltas of currentTime (time between each ontimeupdate call), but these are not constant. I read somewhere else that the browser adapts this rate based on some optimization criterions, so that is probably hard to compute.
Background for this question is a tool that I want to develop. The user is shown a video and he is required to answer some questions for each x second interval of this video.
Unfortunately the timeupdate event doesn't provide a lot of granularity. You can read this answer for more information.
One alternative is to set up an interval manually with setInterval, and on each interval check the time passed with Date.now() since the last time that the timeupdate was updated (also using Date.now() and saving that value somewhere).
That would allow knowing the milliseconds from then, although it would need to handle cases like pausing the video and clearing the interval when necessary to avoid memory leaks with clearTimeout.
Related
From my tests, and from searching to find out more about the problem, my best guess is that css animations may be using a different physical clock from the one used to stream audio. If so perhaps the answer to this is that it can't be done, but am asking in case I am missing anything.
It is for my online metronome here.
It is able to plays notes reasonably accurately in response to an event listener for the css animationiteration event. The eventlistener is set up using e.g.
x.addEventListener("animationstart",playSoundBall2);
See here.
However if I try to synchronize the bounce with the sample precise timing of the AudioContext method that's when I run into problems.
What I do is to use the first css callback just to record the audio context time for the css elapsed time of 0. Then I play the notes using the likes of:
oscillator.start(desired_start_time);
You can try it out with the option on the page: "Schedule notes in advance for sample-precise timing of the audio" on the page here.
You can check how much it drifts by switching on "Add debug info to extra info" on the same page.
On my development machine it works fine with Firefox. But on Edge and Chrome it drifts away from the bounce. And not in a steady way. Can be fine for several minutes and then the bounce starts to slow down relative to the audio stream.
It is not a problem with browser activity - if I move the browser around and try to interrupt the animation the worst that happens is that it may drop notes and if the browser isn't active it is liable to drop notes. But the ones it plays are exactly in time.
My best guess so far, is that it might be that the browser is using the system time, while the audiocontext play method is scheduling it at a precise point in a continuous audio stream. Those may well be using different hardware clocks, from online searches for the problem.
Firefox may for some reason be using the same hardware clock, maybe just on my development machine.
If this is correct, it rather looks as if there is no way to guarantee to precisely synchronize html audio played using AudioContext with css animations.
If that is so I would also think you probably can't guarantee to synchronize it with any javascript animations as it would depend on which clocks the browser uses for the animations, and how that relates to whatever clock is used for streaming audio.
But can this really be the case? What do animators do who need to synchronize sound with animations for long periods of time? Or do they only ever synchronize them for a few minutes at a time?
I wouldn't have noticed if it weren't that the metronome naturally is used for long periods at a time. It can get so bad that the click is several seconds out from the bounce after just two or three minutes.
At other times - well while writing this I've had the metronome going for ten minutes in my Windows 10 app and it has drifted, but only by 20-30 ms relative to the bounce. So it is very irregular, so you can't hope to solve this by adding in some fixed speed up or slow down to get them in time with each other.
I am writing this just in case there is a way to do this in javascript, anything I'm missing. I'm also interested to know if it makes any difference if I use other methods of animation. I can't see how one could use the audio context clock directly for animation as you can only schedule notes in the audio stream, can't schedule a callback at a particular exact time in the future according to the audio stream.
I'm setting a timer in Node.js that waits for 3 hours to emit an event, once that time is reached it emits the event to all browsers that are listening.
The browsers gathers the information that such event before it happens and then calculates the time remaining ticking a countdown every 1 second, and expecting that when the clock reaches 0 the event will be triggered.
So one is using setTimeout (Node.js) and the other is using setInterval (browser) counting per second (countdown).
can I be sure that?
By the time the countdown reaches 0, the event will be triggered with an error range of around 1 to 2 seconds. (browser).
That the Node.js setTimeout is accurate enough to be called with a less than 1 second error range.
I've read about timers being 500ms to even 1000ms innacurate, which is fine for my needs, but I have never heard of them being used for this much time as I want to do.
Are they accurate enough or should I use a different solution? especially in the Node.js side which has to be the most accurate of them all.
Alternatives are to make a interval in Node.js that runs around 4 times per second, calculates the Date milliseconds, and checks if there are events that it needs to call from a list of events.
In the browser is to set the interval so that it calculates the date ms with every callback to try to keep the time synchronized.
The accuracy of the timer will be dependent upon how "busy" the event loop is at the time of the timeout.
It should be good enough if you wanted something like:
setTimeout(done, THREE_HOURS_IN_MS);
If your event loop is blocking for any length of time you have other problems.
But if you are sampling four times a second as part of the countdown, then I would expect a large inaccuracy to accrue.
So you may need to keep the two activities (sampling and countdown) separate, or maintain the elapsed time manually.
Note that when your web app does not have focus, the accuracy of timers degrades.
After starting to play audio from an AudioBuffer using AudioBufferSourceNode, how do I make it stop after a predetermined time in milliseconds? My setup is simple:
// create audio source
var source = audioCtx.createBufferSource();
// play audio from source
source.start();
Using AudioBufferSourceNode.stop() I can only specify this duration in seconds, but supplying anything non-integer will evaluate to zero for some reason, and make the audio stop immediately:
source.stop(0.25); // should be 250 milliseconds, but audio stops immediately
setTimeout is not precise enough, and occasional jittering/drifting occurs, especially if the UI works a lot, or other calculations are being performed elsewhere.
setTimeout(function () {
source.stop(); // very inaccurate
}, 250);
Is there a trivial solution to stop playing an audio after a certain amount of milliseconds, without resorting to hacks like busy wait in a worker thread, or similar?
I think you've misunderstood what the first argument to stop() does.
It represents the amount of time relative to the audio context time, which means that when calling stop(), you need to add that time to it:
source.stop(context.currentTime + 0.25);
Since the function accepts a double, it shouldn't be rounding it to the nearest second. According to the Web Audio API specification:
The when parameter describes at what time (in seconds) the sound should stop playing. It is in the same time coordinate system as the AudioContext's currentTime attribute. If 0 is passed in for this value or if the value is less than currentTime, then the sound will stop playing immediately.
Currently i have 2 animations in two rAF loops. For performance reasons i'm considering moving both to 1 loop. The constantly running animation is a small 3d wireframe in the site header, this is not stoppable and is present the moment the site is loaded. However i also have oscilloscopes and spectrum analysers that are only requesting animation frames when audio is played. The moment i pause, cancelAnimationFrame is called.
I have read that 1 rAF is usually better, but i'm having some second thoughts how i should achieve this. Should the constantly running rAF loop check 60 times a second for a variable having a boolean value, and when the value is true it should also run a function that holds the user triggered animations? Or are there any better solutions? What are your experiences?
I have a server application which renders a 30 FPS video stream then encodes and muxes it in real-time into a WebM Byte Stream.
On the client side, an HTML5 page opens a WebSocket to the server, which starts generating the stream when connection is accepted. After the header is delivered, each subsequent WebSocket frame consists of a single WebM SimpleBlock. A keyframe occurs every 15 frames and when this happens a new Cluster is started.
The client also creates a MediaSource, and on receiving a frame from the WS, appends the content to its active buffer. The <video> starts playback immediately after the first frame is appended.
Everything works reasonably well. My only issue is that the network jitter causes the playback position to drift from the actual time after a while. My current solution is to hook into the updateend event, check the difference between the video.currentTime and the timecode on the incoming Cluster and manually update the currentTime if it falls outside an acceptable range. Unfortunately, this causes a noticeable pause and jump in the playback which is rather unpleasant.
The solution also feels a bit odd: I know exactly where the latest keyframe is, yet I have to convert it into a whole second (as per the W3C spec) before I can pass it into currentTime, where the browser presumably has to then go around and find the nearest keyframe.
My question is this: is there a way to tell the Media Element to always seek to the latest keyframe available, or keep the playback time synchronised with the system clock time?
network jitter causes the playback position to drift
That's not your problem. If you are experiencing drop-outs in the stream, you aren't buffering enough before playback to begin with, and playback just has an appropriately sized buffer, even if a few seconds behind realtime (which is normal).
My current solution is to hook into the updateend event, check the difference between the video.currentTime and the timecode on the incoming Cluster
That's close to the correct method. I suggest you ignore the timecode of incoming cluster and instead inspect your buffered time ranges. What you've received on the WebM cluster, and what's been decoded are two different things.
Unfortunately, this causes a noticeable pause and jump in the playback which is rather unpleasant.
How else would you do it? You can either jump to realtime, or you can increase playback speed to catch up to realtime. Either way, if you want to catch up to realtime, you have to skip in time to do that.
The solution also feels a bit odd: I know exactly where the latest keyframe is
You may, but the player doesn't until that media is decoded. In any case, keyframe is irrelevant... you can seek to non-keyframe locations. The browser will decode ahead of P/B-frames as required.
I have to convert it into a whole second (as per the W3C spec) before I can pass it into currentTime
That's totally false. The currentTime is specified as a double. https://www.w3.org/TR/2011/WD-html5-20110113/video.html#dom-media-currenttime
My question is this: is there a way to tell the Media Element to always seek to the latest keyframe available, or keep the playback time synchronised with the system clock time?
It's going to play the last buffer automatically. You don't need to do anything. You're doing your job by ensuring media data lands in the buffer and setting playback as close to that as reasonable. You can always advance it forward if a network condition changes that allows you to do this, but frankly it sounds as if you just have broken code and a broken buffering strategy. Otherwise, playback would be simply smooth.
Catching up if fallen behind is not going to happen automatically, and nor should it. If the player pauses due to the buffer being drained, a buffer needs to be built back up again before playback can resume. That's the whole point of the buffer.
Furthermore, your expectation of keeping anything in-time with the system clock is not a good idea and is unreasonable. Different devices have different refresh rates, will handle video at different rates. Just hit play and let it play. If you end up being several seconds off, go ahead and set currentTime, but be very confident of what you've buffered before doing so.