I found some examples, where people used a canvas and javascript to take multiple screenshots of a running video.
You can see these examples here or here.
The code sets a time interval, draws the current timeframe to a canvas and uses this to create a screenshot.
I am wondering if it would be possible to use a similar technique, to automatically create a kind of preview for chapters of the video.
But this would require to grab a bunch of screenshots before the video started.
I failed to implement this, so I would like to know, if it is at all possible.
I know that one could use pretaken screenshots for the chapters, but I wanted to automate this process.
Thanks in advance for your answers.
This could be done in theory by jumping to specific times in the video (say every 10 seconds) using video.currentTime, waiting for the frame to be available (using progress events), drawing the frame to a canvas (canvas.drawImage) and storing it in some way (say an array of images having image.src = canvas.toDataURL).
However, this process will take time because at least the relevant parts of the video would need to be loaded in the browser so the frame could be grabbed. The video would not be playable during the process as it is being skipped to different frames.
This behavior is usually not acceptable, but it really depends on your specific use case.
Related
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 6 years ago.
Improve this question
Youtube has this cool feature where you can 'scrub' through the video and they give you the thumbnail view of the video at each second:
Does anybody know how this is achieved? Does the server have to send over each image individually, or can they be extracted from the video on the clients machine with javascript?
Regarding the implementation of the "extraction", I am a bit lost, but I imagine a canvas element could be used to draw the frames once their extracted.
If the server has to be used: since the images are small, I imagine I could use sprites, but if we're talking about every second of video, then that file could get pretty big. (still small in comparison to the video data I suppose, but not exactly optimal)
tl;dr server generated sprite sheets, downloaded as needed.
We can guess going in that the images are generated server side. YouTube does much more intense processing on every video than pulling a few thumbnails. Also, a quick Google suggests this feature has been around for a few years - probably longer than the required HTML5/JS/browser-horsepower to do this client-side.
I hit up Download Tools > Resources in my browser and checked out a newly-posted video from my feed. Interestingly, there were no previews yet (the video was only up for about 20 min when I checked). This indicates the images are probably generated server-side and just haven't finished processing.
Checking an older video and looking at Resources > Images didn't reveal anything interesting. So I switched to Timelines and hit record, then started mousing over the timeline and watching for network traffic. As I moved the mouse, *.jpg files started loading, and they contained 25 thumbnails from a given section of video:
I also noticed that the initial file M0.jpg is the same size image, but contains about 100 thumbnails from across the entire video, rather than 25 thumbnails from one segment. Example:
Testing again with a new video, it looks like the 100-image M0.jpg gets downloaded first and provides the basic lower-res less-granular thumbnail previews. Then, as you mouseover different sections of video, the higher-res M0.jpg, M1.jpg, etc. get downloaded as needed.
Interestingly, this doesn't change for longer videos, which explains why the thumbnails can sometimes suck. If your connection or YouTube is too slow at getting the higher-res thumbnails, then you get stuck with only 100 low-res thumbnails of a really long video. Not sure how this works on shorter videos. Also, it might be interesting to see what distribution they pull the thumbnails from (is it just linear every 1/100th of the video? or something else).
Last tidbit, is I noticed if you use a url with a timecode in it, you don't get a full 100-image M0.jpg sheet but rather a completely different size M#.jpg containing about 25 low-res thumbnails from the timecode to the end of the video.
I guess they're assuming that when people link to a specific timecode, users aren't likely to jump to an earlier point in the video. Also, this is way less granularity than the ~75 images you would've gotten by sending the normal 100-image M0.jpg. On the other hand, it's also about 30% of the size, so maybe speed was that important.
As for generating the thumbnails, ffmpeg is a good way to go:
To make multiple screenshots and place them into a single image file (creating tiles), you can use FFmpeg's tile video filter, like this:
ffmpeg -ss 00:00:10 -i movie.avi -frames 1 -vf "select=not(mod(n\,1000)),scale=320:240,tile=2x3" out.png
That will seek 10 seconds into the movie, select every 1000th frame, scale it to 320x240 pixels and create 2x3 tiles in the output image out.png, which will look like this:
In HLS this is done with #EXT-X-I-FRAMES-ONLY playlists (HLS v4) and is called Trick Play. Hopefully that helps with your research.
https://developer.apple.com/library/ios/technotes/tn2288/_index.html#//apple_ref/doc/uid/DTS40012238-CH1-I_FRAME_PLAYLIST
I am gonna use my webcam as a source and show my view on webpage , than I will manipulate my view like (blacknwhite , fiseye, etc.) and show that manipulated video in my canvas.
An example ( http://photobooth.orange-coding.net/ )
Ok everything is cool for now . I can capture that manipulated canvas as a image.
Is there any way to record that manipulated canvas as video?
I also found an example (https://www.webrtc-experiment.com/ffmpeg/audio-plus-canvas-recording.html)
But when I tried that code on my webcam recording project , it's just recording my source view(not blacknwhite) . It is not implementing my effect to record.
Any idea or is it possible ?
Thank you.
Recording video in the browser is like getting blood out of a stone. If you hit it hard and long enough against your head, there will be blood, eventually. But it's a painful experience, you it will certainly give you a headache!
There is currently no way of recording video in real-time from a canvas element. But there is proposed a Mediastream Recording API which includes video (and it excludes the canvas part). Currently only audio is supported, and only if FF.
You can grab an image as often as possible and use it as a sequence, but there are several issues you will run into:
You will not get full frame-rate if you choose to grab the image as JPEG or PNG (PNG is not very useful with video as there is no alpha)
If you choose to grab the raw data you may achieve full frame rate (note that frame rate for video is typically never above 30 FPS) but you will fill up the memory very quickly, and you would need a point in time to process the frames into something that can be transferred to server or downloaded. JavaScript is single threaded and no matter how you twist and turn this stage, you will get gaps in the video when this process is invoked (unless you have a lot of memory and can wait until the end - but this not good for a public available solution if that's the goal).
You will have no proper sinc like time-code (to sync by) so the video will be like the movies from Chaplins day, variable. You can get close by binding high-resolution timestamps but not accurate enough as you will have no way of getting the stamp at the very time you grab the image.
No sound is recorded; if you do record audio in FF using the API, you have no way to properly sync the audio with the video anyways (which already has its own problems ref. above)
Up until now we are still at single frame sequences. If you record one minute # 30 fps you have 60x30 frames, or 1800 pictures/buffers per minute. If you record in HD720 and choose grabbing the raw buffer (the most realistic option here) you will end up with 1800 x 1280 x 720 x 4 (RGBA) bytes per minute, or 6,635,520,000 bytes, ie. 6.18 GB per minute - and that's just in raw size. Even if you lower the resolution to lets say 720x480 you'll end up with 2.32 GB/min.
You can alternatively process them into a video format, it's possible, but currently there are next to none solutions for this (there has been one, but it had varying result which is probably why it's hard be found...), so you are left to this yourselves - and that is a complete project involving writing encoder, compressor etc. And the memory usage will be quite high as you need to create each frame in separate buffers until you know the full length, then create a storage buffer to hold them all and so forth. And even if you did, compressing more than 6 GB worth of data (or event "just" 2 GB) is not gonna make user or browser very happy (if there is any memory left)...
Or bite the dust and go with a commercial Flash based solution (but that excludes your image processing and pretty much takes over the camera... so not really an option in this case).
The only realistic option, IMO, is to wait for the aforementioned API - this will let your browser do all the hard work, in compiled optimized code, enable frame by frame compression leaving the memory pretty much intact, and give very little headache compared to the alternative(s) above. There may be an option to apply shaders to the stream at one point, or integrate it with some canvas processing (not on the table in this proposal AFAICS) so recording real-time from a canvas will still be a challenge.
This is where server side processing comes in...
(of course, a screen recorder is an option which is of curse completely non-integrated, but will enable you to demo your effects at least...).
Is there any way to have two or more (preferably three) html5 < video > tags playing simultaneously and to be in perfect sync.
If I have let's say three tiles of one video and I want them to appear in browser as one big video. They need to be perfectly synchronized. Without even smallest visual/vertical hint that they are tiled.
Unfortunately I cannot use MediaController because it is not supported well enough.
I've tried some workouts, including canvases, but I still get visual differentiation. Has anyone had any similar problem/solution?
Disclaimer: I'm not a video guy, but here are some thoughts anyway.
If they need to be absolutely perfect...you are fighting several problems at once:
A device might not be powerful enough to acquire, synchronize and render 3 streams at once.
Even if #1 is solved, a device is never totally dedicated to your task. For example, it might pause for garbage collection between processing stream#1 and stream#2--resulting in dropped/unsynchronized frames.
So to give yourself the best chance at perfection, you should first merge your 3 videos into 1 vertical video in the studio (or using studio software).
Then you can use the extended clipping properties of canvas context.drawImage to break each single frame into 2-3 separate frames.
Additionally, buffer a few frames you acquire on the stream (this goes without saying!).
Use requestAnimationFrame (RAF) to control the drawing. RAF does a fairly good job of drawing frames when system resources are available and delaying frames when system resources are lacking.
Your result won't be perfect, but they will be synchronized. You will always have to make the decision whether to drop or delay frames when system resources are unavailable, but at least the frames you do present will be synchronized.
As far as I know it's currently impossible to play HTML5 video frame-by-frame, or seek to a frame accurate time-code. The nearest seek seems to be precise to roughly 1-second.
But you can still get pretty close using the some of the media frameworks:
Popcorn.js library made for synchronizing video with content.
mediagroup.js another library used to add support for mediagroup attributes on HTML5 media elements
The only feature that allowed that is named mediaGroup and it was removed from Chrome(apparently for not being popular enough). It's still present in WebKit. Relevant discussion here and here.
I think you can implement you own "mediagroup"-like tag using wasm though without DOM support it may be tricky.
I know that it's possible to take a series of images and use javascript to set opacity from 1 to 0 very quickly for each one in rapid secession. I have actually done it before very successfully, although I only used 41 images at approx. 720p.
My question though, is whether or not it would be practical to make an entire video (4-10 minutes long) using only html, css, and javascript. Obviously that would be too many images to leave in the cache, so you'd have to empty the cache every so often of particular images, and also this would require a pretty good internet connection, but do you think it would be worth while to attempt?
are there any obvious pro's and cons that you could think of for trying this?
(to be clear, I'm not asking for the code to achieve it as I have already developed most of it, just what you think in terms of practicality of it as opposed to youtube or vimeo etc...)
Plainly put, I don't think it's practical.
Here's a few reasons why:
Number of images. Videos tend to be around 30 frames per second.
For a 4 min video, that's roughly 7,200 images.
Download time. Downloading 7,200 images at 720 pixels high would require a
significant amount of bandwidth, not only for the user, but also for
the server.
DOM load. 7,200 images would require just as many DOM
elements positioned correctly behind each other.
Rendering. Each
time we do an animation (fadeout), the browser has to calculate what
element is visible and what that means to the user (pixel color,
etc.)
Of course, we can optimize this:
Download the images that is needed at that specified time. If the connection can download 30 images/second, then we load what we need and then initiate the play sequence.
Remove elements when it's done. As the sequence goes on, we can assume that those images are no longer needed. When those images are done, we can destroy those elements and thus free up a few resources.
Put images on multiple sub-domains/locations. Browsers tend to download 1 asset per domain at a single moment. Modern browsers may download 6 or more. But if we were to split those assets to multiple subdomains, we can increase that number of simultaneous downloads and thus increase response time.
Create large sprites. Have the sequence on one or several large sprites, and use Javascript to correctly position the element. This allows us to have a single/few downloads and removes the overhead from multiple downloads.
Videos are encoded so that the browser doesn't have to do all those pixel calculations and animations. Each frame isn't a full image, but rather a delta between the current and next frame. (I'm overly simplifying this.)
But this method is constantly used for pieces that need interactivity, or want to get past iOS auto-play hangups. Again, not practical for large sequences, but definitely doable for shorter sequences.
Does anyone know why Javascript performance would be affected by the loading of lots of external JPG/PNG images into HTML5 Image() objects, totalling approx 200Mb-250Mb. Performance also seems to be affected by cache. Ie. if the cache is full(-ish) from previous browsing the performance on the current site is greatly reduced.
There are 2 says i can crudely solve it.
clear cache manually.
minimise browser, wait about 20 secs and re-open the browser after which time the iOS/browser has reclaimed the memory and runs the JS as it should.
I would have expected the iOS to reclaim required memory to run the current task, but it seems not. Another workaround is to load 200Mb of 'cache clearing' images into Image() objects, then remove these by setting the src = "". This does seem to help, but its not an elegant solution...
please help?
First and foremost read the excellent post on LinkedIn Engineering blog. Read it carefully and check if there are some optimizations that you can also try in your application. If you tried all of them and that still haven't solved your performance issues read on.
I assume that you have some image gallery or magazine-style content area on your page
How about having this image area in a separate iframe? What you could do then is this:
Have two iframes. Only one should be visible and active in time.
Load images into first iframe. Track the size of loaded images. If exact size tracking is hard
numberOfLoadedImages * averageImageSize
might be a pretty good aproximation.
As that number approaches some thresshold start preloading the currently visible content into second iframe.
Flip the visibility of iframes so the second one becomes active.
Clear the inner content of the first frame.
Repeat the whole procedure as necessary.
I don't know for sure if this would work for you but I hope that WebKit engine on iPad clears the memory of frames independently.
EDIT: It turned out you're writing a game.
If it's a game I assume that you want to have many game objects on the screen at the same time and you won't be able to simply unload some parts of them. Here are some suggestions for that case:
Don't use DOM for games: it's too memory-heavy. Fortunately, you're using canvas already.
Sprite your images. Image sprites not only help reducing the number of requests. They also let you reduce the number of Image objects and keep the per-file overhead lower. Read about using sprites for canvas animations on IE blog.
Optimize your images. There are several file size optimizers for images. SmushIt is one of them. Try it for your images. Pay attention to other techniques discussed in this great series by Stoyan Stefanov at YUI blog.
Try vector graphics. SVG is awesome and canvg can draw it on top of canvas.
Try simplifying your game world. Maybe some background objects don't need to be that detailed. Or maybe you can get away with fewer sprites for them. Or you can use image filters and masks for different objects of the same group. Like Dave Newton said iPad is a very constrained device and chances are you can get away with a relatively low-quality sprites.
These were all suggestions related to reduction of data you have to load. Some other suggestions that might work for you.
Preload images that you will need and unload images that you no longer need. If your game has "levels" or "missions" load sprites needed only for current one.
Try loading "popular" images first and download the remaining once in background. You can use separate <iframe> for that so your main game loop won't be interrupted by downloads. You can also use cross-frame messaging in order to coordinate your downloader frame.
You can store the very most popular images in localStorage, Application Cache and WebSQL. They can provide you with 5 mb of storage each. That's 15 megs of persistent cache for you. Note that you can use typed arrays for localStorage and WebSQL. Also keep in mind that Application Cache is quite hard to work with.
Try to package your game as a PhoneGap application. This way you can save your users from downloading a huge amount of data before playing the game. 200 megs as a single download just to open a page is way too much. Most people won't even bother to wait for that.
Other than that your initial suggestion to override cache with your images is actually valid. Just don't do it straight away. Explore the possibilities to reduce the download size for your game instead.
I managed to reduce the impact by setting all the images that aren't currently in the viewport to display:none. This was with background images though and I haven't tested over 100Mb of images, so can't say whether this truly helps. But definitely worth of trying.