everybody.
I have a project which envolves object detection (YOLOv6) with video on web-site. Web-site is a Flask app. So site looks like this:
It workes like this:
Select video or type youtube video-link
Video get uploaded on server(Flask) side and gets processed.
Results should be seen in video player.
For video to get processed, it's need to be split into frames (done with cv2), feed to object detection model and get processed frame. Then take VideoWriter and write everything to new file and send it to site with make_response. I used some optimization like process each 3rd frame. Code looks like this:
#app.route('/api/v1/compute/', methods=['GET'])
async def compute():
vid = cv2.VideoCapture(pat)
fourcc = cv2.VideoWriter_fourcc(*'vp09')
out = cv2.VideoWriter('res.mp4', fourcc, 20.0, (int(vid.get(3)), int(vid.get(4))))
i=0
frms = []
while vid.isOpened():
ret, img = vid.read()
if not ret: break
if i%3==0:
img = process(img)
frms.append(img)
for fr in frms:
out.write(fr)
out.release()
But VideoWriter is very slow + saving new video can take a lot of space. So I came up with two possible ideas that skips video-saving part:
Send encoded processed frame to client (using socket-io). Encoded - in base64. Then collect encoded frames in array on client-side using javascript and write array. Then convert everything to video in base64.
Collect processed frames and convert them into video, but without saving it. Then encode it and send encoded (base64) video to client-side.
Now my question: Which is better? Original, 1st or 2nd variant? :
If original is better, then how to make VideoWriter to work faster or how to replace it with smth faster?
If 1st, then how to convert base64 frames to base64 video?
If 2nd, then how to convert frames to video with saving it?
Related
I desire to play an audio-book in my web-page. The audio book is a .zip file, which contains multiple .mp3 files, having one for each chapter of the book. The run time of all the files is several hours, and the their cumulative size is 60MB. The .zip is stored server-side (Express.js)
How can I play each file in succession in the client (in an <audio> element for instance), so that the audio-book plays smoothly, as if 1 file?
Do I need to use a MediaStream object? If so, how?
-Thanks
I'd take a look at this answer on another Stack Overflow question however I have made some modifications in order to match your question:
var audioFileURLs= [];
function preloadAudio(url) {
var audio = new Audio();
// once this file loads, it will call loadedAudio()
// the file will be kept by the browser as cache
audio.addEventListener('canplaythrough', loadedAudio, false);
audio.src = url;
}
var loaded = 0;
function loadedAudio() {
// this will be called every time an audio file is loaded
// we keep track of the loaded files vs the requested files
loaded++;
if (loaded == audioFileURLs.length){
// all have loaded
init();
}
}
var player = document.getElementById('player');
function play(index) {
player.src = audioFiles[index];
player.play();
}
function init() {
// do your stuff here, audio has been loaded
// for example, play all files one after the other
var i = 0;
// once the player ends, play the next one
player.onended = function() {
i++;
if (i >= audioFiles.length) {
// end
return;
}
play(i);
};
// play the first file
play(i);
}
// call node/express server to get a list of links we can hit to retrieve each audio file
fetch('/getAudioUrls/#BookNameOrIdHere#')
.then(r => r.json())
.then(arrayOfURLs => {
audioFileURLs = arrayOfURLs
arrayOfURLs.map(url => preloadAudio(URL))
})
And then just have an audio element on the screen with the id of "player" like <audio id="player"></audio>
With this answer though, the arrayOfURLs array must contain URLs to an API on your server that will open the zip file and return the specified mp3 data. You may also just want to take this answer as a general reference, and not a complete solution because there is optimization to be done. You should probably only load the first audio file at first, and 5 minutes or so before the first file ends you may want to start pre-loading the next and then repeat this process for the entire thing... That all will be up to you but this should hopefully put you on your feet.
You may also run into an issue with the audio element though because it will only show the length of the current audio segment it is on, and not the full length of the audiobook. I would choose to believe this zip file has the book separated by chapter correct? If so you could create a chapter selector, that pretty much allows you to jump to a specific chapter aka getAudioUrls URL.
I hope this helps!
One other note for you... reading your comment on a potential answer down below, you could combine all the audio files into one using some sort of node module (audioconcat is one I found after a quick google search) and return that one file to the client. However, I would not personally take this route because the entire audiobook will be in the server's memory while it combines them, and until it returns it to the client. This could cause some memory issues down the road, so I would avoid it if I could. However, I will admit that this option could be potentially nice because the full length of the audiobook will display in the audio elements timeline.
The best option perhaps is to store the books full length and chapter lengths in a details.json file in the zip file and send that to the client in the first API call along with the URLs to each audio file. This would enable you to build a nice UI.
The only options I can think of is to use either use a javascript mp3 decoder (or compiled a C decoder to asm.js/wasm) and use the audio APIs. Or wrap the mp3 in an mp4 using something like mux.js and use media source extensions to playback.
maybe this will help you
<audio controls="controls">
<source src="track.ogg" type="audio/ogg" />
<source src="track.mp3" type="audio/mpeg" />
Your browser does not support the audio element.
</audio>
I have an array of Blobs (binary data, really -- I can express it however is most efficient. I'm using Blobs for now but maybe a Uint8Array or something would be better). Each Blob contains 1 second of audio/video data. Every second a new Blob is generated and appended to my array. So the code roughly looks like so:
var arrayOfBlobs = [];
setInterval(function() {
arrayOfBlobs.append(nextChunk());
}, 1000);
My goal is to stream this audio/video data to an HTML5 element. I know that a Blob URL can be generated and played like so:
var src = URL.createObjectURL(arrayOfBlobs[0]);
var video = document.getElementsByTagName("video")[0];
video.src = src;
Of course this only plays the first 1 second of video. I also assume I can trivially concatenate all of the Blobs currently in my array somehow to play more than one second:
// Something like this (untested)
var concatenatedBlob = new Blob(arrayOfBlobs);
var src = ...
However this will still eventually run out of data. As Blobs are immutable, I don't know how to keep appending data as it's received.
I'm certain this should be possible because YouTube and many other video streaming services utilize Blob URLs for video playback. How do they do it?
Solution
After some significant Googling I managed to find the missing piece to the puzzle: MediaSource
Effectively the process goes like this:
Create a MediaSource
Create an object URL from the MediaSource
Set the video's src to the object URL
On the sourceopen event, create a SourceBuffer
Use SourceBuffer.appendBuffer() to add all of your chunks to the video
This way you can keep adding new bits of video without changing the object URL.
Caveats
The SourceBuffer object is very picky about codecs. These have to be declared, and must be exact, or it won't work
You can only append one blob of video data to the SourceBuffer at a time, and you can't append a second blob until the first one has finished (asynchronously) processing
If you append too much data to the SourceBuffer without calling .remove() then you'll eventually run out of RAM and the video will stop playing. I hit this limit around 1 hour on my laptop
Example Code
Depending on your setup, some of this may be unnecessary (particularly the part where we build a queue of video data before we have a SourceBuffer then slowly append our queue using updateend). If you are able to wait until the SourceBuffer has been created to start grabbing video data, your code will look much nicer.
<html>
<head>
</head>
<body>
<video id="video"></video>
<script>
// As before, I'm regularly grabbing blobs of video data
// The implementation of "nextChunk" could be various things:
// - reading from a MediaRecorder
// - reading from an XMLHttpRequest
// - reading from a local webcam
// - generating the files on the fly in JavaScript
// - etc
var arrayOfBlobs = [];
setInterval(function() {
arrayOfBlobs.append(nextChunk());
// NEW: Try to flush our queue of video data to the video element
appendToSourceBuffer();
}, 1000);
// 1. Create a `MediaSource`
var mediaSource = new MediaSource();
// 2. Create an object URL from the `MediaSource`
var url = URL.createObjectURL(mediaSource);
// 3. Set the video's `src` to the object URL
var video = document.getElementById("video");
video.src = url;
// 4. On the `sourceopen` event, create a `SourceBuffer`
var sourceBuffer = null;
mediaSource.addEventListener("sourceopen", function()
{
// NOTE: Browsers are VERY picky about the codec being EXACTLY
// right here. Make sure you know which codecs you're using!
sourceBuffer = mediaSource.addSourceBuffer("video/webm; codecs=\"opus,vp8\"");
// If we requested any video data prior to setting up the SourceBuffer,
// we want to make sure we only append one blob at a time
sourceBuffer.addEventListener("updateend", appendToSourceBuffer);
});
// 5. Use `SourceBuffer.appendBuffer()` to add all of your chunks to the video
function appendToSourceBuffer()
{
if (
mediaSource.readyState === "open" &&
sourceBuffer &&
sourceBuffer.updating === false
)
{
sourceBuffer.appendBuffer(arrayOfBlobs.shift());
}
// Limit the total buffer size to 20 minutes
// This way we don't run out of RAM
if (
video.buffered.length &&
video.buffered.end(0) - video.buffered.start(0) > 1200
)
{
sourceBuffer.remove(0, video.buffered.end(0) - 1200)
}
}
</script>
</body>
</html>
As an added bonus this automatically gives you DVR functionality for live streams, because you're retaining 20 minutes of video data in your buffer (you can seek by simply using video.currentTime = ...)
Adding to the previous answer...
make sure to add sourceBuffer.mode = 'sequence' in the MediaSource.onopen event handler to ensure the data is appended based on the order it is received. The default value is segments, which buffers until the next 'expected' timeframe is loaded.
Additionally, make sure that you are not sending any packets with a data.size === 0, and make sure that there is 'stack' by clearing the stack on the broadcasting side, unless you are wanting to record it as an entire video, in which case just make sure the size of the broadcast video is small enough, and that your internet speed is fast. The smaller and lower the resolution the more likely you can keep a realtime connection with a client, ie a video call.
For iOS the broadcast needs to made from a iOS/macOS application, and be in mp4 format. The video chunk gets saved to the app's cache and then removed once it is sent to the server. A client can connect to the stream using either a web browser or app across nearly any device.
I have an ajax application where the PHP side sends uncoded raw image data, coming from a camera, to the client javascript side.
I would like to display the image via html and javascript using img or canvas tags.
Image data are 32*32 unsigned char.
I would like to do, whatever it takes to reach my goal (encoding or everythig else), but I want to do it at client side beacause I cannot handle any other operation at the server side.
I tried to encode raw data into jpeg or png data but without success.
I post an example that doesn't work:
var encoder = new JPEGEncoder(9);
var jpgFile = encoder.encode(rawImage, 9);
document.getElementById("image").src = jpgFile;
jpgFile is like
data:image/jpeg;base64,/9j/4AAQSkZJRgABAQAAAQABAAD/2wCEAFk9Q05DOFlOSE5kXllphd6QhXp6hf/CzaHe//////////////////////////////////////////////////8BXmRkhXWF/5CQ///////////////////////////////////////////////////////////////////////////AABEIAAAAAAMBEQACEQEDEQH/xAGiAAABBQEBAQEBAQAAAAAAAAAAAQIDBAUGBwgJCgsQAAIBAwMCBAMFBQQEAAABfQECAwAEEQUSITFBBhNRYQcicRQygZGhCCNCscEVUtHwJDNicoIJChYXGBkaJSYnKCkqNDU2Nzg5OkNERUZHSElKU1RVVldYWVpjZGVmZ2hpanN0dXZ3eHl6g4SFhoeIiYqSk5SVlpeYmZqio6Slpqeoqaqys7S1tre4ubrCw8TFxsfIycrS09TV1tfY2drh4uPk5ebn6Onq8fLz9PX29/j5+gEAAwEBAQEBAQEBAQAAAAAAAAECAwQFBgcICQoLEQACAQIEBAMEBwUEBAABAncAAQIDEQQFITEGEkFRB2FxEyIygQgUQpGhscEJIzNS8BVictEKFiQ04SXxFxgZGiYnKCkqNTY3ODk6Q0RFRkdISUpTVFVWV1hZWmNkZWZnaGlqc3R1dnd4eXqCg4SFhoeIiYqSk5SVlpeYmZqio6Slpqeoqaqys7S1tre4ubrCw8TFxsfIycrS09TV1tfY2dri4+Tl5ufo6ery8/T19vf4+fr/2gAMAwEAAhEDEQA/AP8A/9k=
I got JPEGEncoder googling on internet.
Tried it for you =)
http://jsfiddle.net/bYGum/
All you have to do is make sure your string is base64 encoded and do the exact same thing you have already written:
document.getElementById("image").src = jpgFile;
In my node.js code, there is a buffer array to which I store the contents of a received image, every time an image being received via TCP connection between the node.js as TCP client and another TCP server:
var byBMP = new Buffer (imageSize);
However
the size of image,imageSize, differs every time which makes the size of byBMP change accordingly. That means something like this happen constantly:
var byBMP = new Buffer (10000);
.... wait to receive another image
byBMP = new Buffer (30000);
.... wait to receive another image
byBMP = new Buffer (10000);
... etc
Question:
Is there any more efficient way of resizing the array byBMP. I looked into this: How to empty an array in JavaScript? which gave me some ideas ebout the efficient way of emptying an array, however for resizing it I need you guys' comments.node.js
The efficient way would be to use Streams rather than a buffer. The buffer will keep the content of the image in your memory which is not very scalable.
Instead, simply pipe the download stream directly to a file output:
imgDownloadStream.pipe( fs.createWriteStream('tmp-img') );
If you're keeping the img in memory to perform transformation, then simply apply the transformation on the stream buffers parts as they pass through.
I'm trying to render MJpeg stream in HTML5 using the img tag.
When I'm running the following, everything works great, meaning, the video starts to play until the video ends:
<img src="http://[some ip]:[port]/mjpg">
My question is how can I get the stream frame by frame.
For each frame, I want to get it, do something (ajax call to the server) and then display the frame as an image.
Thanks.
You can do this without repeatedly making Http requests. Only one will suffice. You can use the fetch api to create a ReadableStream, access it's Reader and keep reading from the stream.
Once you have the reader keep reading chunks from the stream recursively. Look for the SOI ( 0xFF 0xD8) in the byte stream which signals the end of the header and the beginning of the JPEG frame. The header will contain the length of the JPEG in bytes to be read. Read that many bytes from the chunk and any successive chunks and store it into a Uint8Array. Once you've
successfully read the frame convert it into a blob, create a UrlObject out of it and assign it to the src property of your img object.
Keep doing this till the connection is closed.
Shameless plug. Here's a link to a working sample on github.
If the camera exposes raw JPEG images (not .MJPEG extension) you'll have to reaload it manually (if the extension is .MJPEG the browser will do everything, just put the correct src). If you have .MJPEG and want to use the raw .JPEG check your camera documentation. Most cameras expose both the .MJPEG and raw .JPEG streams (just on different URLs).
Unfortunately you won't be able to easily get the image through ajax, but you could change the src of the image periodically.
You can use Date.getTime() and add it to the querystring to force the browser to reload the image, and repeat each time the image loads.
If you use jQuery the code will look something like this:
camera.html
<!DOCTYPE html>
<html>
<head>
<title>ipCam</title>
</head>
<body>
<h1>ipCam</h1>
<img id="motionjpeg" src="http://user:pass#127.0.0.1:8080/" />
<script src="motionjpeg.js"></script>
<script>
//Using jQuery for simplicity
$(document).ready(function() {
motionjpeg("#motionjpeg"); // Use the function on the image
});
</script>
</body>
</html>
motionjpeg.js
function motionjpeg(id) {
var image = $(id), src;
if (!image.length) return;
src = image.attr("src");
if (src.indexOf("?") < 0) {
image.attr("src", src + "?"); // must have querystring
}
image.on("load", function() {
// this cause the load event to be called "recursively"
this.src = this.src.replace(/\?[^\n]*$/, "?") +
(new Date()).getTime(); // 'this' refers to the image
});
}
Note that my example will play the MotionJPEG on page load, but won't allow play/pause/stop functionalities
If you stream source can't return frames on another address (http://[some ip]:[port]/frame/XXX) then you can use MJPEG stream parser on the server. For example, Paparazzo.js parse stream and return single jpeg. Actually it returns only last frame without saving previous, but it can be changed.
Problem can't be solved only in browser with js without some plugins and server.