Rendering MJpeg stream in html5 - javascript

I'm trying to render MJpeg stream in HTML5 using the img tag.
When I'm running the following, everything works great, meaning, the video starts to play until the video ends:
<img src="http://[some ip]:[port]/mjpg">
My question is how can I get the stream frame by frame.
For each frame, I want to get it, do something (ajax call to the server) and then display the frame as an image.
Thanks.

You can do this without repeatedly making Http requests. Only one will suffice. You can use the fetch api to create a ReadableStream, access it's Reader and keep reading from the stream.
Once you have the reader keep reading chunks from the stream recursively. Look for the SOI ( 0xFF 0xD8) in the byte stream which signals the end of the header and the beginning of the JPEG frame. The header will contain the length of the JPEG in bytes to be read. Read that many bytes from the chunk and any successive chunks and store it into a Uint8Array. Once you've
successfully read the frame convert it into a blob, create a UrlObject out of it and assign it to the src property of your img object.
Keep doing this till the connection is closed.
Shameless plug. Here's a link to a working sample on github.

If the camera exposes raw JPEG images (not .MJPEG extension) you'll have to reaload it manually (if the extension is .MJPEG the browser will do everything, just put the correct src). If you have .MJPEG and want to use the raw .JPEG check your camera documentation. Most cameras expose both the .MJPEG and raw .JPEG streams (just on different URLs).
Unfortunately you won't be able to easily get the image through ajax, but you could change the src of the image periodically.
You can use Date.getTime() and add it to the querystring to force the browser to reload the image, and repeat each time the image loads.
If you use jQuery the code will look something like this:
camera.html
<!DOCTYPE html>
<html>
<head>
<title>ipCam</title>
</head>
<body>
<h1>ipCam</h1>
<img id="motionjpeg" src="http://user:pass#127.0.0.1:8080/" />
<script src="motionjpeg.js"></script>
<script>
//Using jQuery for simplicity
$(document).ready(function() {
motionjpeg("#motionjpeg"); // Use the function on the image
});
</script>
</body>
</html>
motionjpeg.js
function motionjpeg(id) {
var image = $(id), src;
if (!image.length) return;
src = image.attr("src");
if (src.indexOf("?") < 0) {
image.attr("src", src + "?"); // must have querystring
}
image.on("load", function() {
// this cause the load event to be called "recursively"
this.src = this.src.replace(/\?[^\n]*$/, "?") +
(new Date()).getTime(); // 'this' refers to the image
});
}
Note that my example will play the MotionJPEG on page load, but won't allow play/pause/stop functionalities

If you stream source can't return frames on another address (http://[some ip]:[port]/frame/XXX) then you can use MJPEG stream parser on the server. For example, Paparazzo.js parse stream and return single jpeg. Actually it returns only last frame without saving previous, but it can be changed.
Problem can't be solved only in browser with js without some plugins and server.

Related

Display pdf file over 2MB using embed element

I am making an application that brings up a preview of PDF files. Embedding with an embed element works well for small PDF files but fails for larger PDF files because of the size limits for data urls. I'm looking for a way to use the browser's native PDF viewer to view PDF files but without using data urls.
My code currently looks something like the following:
<script>
function addToCard(input) {
if (input.files.length <= 0) return;
let fileReader = new FileReader();
fileReader.onload = async function () {
pdfCard.src = fileReader.result;
};
fileReader.readAsDataURL(input.files[0]);
}
</script>
<input type=file oninput="addToCard(this)" />
<embed id=pdfCard style="width:100%;height:100%" />
Example. The original PDF is here.
You could use URL.createObjectURL() on the PDF. It also creates a URL representing the object; however, the difference between an object URL and a data URL is that, while a data URL contains the object itself, an object URL is a reference to the object, which is stored in memory. This means that object URLs are significantly shorter than data URLs and take less time to create.
There are two drawbacks to this approach that may prevent you from using it. The first is that an object URL will only work on the page on which it was created. Attempting to use an object URL on a different page will not work. If you need to access this URL anywhere other than the page it was created on, this approach will not work.
The second is that object URLs keep the object for which they were created stored in memory. You have to revoke the object URL when you are done using it with the URL.revokeObjectURL() method, otherwise it will cause a memory leak. This means that you might have to add some extra code that revokes the object URL once the PDF is loaded. This example may be helpful.
The implementation might look something like this:
function addToCard(input) {
if (input.files.length <= 0) return;
pdfCard.src = URL.createObjectURL(input.files[0])
// gonna have to call revokeObjectURL eventually...
}

Optimising video sending-recieving with Flask app

everybody.
I have a project which envolves object detection (YOLOv6) with video on web-site. Web-site is a Flask app. So site looks like this:
It workes like this:
Select video or type youtube video-link
Video get uploaded on server(Flask) side and gets processed.
Results should be seen in video player.
For video to get processed, it's need to be split into frames (done with cv2), feed to object detection model and get processed frame. Then take VideoWriter and write everything to new file and send it to site with make_response. I used some optimization like process each 3rd frame. Code looks like this:
#app.route('/api/v1/compute/', methods=['GET'])
async def compute():
vid = cv2.VideoCapture(pat)
fourcc = cv2.VideoWriter_fourcc(*'vp09')
out = cv2.VideoWriter('res.mp4', fourcc, 20.0, (int(vid.get(3)), int(vid.get(4))))
i=0
frms = []
while vid.isOpened():
ret, img = vid.read()
if not ret: break
if i%3==0:
img = process(img)
frms.append(img)
for fr in frms:
out.write(fr)
out.release()
But VideoWriter is very slow + saving new video can take a lot of space. So I came up with two possible ideas that skips video-saving part:
Send encoded processed frame to client (using socket-io). Encoded - in base64. Then collect encoded frames in array on client-side using javascript and write array. Then convert everything to video in base64.
Collect processed frames and convert them into video, but without saving it. Then encode it and send encoded (base64) video to client-side.
Now my question: Which is better? Original, 1st or 2nd variant? :
If original is better, then how to make VideoWriter to work faster or how to replace it with smth faster?
If 1st, then how to convert base64 frames to base64 video?
If 2nd, then how to convert frames to video with saving it?

Playing audio broken into multiple files in webpage

I desire to play an audio-book in my web-page. The audio book is a .zip file, which contains multiple .mp3 files, having one for each chapter of the book. The run time of all the files is several hours, and the their cumulative size is 60MB. The .zip is stored server-side (Express.js)
How can I play each file in succession in the client (in an <audio> element for instance), so that the audio-book plays smoothly, as if 1 file?
Do I need to use a MediaStream object? If so, how?
-Thanks
I'd take a look at this answer on another Stack Overflow question however I have made some modifications in order to match your question:
var audioFileURLs= [];
function preloadAudio(url) {
var audio = new Audio();
// once this file loads, it will call loadedAudio()
// the file will be kept by the browser as cache
audio.addEventListener('canplaythrough', loadedAudio, false);
audio.src = url;
}
var loaded = 0;
function loadedAudio() {
// this will be called every time an audio file is loaded
// we keep track of the loaded files vs the requested files
loaded++;
if (loaded == audioFileURLs.length){
// all have loaded
init();
}
}
var player = document.getElementById('player');
function play(index) {
player.src = audioFiles[index];
player.play();
}
function init() {
// do your stuff here, audio has been loaded
// for example, play all files one after the other
var i = 0;
// once the player ends, play the next one
player.onended = function() {
i++;
if (i >= audioFiles.length) {
// end
return;
}
play(i);
};
// play the first file
play(i);
}
// call node/express server to get a list of links we can hit to retrieve each audio file
fetch('/getAudioUrls/#BookNameOrIdHere#')
.then(r => r.json())
.then(arrayOfURLs => {
audioFileURLs = arrayOfURLs
arrayOfURLs.map(url => preloadAudio(URL))
})
And then just have an audio element on the screen with the id of "player" like <audio id="player"></audio>
With this answer though, the arrayOfURLs array must contain URLs to an API on your server that will open the zip file and return the specified mp3 data. You may also just want to take this answer as a general reference, and not a complete solution because there is optimization to be done. You should probably only load the first audio file at first, and 5 minutes or so before the first file ends you may want to start pre-loading the next and then repeat this process for the entire thing... That all will be up to you but this should hopefully put you on your feet.
You may also run into an issue with the audio element though because it will only show the length of the current audio segment it is on, and not the full length of the audiobook. I would choose to believe this zip file has the book separated by chapter correct? If so you could create a chapter selector, that pretty much allows you to jump to a specific chapter aka getAudioUrls URL.
I hope this helps!
One other note for you... reading your comment on a potential answer down below, you could combine all the audio files into one using some sort of node module (audioconcat is one I found after a quick google search) and return that one file to the client. However, I would not personally take this route because the entire audiobook will be in the server's memory while it combines them, and until it returns it to the client. This could cause some memory issues down the road, so I would avoid it if I could. However, I will admit that this option could be potentially nice because the full length of the audiobook will display in the audio elements timeline.
The best option perhaps is to store the books full length and chapter lengths in a details.json file in the zip file and send that to the client in the first API call along with the URLs to each audio file. This would enable you to build a nice UI.
The only options I can think of is to use either use a javascript mp3 decoder (or compiled a C decoder to asm.js/wasm) and use the audio APIs. Or wrap the mp3 in an mp4 using something like mux.js and use media source extensions to playback.
maybe this will help you
<audio controls="controls">
<source src="track.ogg" type="audio/ogg" />
<source src="track.mp3" type="audio/mpeg" />
Your browser does not support the audio element.
</audio>

HTML5 Video: Streaming Video with Blob URLs

I have an array of Blobs (binary data, really -- I can express it however is most efficient. I'm using Blobs for now but maybe a Uint8Array or something would be better). Each Blob contains 1 second of audio/video data. Every second a new Blob is generated and appended to my array. So the code roughly looks like so:
var arrayOfBlobs = [];
setInterval(function() {
arrayOfBlobs.append(nextChunk());
}, 1000);
My goal is to stream this audio/video data to an HTML5 element. I know that a Blob URL can be generated and played like so:
var src = URL.createObjectURL(arrayOfBlobs[0]);
var video = document.getElementsByTagName("video")[0];
video.src = src;
Of course this only plays the first 1 second of video. I also assume I can trivially concatenate all of the Blobs currently in my array somehow to play more than one second:
// Something like this (untested)
var concatenatedBlob = new Blob(arrayOfBlobs);
var src = ...
However this will still eventually run out of data. As Blobs are immutable, I don't know how to keep appending data as it's received.
I'm certain this should be possible because YouTube and many other video streaming services utilize Blob URLs for video playback. How do they do it?
Solution
After some significant Googling I managed to find the missing piece to the puzzle: MediaSource
Effectively the process goes like this:
Create a MediaSource
Create an object URL from the MediaSource
Set the video's src to the object URL
On the sourceopen event, create a SourceBuffer
Use SourceBuffer.appendBuffer() to add all of your chunks to the video
This way you can keep adding new bits of video without changing the object URL.
Caveats
The SourceBuffer object is very picky about codecs. These have to be declared, and must be exact, or it won't work
You can only append one blob of video data to the SourceBuffer at a time, and you can't append a second blob until the first one has finished (asynchronously) processing
If you append too much data to the SourceBuffer without calling .remove() then you'll eventually run out of RAM and the video will stop playing. I hit this limit around 1 hour on my laptop
Example Code
Depending on your setup, some of this may be unnecessary (particularly the part where we build a queue of video data before we have a SourceBuffer then slowly append our queue using updateend). If you are able to wait until the SourceBuffer has been created to start grabbing video data, your code will look much nicer.
<html>
<head>
</head>
<body>
<video id="video"></video>
<script>
// As before, I'm regularly grabbing blobs of video data
// The implementation of "nextChunk" could be various things:
// - reading from a MediaRecorder
// - reading from an XMLHttpRequest
// - reading from a local webcam
// - generating the files on the fly in JavaScript
// - etc
var arrayOfBlobs = [];
setInterval(function() {
arrayOfBlobs.append(nextChunk());
// NEW: Try to flush our queue of video data to the video element
appendToSourceBuffer();
}, 1000);
// 1. Create a `MediaSource`
var mediaSource = new MediaSource();
// 2. Create an object URL from the `MediaSource`
var url = URL.createObjectURL(mediaSource);
// 3. Set the video's `src` to the object URL
var video = document.getElementById("video");
video.src = url;
// 4. On the `sourceopen` event, create a `SourceBuffer`
var sourceBuffer = null;
mediaSource.addEventListener("sourceopen", function()
{
// NOTE: Browsers are VERY picky about the codec being EXACTLY
// right here. Make sure you know which codecs you're using!
sourceBuffer = mediaSource.addSourceBuffer("video/webm; codecs=\"opus,vp8\"");
// If we requested any video data prior to setting up the SourceBuffer,
// we want to make sure we only append one blob at a time
sourceBuffer.addEventListener("updateend", appendToSourceBuffer);
});
// 5. Use `SourceBuffer.appendBuffer()` to add all of your chunks to the video
function appendToSourceBuffer()
{
if (
mediaSource.readyState === "open" &&
sourceBuffer &&
sourceBuffer.updating === false
)
{
sourceBuffer.appendBuffer(arrayOfBlobs.shift());
}
// Limit the total buffer size to 20 minutes
// This way we don't run out of RAM
if (
video.buffered.length &&
video.buffered.end(0) - video.buffered.start(0) > 1200
)
{
sourceBuffer.remove(0, video.buffered.end(0) - 1200)
}
}
</script>
</body>
</html>
As an added bonus this automatically gives you DVR functionality for live streams, because you're retaining 20 minutes of video data in your buffer (you can seek by simply using video.currentTime = ...)
Adding to the previous answer...
make sure to add sourceBuffer.mode = 'sequence' in the MediaSource.onopen event handler to ensure the data is appended based on the order it is received. The default value is segments, which buffers until the next 'expected' timeframe is loaded.
Additionally, make sure that you are not sending any packets with a data.size === 0, and make sure that there is 'stack' by clearing the stack on the broadcasting side, unless you are wanting to record it as an entire video, in which case just make sure the size of the broadcast video is small enough, and that your internet speed is fast. The smaller and lower the resolution the more likely you can keep a realtime connection with a client, ie a video call.
For iOS the broadcast needs to made from a iOS/macOS application, and be in mp4 format. The video chunk gets saved to the app's cache and then removed once it is sent to the server. A client can connect to the stream using either a web browser or app across nearly any device.

HTML5 FileApi + FileReader - Feed <object> with SWF

I want to use the HTML5 FileApi to read a SWF to an OBJECT (or EMBED, if it's better to do?).
My current code crashes on Chrome/Iron (the only stable browser which also supports the xmlhttprequest v2 FormData). I got it to read image data into a on-the-fly created IMG. But the object one crashes the current tab in the browser.
else if (file.type == "application/x-shockwave-flash") {
var show = document.createElement("object");
show.type = "application/x-shockwave-flash"
show.style.width = "100%";
show.style.height = "100%";
show.id = "thumb";
document.getElementById("thumbnails").appendChild(show);
var reader = new FileReader();
reader.onload = (function (aImg) {
return function (e) { aImg.data = e.target.result; };
})(show);
reader.readAsDataURL(file);
Do I really read to the object.data part? How is it done right? Anybody know? Or is this incomplete and I have to wait for better implementation?
A few things I'd recommend trying (in order of increasing complexity):
base64 encode the data with btoa and set it using a data: URI,
instead of creating the object using createElement, construct the <object> tag with all attributes as an HTML string (including the base64 advice above), then inject it into a DOM element with innerHTML,
create a reflector web service where you POST the swf content, it gives you a URL, then pass the URL off to the object,
similar to the previous, create a reflector web service where you POST the swf content, targeting a full-screen IFRAM as the target, have the service spits back an HTML doc including an <object> pointing back to the server.
The later of these options is more intense, and requires round-trips from the server that you'd probably want to avoid - just some more options you might want to consider.
ActionScript 3 has a Loader which may be useful as well. I don't know if it supports data: URI's, but if it does, you could write a boot loader SWF which runs the contents of the local swf file directly.

Categories