I am creating a live stream component that utilizes the videojs-record component. Every x amount of milliseconds, the component triggers an event that returns a blob. As seen, the blob contains data from the video recording. It's not the full recording but a piece, for this got returned x seconds into the recording
After saving it in the backend and playing it back, I find that I am unable to skip through the video; it's not seek-able.
Because this is a task that I'm trying to keep in the frontend, I have to inject this metadata within the browser using ts-ebml. After injecting the metadata, the modified blob is sent to the backend.
The function that receives this blob looks as follows:
timestampHandler(player) {
const { length: recordedDataLength } = player.recordedData;
if (recordedDataLength != 0) {
const { convertStream } = this.converter;
convertStream(player.recordedData[recordedDataLength - 1]).then((blob) => {
console.log(blob);
blob.arrayBuffer().then(async response => {
const bytes = new Uint8Array(response);
let binary = '';
let len = bytes.byteLength;
for (let i = 0; i < len; i++) {
binary += String.fromCharCode(bytes[i]);
}
this.$backend.videoDataSendToServer({ bytes: window.btoa(binary), id: this.videoId })
})
.catch(error => {
console.log('Error Converting:\t', error);
})
})
}
}
convertStream is a function located in a class called TsEBMLEngine. This class looks as follows:
import videojs from "video.js/dist/video";
import { Buffer } from "buffer";
window.Buffer = Buffer;
import { Decoder, tools, Reader } from "ts-ebml";
class TsEBMLEngine {
//constructor(){
//this.chunkDecoder = new Decoder();
//this.chunkReader = new Reader();
//}
convertStream = (data) => {
const chunkDecoder = new Decoder();
const chunkReader = new Reader();
chunkReader.logging = false;
chunkReader.drop_default_duration = false;
// save timestamp
const timestamp = new Date();
timestamp.setTime(data.lastModified);
// load and convert blob
return data.arrayBuffer().then((buffer) => {
// decode
const elms = chunkDecoder.decode(buffer);
elms.forEach((elm) => {
chunkReader.read(elm);
});
chunkReader.stop();
// generate metadata
let refinedMetadataBuf = tools.makeMetadataSeekable(
chunkReader.metadatas,
chunkReader.duration,
chunkReader.cues
);
let body = buffer.slice(chunkReader.metadataSize);
// create new blob
let convertedData = new Blob([refinedMetadataBuf, body], { type: data.type });
// store convertedData
return convertedData;
});
}
}
// expose plugin
videojs.TsEBMLEngine = TsEBMLEngine;
export default TsEBMLEngine;
After recording for more than 10 seconds I stop the recording, go to the DB, and watch the retrieved video. The video is seek-able for the first 3 seconds before the dot reaches the very end of the seek-able line. When I'm watching the video in a live stream, the video freezes after the first 3 seconds.
When I look at the size of the file in the DB, it increases after x seconds which means it's being appended to it, just not properly.
Any help would be greatly appreciated.
For being seekable a video (at least talking about EBMLs) needs to have a SeekHead tag, Cues tags and defined duration in Info tag.
For creating new metadata of the video you can use ts-ebml's exporting function makeMetadataSeekable
Then slice the beginning of video and replace it with new metadata like it was done in the example :
const decoder = new Decoder();
const reader = new Reader();
const webMBuf = await fetch("path/to/file").then(res=> res.arrayBuffer());
const elms = decoder.decode(webMBuf);
elms.forEach((elm)=>{ reader.read(elm); });
reader.stop();
const refinedMetadataBuf = tools.makeMetadataSeekable(reader.metadatas, reader.duration, reader.cues);
const body = webMBuf.slice(reader.metadataSize);
const refinedWebM = new Blob([refinedMetadataBuf, body], {type: "video/webm"});
And voila! new video file becomes seekable
I'm struggling to find documentation or examples of implementing an upload progress indicator using fetch.
This is the only reference I've found so far, which states:
Progress events are a high level feature that won't arrive in fetch for now. You can create your own by looking at the Content-Length header and using a pass-through stream to monitor the bytes received.
This means you can explicitly handle responses without a Content-Length differently. And of course, even if Content-Length is there it can be a lie. With streams you can handle these lies however you want.
How would I write "a pass-through stream to monitor the bytes" sent? If it makes any sort of difference, I'm trying to do this to power image uploads from the browser to Cloudinary.
NOTE: I am not interested in the Cloudinary JS library, as it depends on jQuery and my app does not. I'm only interested in the stream processing necessary to do this with native javascript and Github's fetch polyfill.
https://fetch.spec.whatwg.org/#fetch-api
Streams are starting to land in the web platform (https://jakearchibald.com/2016/streams-ftw/) but it's still early days.
Soon you'll be able to provide a stream as the body of a request, but the open question is whether the consumption of that stream relates to bytes uploaded.
Particular redirects can result in data being retransmitted to the new location, but streams cannot "restart". We can fix this by turning the body into a callback which can be called multiple times, but we need to be sure that exposing the number of redirects isn't a security leak, since it'd be the first time on the platform JS could detect that.
Some are questioning whether it even makes sense to link stream consumption to bytes uploaded.
Long story short: this isn't possible yet, but in future this will be handled either by streams, or some kind of higher-level callback passed into fetch().
My solution is to use axios, which supports this pretty well:
axios.request({
method: "post",
url: "/aaa",
data: myData,
onUploadProgress: (p) => {
console.log(p);
//this.setState({
//fileprogress: p.loaded / p.total
//})
}
}).then (data => {
//this.setState({
//fileprogress: 1.0,
//})
})
I have example for using this in react on github.
fetch: not possible yet
It sounds like upload progress will eventually be possible with fetch once it supports a ReadableStream as the body. This is currently not implemented, but it's in progress. I think the code will look something like this:
warning: this code does not work yet, still waiting on browsers to support it
async function main() {
const blob = new Blob([new Uint8Array(10 * 1024 * 1024)]); // any Blob, including a File
const progressBar = document.getElementById("progress");
const totalBytes = blob.size;
let bytesUploaded = 0;
const blobReader = blob.stream().getReader();
const progressTrackingStream = new ReadableStream({
async pull(controller) {
const result = await blobReader.read();
if (result.done) {
console.log("completed stream");
controller.close();
return;
}
controller.enqueue(result.value);
bytesUploaded += result.value.byteLength;
console.log("upload progress:", bytesUploaded / totalBytes);
progressBar.value = bytesUploaded / totalBytes;
},
});
const response = await fetch("https://httpbin.org/put", {
method: "PUT",
headers: {
"Content-Type": "application/octet-stream"
},
body: progressTrackingStream,
});
console.log("success:", response.ok);
}
main().catch(console.error);
upload: <progress id="progress" />
workaround: good ol' XMLHttpRequest
Instead of fetch(), it's possible to use XMLHttpRequest to track upload progress — the xhr.upload object emits a progress event.
async function main() {
const blob = new Blob([new Uint8Array(10 * 1024 * 1024)]); // any Blob, including a File
const uploadProgress = document.getElementById("upload-progress");
const downloadProgress = document.getElementById("download-progress");
const xhr = new XMLHttpRequest();
const success = await new Promise((resolve) => {
xhr.upload.addEventListener("progress", (event) => {
if (event.lengthComputable) {
console.log("upload progress:", event.loaded / event.total);
uploadProgress.value = event.loaded / event.total;
}
});
xhr.addEventListener("progress", (event) => {
if (event.lengthComputable) {
console.log("download progress:", event.loaded / event.total);
downloadProgress.value = event.loaded / event.total;
}
});
xhr.addEventListener("loadend", () => {
resolve(xhr.readyState === 4 && xhr.status === 200);
});
xhr.open("PUT", "https://httpbin.org/put", true);
xhr.setRequestHeader("Content-Type", "application/octet-stream");
xhr.send(blob);
});
console.log("success:", success);
}
main().catch(console.error);
upload: <progress id="upload-progress"></progress><br/>
download: <progress id="download-progress"></progress>
Update: as the accepted answer says it's impossible now. but the below code handled our problem for sometime. I should add that at least we had to switch to using a library that is based on XMLHttpRequest.
const response = await fetch(url);
const total = Number(response.headers.get('content-length'));
const reader = response.body.getReader();
let bytesReceived = 0;
while (true) {
const result = await reader.read();
if (result.done) {
console.log('Fetch complete');
break;
}
bytesReceived += result.value.length;
console.log('Received', bytesReceived, 'bytes of data so far');
}
thanks to this link: https://jakearchibald.com/2016/streams-ftw/
As already explained in the other answers, it is not possible with fetch, but with XHR. Here is my a-little-more-compact XHR solution:
const uploadFiles = (url, files, onProgress) =>
new Promise((resolve, reject) => {
const xhr = new XMLHttpRequest();
xhr.upload.addEventListener('progress', e => onProgress(e.loaded / e.total));
xhr.addEventListener('load', () => resolve({ status: xhr.status, body: xhr.responseText }));
xhr.addEventListener('error', () => reject(new Error('File upload failed')));
xhr.addEventListener('abort', () => reject(new Error('File upload aborted')));
xhr.open('POST', url, true);
const formData = new FormData();
Array.from(files).forEach((file, index) => formData.append(index.toString(), file));
xhr.send(formData);
});
Works with one or multiple files.
If you have a file input element like this:
<input type="file" multiple id="fileUpload" />
Call the function like this:
document.getElementById('fileUpload').addEventListener('change', async e => {
const onProgress = progress => console.log('Progress:', `${Math.round(progress * 100)}%`);
const response = await uploadFiles('/api/upload', e.currentTarget.files, onProgress);
if (response.status >= 400) {
throw new Error(`File upload failed - Status code: ${response.status}`);
}
console.log('Response:', response.body);
}
Also works with the e.dataTransfer.files you get from a drop event when building a file drop zone.
I don't think it's possible. The draft states:
it is currently lacking [in comparison to XHR] when it comes to request progression
(old answer):
The first example in the Fetch API chapter gives some insight on how to :
If you want to receive the body data progressively:
function consume(reader) {
var total = 0
return new Promise((resolve, reject) => {
function pump() {
reader.read().then(({done, value}) => {
if (done) {
resolve()
return
}
total += value.byteLength
log(`received ${value.byteLength} bytes (${total} bytes in total)`)
pump()
}).catch(reject)
}
pump()
})
}
fetch("/music/pk/altes-kamuffel.flac")
.then(res => consume(res.body.getReader()))
.then(() => log("consumed the entire body without keeping the whole thing in memory!"))
.catch(e => log("something went wrong: " + e))
Apart from their use of the Promise constructor antipattern, you can see that response.body is a Stream from which you can read byte by byte using a Reader, and you can fire an event or do whatever you like (e.g. log the progress) for every of them.
However, the Streams spec doesn't appear to be quite finished, and I have no idea whether this already works in any fetch implementation.
with fetch: now possible with Chrome >= 105 🎉
How to:
https://developer.chrome.com/articles/fetch-streaming-requests/
Currently not supported by other browsers (maybe that will be the case when you read this, please edit my answer accordingly)
Feature detection (source)
const supportsRequestStreams = (() => {
let duplexAccessed = false;
const hasContentType = new Request('', {
body: new ReadableStream(),
method: 'POST',
get duplex() {
duplexAccessed = true;
return 'half';
},
}).headers.has('Content-Type');
return duplexAccessed && !hasContentType;
})();
HTTP >= 2 required
The fetch will be rejected if the connection is HTTP/1.x.
Since none of the answers solve the problem.
Just for implementation sake, you can detect the upload speed with some small initial chunk of known size and the upload time can be calculated with content-length/upload-speed. You can use this time as estimation.
A possible workaround would be to utilize new Request() constructor then check Request.bodyUsed Boolean attribute
The bodyUsed attribute’s getter must return true if disturbed, and
false otherwise.
to determine if stream is distributed
An object implementing the Body mixin is said to be disturbed if
body is non-null and its stream is disturbed.
Return the fetch() Promise from within .then() chained to recursive .read() call of a ReadableStream when Request.bodyUsed is equal to true.
Note, the approach does not read the bytes of the Request.body as the bytes are streamed to the endpoint. Also, the upload could complete well before any response is returned in full to the browser.
const [input, progress, label] = [
document.querySelector("input")
, document.querySelector("progress")
, document.querySelector("label")
];
const url = "/path/to/server/";
input.onmousedown = () => {
label.innerHTML = "";
progress.value = "0"
};
input.onchange = (event) => {
const file = event.target.files[0];
const filename = file.name;
progress.max = file.size;
const request = new Request(url, {
method: "POST",
body: file,
cache: "no-store"
});
const upload = settings => fetch(settings);
const uploadProgress = new ReadableStream({
start(controller) {
console.log("starting upload, request.bodyUsed:", request.bodyUsed);
controller.enqueue(request.bodyUsed);
},
pull(controller) {
if (request.bodyUsed) {
controller.close();
}
controller.enqueue(request.bodyUsed);
console.log("pull, request.bodyUsed:", request.bodyUsed);
},
cancel(reason) {
console.log(reason);
}
});
const [fileUpload, reader] = [
upload(request)
.catch(e => {
reader.cancel();
throw e
})
, uploadProgress.getReader()
];
const processUploadRequest = ({value, done}) => {
if (value || done) {
console.log("upload complete, request.bodyUsed:", request.bodyUsed);
// set `progress.value` to `progress.max` here
// if not awaiting server response
// progress.value = progress.max;
return reader.closed.then(() => fileUpload);
}
console.log("upload progress:", value);
progress.value = +progress.value + 1;
return reader.read().then(result => processUploadRequest(result));
};
reader.read().then(({value, done}) => processUploadRequest({value,done}))
.then(response => response.text())
.then(text => {
console.log("response:", text);
progress.value = progress.max;
input.value = "";
})
.catch(err => console.log("upload error:", err));
}
I fished around for some time about this and just for everyone who may come across this issue too here is my solution:
const form = document.querySelector('form');
const status = document.querySelector('#status');
// When form get's submitted.
form.addEventListener('submit', async function (event) {
// cancel default behavior (form submit)
event.preventDefault();
// Inform user that the upload has began
status.innerText = 'Uploading..';
// Create FormData from form
const formData = new FormData(form);
// Open request to origin
const request = await fetch('https://httpbin.org/post', { method: 'POST', body: formData });
// Get amount of bytes we're about to transmit
const bytesToUpload = request.headers.get('content-length');
// Create a reader from the request body
const reader = request.body.getReader();
// Cache how much data we already send
let bytesUploaded = 0;
// Get first chunk of the request reader
let chunk = await reader.read();
// While we have more chunks to go
while (!chunk.done) {
// Increase amount of bytes transmitted.
bytesUploaded += chunk.value.length;
// Inform user how far we are
status.innerText = 'Uploading (' + (bytesUploaded / bytesToUpload * 100).toFixed(2) + ')...';
// Read next chunk
chunk = await reader.read();
}
});
const req = await fetch('./foo.json');
const total = Number(req.headers.get('content-length'));
let loaded = 0;
for await(const {length} of req.body.getReader()) {
loaded = += length;
const progress = ((loaded / total) * 100).toFixed(2); // toFixed(2) means two digits after floating point
console.log(`${progress}%`); // or yourDiv.textContent = `${progress}%`;
}
Key part is ReadableStream ≪obj_response.body≫.
Sample:
let parse=_/*result*/=>{
console.log(_)
//...
return /*cont?*/_.value?true:false
}
fetch('').
then(_=>( a/*!*/=_.body.getReader(), b/*!*/=z=>a.read().then(parse).then(_=>(_?b:z=>z)()), b() ))
You can test running it on a huge page eg https://html.spec.whatwg.org/ and https://html.spec.whatwg.org/print.pdf . CtrlShiftJ and load the code in.
(Tested on Chrome.)
I had to write a program for facial recognition in JavaScript , for which I used the opencv4nodejs API , since there's NOT many working examples ; Now I somehow want to record and save the stream (for saving on the client-side or uploading on the server) alongwith the audio. This is where I am stuck. Any help is appreciated.
In simple words I need to use the Webcam input for multiple purposes , one for facial recognition and two to somehow save , latter is what i'm unable to do. Also in the worst case, If it's not possible Instead of recording and saving the webcam video I could also save the Complete Screen recording , Please Answer if there's a workaround to this.
Below is what i tried to do, But it doesn't work for obvious reasons.
$(document).ready(function () {
run1()
})
let chunks = []
// run1() for uploading model and for facecam
async function run1() {
const MODELS = "/models";
await faceapi.loadSsdMobilenetv1Model(MODELS)
await faceapi.loadFaceLandmarkModel(MODELS)
await faceapi.loadFaceRecognitionModel(MODELS)
var _stream
//Accessing the user webcam
const videoEl = document.getElementById('inputVideo')
navigator.mediaDevices.getUserMedia({
video: true,
audio: true
}).then(
(stream) => {
_stream = stream
recorder = new MediaRecorder(_stream);
recorder.ondataavailable = (e) => {
chunks.push(e.data);
console.log(chunks, i);
if (i == 20) makeLink(); //Trying to make Link from the blob for some i==20
};
videoEl.srcObject = stream
},
(err) => {
console.error(err)
}
)
}
// run2() main recognition code and training
async function run2() {
// wait for the results of mtcnn ,
const input = document.getElementById('inputVideo')
const mtcnnResults = await faceapi.ssdMobilenetv1(input)
// Detect All the faces in the webcam
const fullFaceDescriptions = await faceapi.detectAllFaces(input).withFaceLandmarks().withFaceDescriptors()
// Training the algorithm with given data of the Current Student
const labeledFaceDescriptors = await Promise.all(
CurrentStudent.map(
async function (label) {
// Training the Algorithm with the current students
for (let i = 1; i <= 10; i++) {
// console.log(label);
const imgUrl = `http://localhost:5500/StudentData/${label}/${i}.jpg`
const img = await faceapi.fetchImage(imgUrl)
// detect the face with the highest score in the image and compute it's landmarks and face descriptor
const fullFaceDescription = await faceapi.detectSingleFace(img).withFaceLandmarks().withFaceDescriptor()
if (!fullFaceDescription) {
throw new Error(`no faces detected for ${label}`)
}
const faceDescriptors = [fullFaceDescription.descriptor]
return new faceapi.LabeledFaceDescriptors(label, faceDescriptors)
}
}
)
)
const maxDescriptorDistance = 0.65
const faceMatcher = new faceapi.FaceMatcher(labeledFaceDescriptors, maxDescriptorDistance)
const results = fullFaceDescriptions.map(fd => faceMatcher.findBestMatch(fd.descriptor))
i++;
}
// I somehow want this to work
function makeLink() {
alert("ML")
console.log("IN MAKE LINK");
let blob = new Blob(chunks, {
type: media.type
}),
url = URL.createObjectURL(blob),
li = document.createElement('li'),
mt = document.createElement(media.tag),
hf = document.createElement('a');
mt.controls = true;
mt.src = url;
hf.href = url;
hf.download = `${counter++}${media.ext}`;
hf.innerHTML = `donwload ${hf.download}`;
li.appendChild(mt);
li.appendChild(hf);
ul.appendChild(li);
}
// onPlay(video) function
async function onPlay(videoEl) {
run2()
setTimeout(() => onPlay(videoEl), 50)
}
I'm not familiar with JavaScript. But in general only one program may communicate with the camera. You will probably need to write a server which will read the data from the camera. Then the server will send the data to your facial recognition, recording, etc.
I am using FFmpeg and html2canvus and trying to create an mp4 video with the screenshots taken from a slider.
here is my worker initialization
const worker = createWorker({
//logger: ({ message }) => console.log(message),
progress: (p) => console.log(p),
});
Then in click, I take the screenshots and putting into the video
const image2video = async () => {
displayLoader();
var zip = new JSZip();
let selectedbanners = $(".selected_templates:checked");
await worker.load();
let promise = new Promise((resolve, reject) => {
let Processed = 0;
selectedbanners.each(async function () {
var dataIndex = $(this).data('index');
let ad = adTemplates[dataIndex];
var innercounter = 0;
$(`.template-container-${ad.name}`).each(async function () {
var imgData;
await html2canvas($(`.template-container-${ad.name}`)[innercounter], {allowTaint: true, width: ad.width, height: ad.height}).then(canvas => {
imgData = canvas.toDataURL('image/jpeg', 1.0).split('base64,')[1];
//await worker.write(`tmp.${ad.name}.${innercounter}.png`, imgData);
});
await worker.write(`tmp.${ad.name}.${innercounter}.png`, imgData);
//await worker.write(`tmp.0.png`, `static/triangle/tmp.0.png`); This is working
});
});
});
};
I have setup a codepen here. It works if I put the image path but doesn't work if I directly pass the base64 string. Here I found that it also supports the base64 string as well as the URL.
This is how it looks in console
Thanks in advance.
I fixed that just changed a line of code.
imgData = `data:image/png;base64,`+canvas.toDataURL('image/png', 1.0).split('base64,')[1];
In this, I just append a data URL and changed the image type from JPEG to png and it worked.
Im trying to read file stored in file system (android) and then put the content into canvas using pdfjs library so I could render it in view. Idid the same with system pdf viewer and it worked, but I need to do later some painting on it and manupulation so it can't be dispsplayed in viewer, it must be within my app.
Rendering of my pdf works fine, since I have tested it with live reload mode.
Below is reading code
readFile( pathToFile ){
this.file.resolveLocalFilesystemUrl( pathToFile).then((fileEntry: any) => {
fileEntry.file( (file) => {
var reader = new FileReader();
reader.onloadend = (event) => {
const x = event.target as any;
// let sliced = x._result.slice(x._result.indexOf(',') + 1, x._result.length);
console.log('item', x)
console.log('item', x.result)
console.log('buffer',new Uint8Array(x.result))
// console.log('64', new Uint8Array(x._result));
// const bytes = this.base64ToUint8Array(sliced)
this.renderPDF(x.result, this.container.nativeElement, 1)
};
reader.readAsArrayBuffer(file);
});
});
}
here are the logs:
as you can see pdf1 ist the last log so the promise from getDocument gets not resolved:
renderPDF(url, canvasContainer, scale) {
console.log('pdf1')
this.pdfCreator.disableWorker = true;
this.pdfCreator
.getDocument(url)
.then((doc) => {
this.doc = doc;
console.log('pdf2')
this.renderPages(canvasContainer, scale);
})
.catch(err => console.log(err))
}
I have spent two days on it without sccess...
I think there is something went wrong with url,
Could you please try this
readFile( pathToFile ){
this.file.resolveLocalFilesystemUrl( pathToFile).then((fileEntry: any) => {
fileEntry.file( (file) => {
var reader = new FileReader();
reader.onloadend = (event) => {
const x = event.target as any;
// let sliced = x._result.slice(x._result.indexOf(',') + 1, x._result.length);
console.log('item', x)
console.log('item', x.result)
console.log('buffer',new Uint8Array(x.result))
// console.log('64', new Uint8Array(x._result));
// const bytes = this.base64ToUint8Array(sliced)
var blob = new Blob([new Uint8Array(x.result)], {type: 'application/pdf'});
var url = URL.createObjectURL(blob);
this.renderPDF(url, this.container.nativeElement, 1)
};
reader.readAsArrayBuffer(file);
});
});
}