Capture video with alpha channel using canvas.captureStream() - javascript

I'm trying to capture the contents of a canvas element with alpha channel. When I do it I get the RGB values correctly but the Alpha channel seems to get dropped when playing back the resulting video. Is there a way to achieve this?
I'm running the following code:
var stream = canvas.captureStream(60);
var recorder = new MediaRecorder(stream);
recorder.addEventListener('dataavailable', finishCapturing);
recorder.addEventListener('stop', function(e) {
video.oncanplay = video.play;
video.src = URL.createObjectURL(new Blob(blobs, {type:"video/webm; codecs=vp9"}));
});
startCapturing();
recorder.start();
Here's a plnkr demonstrating the issue:
http://plnkr.co/edit/z3UL9ikmn6PtVoAHvY0B?p=preview

There is currently no options to enable (VP8/9 transparency channel) from the MediaRecorder API.
One could maybe open an issue on the W3C Mediacapture-Record git repo.
For this, I can guess a few reasons :
From what I understand, webm alpha channel is grossly an hack from chrome, and is not really implemented in the codec itself, nor completely stabilized.
MediaRecorder should be able to encode in many formats even if current implementations only support video webm/VP8 and webm/VP9 (chrome only). So it would mean that they would have to somehow keep the alpha channel in the raw stream, only for this new canvas.captureStream method. Historically, MediaStream mainly came from getUserMedia interface, and there were no need way of getting transparency from there.
[edit: Specs have changed since this answer was written, and MediaStreams should now keep the alpha channel, even if the consumer may not be able to use it, also Chrome now supports more video codecs.]
Chrome, which is the only one to support YUVA webm display in its stable channel (FF supports it in nightly 54), is still not able to include the duration inside their recorded files, let's them fix this before adding the hackish alpha_mode=true.
However, you can achieve it yourself kind of easily :
If you really want a transparent webm file (only readable in chrome and FF nightly), you can use a second canvas to do the recording, set its background (using fillRect) to a chroma that won't appear elsewhere in your drawings, draw the original one on it and record its stream. Once recorded, use ffmpeg to reencode the recorded video, this time with the alpha channel :
// all #00FF00 pixels will become transparent.
ffmpeg -i in.webm -c:v libvpx -vf "chromakey=0x00ff00:0.1:0.1,format=yuva420p" -auto-alt-ref 0 out.webm
I personally needed the -auto-alt-ref 0 flag, not sure everyone needs it though
But because of this other chrome bug, you'll actually have to append this other canvas on screen too, and hide it with css (opacity: 0; width:0px; height:0px; should do).
TL;DR
This API's implementations are far from being stabilized, no one has made the request for such a feature yet, it may come in near future though, and can be done server-side for the time being.

Related

Web video capture (getUserMedia) pixel format with no chroma subsampling (YUV444 or RGB24) on chromium

Despite conducting a good number of experiments and thorough research, I could not find a way to obtain a full color video/webcam stream with no chroma subsampling in chromium.
The goal is to be able to display the a video stream from a capture card and displaying it directly onto a canvas and drawing some overlay elements on top, so network throughput is not an issue, but the stream quality is.
I have been testing by feeding a still image into v4l2 with ffmpeg with RGB24 pixel format (YUV444 doesn't seem to be supported by v4l2). However, whatever I do, I cannot change the resulting stream to be something other than vp8 I420 (YUV420) but I cannot find how to change the vp8 codec used with another one supporting a full color profile.
I attempted using the experimental MediaStreamTrackProcessor since it said it would allow me to access raw stream bytes, however these are already I420 unless I missed a way to change it. But this makes me wonder if there could be some other experimental APIs or ongoing w3c proposals I could have missed and which could achieve what I need.
Here is the code:
navigator.mediaDevices.enumerateDevices().then(async devices => {
console.log(devices);
const videoDevice = devices.find(device => device.kind === "videoinput" && device.label === "Virtual Webcam");
const stream = await navigator.mediaDevices.getUserMedia({
video: {
deviceId: videoDevice.deviceId,
width: {ideal: 1920},
height: {ideal: 1080},
frameRate: {ideal: 25},
}
});
const [track] = stream.getVideoTracks();
const {readable} = new MediaStreamTrackProcessor({track});
const reader = readable.getReader();
const result = await reader.read();
console.log(result.value.format); // Outputs I420 no matter which ffmpeg pix_fmt
});
I know that displaying a real time full color video stream is possible, as reading the stream with OBS or VLC with the right setting, I am able to obtain an RGB24 image back, but I must be able to do this on a web page. My company is willing to spend some people and some time to contribute to chromium to make it happen if it must come to it, but it quite difficult to know if this is realistic as I fear we would need to write a serious and backed proposal first if I understand correctly. If anyone has an idea about the feasibility of this, it would be very welcome.
Otherwise, the last resort idea we are considering would be to use electron to interface with lower level video capture libraries; but the true ideal would be to have it work on a web browser. Any idea on how to achieve that is more than welcome.

Capturing data stream of a JS / Leaflet animation as MP4

How can I capture the datastream of a JS / Leaflet animation and download it to MP4?
I am looking for output that looks something like the smooth path traced in these demos:
https://github.com/IvanSanchez/Leaflet.Polyline.SnakeAnim
Their author appears to have done them in ffcast or some screencasting softare.
However, I am looking for an automated solution that can be run as script, ideally one that works on the data stream itself (not the screen), perhaps with a headless browser.
I have tried puppeteer-gif and puppeteer-gif-cast but the best frame rate is jumpy.
I have tried WebRTC-Experiment but it requires me to set manual permissions. Ditto the Screen Capture API mentioned here, though this at least seems to work on the data stream itself.
The canvas captureStream method combined with the MediaRecorder API should do the trick.
Mind you that Chrome only supports webm as a container format (but does record h264) so you might need a postprocessing step with ffmpeg.

Single stream audio/video in Chrome WebRTC without ssrc tags

Is single stream audio (or video) via Chrome's WebRTC possible when you strip a=ssrc lines from the SDP?
I have tried filtering out a=ssrc lines (with the code below), but single stream audio did not work. I tried also single stream video and renaming instead of removing lines with the same result. I modify both offer and answer SDPs. Interestingly, this filtering works when you try sending SDPs with both audio & video - audio (only) will work in such scenario. However I had issues with re-negotiation in such scenario in our app, so this is probably not a valid solution.
You can see minimum example with the single stream audio / video in this repo: https://github.com/Tev-work/webrtc-audio-demo.
If it is possible, can you please provide minimal example of code with working audio? Preferably using the repo above, what should the modifySdp function (in public/client.js) do?
Currently it modifies sdp with this code:
sdp = sdp.replace(/a=ssrc/g, 'a=xssrc');
sdp = sdp.replace(/a=msid-semantic/g, 'a=xmsid-semantic');
sdp = sdp.replace(/a=mid/g, 'a=xmid');
sdp = sdp.replace(/a=group:BUNDLE/g, 'a=xgroup:BUNDLE');
If it is not possible, do you know whether such limitation has been officialy stated somewhere (please link it), or it just at some point became unworkable? It seems like it was working before (around M29, see comments here https://bugs.chromium.org/p/webrtc/issues/detail?id=1941 - no mention that this was not supposed to be working).
Motivation: We are sometimes sending SDPs via SIP PBXs, which sometimes filter out SSRC lines. Supporting multiple streams in such situations is obviously out of question (maybe with some server side hacking streams?), but supporting at least audio-only for such scenarios would be useful for us.
that should still be possible, even though there are some side-effects like (legacy) getStats not recognizing the stream, see (this bug)[https://bugs.chromium.org/p/webrtc/issues/detail?id=3342].
What you are attempting is to remove the a=ssrc lines before calling setLocalDescription. This is probably not going to work. If you want to simulate the scenario try removing them before calling setRemoteDescription with the SDP.

Where in metadata of a video in html5 is the fps saved?

In order to fully implement my custom html5 video player, I need the the exact frame rate of a video. However I have not been able to find it yet and am using a standard value of 25.
Typically videos have a frame rate value in meta-data so I accessed meta-data using something like this:
var vid = document.getElementById("myVideo");
vid.onloadedmetadata = function(e) {
console.log(e);
};
However I can't find frame rate here. Maybe I am not reading metadata at all.
I can use your help.
Thanks!
Try https://mediainfo.js.org (github)
It works on ui only, no backend needed
I just implemented it and it looks like it worked perfectly fine (at least in Chrome v 70.0.3538.77) for gettting wide media information
It looks like modern browsers beginning to work with some binary libraries
I'm 95% sure the standard html5 video api does not expose the fps information, from what I've read in the past months - other apis like MPEG-DASH and jwplayer do present more / different data.
Your best bet would be to snoop around w3schools.com/tags/ref_av_dom.asp and similar mdn pages.
You can calculate this in realtime yourself and it should work most of the time but I can imagine there's a case or two when it wouldn't. Look at PresentedFrames and then do something like:
fps = video.time / PresentedFrames
view more about PresentedFrames here (currently proposal) and similar attributes at the same link.
mediainfo.js works pretty good - even if used locally in a browser using 'http(s)://'.
To use it locally, just make sure you also download the accompanying mediainfo.wasm and put it into the same directory as mediainfo.min.js.
Alternatively you can install media-info using npm.
The only caveat is, that it doesn't run from the 'file://' protocol.

How to synthesize audio using HTML5/Javascript on iPad

Has anybody out there got working sample code that synthesizes (and plays) audio using HTML5/Javascript on Mobile Safari on the iPad? I have found some examples for javascript-based sound synthesis on the web, but they all seem to work in Firefox only.
Recently I came across this js-library, I think this is what you want ->
https://github.com/oampo/Audiolet
Here is an example that works for me on an iPad:
www.cse.usf.edu/~turnerr/sound_demo.html
You can download the files from http://www.cse.usf.edu/~turnerr/Downloads/Sound_Demo.zip
This demo is on a Unix based server. But I have not been been able to get the same code to work on an IIS server. Hoping someone can provide some help with IIS.
You may be able to use generated data URIs of uniform length, such as 0.1 seconds. This would give you 1/10 of a second delay and you would generate that many "frames" of audio. I'm not sure entirely what formats the iPad supports, but I read it supports uncompressed WAV. Info on this file format is pretty easy to get, I remember generating WAV files a long time ago with some primitive byte manipulation methods.
Please post back with details!
I'm answering a very old question... but modern webkit browsers now support the Web Audio API. I wrote a very simple fiddle that generates chords using sine waves. There are only 4 built-in wave forms, but you can build your own using Fourier coefficients (array of numbers). You have to generate a new oscillator object for each note. They are single use objects. By connecting multiple oscillators to the same destination, you get polyphonic sounds.
let audio = new(window.AudioContext || window.webkitAudioContext)();
let s1 = audio.createOscillator();
let g1 = audio.createGain();
s1.type = 'sine';
s1.frequency.value = 600;
s1.start();
g1.gain.value = 0.5;
g1.connect(audio.destination);
s1.connect(g1);

Categories