How to modify the content of WebRTC MediaStream video track? - javascript

I use WebRTC in a scenario in which the client video stream is recorded on a third-party server https://tokbox.com/. I would like to put some kind of watermark in the recorded video.
Investigation brought me to this page http://w3c.github.io/webrtc-pc/#mediastreamtrack and it seems that it is technically possible since it says that:
A MediaStream acquired using getUserMedia() is, by default, accessible to an application. This means that the application is able to access the contents of tracks, modify their content, and send that media to any peer it chooses.
This is exactly what I need, but I didn't find any examples or explanation of this function. I'd like to get some advice from WebRTC experts.

You need to use a canvas to route the video from getUserMedia to, modify it there, and then use canvas.captureStream() to turn it back into a MediaStream. This is great - except that canvas.captureStream(), while agreed to in the WG hasn't actually been included in the spec yet. (There's a pull request with the proposed verbiage that Mozilla wrote.)
As far as implementations: the initial implementation of captureStream() just landed in Firefox Nightly (41), and it's still behind a pref until a bug or two is fixed. You can enable it with canvas.capturestream.enabled in about:config. You can see a demo at Mozilla's test page for captureStream().
Doing it without canvas.captureStream() would be tough; you're best way would be to do getUserMedia->canvas-> and then use video.captureStream() (or captureStreamUntilEnded()) - however, video.captureStream is also waiting for formal acceptance. Mozilla has had video.captureStream() for some time, however, and I think it works in FF 38 (current release).

Related

How does Disney+ prevent screen recordings of their content?

I tried to take a screenshot of a movie on the Disney+ web app when I realised that the video turns black as soon as I try to take a new screenshot with Snipping Tool. When I tried to do the same thing with OBS and Discord streams, I saw the same effect.
Interestingly, this only works for Chrome on my machine (I also tried Firefox and Edge and they just let me record my screen).
When I saw this, I became really curious on how they achieved this.
Does anyone have any idea how I can recreate this for my own web projects?
I became really curious on how they achieved this.
They use Widevine.
Widevine homepage.
https://ottverse.com/widevine-drm-how-does-it-work/
https://en.wikipedia.org/wiki/Widevine
News reports:
https://www.cordcuttersnews.com/sadly-disney-wont-work-on-chromebooks-linux-some-android-devices-because-of-drm/
https://www.tomsguide.com/news/disney-plus-will-work-on-chromebooks
https://www.androidpolice.com/2019/10/22/disney-will-only-work-on-devices-that-support-the-strictest-widevine-l3-drm/
It's also used by Netflix, Hulu and others.
Widevine is Google's DRM system that's baked-in to Chrome.
All other major browsers have adopted it as well, because no-one will use a browser that can't access Netflix.
Mozilla's and Microsoft's support is less user-hostile and as you noticed.
It's just a standard HTML5 <video> element - when the browser downloads the video stream it will see that it's encrypted with Widevine and that engages the Widevine client-side code which does all the DRM biz.
Though there are HTML and DOM features that facilitate DRM, I'm unsure of the extent that any JavaScript is required to use it - as theoretically everything the browser needs to know to load the DRM system should be embedded in the raw media stream.
On Windows, I understand (though unconfirmed) that Widevine makes use of SetWindowDisplayAffinity to block screenshots.
Nothing stops you from doing this in your own native code (e.g. if you had an Electron fork), but please don't because it's a real dick-move to your users, in addition to not working at all if the user has the DWM disabled (e.g. they're running Windows 7 with Aero disabled).
Has anyone any idea how I can recreate this for my own web projects?
You'll need to license Widevine yourself. This is a complicated process intended only for large media production companies and content rightsholders, not individuals or small businesses.
Anyway, even if you could, please don't. Why would you want to make to harder for users to share and appreciate your media? Just stick it up on YouTube instead.

Is there a way, using streaming in the browser, to toggle the video stream on/off?

The current setup
I'm building a web application which will basically act as an internet radio, but with optional video content too.
The server will be using Icecast to provide the stream (although if required, I could definitely use something else).
The question
Is it possible to implement something, that would allow the user to seamlessly switch the video stream on or off, where the audio will keep playing uninterrupted, and both streams will stay synchronised too.
Synchronisation is only important per-client, not between clients. I don't mean just hiding the video stream either, the idea is to minimize bandwidth usage when not watching.
Watching the stream will happen from within the browser and I do not want to use something like a java plugin.
Support for older browsers doesn't matter, the latest versions of Edge, Safari, Chrome and Firefox is enough.
Turns out this isn't possible to do the way I wanted to (actually turn off the video stream).
Next best workaround is by using DASH and adding effectively an empty (or just black) video stream that can be switched to.

Screensharing over WebRTC and browser support

I am trying to build screensharing over the browser. I am trying to find the best native implementation and did some initial online research
MediaDevices.getUserMedia() - available in FF. In chrome its a little weird
WebRTC Tab Content Capture - I see its in proposal stage
Screensharing a browser tab in HTML5 - A blog explaining other methods
Researching above everything seems to be around 2012 time frame and I want to know what is the latest?
Question: Which current technologies/javascript API can i use and what is its support across browsers
Screensharing is alive and kicking in Firefox, but atm requires the user modifying about:config. See my answer to another question for how. I believe they're working on removing that obstacle.
Chrome is similar but not quite the same, and AFAIK requires the user to install an extension.
I don't believe other browsers support this natively yet.
You can save html document onto <canvas> or <foreignObject> of <svg> element, then send data URL, ArrayBuffer or Blob of <canvas> or <svg>; or alternatively, send html document as encoded string.

Detecting audio in third party online advertisements

Is is possible to detect when an online advertisement is playing audio? In an ideal world, I could do this through a browser in real-time and send a notification that would include the DOM information needed to remove the ad. In this type of model, I could theoretically then use all clients (web visitors) as test subjects. I believe that this might be far fetched, and thus will need to rely on a smaller subset of users. I'm looking for any way to do this. I have even thought about using speakers and a microphone to detect when a sound is playing, and then some how capture a screenshot and the DOM information. Please help. Thank you.
In a previous question, it was stated that this is not possible.
Edit:
No. The best you could do is maybe be able to detect that the browser is or is not playing sound. Even if you could do that, you would have difficulty detecting which tab or page was creating the sound. And if you could, you would have difficulty detecting the offending HTML. Could be flash, javascript, or HTML 5's tag. And even if you could do all of this, IE would surely (being the pain that it is) not support any of your attempts to complete this objective.
Bottom Line: You simply can't do what you are wanting to do. You would need to be able to query the browser for playing audio to be confident that you could handle all supported methods of playing sound in a browser. AFAIK, no browsers support this, so you are out of luck.

HTML5 Local Storage of audio element source - is it possible?

I've been experimenting with the audio and local storage features of html5 of late and have run into something that has me stumped.
I'd like to be able to cache or store the source of the audio element locally to enable speedier and offline playback. The problem is I can't see how this is possible with the current implementation.
I have tried the following using WebKit:
Creating a manifest file to set up local caching but the audio file appears not to be a cacheable item maybe due to the way it is stream or something
I have also attempted to use javascript to put an audio object into local storage but the size of the mp3 makes this impossible due to memory issues (i think).
I have tried to use the data uri and base64 to use the html as a audio transport that can be cached but again the filesize makes this prohibitive. Also the audio element does not seem to like this in WebKit (works fine in mozilla)
I have tried several methods of putting the data into the local database store. Again suffering the same issues as the other cases.
I'd love to hear any other ideas anyone may have as to how I could achieve my goal of offline playback using caching/local storage in WebKit.
I've been trying to do this myself, on the iOS (for iPhone / iPad) but it refuses to cache audio files in offline, even if in Cache Manifest.
It does not error, but instead simply pretends to have played the audio element if invoked via JavaScript without a control. If it's embedded with a control, it displays an alternate control that says "Cannot play audio file.". It works fine if the application is able to go online.
It seems not to cache the audio, playing another sound resource seems to cause it to clear the previous resource from memory - this is pretty worthless functionality even when online.
I have experimented with base64 encoding the audio as data URI's. This works in Safari on the desktop (at least for fairly short samples of around 20-30k that I've been using) but seems not to be supported at all on iOS - it silently does nothing, which is highly annoying.
I don't know about other vendors - Google Chrome used to not support data URI's for audio but perhaps they fixed it... - it seems like it's not possible for now though.
Update: Minor discrepancy with iPhone OS 3.x (tested with 3.1.2): If an audio element is specified in an offline web app but it doesn't have a control, it displays a non-interactive control with a non-animated spinner on it (which it definitely shouldn't do). I assume this is fixed in iOS 4.x (which should be out next week).
So it's been a while since I asked this question and I thought i'd give some info about how we solved it. Basically we encoded the data into PNG's using a similar technique to this:
http://audioscene.org/scene-files/yury/pngencoding/sample.html
Then cached the image on the mobile device using html5 local storage and accessed it as needed. The PNG's were pretty big but this worked for us.
I spent a while trying to do this for a game I'm making, and since as far as I could tell browsers (Firefox and Chrome) still don't support caching of audio elements I thought I'd post the solution I found.
There is a workaround described here: http://dougx.net/plunder/index.php#code
I can confirm it works pretty well, but is probably better suited to smaller files. As he describes here (http://dougx.net/plunder/GameSounds.txt), you encode the audio as base64 strings, and give them a data:audio/ogg;base64 (or any compatible audio format) header, which HTML5 audio can then read in. Because this is just a string, the browser will cache it.
I guess it would be preferable to get the manifest approach working, since this feels like the most relevant mechanism for locally caching the file.
What happens if you alter the audio file's HTTP headers, e.g. Content-Type and Expires? Does the browser do something different if the file extension is changed?
I see you've had no luck so far.
You might want to take a look at JAI (JavaScript Audio Interface) ("the world's first javascript interface for web <audio>"). Or get in touch with Alastair MacDonald, who wrote it.
Failing that, the HTML5 Doctor may be able to assist.
Adding video and audio files to local storage works with iOS 4.3.
I just added a video and an audio file to manifest and they both got downloaded to offline storage on iPad.

Categories