This question is mainly geared towards audio engineers.
I was looking at a few videos here and here concerning the Web Audio API in javascript and how people are using it to make music. I was wondering can this API be used to create sound quality on the level of existing Digital Audio Workstations (DAWs) like Ableton, FL Studio, Reason, Logic, etc.
Or is there some kind of inherent limitation to the browser? I think VSTs cannot be used by Web Audio API, but regardless of VSTs is there something that an audio engine used by DAWs can do that Web Audio cannot do in terms of quality? Or can the sound be just as good. I believe that samples can be played just as effectively (?), but maybe the synthesis would be limited?
Can HTML5 Web Audio be as high quality as a DAW?
Yes! Browsers use a high quality audio engine, with floating point samples all the way through.
Additionally, the API itself is one of the most straightforward to use and is quite extensible. It also supports very specific timing of control.
I think VSTs cannot be used by Web Audio API
That is correct.
but maybe the synthesis would be limited?
There are no real fundamental limitations. If you can code it, you can do it.
Now, there are a couple things you didn't ask about but still should know about. The first is an issue of latency. Browsers don't use ASIO or anything like that, so if you're using the Web Audio API to process and return low latency audio, you're going to have a bad time. In Chrome, you can improve this by letting Chrome open the audio device in exclusive mode.
Another issue is input/output channel count. This typically maxes out at 8 channels. Internally though, you can do pretty much whatever you want. The limitation is from the browser, not the Web Audio API itself.
There are some high quality audio applications that could run in the browser, but you can only do realtime audio in Chromium-based browsers, and you can't get realtime input even then.
Only Chromium supports high-priority audio threads in Audio Worklets, which you need for stable, realtime audio.
The WebMIDI API only runs on the main thread, which makes it useless for realtime input.
The WebUSB API is threadable, but does not permit communication with audio-class devices (which includes MIDI controllers (and soundcards)).
Keyboard, touch and mouse events must (currently) be handled by the main thread.
So, in short, you have no (practical) way to control realtime audio in Chromium, and cannot even do realtime audio (reliably) on any other platform.
Note: Adding support for WebMIDI workers was requested nine years ago, and the issue is still open (just for adding it to the spec).
Note: Electron will allow you to access protected USB device classes (Audio/MIDI), but Electron WebUSB support is still in progress. NWJS has WebUSB support, but no way to access protected device classes (though I opened an issue recently). WebUSB will work on these platforms eventually, but you would still need to implement web drivers for your controllers (even class-compliant ones), so this will not be an ideal solution, even once it it works.
Related
I tried to take a screenshot of a movie on the Disney+ web app when I realised that the video turns black as soon as I try to take a new screenshot with Snipping Tool. When I tried to do the same thing with OBS and Discord streams, I saw the same effect.
Interestingly, this only works for Chrome on my machine (I also tried Firefox and Edge and they just let me record my screen).
When I saw this, I became really curious on how they achieved this.
Does anyone have any idea how I can recreate this for my own web projects?
I became really curious on how they achieved this.
They use Widevine.
Widevine homepage.
https://ottverse.com/widevine-drm-how-does-it-work/
https://en.wikipedia.org/wiki/Widevine
News reports:
https://www.cordcuttersnews.com/sadly-disney-wont-work-on-chromebooks-linux-some-android-devices-because-of-drm/
https://www.tomsguide.com/news/disney-plus-will-work-on-chromebooks
https://www.androidpolice.com/2019/10/22/disney-will-only-work-on-devices-that-support-the-strictest-widevine-l3-drm/
It's also used by Netflix, Hulu and others.
Widevine is Google's DRM system that's baked-in to Chrome.
All other major browsers have adopted it as well, because no-one will use a browser that can't access Netflix.
Mozilla's and Microsoft's support is less user-hostile and as you noticed.
It's just a standard HTML5 <video> element - when the browser downloads the video stream it will see that it's encrypted with Widevine and that engages the Widevine client-side code which does all the DRM biz.
Though there are HTML and DOM features that facilitate DRM, I'm unsure of the extent that any JavaScript is required to use it - as theoretically everything the browser needs to know to load the DRM system should be embedded in the raw media stream.
On Windows, I understand (though unconfirmed) that Widevine makes use of SetWindowDisplayAffinity to block screenshots.
Nothing stops you from doing this in your own native code (e.g. if you had an Electron fork), but please don't because it's a real dick-move to your users, in addition to not working at all if the user has the DWM disabled (e.g. they're running Windows 7 with Aero disabled).
Has anyone any idea how I can recreate this for my own web projects?
You'll need to license Widevine yourself. This is a complicated process intended only for large media production companies and content rightsholders, not individuals or small businesses.
Anyway, even if you could, please don't. Why would you want to make to harder for users to share and appreciate your media? Just stick it up on YouTube instead.
HTML5 has an onboard volume detection.
<video controls width="320" height="176" onvolumechange="alert('The volume has been changed')">
This is for the player only.
Is it possible to detect volume from the system? Say a phone. Can I get volume readings of my phone from the mobile browser?
Or can I use JS to detect if the audio jack is plugged in?
Can Volume level be detected for audio out using GetUserMedia?
Short answer for this is no!
This is how web-browsers have been made for a very long time now, and Microsoft/Google/Apple/Mozilla/Opera have done this to avoid exploits through File.IO and such. Actually, we should be thankful for this - as it isolates you computer (with all your personal stuff) from the web-application (which might be hosted on a server with hostile intentions).
Actually, there is one way you could do this, but it would only work for IE users - and the answer is activeX plugins. Now, you could make your javascript create such a plugin and communicate with it - but who knows if activeX will be supported by IE in the future.
So per se, there is no way this is achievable with HTML5/Javascript alone - just think of the browser as the OS on which your web-app is running. So Google, Microsoft etc + the rest of the people who decides the HTML5 standards are the ones who decides how much access the developer gets. Camera hardware, GPS hardware etc. we do get to access, but we are not allowed to do whatever we want with that hardware - its not always the developers choice.
To preface this, all of my experience has been with developing Ruby on Rails web apps serving, essentially, text data - I don't have any experience yet with rich media.
I'm looking into developing a music notation app (for practice and because I'm dissatisfied with many of the current options), and am trying to determine whether or not to build a native app or a web app. Since all of my programming experience lies in web development, I would prefer to make this a web app if I can, because if I chose to go native I would need to find a decent cross-platform solution in addition to the usual pitfalls of developing natively.
I suspect that even with HTML5 I wouldn't have access to enough processing power to make a browser-based audio engine feasible. It would need to have the ability to play and sequence not a single large .wav file of a song, but many short .wav files concurrently and in sequence. There would be no need for recording - only playback.
But since I've never done this before I really don't know if this is possible. So, I wanted to pose this to more experienced HTML5 media developers:
For modern browsers (I'm not too concerned with support for older browsers), would such an audio engine be feasible? (As in, is the current landscape of JS libraries and the HTML5 audio API sufficient for building something like this?) Or would something like this be far too CPU / memory hungry in a browser?
Thanks!
In my experience Chrome and Firefox for desktop both work great with multiple, sequenced, .wav files embedded in audio tags. You can trigger them with JavaScript and it all seems to work great.
Unfortunately, Mobile Safari seemed to be unhappy with more than one audio tag playing at once, and Chrome for Android had similar limitations. This was a year ago, so things might be different now.
Since we were targeting iPads, we wound up using Phonegap to mix in some native sound libraries into our HTML & JavaScript app. One gotcha that tripped us up: if you want looping audio samples, stick to .wav files; .mp3 files have some silence at the start and end that's really tricky to get rid of.
I've been investigating Web Audio Api to build synths, and it's still very primitive and difficult to use. The infrastructure seems to have had a boost when Web Audio was introduced, but not much development since. Mobile audio on the web is rarely mentioned, too clunky, or unsupported.
At the same time, JS has become significantly faster with the V8 engine, JS loading can happen as needed, asynchronously, and non-blocking, which may solve many of the CPU/memory concerns you have. Look into NodeJS and various module loaders such as webpack, ES6, for more info.
I think there will be attention to web-based music apps because the web can support multiple users. That could be a good reason to keep pushing through these murky waters.. especially if you're talking concurrent connections.
See Molgav tracker. It works on desktop and mobile, it uses hundred of sampled instruments, it has UI adapted for small mobile touchscreens. See how examples sounds.
Since HTML5 isn't ready yet, and getUserMedia doesn't work in browsers i have tested, I am asking that is there any "wrappers" so I could grap audio stream from microphone and send it to server with Javascript. Similar wrapper has been made with web cameras: (google for a "jquery-webcam-plugin". Two link restriction), but I haven't found any similar things for microphone audio streaming.
Audio can be in any commonly known format.
Flash is not my daily basis, flash tips might not help.
Since Google gears has been deprecated, I think it is not wise to use AudioApi. This is not a good news either, so I am guessing that only choices are Java and Flash.
I will appreciate every hint and tip I get.
I'm not sure what you mean by "Flash is not my daily basis". Your other option is Java, but that's less user-friendly. If you're going to require a plug-in (and you'll have to until getUserMedia() is implemented), you might as well make it Flash.
Here is a project that wraps a simple Flash app with a Javascript API that streams audio via an HTTP post to a web-server of your choice:
https://code.google.com/p/wami-recorder/
It's not ideal, but it works and does not require a clunky Flash Media Server.
I am developing a web application that is supposed to display land traffic in real time in any part of the world. For a couple months I've been developing it using JavaScript and OpenLayers (http://www.openlayers.org) framework.
Unfortunately, this solution appears to be inefficient. There are hundreds (200-300) of objects on the map that are updated every couple minutes. The sheer operation of refreshing and rendering them takes significant amount of time that makes the
application less usable (slow responsivity to user actions).
At the moment I am considering changing the technology. Adobe Flex seems to be the most reasonable solution. There is at least one application written in it that does similar things to mine (http://casper.frontier.nl/).
However, I have a couple of concerns regarding Flex:
can it be easily integrated with the
HTML/CSS/JavaScript based part of the
application (for example, the
graphical interface should be
coherent when it comes to styles and
colors)?
I get an impression that
with latest browsers (mainly Chrome
9.0) JavaScript and CSS becoming more efficient. What are the chances that
in a couple of months JavaScript+CSS
will make it possible to implement
efficient, flash-like rich internet
application? (A note is needed here:
famous Canvas tag is not a solution
for my problem, at least not for now.
Rendering objects on map with canvas
proved to be less efficient than with
traditional SVG because the size of
the canvas is really big - a whole
browser window)
What are the chances
that Flash technology will be
abandoned (Apple policy, HTML5
growing support etc.) in
not-so-near-future (a couple of
years)?
There is a possibility that my client would like to view this application on mobile devices, including iPhone.
Any other solution for web-based interactive maps?
Can anybody address these issues?
Lazy comment repost:
I've used Google Maps JavaScript API + a custom canvas tile layer (see here and here) to draw maps with 10k+ markers, really quickly. Perhaps you just need to rethink your particular approach rather than totally rewriting your maps.
JavaScript running on a modern browser (say, IE7 or later) should easily be able to handle 200 or 300 object updates every few minutes. Granted, if you want to do all 300 updates at the same time, things might get a little slow. But if those updates occur spread out over that period, then you shouldn't have any trouble.
There are Asteroids games and 3D shooters written in JavaScript and that are very playable. They do dozens of updates per second.
I would suspect your framework (I know nothing about OpenLayers) or the way in which you're doing the updates before I suspected the platform.
My experience with Flash has been less that positive. Although it will interoperate with JavaScript, there are some strange edge cases that will trip you up. And my experience is that it's almost impossible not to trip over those edge cases unless what you're doing is truly trivial. And, of course, the lack of Flash support on the iPad and iPhone will make supporting those platforms impossible.
I think it's unlikely that Flash will be abandoned any time soon, as there are too many customers who continue to believe the silly notion that Flash is the way to build interactive Web apps. Although that was almost certainly true four years ago, browsers, computers, and JavaScript techniques have advanced to the point that the only use I currently have for Flash is to play video. And that use will go away in the next few years when HTML5 video becomes more prevalent. With Google's WebM video format and the expected high-quality tools to build WebM, Flash becomes almost irrelevant as a movie player, except for older content.
My advice would be to take a long hard look at your current implementation, study some other JavaScript applications that do frequent updates, and determine if it's really the platform rather than the way you're using it that is causing your performance problem.
No idea how many objects you can manage and update in js, but in my company (flashmaps.com) we have built flash-based maps handling many thousands of objects. The key issue in many cases in fact is that the map is completely overlapped by the markers. We use to recommend filtering the markers in those cases. We have a huge experience in building flash/flex-based maps, so on't hesitate to ask me any question on that.
By the way, I don't think Flash will get out of use soon. Apple's strategy on controlling iPhone/iPad apps (the real reason behind Flash banning) is causing a lot of trouble to web developers, that need to create specific versions of their websites for these devices, it's crazy. But I'm sure Apple will permit Flash, someday... Probably when many other tablets hit the streets supporting Flash. We'll see.
The awesome thing about MVC architecture is as long as you keep your domain logic separate from your business logic and UI, then it's relatively easy to create platform specific apps that access the same data. For example, you could build the same UI to run in the web browser (via html/javascript or flash player), on the desktop (via Air), and on an iPhone/iPad (via iOS) that all connect to the same server-side scripts. It's all a matter of personal choice which platform you choose. If a platform happens to fall out of fashion in the future, then you simply create a new UI on another platform.