How to synthesize audio using HTML5/Javascript on iPad - javascript

Has anybody out there got working sample code that synthesizes (and plays) audio using HTML5/Javascript on Mobile Safari on the iPad? I have found some examples for javascript-based sound synthesis on the web, but they all seem to work in Firefox only.

Recently I came across this js-library, I think this is what you want ->
https://github.com/oampo/Audiolet

Here is an example that works for me on an iPad:
www.cse.usf.edu/~turnerr/sound_demo.html
You can download the files from http://www.cse.usf.edu/~turnerr/Downloads/Sound_Demo.zip
This demo is on a Unix based server. But I have not been been able to get the same code to work on an IIS server. Hoping someone can provide some help with IIS.

You may be able to use generated data URIs of uniform length, such as 0.1 seconds. This would give you 1/10 of a second delay and you would generate that many "frames" of audio. I'm not sure entirely what formats the iPad supports, but I read it supports uncompressed WAV. Info on this file format is pretty easy to get, I remember generating WAV files a long time ago with some primitive byte manipulation methods.
Please post back with details!

I'm answering a very old question... but modern webkit browsers now support the Web Audio API. I wrote a very simple fiddle that generates chords using sine waves. There are only 4 built-in wave forms, but you can build your own using Fourier coefficients (array of numbers). You have to generate a new oscillator object for each note. They are single use objects. By connecting multiple oscillators to the same destination, you get polyphonic sounds.
let audio = new(window.AudioContext || window.webkitAudioContext)();
let s1 = audio.createOscillator();
let g1 = audio.createGain();
s1.type = 'sine';
s1.frequency.value = 600;
s1.start();
g1.gain.value = 0.5;
g1.connect(audio.destination);
s1.connect(g1);

Related

Capturing data stream of a JS / Leaflet animation as MP4

How can I capture the datastream of a JS / Leaflet animation and download it to MP4?
I am looking for output that looks something like the smooth path traced in these demos:
https://github.com/IvanSanchez/Leaflet.Polyline.SnakeAnim
Their author appears to have done them in ffcast or some screencasting softare.
However, I am looking for an automated solution that can be run as script, ideally one that works on the data stream itself (not the screen), perhaps with a headless browser.
I have tried puppeteer-gif and puppeteer-gif-cast but the best frame rate is jumpy.
I have tried WebRTC-Experiment but it requires me to set manual permissions. Ditto the Screen Capture API mentioned here, though this at least seems to work on the data stream itself.
The canvas captureStream method combined with the MediaRecorder API should do the trick.
Mind you that Chrome only supports webm as a container format (but does record h264) so you might need a postprocessing step with ffmpeg.

Single stream audio/video in Chrome WebRTC without ssrc tags

Is single stream audio (or video) via Chrome's WebRTC possible when you strip a=ssrc lines from the SDP?
I have tried filtering out a=ssrc lines (with the code below), but single stream audio did not work. I tried also single stream video and renaming instead of removing lines with the same result. I modify both offer and answer SDPs. Interestingly, this filtering works when you try sending SDPs with both audio & video - audio (only) will work in such scenario. However I had issues with re-negotiation in such scenario in our app, so this is probably not a valid solution.
You can see minimum example with the single stream audio / video in this repo: https://github.com/Tev-work/webrtc-audio-demo.
If it is possible, can you please provide minimal example of code with working audio? Preferably using the repo above, what should the modifySdp function (in public/client.js) do?
Currently it modifies sdp with this code:
sdp = sdp.replace(/a=ssrc/g, 'a=xssrc');
sdp = sdp.replace(/a=msid-semantic/g, 'a=xmsid-semantic');
sdp = sdp.replace(/a=mid/g, 'a=xmid');
sdp = sdp.replace(/a=group:BUNDLE/g, 'a=xgroup:BUNDLE');
If it is not possible, do you know whether such limitation has been officialy stated somewhere (please link it), or it just at some point became unworkable? It seems like it was working before (around M29, see comments here https://bugs.chromium.org/p/webrtc/issues/detail?id=1941 - no mention that this was not supposed to be working).
Motivation: We are sometimes sending SDPs via SIP PBXs, which sometimes filter out SSRC lines. Supporting multiple streams in such situations is obviously out of question (maybe with some server side hacking streams?), but supporting at least audio-only for such scenarios would be useful for us.
that should still be possible, even though there are some side-effects like (legacy) getStats not recognizing the stream, see (this bug)[https://bugs.chromium.org/p/webrtc/issues/detail?id=3342].
What you are attempting is to remove the a=ssrc lines before calling setLocalDescription. This is probably not going to work. If you want to simulate the scenario try removing them before calling setRemoteDescription with the SDP.

Capture video with alpha channel using canvas.captureStream()

I'm trying to capture the contents of a canvas element with alpha channel. When I do it I get the RGB values correctly but the Alpha channel seems to get dropped when playing back the resulting video. Is there a way to achieve this?
I'm running the following code:
var stream = canvas.captureStream(60);
var recorder = new MediaRecorder(stream);
recorder.addEventListener('dataavailable', finishCapturing);
recorder.addEventListener('stop', function(e) {
video.oncanplay = video.play;
video.src = URL.createObjectURL(new Blob(blobs, {type:"video/webm; codecs=vp9"}));
});
startCapturing();
recorder.start();
Here's a plnkr demonstrating the issue:
http://plnkr.co/edit/z3UL9ikmn6PtVoAHvY0B?p=preview
There is currently no options to enable (VP8/9 transparency channel) from the MediaRecorder API.
One could maybe open an issue on the W3C Mediacapture-Record git repo.
For this, I can guess a few reasons :
From what I understand, webm alpha channel is grossly an hack from chrome, and is not really implemented in the codec itself, nor completely stabilized.
MediaRecorder should be able to encode in many formats even if current implementations only support video webm/VP8 and webm/VP9 (chrome only). So it would mean that they would have to somehow keep the alpha channel in the raw stream, only for this new canvas.captureStream method. Historically, MediaStream mainly came from getUserMedia interface, and there were no need way of getting transparency from there.
[edit: Specs have changed since this answer was written, and MediaStreams should now keep the alpha channel, even if the consumer may not be able to use it, also Chrome now supports more video codecs.]
Chrome, which is the only one to support YUVA webm display in its stable channel (FF supports it in nightly 54), is still not able to include the duration inside their recorded files, let's them fix this before adding the hackish alpha_mode=true.
However, you can achieve it yourself kind of easily :
If you really want a transparent webm file (only readable in chrome and FF nightly), you can use a second canvas to do the recording, set its background (using fillRect) to a chroma that won't appear elsewhere in your drawings, draw the original one on it and record its stream. Once recorded, use ffmpeg to reencode the recorded video, this time with the alpha channel :
// all #00FF00 pixels will become transparent.
ffmpeg -i in.webm -c:v libvpx -vf "chromakey=0x00ff00:0.1:0.1,format=yuva420p" -auto-alt-ref 0 out.webm
I personally needed the -auto-alt-ref 0 flag, not sure everyone needs it though
But because of this other chrome bug, you'll actually have to append this other canvas on screen too, and hide it with css (opacity: 0; width:0px; height:0px; should do).
TL;DR
This API's implementations are far from being stabilized, no one has made the request for such a feature yet, it may come in near future though, and can be done server-side for the time being.

Where in metadata of a video in html5 is the fps saved?

In order to fully implement my custom html5 video player, I need the the exact frame rate of a video. However I have not been able to find it yet and am using a standard value of 25.
Typically videos have a frame rate value in meta-data so I accessed meta-data using something like this:
var vid = document.getElementById("myVideo");
vid.onloadedmetadata = function(e) {
console.log(e);
};
However I can't find frame rate here. Maybe I am not reading metadata at all.
I can use your help.
Thanks!
Try https://mediainfo.js.org (github)
It works on ui only, no backend needed
I just implemented it and it looks like it worked perfectly fine (at least in Chrome v 70.0.3538.77) for gettting wide media information
It looks like modern browsers beginning to work with some binary libraries
I'm 95% sure the standard html5 video api does not expose the fps information, from what I've read in the past months - other apis like MPEG-DASH and jwplayer do present more / different data.
Your best bet would be to snoop around w3schools.com/tags/ref_av_dom.asp and similar mdn pages.
You can calculate this in realtime yourself and it should work most of the time but I can imagine there's a case or two when it wouldn't. Look at PresentedFrames and then do something like:
fps = video.time / PresentedFrames
view more about PresentedFrames here (currently proposal) and similar attributes at the same link.
mediainfo.js works pretty good - even if used locally in a browser using 'http(s)://'.
To use it locally, just make sure you also download the accompanying mediainfo.wasm and put it into the same directory as mediainfo.min.js.
Alternatively you can install media-info using npm.
The only caveat is, that it doesn't run from the 'file://' protocol.

Why CreateJS code not working in firefox?

Have exported my flash cs6 game using createjs using "toolkit for createjs". All sounds exported to directory successfully.
Following code calls sounds
var manifest = [
{src:"sounds/cutter.wav", id:"cutter"}
];
var loader = new createjs.PreloadJS(false);
loader.installPlugin(createjs.SoundJS);
loader.onComplete = handleComplete;
loader.loadManifest(manifest);
function playSound(name, loop) {
createjs.SoundJS.play(name, createjs.SoundJS.INTERRUPT_EARLY, 0, 0, loop);
}
chorme and opera plays sound correctly but firefox.
Thanks in advance :)
I would recommend trying the latest code available at http://www.soundjs.com. You'll also find helpful tutorials and examples that work in firefox. It doesn't offer direct toolkit support, but it can help you understand what the exported code is doing and how to alter it.
My best guess without seeing the code in context is you are trying to call play without waiting for load to complete. This creates a race condition, where sometimes if the sound is cached it will work, and other times it will fail.
It's also possible that it has something to do with the wav encoding. With mp3s we've found you mostly need to stick to default encoding to ensure browsers can actually play the audio. You might also want to consider using mp3 and ogg files for the broadest audio support.
Hope that helps.

Categories