web audio, createMediaElementSource variable sources - javascript

lets say i wan't to have an app that has variable audio sources as audio tags like so:
<audio preload="auto" src="1.mp3" controls="" class="muzz"></audio>
<audio preload="auto" src="track.mp3" controls="" class="muzz"></audio>
Depending on which of them is played it should be passed to createMediaElementSource and then the sound would be sent to analyser and various things would be done with it, but it doesn't work:
var trackName;
//get the source of clicked track
$(".muzz").on("play", function(){
trackName = $(this).attr("src");
console.log("got a source: ", trackName);
audio = new Audio();
audio.src=trackName;
context = new AudioContext();
analyser = context.createAnalyser();
source = context.createMediaElementSource(audio);
source.connect(analyser);
analyser.connect(context.destination);
letsDraw();
});
the console log displays the correct source name, the letsDraw() method is supposed to draw a spectrogram of the audio playing:
function letsDraw(){
console.log("draw called");
window.requestAnimationFrame(letsDraw);
fbc_array = new Uint8Array(analyser.frequencyBinCount);
analyser.getByteFrequencyData(fbc_array); //get frequency from the analyser node
ctx.clearRect(0, 0, canvas.width, canvas.height);
ctx.fillStyle="white";
ctx.font = "bold 12px Arial";
ctx.fillText("currently playing:" + trackName, 10, 20);//this works
bars = 150;
for(var i = 0; i < analyser.frequencyBinCount; i++){ //but this doesn't
/*fill the canvas*/
x = i *2;
barWidth = 1;
barHeight = -(fbc_array[i]/1.8);
//colours react to the frequency loudness
hue = parseInt(500 * (1 - (barHeight / 200)), 10);
ctx.fillStyle = 'hsl(' + hue + ',75%,50%)';
ctx.fillRect(x, canvas.height, barWidth, barHeight);
}
}
it was working fine with one set audio source, but fails with variable sources, any ideas would be very appreciated.
of course, no errors are even thrown in the console at all.

What I don't get is why you take the src and put that in a new Audio object, as you already have them.
It is also way better to create a source from both <audio> tags. You just create a function that runs on page load (so when everything on the page is ready so you won't get any errors about elements not yet existing etc.).
Before I start writing a piece of code, what do you expect to happen? Should it be possible to have both tags playing at the same time, or should one be stopped if you click play on the other? If it shouldn't play at the same time, you'd better make one <audio> tag and create two buttons which each set the src of the tag.
Another problem with your code is that you already have the <audio> elements, and when you want them to play you just create a new audio element and append the src to it.. What is the logic behind that?
EDIT:
Here is an example of using multiple sources with only one <audio> element.
The HTML code should look like this:
<audio id="player" src="" autoplay="" controls=""></audio>
<div id="buttons">
<!--The javascript code will generate buttons with which you can play the audio-->
</div>
Then you use this JS code:
onload = function () { //this will be executed when the page is ready
window.audioFiles = ['track1.mp3', 'track2.mp3']; //this is gonna be the array with all file names
window.player = document.getElementById('player');
window.AudioContext = window.AudioContext || window.webkitAudioContext;
context = new AudioContext();
source = context.createMediaElementSource(player);
analyser = context.createAnalyser();
source.connect(analyser);
analyser.connect(context.destination);
//now we take all the files and create a button for every file
for (var x in audioFiles) {
var btn = document.createElement('button');
btn.innerHTML = audioFiles[x];
btn.onclick = function () {
player.src = audioFiles[x];
//so when the user clicks the button, the new source gets appended to the audio element
}
document.getElementById('buttons').appendChild(btn);
}
}
Hope the comments explain it good enough
EDIT2:
You want to know how to do this for multiple elements. What you want to do is to create all the audio elements when the page loads, and create all sources for it. This will decrease mess when starting to play audio. The code is here.
All you need to do is have the for loop that runs for every media file you have, which it will create an audio element for with the appropriate source, and then create a sourcenode for that (createMediaElementSource), and connect that sourcenode to the analyser.
I also want to say something about your visualiser code though. If you do not override the font, color or anything, you don't need to execute that every animationframe. Once on init is enough.

At the top of your first code block, try this:
var trackName,
context = new AudioContext();
And remove context = new AudioContext(); from the click handler.
You should only have one AudioContext for the page.

Related

Using Canvas for rendering video in React

I'm new to React. I'm trying to create a <canvas> that will render a video in React so that I can provide a restriction on video-downloading as mentioned here(the paint on canvas part). (I know I can't prevent users from downloading it, just a preventive measure from my side for newbie users).
I was using this for HTML-JS :
var canvas = document.getElementById("canV");
var ctx = canvas.getContext("2d");
var video = document.createElement("video");
video.src = "http://techslides.com/demos/sample-videos/small.mp4";
video.addEventListener('loadeddata', function() {
video.play(); // start playing
update(); //Start rendering
});
function update(){
ctx.drawImage(video,0,0,256,256);
requestAnimationFrame(update); // wait for the browser to be ready to present another animation fram.
}
and
<canvas id="canV" width='256' height='256'></canvas>
Now, How can I do this in React? Also, any other simple preventive measures to prevent the videos from getting downloaded?
Also, there is this answer, Can I do something like this in react/gatsby? If yes, then please guide.
Thanks.

First frame from captureStream() not sending

We're working on a project where people can be in a chat room with their webcams, and they can grab a snapshot of someone's cam at that moment, do some annotations on top of it, and then share that modified picture as if it was their own webcam (like sharing a whiteboard).
Capturing the webcam stream into a canvas element where it can be edited was relatively easy. Finding the canvas element on our page and doing a .getContext('2d') on it,
Used an open library to add editing tools to it. Grabbing a stream from that canvas was done like so:
var canvasToSend = document.querySelector('canvas');
var stream = canvasToSend.captureStream(60);
var room = osTwilioVideoWeb.getConnectedRoom();
var mytrack = null;
room.localParticipant.publishTrack(stream.getTracks()[0]).then((publication) => {
mytrack = publication.track;
var videoElement = mytrack.attach();
});
This publishes the stream alright, but the first frame will not get sent unless you draw something else on the canvas. Let's say you drew 2 circles and then hit Share, the stream will start but will not be shown on the recipients' side unless you draw a line, or another circle, or anything. It seems like it needs a frame change for it to send data over.
I was able to force this with developer tools by doing something like context.fill();, but when I tried adding this after the publishing function, even in a then()... no luck.
Any ideas on how to force this "refresh" to happen?
So it seems it is expected behavior (and thus would make my FF buggy).
From the specs about the frame request algorithm:
A new frame is requested from the canvas when frameCaptureRequested is true and the canvas is painted.
Let's put some emphasis on the "and the canvas as been painted". This means that we need both these conditions, and while captureStream itself, or its frameRate argument ellapsing or a method like requestFrame would all set the frameCaptureRequested flag to true, we still need the new painting...
The specs even have a note stating
This algorithm results in a captured track not starting until something changes in the canvas.
And Chrome indeed seems to generate an empty CanvasCaptureMediaStreamTrack if the call to captureStream has been made after the canvas has been painted.
const ctx = document.createElement('canvas')
.getContext('2d');
ctx.fillRect(0,0,20,20);
// let's request a stream from before it gets painted
// (in the same frame)
const stream1 = ctx.canvas.captureStream();
vid1.srcObject = stream1;
// now let's wait that a frame ellapsed
// (rAF fires before next painting, so we need 2 of them)
requestAnimationFrame(()=>
requestAnimationFrame(()=> {
const stream2 = ctx.canvas.captureStream();
vid2.srcObject = stream1;
})
);
<p>stream initialised in the same frame as the drawings (i.e before paiting).</p>
<video id="vid1" controls autoplay></video>
<p>stream initialised after paiting.</p>
<video id="vid2" controls autoplay></video>
So to workaround this, you should be able to get a stream with a frame by requesting the stream from the same operation as a first drawing on the canvas, like stream1 in above example.
Or, you could redraw the canvas context over itself (assuming it is a 2d context) by calling ctx.drawImage(ctx.canvas,0,0) after having set its globalCompositeOperation to 'copy' to avoid transparency issues.
const ctx = document.createElement('canvas')
.getContext('2d');
ctx.font = '15px sans-serif';
ctx.fillText('if forced to redraw it should work', 20, 20);
// produce a silent stream again
requestAnimationFrame(() =>
requestAnimationFrame(() => {
const stream = ctx.canvas.captureStream();
forcePainting(stream);
vid.srcObject = stream;
})
);
// beware will work only for canvas intialised with a 2D context
function forcePainting(stream) {
const ctx = (stream.getVideoTracks()[0].canvas ||
stream.canvas) // FF has it wrong...
.getContext('2d');
const gCO = ctx.globalCompositeOperation;
ctx.globalCompositeOperation = 'copy';
ctx.drawImage(ctx.canvas, 0, 0);
ctx.globalCompositeOperation = gCO;
}
<video id="vid" controls autoplay></video>

Queue getting multiple video frames

I'm creating a PDF output tool using jsPDF but need to add multiple pages, each holding a canvas image of a video frame.
I am stuck on the logic as to the best way to achieve this as I can't reconcile how to queue the operations and wait on events to achieve the best result.
To start I have a video loaded into a video tag and can get or set its seek point simply with:
video.currentTime
I also have an array of video seconds like the following:
var vidSecs = [1,9,13,25,63];
What I need to do is loop through this array, seek in the video to the seconds defined in the array, create a canvas at these seconds and then add each canvas to a PDF page.
I have a create canvas from video frame function as follows:
function capture_frame(video_ctrl, width, height){
if(width == null){
width = video_ctrl.videoWidth;
}
if(height == null){
height = video_ctrl.videoHeight;
}
canvas = document.createElement('canvas');
canvas.width = width;
canvas.height = height;
var ctx = canvas.getContext('2d');
ctx.drawImage(video_ctrl, 0, 0, width, height);
return canvas;
}
This function works fine in conjunction with the following to add an image to the PDF:
function addPdfImage(pdfObj, videoObj){
pdfObj.addPage();
pdfObj.text("Image at time point X:", 10, 20);
var vidImage = capture_frame(videoObj, null, null);
var dataURLWidth = 0;
var dataURLHeight = 0;
if(videoObj.videoWidth > pdfObj.internal.pageSize.width){
dataURLWidth = pdfObj.internal.pageSize.width;
dataURLHeight = (pdfObj.internal.pageSize.width/videoObj.videoWidth) * videoObj.videoHeight;
}else{
dataURLWidth = videoObj.videoWidth;
dataURLHeight = videoObj.videoHeight;
}
pdfObj.addImage(vidImage.toDataURL('image/jpg'), 'JPEG', 10, 50, dataURLWidth, dataURLHeight);
}
My logic confusion is how best to call these bits of code while looping through the vidSecs array as the problem is that setting the video.currentTime needs the loop to wait for the video.onseeked event to fire before code to capture the frame and add it to the PDF can be run.
I've tried the following but only get the last image as the loop has completed before the onseeked event fires and calls the frame capture code.
for(var i = 0; i < vidSecs.length; i++){
video.currentTime = vidSecs[i];
video.onseeked = function() {
addPdfImage(jsPDF_Variable, video);
};
}
Any thoughts much appreciated.
This is not a real answer but a comment, since I develop alike application and got no solution.
I am trying to extract viddeo frames from webcam live video stream and save as canvas/context, updated every 1 - 5 sec.
How to loop HTML5 webcam video + snap photo with delay and photo refresh?
I have created 2 canvases to be populated by setTimeout (5000) event and on test run I don't get 5 sec delay between canvas/contextes, sometimes, 2 5 sec. delayed contextes get populated with image at the same time.
So I am trying to implement
Draw HTML5 Video onto Canvas - Google Chrome Crash, Aw Snap
var toggle = true;
function loop() {
toggle = !toggle;
if (toggle) {
if (!v.paused) requestAnimationFrame(loop);
return;
}
/// draw video frame every 1/30 frame
ctx.drawImage(v, 0, 0);
/// loop if video is playing
if (!v.paused) requestAnimationFrame(loop);
}
to replace setInterval/setTimeout to get video and video frames properly synced
"
Use requestAnimationFrame (rAF) instead. The 20ms makes no sense. Most video runs at 30 FPS in the US (NTSC system) and at 25 FPS in Europe (PAL system). That would be 33.3ms and 40ms respectively.
"
I am afraid HTML5 provided no quality support for synced real time live video processing via canvas/ context, since HTML5 offers no proper timing since was intended to be event/s controlled and not run as real time run app code ( C, C++ ...).
My 100+ queries via search engine resulted in not a single HTML5 app I intend to develop.
What worked for me was Snap Photo from webcam video input, Click button event controlled.
If I am wrong, please correct me.
Two approaches:
create a new video element for every seek event, code provided by Chris West
reuse the video element via async/await, code provided by Adrian Wong

oddity about createMediaElementSource()

I was building an audio program and hit a stumbling block on the .createMediaElementSource method. I was able to solve the problem, but I do not quite know why the solution works.
In my HTML, I created an audio player: <audio id="myAudio><source src="music.mp3"></audio>
Now in my JS:
context = new AudioContext();
audio = document.getElementById('myAudio');
source = context.createMediaElementSource(audio);
audio.play();
doesn't work. The audio element loads, but doesn't play the song, nor is there audio.
However! This JS code works:
context = ...; //same as above
audio...;
source = context.createMediaElementSource(audio[0]);
audio.play();
All I changed was adding a [0] to the audio and the program suddenly works again. Since .getElementById doesn't return an array, I don't know why referring to audio as an array works, but just referring to audio does not.
A few months late, but in case others stumble upon this and want an answer:
This behaviour is described in the Web Audio API spec:
The createMediaElementSource method
Creates a MediaElementAudioSourceNode given an HTMLMediaElement. As a consequence of calling this method, audio playback from the HTMLMediaElement will be re-routed into the processing graph of the AudioContext.
Emphasis mine. Since the output from the audio element is now routed into the newly created MediaElementAudioSourceNode instance (instead of the original destination, usually your speakers), you need to route the output of the instance back to the original destination:
var audio = document.getElementById('myAudio');
var ctx = new AudioContext();
var src = ctx.createMediaElementSource(audio);
src.connect(ctx.destination); // connect the output of the source to your speakers
audio.play();
The reason it worked when you added [0] is that document.getElementById doesn't return an array, or an element with a defined key of "0". As such, you might as well have written ctx.createMediaElementSource(undefined), which doesn't re-route the audio from the #myAudio element.

HTML5 object selection and audio playback

In my software engineering course we're designing a language-learning website in HTML5 but none of us have experience with it.
The main functionality of the website is essentially playing an audio clip, having the user select an image based on what they think the audio clip corresponds to, and then playing (and internally recording) a noise based on whether they correctly identified the object.
I'm just looking for some general ideas as to how this can be done. I'm assuming the use of javascript is a given but not entirely sure about how to 'link' an image to the audio clip to determine if the user selection was correct.
Any help is appreciated and thanks a lot for taking a look!
This is a sketch: http://jsfiddle.net/pimvdb/eZWq5/.
Define a pair, and each pair can be "displayed"; it will make the audio and images appear. Clicking on an image will check whether it was the image of the same pair that the audio was from.
It does not have the noise implemented among other things you'd probably like to alter, but it might be a start.
function Pair(image, sound) {
this.image = image;
this.sound = sound;
}
Pair.prototype.display = function() {
var div1 = document.getElementById("div1");
var div2 = document.getElementById("div2");
div1.innerHTML = "";
div2.innerHTML = "";
var audio = document.createElement("audio"); // make audio
audio.src = this.sound;
audio.autoplay = true;
audio.controls = true;
div1.appendChild(audio);
var thisPair = this;
for(var i = 0; i < pairs.length; i++) { // make images
var img = document.createElement("img");
img.src = pairs[i].image;
img.onclick = function() {
if(this.src === thisPair.image) {
alert('yey');
} else {
alert('nop');
}
showRandom();
};
div2.appendChild(img);
}
};
var pairs = [
new Pair("http://lorempixel.com/100/100/?1",
"http://upload.wikimedia.org/wikipedia/"
+ "commons/a/a9/Tromboon-sample.ogg"),
new Pair("http://lorempixel.com/100/100/?2",
"http://upload.wikimedia.org/wikipedia/"
+ "commons/c/c8/Example.ogg")
];
function showRandom() {
pairs[Math.random() * pairs.length | 0].display();
}
showRandom();
To learn more about the APIs for javascript and the HTML5 audio check out Mozilla Docs.
To answer your question on how to 'link' things here is a rough sketch about how it would work. You have a list of audio clips and a list of images that goes with each and which image is correct (like a multiple choice question). Then you display the images and play the sound clip. Wait for the user to click on one of the images and then check whether it is right or not. This all can be accomplished with pretty straightforward javscript. I haven't worked with the audio API as much but you can come back with a specific question if you need more help.

Categories