We Keep Coding

How to render videos in PixiJS without lowering performance

I am building a game with PixiJS and need to use videos as the background. I am able to load the video and have it fill the canvas but when the video plays I get a major performance hit on the rest of the game.
I have seen something about rendering the video with its own ticker so it's not rendering at ~60fps but I have not been able to figure out how to add multiple ticker instances in PixiJS.
Here is my code for the video:
let video = document.createElement('video');
video.preload = 'auto';
video.loop = true;
video.autoplay = false;
video.src = '../assets/placeholder/VIDEO FILE.mp4';
let texture = PIXI.Texture.from(video);
let videoSprite = new PIXI.Sprite(texture);
app.stage.addChild(videoSprite);
And here is my render code:
function render() {
app.ticker.add((delta) => {
elapsed += delta;
gameLoop()
});
}
Is this the proper way to use video textures in PixiJS?

I was able to fix this issue using VideoResource and setting the FPS:
const videoResource = new PIXI.resources.VideoResource(video);
videoResource.updateFPS = 30 // set video to render at 30FPS to avoid performance issues
let texture = PIXI.Texture.from(videoResource);
let videoSprite = new PIXI.Sprite(texture);
this.backgroundVideo = videoSprite
this.backgroundVideo.height = h
this.backgroundVideo.width = w
app.stage.addChild(this.backgroundVideo);

How to screen record a rolling n-second window

I'm aware of the MediaRecorder API and how to record screen/audio/video, and then download those recordings. I'm also aware of npm modules such as react-media-recorder that leverage that API.
I would like to record a rolling n-second window of screen recording, to allow the user to create clips and then be able to share those clips. I cannot record the entire session as I don't know how long they will last, meaning I don't know how big the recordings might get (I assume there is a limit to what the recording can have in memory.)
Is there any easy way to use MediaRecorder to record a rolling window (i.e. to always have in memory the last 30 seconds recorded)?

I spent quite a while trying to make this work. Unfortunately, the only solution that works for me involves making 30 recorders.
The naive solution to this problem is to call recorder.start(1000) to record data in one second intervals, then maintain a circular buffer on the dataavailable event. The issue with this is that MediaRecorder supports a very, very limited number of encodings. None of these encodings allow data packets to be dropped from the beginning, since they contain important metadata. With better understanding of the protocols, I'm sure that it is to some extent possible to make this strategy work. However, simply concatenating the packets together (when some are missing) does not create a valid file.
Another attempt I made used two MediaRecorder objects at once. One of them would record second-long start packets, and the other would record regular data packets. When taking a clip, this then combined a start packet from the first recorder with the packets from the second. However, this usually resulted in corrupted recordings.
This solution is not fantastic, but it does work: the idea is to keep 30 MediaRecorder objects, each offset by one second. For the sake of this demo, the clips are 5 seconds long, not 30:
<canvas></canvas><button>Clip!</button>
<style>
canvas, video, button {
display: block;
}
</style>
<!-- draw to the canvas to create a stream for testing -->
<script>
const canvas = document.querySelector('canvas');
const ctx = canvas.getContext('2d');
// fill background with white
ctx.fillStyle = 'white';
ctx.fillRect(0, 0, canvas.width, canvas.height);
// randomly draw stuff
setInterval(() => {
const x = Math.floor(Math.random() * canvas.width);
const y = Math.floor(Math.random() * canvas.height);
const radius = Math.floor(Math.random() * 30);
ctx.beginPath();
ctx.arc(x, y, radius, 0, Math.PI * 2);
ctx.stroke();
}, 100);
</script>
<!-- actual recording -->
<script>
// five second clips
const LENGTH = 5;
const codec = 'video/webm;codecs=vp8,opus'
const stream = canvas.captureStream();
// circular buffer of recorders
let head = 0;
const recorders = new Array(LENGTH)
.fill()
.map(() => new MediaRecorder(stream, { mimeType: codec }));
// start them all
recorders.forEach((recorder) => recorder.start());
let data = undefined;
recorders.forEach((r) => r.addEventListener('dataavailable', (e) => {
data = e.data;
}));
setInterval(() => {
recorders[head].stop();
recorders[head].start();
head = (head + 1) % LENGTH;
}, 1000);
// download the data
const download = () => {
if (data === undefined) return;
const url = URL.createObjectURL(data);
// download the url
const a = document.createElement('a');
a.download = 'test.webm';
a.href = url;
a.click();
URL.revokeObjectURL(url);
};
// stackoverflow doesn't allow downloads
// we show the clip instead
const show = () => {
if (data === undefined) return;
const url = URL.createObjectURL(data);
// display url in new video element
const v = document.createElement('video');
v.src = url;
v.controls = true;
document.body.appendChild(v);
};
document.querySelector('button').addEventListener('click', show);
</script>

How to create video srcObject from VideoFrame?

I'm learning webcodecs now, and I saw things as below:
So I wonder maybe it can play video on video element with several pictures. I tried many times but it still can't work.
I create videoFrame from pictures, and then use MediaStreamTrackGenerator to creates a media track. But the video appears black when call play().
Here is my code：
const test = async () => {
const imgSrcList = [
'https://gw.alicdn.com/imgextra/i4/O1CN01CeTlwJ1Pji9Pu6KW6_!!6000000001877-2-tps-62-66.png',
'https://gw.alicdn.com/imgextra/i3/O1CN01h7tWZr1ZiTEk1K02I_!!6000000003228-2-tps-62-66.png',
'https://gw.alicdn.com/imgextra/i4/O1CN01CSwWiA1xflg5TnI9b_!!6000000006471-2-tps-62-66.png',
];
const imgEleList: HTMLImageElement[] = [];
await Promise.all(
imgSrcList.map((src, index) => {
return new Promise((resolve) => {
let img = new Image();
img.src = src;
img.crossOrigin = 'anonymous';
img.onload = () => {
imgEleList[index] = img;
resolve(true);
};
});
}),
);
const trackGenerator = new MediaStreamTrackGenerator({ kind: 'video' });
const writer = trackGenerator.writable.getWriter();
await writer.ready;
for (let i = 0; i < imgEleList.length; i++) {
const frame = new VideoFrame(imgEleList[i], {
duration: 500,
timestamp: i * 500,
alpha: 'keep',
});
await writer.write(frame);
frame.close();
}
// Call ready again to ensure that all chunks are written before closing the writer.
await writer.ready.then(() => {
writer.close();
});
const stream = new MediaStream();
stream.addTrack(trackGenerator);
const videoEle = document.getElementById('video') as HTMLVideoElement;
videoEle.onloadedmetadata = () => {
videoEle.play();
};
videoEle.srcObject = stream;
};
Thanks!

Disclaimer:
I am not an expert in this field and it's my first use of this API in this way. The specs and the current implementation don't seem to match, and it's very likely that things will change in the near future. So take this answer with all the salt you can, it is only backed by trial.
There are a few things that seems wrong in your implementation:
duration and timestamp are set in micro-seconds, that's 1/1,000,000s. Your 500 duration is then only half a millisecond, that would be something like 2000FPS and your three images would get all displayed in 1.5ms. You will want to change that.
In current Chrome's implementation, you need to specify the displayWidth and displayHeight members of the VideoFrameInit dictionary (though if I read the specs correctly that should have defaulted to the source image's width and height).
Then there is something I'm less sure about, but it seems that you can't batch-write many frames. It seems that the timestamp field is kind of useless in this case (even though it's required to be there, even with nonsensical values). Once again, specs have changed so it's hard to know if it's an implementation bug, or if it's supposed to work like that, but anyway it is how it is (unless I too missed something).
So to workaround that limitation you'll need to write periodically to the stream and append the frames when you want them to appear.
Here is one example of this, trying to keep it close to your own implementation by writing a new frame to the WritableStream when we want it to be presented.
const test = async() => {
const imgSrcList = [
'https://gw.alicdn.com/imgextra/i4/O1CN01CeTlwJ1Pji9Pu6KW6_!!6000000001877-2-tps-62-66.png',
'https://gw.alicdn.com/imgextra/i3/O1CN01h7tWZr1ZiTEk1K02I_!!6000000003228-2-tps-62-66.png',
'https://gw.alicdn.com/imgextra/i4/O1CN01CSwWiA1xflg5TnI9b_!!6000000006471-2-tps-62-66.png',
];
// rewrote this part to use ImageBitmaps,
// using HTMLImageElement works too
// but it's less efficient
const imgEleList = await Promise.all(
imgSrcList.map((src) => fetch(src)
.then(resp => resp.ok && resp.blob())
.then(createImageBitmap)
)
);
const trackGenerator = new MediaStreamTrackGenerator({
kind: 'video'
});
const duration = 1000 * 1000; // in µs (1/1,000,000s)
let i = 0;
const presentFrame = async() => {
i++;
const writer = trackGenerator.writable.getWriter();
const img = imgEleList[i % imgEleList.length];
await writer.ready;
const frame = new VideoFrame(img, {
duration, // value doesn't mean much, but required
timestamp: i * duration, // ditto
alpha: 'keep',
displayWidth: img.width * 2, // required
displayHeight: img.height * 2, // required
});
await writer.write(frame);
frame.close();
await writer.ready;
// unlock our Writable so we can write again at next frame
writer.releaseLock();
setTimeout(presentFrame, duration / 1000);
}
presentFrame();
const stream = new MediaStream();
stream.addTrack(trackGenerator);
const videoEle = document.getElementById('video');
videoEle.srcObject = stream;
};
test().catch(console.error)
<video id=video controls autoplay muted></video>

How to detect when `canvas.captureStream()` is not supported?

It is known that iOS Safari does not support canvas.captureStream() to (e.g.) pipe its content into a video element, see this demo not working in iOS Safari.
However, canvas.captureStream() is a valid function in iOS Safari, and correctly returns a CanvasCaptureMediaStreamTrack, it just doesn't function as intended. In order to detect browsers that don't support canvas.captureStream, it would have been easy to do a test typeof canvas.captureStream === 'function', but at least for iOS Safari, we can't rely on that. Neither can we rely on the type of the returned value.
How do I write JavaScript that detects whether the current browser effectively supports canvas.captureStream()?

No iOS to test it here, but according to the comments on the issue you linked to, captureStream() actually works, what doesn't is the HTMLVideoElement's reading of this MediaStream. So that's what you actually want to test.
According to the messages there, the video doesn't even fail to load the video (i.e the metadata are correctly set and I don't expect events like error to fire, though if it does, then it's quite simple to test: check if a video is able to play such a MediaStream.
function testReadingOfCanvasCapturedStream() {
// first check the DOM API is available
if( !testSupportOfCanvasCapureStream() ) {
return Promise.resolve(false);
}
// create a test canvas
const canvas = document.createElement("canvas");
// we need to init a context on the canvas
const ctx = canvas.getContext("2d");
const stream = canvas.captureStream();
const vid = document.createElement("video");
vid.muted = true;
vid.playsInline = true;
vid.srcObject = stream;
let supports = false;
// Safari needs we draw on the canvas
// asynchronously after we requested the MediaStream
setTimeout(() => ctx.fillRect(0,0,5,5));
// if it failed, .play() would be enough
// but according to the comments on the issue, it isn't
return vid.play()
.then(() => supports = true)
.finally(() => {
// clean
stream.getTracks().forEach(track => track.stop());
return supports;
});
}
function testSupportOfCanvasCapureStream() {
return "function" === typeof HTMLCanvasElement.prototype.captureStream;
}
testReadingOfCanvasCapturedStream()
.then(supports => console.log(supports));
But if the video is able to play, but no images is painted, then we have to go a bit deeper and check what has been painted on the video. To do this, we'll draw some color on the canvas, wait for the video to have loaded and draw it back on the canvas before checking the color of the frame on the canvas:
async function testReadingOfCanvasCapturedStream() {
// first check the DOM API is available
if( !testSupportOfCanvasCapureStream() ) {
return false;
}
// create a test canvas
const canvas = document.createElement("canvas");
// we need to init a context on the canvas
const ctx = canvas.getContext("2d");
const stream = canvas.captureStream();
const clean = () => stream.getTracks().forEach(track => track.stop());
const vid = document.createElement("video");
vid.muted = true;
vid.srcObject = stream;
// Safari needs we draw on the canvas
// asynchronously after we requested the MediaStream
setTimeout(() => {
// we draw in a well knwown color
ctx.fillStyle = "#FF0000";
ctx.fillRect(0,0,300,150);
});
try {
await vid.play();
}
catch(e) {
// failed to load, no need to go deeper
// it's not supported
clean();
return false;
}
// here we should have our canvas painted on the video
// let's keep this image on the video
await vid.pause();
// now draw it back on the canvas
ctx.clearRect(0,0,300,150);
ctx.drawImage(vid,0,0);
const pixel_data = ctx.getImageData(5,5,1,1).data;
const red_channel = pixel_data[0];
clean();
return red_channel > 0; // it has red
}
function testSupportOfCanvasCapureStream() {
return "function" === typeof HTMLCanvasElement.prototype.captureStream;
}
testReadingOfCanvasCapturedStream()
.then(supports => console.log(supports));

Struggling to playback a Float 32 Array (Web Audio API)

I'm building a simple looper, to help me come to an understanding of the Web Audio API however struggling to to get a buffer source to play back the recorded audio.
The code has been simplified as much as possible however with annotation it's still 70+ lines, ommitting the CSS and HTML, so apologies for that. A version including the CSS and HTML can be found on JSFiddle:
https://jsfiddle.net/b5w9j4yk/10/
Any help would be much appreciated. Thank you :)
// Aim of the code is to record the input from the mike to a float32 array. then prass that to a buffer which is linked to a buffer source, so the audio can be played back.
// Grab DOM Elements
const playButton = document.getElementById('play');
const recordButton = document.getElementById('record');
// If allowed access to microphone run this code
const promise = navigator.mediaDevices.getUserMedia({audio: true, video: false})
.then((stream) => {
recordButton.addEventListener('click', () => {
// when the record button is pressed clear enstanciate the record buffer
if (!recordArmed) {
recordArmed = true;
recordButton.classList.add('on');
console.log('recording armed')
recordBuffer = new Float32Array(audioCtx.sampleRate * 10);
}
else {
recordArmed = false;
recordButton.classList.remove('on');
// After the recording has stopped pass the recordBuffer the source's buffer
myArrayBuffer.copyToChannel(recordBuffer, 0);
//Looks like the buffer has been passed
console.log(myArrayBuffer.getChannelData(0));
}
});
// this should stat the playback of the source intended to be used adter the audio has been recorded, I can't get it to work in this given context
playButton.addEventListener('click', () => {
playButton.classList.add('on');
source.start();
});
//Transport variables
let recordArmed = false;
let playing = false;
// this buffer will later be assigned a Float 32 Array / I'd like to keep this intimediate buffer so the audio can be sliced and minipulated with ease later
let recordBuffer;
// Declear Context, input source and a block processor to pass the input sorce to the recordBuffer
const audioCtx = new AudioContext();
const audioIn = audioCtx.createMediaStreamSource(stream);
const processor = audioCtx.createScriptProcessor(512, 1, 1);
// Create a source and corrisponding buffer for playback and then assign link
const myArrayBuffer = audioCtx.createBuffer(1, audioCtx.sampleRate * 10, audioCtx.sampleRate);
const source = audioCtx.createBufferSource();
source.buffer = myArrayBuffer;
// Audio Routing
audioIn.connect(processor);
source.connect(audioCtx.destination);
// When recording is armed pass the samples of the block one at a time to the record buffer
processor.onaudioprocess = ((audioProcessingEvent) => {
let inputBuffer = audioProcessingEvent.inputBuffer;
let i = 0;
if (recordArmed) {
for (let channel = 0; channel < inputBuffer.numberOfChannels; channel++) {
let inputData = inputBuffer.getChannelData(channel);
let avg = 0;
inputData.forEach(sample => {
recordBuffer.set([sample], i);
i++;
});
}
}
else {
i = 0;
}
});
})