I have been trying to set up webcam recording in Google Colab. Through adapting code found here: Is there any way to capture live video using webcam in google colab? I was able to get some video, however, if I try to record for more than 60 seconds the environment crashes.
I've tried a few rough workarounds:
Recording numerous smaller videos and concatenating them. This led to gaps in the footage.
Calling the 'wait' function a few time... 30 seconds, then another 30 seconds ect. This still led to crashing with anything over 60 seconds.
This work has led me to believe that the problem isn't a timeout, it's caused by the video buffer size being exceeded by the incoming video stream.
I'm new to javascript and realize this isn't really what Colab is made for, but would appreciate any help - even if it's just letting me know that this isn't a feasible thing to do! My code can be found below, thanks in advance!
# Run the function, get the video path as saved in your notebook, and play it back here.
from IPython.display import HTML, display, Javascript
from base64 import b64encode, b64decode
from google.colab.output import eval_js
def record_video_timed(filename='video.mp4'):
js = Javascript("""
async function wait(ms) {
return new Promise(resolve => {
setTimeout(resolve, ms);
});
}
async function recordVideo() {
// mashes together the advanced_outputs.ipynb function provided by Colab,
// a bunch of stuff from Stack overflow, and some sample code from:
// https://developer.mozilla.org/en-US/docs/Web/API/MediaStream_Recording_API
// Optional frames per second argument.
const options = { mimeType: "video/webm; codecs=vp9" };
const stream = await navigator.mediaDevices.getUserMedia({video: true});
let recorder = new MediaRecorder(stream, options);
recorder.start();
await wait(10000);//less than 60 or it crashes
recorder.stop();
let recData = await new Promise((resolve) => recorder.ondataavailable = resolve);
let arrBuff = await recData.data.arrayBuffer();
let binaryString = "";
let bytes = new Uint8Array(arrBuff);
bytes.forEach((byte) => {
binaryString += String.fromCharCode(byte);
})
return btoa(binaryString);
}
""")
try:
display(js)
data = eval_js('recordVideo({})')
binary = b64decode(data)
with open(filename, "wb") as video_file:
video_file.write(binary)
print(
f"Finished recording video. Saved binary under filename in current working directory: {filename}"
)
except Exception as err:
# In case any exceptions arise
print(str(err))
return filename
#######
video_path = record_video_timed()
Related
I'm trying to create a 1ch (mono) MediaStreamTrack with a MediaStreamAudioDestinationNode. According to the standard, this should be possible.
const ctx = new AudioContext();
const destinationNode = new MediaStreamAudioDestinationNode(ctx, {
channelCount: 1,
channelCountMode: 'explicit',
channelInterpretation: 'speakers',
});
await ctx.resume(); // doesn't make a difference
// this fails
expect(destinationNode.stream.getAudioTracks()[0].getSettings().channelCount).equal(1);
Result:
Chrome 92.0.4515.107 always creates a stereo track
Firefox 90 returns nothing for destinationNode.stream.getAudioTracks()[0].getSettings() even though getSettings() should be fully supported
What am I doing wrong here?
Edit:
Apparently both firefox and chrome actually produce a mono track, they just don't tell you the truth. Here's a workaround solution for Typescript:
async function getNumChannelsInTrack(track: MediaStreamTrack): Promise<number> {
// unfortunately, we can't use track.getSettings().channelCount, because
// - Firefox 90 returns {} from getSettings() => see: https://bugzilla.mozilla.org/show_bug.cgi?id=1307808
// - Chrome 92 always reports 2 channels, even if that's incorrect => see: https://bugs.chromium.org/p/chromium/issues/detail?id=1044645
// Workaround: Record audio and look at the recorded buffer to determine the number of audio channels in the buffer.
const stream = new MediaStream();
stream.addTrack(track);
const mediaRecorder = new MediaRecorder(stream);
mediaRecorder.start();
return new Promise<number>((resolve) => {
setTimeout(() => {
mediaRecorder.stop();
mediaRecorder.ondataavailable = async ({ data }) => {
const offlineAudioContext = new OfflineAudioContext({
length: 1,
sampleRate: 48000,
});
const audioBuffer = await offlineAudioContext.decodeAudioData(
await data.arrayBuffer()
);
resolve(audioBuffer.numberOfChannels);
};
}, 1000);
});
}
I don't think you're doing anything wrong. It's a known issue (https://bugs.chromium.org/p/chromium/issues/detail?id=1044645) in Chrome which just didn't get fixed so far.
I think in Firefox it isn't even implemented. This bug (https://bugzilla.mozilla.org/show_bug.cgi?id=1307808) indicates that getSettings() only returns those values that can be changed so far.
I think it would be helpful if you star/follow theses issues or comment on them to make sure they don't get forgotten about.
I am working on a video editor, and the video is rendered using the canvas, so I use the JS MediaRecorder API, but I have run into an odd problem, where, because the MediaRecorder API is primarily designed for live streams, my exported WebM file doesn't show how long it is until it's done, which is kinda annoying.
This is the code I am using:
function exportVideo() {
const stream = preview.captureStream();
const dest = audioContext.createMediaStreamDestination();
const sources = []
.concat(...layers.map((layer) => layer.addAudioTracksTo(dest)))
.filter((source) => source);
// exporting doesn't work if there's no audio and it adds the tracks
if (sources.length) {
dest.stream.getAudioTracks().forEach((track) => stream.addTrack(track));
}
const recorder = new MediaRecorder(stream, {
mimeType: usingExportType,
videoBitsPerSecond: exportBitrate * 1000000,
});
let download = true;
recorder.addEventListener("dataavailable", (e) => {
const newVideo = document.createElement("video");
exportedURL = URL.createObjectURL(e.data);
if (download) {
const saveLink = document.createElement("a");
saveLink.href = exportedURL;
saveLink.download = "video-export.webm";
document.body.appendChild(saveLink);
saveLink.click();
document.body.removeChild(saveLink);
}
});
previewTimeAt(0, false);
return new Promise((res) => {
recorder.start();
audioContext.resume().then(() => play(res));
}).then((successful) => {
download = successful;
recorder.stop();
sources.forEach((source) => {
source.disconnect(dest);
});
});
}
And if this is too vague, please tell me what is vague about it.
Thanks!
EDIT: Narrowed down the problem, this is a chrome bug, see https://bugs.chromium.org/p/chromium/issues/detail?id=642012. I discovered a library called https://github.com/legokichi/ts-ebml that may be able to make the webm seekable, but unfortunately, this is a javascript project, and I ain't setting up Typescript.
JS MediaRecorder API exports a non seekable WebM file
Yes, it does. It's in the nature of streaming.
In order to make that sort of stream seekable you need to post process it. There's a npm embl library pre-typescript if you want to attempt it.
I'd like to be able to end a Google speech-to-text stream (created with streamingRecognize), and get back the pending SR (speech recognition) results.
In a nutshell, the relevant Node.js code:
// create SR stream
const stream = speechClient.streamingRecognize(request);
// observe data event
const dataPromise = new Promise(resolve => stream.on('data', resolve));
// observe error event
const errorPromise = new Promise((resolve, reject) => stream.on('error', reject));
// observe finish event
const finishPromise = new Promise(resolve => stream.on('finish', resolve));
// send the audio
stream.write(audioChunk);
// for testing purposes only, give the SR stream 2 seconds to absorb the audio
await new Promise(resolve => setTimeout(resolve, 2000));
// end the SR stream gracefully, by observing the completion callback
const endPromise = util.promisify(callback => stream.end(callback))();
// a 5 seconds test timeout
const timeoutPromise = new Promise(resolve => setTimeout(resolve, 5000));
// finishPromise wins the race here
await Promise.race([
dataPromise, errorPromise, finishPromise, endPromise, timeoutPromise]);
// endPromise wins the race here
await Promise.race([
dataPromise, errorPromise, endPromise, timeoutPromise]);
// timeoutPromise wins the race here
await Promise.race([dataPromise, errorPromise, timeoutPromise]);
// I don't see any data or error events, dataPromise and errorPromise don't get settled
What I experience is that the SR stream ends successfully, but I don't get any data events or error events. Neither dataPromise nor errorPromise gets resolved or rejected.
How can I signal the end of my audio, close the SR stream and still get the pending SR results?
I need to stick with streamingRecognize API because the audio I'm streaming is real-time, even though it may stop suddenly.
To clarify, it works as long as I keep streaming the audio, I do receive the real-time SR results. However, when I send the final audio chunk and end the stream like above, I don't get the final results I'd expect otherwise.
To get the final results, I actually have to keep streaming silence for several more seconds, which may increase the ST bill. I feel like there must be a better way to get them.
Updated: so it appears, the only proper time to end a streamingRecognize stream is upon data event where StreamingRecognitionResult.is_final is true. As well, it appears we're expected to keep streaming audio until data event is fired, to get any result at all, final or interim.
This looks like a bug to me, filing an issue.
Updated: it now seems to have been confirmed as a bug. Until it's fixed, I'm looking for a potential workaround.
Updated: for future references, here is the list of the current and previously tracked issues involving streamingRecognize.
I'd expect this to be a common problem for those who use streamingRecognize, surprised it hasn't been reported before. Submitting it as a bug to issuetracker.google.com, as well.
My bad — unsurprisingly, this turned to be an obscure race condition in my code.
I've put together a self-contained sample that works as expected (gist). It helped me tracking down the issue. Hopefully, it may help others and my future self:
// A simple streamingRecognize workflow,
// tested with Node v15.0.1, by #noseratio
import fs from 'fs';
import path from "path";
import url from 'url';
import util from "util";
import timers from 'timers/promises';
import speech from '#google-cloud/speech';
export {}
// need a 16-bit, 16KHz raw PCM audio
const filename = path.join(path.dirname(url.fileURLToPath(import.meta.url)), "sample.raw");
const encoding = 'LINEAR16';
const sampleRateHertz = 16000;
const languageCode = 'en-US';
const request = {
config: {
encoding: encoding,
sampleRateHertz: sampleRateHertz,
languageCode: languageCode,
},
interimResults: false // If you want interim results, set this to true
};
// init SpeechClient
const client = new speech.v1p1beta1.SpeechClient();
await client.initialize();
// Stream the audio to the Google Cloud Speech API
const stream = client.streamingRecognize(request);
// log all data
stream.on('data', data => {
const result = data.results[0];
console.log(`SR results, final: ${result.isFinal}, text: ${result.alternatives[0].transcript}`);
});
// log all errors
stream.on('error', error => {
console.warn(`SR error: ${error.message}`);
});
// observe data event
const dataPromise = new Promise(resolve => stream.once('data', resolve));
// observe error event
const errorPromise = new Promise((resolve, reject) => stream.once('error', reject));
// observe finish event
const finishPromise = new Promise(resolve => stream.once('finish', resolve));
// observe close event
const closePromise = new Promise(resolve => stream.once('close', resolve));
// we could just pipe it:
// fs.createReadStream(filename).pipe(stream);
// but we want to simulate the web socket data
// read RAW audio as Buffer
const data = await fs.promises.readFile(filename, null);
// simulate multiple audio chunks
console.log("Writting...");
const chunkSize = 4096;
for (let i = 0; i < data.length; i += chunkSize) {
stream.write(data.slice(i, i + chunkSize));
await timers.setTimeout(50);
}
console.log("Done writing.");
console.log("Before ending...");
await util.promisify(c => stream.end(c))();
console.log("After ending.");
// race for events
await Promise.race([
errorPromise.catch(() => console.log("error")),
dataPromise.then(() => console.log("data")),
closePromise.then(() => console.log("close")),
finishPromise.then(() => console.log("finish"))
]);
console.log("Destroying...");
stream.destroy();
console.log("Final timeout...");
await timers.setTimeout(1000);
console.log("Exiting.");
The output:
Writting...
Done writing.
Before ending...
SR results, final: true, text: this is a test I'm testing voice recognition This Is the End
After ending.
data
finish
Destroying...
Final timeout...
close
Exiting.
To test it, a 16-bit/16KHz raw PCM audio file is required. An arbitrary WAV file wouldn't work as is because it contains a header with metadata.
This: "I'm looking for a potential workaround." - have you considered extending from SpeechClient as a base class? I don't have credential to test, but you can extend from SpeechClient with your own class and then call the internal close() method as needed. The close() method shuts down the SpeechClient and resolves the outstanding Promise.
Alternatively you could also Proxy the SpeechClient() and intercept/respond as needed. But since your intent is to shut it down, the below option might be your workaround.
const speech = require('#google-cloud/speech');
class ClientProxy extends speech.SpeechClient {
constructor() {
super();
}
myCustomFunction() {
this.close();
}
}
const clientProxy = new ClientProxy();
try {
clientProxy.myCustomFunction();
} catch (err) {
console.log("myCustomFunction generated error: ", err);
}
Since it's a bug, I don't know if this is suitable for you but I have used this.recognizeStream.end(); several times in different situations and it worked. However, my code was a bit different...
This feed may be something for you:
https://groups.google.com/g/cloud-speech-discuss/c/lPaTGmEcZQk/m/Kl4fbHK2BQAJ
I'm creating a button to record a canvas using FFMPEG. Here's the code that finalizes the download process.
const recordButton = document.querySelector("#record")
recordButton.addEventListener('click', function () {
function startRecording() {
const { createFFmpeg } = FFmpeg;
const ffmpeg = createFFmpeg({
log: true
});
var transcode = async (webcamData) => {
var name = `record${id}.webm`;
await ffmpeg.load();
await ffmpeg.write(name, webcamData);
await ffmpeg.transcode(name, `output${id}.mp4`);
var data = ffmpeg.read(`output${id}.mp4`);
var video = document.getElementById('output-video');
video.src = URL.createObjectURL(new Blob([data.buffer], { type: 'video/mp4' }));
dl.href = video.src;
}
fn().then(async ({ url, blob }) => {
transcode(new Uint8Array(await (blob).arrayBuffer()));
})
...
id += 1}
The problem arises with the transcode variable. While the button works initially, every other attempt (on a single page load) fails just the async function. I'm not well versed enough in the function to know why it would only work once. That said, I do know it is the only bit of code that does not fire off upon second attempt.
It could be a few things. This is borrowed code, and I've repurposed it for multiple uses. I may have messed up the declarations. It may be an async issue. I tried to use available values to rig up a secondary, similar function, but that would defeat the purpose of the first.
I tried clearing and appending the DOM elements affected, but that doesn't do anything.
It seems to have something to do with:
await ffmpeg.load();
While FFMPEG has to download and load the library initially, it does not have to do so the second initialization. That my be the trigger that is not activating with successive uses.
I am implementing the MediaRecorder API as a way to record webm blobs for use as segments in a livestream. I have gotten the functionality I need but ran into a problem with Chrome crashing when calling MediaRecorder.stop() and MediaRecorder.start() multiple times in regular intervals.
Here is the recording code:
let Recorder = null;
let segmentBuffer = [];
let recordInterval = null;
let times = 0; //limiter for crashes
function startRecording() {
Recorder = new MediaRecorder(LocalStream, { mimeType: 'video/webm;codecs=opus, vp8', audioBitsPerSecond: 50000, videoBitsPerSecond: 1000000, });
//error evt
Recorder.onerror = (evt) => {
console.error(evt.error);
}
//push blob data to segments buffer
Recorder.ondataavailable = (evt) => {
segmentBuffer.push(evt.data);
}
//start initial recording
Recorder.start();
//set stop/start delivery interval every 5 seconds
recordInterval = setInterval(() => {
//stop recording
Recorder.stop();
//here to prevent crash
if (times > 5) {
Recorder = null;
console.log('end')
return;
}
times++;
//check if has segments
if (segmentBuffer.length) {
//produce segment, this segment is playable and not just a byte-stream due to start/stop
let webm = segmentBuffer.reduce((a, b) => new Blob([a, b], { type: "video/webm;codecs=opus, vp8" }));
//unset buffer
segmentBuffer = [];
//handle blob ie. send to server
handleBlob(webm)
}
//restart recorder
Recorder.start();
}, 5000);
}
I've also gone into the performance and discovered that a new audio and video encoder thread is started for each start/stop. I think this is the major problem as setting the interval to 10s vs. 5s creates fewer encoding threads. The buildup of multiple encoding threads causes chrome to lag and then finally crash afer a few passes.
How do I prevent multiple encoding threads from occurring while still being able to start/stop MediaRecorder (start/stop is the only way I found to achieve webm files that can be playable separately, otherwise each subsequent blob is missing the webm header part).
It appears that this is a bug in chrome:
https://bugs.chromium.org/p/chromium/issues/detail?id=1012378&q=mediaRecorder%20thread&can=2
I'm not sure there is anything you can do to fix it.