My goal is to render a canvas in Node, and stream that canvas to an RTMP server (Twitch ultimately, but testing on a local RTMP server). The standard way to stream to RTMP seems to be ffmpeg, so I'm using that, spawned as a child process from within node. I've tried a bunch of different combinations of techniques and ffmpeg params to get a consistent framerate and a stream at "realtime" speed, but can't figure it out. Here's the paths I've gone down so far
Render canvas and send input in continuous interval
import { createCanvas } from 'canvas';
const canvas = createCanvas(1920, 1080);
const ctx = canvas.getContext('2d');
const fps = 30;
const ffmpeg = spawn('ffmpeg', [
'-re',
'-framerate', String(.fps),
'-r', String(fps),
'-i', '-',
'-vcodec', 'libx264',
'-r', String(fps),
'-s', '1920x1080',
'-g:v', String(2*fps),
'-c:a', 'aac',
'-f', 'flv', 'rtmp://127.0.0.1/live'
]);
ffmpeg.stdout.pipe(process.stdout)
ffmpeg.stderr.pipe(process.stderr)
const send = () => {
ctx.fillStyle = 'red'
ctx.fillRect(0, 0, 1920, 1080);
ctx.font = '100px Arial';
ctx.fillStyle = 'black'
ctx.fillText(new Date().toLocaleString(), 500, 500);
ffmpeg.stdin.write(canvas.toBuffer())
setImmediate(() => send())
}
send()
Observations
Took about 35 seconds for the stream to actually start (I think because of ffmpeg needing some amount of time to analyze the input?)
Frame rate extremely below what I set it to, and "speed" also very low, although I'm not 100% sure what this means. example log Frame= 906 fps=3.9 q=29.0 size= 311kB time=00:00:27.83 bitrate= 91.5kbits/s speed=0.119x
Stream behavior
Takes about a minute to load once opened in VLC
Timer on the stream starts about 1 minute behind real time, stays stuck on a single second for 30+ seconds, then shoots up a few seconds quickly, and gets stuck again
I had a hunch here that at least some of the reason for the strange behavior was that rendering the canvas in the same loop that I send input to ffmpeg in was too slow to achieve 30 FPS.
Render canvas in separate interval from ffmpeg input interval
Only render canvas FPS-times per second
Continue sending input to ffmpeg as fast as possible
import { createCanvas } from 'canvas';
const canvas = createCanvas(1920, 1080);
const ctx = canvas.getContext('2d');
let buffer = canvas.toBuffer();
const fps = 30;
const ffmpeg = spawn('ffmpeg', [
...same as before
]);
const render = () => {
ctx.fillStyle = 'red'
ctx.fillRect(0, 0, 1920, 1080);
ctx.font = '100px Arial';
ctx.fillStyle = 'black'
ctx.fillText(new Date().toLocaleString(), 500, 500);
buffer = canvas.toBuffer();
setTimeout(() => render(), 1/fps)
}
render();
const send = () => {
ffmpeg.stdin.write(buffer)
setImmediate(() => send())
}
send()
Observations
ffmpeg starts streaming almost immediately
fps starts out around 16, takes a couple seconds to hit 28, and then ~30 more seconds to hit 30fps. speed much closer to 1x, but not quite all the way. example log frame=15421 fps= 30 q=29.0 size= 4502kB time=00:08:31.66 bitrate= 72.1kbits/s speed=0.994x
Stream behavior
Takes about 5 seconds to load once opened in VLC
Timer stays stuck on the same second for multiple minutes
My hunch here for the stream being stuck on 1 timestamp is that while ffmpeg is sending frames out at 30 frames per second, I'm sending it frames much quicker than that. So in the first 1st of a second of streaming
Canvas renders with timestamp T 30 times
send runs N times where N is likely way higher than 30, sending ffmpeg N frames with the current timestamp
ffmpeg now has N frames with timestamp T on them, but can only send them out 30 per second, so it takes more than 1 second for the timestamp on the screen to change
Only send ffmpeg a frame every 1/FPS second
Same as before, but instead of sending ffmpeg frames as quickly as possible, only send it FPS frames every second.
import { createCanvas } from 'canvas';
const canvas = createCanvas(1920, 1080);
const ctx = canvas.getContext('2d');
let buffer = canvas.toBuffer();
const fps = 30;
const ffmpeg = spawn('ffmpeg', [
...same as before
]);
const render = () => {
...same as before
}
render();
const send = () => {
ffmpeg.stdin.write(buffer)
setTimeout(() => send(), 1/fps)
}
send()
Observations
ffmpeg takes a few seconds to start streaming
fps starts out high, around 28, and over the next minute or so drops down to 16. Speed drops along with it. example log frame= 1329 fps= 16 q=29.0 size= 463kB time=00:00:41.93 bitrate= 90.5kbits/s speed= 0.5x
Stream behavior
Takes about 10 seconds to load once opened in VLC
Timer increases about twice as fast as expected, then gets hung on one second for a bit, and then starts increasing again at same rate
I'll stop there, but tl;dr I've tried a whole bunch of different combinations of -re, -framerate, -fps_mode, -r ffmpeg args, and some other techniques in the code like continuing to use setImmediate to send frames, but use a date comparison to actually send a frame at an FPS rate. I'm sure there's probably some fundamental video streaming knowledge I'm missing, so I'm just looking for any sort of guidance on how I can get my canvas to stream at a "realtime" speed, or whatever I could be missing here.
Related
So I have a new stream to which I will push 2048 bytes of audio buffer every 128ms (i.e. 16KB every second) but I seem to have more than 1 second of data pushed to the stream. (I think so by finding ffmpeg is still streaming sound even tens of seconds after I stop pushing data in it)
When I changed it to 1024 bytes/128ms (8KB/s), the sound stop right after I stop pushing data.
Please correct me if I do anything wrong!
Some background story
I am using ffmpeg to stream to a rtp server. It is one-time used, so I can't stop ffmpeg and start again. I don't want to use the ZeroMQ way because of latency. The target I am trying to archive is to have the same readable stream to ffmpeg and change the audio content on the go by stop pushing chunks of previous audio file and switch to the new one.
If you know some other ways to archive the goal, I would be very pleased to know. Thank you in advance!
My current code
const stream = new Stream.Readable();
// ...
// (send stream to ffmpeg)
// ...
fileP = fs.createReadStream(input);
fileP.on('readable', async () => {
var chunk;
while (previousStream && (chunk = fileP?.read(16 * 128))) {
if (!previousStream) break;
stream.push(chunk);
await delay(125);
}
});
I'm working with a big company, and part of their flow is scanning a QR code to register some data. The problem is, in order to test this, I need to generate a QR code from the data, photograph it on my phone, and scan it in through my laptop's camera.
There are NPM modules for creating QR codes from data so that's okay, but I was wondering if it's somehow possible to override getUserMedia to return a stream of bytes that is just a QR code? I was thinking of maybe encapsulating all this into one nice chrome extension, but from looking around online, I'm not sure how I'd 'override' the camera input and replace it with a stream of QR code bytes instead.
Thanks
The HTMLCanvasElement has a captureStream() method that does produce a MediaStream with a VideoTrack similar to what getUserMedia({video: true}) produces.
This is a convenient way to test various things with a video stream, without needing an human in the loop:
const width = 1280;
const height = 720
const canvas = Object.assign(document.createElement("canvas"), {width, height});
const ctx = canvas.getContext("2d");
// you'd do the drawing you wish, here I prepare some noise
const img = new ImageData(width, height);
const data = new Uint32Array(img.data.buffer);
const anim = () => {
for(let i = 0; i<data.length; i++) {
data[i] = 0xFF000000 + Math.random()*0xFFFFFF;
}
ctx.putImageData(img, 0, 0);
requestAnimationFrame(anim);
};
requestAnimationFrame(anim);
// extract the MediaStream from the canvas
const stream = canvas.captureStream();
// Use it in your test (here I'll just display it in the <video>)
document.querySelector("video").srcObject = stream;
video { height: 100vh }
<video controls autoplay></video>
But in your case, you need to separate the concerns.
The QR code detection tests should be done on their own, and these can certainly use still images instead of a MediaStream.
I am building an web application for my experiment purpose. The aim here is to capture ~15-20 frames per second from the webcam and send it to the server. Once the frame is captured, it is converted to base64 and added to an array. After certain time, it is sent back to the server. Currently I am using imageCapture.takePhoto() to achieve this functionality. I get blob as a result which is then changed to base64. The application runs for ~5 seconds and during this time, frames are captured and sent to the server.
What are the more efficient ways to capture the frames through webcam to achieve this?
You can capture still images directly from the <video> element used to preview the stream from .getUserMedia(). You set up that preview, of course, by doing this sort of thing (pseudocode).
const stream = await navigator.getUserMedia(options)
const videoElement = document.querySelector('video#whateverId')
videoElement.srcObject = stream
videoElement.play()
Next, make yourself a canvas object and a context for it. It doesn't have to be visible.
const scratchCanvas = document.createElement('canvas')
scratchCanvas.width = video.videoWidth
scratchCanvas.height = video.videoHeight
const scratchContext = scratchCanvas.getContext('2d')
Now you can make yourself a function like this.
function stillCapture(video, canvas, context) {
context.drawImage( video, 0, 0, video.videoWidth, video.videoHeight)
canvas.toBlob(
function (jpegBlob) {
/* do something useful with the Blob containing jpeg */
}, 'image/jpeg')
}
A Blob containing a jpeg version of a still capture shows up in the callback. Do with it whatever you need to do.
Then, invoke that function every so often. For example, to get approximately 15fps, do this.
const howOften = 1000.0 / 15.0
setInterval (stillCapture, howOften, videoElement, scratchCanvas, scratchContext)
All this saves you the extra work of using .takePhoto().
I want to record the audio at 16000Hz and get the spectrogram of it. My model takes input of [null.1998.101]. I am unable to achieve it in javascript
const mic = await tf.data.microphone({
fftSize: 256,
columnTruncateLength: 101,
numFramesPerSpectrogram: 1998 ,
sampleRateHz:16000,
includeSpectrogram: true,
includeWaveform: true
});
const audioData = await mic.capture();
console.log(audioData)
const spectrogramTensor = audioData.spectrogram;
console.log(spectrogramTensor)
spectrogramTensor.print();
const waveformTensor = audioData.waveform;
waveformTensor.print();
mic.stop();
My model is trigger word detection.
In Python I have used the following code.
def graph_spectrogram(wav_file):
rate, data = get_wav_info(wav_file)
print(data)
print(len(data))
nfft = 200 # Length of each window segment
fs = 8000 # Sampling frequencies
noverlap = 120 # Overlap between windows
nchannels = data.ndim
if nchannels == 1:
pxx, freqs, bins, im = plt.specgram(data, nfft, fs, noverlap = noverlap)
elif nchannels == 2:
pxx, freqs, bins, im = plt.specgram(data[:,0], nfft, fs, noverlap = noverlap)
return pxx
The browser has a default and fixed value of the sampling rate of audio recording. The following will output the frequency rate of the browser.
new window.AudioContext().sampleRate
The error is thrown because 16000 does not match the browser sampling rate. It is currently not possible to change the sampling rate of an audio recording from the browser.
What can be done is
to train the model using the frequency rate
reshape (or slice ) the tensor to the model inputShape
record the audio and resample it (using this answer) and create a tensor from an audio recording (using this answer
Though I haven't tried, it seems that the value of the sample rate comes from the settings of the operating system. Changing it will allow the recording to have the right sample rate. On linux, the recording frequency can be set in the /etc/pulse/daemon.conf file
I am trying to buffer MP3 songs using node js and socket io in real time. I basically divide the MP3 into segments of bytes and send it over to the client side where the web audio API will receive it, decode it and start to play it. The issue here is that the sound does not play continuously, there is a something like a 0.5 seconds gap between every buffered segment. How can I solve this problem
// buffer is a 2 seconds decoded audio ready to be played
// the function is called when a new buffer is recieved
function stream(buffer)
{
// creates a new buffer source and connects it to the Audio context
var source = context.createBufferSource();
source.buffer = buffer;
source.connect(context.destination);
source.loop = false;
// sets it and updates the time
source.start(time + context.currentTime);
time += buffer.duration; // time is global variable initially set to zero
}
The part where stream is called
// where stream bufferQ is an array of decoded MP3 data
// so the stream function is called after every 3 segments that are recieved
// the Web audio Api plays it with gaps between the sound
if(bufferQ.length == 3)
{
for(var i = 0, n = bufferQ.length ; i < n; i++)
{
stream(bufferQ.splice(0,1)[0]);
}
}
should I use a different API other than the web audio API or is there a way to schedule my buffer so that it would be played continuously ?
context.currentTime will vary depending on when it is evaluated, and every read has an implicit inaccuracy due to being rounded to the nearest 2ms or so (see Firefox BaseAudioContext/currentTime#Reduced time precision). Consider:
function stream(buffer)
{
...
source.start(time + context.currentTime);
time += buffer.duration; // time is global variable initially set to zero
Calling source.start(time + context.currentTime) for every block of PCM data will always start the playback of that block at whatever the currentTime is now (which is not necessarily related to the playback time) rounded to 2ms, plus the time offset.
For playing back-to-back PCM chunks, read currentTime once at the beginning of the stream, then add each duration to it after scheduling the playback. For example, PCMPlayer does:
PCMPlayer.prototype.flush = function() {
...
if (this.startTime < this.audioCtx.currentTime) {
this.startTime = this.audioCtx.currentTime;
}
...
bufferSource.start(this.startTime);
this.startTime += audioBuffer.duration;
};
Note startTime is only reset when it represents a time in the past - for continuous buffered streaming it is not reset as it will be a value some time in the future. In each call to flush, startTime is used to schedule playback, and is only increased by each PCM data duration, it does not depend on currentTime.
Another potential issue is that the sample rate of the PCM buffer that you are decoding may not match the sample rate of the AudioContext. In this case, the browser resamples each PCM buffer separately, resulting in discontinuities at the boundaries of the chunks. See Clicking sounds in Stream played with Web Audio Api.
It's an issue with mp3 files, each mp3 file has a few frames of silence at the start and end.
If you use wav files or time the start and stop of each file properly you can fix it