I am trying to build a real-time streaming app using websockets and ffmpeg.
I have 2 processes on the host machine for capturing video(screen) and audio(microphone) every second. Those 2 files (1 file for a second of audio and 1 file for a second of video) are combined into one mp4 file using ffmpeg.
ffmpeg -y -i video.mkv -i audio.wav -map 0:v -map 1:a -c:v libx264 -tune zerolatency -preset ultrafast -shortest -movflags frag_keyframe+empty_moov+default_base_moof -r 25 -g 50 -pass 1 -f mp4 -b:v 2M pipe:1
This file is sent to the client machine using a websocket.
On the web part I initialize a MediaSource object with a sourceBuffer.
videoSourceBuffer = mediaSource.addSourceBuffer('video/mp4; codecs="avc1.4D0033, mp4a.40.2"');
But I had a problem. Both audio and video were lagging behind. When I started the stream the delay was about 2 seconds. After 2 minutes the delay was 10 seconds.
I managed to see how many buffered frames I had in the sourceBuffer and speed up the stream to keep up with the source using this piece of code from How can I find remaing frames in a MediaSource SourceBuffer?.
let getBufferedLength = () => videoSourceBuffer.buffered.end(0) - videoElement.currentTime;
Now the video stream has always a delay about 1.5 seconds but the audio lags behind.
I cannot find a way to access how many audio frames are buffered or something so I can sync both the audio and video
Related
I am creating a streaming app like youtube while I am creating it I am facing many challenges related to different quality video converting.
My question is
Should I convert orginal video file into multiple video file (like 240p, 480p and 720p) and storage them? Or there is anyway where I can create a single video file which can be play in multiple qualities like youtube.
Multiple video files is the way to go. Currently, the most common approach to adaptive streaming is MPEG-DASH. The different video sizes and a MPD manifest, which is like a playlist for the different video sizes, can be generated using ffmpeg and mp4box. Many videoplayers, e.g. Video.js or Dash.js support adaptive streaming with MPEG-DASH.
Generate video files:
ffmpeg -y -i movie.avi -an -c:v libx264 -x264opts 'keyint=24:min-keyint=24:no-scenecut' -b:v 1500k -maxrate 1500k -bufsize 3000k -vf "scale=-1:720" movie-720.mp4
ffmpeg -y -i movie.avi -an -c:v libx264 -x264opts 'keyint=24:min-keyint=24:no-scenecut' -b:v 800k -maxrate 800k -bufsize 1600k -vf "scale=-1:540" movie-540.mp4
ffmpeg -y -i movie.avi -an -c:v libx264 -x264opts 'keyint=24:min-keyint=24:no-scenecut' -b:v 400k -maxrate 400k -bufsize 800k -vf "scale=-1:360" movie-360.mp4
ffmpeg -y -i movie.avi -vn -c:a aac -b:a 128k movie.m4a
Generate the manifest:
mp4box -dash-strict 2000 -rap -frag-rap -bs-switching no -profile "dashavc264:live" -out movie-dash.mpd movie-720.mp4 movie-540.mp4 movie-360.mp4 movie.m4a
Original source: https://gist.github.com/andriika/8da427632cf6027a3e0036415cce5f54
I have an audio file with extension .raw and recorded in stereo, and I need to convert it to mono with Node. I'm not able to find a example of how to do it this process or a library.
Any help would be great
You can use the following library
https://github.com/fluent-ffmpeg/node-fluent-ffmpeg to use ffmpeg on node.
And following to convert to stereo to mono via ffmpeg
ffmpeg -i stereo.flac -ac 1 mono.flac
Just pass the above options to ffmpeg via the library.
Here's my code that seems to work
var FFmpeg = require('fluent-ffmpeg');
var command = FFmpeg({
source: 'test.webm'
}).addOption('-ac', 1)
.saveToFile('out.mp3');
Let me Explain by my Code what issue i am facing...
This is my js file for using with PhantomJS. It simple tell it to open a page and take screenshots of it and store them in stdout.
var page = require("webpage").create();
page.viewportSize = { width: 640, height: 480 };
page.open("http://www.goodboydigital.com/pixijs/examples/12-2/", function() {
setInterval(function() {
page.render("/dev/stdout", { format: "png" });
}, 25);
});
And this is the cmd command I'm running to receive the captured images in ffmpeg in Windows Command Prompt.
phantomjs runner.js | ffmpeg -y -c:v png -f image2pipe -r 25 -t 10 -i - -c:v libx264 -pix_fmt yuv420p -movflags +faststart dragon.mp4
This command successfully starts the processes of PhantomJS and ffmpeg. But nothing happens for quite some time, after 15 minutes it gives an error saying:
"Failed to reallocate parser buffer"
thats it. I have referenced this code from this site on which the developer claims that it works
https://mindthecode.com/recording-a-website-with-phantomjs-and-ffmpeg/
Please see the attached Image for more explanation.
Image of Code
It could be related to stdout the ffmpeg process as it is being stdin through the pipe and after taking continuous image buffer is filled up and gives error.
You can review this from a well organized canvas recording application "puppeteer-recorder" on nodeJS
https://github.com/clipisode/puppeteer-recorder
Google just changed how Chrome preloads audio and video; see: https://googlechrome.github.io/samples/media/preload-metadata
It's my understanding that simply setting preload attribute to auto should fix the problem, however, I have been unable to do so:
https://jsfiddle.net/NinoSkopac/f4zscrdy/1/
let mp3 = 'https://s3-staging.read2me.online/audio/5a745d88483d86.76121223.mp3';
let audio = new Audio(mp3);
audio.preload = 'auto';
audio.play();
https://jsfiddle.net/NinoSkopac/rst8aspm/
<audio src="https://s3-staging.read2me.online/audio/5a745d88483d86.76121223.mp3" preload="auto" autoplay></audio>
Both of these will stop playing within a minute on Chrome 64 and Chrome 65-dev (other browsers and older Chromes are unaffected). I have replicated this issue on Mac, Windows and Android.
During my debug process, I have attached all possible media events to the JS object (i.e. audio.addEventListener('timeupdate', () => { console.log('timeupdate') })) and at first the events were firing like this:
progress
timeupdate
progress
timeupdate
[...]
Later like this:
timeupdate
timeupdate
timeupdate
[...]
When the audio playback stopped, I got a handful of error events, and dumping audio.error returns: PIPELINE_ERROR_DECODE: Failed to send audio packet for decoding: timestamp=81763265 duration=26122 size=201 side_data_size=0 is_key_frame=1 encrypted=0 discard_padding (ms)=(0, 0)
How do I fix this? Is this a Chrome bug?
UPDATE:
OGG plays fine: https://jsfiddle.net/NinoSkopac/2hktqcqt/1/
This does seem to be a Chrome bug: https://bugs.chromium.org/p/chromium/issues/detail?id=794782
A similar error on Github: https://github.com/video-dev/hls.js/issues/1529
UPDATE 2:
chrome://media-internals/ reveals this:
UPDATE 3:
This issue has been fixed in Chrome 65.
After a couple of days of trial and error and research, I have confirmed what doesn't and does work.
Doesn't work
mp3wrap
mp3wrap output.mp3 *.mp3
the output file is still corrupted and halts
ffmpeg
ffmpeg -i "concat:0.mp3|1.mp3" -acodec copy output.mp3
the output file is still corrupted and halts
Does work
mp3val with -f argument
Simply concatenate/implode your audio binaries (in PHP I do implode('', $audioBinaries) and then run mp3val -f concatenated-audio-file.mp3. The -f argument is essential and it means "try to fix errors".
How to install mp3val?
On MacOS: brew install mp3val
On Deb/Ubu: apt-get install mp3val
Same issue I am facing with concatenating technique. With ffmpeg, it works fine. Try ffmpeg with this command.
ffmpeg -f concat -i "{textfile}" -c:v copy -ab 48k -y "{output}"
textfile will have a list of files written per line.
I'm trying to do gapless playback of segments generated using ffmpeg:
I use ffmpeg to encode 3 files from a source with exactly 240000 samples # 48kHz, i.e. 5 seconds.
ffmpeg -i tone.wav -af atrim=start_sample=24000*0:end_sample=240000*1 -c:a opus 0.webm
ffmpeg -i tone.wav -af atrim=start_sample=24000*1:end_sample=240000*2 -c:a opus 1.webm
ffmpeg -i tone.wav -af atrim=start_sample=24000*2:end_sample=240000*3 -c:a opus 2.webm
When looking at the meta data (using ffprobe and ffmpeg -loglevel debug) from the file I get the following which seems to me inconsistent values:
Duration: 5.01,
Start 0.007
discard 648/900 samples
240312 samples decoded
If I have several of these files how would I play them seamlessly without gaps?
i.e. in a browser I've tried:
sourceBuffer.timestampOffset = 5 * n - 648/48000;
sourceBuffer.appendWindowStart = 5 * n;
sourceBuffer.appendWindowEnd = 5 * (n+1);
sourceBuffer.appendBuffer(new Uint8Array(buffer[n]));
However, there are audible gaps.
How many samples am I actually supposed to discard? 0.007 * 48000, 648, or 240312 - 240000?
Here is a html page which can be opened in Chrome to test.
You need a simple http server to run it:
<< ls
>> index.html 0.webm 1.webm 2.webm
<< npm install -g http-server
<< http-server --cors