I am trying to stream audio through a websocket on a node.js (express) server to a web browser. The audio is coming from an iOS device as 16-bit, mono wav files sampled at 4k (4000 samples per second).
Here's my code:
Server Code:
webSocketServer.on('connection', function connection(client) {
client.on('message', function(message) {
webSocketServer.clients.forEach(function each(connection) {
connection.send(message, { binary: true }
);
});
});
Client Code:
webSocket = new WebSocket('ws://' + window.location.hostname + ':8080/');
webSocket.binaryType = 'arraybuffer'
webSocket.onmessage = function(message) {
var arrayBuffer = message.data // wav from server, as arraybuffer
var source = audioContext.createBufferSource();
audioContext.decodeAudioData(arrayBuffer, function(buffer){
source.buffer = buffer
source.connect(audioContext.destination)
source.start(time);
time += source.buffer.duration
}, function(){
console.log('error')
})
};
decodeAudioData()appears to be working, however the audio buffer it returns is half the length I was expecting. (eg 4000 samples will only give me 0.5 seconds of audio. I originally thought this was because the wav is 16 bit and not 32, but switching to 32 caused decodeAudioData() to trigger it's error callback.
I figured this workaround could be added to the success callback:
source.playbackRate.value = 0.5 // play at half speed
time += source.buffer.duration * 2 // double duration
This gets the timing to work perfectly, but I am left with one problem: There is an audible 'click' or 'pop' between audio chunks. After spacing out the chunks by one second (time += (source.buffer.duration * 2) + 1), I was able to find that the click happens at the very beginning of each chunk.
So my main two head-scratchers are:
1) Why is the decoded audio playing at twice the speed I am expecting? Is my sampling rate of 4k too low for the Web Audio API? Why can't I decode 32-bit wav's?
2) I have some experience with digital audio workstations (ableton, logic) and I know that clicking sounds can arise if a wave 'jumps' from a sample back down to zero or vice versa (ie: starting/ending a sine wave in the midst of a phase). Is that what's going on here? Is there a way to get around this? Crossfading each individual sample seems silly. Why doesn't each chunk pickup where the last one left off?
1) The audio I was receiving was actually at 2k by mistake, but the wav header still said 4k, thus the double speed error.
2) See the last paragraph of Chris Wilsons answer here:
Finally - this is not going to work well if the sound stream does not match the default audio device's sample rate; there are always going to be clicks, because decodeAudioData will resample to the device rate, which will not have a perfect duration. It will work, but there will likely be artifacts like clicks at the boundaries of chunks. You need a feature that's not yet spec'ed or implemented - selectable AudioContext sample rates - in order to fix this.
Brion Vibbers AudioFeeder.js works great without any clicks but requires raw 32bit pcm data. Also be wary of upsampling artifacts!
Another option :
You can use the MediaSource API to overcome those glitches between the audio.
If you need full fledged research on this, use this : MSE for Audio
Related
So I have a new stream to which I will push 2048 bytes of audio buffer every 128ms (i.e. 16KB every second) but I seem to have more than 1 second of data pushed to the stream. (I think so by finding ffmpeg is still streaming sound even tens of seconds after I stop pushing data in it)
When I changed it to 1024 bytes/128ms (8KB/s), the sound stop right after I stop pushing data.
Please correct me if I do anything wrong!
Some background story
I am using ffmpeg to stream to a rtp server. It is one-time used, so I can't stop ffmpeg and start again. I don't want to use the ZeroMQ way because of latency. The target I am trying to archive is to have the same readable stream to ffmpeg and change the audio content on the go by stop pushing chunks of previous audio file and switch to the new one.
If you know some other ways to archive the goal, I would be very pleased to know. Thank you in advance!
My current code
const stream = new Stream.Readable();
// ...
// (send stream to ffmpeg)
// ...
fileP = fs.createReadStream(input);
fileP.on('readable', async () => {
var chunk;
while (previousStream && (chunk = fileP?.read(16 * 128))) {
if (!previousStream) break;
stream.push(chunk);
await delay(125);
}
});
I have working on streaming live video using WebRTC based on RTCConnection with library called simple-peer, but I have faced with some lag between live stream video (with MediaRecorder) and that was played on using MediaSource
Here is recorder:
var mediaRecorder = new MediaRecorder(stream, options);
mediaRecorder.ondataavailable = handleDataAvailable;
function handleDataAvailable(event) {
if (connected && event.data.size > 0) {
peer.send(event.data);
}
}
...
peer.on('connect', () => {
// wait for 'connect' event before using the data channel
mediaRecorder.start(1);
});
Here is source that is played:
var mediaSource = new MediaSource();
var sourceBuffer;
mediaSource.addEventListener('sourceopen', args => {
sourceBuffer = mediaSource.addSourceBuffer(mimeCodec);
});
...
peer.on('data', data => {
// got a data channel message
sourceBuffer.appendBuffer(data);
});
I open two tabs and connect to myself and I see delay in playing video ...
Seems like I configured badly MediaRecorder or MediaSource
Any help will be appreciated ;)
You've combined two completely unrelated techniques for streaming the video, and are getting the worst tradeoffs of both. :-)
WebRTC has media stream handling built into it. If you expect realtime video, the WebRTC stack is what you want to use. It handles codec negotiation, auto-scales bandwidth, frame size, frame rate, and encoding parameters to match network conditions, and will outright drop chunks of time to keep playback as realtime as possible.
On the other hand, if retaining quality is more desirable than being realtime, MediaRecorder is what you would use. It makes no adjustments based on network conditions because it is unaware of those conditions. MediaRecorder doesn't know or care where you put the data after it gives you the buffers.
If you try to play back video as it's being recorded, will inevitably lag further and further behind because there is no built-in catch-up method. The only thing that can happen is a buffer underrun, where the playback side waits until there is enough data to begin playback again. Even if it becomes minutes behind, it isn't going to automatically skip ahead.
The solution is to use the right tool. It sounds like from your question that you want realtime video. Therefore, you need to use WebRTC. Fortunately simple-peer makes this... simple.
On the recording side:
const peer = new Peer({
initiator: true,
stream
});
Then on the playback side:
peer.on('stream', (stream) => {
videoEl.srcObject = stream;
});
Much simpler. The WebRTC stack handles everything for you.
Background:
On Windows 10 I'm using getUserMedia (gUM) and MediaRecorder in Google Chrome (v71) to capture and encode a video stream.
I'm using the constraints parameter to gUM to tell it I want a video format of 352x288.
I'm requesting video/webm; codecs="avc1.42E01E" as the MIME type of the encoded stream (that's H.264 boxed in Matroska).
I'm selecting a cheezy webcam built into a laptop as the video source. It's called "EasyCamera" made by DMAX-AVC. It's tempting to call it CheezyCamera.
The video stream gets generated just fine.
Problem:
The dimensions of the encoded video in the stream are 440x360 rather than my requested 352x288. This information is embedded in the recorded stream, and only visible from the consumer of that data. Use of the various APIs reveals the gUM stream, MediaRecorder, and <video> element metadata all think the dimensions are the ones I asked for.
Of course, webcam, gUM, and MediaRecorder treat the constraints parameter as suggestions, and are free to respond with something different. In this case they respond with 440x360 when I request 352x288. This system functions as designed; that is not my problem.
To clarify, the unexpected 440x360 dimensions are only visible to the consumer of the recorded stream. I hope to find a way to know the producer-side webcam, gUM, and MediaEncoder signal chain is producing a different resolution than I requested.
How does the stream consumer know the stream dimensions? They're in the 'PixelWidth' and 'PixelHeight' Matroska boxes, and they're baked in to the H.264 stream. (Oddly enough, considering this is a software-chosen resolution, it isn't an integral number of 16x16 macroblocks. It still works of course.)
I can't parse the recorded data in the browser because it's stored in opaque blobs.
When I use a different, better, webcam (a Logitech C615) my encoded video stream is the size I requested.
My question:
Is there any way in the webcam / gUM / MediaRecorder / <video> signal chain to find the actual dimensions of the encoded stream in the browser actually recording the stream? That is, can I find the signal chain's response to my requested dimensions without decoding the generated stream?
Use MediaStream.getVideoTracks() method to get the video track (MediaStreamTrack), then use MediaStreamTrack.getSettings() to get MediaTrackSettings object, which contains the height and width of the video of the stream.
So if I request a video of 0 height specified as Constraints, I get a video of height 1 pixels. While streaming we can retrieve both the height I requested and the height I am getting as output.
function handleMediaStream(mystream){
let videoStreamTrack = mystream.getVideoTracks()[0];
let constraints = videoStreamTrack.getConstraints();
console.log(constraints.width, constraints.height);
// outputs: 640 0
let settings = videoStreamTrack.getSettings();
console.log(settings.width, settings.height);
// outputs: 640 1
}
let videoConstraints = { width: 640, height: 0 }
navigator.mediaDevices.getUserMedia({ video: videoConstraints })
.then(function create_media_recorder(mystream) {
handleMediaStream(mystream);
});
You might need to use the exact constraint keyword like in this test
const sdPalConstraints = {
video: {width: {exact: 352}, height: {exact: 288}}
};
// Assuming |video| exists and represents a <video> element.
navigator.mediaDevices.getUserMedia(sdPalConstraints)
.then(stream) => {video.srcObject = stream};
This is no guarantee that the WebCam will be streaming in that resolution, but if the produced one differs from the requested, VideoTrackAdapter will be engaged to adapt it for you (and for MediaRecorder).
After having tried the various things offered by other answerers of this question, I have not succeeded in solving this problem.
I conclude it's a bug somewhere in the signal chain webcam to Google Chrome to gUM.
I am trying to buffer MP3 songs using node js and socket io in real time. I basically divide the MP3 into segments of bytes and send it over to the client side where the web audio API will receive it, decode it and start to play it. The issue here is that the sound does not play continuously, there is a something like a 0.5 seconds gap between every buffered segment. How can I solve this problem
// buffer is a 2 seconds decoded audio ready to be played
// the function is called when a new buffer is recieved
function stream(buffer)
{
// creates a new buffer source and connects it to the Audio context
var source = context.createBufferSource();
source.buffer = buffer;
source.connect(context.destination);
source.loop = false;
// sets it and updates the time
source.start(time + context.currentTime);
time += buffer.duration; // time is global variable initially set to zero
}
The part where stream is called
// where stream bufferQ is an array of decoded MP3 data
// so the stream function is called after every 3 segments that are recieved
// the Web audio Api plays it with gaps between the sound
if(bufferQ.length == 3)
{
for(var i = 0, n = bufferQ.length ; i < n; i++)
{
stream(bufferQ.splice(0,1)[0]);
}
}
should I use a different API other than the web audio API or is there a way to schedule my buffer so that it would be played continuously ?
context.currentTime will vary depending on when it is evaluated, and every read has an implicit inaccuracy due to being rounded to the nearest 2ms or so (see Firefox BaseAudioContext/currentTime#Reduced time precision). Consider:
function stream(buffer)
{
...
source.start(time + context.currentTime);
time += buffer.duration; // time is global variable initially set to zero
Calling source.start(time + context.currentTime) for every block of PCM data will always start the playback of that block at whatever the currentTime is now (which is not necessarily related to the playback time) rounded to 2ms, plus the time offset.
For playing back-to-back PCM chunks, read currentTime once at the beginning of the stream, then add each duration to it after scheduling the playback. For example, PCMPlayer does:
PCMPlayer.prototype.flush = function() {
...
if (this.startTime < this.audioCtx.currentTime) {
this.startTime = this.audioCtx.currentTime;
}
...
bufferSource.start(this.startTime);
this.startTime += audioBuffer.duration;
};
Note startTime is only reset when it represents a time in the past - for continuous buffered streaming it is not reset as it will be a value some time in the future. In each call to flush, startTime is used to schedule playback, and is only increased by each PCM data duration, it does not depend on currentTime.
Another potential issue is that the sample rate of the PCM buffer that you are decoding may not match the sample rate of the AudioContext. In this case, the browser resamples each PCM buffer separately, resulting in discontinuities at the boundaries of the chunks. See Clicking sounds in Stream played with Web Audio Api.
It's an issue with mp3 files, each mp3 file has a few frames of silence at the start and end.
If you use wav files or time the start and stop of each file properly you can fix it
How to jump to certain time offsets in HTML5 Audio elements?
They say you can simply set their currentTime property (emphasis mine):
The currentTime attribute must, on getting, return the current
playback position, expressed in seconds. On setting, if the media
element has a current media controller, then it must throw an
INVALID_STATE_ERR exception; otherwise, the user agent must seek to
the new value (which might raise an exception).
Alas, it doesn't seem to work (I need it in Chrome).
There are similar questions, although, no answers.
To jump around an audio file, your server must be configured properly.
The client sends byte range requests to seek and play certain regions of a file, so the server must response adequately:
In order to support seeking and playing back regions of the media that
aren't yet downloaded, Gecko uses HTTP 1.1 byte-range requests to
retrieve the media from the seek target position. In addition, if you
don't serve X-Content-Duration headers, Gecko uses byte-range requests
to seek to the end of the media (assuming you serve the Content-Length
header) in order to determine the duration of the media.
Then, if the server responses to byte range requests correctly, you can set the position of audio via currentTime:
audio.currentTime = 30;
See MDN's Configuring servers for Ogg media (the same applies for other formats, actually).
Also, see Configuring web servers for HTML5 Ogg video and audio.
Works on my chrome...
$('#audio').bind('canplay', function() {
this.currentTime = 29; // jumps to 29th secs
});
Both audio and video media accept the #t URI Time range property
song.mp3#t=8.5
To dynamically skip to a specific point use HTMLMediaElement.currentTime:
audio.currentTime = 8.5;
A much easier solution is
var element = document.getElementById('audioPlayer');
//first make sure the audio player is playing
element.play();
//second seek to the specific time you're looking for
element.currentTime = 226;
Make sure you attempt to set the currentTime property after the audio element is ready to play. You can bind your function to the oncanplay event attribute defined in the specification.
Can you post a sample of the code that fails?
I was facing problem that progress bar of audio was not working but audio was working properly. This code works for me. Hope it will help you too.
Here song is the object of audio component.
HTML Part
<input type="range" id="seek" value="0" max=""/>
JQuery Part
$("#seek").bind("change", function() {
song.currentTime = $(this).val();
});
song.addEventListener('timeupdate',function (){
$("#seek").attr("max", song.duration);
$('#seek').val(song.currentTime);
});
Firefox also makes byte range requests when seeking content that it has not yet loaded- it is not just a chrome issue.
Set the response header "Accept-Ranges: bytes" and return a 206 Partial Content status code to allow any client to make byte range requests.
See https://developer.mozilla.org/en-US/docs/Web/HTTP/Configuring_servers_for_Ogg_media#Handle_HTTP_1.1_byte_range_requests_correctly
The #katspaugh's answer is correct, but there is a workaround that does not require any additional server configuration. The idea is to get the audio file as a blob, transform it to dataURL and use it as the src for the audio element.
Here is solution for angular $http, but if needed I can add vanilla JS version as well:
$http.get(audioFileURL,
{responseType:'blob'})
.success(function(data){
var fr = new FileReader;
fr.readAsDataURL(data);
fr.onloadend = function(){
domObjects.audio.src = fr.result;
};
});
cautions
This workaround is not suitable for large files.
It will not work cross-origin unless CORS are set properly.
Set time position to 5 seconds:
var vid = document.getElementById("myAudio");
vid.currentTime = 5;
In order to fix video rewind and fast forward on chrome just add /stream? to your html request for example:
<video src="youre.website.ext/{fileId}">
fix. <video src="your.website./{fileId}/stream?">
My problem was video rewind and forward didnt work on chrome but worked well on mozzila.