Node JS access audio interface - javascript

I want to send audio signal coming from my audio interface (focusrite saffire) to my nodejs server. How should I go about doing this? The easiest way would be to access the audio interface from the browser (html5) like capturing microphone output with getUserMedia (https://developer.mozilla.org/en-US/docs/Web/API/MediaDevices/getUserMedia), but couldn't find a way to access my audio interface through that library. Otherwise, I'm planning on creating a desktop application, but don't know if there is a function/library to allow access to my usb-connected audio interface.

This probably has little to do with the Javascript or the MediaDevices API. On Linux, using Firefox, PulseAudio is required to interface with your audio hardware. Since your soundcard is an Input/Output interface, you should be able to test it pretty easily by simply playing any sound file in the browser.
Most of PulseAudio configuration can be achieved using pavucontrol GUI. You should check the "configuration" tab and both "input device" & "output device" tabs to make sure your Focusrite is correctly set up and used as sound I/O.
Once this is done, you should be able to access an audio stream using the following (only available if on a "secure context", ie localhost or served through HTTPS, as stated in the MDN page you mentionned):
navigator.mediaDevices.getUserMedia({ audio: true })
.then(function(stream) {
// do whatever you want with this audio stream
})
(code snippet taken from the MDN page about MediaDevices)
Sending audio to the server is a whole other story. Have you tried anything? Are you looking for a real-time communication with the server? If so, I'd start by having a look at the WebSockets API. The WebRTC docs might be worth reading too, but it is more oriented to client-to-client communications.

How to use exact device id
Use media device constrain to pass exact device id.
An sample would be
const preferedDeviceName = '<your_device_name>'
const deviceList = await navigator.mediaDevices.enumerateDevices()
const audio = devices.find((device) => device.kind === 'audioinput' && device.label === preferedDeviceName)
const {deviceId} = audio;
navigator.mediaDevices.getUserMedia({audio: { deviceId }}
How to process and send to the backend
Possible duplicate of this
Still not clear follow this

Related

Sending a stream from the browser to a Node JS server

The general idea: I created a Node JS program that interacts with multiple APIs to recreate a home assistant (like Alexia or Siri). It interacts mainly with IBM Watson. My first goal was to setup Dialogflow so that I could have a real AI processing the questions but due to the update to Dialogflow v2, I have to use Google Cloud and It's too much trouble for me so I just got with a hand-made script that reads possible responses from a configurable list.
My actual goal is to get an audio stream from the user and send it inside my main program. I have set up an express server. It responds with a HTML page when you GET on '/'. The page is the following:
<!DOCTYPE html>
<html lang='fr'>
<head>
<script>
let state = false
function button() {
navigator.mediaDevices.getUserMedia({audio: true})
.then(function(mediaStream) {
// And here I got my stream. So now what do I do?
})
.catch(function(err) {
console.log(err)
});
}
</script>
<title>Audio recorder</title>
</head>
<body>
<button onclick='button()'>Lancer l'audio</button>
</body>
</html>
It records audio from the user when they click the button with mediaDevices.getUserMedia()
My configuration looks like this:
What I'm looking for is a way to launch the recording, then press the stop button and when the stop button is pressed, it automatically send the stream to the Node program. It's preferable if the output is a stream because it's the input type for IBM Watson (or else I will have to store the file, then read it and then delete it).
Thanks for your attention.
Fun fact: The imgur ID of my image starts with "NUL", which means "NOOB" in French lol
Most browsers, but not all (I'm looking at you, Mobile Safari), support the capture and streaming of audio (and video, which you don't care about) using the getUserMedia() and MediaRecorder APIs. With these APIs you can transmit your captured audio in small chunks via WebSockets, or socket.io, or a series of POST requests, to your nodejs server. Then the nodejs server can send them along to your recognition service. The challenge here: the audio is compressed and encapsulated in webm. If your service accepts audio in that format, this strategy will work for you.
Or you can try using node-ogg and node-vorbis to accept and decode. (I haven't done this.)
There may be other ways. Maybe somebody who knows one will answer.

How can I prevent breakup/choppiness/glitches when using an AudioWorklet to stream captured audio?

We've been working on a JavaScript-based audio chat client that runs in the browser and sends audio samples to a server via a WebSocket. We previously tried using the Web Audio API's ScriptProcessorNode to obtain the sample values. This worked well on our desktops and laptops, but we experienced poor audio quality when transmitting from a handheld platform we must support. We've attributed this to the documented script processor performance issues (https://developer.mozilla.org/en-US/docs/Web/API/Web_Audio_API). On the handheld, with a script processor buffer size of 2048, audio consistently had breakups. At the next highest size interval (4096), the audio was smooth (no breakups), but there was too much latency (around two seconds).
Our results from ScriptProcessorNode prompted experimentation with Audio Worklet. Unfortunately, with our worklet implementation, audio quality is worse: both breakups and latency, even on our laptops. I'm wondering if there's a way to tweak our worklet implementation to get better performance, or if what we're experiencing is to be expected from (is "par for the course" for) the current state of audio worklets (Chromium issues 796330, 813825, and 836306 seem relevant).
Here's a little more detail on what the code does:
Create a MediaStreamStreamSourceNode with the MediaStream obtained from getUserMedia.
Connect the source node to our worklet node implementation (extends AudioWorkletNode).
Our worklet processor implementation (extends AudioWorkletProcessor) buffers blocks that arrive as the "input" argument to its process method.
When buffer is full, use MessagePort to send the buffer contents to the worklet node.
Worklet node transmits the buffer contents over a WebSocket connection.
The process method is below. The var "samples" is a Float32Array, which gets initialized to the buffer size and reused. I've experimented with buffer size a bit, but it doesn't seem to have an impact. The approach is based on the guidance in section 4.1 of AudioWorklet: The future of web audio to minimize memory allocations.
if (micKeyed == true) {
if (inputs[0][0].length == framesPerBlock) {
samples.set(inputs[0][0], currentBlockIndex * framesPerBlock);
currentBlockIndex++;
if (currentBlockIndex == lastBlockIndex) {
// console.log('About to send buffer.');
this.port.postMessage(samples);
currentBlockIndex = 0;
}
} else {
console.error("Got a block of unexpected length!!!");
}
}
return true;
Currently testing with PCs running Chrome 72.0.3626.109 on CentOS 7. Our handhelds are Panasonic FZ-N1 running Chrome 72.0.3626.105 on Android 6.0.1.
Thank you for reading and any suggestions you may be able to provide.

how to set audio out (speaker) in chrome (CEF3) in windows7 (WebRTC, C++)

Chrome or CEF3 should not use the default speaker of the system to play the sound (audio out).
Is there any way that user or developer can set default speaker in chrome or CEF3 other than system default?
e.g during WebRTC call, (in current scenario) audio sound is used to play by default audio sound device (speaker).
But in my scenario it should be played by selected speaker not system default.
This feature is not available in Chrome/Chromium/CEF.
Also, in Windows there is no public API to set which sound card use for output (it is left to the user to choose that):
This is a deliberate design, since we do not want applications to override audio setting set by the user
...However, here are some workarounds:
https://stackoverflow.com/a/2216886/833188
https://stackoverflow.com/a/20270535/833188
You should be able to do this purely in the JavaScript side of your application without having to mess with the CEF side. First, find what device ID you want to output to:
const devices = await navigator.mediaDevices.enumerateDevices();
const audioDevices = devices.filter(device => device.kind === 'audiooutput');
audioDevices will now contain all outputs. Then, set your Audio object's sink:
const audio = document.querySelector('audio');
audio.setSinkId(audioDevices[0].deviceId);
For more information: https://developer.mozilla.org/en-US/docs/Web/API/HTMLMediaElement/setSinkId

Change the VideoTrack of a MediaStream object

In a Nutshell: I'm trying to change the VideoTrack of a MediaStream object.
(Documentation: https://developer.mozilla.org/en-US/docs/WebRTC/MediaStream_API)
I have a MediaStream object __o_jsep_stream_audiovideo which is created by the sipml library.
__o_jsep_stream_audiovideo looks like this:
So it has one AudioTrack and one VideoTrack. At first the VideoTrack comes from the users camera (e.g label: "FaceTime Camera").
According to the Documentation:
A MediaStream consists of zero or more MediaStreamTrack objects, representing various audio or video tracks.
So we should be fine adding more Tracks to this Stream.
I'm trying to switch/exchange the VideoTrack with that from another stream. The other stream (streamB) originates from Chromes ScreenCapture api (label: "Screen")
I tried:
__o_jsep_stream_audiovideo.addTrack(streamB.getVideoTracks()[0])
which doesn't seem to have any effect.
I also tried assigning the videoTracks directly (which was desperate I know).
I must be missing something obvious could you point me in the right direction?
I'm running
Chrome (Version 34.0.1847.131) and
Canary (Version 36.0.1976.2 canary)
OSX 10.9.2
When you talk about change video track, we mean 2 areas:
change the remote video track (what the others can see from u)
WebRTC gets new version of doing that, since it deprecates addStream/removeStream.
However, the excelence is that they introduce new interface replaceTrack
stream.getTracks().forEach(function(track) {
// remote
qcClient.calls.values().forEach(function(call) {
var sender = call.pc.getSenders().find(function(s) {
return s.track.kind == track.kind;
});
sender.replaceTrack(track);
});
});
change your display video (You see yourself)
Better to just add a new video element (or using existing video element) But assign srcObject to the new captured stream
Adding and removing tracks on a MediaStream object do not signal a renegotiation and there are also issues with a MediaStream having two tracks of the same type in chrome.
You should probably just add the separate mediastream to the peer connection so that it can fire a re-negotiation and handle the streams. The Track add/remove functionality in chrome is very naive and not very granular and you should move away from it as much as you can.

Streaming Video to Web Browser

I would like to display a live video stream in a web browser. (Compatibility with IE, Firefox, and Chrome would be awesome, if possible.) Someone else will be taking care of streaming the video, but I have to be able to receive and display it. I will be receiving the video over UDP, but for now I am just using VLC to stream it to myself for testing purposes. Is there an open source library that might help me accomplish this using HTML and/or JavaScript? Or a good website that would help me figure out how to do this on my own?
I've read a bit about RTSP, which seems like the traditional option for something like this. That might be what I have to fall back on if I can't accomplish this using UDP, but if that is the case I still am unsure of how to go about this using RTSP/RTMP/RTP, or what the differences between all those acronyms are, if any.
I thought HTTP adaptive streaming might be the best option for a while, but it seemed like all the solutions using that were proprietary (Microsoft IIS Smooth Streaming, Apple HTTP Live Streaming, or Adobe HTTP Dynamic Streaming), and I wasn't having much luck figuring out how to accomplish it on my own. MPEG-DASH sounded like an awesome solution as well, but it doesn't seem to be in use yet since it is still so new. But now I am told that I should expect to receive the video over UDP anyways, so those solutions probably don't matter for me anymore.
I've been Googling this stuff for several days without much luck on finding anything to help me implement it. All I can find are articles explaining what the technologies are (e.g. RTSP, HTTP Adaptive Streaming, etc.) or tools that you can buy to stream your own videos over the web. Your guidance would be greatly appreciated!
It is incorrect that most video sites use FLV, MP4 is the most widely supported format and it is played via Flash players as well.
The easiest way to accomplish what you want is to open a S3Amzon/cloudFront account and work with JW player. Then you have access to RTMP software to stream video and audio. This service is very cheap. if you want to know more about this, check out these tutorials:
http://www.miracletutorials.com/category/s3-amazon-cloudfront/ Start at the bottom and work your way up to the tutorials higher up.
I hope this will help you get yourself on your way.
If you don't need sound, you can send JPEGs with header like this:
Content-Type: multipart/x-mixed-replace
This is a simple demo with nodejs, it uses library opencv4nodejs to generate images. You can use any other HTTP server which allows to append data to the socket while keeping connection opened. Tested on Chrome and FF on Ubuntu Linux.
To run the sample you will need to install this library with npm install opencv4nodejs, it might take while, then start the server like this: node app.js, here is app.js itself
var http = require('http')
const cv = require('opencv4nodejs');
var m=new cv.Mat(300, 300, cv.CV_8UC3);
var cnt = 0;
const blue = new cv.Vec3(255, 220, 120);
const yellow = new cv.Vec3(255, 220, 0);
var lastTs = Date.now();
http.createServer((req, res) => {
if (req.url=='/'){
res.end("<!DOCTYPE html><style>iframe {transform: scale(.67)}</style><html>This is a streaming video:<br>" +
"<img src='/frame'></img></html>")
} else if (req.url=='/frame') {
res.writeHead(200, { 'Content-Type': 'multipart/x-mixed-replace;boundary=myboundary' });
var x =0;
var fps=0,fcnt=0;
var next = function () {
var ts = Date.now();
var m1=m.copy();
fcnt++;
if (ts-lastTs > 1000){
lastTs = ts;
fps = fcnt;
fcnt=0;
}
m1.putText(`frame ${cnt} FPS=${fps}`, new cv.Point2(20,30),1,1,blue);
m1.drawCircle(new cv.Point2(x,50),10,yellow,-1);
x+=1;
if (x>m.cols) x=0;
cnt++;
var buf = cv.imencode(".jpg",m1);
res.write("--myboundary\r\nContent-type:image/jpeg\r\nDaemonId:0x00258009\r\n\r\n");
res.write(buf,function () {
next();
});
};
next();
}
}).listen(80);
A bit later I've found this example with some more details in python: https://blog.miguelgrinberg.com/post/video-streaming-with-flask
UPDATE: it also works, if you stream this into html img tag.
True cross-browser streaming is only possible through "rich media" clients like Flash, which is why almost all video websites default to serving video using Adobe's proprietary .flv format.
For non-live video the advent of video embeds in HTML5 shows promise, and using Canvas and JavaSCript streaming should be technically possible, but handling streams and preloading binary video objects would have to be done in JavaScript and would not be very straightforward.

Categories