I have a video element, with data being added via MSE. I'm trying to determine how many audio channels there are in each track.
The AudioTrack objects themselves don't have a property with this information. The only way I know to go about it is to use the Web Audio API:
const v = document.querySelector('video');
const ctx = new OfflineAudioContext(32, 48000, 48000);
console.log(Array.from(v.audioTracks).map((track) => {
return ctx.createBufferSource(track.sourceBuffer).channelCount;
}));
For a video with a single mono track, I expect to get [1]. For a video with a single stereo track, I expect to get [2]. Yet, every time I get [2] no matter what the channel count is in the original source.
Questions:
Is there a proper direct way to get the number of channels in an AudioTrack?
Is there something else I could be doing with the Web Audio API to get the correct number of channels?
I stumbled upon an answer for this that seems to be working. It looks like by using decodeAudioData we can grab some buffer data about a file. I built a little function that returns a Promise with the buffer data that should return the correct number of channels of an audio file:
function loadBuffer(path) {
return fetch(path)
.then(response => response.arrayBuffer())
.then(
buffer =>
new Promise((resolve, reject) =>
audioContext.decodeAudioData(
buffer,
data => resolve(data),
err => reject(err)
)
)
)
}
Then you can use it like this:
loadBuffer(audioSource).then(data => console.log(data.numberOfChannels))
Might be best to store and reuse the data if it can be called multiple times.
Related
I want to make a recording where, I get multiple audio tracks from different mediaStream objects (some of them, remote). Use the getAudioTracks () method and add them to a mediaStream object using addTrack (). At the moment of passing this last object as a parameter for mediaRecorder I realize that it only records the audio track located in position [0]. That gives me to understand that mediaRecorder is capable of recording a track by type, is there any way to join these tracks into one to record them correctly using mediaRecorder? I would be grateful for any page that explains this if possible and if it exists
I was battling with this for a while and took me ages to realise that the MediaStream only ever recorded the first track I added. My solution was to get the Web Audio API involved. This example uses two UserMedia (e.g. a mic & Stereo Mix) and merges them. The UserMedia are identified by their deviceId as shown when you use await navigator.mediaDevices.enumerateDevices().
In summary:
Create an AudioContext()
Get your media using navigator.mediaDevices.getUserMedia()
Add these as a stream source to the AudioContext
Create an AudioContext stream destination object
Connect your sources to this single destination
And your new MediaRecorder() takes this destination as its MediaStream
Now you can record yourself singing along to your favourite song as it streams ;)
const audioContext = new AudioContext();
audioParams_01 = {
deviceId: "default",
}
audioParams_02 = {
deviceId: "7079081697e1bb3596fad96a1410ef3de71d8ccffa419f4a5f75534c73dd16b5",
}
mediaStream_01 = await navigator.mediaDevices.getUserMedia({ audio: audioParams_01 });
mediaStream_02 = await navigator.mediaDevices.getUserMedia({ audio: audioParams_02 });
audioIn_01 = audioContext.createMediaStreamSource(mediaStream_01);
audioIn_02 = audioContext.createMediaStreamSource(mediaStream_02);
dest = audioContext.createMediaStreamDestination();
audioIn_01.connect(dest);
audioIn_02.connect(dest);
const recorder = new MediaRecorder(dest.stream);
chunks = [];
recorder.onstart = async (event) => {
// your code here
}
recorder.ondataavailable = (event) => {
chunks.push(event.data);
}
recorder.onstop = async (event) => {
// your code here
}
Finish using a library built by muazKhan, which allows you to merge the streams and return them in one!
It's incredibly easy!
https://github.com/muaz-khan/MultiStreamsMixer
I'm building a data visualization which relies on a lot of small raster images, delivered as AWS URLs via JSON API.
This works fairly well, until I try to implement my next step, which is rendering the data visualization as a PNG to download. In the PNG, the raster images are broken.
I've understood that to solve this, I need to embed images as Data URLs.
Here's what I've got so far:
const companies_base64 = companies.map(c => {
var o = Object.assign({}, c)
o.base64 = imageToBase64(c.mimetype, c.logo)
return o
})
Where companies is an array of objects. Here's imageToBase64, the Heroku app being a clone of CORS anywhere:
function imageToBase64(mimetype, logo) {
var url = 'https://example.herokuapp.com/' + logo
return d3.blob(url)
.then(blob => blobToBase64(blob))
.then(base64 => mimetype + base64)
.catch(error => console.error(error))
}
function blobToBase64(blob) {
return new Promise((resolve, reject) => {
let reader = new FileReader()
reader.onload = () => {
let dataUrl = reader.result
let base64 = dataUrl.split(',')[1]
resolve(base64)
}
reader.onerror = () => {
reject("Error")
}
reader.readAsDataURL(blob)
})
}
Which results in a Promise being returned when calling base64 on any of the objects in companies_base64, the [[PromiseValue]] being of course what I'm after. How am I supposed to make sure it is what gets returned so I can, ultimately, place it inside the xlink:href attributes of the <image>s in my <svg>?
I think that once it works and I can call imageToBase64 wherever, it's something I want to do only when the user presses Download. I imagine I can do this using D3, iterating over the <image>s and swapping out their xlink:href. Or should I go about it another way?
I have also tried getting the images as objects and then converting them to base64 in my RoR backend so they come packaged with the JSON, via an Image#to_base64 method. This does work, but it A) feels very wrong and B) is obviously very slow on initial load.
Thank you for your time and please bear with me as I am a beginner.
Your imageToBase64 function returns a promise, not the resolved data URL. That means you have to wait before you can attach them to the companies_base64 members. It is your choice if you do that as soon as the individual base64 string is ready, or if you wait for them all:
Promise.all(companies.map(c => {
return imageToBase64(c.mimetype, c.logo)
.then(u => Object.assign({ base64: u }, c))
.then(/* change the image reference here one by one... */)
}))
.then(companies_base64 => /* ...or here, in a loop over the array */)
.catch(error => console.error(error))
I tried to write a program with highland.js to download several files, unzip them and parse into objects, then merge object streams into one stream by flatMap and print out.
function download(url) {
return _(request(url))
.through(zlib.createGunzip())
.errors((err) => console.log('Error in gunzip', err))
.through(toObjParser)
.errors((err) => console.log('Error in OsmToObj', err));
}
const urlList = ['url_1', 'url_2', 'url_3'];
_(urlList)
.flatMap(download)
.each(console.log);
When all URLs are valid, it works fine. If a URL is invalid there is no file downloaded, then gunzip reports error. I suspect that the stream closes when error occurs. I expect that flatMap will continue with other streams, however the program doesn't download other files and there is nothing printed out.
What's the correct way to handle error in stream and how to make flatMap not stop after one stream has error?
In imperative programming, I can add debug logs to trace where error happens. How to debug streaming code?
PS. toObjParser is a Node Transform Stream. It takes a readable stream of OSM XML and outputs a stream of objects compatible with Overpass OSM JSON. See https://www.npmjs.com/package/osm2obj
2017-12-19 update:
I tried to call push in errors as #amsross suggested. To verify if push really works, I pushed a XML document and it was parsed by following parser and I saw it from output. However, stream still stopped and url_3 was not downloaded.
function download(url) {
console.log('download', url);
return _(request(url))
.through(zlib.createGunzip())
.errors((err, push) => {
console.log('Error in gunzip', err);
push(null, Buffer.from(`<?xml version='1.0' encoding='UTF-8'?>
<osmChange version="0.6">
<delete>
<node id="1" version="2" timestamp="2008-10-15T10:06:55Z" uid="5553" user="foo" changeset="1" lat="30.2719406" lon="120.1663723"/>
</delete>
</osmChange>`));
})
.through(new OsmToObj())
.errors((err) => console.log('Error in OsmToObj', err));
}
const urlList = ['url_1_correct', 'url_2_wrong', 'url_3_correct'];
_(urlList)
.flatMap(download)
.each(console.log);
Update 12/19/2017:
Ok, so I can't give you a good why on this, but I can tell you that switching from consuming the streams resulting from download in sequence to merge'ing them together will probably give you the result you're after. Unfortunately (or not?), you will no longer be getting the results back in any prescribed order.
const request = require('request')
const zlib = require('zlib')
const h = require('highland')
// just so you can see there isn't some sort of race
const rnd = (min, max) => Math.floor((Math.random() * (max - min))) + min
const delay = ms => x => h(push => setTimeout(() => {
push(null, x)
push(null, h.nil)
}, ms))
const download = url => h(request(url))
.flatMap(delay(rnd(0, 2000)))
.through(zlib.createGunzip())
h(['urlh1hcorrect', 'urlh2hwrong', 'urlh3hcorrect'])
.map(download).merge()
// vs .flatMap(download) or .map(download).sequence()
.errors(err => h.log(err))
.each(h.log)
Update 12/03/2017:
When an error is encountered on the stream, it ends that stream. To avoid this, you need to handle the error. You are currently using errors to report the error, but not handle it. You can do something like this to move on to the next value in the stream:
.errors((err, push) => {
console.log(err)
push(null) // push no error forward
})
Original:
It's difficult to answer without knowing the input and output types of toObjParser are.
Because through passes a stream of values to the provided function and expects a stream of values in return, your issue may reside in toObjParser having a signature like Stream -> Object, or Stream -> Stream Object, where the errors are occurring on the inner stream, which will not emit any errors until it is consumed.
What is the output of .each(console.log)? If it is logging a stream, that is most likely your problem.
While trying to determine a solution to How to use Web Speech API at chromium? found that
var voices = window.speechSynthesis.getVoices();
returns an empty array for voices identifier.
Not certain if lack of support at chromium browser is related to this issue Not OK, Google: Chromium voice extension pulled after spying concerns?
Questions:
1) Are there any workarounds which can implement the requirement of creating or converting audio from text at chromium browser?
2) How can we, the developer community, create an open source database of audio files reflecting both common and uncommon words; served with appropriate CORS headers?
There are several possible workarounds that have found which provide the ability to create audio from text; two of which require requesting an external resource, the other uses meSpeak.js by #masswerk.
Using approach described at Download the Audio Pronunciation of Words from Google, which suffers from not being able to pre-determine which words actually exist as a file at the resource without writing a shell script or performing a HEAD request to check if a network error occurs. For example, the word "do" is not available at the resource used below.
window.addEventListener("load", () => {
const textarea = document.querySelector("textarea");
const audio = document.createElement("audio");
const mimecodec = "audio/webm; codecs=opus";
audio.controls = "controls";
document.body.appendChild(audio);
audio.addEventListener("canplay", e => {
audio.play();
});
let words = textarea.value.trim().match(/\w+/g);
const url = "https://ssl.gstatic.com/dictionary/static/sounds/de/0/";
const mediatype = ".mp3";
Promise.all(
words.map(word =>
fetch(`https://query.yahooapis.com/v1/public/yql?q=select * from data.uri where url="${url}${word}${mediatype}"&format=json&callback=`)
.then(response => response.json())
.then(({query: {results: {url}}}) =>
fetch(url).then(response => response.blob())
.then(blob => blob)
)
)
)
.then(blobs => {
// const a = document.createElement("a");
audio.src = URL.createObjectURL(new Blob(blobs, {
type: mimecodec
}));
// a.download = words.join("-") + ".webm";
// a.click()
})
.catch(err => console.log(err));
});
<textarea>what it does my ninja?</textarea>
Resources at Wikimedia Commons Category:Public domain are not necessary served from same directory, see How to retrieve Wiktionary word content?, wikionary API - meaning of words.
If the precise location of the resource is known, the audio can be requested, though the URL may include prefixes other than the word itself.
fetch("https://upload.wikimedia.org/wikipedia/commons/c/c5/En-uk-hello-1.ogg")
.then(response => response.blob())
.then(blob => new Audio(URL.createObjectURL(blob)).play());
Not entirely sure how to use the Wikipedia API, How to get Wikipedia content using Wikipedia's API?, Is there a clean wikipedia API just for retrieve content summary? to get only the audio file. The JSON response would need to be parsed for text ending in .ogg, then a second request would need to be made for the resource itself.
fetch("https://en.wiktionary.org/w/api.php?action=parse&format=json&prop=text&callback=?&page=hello")
.then(response => response.text())
.then(data => {
new Audio(location.protocol + data.match(/\/\/upload\.wikimedia\.org\/wikipedia\/commons\/[\d-/]+[\w-]+\.ogg/).pop()).play()
})
// "//upload.wikimedia.org/wikipedia/commons/5/52/En-us-hello.ogg\"
which logs
Fetch API cannot load https://en.wiktionary.org/w/api.php?action=parse&format=json&prop=text&callback=?&page=hello. No 'Access-Control-Allow-Origin' header is present on the requested resource
when not requested from same origin. We would need to try to use YQL again, though not certain how to formulate the query to avoid errors.
The third approach uses a slightly modified version of meSpeak.js to generate the audio without making an external request. The modification was to create a proper callback for .loadConfig() method
fetch("https://gist.githubusercontent.com/guest271314/f48ee0658bc9b948766c67126ba9104c/raw/958dd72d317a6087df6b7297d4fee91173e0844d/mespeak.js")
.then(response => response.text())
.then(text => {
const script = document.createElement("script");
script.textContent = text;
document.body.appendChild(script);
return Promise.all([
new Promise(resolve => {
meSpeak.loadConfig("https://gist.githubusercontent.com/guest271314/8421b50dfa0e5e7e5012da132567776a/raw/501fece4fd1fbb4e73f3f0dc133b64be86dae068/mespeak_config.json", resolve)
}),
new Promise(resolve => {
meSpeak.loadVoice("https://gist.githubusercontent.com/guest271314/fa0650d0e0159ac96b21beaf60766bcc/raw/82414d646a7a7ef11bb04ddffe4091f78ef121d3/en.json", resolve)
})
])
})
.then(() => {
// takes approximately 14 seconds to get here
console.log(meSpeak.isConfigLoaded());
meSpeak.speak("what it do my ninja", {
amplitude: 100,
pitch: 5,
speed: 150,
wordgap: 1,
variant: "m7"
});
})
.catch(err => console.log(err));
one caveat of the above approach being that it takes approximately 14 and a half seconds for the three files to load before the audio is played back. However, avoids external requests.
It would be a positive to either or both 1) create a FOSS, developer maintained database or directory of sounds for both common and uncommon words; 2) perform further development of meSpeak.js to reduce load time of the three necessary files; and use Promise based approaches to provide notifications of the progress of of the loading of the files and readiness of the application.
In this users' estimation, it would be a useful resource if developers themselves created and contributed to an online database of files which responded with an audio file of the specific word. Not entirely sure if github is the appropriate venue to host audio files? Will have to consider the possible options if interest in such a project is shown.
I have a webRTC connection established with audio and video.
On the caller side, I'd like to change the audio input.
e.g. the User changes the audioinput from a dropdown list.
What's the workflow to substitute the audio track of an existing stream?
Can I add another audio track and make one active over the other? how?
Looks like I may need to call getUserMedia again passing constraints (?), which to my understanding comes to create a New mediaStream instances and not modify the existing.
For us it looks something like this:
const replaceTrack = async (peerConnection, oldSender, track, stream) => {
peerConnection.removeTrack(oldSender);
const newSender = peerConnection.addTrack(track, stream);
const localSdp = await peerConnection.createOffer({ offerToReceiveAudio: 1 });
await peerConnection.setLocalDescription(reply);
const response = await sendOffer(peerConnection.localDescription);
const description = new RTCSessionDescription(response);
peerConnection.setRemoteDescription(description);
return newSender;
}
There is now a much simpler API for this operation: RTCRtpSender.replaceTrack().
It could look something like this:
const currentSenders = peerConnection.getSenders();
const currentAudioSender = currentSenders.find((s) => s.track.kind === 'audio');
currentAudioSender.replaceTrack(newAudioTrack);