audio file doesn't work on ios iphone neither safari nor chrome - javascript

I'm creating a chat with the ability to send voice notes.
and voice notes work perfectly on desktop and android but on ios things start to crash
once the audio files load, the chrome console on ios shows an error
mediaError {code:4, message:Unsupported source type, MEDIA_ERR_ABORTED:1, MEDIA_ERR_NETWORK:2, MEDIA_ERR_DECODE:3}
and if I click on the play button it gives the error DOMException
This is the function that records audio
const recordAudio = async (_) => {
const stream = await navigator.mediaDevices.getUserMedia({ audio: true })
const mediaRecorder = new MediaRecorder(stream, {
mimeType:'audio/mp4',
audioBitrate: '128000',
})
mediaRecorder.start()
const audioChunks = []
mediaRecorder.addEventListener('dataavailable', (event) => {
audioChunks.push(event.data)
})
mediaRecorder.addEventListener('stop', () => {
const audioBlob = new Blob(audioChunks, { type: 'audio/mp3' })
composeMessage('audio', audioBlob)
setIsRecording(false)
})
setTimeout(() => {
mediaRecorder.stop()
}, 30000)
}
The function that creates audio file
const createAudioFile = () => {
const audio = new Audio()
audio.setAttribute('preload', 'metadata')
const source = document.createElement('source')
source.setAttribute('src', URL)
source.setAttribute('type', 'audio/mp3')
audio.appendChild(source)
setAudioFile(audio)
}
and this is the function that triggers the audio file
const playAudioHandler = () => {
const playPromise = audioFile.play()
if (playPromise !== undefined) {
playPromise
.then((_) => {
audioFile.play()
setIsPlaying(true)
})
.catch((error) => {
pauseAudioHandler()
})
}
}

Related

The video stops and does not resume after track.stop()

I'm creating simple video-chat one-to-one using PeerJS and React. Everything is working fine except the camera muting. When participant A muting camera, it's muting on both clients but when participant A unmutes it, participant B can see only the static image instead of the opponent video.
I have a global state webcamState which changed when the corresponding button was clicked:
const videoRef = useRef<any>(null);
const webcamState = useSelector(getWebcamState);
const [stream, setStream] = useState<MediaStream | null>(null);
const [isWebcamLoading, setIsWebcamLoading] = useState(true);
const loadUserMedia = () => {
setIsWebcamLoading(true);
getUserMedia(
{ video: webcamState, audio: micState },
(newStream: MediaStream) => {
videoRef.current.srcObject = newStream;
setStream(newStream);
setIsWebcamLoading(false);
},
(err: any) => {
setIsWebcamLoading(false);
},
);
};
useEffect(() => {
if (videoRef?.current?.srcObject) {
videoRef.current.srcObject.getVideoTracks().forEach((track: any) => (track.enabled = webcamState));
if (!webcamState) {
videoRef.current.srcObject.getVideoTracks().forEach((track: any) => track.stop());
videoRef.current.pause();
} else {
loadUserMedia();
}
}
}, [webcamState]);
Then this stream exporting from this hook and passed into another to initialize Peer call (and peer answer as well):
export interface IPeerOptions {
opponentVideoRef: React.MutableRefObject<null>;
currentVideoRef: React.MutableRefObject<null>;
currentStream: MediaStream;
userId: string;
}
export const initializePeerCall = (options: IPeerOptions): Peer => {
const call = options.peer.call(options.opponentId, options.currentStream);
call.on('stream', stream => {
options.opponentVideoRef = setVideo(options.opponentVideoRef, stream);
});
call.on('error', err => {
console.log(err);
});
call.on('close', () => {
console.log('Connection closed');
});
return options.peer;
};
No errors appear in the console
But if I will remove the following line: videoRef.current.srcObject.getVideoTracks().forEach((track: any) => track.stop()); everything will work fine as expected
Maybe anyone faced something similar?
UPD: I tried this but the result was the same:
useEffect(() => {
if (videoRef?.current?.srcObject) {
videoRef.current.srcObject.getVideoTracks().forEach((track: any) => (track.enabled = webcamState));
if (!webcamState) {
videoRef.current.srcObject.getVideoTracks().forEach((track: any) => track.stop());
loadUserMedia();
} else {
loadUserMedia();
}
}
}, [webcamState]);
Notification:
const onOpponentVideoStatusChanged = (newStatus: boolean) => {
setOpponentState(prev => {
return { microphoneState: !!prev?.microphoneState, webcamState: newStatus };
});
options.opponentVideoRef.current.srcObject.getVideoTracks().forEach((track: any) => (track.enabled = newStatus));
};
As I understand after long investigation, user B still getting the same stream after user A created a new one. How can I fix it?
If you turn track off using track.stop(), you can not resume it.
I did get new stream when I have to resume it.
React.useEffect(()=>{
if(mediaStream) { // mediaStream could be state locally or globally.
const videoTrack = mediaStream.getVideoTracks();
videoTrack.forEach((t) => {
t.enabled = false;
t.stop();
});
}
// get MediaStream again with contraints
}, [isAudioOnly]);

recorded Audio file is not working in ios

I am developing a web admin application and backend for IOS & android mobile apps.
I am recording audio from the web using navigation.mediaDevices api.
When I record audio from chrome and firefox and upload to the server then this recorded audio file is playing in android but not playing in ios app.
and when I record audio from safari then audio file played in ios & android
Note - I am statically serving audio file from server
below I have written code to record audio from web using browser navigation.mediaDevices api
import * as toast from "../Toast";
export default function recordAudio() {
return new Promise((resolve) => {
navigator.mediaDevices
.getUserMedia({ audio: true })
.then((stream) => {
const mediaRecorder = new MediaRecorder(stream, {
audioBitsPerSecond: 32000,
// bitsPerSecond: 32000
// mimeType: "audio/webm;codecs=opus",
});
const audioChunks = [];
mediaRecorder.addEventListener("dataavailable", (event) => {
audioChunks.push(event.data);
});
function start() {
mediaRecorder.start();
}
function pause() {
mediaRecorder.pause();
}
function resume() {
mediaRecorder.resume();
}
function stop() {
return new Promise((resolve) => {
mediaRecorder.addEventListener("stop", () => {
const audioName = "audio.mp3";
const blobData = new Blob(audioChunks, {
type: "audio/mpeg",
});
const audioFile = new File([blobData], audioName, {
type: "audio/mpeg",
});
const audioUrl = URL.createObjectURL(audioFile);
const audio = new Audio(audioUrl);
function play() {
audio.play();
}
stream.getTracks()[0].stop();
resolve({ audioFile, audioUrl, play, audioName });
});
mediaRecorder.stop();
});
}
resolve({ start, stop, pause, resume });
})
.catch((error) => {
toast.info(
"To record Voice Messages, application needs access to your computer’s microphone. Click in the URL bar and set “Use the Microphone” to “Allow.” "
);
});
});
}
Please feel free to send in your suggestions and solutions.
Thank you

How do you initialize a WebRTC call using the transceiver API but only enable audio&video later after signaling completed?

I am trying to first connect two WebRTC peers. Once the connection is established I want to give the users on both sides the option to enable/disable video and audio. This should happen without triggering the signaling process again.
I do run into an issue though: If I call replaceTrack(audioTack) the remote peer will not playback audio until I also call replaceTrack(video).
I am unsure why this happen and can not find any clue in the documentation. It does play fine after 10 seconds once I also attach the video track. Without video track there is no audio playback. Why?
function createVideoElement() {
const vid = document.createElement("video")
vid.width = 320;
vid.controls = true;
vid.autoplay = true;
const root = document.body;
document.body.appendChild(vid);
return vid;
}
async function RunTestInit() {
console.log("get media access");
const p1_stream_out = await navigator.mediaDevices.getUserMedia({
video: true,
audio: true
});
const p2_stream_out = await navigator.mediaDevices.getUserMedia({
video: true,
audio: true
});
console.log("stream setup");
const p1_stream_in = new MediaStream();
const p2_stream_in = new MediaStream();
const p1_video_in = createVideoElement();
const p2_video_in = createVideoElement();
console.log("peer setup");
const p1 = new RTCPeerConnection();
const p2 = new RTCPeerConnection();
const p1_tca = p1.addTransceiver("audio", {
direction: "sendrecv"
});
const p1_tcv = p1.addTransceiver("video", {
direction: "sendrecv"
});
p1.onicecandidate = (ev) => {
p2.addIceCandidate(ev.candidate);
}
p2.onicecandidate = (ev) => {
p1.addIceCandidate(ev.candidate);
}
p1.onconnectionstatechange = (ev) => {
console.log("p1 state: ", p1.connectionState);
}
p2.onconnectionstatechange = async (ev) => {
console.log("p2 state: ", p2.connectionState);
}
p1.onnegotiationneeded = () => {
//triggers once
console.warn("p1.onnegotiationneeded");
}
p2.onnegotiationneeded = () => {
//should never trigger
console.warn("p2.onnegotiationneeded");
}
p1.ontrack = (ev) => {
console.log("p1.ontrack", ev);
p1_stream_in.addTrack(ev.track);
p1_video_in.srcObject = p1_stream_in;
}
p2.ontrack = (ev) => {
console.log("p2.ontrack", ev);
p2_stream_in.addTrack(ev.track);
p2_video_in.srcObject = p2_stream_in;
}
console.log("signaling");
const offer = await p1.createOffer();
await p1.setLocalDescription(offer);
await p2.setRemoteDescription(offer);
const p2_tca = p2.getTransceivers()[0];
const p2_tcv = p2.getTransceivers()[1];
p2_tca.direction = "sendrecv"
p2_tcv.direction = "sendrecv"
const answer = await p2.createAnswer();
await p2.setLocalDescription(answer);
await p1.setRemoteDescription(answer);
console.log("signaling done");
//send audio from p2 to p1 (direction doesn't matter)
//after this runs nothing will happen and no audio plays
setTimeout(async () => {
await p2_tca.sender.replaceTrack(p2_stream_out.getAudioTracks()[0]);
console.warn("audio playback should start now but nothing happens");
}, 1000);
//audio starts playing once this runs
setTimeout(async () => {
//uncomment this and it works just fine
await p2_tcv.sender.replaceTrack(p2_stream_out.getVideoTracks()[0]);
console.warn("now audio playback starts");
}, 10000);
}
function start() {
setTimeout(async () => {
console.log("Init test case");
await RunTestInit();
}, 1);
}
Same example in the js fiddle (needs camera and microphone access):
https://jsfiddle.net/vnztcx5p/5/
Once audio works this will cause an echo.
that is a known issue. https://bugs.chromium.org/p/chromium/issues/detail?id=813243 and https://bugs.chromium.org/p/chromium/issues/detail?id=403710 have some background information.
In a nutshell the video element expect you to send audio and video data and these need to be synchronized. But you don't send any video data and the element needs to fire a loadedmetadata and resize event because that is what the specification says. Hence it will block audio indefinitely
You can enable/disable audio and video tracks, so you dont have to renegotiate. Note that this tracks have to be added before negotiation starts. You can achieve it with:
mediaStream.getAudioTracks()[0].enabled = false; // or true to enable it.
Or if you want to disable video:
mediaStream.getVideoTracks()[0].enabled = false; // or true to enable it.
Here is the documentation
getAudioTracks()
getVideoTracks()
I got this working. It looks like more a problem with how HTMLVideoElement works rather than WebRTC.
If I set
p1_video_in.srcObject = p1_stream_in;
p2_video_in.srcObject = p2_stream_in;
before I add the tracks to the stream it works.
Complete example looks like this:
function createVideoElement() {
const vid = document.createElement("video")
vid.width = 320;
vid.controls = true;
vid.autoplay = true;
const root = document.body;
document.body.appendChild(vid);
return vid;
}
async function RunTestInit() {
console.log("get media access");
const p1_stream_out = await navigator.mediaDevices.getUserMedia({
video: true,
audio: true
});
const p2_stream_out = await navigator.mediaDevices.getUserMedia({
video: true,
audio: true
});
console.log("stream setup");
const p1_stream_in = new MediaStream();
const p2_stream_in = new MediaStream();
const p1_video_in = createVideoElement();
const p2_video_in = createVideoElement();
p1_video_in.srcObject = p1_stream_in;
p2_video_in.srcObject = p2_stream_in;
console.log("peer setup");
const p1 = new RTCPeerConnection();
const p2 = new RTCPeerConnection();
const p1_tca = p1.addTransceiver("audio", {
direction: "sendrecv"
});
const p1_tcv = p1.addTransceiver("video", {
direction: "sendrecv"
});
p1.onicecandidate = (ev) => {
p2.addIceCandidate(ev.candidate);
}
p2.onicecandidate = (ev) => {
p1.addIceCandidate(ev.candidate);
}
p1.onconnectionstatechange = (ev) => {
console.log("p1 state: ", p1.connectionState);
}
p2.onconnectionstatechange = async (ev) => {
console.log("p2 state: ", p2.connectionState);
}
p1.onnegotiationneeded = () => {
//triggers once
console.warn("p1.onnegotiationneeded");
}
p2.onnegotiationneeded = () => {
//should never trigger
console.warn("p2.onnegotiationneeded");
}
p1.ontrack = (ev) => {
console.log("p1.ontrack", ev);
p1_stream_in.addTrack(ev.track);
}
p2.ontrack = (ev) => {
console.log("p2.ontrack", ev);
p2_stream_in.addTrack(ev.track);
}
console.log("signaling");
const offer = await p1.createOffer();
await p1.setLocalDescription(offer);
await p2.setRemoteDescription(offer);
const p2_tca = p2.getTransceivers()[0];
const p2_tcv = p2.getTransceivers()[1];
p2_tca.direction = "sendrecv"
p2_tcv.direction = "sendrecv"
const answer = await p2.createAnswer();
await p2.setLocalDescription(answer);
await p1.setRemoteDescription(answer);
console.log("signaling done");
//send audio from p2 to p1 (direction doesn't matter)
//after this runs nothing will happen and no audio plays
setTimeout(async () => {
await p2_tca.sender.replaceTrack(p2_stream_out.getAudioTracks()[0]);
console.warn("audio playback should start now but nothing happens");
}, 1000);
//audio starts playing once this runs
setTimeout(async () => {
//uncomment this and it works just fine
await p2_tcv.sender.replaceTrack(p2_stream_out.getVideoTracks()[0]);
console.warn("now audio playback starts");
}, 10000);
}
function start() {
setTimeout(async () => {
console.log("Init test case");
await RunTestInit();
}, 1);
}

Duration of recordered file WebRTC

I implement a video streaming service with WebRTC an signaling WebSockets-Server. It is a electron-App. I want to record some video fragments and get a webm file to download. It works fine if i record a local stream (from navigator.mediaDevices.getUserMedia), but if I try to record a remote stream, it will be a video file generated with duration < 1 sec, this trouble is only in chrome (and electron) in Firefox works it fine. If I call console.log(blob) in stopRecording () it gives somethink like this:
Blob(1875185) {size: 1875185, type: "video/webm"}
So blob size is correct and after downloading is file size is also correct, what in this case?
class WebRTC {
...........
constructor () {
navigator.mediaDevices.getUserMedia({
'audio': true,
'video': false
}).then(
(stream) => {
this.peerConnection = new RTCPeerConnection(peerConnectionConfig)
this.peerConnection.addEventListener('addstream', event => this.gotRemoteMediaStream(event))
this.peerConnection.addEventListener('icecandidate', e => this.handleConnection(e))
this.peerConnection.addStream(stream)
}
).catch(e => console.log('error', e))
}
gotRemoteMediaStream (event) {
console.log(event)
const mediaStream = event.stream
this.video.srcObject = mediaStream
this.video.play()
this.remoteStream = mediaStream
console.log('Remote peer connection received remote stream.')
}
startRecording () {
console.log('RECORD')
this.recorder = new MediaRecorder(this.remoteStream)
this.recorder.addEventListener('dataavailable', e => this.onRecordingReady(e))
this.recorder.start(15)
}
onRecordingReady (e) {
console.log(e)
this.chunks.push(e.data)
}
stopRecording () {
this.recorder.stop()
var a = document.createElement('a')
document.body.appendChild(a)
a.style = 'display: none'
a.download = 'test.webm'
setTimeout(() => {
console.log(this)
var blob = new Blob(this.chunks, { type: 'video/webm' })
console.log(this.chunks)
console.log(blob)
let url = URL.createObjectURL(blob)
a.href = url
a.click()
window.URL.revokeObjectURL(url)
}, 500)
}
}

How to capture generated audio from window.speechSynthesis.speak() call?

Previous questions have presented this same or similar inquiry
Can Web Speech API used in conjunction with Web Audio API?
How to access audio result from Speech Synthesis API?
Record HTML5 SpeechSynthesisUtterance generated speech to file
generate audio file with W3C Web Speech API
yet no workarounds appear to be have been created using window.speechSynthesis(). Though there are workarounds using epeak , meSpeak How to create or convert text to audio at chromium browser? or making requests to external servers.
How to capture and record audio output of window.speechSynthesis.speak() call and return result as a Blob, ArrayBuffer, AudioBuffer or other object type?
The Web Speech API Specification does not presently provide a means or hint on how to achieve returning or capturing and recording audio output of window.speechSynthesis.speak() call.
See also
MediaStream, ArrayBuffer, Blob audio result from speak() for recording?
Re: MediaStream, ArrayBuffer, Blob audio result from speak() for recording?
Re: MediaStream, ArrayBuffer, Blob audio result from speak() for recording?. In pertinent part, use cases include, but are not limited to
Persons who have issues speaking; i.e.g., persons whom have suffered a
stroke or other communication inhibiting afflictions. They could convert
text to an audio file and send the file to another individual or group.
This feature would go towards helping them communicate with other persons,
similar to the technologies which assist Stephen Hawking communicate;
Presently, the only person who can hear the audio output is the person
in front of the browser; in essence, not utilizing the full potential of
the text to speech functionality. The audio result can be used as an
attachment within an email; media stream; chat system; or other
communication application. That is, control over the generated audio output;
Another application would be to provide a free, libre, open source audio
dictionary and translation service - client to client and client to server,
server to client.
It is possible to capture the output of audio output of window.speechSynthesis.speak() call utilizing navigator.mediaDevices.getUserMedia() and MediaRecorder(). The expected result is returned at Chromium browser. Implementation at Firefox has issues. Select Monitor of Built-in Audio Analog Stereo at navigator.mediaDevices.getUserMedia() prompt.
The workaround is cumbersome. We should be able to get generated audio, at least as a Blob, without navigator.mediaDevices.getUserMedia() and MediaRecorder().
More interest is evidently necessary by users of browsers, JavaScript and C++ developers, browser implementers and specification authors for further input; to create a proper specification for the feature, and consistent implementation at browsers' source code; see How to implement option to return Blob, ArrayBuffer, or AudioBuffer from window.speechSynthesis.speak() call.
At Chromium a speech dispatcher program should be installed and the instance launched with --enable-speech-dispatcher flag set, as window.speechSynthesis.getVoices() returns an empty array, see How to use Web Speech API at chromium?.
Proof of concept
// SpeechSynthesisRecorder.js guest271314 6-17-2017
// Motivation: Get audio output from `window.speechSynthesis.speak()` call
// as `ArrayBuffer`, `AudioBuffer`, `Blob`, `MediaSource`, `MediaStream`, `ReadableStream`, or other object or data types
// See https://lists.w3.org/Archives/Public/public-speech-api/2017Jun/0000.html
// https://github.com/guest271314/SpeechSynthesisRecorder
// Configuration: Analog Stereo Duplex
// Input Devices: Monitor of Built-in Audio Analog Stereo, Built-in Audio Analog Stereo
class SpeechSynthesisRecorder {
constructor({text = "", utteranceOptions = {}, recorderOptions = {}, dataType = ""}) {
if (text === "") throw new Error("no words to synthesize");
this.dataType = dataType;
this.text = text;
this.mimeType = MediaRecorder.isTypeSupported("audio/webm; codecs=opus")
? "audio/webm; codecs=opus" : "audio/ogg; codecs=opus";
this.utterance = new SpeechSynthesisUtterance(this.text);
this.speechSynthesis = window.speechSynthesis;
this.mediaStream_ = new MediaStream();
this.mediaSource_ = new MediaSource();
this.mediaRecorder = new MediaRecorder(this.mediaStream_, {
mimeType: this.mimeType,
bitsPerSecond: 256 * 8 * 1024
});
this.audioContext = new AudioContext();
this.audioNode = new Audio();
this.chunks = Array();
if (utteranceOptions) {
if (utteranceOptions.voice) {
this.speechSynthesis.onvoiceschanged = e => {
const voice = this.speechSynthesis.getVoices().find(({
name: _name
}) => _name === utteranceOptions.voice);
this.utterance.voice = voice;
console.log(voice, this.utterance);
}
this.speechSynthesis.getVoices();
}
let {
lang, rate, pitch
} = utteranceOptions;
Object.assign(this.utterance, {
lang, rate, pitch
});
}
this.audioNode.controls = "controls";
document.body.appendChild(this.audioNode);
}
start(text = "") {
if (text) this.text = text;
if (this.text === "") throw new Error("no words to synthesize");
return navigator.mediaDevices.getUserMedia({
audio: true
})
.then(stream => new Promise(resolve => {
const track = stream.getAudioTracks()[0];
this.mediaStream_.addTrack(track);
// return the current `MediaStream`
if (this.dataType && this.dataType === "mediaStream") {
resolve({tts:this, data:this.mediaStream_});
};
this.mediaRecorder.ondataavailable = event => {
if (event.data.size > 0) {
this.chunks.push(event.data);
};
};
this.mediaRecorder.onstop = () => {
track.stop();
this.mediaStream_.getAudioTracks()[0].stop();
this.mediaStream_.removeTrack(track);
console.log(`Completed recording ${this.utterance.text}`, this.chunks);
resolve(this);
}
this.mediaRecorder.start();
this.utterance.onstart = () => {
console.log(`Starting recording SpeechSynthesisUtterance ${this.utterance.text}`);
}
this.utterance.onend = () => {
this.mediaRecorder.stop();
console.log(`Ending recording SpeechSynthesisUtterance ${this.utterance.text}`);
}
this.speechSynthesis.speak(this.utterance);
}));
}
blob() {
if (!this.chunks.length) throw new Error("no data to return");
return Promise.resolve({
tts: this,
data: this.chunks.length === 1 ? this.chunks[0] : new Blob(this.chunks, {
type: this.mimeType
})
});
}
arrayBuffer(blob) {
if (!this.chunks.length) throw new Error("no data to return");
return new Promise(resolve => {
const reader = new FileReader;
reader.onload = e => resolve(({
tts: this,
data: reader.result
}));
reader.readAsArrayBuffer(blob ? new Blob(blob, {
type: blob.type
}) : this.chunks.length === 1 ? this.chunks[0] : new Blob(this.chunks, {
type: this.mimeType
}));
});
}
audioBuffer() {
if (!this.chunks.length) throw new Error("no data to return");
return this.arrayBuffer()
.then(ab => this.audioContext.decodeAudioData(ab))
.then(buffer => ({
tts: this,
data: buffer
}))
}
mediaSource() {
if (!this.chunks.length) throw new Error("no data to return");
return this.arrayBuffer()
.then(({
data: ab
}) => new Promise((resolve, reject) => {
this.mediaSource_.onsourceended = () => resolve({
tts: this,
data: this.mediaSource_
});
this.mediaSource_.onsourceopen = () => {
if (MediaSource.isTypeSupported(this.mimeType)) {
const sourceBuffer = this.mediaSource_.addSourceBuffer(this.mimeType);
sourceBuffer.mode = "sequence"
sourceBuffer.onupdateend = () =>
this.mediaSource_.endOfStream();
sourceBuffer.appendBuffer(ab);
} else {
reject(`${this.mimeType} is not supported`)
}
}
this.audioNode.src = URL.createObjectURL(this.mediaSource_);
}));
}
readableStream({size = 1024, controllerOptions = {}, rsOptions = {}}) {
if (!this.chunks.length) throw new Error("no data to return");
const src = this.chunks.slice(0);
const chunk = size;
return Promise.resolve({
tts: this,
data: new ReadableStream(controllerOptions || {
start(controller) {
console.log(src.length);
controller.enqueue(src.splice(0, chunk))
},
pull(controller) {
if (src.length = 0) controller.close();
controller.enqueue(src.splice(0, chunk));
}
}, rsOptions)
});
}
}
Usage
let ttsRecorder = new SpeechSynthesisRecorder({
text: "The revolution will not be televised",
utternanceOptions: {
voice: "english-us espeak",
lang: "en-US",
pitch: .75,
rate: 1
}
});
// ArrayBuffer
ttsRecorder.start()
// `tts` : `SpeechSynthesisRecorder` instance, `data` : audio as `dataType` or method call result
.then(tts => tts.arrayBuffer())
.then(({tts, data}) => {
// do stuff with `ArrayBuffer`, `AudioBuffer`, `Blob`,
// `MediaSource`, `MediaStream`, `ReadableStream`
// `data` : `ArrayBuffer`
tts.audioNode.src = URL.createObjectURL(new Blob([data], {type:tts.mimeType}));
tts.audioNode.title = tts.utterance.text;
tts.audioNode.onloadedmetadata = () => {
console.log(tts.audioNode.duration);
tts.audioNode.play();
}
})
// AudioBuffer
ttsRecorder.start()
.then(tts => tts.audioBuffer())
.then(({tts, data}) => {
// `data` : `AudioBuffer`
let source = tts.audioContext.createBufferSource();
source.buffer = data;
source.connect(tts.audioContext.destination);
source.start()
})
// Blob
ttsRecorder.start()
.then(tts => tts.blob())
.then(({tts, data}) => {
// `data` : `Blob`
tts.audioNode.src = URL.createObjectURL(blob);
tts.audioNode.title = tts.utterance.text;
tts.audioNode.onloadedmetadata = () => {
console.log(tts.audioNode.duration);
tts.audioNode.play();
}
})
// ReadableStream
ttsRecorder.start()
.then(tts => tts.readableStream())
.then(({tts, data}) => {
// `data` : `ReadableStream`
console.log(tts, data);
data.getReader().read().then(({value, done}) => {
tts.audioNode.src = URL.createObjectURL(value[0]);
tts.audioNode.title = tts.utterance.text;
tts.audioNode.onloadedmetadata = () => {
console.log(tts.audioNode.duration);
tts.audioNode.play();
}
})
})
// MediaSource
ttsRecorder.start()
.then(tts => tts.mediaSource())
.then(({tts, data}) => {
console.log(tts, data);
// `data` : `MediaSource`
tts.audioNode.srcObj = data;
tts.audioNode.title = tts.utterance.text;
tts.audioNode.onloadedmetadata = () => {
console.log(tts.audioNode.duration);
tts.audioNode.play();
}
})
// MediaStream
let ttsRecorder = new SpeechSynthesisRecorder({
text: "The revolution will not be televised",
utternanceOptions: {
voice: "english-us espeak",
lang: "en-US",
pitch: .75,
rate: 1
},
dataType:"mediaStream"
});
ttsRecorder.start()
.then(({tts, data}) => {
// `data` : `MediaStream`
// do stuff with active `MediaStream`
})
.catch(err => console.log(err))
plnkr
This is an updated code from previous answer which works in Chrome 96:
make sure to select "Share system audio" checkbox in "Choose what to share" window
won't run via SO code snippet (save to demo.html)
<script>
(async () => {
const text = "The revolution will not be televised";
const blob = await new Promise(async resolve => {
console.log("picking system audio");
const stream = await navigator.mediaDevices.getDisplayMedia({video:true, audio:true});
const track = stream.getAudioTracks()[0];
if(!track)
throw "System audio not available";
stream.getVideoTracks().forEach(track => track.stop());
const mediaStream = new MediaStream();
mediaStream.addTrack(track);
const chunks = [];
const mediaRecorder = new MediaRecorder(mediaStream, {bitsPerSecond:128000});
mediaRecorder.ondataavailable = event => {
if (event.data.size > 0)
chunks.push(event.data);
}
mediaRecorder.onstop = () => {
stream.getTracks().forEach(track => track.stop());
mediaStream.removeTrack(track);
resolve(new Blob(chunks));
}
mediaRecorder.start();
const utterance = new SpeechSynthesisUtterance(text);
utterance.onend = () => mediaRecorder.stop();
window.speechSynthesis.speak(utterance);
console.log("speaking...");
});
console.log("audio available", blob);
const player = new Audio();
player.src = URL.createObjectURL(blob);
player.autoplay = true;
player.controls = true;
})()
</script>

Categories