I am making an application where I am taking mic data from the inputBuffer and I want to stream to another client and play it. However, I cannot get it wokring.
My recording/capturing works fine so I will skip to relevant parts of the code
function recorderProcess(e) {
var left = e.inputBuffer.getChannelData(0);
var convert = convertFloat32ToInt16(left);
window.stream.write(convert);
var src = window.URL.createObjectURL(lcm);
playsound(convert);
ss(socket).emit('file',convert, {size: src.size},currentgame);
ss.createBlobReadStream(convert).pipe(window.stream);
//ss.createReadStream(f).pipe(widnow.stream);
}
function playsound(raw) {
console.log("now playing a sound, that starts with", new Uint8Array(raw.slice(0, 10)));
context.decodeAudioData(raw, function (buffer) {
if (!buffer) {
console.error("failed to decode:", "buffer null");
return;
}
var source = context.createBufferSource();
source.buffer = buffer;
source.connect(context.destination);
source.start(0);
console.log("started...");
}, function (error) {
console.error("failed to decode:", error);
});
}
I am able to successfully create an array buffer using the float32toint16 function, however when I use the init sound function I get an error "null" meaning that the arraybuffer will not decode into an audio stream? Has anyone else had this issue? I have scoured the internet with no answer on how to do this. I am trying to play it this way because ultimately I will be streaming from client to client so I will be sending arraybufers via sockets.
thanks in advance.
If I'm understanding this correctly (there are some missing pieces in your code sample)...
decodeAudioData can only decode things like MP3 or WAV. It looks like you're passing it a raw Int16Array or Uint16Array. Because the underlying ArrayBuffer isn't a format that decodeAudioData understands, it gives up.
I think what you want to do is something like this:
function playsound( raw ) {
// i'll assume you know how to convert in this direction
// since you have convertFloat32ToInt16
var buffer = convertInt16ToFloat32( raw ),
src = context.createBufferSource(),
audioBuffer = context.createBuffer( 1, buffer.length, context.sampleRate );
audioBuffer.getChannelData( 0 ).set( buffer );
src.buffer = audioBuffer;
src.connect( context.destination );
src.start( 0 );
}
Basically, you already have a way to create the raw Float32Array that the Web Audio API likes, so there's no need to decode (and you can't decode anyway, since your data isn't a valid file format). So you just convert back to Float32Array, create your own AudioBuffer, write in the data from buffer, and go from there.
For converting from float32 to unsigned int 16 you can multiply each float32 value with 0xffff(which is 16 bit max value). and for int16 to float32 do this reversely which means divide by 0xffff. audio should be fine now.
I am new in stackoverflow. I should write this as a comment but due to lack of reputation point i can't. Thats why i have to write it as a answer. sorry for inconvenience.
Related
the title is indicating what I want, I just want to record audio as buffer and play that byte with only javascript at the same time (not node js) .
I researched about that for a while, and finally, I tried this method, and works for recording
const handleSuccess = function(stream) {
const context = new AudioContext();
const source = context.createMediaStreamSource(stream);
const processor = context.createScriptProcessor(1024*4, 1, 1);
source.connect(processor);
processor.connect(context.destination);
processor.onaudioprocess = function(e) {
console.log(e);
};
};
navigator.mediaDevices.getUserMedia({ audio: true, video: false })
.then(handleSuccess);
now the problem is playing this e
I tried decoding it with context.decodeAudioData(e.inputBuffer.getChannelData(0).buffer) but it throws an error :
DOMException: Failed to execute 'decodeAudioData' on 'BaseAudioContext': Unable to decode audio data
why it's not working? e.inputBuffer.getChannelData(0).buffer returns an object of type ArrayBuffer which is exactly what decodeAudioData wants, I also ensured that the output array is not empty.
Please help me to solve this problem.
Thank you
The audioprocess event of a ScriptProcessorNode gives you access to the current audio as an AudioBuffer. If you want to play that AudioBuffer you can use it as is and don't need to decode it anymore. It's already decoded. It can be played with an AudioBufferSourceNode.
The ScriptProcessorNode is officially deprecated but I guess it will never go away. Its successor - the AudioWorkletProcessor - gives you access to the raw channel data.
In trying to send document scans as binary data over a web socket the first block of code below works fairly well. Since the first byte contains information concerning what exactly to do with the blob, the image data starts at an offset of one byte; and because it appears that blob.slice() returns a copy of the original blob rather than simply reading it, this likely is not the most efficient method of processing the blob because it makes a copy of the entire blob less one byte.
socket.onmessage = function(evt) {
if (evt.data instanceof Blob) {
evt.data.slice(0,1).arrayBuffer()
.then( (b) => {
let v = new DataView(b),
r = v.getUint8(0);
// Do something based on r and then display scan.
let objectURL = URL.createObjectURL( evt.data.slice(1) );
imgscan.onload = () => {
URL.revokeObjectURL( objectURL );
};
imgscan.src = objectURL;
})
If the websocket's binaryType is changed to arraybuffer, it's a bit easier to read the first byte and apparently does not make a copy, but I do not understand how to get the image data out of the buffer to display it; that is, I don't see which method to apply to the DataView to get the raw binary data of the image. Would you please explain or point me to the correct documentation? Thank you.
This SO question seems similar but was not helpful to me anyway.
socket.onmessage = function(evt) {
if ( evt.data instanceof ArrayBuffer ) {
let v = new DataView(evt.data),
r = v.getUint8(0);
// How to get the raw binary image data out of
// the array buffer starting at the first byte?
I am trying to record and upload audio from javascript. I can successfullly record audio Blobs from a MediaRecorder. My understanding is that after recording several chunks into blobs, I would concatenate them as a new Blob(audioBlobs) and upload that. Unfortunately, the result on the server-side keeps being more or less gibberish. I'm currently running a localhost connection, so converting to uncompressed WAV isn't a problem (might be come one later, but that's a separate issue). Here is what I have so far
navigator.mediaDevices.getUserMedia({audio: true, video: false})
.then(stream => {
const mediaRecorder = new MediaRecorder(stream);
mediaRecorder.start(1000);
const audioChunks = [];
mediaRecorder.addEventListener("dataavailable", event => {
audioChunks.push(event.data);
});
function sendData () {
const audioBlob = new Blob(audioChunks);
session.call('my.app.method', [XXXXXX see below XXXXXX])
}
})
The session object here is an autobahn.js websockets connection to a python server (using soundfile. I tried a number of arguments in the place that was labelled by XXXXX in the code.
Just pass the audioBlob. In that case, the python side just receives an empty dictionary.
Pass audioBlob.text() in that case, I get something that looks somewhat binary (starts with OggS), but it can't be decoded.
Pass audioBlob.arrayBuffer(). In that case the python side receives an empty dictionary.
A possible solution could be to convert the data to WAV on the serverside (just changing the mime-type on the blob doesn't work) or to find a way to interpret the .text() output on the server side.
The solution was to use recorder.js and then use the getBuffer method in there to get the wave data as a Float32Array.
I have a .net core WebSocket server that receives live stream audio from client A, then I need to stream this live audio to client B (Browser). So I've received the byte array from client A, and I sent the byte array to client B (Browser) *The byte array is correct as I can convert it into .wav and play it without a problem.
In client B (Browser), I try to decode the array buffer into the audio buffer so it can be put into output and play.
The mediastreamhandler.SendArraySegToAllAsync is where I start to send out the byte array from the server to client B. I use to send to all method 1st, later will be modified and send out data by matching websocket connection ID.
private async Task Echo(HttpContext context, WebSocket webSocket)
{
Debug.WriteLine("Start Echo between Websocket server & client");
var buffer = new byte[1024 * 4];
WebSocketReceiveResult result = await webSocket.ReceiveAsync(new ArraySegment<byte>(buffer), CancellationToken.None);
while (!result.CloseStatus.HasValue)
{
await webSocket.SendAsync(new ArraySegment<byte>(buffer, 0, result.Count), result.MessageType, result.EndOfMessage, CancellationToken.None);
result = await webSocket.ReceiveAsync(new ArraySegment<byte>(buffer), CancellationToken.None);
await mediastreamhandler.SendArraySegToAllAsync(new ArraySegment<byte>(buffer, 0, result.Count));
}
Debug.WriteLine("Close Echo");
await webSocket.CloseAsync(result.CloseStatus.Value, result.CloseStatusDescription, CancellationToken.None);
}
I then receive the audio byte array through websocket.onmessage in Javascript. Then I pass the byte array to decode and play. But here it says "unable to decode data". While in Mozilla, it did says that content format was unknown (Do I need to reformat the byte array that I receive?) The byte array itself is fine because I've used the same byte to create .wav file locally and play it without any problem.
var ctx = new AudioContext();
function playSound(arrBuff) {
var myAudioBuffer;
var src = ctx.createBufferSource();
ctx.decodeAudioData(arrBuff, function (buffer) {
myAudioBuffer = buffer;
});
src.buffer = myAudioBuffer;
src.connect(ctx.destination);
src.start();
}
I then try another method to decode and play the audio, this time, it has played out some whitenoise sounds instead of streaming out the audio from Client A.
var ctx = new AudioContext();
function playSound(arrBuff) {
var myAudioBuffer;
var src = ctx.createBufferSource();
myAudioBuffer = ctx.createBuffer(1, arrBuff.byteLength, 8000);
var nowBuffering = myAudioBuffer.getChannelData(0);
for (var i = 0; i < arrBuff.byteLength; i++) {
nowBuffering[i] = arrBuff[i];
}
src.buffer = myAudioBuffer;
src.connect(ctx.destination);
src.start();
}
I think I really need some help over here guys, trying to play out the array buffer in weeks and still, couldn't have any breakthrough. Stuck here. Not sure what I've done, could you guys kindly guide me or tell me any other approach to this? Thanks in advance very much, really mean it.
decodeAudioData() requires complete files, so it can't be used to decode partial chunks of data as they are received from a websocket. If you can stream Opus audio files over your websocket, you can play back with an available WebAssembly decoder. See:
https://fetch-stream-audio.anthum.com/
https://github.com/AnthumChris/fetch-stream-audio
I've solved the issue months ago, just here to post my solution.
Steps:
Server receive payload string from Twilio
Send payload string from server to client (browser).
public async Task SendMessageAsync(WebSocket socket, string message)
{
if (socket.State != WebSocketState.Open)
return;
await socket.SendAsync(buffer: new ArraySegment<byte>(array: Encoding.ASCII.GetBytes(message),
offset: 0,
count: message.Length),
messageType: WebSocketMessageType.Text,
endOfMessage: true,
cancellationToken: CancellationToken.None);
}
Add wavheader to payload string at the client side before play out the audio.
function playSound(payloadBase64) {
/* You can generate the wav header here --> https://codepen.io/mxfh/pen/mWLMrJ */
var Base64wavheader = "UklGRgAAAABXQVZFZm10IBIAAAAHAAEAQB8AAEAfAAABAAgAAABmYWN0BAAAAAAAAABkYXRh";
var audio = new Audio('data:audio/wav;base64,' + Base64wavheader + payloadBase64);
audio.play();
};
Context: JavaScript, as part of a SDK (can be on node.js or browser).
Start point: I have a base64 string that's actually a base64 encoded PNG image (I got it from selenium webdriver - takeScreenshot).
Question: How do I crop it?
The techniques involving the canvas seem irrelevant (or am I wrong?). My code runs as part of tests - probably on node.js. The canvas approach doesn't seem to fit here and might also cause additional noise in the image.
All the libraries I found either deal with streams (maybe I should convert the string to stream somehow?) or deal directly with the UI by adding a control (irrelevant for me).
Isn't there something like (promises and callbacks omitted for brevity):
var base64png = driver.takeScreenshot();
var png = new PNG(base64png);
return png.crop(50, 100, 20, 80).toBase64();
?
Thanks!
Considering you wish to start with base64 string and end with cropped base64 string (image), here is the following code:
var Stream = require('stream');
var gm = require('gm');
var base64png = driver.takeScreenshot();
var stream = new Stream();
stream.on('data', function(data) {
print data
});
gm(stream, 'my_image.png').crop(WIDTH, HEIGHT, X, Y).stream(function (err, stdout, stderr) {
var data = '';
stdout.on('readable', function() {
data += stream.read().toString('base64');
});
stream.on('end', function() {
// DO something with your new base64 cropped img
});
});
stream.emit('data', base64png);
Be aware that it is unfinished, and might need some polishing or debugging (I am in no means a node.js guru), but the idea is next:
Convert string into stream
Read stream into GM module
Manipulate the image
Save it into a stream
Convert stream back into 64base string
Adding my previous comment as an answer:
Anyone looking to do this will need to decode the image to get the raw image data using a library such as node-pngjs and manipulate the data yourself (perhaps there is a library for such operations that doesn't rely on the canvas).