The following code snippet receives websocket messages. It tries to readAsText the blob and also display it as a JPEG.
ws.onmessage = function(message) {
console.log('websocket blob:', message.data);
var text = reader.readAsText(message.data);
img.src = window.URL.createObjectURL(message.data);
console.log("jpeg received");
ctx.drawImage(img, 0, 0);
}
}
reader.onload = function(progressEvent) {
var text = this.result;
if (text.startsWith('{"')) {
console.log("status received", text);
document.querySelectorAll('pre')[0].textContent=text;
}
}
(The code is broken as it is trying to display plain text messages as JPEGs.)
How can I discriminate between two different sorts of incoming websocket packets? I am in control of both ends, so I can change what the NodeJS server sends, if required.
My JSON packets always begin with {" and my JPEG packets always begin with FF D8 FF.
Is there a way of attaching some meta-data to the websocket messages, when sending, that I could query in the page?
Can I do any processing of websocket blobs without callback methods?
My current workaround is now:
ws.binaryType = 'arraybuffer';
...
ws.onmessage = function(message) { // FIXME: utf8-unsafe
var arrayView = new Uint8Array(message.data);
console.log('websocket message:', arrayView);
if (arrayView[0] === 0xFF) { // jpeg
console.log("jpeg received");
var blob = new Blob([arrayView], {type: "image/jpeg"});
img.src = window.URL.createObjectURL(blob);
ctx.drawImage(img, 0, 0);
} else { // text
console.log("text received");
document.querySelectorAll('pre')[0].textContent=String.fromCharCode.apply(null, arrayView);
}
}
I would prefer a UTF8 safe handler, but that can wait until other parts of the project are working.
Related
Im trying to send audio from a client (javascript) to a server (java). I take the user audio from the microphone and then make a blob from it (and a url for the blob). The project is a spring boot project so i am looking for a way to send it as a parameter in a method to upload it to the server.
Was hoping that it would be possible to upload the blob to the server, but it seems to only be avalible localy on the browser and since the url for the blob starts with "blob:" before "http" it causes problems.
I have also looked at serialization but dont seem to find a way to do that with a blob in js.
Just passing the blob url here between the client and the server
Client side in js
// Convert the audio data in to blob
// after stopping the recording
mediaRecorder.onstop = function (ev) {
console.log(dataArray);
// blob of type mp3
let audioData = new Blob(dataArray,
{ 'type': 'audio/mp3;' });
// After fill up the chunk
// array make it empty
dataArray = [];
// Creating audio url with reference
// of created blob named 'audioData'
let audioSrc = window.URL
.createObjectURL(audioData);
//console.log(audioSrc);
// Pass the audio url to the 2nd video tag
playAudio.src = audioSrc;
const url = "http://localhost:8080/speech?url=" + audioSrc;
console.log(url);
$.get(url, function(data) {
$("#resultat").html("transcribed tekst: " + data);
});
}
Server in Java
#GetMapping("/speech")
public String speechToText(String url) throws IOException {
try (SpeechClient speechClient = SpeechClient.create()) {
// The path to the audio file to transcribe
String gcsUri = url;
// Builds the sync recognize request
RecognitionConfig config =
RecognitionConfig.newBuilder()
.setEncoding(RecognitionConfig.AudioEncoding.LINEAR16)
.setSampleRateHertz(16000)
.setLanguageCode("en-US")
.build();
RecognitionAudio audio = RecognitionAudio.newBuilder().setUri(gcsUri).build();
// Performs speech recognition on the audio file
RecognizeResponse response = speechClient.recognize(config, audio);
List<SpeechRecognitionResult> results = response.getResultsList();
for (SpeechRecognitionResult result : results) {
// There can be several alternative transcripts for a given chunk of speech. Just use the
// first (most likely) one here.
SpeechRecognitionAlternative alternative = result.getAlternativesList().get(0);
System.out.printf("Transcription: %s%n", alternative.getTranscript());
return alternative.getTranscript();
}
return "idk";
} catch (IOException e) {
e.printStackTrace();
return "noe ble feil";
}
}
I'm using the gcloud API on a Nodejs web server to upload files. I'd prefer the files not be uploaded on the client side and instead uploaded on the server. Currently, I am producing a blob on the client side, then converting it to text and passing that to the server through a POST request. All of the information gets successfully passed from the client to the server as expected. This data is also uploaded to gcloud, however, Gcloud does not recognize this as a valid file nor does my computer when I download it.
What is the best way to get the contents of the file to gcloud from the server side? I've tried using dataURIs and reading the orignal file by text and both produce similiar issues. I've also explored piping a readFileStream from the blob on the server end but blobs are not natively supported by node so I have not done so yet.
Client Side
function readSingleFile(e, func, func2){
var file = e.target.files[0];
if(!file){
return; // Add error msg_here
}
var reader = new FileReader();
reader.onload = function(e){
let contents = e.target.result;
let img = document.createElement('img')
let cvs = document.createElement('canvas');
img.onload = ()=>{
cvs.width = img.width;
cvs.height= img.height;
let ctx = cvs.getContext('2d');
ctx.drawImage(img,0,0);
cvs.toBlob((res)=>{res.text().then((text)=>{func2(text)})}, "image/jpeg", 0.92);
}
img.src=contents;
func(contents);
}
reader.readAsDataURL(file);
}
Server Side
function publishPrintjob(dataObj){
try{
var newElemKey = database.ref().child('queue').push().key; // Get random Key
// Create a new blob in the bucket and upload the file data.
const gcloudFile = storage.file('images/' + newElemKey + '.jpg');
gcloudFile.save(dataObj.sockImageFile, function(err) {
if (!err) {
Console.log("File Uploaded!")
}
});
var data = {
date: dataObj.Date,
email: dataObj.email,
design: dataObj.Design,
author: dataObj.Author,
address: dataObj.address,
imageKey: newElemKey,
}
admin.database().ref('queue/' + newElemKey).set(data);
} catch(err){
console.log(err)
}
}
Note: func simply shows the image on the client side, func2 just adds the contents to the POST object.
Uploading a file directly from the computer would be easiest using the storage.bucket(bucketName).upload() function from the cloud storage library. However, this uses location of a file locally and thus will not work unless a file is transferred to the server and saved first. This could be achieved using multi-part form data. Using multipart or uploading locally are better methods for uploading to google storage.
Instead, I solve this by first converting the image to a dataURI, sending the data URI to the server via the body of a GET request, and then converting it to a buffer with a readable stream that can be piped to google storage.
Client
let formData = getFormData('myForm');
var xhttp = new XMLHttpRequest();
xhttp.onreadystatechange = function() {
if (this.readyState == 4 && this.status == 200) {
// Typical action to be performed when the document is ready:
}
};
xhttp.open("POST", "dashboard", true);
xhttp.setRequestHeader('Content-Type', 'application/json');
xhttp.send(JSON.stringify(formData));
xhttp.onload = ()=> {
console.log(JSON.parse(xhttp.response))
// Handle server response here
};
}
Server
// DataObject is the body of the GET request, the property imageFile is the URI from readFileAsURI
function uploadImageOnServer(dataObj){
try{
var newElemKey = database.ref().child('queue').push().key; // Get random Key to use as filename
// Create a new blob in the bucket and upload the file data.
const gcloudFile = storage.file('images/' + newElemKey + '.jpeg');
var fs = require('fs'); // Should be required at the top of the file
var string = dataObj.ImageFile;
var regex = /^data:.+\/(.+);base64,(.*)$/;
var matches = string.match(regex);
var ext = matches[1];
var data = matches[2];
var buffer = Buffer.from(data, 'base64');
// Create the readstream
const readableInstanceStream = new Readable({
read() {
this.push(buffer);
this.push(null);
}
});
readableInstanceStream.pipe(gcloudFile.createWriteStream()) // link to gcloud storage api
.on('error', function(err) {
console.log('error')
})
.on('finish', function() {
console.log('upload complete')
});
} catch(err){
console.log(err)
}
}
I've tried several different options, so many I've lost track of them all. I'm making an AJAX request and the response is of Content-Type: image/png, and the contents are the actual image.
I would absolutely love to display the image, but nothing seems to work the way I want:
// imgdata contains a string that looks like this: "�PNG..."
var img = document.createElement('img');
// no good
img.src = 'data:image/png;base64,' + btoa(unescape(encodeURIComponent(data)));
// also no good
img.src = 'data:image/png;base64,' + btoa(encodeURIComponent(data));
// also no good
img.src = 'data:image/png;base64,' + btoa($.map(d, function(x){ return x.charCodeAt(0); }))
I've tried a few other things, but still no dice.
Is there any simple (or even complciated) way to do this in Javascript?
This isn't done with base64 but with blob, but you'll get exactly the same result:
var xhr = new XMLHttpRequest();
xhr.onreadystatechange = function(){
if (this.readyState == 4 && this.status == 200){
var img = document.getElementById('image');
var url = window.URL || window.webkitURL;
img.src = url.createObjectURL(this.response);
}
}
// Relative path because :
// No 'Access-Control-Allow-Origin' header is present...
xhr.open('GET', '/img/logo.png');
xhr.responseType = 'blob';
xhr.send();
Demo here : http://jsfiddle.net/sparkup/sp4cukee/
If you're serving this file via node.js or PHP you could always write the file to disk in temporary location server-side, serve it, and then immediately delete it afterwards.
var tempFile = 'path/to/file.jpg';
// Write the file to disk
fs.writeFile(tempFile, base64versionOfPhoto, function (err) {
if (err) {
res.header("Content-Type","application/json");
return res.send(JSON.stringify(err));
} else {
// Pipe the jpg to the user
res.header("Content-Type","image/jpeg");
var readStream = fs.createReadStream(tempFile);
readStream.pipe(res, {end: false});
readStream.on("end", function () {
res.end();
// Delete the file
fs.unlink(tempFile, function (err) {
if (err) console.log(err);
});
});
}
});
How would I correctly receive/send raw audio data?
I currently receive raw audio data over WebSockets (with Node.js) and write them to a file. Data is sent as byte[] array of 1024 byte in size from a Java client. Audio format is PCM, Mono, 16bit, 44100Hz.
I'm trying to pass on the raw audio data directly to browser, but it doesn't seem to work. What am I missing? Sorry for being new at this. The browsers I tested all support (Webkit) AudioContext. For each package received in browser, I get the following message:
Uncaught SyntaxError: An invalid or illegal string was specified. audio.php:19 1024
Node.js (server):
var WebSocketServer = require('ws').Server
, wss = new WebSocketServer({port: 8081});
var Clients = [];
function findClient(url) {
return (Clients.indexOf(url));
}
wss.on('connection', function(ws) {
Clients.push(ws);
console.log('Connected: %s', ws.upgradeReq.url);
console.log('Clients: %s', Clients.length);
ws.on('message', function(message, flags) {
if(flags.binary) {
ws.send(message, {binary:true});
}
console.log('data arrived: %s', message);
});
ws.on('close', function(user) {
console.log('Disconnected: %s', ws.upgradeReq.url);
Clients.splice(findClient(ws));
console.log('Clients: %s', Clients.length);
});
Client browser:
<script language="javascript">
if (window.webkitAudioContext) {
var ctx = new webkitAudioContext();
} else if (window.AudioContext) {
var ctx = new AudioContext();
}
function testSound(buff) {
var src = ctx.createBufferSource();
src.buffer = ctx.createBuffer(buff, false);
//src.looping = false;
src.connect(ctx.destination);
src.noteOn(0);
}
var ws = new WebSocket('ws://localhost:8081');
ws.binaryType = 'arraybuffer';
ws.onmessage = function(e) {
console.log(e.data.byteLength);
testSound(e.data);
}
</script>
I'm trying to make an application that stores and retrieves Video files to and from IndexedDB. However, I am having issues while retrieving in Firefox and while storing in Chrome. I'll post the code:
(function () {
// IndexedDB
var indexedDB = window.indexedDB || window.webkitIndexedDB || window.mozIndexedDB || window.OIndexedDB || window.msIndexedDB,
IDBTransaction = window.IDBTransaction || window.webkitIDBTransaction || window.OIDBTransaction || window.msIDBTransaction,
dbVersion = 1.0;
// Create/open database
var request = indexedDB.open("videoFiles", dbVersion);
var db;
var createObjectStore = function (dataBase) {
// Create an objectStore
console.log("Creating objectStore")
dataBase.createObjectStore("earth");
},
getVideoFile = function () {
// Create XHR
var xhr = new XMLHttpRequest(),
blob;
xhr.open("GET", "day_the_earth_stood_still.ogv", true);
// Set the responseType to blob
xhr.responseType = "blob";
xhr.addEventListener("load", function () {
if (xhr.status === 200) {
console.log("Video retrieved");
// Blob as response
blob = xhr.response;
console.log("Blob:" + blob);
// Put the received blob into IndexedDB
putEarthInDb(blob);
}
}, false);
// Send XHR
xhr.send();
},
putEarthInDb = function (blob) {
console.log("Putting earth in IndexedDB");
// Open a transaction to the database
var transaction = db.transaction(["earth"], "readwrite");
// Put the blob into the dabase
var put = transaction.objectStore("earth").put(blob, "video");
// Retrieve the file that was just stored
transaction.objectStore("earth").get("video").onsuccess = function (event) {
var vidFile = event.target.result;
console.log("Got earth!" + vidFile);
console.log('File Type: ' + vidFile.type); /// THIS SHOWS : application/xml
// Get window.URL object
var URL = window.URL || window.webkitURL;
// Create and revoke ObjectURL
var vidURL = URL.createObjectURL(vidFile);
// Set vid src to ObjectURL
var vidEarth = document.getElementById("earth");
vidEarth.setAttribute("src", vidURL);
// Revoking ObjectURL
URL.revokeObjectURL(vidURL);
};
};
request.onerror = function (event) {
console.log("Error creating/accessing IndexedDB database");
};
request.onsuccess = function (event) {
console.log("Success creating/accessing IndexedDB database");
db = request.result;
db.onerror = function (event) {
console.log("Error creating/accessing IndexedDB database");
};
// Interim solution for Google Chrome to create an objectStore. Will be deprecated
if (db.setVersion) {
if (db.version != dbVersion) {
var setVersion = db.setVersion(dbVersion);
setVersion.onsuccess = function () {
createObjectStore(db);
getVideoFile();
};
}
else {
getVideoFile();
}
}
else {
getVideoFile();
}
}
// For future use. Currently only in latest Firefox versions
request.onupgradeneeded = function (event) {
createObjectStore(event.target.result);
};
})();
Problem 1(Firefox): In Firefox, the line console.log('File Type: ' + vidFile.type); above shows "application/xml" while GETTING a video file (mp4, ogv, webm) and so the Video tag says "video format or mime type is not supported".
However when I GET an image file like png it shows "image/png" and works well if the src of an img tag is set.
Problem 2(Chrome): In Chrome, both image and video are not even getting stored into the IndexedDB. At the following line:
var put = transaction.objectStore("earth").put(blob, "video");
Uncaught Error: DataCloneError: DOM IDBDatabase Exception 25 is thrown.
I am new to IndexedDB and have no clue on how to solve this. All I need to do is store video files into indexedDB, retrieve it and show in Video tag.
The HTML is shown below:
(mp4):
<div class="myVidDiv">
<video id="earth" type="video/mp4" codecs="avc1.42e01e, mp4a.40.2" controls> </video>
</div>
(ogv):
<div class="myVidDiv">
<video id="earth" type="video/ogg" codecs="theora, vorbis" controls></video>
</div>
Also tried without "codecs" attribute. Nothing works. I've been stuck with this for dayss together... Couldn't find any working example via google as well. Someone kindly help me with this.
Ok, I'll try to sum up what it came out from the comments.
1. Firefox
It seems that, originally, the Blob object you get from the AJAX request has content type application/xml, because that's what you get in response from the server. It can be a problem of misconfiguration.
If you have access to the HTTP server's configuration, it may be solved quite easily. If it's Apache, you can simply add this line:
AddType video/ogg .ogv
Save, restart Apache and you should be ok. If you can't change the server's configuration, you'll have to change the Blob's content type in order to match the desired one:
blob = xhr.response.slice(0, xhr.response.size, "video/ogg");
Note that this could be memory expensive because you're making a copy of a (probably) large file, but xhr.response should be sent to garbage after a couple of steps.
2. Chrome
It seems that Chrome still doesn't support Blob and File storing.
It also seems that they've already fixed the problem, but haven't deployed the fix yet. I wonder what they're waiting for :[
UPDATE: as of July 1st, 2014, Chrome dev supports storing blobs into IndexedDB. It's expected to land soon on the stable channel.