We Keep Coding

How do I store and fetch bytea data through hasura?

I've got a blob of audio data confirmed to play in the browser but fails to play after storing, retrieving, and conversion of the same data. I've tried a few methods without success, each time returning the error:
Uncaught (in promise) DOMException: Failed to load because no supported source was found
Hasura notes that bytea data must be passed in as a String, so I tried a couple things.
Converting the blob into base64 stores fine but the retrieval and playing of the data doesn't work. I've tried doing conversions within the browser to base64 and then back into blob. I think it's just the data doesn't store properly as bytea if I convert it to base64 first:
// Storing bytea data as base64 string
const arrayBuffer = await blob.arrayBuffer();
const byteArray = new Uint8Array(arrayBuffer);
const charArray = Array.from(byteArray, (x: number) => String.fromCharCode(x));
const encodedString = window.btoa(charArray.join(''));
hasuraRequest....
`
mutation SaveAudioBlob ($input: String) {
insert_testerooey_one(
object: {
blubberz: $input
}
) {
id
blubberz
}
}
`,
{ input: encodedString }
);
// Decoding bytea data
const decodedString = window.atob(encodedString);
const decodedByteArray = new Uint8Array(decodedString.length).map((_, i) =>
decodedString.charCodeAt(i)
);
const decodedBlob = new Blob([decodedByteArray.buffer], { type: 'audio/mpeg' });
const audio4 = new Audio();
audio4.src = URL.createObjectURL(decodedBlob);
audio4.play();
Then I came across a Github issue (https://github.com/hasura/graphql-engine/issues/3336) suggesting the use of a computed field to convert the bytea data to base64, so I tried using that instead of my decoding attempt, only to be met with the same error:
CREATE OR REPLACE FUNCTION public.content_base64(mm testerooey)
RETURNS text
LANGUAGE sql
STABLE
AS $function$
SELECT encode(mm.blobberz, 'base64')
$function$
It seemed like a base64 string was not the way to store bytea data, so I tried converting the data to a hex string prior to storing. It stores ok, I think, but upon retrieval the data doesn't play, and I think it's a similar problem as storing as base64:
// Encoding to hex string
const arrayBuffer = await blob.arrayBuffer();
const byteArray = new Uint8Array(arrayBuffer);
const hexString = Array.from(byteArray, (byte) =>
byte.toString(16).padStart(2, '0')
).join('');
But using the decoded data didn't work again, regardless of whether I tried the computed field method or my own conversion methods. So, am I just not converting it right? Is my line of thinking incorrect? Or what is it I'm doing wrong?
I've got it working if I just convert to base64 and store as a text field but I'd prefer to store as bytea because it takes up less space. I think something's wrong with how the data is either stored, retrieved, or converted, but I don't know how to do it. I know the blob itself is fine because when generated I can play audio with it, it only bugs out after fetching and attempted conversion its stored value. Any ideas?
Also, I'd really like to not store the file in another service like s3, even if drastically simpler.

How to convert uploaded file from charset=iso-8859-1 to charset=utf-8?

When user uploads a file that has charset=iso-8859-1 it comes with question marks and gibberish.
I have seen that there online web converts it successfully to utf-8 - so after uploading the file after this conversion the file is getting uploaded properly. This is the web: https://subtitletools.com/convert-text-files-to-utf8-online
This is my code:
const file = document.getElementById('some-id').files[0];
const reader = new FileReader();
reader.onloadend = event => {
let data = event.target.result;
console.log(`[data]:`, data); // question marks / gibberish
}
reader.readAsText(file);
I have also tried to use reader.readAsBinaryString but got gibberish instead of question marks.
I have also tried to use the utf8 library: https://www.npmjs.com/package/utf8 but it didn't work.
How the site that I mentioned above achieves to convert the file to the desired charset so its data is not in question marks or gibberish? BTW also Google Drive does it well.

You can use TextDecoder with you own charset
var data = new TextDecoder('iso-8859-1').decode(await file.arrayBuffer())
guess the harder part is to figure out what charset it's
If you are not using async/await then you can instead do this:
const file = document.getElementById('some-id').files[0];
file.arrayBuffer().then(ab => {
const data = new TextDecoder('iso-8859-1').decode(ab)
})

Some Random characters appearing at beginning of .txt file after converting from base64 node js

So i am dropping a .txt file in an uploader which is converting it into base64 data like this:
const {getRootProps, getInputProps} = useDropzone({
onDrop: async acceptedFiles => {
let font = ''; // its not actually a font just reusing some code i'll change it later its a .txt file so wherever you see font assume its NOT a font.
let reader = new FileReader();
let filename = acceptedFiles[0].name.split(".")[0];
console.log(filename);
reader.readAsDataURL(acceptedFiles[0]);
reader.onload = await function (){
font = reader.result;
console.log(font);
dispatch({type:'SET_FILES',payload:font})
};
setFontSet(true);
}
});
Then a POST request is made to the node js server and I indeed receive the base64 value. I then proceed to convert it back into a .txt file by writing it into a file called signals.txt like this:
server.post('/putInDB',(req,res)=>{
console.log(req.body);
var bitmap = new Buffer(req.body.data, 'base64');
let dirpath = `${process.cwd()}/signals.txt`;
let signalPath = path.normalize(dirpath);
connection.connect();
fs.writeFile(signalPath, bitmap, async (err) => {
if (err) throw err;
console.log('Successfully updated the file data');
//all the ending brackets and stuff
Now the thing is the orignal file looks like this :
Time,1,2,3,4,5,6,7,8,9,10,11,12
0.000000,7.250553,14.951141,5.550423,2.850217,-1.050080,-3.050233,1.850141,2.850217,-3.150240,1.350103,-2.950225,1.150088
But the file when writing back from base64 looks like this :
u«Zµìmþ™ZŠvÚ±î¸Time,1,2,3,4,5,6,7,8,9,10,11,12
0.000000,1.250095,0.250019,-4.150317,-0.350027,3.650278,1.950149,0.950072,-1.250095,-1.150088,-7.750591,-1.850141,-0.050004
See the weird characters in the beginning ? Why is this happening.

Remember to read up on what the functions you use do, because you're using readAsDataURL which does not give you the base64 encoded version of your data: it gives you Data-URL, and Data-URLs have a header prefix to tell URL parsers what kind of data this will be, and how to decode the data directly following the header.
To quote the MDN article:
Note: The blob's result cannot be directly decoded as Base64 without first removing the Data-URL declaration preceding the Base64-encoded data. To retrieve only the Base64 encoded string, first remove data:*/*;base64, from the result.
If you don't, blindly converting the Data-URL from base64 to plain text will give you some nonsense data at the start:
> Buffer.from('data:*/*;base64', 'base64').toString('utf-8')
'u�Z���{�'
Which raises another point: you would have caught this with POST data validation, because the Data-URL that you sent contains characters that are not allowed in base64. POST validation is always a good idea.

I know this isn't the exact code, but it is difficult to reproduce your problem with the code you provided. But the data you are sending needs to be a URL/URI encoded form.
So essentially:
encodeURI(base64data);
Encode URI is built into javascript: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/encodeURI
EDIT:
I saw you used the function readDataAsUrl(), but try using the encodeURI function and then readDataAsUrl().

Calculate MD5 or SHA of an Image using CryptoJS

I have a requirement where I need to calculate hash of an Image and upload it to a webservice. The image is captured using Cordova API and the API returns a URI of that image.
I checked the CryptoJS but it is taking a string as input.
var hash = CryptoJS.MD5("Message");
Any way we can calculate hash for the image ?
CryptoJS Lib : CryptoJS Home Page
Any other library is also fine as long as it can be used from Cordova App.

Using Cordova API you can get a Base64 encoded image instead of the URI. Which you can easily hash using CryptoJS.
When calling getPicture API use the DATA_URL option to get Base64 encoded image as the result:
navigator.camera.getPicture(onSuccess, onFail, {
destinationType: Camera.DestinationType.DATA_URL });
This will return the base64 encoded image in the onSuccess method as the param:
function onSuccess(imageURI) {
var hash = CryptoJS.MD5(imageURI);
}

Here is my way to generate the MD5 of the image file in HTML:
function onFileChange (e) {
const readerBuffer = new FileReader()
readerBuffer.readAsBinaryString(e.target.files[0])
readerBuffer.onloadend = function (e) {
const hash = CryptoJS.MD5(CryptoJS.enc.Latin1.parse(e.target.result))
const content-md5 = hash.toString(CryptoJS.enc.Base64)
}
}
Need to read the image as binary.
After reading this, I got that we need to convert the string to bytes using Latin-1.
This result can be used as content-md5 by AWS required.

encode/decode image with base64 breaks image

I am trying to encode and decode an image. I am using the FileReader's readAsDataURL method to convert the image to base64. Then to convert it back I have tried using readAsBinaryString() and atob() with no luck. Is there another way to persist images without base64 encoding them?
readAsBinaryString()
Starts reading the contents of the specified Blob, which may be a
File. When the read operation is finished, the readyState will become
DONE, and the onloadend callback, if any, will be called. At that
time, the result attribute contains the raw binary data from the file.
Any idea what I'm doing wrong here?
Sample Code
http://jsfiddle.net/qL86Z/3/
$("#base64Button").on("click", function () {
var file = $("#base64File")[0].files[0]
var reader = new FileReader();
// callback for readAsDataURL
reader.onload = function (encodedFile) {
console.log("reader.onload");
var base64Image = encodedFile.srcElement.result.split("data:image/jpeg;base64,")[1];
var blob = new Blob([base64Image],{type:"image/jpeg"});
var reader2 = new FileReader();
// callback for readAsBinaryString
reader2.onloadend = function(decoded) {
console.log("reader2.onloadend");
console.log(decoded); // this should contain binary format of the image
// console.log(URL.createObjectURL(decoded.binary)); // Doesn't work
};
reader2.readAsBinaryString(blob);
// console.log(URL.createObjectURL(atob(base64Image))); // Doesn't work
};
reader.readAsDataURL(file);
console.log(URL.createObjectURL(file)); // Works
});
Thanks!

After some more research I found the answer from here
I basically needed to wrap the raw binary in an arraybuffer and convert the binary chars to Unicode.
This is the code that was missing,
var binaryImg = atob(base64Image);
var length = binaryImg.length;
var ab = new ArrayBuffer(length);
var ua = new Uint8Array(ab);
for (var i = 0; i < length; i++) {
ua[i] = binaryImg.charCodeAt(i);
}
The full sample code is here

URL.createObjectURL expects a Blob (which can be a File) as its argument. Not a string. That's why URL.createObjectURL(file) works.
Instead, you are creating a FileReader reader that reads file as a data url, then you use that data url to create another Blob (with the same contents). And then you even create a reader2 to get a binary string from the just constructed blob. However, neither the base64Image url string part (even if btoa-decoded to a larger string) nor the decoded.binary string are vaild arguments to URL.createObjectURL!