We Keep Coding

How to save to disk a file of unknown content type, when receiving it as a binary stream in a string. (All my attempts corrupt the file)

I receive a string from an API that is a binary stream and a filename, here is an example (Shortened for brevity).
{ filename: '002-000017.pdf' body: '%PDF-1.7\n\n4 0 obj\n(Identity)\nendobj\n5 0 obj\n(Adobe)\nendobj\n8 0 obj\n<<\n/Filter /FlateDecode\n/Length 181864\n/Length1 542744\n/Type /Stream\n>>\nstream\nx��}\t`�������}dw����~��\t9 #H6'\t�9 e.t.c ' }
Other files simply have \u001c\u0004N�v�$$\u0010$�\u0000
In the above example it is a PDF file, but I dont know what content type the file will be, it could be a .docx, .txt or even no file extension. Also the name will not always have an extension. I take this file and convert it to blob, that I attach to and click an anchor.
I have tried specifying no type, 'application/unknown' and 'application/octet-stream'. I even tried hardcoding it to 'application/pdf' for the above sample, just to see if that would work. In every scenario the pdf is blank. (Or unreadable with other file types, docx for example just crashes out when opening with word).
The exception is .txt, any .txt files work perfectly.
I have confirmed the string is convertable manually and readable, I did this using PowerShell without issue. The issue is im doing something wrong in my web side javascript code. What am I missing here?
Code:
const { filename } = response
const blob = new Blob([response.body])
const blobURL = window.URL.createObjectURL(blob);
// Using React Ref to handle the anchor.
downloadAnchor!.current!.href = blobURL;
downloadAnchor!.current!.download = 'filename';
downloadAnchor!.current!.click();

Download a 'data:' image/file using puppeteer and node.js

I'm trying to download an image using node.js and puppeteer but I'm running into some issues. I'm using a webscraper to gather the links of the images from the site and then using the https/http package to download the image.
This works for the images using http and https sources but some images have links that look like this (the whole link is very long so I cut the rest):
data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAw8AAADGCAYAAACU07w3AAAZuUlEQVR4Ae3df4yU930n8Pcslu1I1PU17okdO1cLrTD+g8rNcvRyti6247K5NG5S5HOl5hA2uZ7du6RJEGYPTFy1Nv4RUJy0cWVkeQ9ErqqriHNrR8niZuVIbntBS886rBZWCGHVsNEFRQ5BloPCzGn2B+yzZMLyaP........
I'm not sure how to handle these links or how to download the image. Any help would be appreciated.

You need to first decode the url from base64 using node.js Buffer.
// the content type image/png has to be removed first
const data = 'iVBORw0KGgoAAAANSUhEUgAAAw8AAADGCAYAAACU07w3AAAZuUlEQVR4Ae3df4yU930n8Pcslu1I1PU17okdO1cLrTD+g8rNcvRyti6247K5NG5S5HOl5hA2uZ7du6RJEGYPTFy1Nv4RUJy0cWVkeQ9ErqqriHNrR8niZuVIbntBS886rBZWCGHVsNEFRQ5BloPCzGn2B+yzZMLyaP';
const buffer = new Buffer(data);
const base64data = buff.toString('base64');
// after this you will get the url string and continue to fetch the image

These are the base64 encoded images (mostly used for icons and small images).
you can ignore it.
if(url.startsWith('data:')){
//base 64 image
} else{
// an image url
}
if you really want to mess with base64 I can give you a workaround.
import { parseDataURI } from 'dauria';
import mimeTypes from 'mime-types';
const fileContent = parseDataURI(file);
// you probably need an extension for that image.
let ext = mimeTypes.extension(fileContent.MIME) || 'bin';
fs.writeFile("a random file"+"."+ext, fileContent.buffer, function (err) {
console.log(err); // writes out file without error, but it's not a valid image
});

Convert HEIC to JPG , using php or JS

Anyone tried to convert a heic to jpg?
I looked at the official repository, but I did'nt understand how it works.
All examples in the repository are working. But when I try to process my photo, made on the iphone, the script refuses to process it.

I've had some luck recently with the conversion using libheif. So I made this library which should greatly simplify the whole process
https://github.com/alexcorvi/heic2any
The only caveat is that the resulting PNG/JPG doesn't retain any of the meta-data that were in the original HEIC.

I managed to convert heic to jpg with the help of heic2any js library (https://github.com/alexcorvi/heic2any/blob/master/docs/getting-started.md)
I converted the picture on client side, then gave it to the input in client side.
Server is seeing it as it was originally uploaded as jpg.
function convertHeicToJpg(input)
{
var fileName = $(input).val();
var fileNameExt = fileName.substr(fileName.lastIndexOf('.') + 1);
if(fileNameExt == "heic") {
var blob = $(input)[0].files[0]; //ev.target.files[0];
heic2any({
blob: blob,
toType: "image/jpg",
})
.then(function (resultBlob) {
var url = URL.createObjectURL(resultBlob);
$(input).parent().find(".upload-file").css("background-image", "url("+url+")"); //previewing the uploaded picture
//adding converted picture to the original <input type="file">
let fileInputElement = $(input)[0];
let container = new DataTransfer();
let file = new File([resultBlob], "heic"+".jpg",{type:"image/jpeg", lastModified:new Date().getTime()});
container.items.add(file);
fileInputElement.files = container.files;
console.log("added");
})
.catch(function (x) {
console.log(x.code);
console.log(x.message);
});
}
}
$("#input").change(function() {
convertHeicToJpg(this);
});
What I am doing is converting the heic picture to jpg, then previewing it.
After that I add it to the original input. Server side will consider it as an uploaded jpg.
Some delay can appear while converting, therefore I placed a loader gif while uploading.

Adding mp3s into zip using jszip

everyone. Trying to unsuccesfully add mp3s into my zip file using the wonderful JSZIP lib. Right now, it only created the zip file with the correct file name, but the mp3 is always blank.
This is the code I have so far:
//init
var zip = new JSZip();
//add an mp3 titled "any other way" and decode binary to base64
zip.file("any other way.mp3", btoa("absolutepath/to/my/file/any_other_way.mp3"), {base64: true});
//generate zip
var content = zip.generate();
//download zip
location.href="data:application/zip;base64,"+content;

So, I ended up including another js file with the JSZIP utils in my project and called the following method and it downloaded fine on Chrome. However, if you want to make it work on IE and safari you will have to implement Downloadify (http://stuk.github.io/jszip/documentation/howto/write_zip.html#toc_3):
// loading a file and add it in a zip file
JSZipUtils.getBinaryContent("path/to/audio.mp3", function (err, data) {
if(err) {
throw err; // or handle the error
}
var zip = new JSZip();
zip.file("audio.mp3", data, {binary:true});
});
http://stuk.github.io/jszip-utils/documentation/api/getbinarycontent.html
Also, here is a link to the issue: https://github.com/Stuk/jszip/issues/176#issuecomment-57266207

Image not showing using chrome filesystem toURL

I have the following code to write an image into the filesystem, and read it back for display. Prior to trying out the filesystem API, I loaded the whole base64 image into the src attribute and the image displayed fine. Problem is the images can be large so if you add a few 5MB images, you run out of memory. So I thought I'd just write them to the tmp storage and only pass the URL into the src attribute.
Trouble is, nothing gets displayed.
Initially I thought it might be something wrong with the URL, but then I went into the filesystem directory, found the image it was referring to and physically replaced it with the real binary image and renamed it to the same as the replaced image. This worked fine and the image is displayed correctly, so the URL looks good.
The only conclusion I can come to is that the writing of the image is somehow wrong - particularly the point where the blob is created. I've looked through the blob API and can't see anything that I may have missed, however I'm obviously doing something wrong because it seems to be working for everyone else.
As an aside, I also tried to store the image in IndexedDB and use the createObjectURL to display the image - again, although the URL looks correct, nothing is displayed on the screen. Hence the attempt at the filesystem API. The blob creation is identical in both cases, with the same data.
The source data is a base64 encoded string as I mentioned. Yes, I did also try to store the raw base64 data in the blob (with and without the prefix) and that didn't work either.
Other info - chrome version 28, on linux Ubuntu
//strip the base64 `enter code here`stuff ...
var regex = /^data.+;base64,/;
if (regex.test(imgobj)) { //its base64
imgobj = imgobj.replace(regex,"");
//imgobj = B64.decode(imgobj);
imgobj = window.atob(imgobj);
} else {
console.log("it's already :", typeof imgobj);
}
// store the object into the tmp space
window.requestFileSystem(window.TEMPORARY, 10*1024*1024, function(fs) {
// check if the file already exists
fs.root.getFile(imagename, {create: false}, function(fileEntry) {
console.log("File exists: ", fileEntry);
callback(fileEntry.toURL(), fileEntry.name);
//
}, function (e) { //file doesn't exist
fs.root.getFile(imagename, {create: true}, function (fe) {
console.log("file is: ", fe);
fe.createWriter(function(fw){
fw.onwriteend = function(e) {
console.log("write complete: ", e);
console.log("size of file: ", e.total)
callback(fe.toURL(), fe.name);
};
fw.onerror = function(e) {
console.log("Write failed: ", e.toString());
};
var data = new Blob([imgobj], {type: "image/png"});
fw.write(data);
}, fsErrorHandler);
}, fsErrorHandler);
});
// now create a file
}, fsErrorHandler);
Output from the callback is:
<img class="imgx" src="filesystem:file:///temporary/closed-padlock.png" width="270px" height="270px" id="img1" data-imgname="closed-padlock.png">
I'm at a bit of a standstill unless someone can provide some guidance...
UPDATE
I ran a test to encode and decode the base64 image with both the B64encoder/decoder and atob/btoa -
console.log(imgobj); // this is the original base64 file from the canvas.toDataURL function
/* B64 is broken*/
B64imgobjdecode = B64.decode(imgobj);
B64imgobjencode = B64.encode(B64imgobjdecode);
console.log(B64imgobjencode);
/* atob and btoa decodes and encodes correctly*/
atobimgobj = window.atob(imgobj);
btoaimgobj = window.btoa(atobimgobj);
console.log(btoaimgobj);
The results show that the btoa/atob functions work correctly but the B64 does not - probably because the original encoding didn't use the B64.encode function...
The resulting file in filesystem TEMPORARY, I ran through an online base64 encoder for comparison and the results are totally different. So the question is - while in the filesystem temp storage, is the image supposed to be an exact image, or is it padded with 'something' which only the filesystem API understands? Remember I put the original PNG in the file system directory and the image displayed correctly, which tends to indicate that the meta-data about the image (eg. the filename) is held elsewhere...
Can someone who has a working implementation of this confirm if the images are stored as images in the filesystem, or are padded with additional meta-data?

So to answer my own question - the core problem was in the base64 encoding/decoding - I've since then changed this to use things like ajax and responseTypes like arraybuffer and blob and things have started working.
To answer the last part of the question, this is what I've found - in the filesystem tmp storage, yes the file is supposed to be an exact binary copy - verified this in chrome and phonegap.