node.js download image with unknown extension from url - javascript

My issue is download image from given link, but there is one trouble - I dont know image extension(bmp, jpg, png etc.)
I am using request module, and succesfully download image of png extension:
var fs = require('fs');
var request = require('request');
var url = 'www.images.com/image01.';
request(url+ '.png').pipe(fs.createWriteStream('./image.png'));
But what to do when I dont know image extension. I think maybe I can check responce and if it's = '404' , then second try to download image from: url + '.[another extension]' and so on. By its look likes callback hell.

Make a Request to the URL, check the Content-type in the header of the request for the Mime-type of the file. You can have a file in your project, called mime-types.js
Where you download the file, you can make a request of that file:
var mime = request('mime-types.js');
And the content of your mime-types.js file should be something like this
const JPG = "image/jpeg";
module.exports = {
JPG: JPG
}
Then you can switch between the different mime-types you have and the result of the request.
You can find mime-types here: https://en.wikipedia.org/wiki/Internet_media_type#Type_image

Perform HEAD requests and check HTTP response message Content-Type header. It should include one of the specified MIME-types for images.

Related

How to download .js file by JavaScript from an url?

I know this question is asked hundreds of time in this forum, But I'm trying to download a .js file from an url in my Vue 2 application, but it's not working. Here is what I'm trying:
downloadScript() {
ApiService.post(`url`).then((res) => { // axios
try {
// Create a new link
const url = window.URL.createObjectURL(new Blob([res.data.path]));
const anchor = document.createElement("a");
anchor.href = url;.js";
anchor.setAttribute("download", "script.js");
document.body.appendChild(anchor);
anchor.click();
} catch {
//
}
});
},
This downloads a file which consists nothing but the url I've provided to the axios post request.
I'm getting API response like following:
{
"success": true,
"path": "https://something.com/files/iq-return.min.js"
}
I've to donwload the script in a file from the path
new Blob([res.data.path]) creates a Blob (which is sort-of-like-a-file) containing the text in the string you pass it.
Since that text is a URL, the file you download is a text file containing that URL.
If you want to create a Blob containing the JavaScript source code, then you need to get the JS source code. Make an HTTP request to the URL (e.g. with fetch) and put the response body in the Blob.
(Aside: don't append .js to the generated URL with you set href, that modifies the contents of the file!)
This will, of course, require permission from CORS if this is a cross-origin request.
If it isn't a cross-origin request then you can just set the href attribute to res.data.path without leaping through all these hoops.

Download a 'data:' image/file using puppeteer and node.js

I'm trying to download an image using node.js and puppeteer but I'm running into some issues. I'm using a webscraper to gather the links of the images from the site and then using the https/http package to download the image.
This works for the images using http and https sources but some images have links that look like this (the whole link is very long so I cut the rest):
data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAw8AAADGCAYAAACU07w3AAAZuUlEQVR4Ae3df4yU930n8Pcslu1I1PU17okdO1cLrTD+g8rNcvRyti6247K5NG5S5HOl5hA2uZ7du6RJEGYPTFy1Nv4RUJy0cWVkeQ9ErqqriHNrR8niZuVIbntBS886rBZWCGHVsNEFRQ5BloPCzGn2B+yzZMLyaP........
I'm not sure how to handle these links or how to download the image. Any help would be appreciated.
You need to first decode the url from base64 using node.js Buffer.
// the content type image/png has to be removed first
const data = 'iVBORw0KGgoAAAANSUhEUgAAAw8AAADGCAYAAACU07w3AAAZuUlEQVR4Ae3df4yU930n8Pcslu1I1PU17okdO1cLrTD+g8rNcvRyti6247K5NG5S5HOl5hA2uZ7du6RJEGYPTFy1Nv4RUJy0cWVkeQ9ErqqriHNrR8niZuVIbntBS886rBZWCGHVsNEFRQ5BloPCzGn2B+yzZMLyaP';
const buffer = new Buffer(data);
const base64data = buff.toString('base64');
// after this you will get the url string and continue to fetch the image
These are the base64 encoded images (mostly used for icons and small images).
you can ignore it.
if(url.startsWith('data:')){
//base 64 image
} else{
// an image url
}
if you really want to mess with base64 I can give you a workaround.
import { parseDataURI } from 'dauria';
import mimeTypes from 'mime-types';
const fileContent = parseDataURI(file);
// you probably need an extension for that image.
let ext = mimeTypes.extension(fileContent.MIME) || 'bin';
fs.writeFile("a random file"+"."+ext, fileContent.buffer, function (err) {
console.log(err); // writes out file without error, but it's not a valid image
});

Is there a way to download any kind of file using angular filesaver?

I'm using node.js and angular.js for my app and I'm trying to download files through the browser using Blob and fileSaver.js.
I've used these in other sections of my app to download text files and pdf files specifying the correct type when creating the Blob object without any problem, but in the current section I need to support any type of file and I don't know if it's possible.
For example, I've tried downloading an image file with and without type:image/png and the result was a corrupted image - inspecting it in a text editor and comparing it with the original file shows that many of the bytes were changed.
Here are the code snippets I use:
Server:
fs.readFile(/* snipped file path */, function(err, data){
if(err){
/* handle error */
}
else{
res.send(data);
}
});
Client:
$http.get(/* endPoint URL */)
.success(function(result){
var data = new Blob([result], {type: 'image/png'});
FileSaver.saveAs(data, filename);
});
A few questions:
Do I need to specify type for Blob? If so, do I need to specify it at server, too (it's a pain to determine it)? Can't I just skip it on both ends?
What causes the image test to result in corrupted file? Am I missing some content-type header or something?
Try adding {contentType: 'arraybuffer'} to your GET request and remove type from Blob definition, like so:
$http.get(/* endPoint URL */, {contentType: 'arraybuffer'})
.success(function(result){
var data = new Blob([result]);
FileSaver.saveAs(data, filename);
});
(Edit: deleted redundant type definition from Blob)

Saving file with JavaScript File API results wrong encoding

I have a problem (or may be two) with saving files using HTML5 File API.
A files comes from the server as a byte array and I need to save it. I tried several ways described on SO:
creating blob and opening it in a new tab
creating a hidden anchor tag with "data:" in href attribute
using FileSaver.js
All approaches allow to save the file but with breaking it by changing the encoding to UTF-8, while the file (in current test case) is in ANSI. And it seems that I have to problems: at the server side and at the client side.
Server side:
Server side is ASP.NET Web API 2 app, which controller sends the file using HttpResponseMessage with StreamContent. The ContentType is correct and corresponds with actual file type.
But as can be seen on the screenshot below server's answer (data.length) is less then actual file size calculated at upload (file.size). Also here could be seen that HTML5 File object has yet another size (f.size).
If I add CharSet with value "ANSI" to server's response message's ContentType property, file data will be the same as it was uploaded, but on saving result file still has wrong size and become broken:
Client side:
I tried to set charset using the JS File options, but it didn't help. As could be found here and here Eli Grey, the author of FileUplaod.js says that
The encoding/charset in the type is just metadata for the browser, not an encoding directive.
which means, if I understood it right, that it is impossible to change the encoding of the file.
Issue result: at the end I can successfully download broken files which are unable to open.
So I have two questions:
How can I save file "as is" using File API. At present time I cannot use simple way with direct link and 'download' attribute because of serverside check for access_token in request header. May be this is the "bottle neck" of the problem?
How can I avoid setting CharSet at server side and also send byte array "as is"? While this problem could be hacked in some way I guess it's more critical. For example, while "ANSI" charset solves the problem with the current file, WinMerge shows that it's encoding is Cyrillic 'Windows-1251' and also can any other.
P.S. the issue is related to all file types (extensions) except *.txt.
Update
Server side code:
public HttpResponseMessage DownloadAttachment(Guid fileId)
{
var stream = GetFileStream(fileId);
var message = new HttpResponseMessage(HttpStatusCode.OK);
message.Content = new StreamContent(stream);
message.Content.Headers.ContentLength = file.Size;
message.Content.Headers.ContentType = new MediaTypeHeaderValue(file.ContentType)
{
// without this charset files sent with bigger size
// than they are as shown on image 1
CharSet = "ANSI"
};
message.Content.Headers.ContentDisposition = new ContentDispositionHeaderValue("attachment")
{
FileName = file.FileName + file.Extension,
Size = file.Size
};
return message;
}
Client side code (TypeScript):
/*
* Handler for click event on download <a> tag
*/
private downloadFile(file: Models.File) {
var self = this;
this.$service.downloadAttachment(this.entityId, file.fileId).then(
// on success
function (data, status, headers, config) {
var fileName = file.fileName + file.extension;
var clientFile = new File([data], fileName);
// here's the issue ---^
saveAs(clientFile, fileName);
},
// on fail
function (error) {
self.alertError(error);
});
}
My code is almost the same as in answers on related questions on SO: instead of setting direct link in 'a' tag, I handle click on it and download file content via XHR (in my case using Angularjs $http service). Getting the file content I create a Blob object (in my case I use File class that derives from Blob) and then try to save it using FileSaver.js. I also tried approach with encoded URL to Blob in href attribute, but it only opens a new tab with a file broken the same way. I found that the problem is in Blob class - calling it's constructor with 'normal' file data I get an instance with 'wrong' size as could be seen on first two screenshots. So, as I understand, my problem not in the way I try to save my file, but in the way I create it - File API

Upload file to Windows Azure with only the link of the file is provided

How can I upload a file in azure if I only have the URL of the file to upload. In this case, i 'm using Dropbox file chooser which selects file from dropbox and returns its url path.
eq
https://www.dropbox.com/s/o9myet72y19iaan/Getting%20Started.pdf
Now we need the file to be stored in Windows Azure blob. What is the easiest way to do this without downloading the file first.
I'm planning to use a asp.net web api for the uploading of file to azure blob.
At first, I thought it should be quite straight forward as Azure Blob Storage support copying blobs from external URL however I don't think this would work in case of Dropbox files. I just tried it and got an error even though.
The link you mentioned above is not the direct link to the file. It's a link to a page on Dropbox's website from where you can download a file. This is obviously you don't want. Here's an alternate solution which you can try:
Replace www.dropbox.com in your URL with dl.dropboxusercontent.com (based on #smarx's comments below) and use that URL in the following code:
First you would need to append dl=1 to your request URL as query string. So your Dropbox URL would be https://www.dropbox.com/s/o9myet72y19iaan/Getting%20Started.pdf?dl=1. dl query string parameter indicates the file needs to be downloaded.
Next, using HTTPWebRequest try accessing this URL. Dropbox will respond back with another link and 302 status code. This link would be something like https://dl.dropboxusercontent.com/s/o9myet72y19iaan/Getting%20Started.pdf?token_hash=<tokenhash>.
Use this link in the code below to copy file. This would work.
CloudStorageAccount acc = new CloudStorageAccount(new StorageCredentials("account", "key"), false);
var client = acc.CreateCloudBlobClient();
var container = client.GetContainerReference("container-name");
container.CreateIfNotExists();
var blob = container.GetBlockBlobReference("dropbox-file-name");
blob.StartCopyFromBlob(new Uri("dropbox URL with dl.dropboxusercontent.com"));
Console.WriteLine("Copy request accepted");
Console.WriteLine("Now checking for copy state");
bool continueLoop = true;
do
{
blob.FetchAttributes();
var copyState = blob.CopyState;
switch (copyState.Status)
{
case CopyStatus.Pending:
Console.WriteLine("Copy is still pending. Will check status again after 1 second.");
System.Threading.Thread.Sleep(1000);//Copy is still pending...check after 1 second
break;
default:
Console.WriteLine("Terminating process with copy state = " + copyState.Status);
continueLoop = false;
break;
}
}
while (continueLoop);
Console.WriteLine("Press any key to continue.");

Categories