There exists a url which dynamically creates a file (zip) for download.
http://example.com/generateZipFile.php
In a browser, opening the url simply opens the download dialog box, and a user can save the file to their hard drive.
I need to automatically download this file using node.js
I have tried the following code:
var file = fs.createWriteStream("http://example.com/generateZip.php");
var request = http.get(URL, function(response) {
response.pipe(file);
});
However it only downloads the content of what appears to be a HTML redirect page to the current incantation of the zip file, not the zip file itself:
> <head><title>Document Moved</title></head> <body><h1>Object
> Moved</h1>This document may be found <a
> HREF="http://example.com/generated12345.zip">here</a></body>
I can not hard code the generated name of the zip file, as it is subject to change.
http module does not support redirection for queries. You can use request module:
require('request')
.get('http://example.com/generateZip.php')
.on('error', function(error) {
console.log(error)
})
.pipe(require('fs').createWriteStream(__dirname + '/tmp.file'))
Related
Over the years on snapchat I have saved lots of photos that I would like to retrieve now, The problem is they do not make it easy to export, but luckily if you go online you can request all the data (thats great)
I can see all my photos download link and using the local HTML file if I click download it starts downloading.
Here's where the tricky part is, I have around 15,000 downloads I need to do and manually clicking each individual one will take ages, I've tried extracting all of the links through the download button and this creates lots of Urls (Great) but the problem is, if you past the url into the browser then ("Error: HTTP method GET is not supported by this URL") appears.
I've tried a multitude of different chrome extensions and none of them show the actually download, just the HTML which is on the left-hand side.
The download button is a clickable link that just starts the download in the tab. It belongs under Href A
I'm trying to figure out what the best way of bulk downloading each of these individual files is.
So, I just watched their code by downloading my own memories. They use a custom JavaScript function to download your data (a POST request with ID's in the body).
You can replicate this request, but you can also just use their method.
Open your console and use downloadMemories(<url>)
Or if you don't have the urls you can retrieve them yourself:
var links = document.getElementsByTagName("table")[0].getElementsByTagName("a");
eval(links[0].href);
UPDATE
I made a script for this:
https://github.com/ToTheMax/Snapchat-All-Memories-Downloader
Using the .json file you can download them one by one with python:
req = requests.post(url, allow_redirects=True)
response = req.text
file = requests.get(response)
Then get the correct extension and the date:
day = date.split(" ")[0]
time = date.split(" ")[1].replace(':', '-')
filename = f'memories/{day}_{time}.mp4' if type == 'VIDEO' else f'memories/{day}_{time}.jpg'
And then write it to file:
with open(filename, 'wb') as f:
f.write(file.content)
I've made a bot to download all memories.
You can download it here
It doesn't require any additional installation, just place the memories_history.json file in the same directory and run it. It skips the files that have already been downloaded.
Short answer
Download a desktop application that automates this process.
Visit downloadmysnapchatmemories.com to download the app. You can watch this tutorial guiding you through the entire process.
In short, the app reads the memories_history.json file provided by Snapchat and downloads each of the memories to your computer.
App source code
Long answer (How the app described above works)
We can iterate over each of the memories within the memories_history.json file found in your data download from Snapchat.
For each memory, we make a POST request to the URL stored as the memories Download Link. The response will be a URL to the file itself.
Then, we can make a GET request to the returned URL to retrieve the file.
Example
Here is a simplified example of fetching and downloading a single memory using NodeJS:
Let's say we have the following memory stored in fakeMemory.json:
{
"Date": "2022-01-26 12:00:00 UTC",
"Media Type": "Image",
"Download Link": "https://app.snapchat.com/..."
}
We can do the following:
// import required libraries
const fetch = require('node-fetch'); // Needed for making fetch requests
const fs = require('fs'); // Needed for writing to filesystem
const memory = JSON.parse(fs.readFileSync('fakeMemory.json'));
const response = await fetch(memory['Download Link'], { method: 'POST' });
const url = await response.text(); // returns URL to file
// We can now use the `url` to download the file.
const download = await fetch(url, { method: 'GET' });
const fileName = 'memory.jpg'; // file name we want this saved as
const fileData = download.body; // contents of the file
// Write the contents of the file to this computer using Node's file system
const fileStream = fs.createWriteStream(fileName);
fileData.pipe(fileStream);
fileStream.on('finish', () => {
console.log('memory successfully downloaded as memory.jpg');
});
My app is created with mean and I am a user of docker too. The purpose of my app is to create and download a CSV file. I already created my file, compressed it and placed it in a temp folder (the file will be removed after the download). This part is in the nodejs server side and works without problems.
I already use several things like (res.download) which is supposed to download directly the file in the browser but nothing append. I tried to use blob in the angularjs part but it doesn't work.
The getData function creates and compresses the file (it exists I can reach it directly when I look where the app is saved).
exports.getData = function getData(req, res, next){
var listRequest = req.body.params.listURL;
var stringTags = req.body.params.tagString;
//The name of the compressed CSV file
var nameFile = req.body.params.fileName;
var query = url.parse(req.url, true).query;
//The function which create the file
ApollineData.getData(listRequest, stringTags, nameFile)
.then(function (response){
var filePath = '/opt/mean.js/modules/apolline/client/CSVDownload/'+response;
const file = fs.createReadStream(filePath);
res.download(filePath, response);
})
.catch(function (response){
console.log(response);
});
};
My main problem is to download this file directly in the browser without using any variable because it could be huge (like several GB). I want to download it and then delete it.
There is nothing wrong with res.download
Probably the reason why res.download don't work for you is b/c you are using AJAX to fetch the resource, Do a regular navigation. Or if it requires some post data and another method: create a form and submit.
I am trying to download a video file from a friends server. I manage to download the subtitle file doing this:
var file = fs.createWriteStream('sub.srt');
var request = https.get(subtitleTrackURL, function(response) {
response.pipe(file);
});
But when I try to get the video file using the same method all I get is an empty file:
var file = fs.createWriteStream('video.mp4');
var request = https.get(videoFileURL, function(response) {
response.pipe(file);
});
The video "downloads" instantly (more like not at all since the file is empty) but it is supposed to be about 400MB and should as such take a bit of time.
I am thinking that there must be some encoding or content type that I have to provide for the video file request (the subtitle file is after all just text), but I can't figure out how or what I need to provide. Google was surprisingly unhelpful in how to download a video file using node. So if I should be using something other than https, I am open to suggestions.
I used this link to make an original link into a download link:
https://milanaryal.com/2015/direct-linking-to-your-files-on-dropbox-google-drive-and-onedrive/
Now how do I actually use that download link to download the file in JavaScript? I want to do something like:
link = 'https://drive.google.com/uc?export=download&id=FILE_ID';
let x = download(link); //now x is the download file
I looked it up and it seems like there are ways of doing this with HTML/jQuery, but I am not using those because I am working on the server side with Nodejs. I am doing this download thing because I want to check if the file is a pdf or text, parse the text, and then search through it using Elasticsearch.
It's easiest to use a module such as Request to do a HTTP get from a node script.
For example:
var request = require('request');
request.get('https://drive.google.com/uc?export=download&id=FILE_ID',
function(err, res, body){
if(err) return console.log(err);
console.log(body);
});
Once the file has downloaded, the callback function is run with the downloaded file in the body variable
If you only want to download the file, open it, search for data and delete it, you can easily edit this code snippet: https://stackoverflow.com/a/11944984/642977
How can I upload a file in azure if I only have the URL of the file to upload. In this case, i 'm using Dropbox file chooser which selects file from dropbox and returns its url path.
eq
https://www.dropbox.com/s/o9myet72y19iaan/Getting%20Started.pdf
Now we need the file to be stored in Windows Azure blob. What is the easiest way to do this without downloading the file first.
I'm planning to use a asp.net web api for the uploading of file to azure blob.
At first, I thought it should be quite straight forward as Azure Blob Storage support copying blobs from external URL however I don't think this would work in case of Dropbox files. I just tried it and got an error even though.
The link you mentioned above is not the direct link to the file. It's a link to a page on Dropbox's website from where you can download a file. This is obviously you don't want. Here's an alternate solution which you can try:
Replace www.dropbox.com in your URL with dl.dropboxusercontent.com (based on #smarx's comments below) and use that URL in the following code:
First you would need to append dl=1 to your request URL as query string. So your Dropbox URL would be https://www.dropbox.com/s/o9myet72y19iaan/Getting%20Started.pdf?dl=1. dl query string parameter indicates the file needs to be downloaded.
Next, using HTTPWebRequest try accessing this URL. Dropbox will respond back with another link and 302 status code. This link would be something like https://dl.dropboxusercontent.com/s/o9myet72y19iaan/Getting%20Started.pdf?token_hash=<tokenhash>.
Use this link in the code below to copy file. This would work.
CloudStorageAccount acc = new CloudStorageAccount(new StorageCredentials("account", "key"), false);
var client = acc.CreateCloudBlobClient();
var container = client.GetContainerReference("container-name");
container.CreateIfNotExists();
var blob = container.GetBlockBlobReference("dropbox-file-name");
blob.StartCopyFromBlob(new Uri("dropbox URL with dl.dropboxusercontent.com"));
Console.WriteLine("Copy request accepted");
Console.WriteLine("Now checking for copy state");
bool continueLoop = true;
do
{
blob.FetchAttributes();
var copyState = blob.CopyState;
switch (copyState.Status)
{
case CopyStatus.Pending:
Console.WriteLine("Copy is still pending. Will check status again after 1 second.");
System.Threading.Thread.Sleep(1000);//Copy is still pending...check after 1 second
break;
default:
Console.WriteLine("Terminating process with copy state = " + copyState.Status);
continueLoop = false;
break;
}
}
while (continueLoop);
Console.WriteLine("Press any key to continue.");