I'm starting to learn Django and one of the projects I have chosen for myself is one that has an HTML front page, in which the user inputs some search text and then that calls to an API using Python and Pytrends, then its saves that information as a CSV file.
I'm having a hard time with the next part which is importing that CSV file into another HTML page, where I would like to use my chart.js file to turn it into a graph. I can't figure out how to open up the file and run it inside of javascript. I have tried a static load but it's not opening up the file, just the path to the file. Please let me know if you need to see any other files or know anything else.
async function getData() {
const data = '{% static "interest.csv" %}';
console.log(data.text());
const table = data.split("\n").slice(1);
table.forEach((row) => {
const columns = row.split(",");
const year = columns[0];
const interest = columns[1];
xlabel.push(year);
console.log(year, interest);
});
Related
I have a process where a client uploads a document. This document can be in the form of a PDF, JPG or PNG file only and it should be reuploaded once a year (it is an errors and omissions insurance policy).
I am saving this file in a container.
For deleting files from anywhere at the application, I have this function (Node):
deleteFromBlob = async function (account, accountKey, containerName, blobFolder, blobName) {
try {
const {
BlobServiceClient,
StorageSharedKeyCredential
} = require("#azure/storage-blob");
const sharedKeyCredential = new StorageSharedKeyCredential(account, accountKey);
const blobServiceClient = new BlobServiceClient(
`https://${account}.blob.core.windows.net`,
sharedKeyCredential
);
const containerClient = blobServiceClient.getContainerClient(containerName);
const blockBlobClient = containerClient.getBlockBlobClient(blobFolder + '/' + blobName);
const uploadblobResponse = await blockBlobClient.deleteIfExists()
return true
}
catch(e) {
return false
}
}
And this works perfect when I know the file name and extension I want to delete, like "2448.pdf":
let deleteFile = await utils.deleteFromBlob(account, accountKey, "agents", "/eopolicies/", userData.agentid.toString() + ".pdf" )
But the problem Im facing is that the function above is to delete a file I know exists; for example, if the agent ID is 2448 and he uploads "policy.pdf" I save it as "2448.pdf" for easy file identification.
The problem Im facing is if the agent uploaded a .PNG last year. a .DOC a year before, and a .PDF now. If that's the case, I want to delete 2448.* and keep only the latest version of the document.
So I tried changing my function to
let deleteFile = await utils.deleteFromBlob(account, accountKey, "agents", "/eopolicies/", userData.agentid.toString() + ".*" )
And of course it is not working...
I tried to find a solution and all I found is one to list the content of a folder, then loop it and delete the specific file I want; but that will not work for me since there are 37,000 EO policies on that folder.
Is there a way to delete files with a specific name, and whatever extension?
Thanks.
I've never tried using a wildcard on the extension side of the file name. However, I would iterate through the files in the directory and find the one that contains the specific string you are looking for. Get it's index, and delete from there.
I want to use fs.WriteFile in my JS project. I am building an algorithm that outputs random data and I want to give the user the opportunity to save the data as a txt file. I have been able to implement fs.WriteFile in my project, but I have a couple of questions on how to proceed next as the function remains somewhat unclear.
How do I specify that I want to include the contents of various vars? Is it as simple as data = let1 + let2 + let3 and all of the data will be included?
can I add the current date and time in the .txt file name? If so, how?
How do I tell writeFile to save the contents to a .txt file and open a download blob so that people can specify their own download locations?
Thanks in advance!
I've tried looking at basic documentation but its mainly the same: a template using a simple string that saves into the same directory, which is what I don't want.
For you first question, you are correct. You can just combine different string variables into a larger string variable. See the documentation for string concatenation for more information.
For your second question, yes you can. You can get the current date and time with new Date() and turn it into a variety of formats. For file names, using mydate.toISOString() will probably be the most clean.
Here's an example of both of these in practice:
import fs from 'fs';
// Here's some data that we want to put in the file.
const name = "Bob";
const age = 43;
// Create the data we want to put in our file.
const data = name + '\n' + age;
// Let's grab the date and use it as part of our file name.
const date = new Date();
const fileName = `${date.toISOString()}.txt`;
// Call fs.writeFile to put the data in the file.
fs.writeFile(fileName, data, () => {
console.log(`Wrote data to ${fileName}.`);
});
Your third question is more complicated and probably worth a separate post. fs.writeFile can't do this for you. You'll have to come up with some mechanism for the user to enter their own file name and build off of that.
Edit:
To address your question in the comments, you might be a little confused with how NodeJS works. NodeJS runs on the server and doesn't have any way to deal with buttons or UIs by default like browser JavaScript does. It might be helpful to look at the differences between the two. So you won't be able to save it to the downloads folder on a button click.
With that said, we can save the file to the user's Downloads folder with the same script I posted above by adding the path to the Downloads folder to the beginning of the file name.
Here's the code above adjusted to do that:
import fs from 'fs';
import os from 'os'; // NEW
import path from 'path'; // NEW
const name = "Bob";
const age = 43;
const data = name + '\n' + age;
const date = new Date();
const fileName = `${date.toISOString()}.txt`;
// Get the user's home directory.
const homedir = os.homedir();
// Append the Downloads directory and fileName to the user's home directory.
const fullPath = path.join(homedir, 'Downloads', fileName);
// Use fullPath here instead of fileName.
fs.writeFile(fullPath, data, () => {
console.log(`Wrote data to ${fileName}.`);
});
Over the years on snapchat I have saved lots of photos that I would like to retrieve now, The problem is they do not make it easy to export, but luckily if you go online you can request all the data (thats great)
I can see all my photos download link and using the local HTML file if I click download it starts downloading.
Here's where the tricky part is, I have around 15,000 downloads I need to do and manually clicking each individual one will take ages, I've tried extracting all of the links through the download button and this creates lots of Urls (Great) but the problem is, if you past the url into the browser then ("Error: HTTP method GET is not supported by this URL") appears.
I've tried a multitude of different chrome extensions and none of them show the actually download, just the HTML which is on the left-hand side.
The download button is a clickable link that just starts the download in the tab. It belongs under Href A
I'm trying to figure out what the best way of bulk downloading each of these individual files is.
So, I just watched their code by downloading my own memories. They use a custom JavaScript function to download your data (a POST request with ID's in the body).
You can replicate this request, but you can also just use their method.
Open your console and use downloadMemories(<url>)
Or if you don't have the urls you can retrieve them yourself:
var links = document.getElementsByTagName("table")[0].getElementsByTagName("a");
eval(links[0].href);
UPDATE
I made a script for this:
https://github.com/ToTheMax/Snapchat-All-Memories-Downloader
Using the .json file you can download them one by one with python:
req = requests.post(url, allow_redirects=True)
response = req.text
file = requests.get(response)
Then get the correct extension and the date:
day = date.split(" ")[0]
time = date.split(" ")[1].replace(':', '-')
filename = f'memories/{day}_{time}.mp4' if type == 'VIDEO' else f'memories/{day}_{time}.jpg'
And then write it to file:
with open(filename, 'wb') as f:
f.write(file.content)
I've made a bot to download all memories.
You can download it here
It doesn't require any additional installation, just place the memories_history.json file in the same directory and run it. It skips the files that have already been downloaded.
Short answer
Download a desktop application that automates this process.
Visit downloadmysnapchatmemories.com to download the app. You can watch this tutorial guiding you through the entire process.
In short, the app reads the memories_history.json file provided by Snapchat and downloads each of the memories to your computer.
App source code
Long answer (How the app described above works)
We can iterate over each of the memories within the memories_history.json file found in your data download from Snapchat.
For each memory, we make a POST request to the URL stored as the memories Download Link. The response will be a URL to the file itself.
Then, we can make a GET request to the returned URL to retrieve the file.
Example
Here is a simplified example of fetching and downloading a single memory using NodeJS:
Let's say we have the following memory stored in fakeMemory.json:
{
"Date": "2022-01-26 12:00:00 UTC",
"Media Type": "Image",
"Download Link": "https://app.snapchat.com/..."
}
We can do the following:
// import required libraries
const fetch = require('node-fetch'); // Needed for making fetch requests
const fs = require('fs'); // Needed for writing to filesystem
const memory = JSON.parse(fs.readFileSync('fakeMemory.json'));
const response = await fetch(memory['Download Link'], { method: 'POST' });
const url = await response.text(); // returns URL to file
// We can now use the `url` to download the file.
const download = await fetch(url, { method: 'GET' });
const fileName = 'memory.jpg'; // file name we want this saved as
const fileData = download.body; // contents of the file
// Write the contents of the file to this computer using Node's file system
const fileStream = fs.createWriteStream(fileName);
fileData.pipe(fileStream);
fileStream.on('finish', () => {
console.log('memory successfully downloaded as memory.jpg');
});
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 2 years ago.
Improve this question
So I am doing a personal project for my home administration. I would like to automatically when opening my spread sheet (adminSheet.gs) to import .csv files. It is possible to do this manually, but I would like to automate this as I have to import 30+ .csv files, and more coming. I have enabled google apps script (GAS). However GAS is new for me, I am a bit comfortable with Javascript, however it has been a while since I was in college.. I hope you can help me out!
Here are the facts.
I have multiple .csv files, all with a different name. They are located in the same folder called CSVfolder. This folder is within the main folder MAINfolder in Google Drive. They are seperated by the delimiter ; (semicolon). The CSV files have the same data structure and the first row of each .csv file is the same. This first row has been put already in the sheet adminSheet.gs, serving as the header row called HEADER. So the first row needs to be deleted of each .csv file when imported.
I will upload new .csv files in the CSVfolder. These file name follow the following naming structure. ID_01-09-2020_30-09-2020.csv or ID_01-10-2020_31-10-2020.csv. ID followed by a changing date. (the .csv data will be for that period of time. overlapping or duplicated data will not occur, as I will choose the dataset manually). However this would make it of course very neat.
When I open the adminSheet.gs (located in the MAINfolder), it needs to automatically recognise any new .csv file and merge that into the existing adminSheet.gs with all the other (previously imported) .csv data. The new .csv file should merged below the fixed header row HEADER and above the previously imported .csv file. (this can also be explained as: between the HEADER and the data already in the adminSheet.gs) (this can also be explained as: the new data from the .csv should come above the old data already in the adminSheet.gs).
What Google Apps Script could do this?
Here's how the following code works:
gets data from the csv file as an array
inserts a number of rows after the header row
sets the array in the sheet
moves the csv files to an archive folder after importing them so they are not imported again
then moves to the next file until there are no more files to import
You will need to create an archive folder for the csv files to be stored in after they have been imported. I generally make one inside the folder containing the csv folders for this purpose.
In the code, you will need to provide the folder IDs for the archive folder and the root folder (folder that contains your CSV folder) and the name of the destination sheet in your spreadsheet. These can all be set in the first four variable in the code. (I already set csvFolderName to 'CSVfolder' per the information you provided above.
After saving this code, create a trigger on it to run on open.
Here is the code:
function importCSVFiles()
{
var ss = SpreadsheetApp.getActiveSpreadsheet();
const csvFolderName = 'CSVfolder'; // Folder containing csv files and the Archive folder
const rootFolderId = 'id_of_folder_containing_CSVfolder'; // Root folder
const archiveFolderId = 'id_of_archiveFolder'; // Archive Folder
const destinationSheetName = 'name_of_destination_sheet_in_spreadsheet'; // destination sheet in spreadsheet
const root = DriveApp.getFolderById(rootFolderId);
const archiveFolder = DriveApp.getFolderById(archiveFolderId);
const fileName =/ *.csv/;
var rootFolders = root.getFolders();
const sheet = ss.getSheetByName(destinationSheetName);
var csvArray = [];
while (rootFolders.hasNext())
{
var folder = rootFolders.next ();
Logger.log('folder.getName()',folder.getName());
if (folder.getName() === csvFolderName)
{
importCSV(folder);
}
else
{
var csvFolder = folder.getFoldersByName(csvFolderName);
while (csvFolder.hasNext())
{
importCSV(csvFolder.next());
}
}
}
function importCSV(folder)
{
var files = folder.getFiles();
while (files.hasNext())
{
var file = files.next();
Logger.log('file.getName()',file.getName());
csvArray = file.getBlob().getDataAsString("UTF-8");
csvArray = Utilities.parseCsv(csvArray);
csvArray.shift();
sheet.insertRowsAfter(1,csvArray.length);
sheet.getRange (2,1,csvArray.length,csvArray[0].length).setValues(csvArray);
file.moveTo(archiveFolder);
}
}
}
If it seems like you are having issues with the delimiter, you may need to set it specifically as a custom delimiter by substituting the above line of code csvArray = Utilities.parseCsv(csvArray); for csvArray = Utilities.parseCsv(csvArray,';');
I have written a script within a gsheet that can create a gdoc based on the information in a row in the spreadsheet.
I am then trying to create a PDF of the gdoc that has been generated by the script in the gsheet. I am trying to make this all one seamless function where the gdoc is created with the pertinent information and then a pdf is automatically created as well. However, the PDF generated is always just a blank page.
I am able to add a script to a gdoc that will function correctly and create a pdf version of itself (or another gdoc), but when I run a function from a gsheet, it is always just the blank one-page pdf, Does anybody know why this is happening, or have a solution?
Here is an example of one of the simpler scripts I've tried and works in the gdoc script - obviously when running it from the gsheet I have to know the ID first and then open the gdoc from that.
function convertPDF(docId)
{
var doc = DocumentApp.getActiveDocument();
var docId = doc.getId();
var docFolder = DriveApp.getFileById(docId).getParents().next().getId();
var docblob = DocumentApp.getActiveDocument().getAs('application/pdf');
docblob.setName(doc.getName() + ".pdf");
var file = DriveApp.createFile(docblob);
var fileId = file.getId();
}
Thanks!
Chris