I am trying to upload file to s3, before that I am altering the name of the file. Now I am accepting 2 files from request form-data object, renaming the filename, and uploading the file to s3. And end of the task I need to return the renamed file list which is uploaded successfully.
I am using S3.upload() function. But the problem is, the variable which is assigned as empty array initially, that will contain the renamed file list. But the array is returning empty response. The s3.upload() is taking much time. is there any probable solution where I can store the file name if upload is successful and return those names in response.
Please help me to fix this. The code looks like this,
if (formObject.files.document && formObject.files.document.length > 0) {
const circleCode = formObject.fields.circleCode[0];
let collectedKeysFromAwsResponse = [];
formObject.files.document.forEach(e => {
const extractFileExtension = ".pdf";
if (_.has(FILE_EXTENSIONS_INCLUDED, _.lowerCase(extractFileExtension))) {
console.log(e);
//change the filename
const originalFileNameCleaned = "cleaning name logic";
const _id = mongoose.Types.ObjectId();
const s3FileName = "s3-filename-convension;
console.log(e.path, "", s3FileName);
const awsResponse = new File().uploadFileOnS3(e.path, s3FileName);
if(e.hasOwnProperty('ETag')) {
collectedKeysFromAwsResponse.push(awsResponse.key.split("/")[1])
}
}
});
};
use await s3.upload(params).promise(); is the solution.
Use the latest code - which is AWS SDK for JavaScript V3. Here is the code you should be using
// Import required AWS SDK clients and commands for Node.js.
import { PutObjectCommand } from "#aws-sdk/client-s3";
import { s3Client } from "./libs/s3Client.js"; // Helper function that creates Amazon S3 service client module.
import {path} from "path";
import {fs} from "fs";
const file = "OBJECT_PATH_AND_NAME"; // Path to and name of object. For example '../myFiles/index.js'.
const fileStream = fs.createReadStream(file);
// Set the parameters
export const uploadParams = {
Bucket: "BUCKET_NAME",
// Add the required 'Key' parameter using the 'path' module.
Key: path.basename(file),
// Add the required 'Body' parameter
Body: fileStream,
};
// Upload file to specified bucket.
export const run = async () => {
try {
const data = await s3Client.send(new PutObjectCommand(uploadParams));
console.log("Success", data);
return data; // For unit tests.
} catch (err) {
console.log("Error", err);
}
};
run();
More details can be found in the AWS JavaScript V3 DEV Guide.
Related
How do I get uploaded image in next.js API route and save it on public folder? I have front end ready. I'm uploading images to an endpoint using plain JavaScript.
here is the onSubmit function for uploading images. Suggest me if I'm doing it wrong here. The main question is how do I retrieve it?
const onSubmit=async(e)=>{
e.preventDefault();
const fd=new FormData()
fd.append('myfile',image.name)
let res=await fetch(`http://localhost:3000/api/upload`,{
method: 'POST',
headers: {
"Content-Type": "image/jpeg",
},
body: fd,
})
let response=await res.json();
one more bonus question, it's surely not a good idea to save the uploaded images on public folder. I have save it somewhere on the cloud.
This is the endpoint code I used for uploading image in nextjs, it requires some additional packages I will list them bellow also.
next-connect
multer
uuid
import nextConnect from "next-connect";
import multer from "multer";
import { v4 as uuidv4 } from "uuid";
let filename = uuidv4() + "-" + new Date().getTime();
const upload = multer({
storage: multer.diskStorage({
destination: "./public/uploads/profiles", // destination folder
filename: (req, file, cb) => cb(null, getFileName(file)),
}),
});
const getFileName = (file) => {
filename +=
"." +
file.originalname.substring(
file.originalname.lastIndexOf(".") + 1,
file.originalname.length
);
return filename;
};
const apiRoute = nextConnect({
onError(error, req, res) {
res
.status(501)
.json({ error: `Sorry something Happened! ${error.message}` });
},
onNoMatch(req, res) {
res.status(405).json({ error: `Method '${req.method}' Not Allowed` });
},
});
apiRoute.use(upload.array("file")); // attribute name you are sending the file by
apiRoute.post((req, res) => {
res.status(200).json({ data: `/uploads/profiles/${filename}` }); // response
});
export default apiRoute;
export const config = {
api: {
bodyParser: false, // Disallow body parsing, consume as stream
},
};
no Need to use any packages to handle file uploading you can use base64 to convert file to string and return it back to file by using "fs" module
why This way is beterr then using formData ?
because you duleing with normal post request where you can send any kind of data with it and use body parsere .
converting
const toBase64 = (file: File) => new Promise((resolve, reject) => {
const reader = new FileReader();
reader.readAsDataURL(file);
reader.onload = () => resolve(reader.result);
reader.onerror = error => reject(error);
}
send a post request to server
const base64: string = await toBase64(file) as string;
const fileData = { base64, fileName: file.name };
const result = await api.post("/foo", fileData, name: "Salih", massage: "Hello World"})
converting base64 to file in server
function base64ToFile(file: { base64: string, fileName: string }) {
const fileContents = file.base64.replace(/^data:image\/png;base64,/, "");
fs.mkdirSync("./public/uploads", { recursive: true });
const fileName = `./public/uploads/${Date.now().toString() + file.fileName}`
fs.writeFile(fileName, fileContents, 'base64', function (err) { console.log(err) });
}
I suggest the popular and lightweight formidable library:
# install
yarn add formidable#v3 #types/formidable
// pages/api/file-upload.ts
import fs from "fs";
import path from "path";
import { File } from "formidable";
// Important for NextJS!
export const config = {
api: {
bodyParser: false,
},
};
export default async function handler(
req: NextApiRequest,
res: NextApiResponse<string>
) {
try {
// Parse request with formidable
const { fields, files } = await parseFormAsync(req);
// Files are always arrays (formidable v3+)
const myfile = (files["myfile"] as any as File[])[0];
// Save file in the public folder
saveFile(myfile, "./public/uploads");
// Return success
res.status(200).json("success!");
} catch (e) {
return res.status(500).json(e);
}
}
function saveFile(file: File, publicFolder: string): void {
const fileExt = path.extname(file.originalFilename || "");
fs.renameSync(file.filepath, `${publicFolder}/${file.newFilename}${fileExt}`);
}
// ./helpers/formidable.ts
import type { NextApiRequest } from "next";
import formidable from "formidable";
export type FormidableParseReturn = {
fields: formidable.Fields;
files: formidable.Files;
};
export async function parseFormAsync(
req: NextApiRequest,
formidableOptions?: formidable.Options
): Promise<FormidableParseReturn> {
const form = formidable(formidableOptions);
return await new Promise<FormidableParseReturn>((resolve, reject) => {
form.parse(req, async (err, fields, files) => {
if (err) {
reject(err);
}
resolve({ fields, files });
});
});
}
Bonus question
one more bonus question, it's surely not a good idea to save the uploaded images on public folder. I have save it somewhere on the cloud.
S3 and other cloud services
You can save on cloud services with Formidable.
See the official examples: https://github.com/node-formidable/formidable/blob/master/examples/store-files-on-s3.js
But you don't need to use cloud storage to protect private uploads. You can store them locally.
Working with private uploads locally
Saving:
Store the uploads in a non-public folder;
Ex. /private-uploads/{logged_user_id}/;
Reading:
Create an API page to fetch the file
Ex. https://.../uploads/{filename}
Fail if the file doesn't belong to the authenticated user;
Send the file as the response;
Security:
With the above folder scheme, hackers can use .. and similar on the filename to obtain unauthorized access;
Sanitize the filename having this in mind (ex. only allow alphanumeric characters);
Alternatively, use a database table to control ownership instead of a folder scheme;
I have managed to use fleek to update IPFS via straight javascript. I am now trying to add this functionality to a clean install of a svelteKit app. I think I am having trouble with the syntax around imports, but am not sure what I am doing wrong. When I click the button on the index.svelte I get the following error
Uncaught ReferenceError: require is not defined
uploadIPFS upload.js:3
listen index.mjs:412..........(I truncated the error here)
A few thoughts
I am wondering if it could be working in javascript because it is being called in node (running on the server) but running on the client in svelte?
More Details
The index.svelte file looks like this
<script>
import {uploadIPFS} from '../IPFS/upload'
</script>
<button on:click={uploadIPFS}>
upload to ipfs
</button>
the upload.js file looks like this
export const uploadIPFS = () => {
const fleek = require('#fleekhq/fleek-storage-js');
const apiKey = 'cZsQh9XV5+6Nd1+Bou4OuA==';
const apiSecret = '';
const data = 'pauls test load';
const testFunctionUpload = async (data) => {
const date = new Date();
const timestamp = date.getTime();
const input = {
apiKey,
apiSecret,
key: `file-${timestamp}`,
data
};
try {
const result = await fleek.upload(input);
console.log(result);
} catch (e) {
console.log('error', e);
}
};
testFunctionUpload(data);
};
I have also tried using the other import syntax and when I do I get the following error
500
global is not defined....
import with the other syntax is
import fleekStorage from '#fleekhq/fleek-storage-js';
function uploadIPFS() {
console.log('fleekStorage',fleekStorage)
};
export default uploadIPFS;
*I erased the api secret in the code above. In future I will store these in a .env file.
Even more details (if you need them)
The file below will update IPFS and runs via the command
npm run upload
That file is below. For my version that I used in svelte I simplified the file by removing all the file management and just loading a variable instead of a file (as in the example below)
const fs = require('fs');
const path = require('path');
const fleek = require('#fleekhq/fleek-storage-js');
require('dotenv').config()
const apiKey = process.env.FLEEK_API_KEY;
const apiSecret = process.env.FLEEK_API_SECRET;
const testFunctionUpload = async (data) => {
const date = new Date();
const timestamp = date.getTime();
const input = {
apiKey,
apiSecret,
key: `file-${timestamp}`,
data,
};
try {
const result = await fleek.upload(input);
console.log(result);
} catch(e) {
console.log('error', e);
}
}
// File management not used a my svelte version to keep it simple
const filePath = path.join(__dirname, 'README.md');
fs.readFile(filePath, (err, data) => {
if(!err) {
testFunctionUpload(data);
}
})
I am trying so hard to upload one image from cloud functions
I am sending an image from the web to the cloud function using onRequest. I am sending a base64 string and the fileName. Now I was following different tutorials on the internet and couldn't seem to solve my problem.
Here is my code. I think I am doing something wrong with the service account json. Although i generated the json file and used it but still it didn't work.
I get the error of The caller does not have permission at Gaxios._request when i don't use service account json
And when i do use serviceAccount.json then i get this error The "path" argument must be of type string. Received an instance of Object which is from file.createWriteStream() i think
Anyway here is the code can anyone please help me with this
The projectId that I am using is shown in the picture below
const functions = require("firebase-functions");
const admin = require("firebase-admin");
const projectId = functions.config().apikeys.projectid; // In the picture below
const stream = require("stream");
const cors = require("cors")({ origin: true });
const { Storage } = require("#google-cloud/storage");
// Enable Storage
const storage = new Storage({
projectId: projectId, // I did use serviceAccount json here but that wasn't working
});
// With serviceAccount.json code
// const storage = new Storage({
// projectId: projectId,
// keyFilename: serviceAccount,
// });
// This is giving the error of: The "path" argument must be of type string. Received an instance of Object
exports.storeUserProfileImage = functions.https.onRequest((req, res) => {
cors(req, res, async () => {
try {
const bucket = storage.bucket(`gs://${projectId}.appspot.com`);
let pictureURL;
const image = req.body.image;
const userId = req.body.userId;
const fileName = req.body.fileName;
const mimeType = image.match(
/data:([a-zA-Z0-9]+\/[a-zA-Z0-9-.+]+).*,.*/
)[1];
//trim off the part of the payload that is not part of the base64 string
const base64EncodedImageString = image.replace(
/^data:image\/\w+;base64,/,
""
);
const imageBuffer = Buffer.from(base64EncodedImageString, "base64");
const bufferStream = new stream.PassThrough();
bufferStream.end(imageBuffer);
// Define file and fileName
const file = bucket.file("images/" + fileName);
bufferStream
.pipe(
file.createWriteStream({
metadata: {
contentType: mimeType,
},
public: true,
validation: "md5",
})
)
.on("error", function (err) {
console.log("error from image upload", err.message);
})
.on("finish", function () {
// The file upload is complete.
console.log("Image uploaded");
file
.getSignedUrl({
action: "read",
expires: "03-09-2491",
})
.then((signedUrls) => {
// signedUrls[0] contains the file's public URL
console.log("Signed urls", signedUrls[0]);
pictureURL = signedUrls[0];
});
});
console.log("image url", pictureURL);
res.status(200).send(pictureURL);
} catch (e) {
console.log(e);
return { success: false, error: e };
}
});
});
const storage = new Storage({
projectId: projectId
keyFilename: "" // <-- Path to a .json, .pem, or .p12 key file
});
keyFilename accepts path to where your service account is stored and the credentials themselves.
folder
|-index.js
|-credentials
|-serviceAccountKey.json
If your directory structure looks like about then the path should be like this:
const storage = new Storage({
projectId: projectId
keyFilename: "./credentials/serviceAccountKey.json"
});
Do note that if you are using Cloud functions then the SDK will use Application Default Credentials so you don't have to pass those params. Simply initialize as shown below:
const storage = new Storage()
So first of all I didn't give any serviceaccounts because I am using the firebase cloud functions as #Dharmaraj said in his answer
Secondly, this was a permission problem in the google cloud platform which can be solved by going through the following steps
Go to your project's Cloud Console (https://console.cloud.google.com/) > IAM & admin > IAM, Find the App Engine default service account then click on the pencil at far left > Click on add role > In the filter field enter Service Account Token Creator and click on it save and you are good to go
Found this solution from here
https://github.com/firebase/functions-samples/issues/782
I have been trying to upload a file to Firebase storage using a callable firebase cloud function.
All i am doing is fetching an image from an URL using axios and trying to upload to storage.
The problem i am facing is, I don't know how to save the response from axios and upload it to storage.
First , how to save the received file in the temp directory that os.tmpdir() creates.
Then how to upload it into storage.
Here i am receiving the data as arraybuffer and then converting it to Blob and trying to upload it.
Here is my code. I have been missing a major part i think.
If there is a better way, please recommend me. Ive been looking through a lot of documentation, and landed up with no clear solution. Please guide. Thanks in advance.
const bucket = admin.storage().bucket();
const path = require('path');
const os = require('os');
const fs = require('fs');
module.exports = functions.https.onCall((data, context) => {
try {
return new Promise((resolve, reject) => {
const {
imageFiles,
companyPIN,
projectId
} = data;
const filename = imageFiles[0].replace(/^.*[\\\/]/, '');
const filePath = `ProjectPlans/${companyPIN}/${projectId}/images/${filename}`; // Path i am trying to upload in FIrebase storage
const tempFilePath = path.join(os.tmpdir(), filename);
const metadata = {
contentType: 'application/image'
};
axios
.get(imageFiles[0], { // URL for the image
responseType: 'arraybuffer',
headers: {
accept: 'application/image'
}
})
.then(response => {
console.log(response);
const blobObj = new Blob([response.data], {
type: 'application/image'
});
return blobObj;
})
.then(async blobObj => {
return bucket.upload(blobObj, {
destination: tempFilePath // Here i am wrong.. How to set the path of downloaded blob file
});
}).then(buffer => {
resolve({ result: 'success' });
})
.catch(ex => {
console.error(ex);
});
});
} catch (error) {
// unknown: 500 Internal Server Error
throw new functions.https.HttpsError('unknown', 'Unknown error occurred. Contact the administrator.');
}
});
I'd take a slightly different approach and avoid using the local filesystem at all, since its just tmpfs and will cost you memory that your function is using anyway to hold the buffer/blob, so its simpler to just avoid it and write directly from that buffer to GCS using the save method on the GCS file object.
Here's an example. I've simplified out a lot of your setup, and I am using an http function instead of a callable. Likewise, I'm using a public stackoverflow image and not your original urls. In any case, you should be able to use this template to modify back to what you need (e.g. change the prototype and remove the http response and replace it with the return value you need):
const functions = require('firebase-functions');
const axios = require('axios');
const admin = require('firebase-admin');
admin.initializeApp();
exports.doIt = functions.https.onRequest((request, response) => {
const bucket = admin.storage().bucket();
const IMAGE_URL = 'https://cdn.sstatic.net/Sites/stackoverflow/company/img/logos/so/so-logo.svg';
const MIME_TYPE = 'image/svg+xml';
return axios.get(IMAGE_URL, { // URL for the image
responseType: 'arraybuffer',
headers: {
accept: MIME_TYPE
}
}).then(response => {
console.log(response); // only to show we got the data for debugging
const destinationFile = bucket.file('my-stackoverflow-logo.svg');
return destinationFile.save(response.data).then(() => { // note: defaults to resumable upload
return destinationFile.setMetadata({ contentType: MIME_TYPE });
});
}).then(() => { response.send('ok'); })
.catch((err) => { console.log(err); })
});
As a commenter noted, in the above example the axios request itself makes an external network access, and you will need to be on the Blaze or Flame plan for that. However, that alone doesn't appear to be your current problem.
Likewise, this also defaults to using a resumable upload, which the documentation does not recommend when you are doing large numbers of small (<10MB files) as there is some overhead.
You asked how this might be used to download multiple files. Here is one approach. First, lets assume you have a function that returns a promise that downloads a single file given its filename (I've abridged this from the above but its basically identical except for the change of INPUT_URL to filename -- note that it does not return a final result such as response.send(), and there's sort of an implicit assumption all the files are the same MIME_TYPE):
function downloadOneFile(filename) {
const bucket = admin.storage().bucket();
const MIME_TYPE = 'image/svg+xml';
return axios.get(filename, ...)
.then(response => {
const destinationFile = ...
});
}
Then, you just need to iteratively build a promise chain from the list of files. Lets say they are in imageUrls. Once built, return the entire chain:
let finalPromise = Promise.resolve();
imageUrls.forEach((item) => { finalPromise = finalPromise.then(() => downloadOneFile(item)); });
// if needed, add a final .then() section for the actual function result
return finalPromise.catch((err) => { console.log(err) });
Note that you could also build an array of the promises and pass them to Promise.all() -- that would likely be faster as you would get some parallelism, but I wouldn't recommend that unless you are very sure all of the data will fit inside the memory of your function at once. Even with this approach, you need to make sure the downloads can all complete within your function's timeout.
I want to download an entire folder from Firebase storage. I can download single files using DownloadURL as follows, but it does not work for folders.
var storageRef = firebase.storage().ref();
// Create a reference to the file we want to download
var starsRef = storageRef.child(path);
// Get the download URL
starsRef.getDownloadURL().then(function(url) {
// Insert url into an <img> tag to "download"
ImageUrl = url;
console.log(ImageUrl);
}).catch(function(error) {
switch (error.code) {
case 'storage/object_not_found':
// File doesn't exist
break;
case 'storage/unauthorized':
// User doesn't have permission to access the object
break;
case 'storage/canceled':
// User canceled the upload
break;
case 'storage/unknown':
// Unknown error occurred, inspect the server response
break;
}
});
How to download entire folder from Firebase?
You can use gsutil to download the whole storage bucket
gsutil -m cp -R gs://<bucket_name> .
There is no API in Firebase Storage to download all files in a folder. You will have to download the files one by one, or create a zip file that contains all the files.
As Lahiru's answer shows it can be accomplished with gsutils, but that's a server-side operation - not something you'd run in your client-side application.
Related:
How to get a list of all files in Cloud Storage in a Firebase app?
Command gustil for Windows !!!
gsutil cp -r gs://<bucket_name>.appspot.com/OBJECT_NAME "D:\path"
Use Cloud tools for PowerShell
REF for install windows >> https://cloud.google.com/storage/docs/gsutil_install
You can download the folder by creating a zip file of it.
Here is a sample function:
import JSZip from 'jszip';
import { saveAs } from 'file-saver';
import {
getStorage,
listAll,
ref,
getDownloadURL,
getMetadata,
} from 'firebase/storage';
import { auth } from '../../Firebase';
export const downloadFolderAsZip = async () => {
const jszip = new JSZip();
const storage = getStorage();
const folderRef = ref(
storage,
'images'
);
const folder = await listAll(folderRef);
const promises = folder.items
.map(async (item) => {
const file = await getMetadata(item);
const fileRef = ref(storage, item.fullPath);
const fileBlob = await getDownloadURL(fileRef).then((url) => {
return fetch(url).then((response) => response.blob());
});
jszip.file(file.name, fileBlob);
})
.reduce((acc, curr) => acc.then(() => curr), Promise.resolve());
await promises;
const blob = await jszip.generateAsync({ type: 'blob' });
saveAs(blob, 'download.zip');
};
For a recursive solution that includes subfolders in the zip file, see the following sample. You'll instantiate a jszip object, await promises from a function that zips files and traverses the directories, then save the zip. If the content is a file ("item"), it is zipped into the jszip object. If it is a folder ("prefix"), the function is called again with a new subpath, passing in the same jszip object. For further improvement, you may want to get contents with list and pagination if your contents are too many for listAll, since listAll limits retrievals.
import JSZip from 'jszip';
import { saveAs } from 'file-saver';
import {
getStorage, ref, getBlob, listAll,
} from "firebase/storage";
const addFilesFromDirectoryToZip = async (directoryPath = "", zip) => {
const storage = getStorage();
const directoryContentsRef = ref(
storage,
directoryPath
);
const directoryContents = await listAll(directoryContentsRef);
for (const file of directoryContents.items) {
const fileRef = ref(storage, file.fullPath);
const fileBlob = await getBlob(fileRef)
zip.file(file.fullPath, fileBlob);
}
for (const folder of directoryContents.prefixes) {
await addFilesFromDirectoryToZip(folder.fullPath, zip);
};
};
export const downloadFolderAsZip = async (directoryPath = "") => {
const zip = new JSZip();
await addFilesFromDirectoryToZip(directoryPath, zip);
const blob = await zip.generateAsync({ type: "blob" });
const name = directoryPath.split('/').pop();
saveAs(blob, name);
};