We Keep Coding

Firebase Storage upload always times out ("storage/retry-limit-exceeded" error)

I'm writing a script using the Firebase JavaScript (client, not admin) SDK to handle uploading a large quantity of images (~1.5 mb each) from an external download link to Google Firebase Storage, and I cannot successfully upload a single image.
Here's the piece of code that I'm testing out with a single image URL:
const firebaseConfig = {
// firebase config details
};
const app = initializeApp(firebaseConfig);
const storage = getStorage(app);
storage.maxOperationRetryTime = 120000;
storage.maxUploadRetryTime = 120000;
const addImageToFirebase = async (url) => {
const download = await fetch(url);
const blob = await download.blob();
const path = "test-folder/test-file.png";
const imageRef = ref(storage, path);
try {
await uploadBytes(imageRef, blob);
console.log('success!');
} catch (error) {
console.log(error);
}
}
const imageUrl = // download link for test url ;
addImageToFirebase(imageUrl);
When running the code, I get the following error in the terminal, no matter how long I set the maxOperationRetyTime or maxUploadRetryTime:
[StorageError [FirebaseError]: Firebase Storage: Max retry time for operation exceeded, please try again. (storage/retry-limit-exceeded)] {
code: 'storage/retry-limit-exceeded',
customData: { serverResponse: null },
_baseMessage: 'Firebase Storage: Max retry time for operation exceeded, please try again. (storage/retry-limit-exceeded)'
}
Does anyone know why the upload would fail to go through?

Uploading an mp3 to Firebase Storage with React Native Expo

I am attempting to upload an mp3 to firebase storage using expo and react native. So far I've got the file into firebase storage, but it's only 9bytes large, so I'm doing something wrong. I've attempted this with blob as shown below with no success.
Here is a screenshot of the firebase storage folder showing the file uploaded but not the data of said file:
Any help is greatly appreciated, I feel like I'm missing a step to actually upload the data along with the file.
export default function SongPicker() {
const [song, setSong] = useState(null);
//Get current user through authentication
const user = auth.currentUser;
const pickDocument = async () => {
let result = await DocumentPicker.getDocumentAsync({});
// Fetch the photo with it's local URI
const response = fetch(result.uri);
alert(result.uri);
console.log(result);
const file = new Blob(
[response.value], {
type: 'audio/mpeg'
});
console.log('do we see this?');
try {
//Create the file reference
const storage = getStorage();
const storageRef = ref(storage, `songs/${user.uid}/${result.name}`);
// Upload Blob file to Firebase
const snapshot = uploadBytes(storageRef, file, 'blob').then((snapshot) => {
console.log('Uploaded a song to firebase storage!');
});
setSong(result.uri);
} catch (error) {
console.log(error);
}
}

The fetch() returns a Promise so you should add an await for that as well.
const response = await fetch(result.uri);
Then try using blob() method on the Response:
const file = await response.blob()
The third param in uploadBytes should be upload metadata object but you can skip that here:
const snapshot = await uploadBytes(storageRef, file).

amazon s3.upload is taking time

I am trying to upload file to s3, before that I am altering the name of the file. Now I am accepting 2 files from request form-data object, renaming the filename, and uploading the file to s3. And end of the task I need to return the renamed file list which is uploaded successfully.
I am using S3.upload() function. But the problem is, the variable which is assigned as empty array initially, that will contain the renamed file list. But the array is returning empty response. The s3.upload() is taking much time. is there any probable solution where I can store the file name if upload is successful and return those names in response.
Please help me to fix this. The code looks like this,
if (formObject.files.document && formObject.files.document.length > 0) {
const circleCode = formObject.fields.circleCode[0];
let collectedKeysFromAwsResponse = [];
formObject.files.document.forEach(e => {
const extractFileExtension = ".pdf";
if (_.has(FILE_EXTENSIONS_INCLUDED, _.lowerCase(extractFileExtension))) {
console.log(e);
//change the filename
const originalFileNameCleaned = "cleaning name logic";
const _id = mongoose.Types.ObjectId();
const s3FileName = "s3-filename-convension;
console.log(e.path, "", s3FileName);
const awsResponse = new File().uploadFileOnS3(e.path, s3FileName);
if(e.hasOwnProperty('ETag')) {
collectedKeysFromAwsResponse.push(awsResponse.key.split("/")[1])
}
}
});
};

use await s3.upload(params).promise(); is the solution.

Use the latest code - which is AWS SDK for JavaScript V3. Here is the code you should be using
// Import required AWS SDK clients and commands for Node.js.
import { PutObjectCommand } from "#aws-sdk/client-s3";
import { s3Client } from "./libs/s3Client.js"; // Helper function that creates Amazon S3 service client module.
import {path} from "path";
import {fs} from "fs";
const file = "OBJECT_PATH_AND_NAME"; // Path to and name of object. For example '../myFiles/index.js'.
const fileStream = fs.createReadStream(file);
// Set the parameters
export const uploadParams = {
Bucket: "BUCKET_NAME",
// Add the required 'Key' parameter using the 'path' module.
Key: path.basename(file),
// Add the required 'Body' parameter
Body: fileStream,
};
// Upload file to specified bucket.
export const run = async () => {
try {
const data = await s3Client.send(new PutObjectCommand(uploadParams));
console.log("Success", data);
return data; // For unit tests.
} catch (err) {
console.log("Error", err);
}
};
run();
More details can be found in the AWS JavaScript V3 DEV Guide.

How to upload a file into Firebase Storage from a callable https cloud function

I have been trying to upload a file to Firebase storage using a callable firebase cloud function.
All i am doing is fetching an image from an URL using axios and trying to upload to storage.
The problem i am facing is, I don't know how to save the response from axios and upload it to storage.
First , how to save the received file in the temp directory that os.tmpdir() creates.
Then how to upload it into storage.
Here i am receiving the data as arraybuffer and then converting it to Blob and trying to upload it.
Here is my code. I have been missing a major part i think.
If there is a better way, please recommend me. Ive been looking through a lot of documentation, and landed up with no clear solution. Please guide. Thanks in advance.
const bucket = admin.storage().bucket();
const path = require('path');
const os = require('os');
const fs = require('fs');
module.exports = functions.https.onCall((data, context) => {
try {
return new Promise((resolve, reject) => {
const {
imageFiles,
companyPIN,
projectId
} = data;
const filename = imageFiles[0].replace(/^.*[\\\/]/, '');
const filePath = `ProjectPlans/${companyPIN}/${projectId}/images/${filename}`; // Path i am trying to upload in FIrebase storage
const tempFilePath = path.join(os.tmpdir(), filename);
const metadata = {
contentType: 'application/image'
};
axios
.get(imageFiles[0], { // URL for the image
responseType: 'arraybuffer',
headers: {
accept: 'application/image'
}
})
.then(response => {
console.log(response);
const blobObj = new Blob([response.data], {
type: 'application/image'
});
return blobObj;
})
.then(async blobObj => {
return bucket.upload(blobObj, {
destination: tempFilePath // Here i am wrong.. How to set the path of downloaded blob file
});
}).then(buffer => {
resolve({ result: 'success' });
})
.catch(ex => {
console.error(ex);
});
});
} catch (error) {
// unknown: 500 Internal Server Error
throw new functions.https.HttpsError('unknown', 'Unknown error occurred. Contact the administrator.');
}
});

I'd take a slightly different approach and avoid using the local filesystem at all, since its just tmpfs and will cost you memory that your function is using anyway to hold the buffer/blob, so its simpler to just avoid it and write directly from that buffer to GCS using the save method on the GCS file object.
Here's an example. I've simplified out a lot of your setup, and I am using an http function instead of a callable. Likewise, I'm using a public stackoverflow image and not your original urls. In any case, you should be able to use this template to modify back to what you need (e.g. change the prototype and remove the http response and replace it with the return value you need):
const functions = require('firebase-functions');
const axios = require('axios');
const admin = require('firebase-admin');
admin.initializeApp();
exports.doIt = functions.https.onRequest((request, response) => {
const bucket = admin.storage().bucket();
const IMAGE_URL = 'https://cdn.sstatic.net/Sites/stackoverflow/company/img/logos/so/so-logo.svg';
const MIME_TYPE = 'image/svg+xml';
return axios.get(IMAGE_URL, { // URL for the image
responseType: 'arraybuffer',
headers: {
accept: MIME_TYPE
}
}).then(response => {
console.log(response); // only to show we got the data for debugging
const destinationFile = bucket.file('my-stackoverflow-logo.svg');
return destinationFile.save(response.data).then(() => { // note: defaults to resumable upload
return destinationFile.setMetadata({ contentType: MIME_TYPE });
});
}).then(() => { response.send('ok'); })
.catch((err) => { console.log(err); })
});
As a commenter noted, in the above example the axios request itself makes an external network access, and you will need to be on the Blaze or Flame plan for that. However, that alone doesn't appear to be your current problem.
Likewise, this also defaults to using a resumable upload, which the documentation does not recommend when you are doing large numbers of small (<10MB files) as there is some overhead.
You asked how this might be used to download multiple files. Here is one approach. First, lets assume you have a function that returns a promise that downloads a single file given its filename (I've abridged this from the above but its basically identical except for the change of INPUT_URL to filename -- note that it does not return a final result such as response.send(), and there's sort of an implicit assumption all the files are the same MIME_TYPE):
function downloadOneFile(filename) {
const bucket = admin.storage().bucket();
const MIME_TYPE = 'image/svg+xml';
return axios.get(filename, ...)
.then(response => {
const destinationFile = ...
});
}
Then, you just need to iteratively build a promise chain from the list of files. Lets say they are in imageUrls. Once built, return the entire chain:
let finalPromise = Promise.resolve();
imageUrls.forEach((item) => { finalPromise = finalPromise.then(() => downloadOneFile(item)); });
// if needed, add a final .then() section for the actual function result
return finalPromise.catch((err) => { console.log(err) });
Note that you could also build an array of the promises and pass them to Promise.all() -- that would likely be faster as you would get some parallelism, but I wouldn't recommend that unless you are very sure all of the data will fit inside the memory of your function at once. Even with this approach, you need to make sure the downloads can all complete within your function's timeout.

How to softcode firebase storage image url's

I am using Firebase Storage to create and send image url's to Firebase Database. I have this working well when I have both a 'currentUser' and a Firebase Storage child reference hardcoded. But, how do I grab the actual currentUser from Firebase Database and put a fluid name to the child reference/references? I've seen some possible answers to this in Java. But not Javascript. Here's my code:
// openImage() is the function that grabs a photo from the camera roll
// I'm using react-native btw, so some of this code is handling the
// blob.
openImage() {
this.setState({ loading: true });
const Blob = RNFetchBlob.polyfill.Blob;
const fs = RNFetchBlob.fs;
window.XMLHttpRequest = RNFetchBlob.polyfill.XMLHttpRequest
window.Blob = Blob
// if I use this code, it won't return the urls: const {currentUser}
// = firebase.auth()
// So, I've got the 'currentUser' hardcoded in as "12345"
const { currentUser } = "12345"
// ImagePicker.openPicker() is just a library to access the camera roll
ImagePicker.openPicker({
width: 400,
height: 300,
cropping: true,
mediaType: 'photo',
}).then(image => {
const imagePath = image.path;
let uploadBlob = null;
const storage = firebase.storage();
const storageRef = storage.ref(currentUser);
//I also have the .child reference name hardcoded as 'dp.jpg'
const imageRef = storageRef.child('dp.jpg');
const mime = 'image/jpg';
fs.readFile(imagePath, 'base64')
.then((data) => {
return Blob.build(data, { type: `${mime};BASE64` });
})
.then((blob) => {
uploadBlob = blob;
return imageRef.put(blob, { contentType: mime });
})
.then(() => {
uploadBlob.close();
return imageRef.getDownloadURL();
})
.then((url) => {
const { image } = this.props;
this.props.entryUpdate({ prop: 'image', value: url })
});
});
}
The image url is then passed to Firebase Database and sets it as a key of 'image' and a value of 'dp.jpg'. With the hardcoding, this works fine, but for only one image and one user (or folder in Firebase Storage) of course. I'm aware of how Firebase Realtime Database will assign it's own id number to an item as in:
const { currentUser } = firebase.auth();
const item = firebase.database().ref(`/users/${currentUser.uid}/items`)
.push();
return () => {
item.set({ make, model, year, uid: item.key, image })
where item.key is generated by Firebase, thus it doesn't have to be hardcoded in. Is it possible to achieve this in Firebase Storage for both the image child name and the 'currentUser'? And do I really need to assign each image to a 'currentUser' folder in Firebase storage since the mobile app only is fetching the url from the database?

#parohy Thanks for taking the time to search through the firebase docs. I think .once() only works with Firebase Realtime Database. And I'm looking for a unique name in Firebase Storage. However, your answer helped me come to my solution. I was trying to tap into the fact that Firebase Database already creates perfectly unique ids. However, I'm don't think I can access those in Firebase Storage. So I just did:
firebase.storage().ref().child(unique + '.jpg')
const unique = Date.now() + Math.random();
not perfectly unique, but close enough.