I'm trying to download images from S3 bucket in a loop. My bucket is not public and using the direct getSignedURL doesn't work (Forbidden error). I need to download (between 10 - 30) images from S3 upon user selection from the user interface (and then later just delete after creating a GIF).
It downloads correct amount of images (with correct names) but the content of all the images is replaced with the last image on the local machine. I even wrote a Promise to call within the loop (hoping that each getObject call will complete first before going to the next) but didn't work. Except bluebird, I tried all solutions from this, but same result. My code looks like this:
var urlParams = {Bucket: 'bucket_name', Key: ''};
for (i = 0; i < imageNames.length; i+=increment) {
urlParams.Key = imageNames[i]+'.jpg';
pathToSave = '/local-files/'+urlParams.Key;
var tempFile = fs.createWriteStream(pathToSave);
// I tried a Promise (and setTimeout) here too but gives me the same result
var stream = s3.getObject(urlParams).createReadStream().pipe(tempFile);
var had_error = false;
stream.on('error', function(err){
had_error = true;
});
stream.on('close', function(){
if (!had_error) {
console.log("Image saved");
}
});
}
After the above code finishes, as I mentioned all images with correct names are saved but due to the non-blocking issue here, all images contain the content of last image in the array (imageNames). The Promise I wrote and tried is below
function getBucketObject(urlParams){
return new Promise ((resolve, reject)=> {
var pathToSave = '/local-files/'+params.Key;
var tempFile = fs.createWriteStream(pathToSave);
var stream = s3.getObject(params).createReadStream().pipe(tempFile);
var had_error = false;
stream.on('error', function(err){
had_error = true;
});
stream.on('close', function(){
if (!had_error) {
resolve(pathToSave);
}
});
})
}
Both setTimeout and Promise are not working for my issue. Any help will be appreciated. Thanks
You should use let instead of var
Modify your code as:
for (var i = 0; i < imageNames.length; i+=increment) {
let urlParams = {Bucket: 'bucket_name', Key: imageNames[i]+'.jpg'};
let pathToSave = 'img/analysis/'+urlParams.Key;
getBucketObject(urlParams).then(function(pathToSave){
console.log("image saved");
})
}
function getBucketObject(urlParams){
return new Promise ((resolve, reject)=> {
let pathToSave = '/local-files/'+params.Key;
let tempFile = fs.createWriteStream(pathToSave);
let stream = s3.getObject(params).createReadStream().pipe(tempFile);
let had_error = false;
stream.on('error', function(err){
had_error = true;
});
stream.on('close', function(){
if (!had_error) {
resolve(pathToSave);
}
});
})
}
Following what #Anshuman Jaiswal suggested in the comment, I tried the code with let instead of var and it works now. Thanks Anshuman. The code in the loop looks like below
for (var i = 0; i < imageNames.length; i+=increment) {
let urlParams = {Bucket: 'bucket_name', Key: imageNames[i]+'.jpg'};
let pathToSave = '/local-files/'+urlParams.Key;
getBucketObject(urlParams).then(function(pathToSave){
console.log("image saved");
})
}
And the promise function is below
function getBucketObject(urlParams){
return new Promise ((resolve, reject)=> {
let pathToSave = '/local-files/'+params.Key;
let tempFile = fs.createWriteStream(pathToSave);
let stream = s3.getObject(params).createReadStream().pipe(tempFile);
let had_error = false;
stream.on('error', function(err){
had_error = true;
});
stream.on('close', function(){
if (!had_error) {
resolve(pathToSave);
}
});
})
}
Related
I'm trying to upload multiple images to firebase at once. It is doing that, but the returned url array of those cloud image links are too late since the post is already being sent with an empty array. Here is my code:
// uploading media files using promises
async uploadMedia(mediaFile: string){
const extension = mediaFile.split('.')[mediaFile.split('.').length - 1];
const mediaFileName = `${Math.round(Math.random()*100000000000)}.${extension}`;
this.uploadProgress = 0;
const response = await fetch(mediaFile);
const blob = await response.blob();
const storageRef = storage.ref(`${mediaFileName}`).put(blob);
return storageRef.on(`state_changed`,snapshot=>{
this.uploadProgress = (snapshot.bytesTransferred/snapshot.totalBytes);
}, error=>{
this.error = error.message;
this.submitting = false;
this.uploadingMedia = false;
return;
},
async () => {
// check whether the media is an image or a video and add to correct arrays
if(extension == "png" || extension == "jpg"){
return storageRef.snapshot.ref.getDownloadURL().then(async (url)=>{
this.firebaseImageUrls = [...this.firebaseImageUrls, url];
return;
});
}
else{
return storageRef.snapshot.ref.getDownloadURL().then(async (url)=>{
this.firebaseVideoUrls = [...this.firebaseVideoUrls, url];
return;
});
}
});
}
Where everything is being called:
await Promise.all(this.props.store.selectedImagesArray.map(async (file:string) => {
await this.uploadMedia(file);
}))
this.submitPost(); // this submits everything with the firebaseImageUrls
any help is appreciated
The problem seems to be that storageRef.on() does not return a promise. It just registers the handlers. I'm not an expert on firebase. Maybe the put(blob) returns a promise that you can use.
Figured it out. I had to make a promise and resolve the promise for each upload task and then loop through all the files doing this. Then when all the files are completely uploaded and the loop is completed, then I can submit the post with the files that are in firebaseImageUrls.
async uploadMedia(mediaFile: string){
return new Promise(async (resolve, reject) => {
//making the uploading task for one file
const extension = mediaFile.split('.')[mediaFile.split('.').length - 1];
const mediaFileName = `${Math.round(Math.random()*100000000000)}.${extension}`;
const response = await fetch(mediaFile);
const blob = await response.blob();
const storageRef = storage.ref(`${mediaFileName}`);
const task = storageRef.put(blob);
task.on(`state_changed`,snapshot=>{
this.uploadProgress = (snapshot.bytesTransferred/snapshot.totalBytes);
}, error=>{
this.error = error.message;
this.submitting = false;
this.uploadingMedia = false;
return;
},
async () => {
if(extension == "png" || extension == "jpg"){
task.snapshot.ref.getDownloadURL().then((url:any)=>{
console.log(url);
resolve(url);
});
}
else{
task.snapshot.ref.getDownloadURL().then((url:any)=>{
console.log(url);
resolve(url);
});
}
});
})
}
The loop:
for(var i = 0; i < this.props.store.selectedImagesArray.length; i++){
const imageUrl = await this.uploadMedia(this.props.store.selectedImagesArray[i]);
this.firebaseImageUrls = [...this.firebaseImageUrls, imageUrl];
}
this.submitPost();
WebAssembly.instantiateStreaming is the fastest way to download and instantiate a .wasm module however for large .wasm files it can still take a long time. Simply displaying a spinner does not provide enough user feedback in this case.
Is there a way to use the WebAssembly.instantiateStreaming api and get some form of progress event so that an eta can displayed to the user? Ideally I would like to be able to display a percentage progress bar / estimated time left indicator so user's know how long they will have to wait.
Building off the answer here.
To get the progress of WebAssembly.instantiateStreaming / WebAssembly.compileStreaming create a new Fetch Response with a custom ReadableStream which implements it's own controller.
Example:
// Get your normal fetch response
var response = await fetch('https://www.example.com/example.wasm');
// Note - If you are compressing your .wasm file the Content-Length will be incorrect
// One workaround is to use a custom http header to manually specify the uncompressed size
var contentLength = response.headers.get('Content-Length');
var total = parseInt(contentLength, 10);
var loaded = 0;
function progressHandler(bytesLoaded, totalBytes)
{
// Do what you want with this info...
}
var res = new Response(new ReadableStream({
async start(controller) {
var reader = response.body.getReader();
for (;;) {
var {done, value} = await reader.read();
if (done)
{
progressHandler(total, total)
break
}
loaded += value.byteLength;
progressHandler(loaded, total)
controller.enqueue(value);
}
controller.close();
},
}, {
"status" : response.status,
"statusText" : response.statusText
}));
// Make sure to copy the headers!
// Wasm is very picky with it's headers and it will fail to compile if they are not
// specified correctly.
for (var pair of response.headers.entries()) {
res.headers.set(pair[0], pair[1]);
}
// The response (res) can now be passed to any of the streaming methods as normal
var promise = WebAssembly.instantiateStreaming(res)
Building off of various other SO answers, here is what I ended up with.
My solution also has decent fallback for Firefox, which doesn't yet have proper stream support. I opted for falling back to a good old XHR and WebAssembly.Instantiate there, as I really do want to show a loading bar, even if it means slightly slower startup just on FF.
async function fetchWithProgress(path, progress) {
const response = await fetch(path);
// May be incorrect if compressed
const contentLength = response.headers.get("Content-Length");
const total = parseInt(contentLength, 10);
let bytesLoaded = 0;
const ts = new TransformStream({
transform (chunk, ctrl) {
bytesLoaded += chunk.byteLength;
progress(bytesLoaded / total);
ctrl.enqueue(chunk)
}
});
return new Response(response.body.pipeThrough(ts), response);
}
async function initWasmWithProgress(wasmFile, importObject, progress) {
if (typeof TransformStream === "function" && ReadableStream.prototype.pipeThrough) {
let done = false;
const response = await fetchWithProgress(wasmFile, function() {
if (!done) {
progress.apply(null, arguments);
}
});
await WebAssembly.InstantiateStreaming(response, importObject);
done = true;
progress(1);
} else {
// xhr fallback, this is slower and doesn't use WebAssembly.InstantiateStreaming,
// but it's only happening on Firefox, and we can probably live with the game
// starting slightly slower there...
const xhr = new XMLHttpRequest();
await new Promise(function(resolve, reject) {
xhr.open("GET", wasmFile);
xhr.responseType = "arraybuffer";
xhr.onload = resolve;
xhr.onerror = reject;
xhr.onprogress = e => progress(e.loaded / e.total);
xhr.send();
});
await WebAssembly.Instantiate(xhr.response, importObject);
progress(1);
}
}
const wasmFile = "./wasm.wasm";
await initWasmWithProgress(wasmFile, importObject, p => console.log(`progress: ${p*100}%`));
console.log("Initialized wasm");
I have an input element on my html page, with which I can select 1/multiple file(s).
Once I have chosen my file(s), I would like to read the content of each file using a FileReader to make a SHA1 from it.
Once I have the SHA1 value, I would like to save it somewhere.
The problem is that I receive the SHA1 value only after the .onload of the FileReader is finished and that happens after I try to safe it's value.
I have tried to make the function async and using an await to wait until the file is read but that didn't work.
I have tried to add a Promise but that didn't work either.
I really don't know what to do to have the desired outcome. Please help.
This is my angular function that I call when I have choosen my file(s):
hashFiles(files: Array<any>){
console.log('start hashing');
for (const file of files) {
const myHash = hashFile(file);
console.log('hash: ', myHash);
/*I would like to save myHash here*/
}
console.log('done hashing');
}
This is my javascript function that's called from angular that will read the file with a FileReader and then make a sha1 hash from it's content
function hashFile(fileToHandle) {
console.log('1');
var reader = new FileReader();
console.log('2');
reader.onload = (function() {
return function(e) {
console.log('4');
const hash = CryptoJS.SHA1(arrayBufferToWordArray(e.target.result)).toString();
console.log('hash result in fileReader: ', hash);
return hash;
};
}) (fileToHandle);
reader.onerror = function(e) {
console.error(e);
};
console.log('3');
reader.readAsArrayBuffer(fileToHandle);
console.log('5');
}
function arrayBufferToWordArray(ab) {
var i8a = new Uint8Array(ab);
var a = [];
for (var i = 0; i < i8a.length; i += 4) {
a.push(i8a[i] << 24 | i8a[i + 1] << 16 | i8a[i + 2] << 8 | i8a[i + 3]);
}
return CryptoJS.lib.WordArray.create(a, i8a.length);
}
When running this code I have the following in my console:
start hashing
1
2
3
5
hash: undefined
done hashing
4
hash result in fileReader: 327c468b64b4ca54377546f8a214d703ccbad64b
And I need it to be:
start hashing
1
2
3
hash result in fileReader: 327c468b64b4ca54377546f8a214d703ccbad64b
4
5
hash: 327c468b64b4ca54377546f8a214d703ccbad64b
done hashing
Here is a sample of my code:
https://stackblitz.com/edit/sha1-from-file
You need to either work with call backs or add a promise wrapper (observable would also work) to the native callbacks of the FileReader. Here is how you could do it just by adding a call back.
Stackblitz
app.component.ts
hashFiles(event) {
console.log('start hashing');
const numberOfFiles = event.target.files.length;
var fileCounter = 0;
for (const file of event.target.files) {
hashFile(file, (hashedFile, hash) => {
console.log('hash: ', hash);
fileCounter++;
if(fileCounter === numberOfFiles)
console.log('Done Hashing!');
});
}
}
script.js
export function hashFile(fileToHandle, callback) {
var CryptoJS = require("crypto-js");
console.log('1');
var reader = new FileReader();
console.log('2');
reader.onloadend = (function() {
return function(e) {
console.log('4');
const hash = CryptoJS.SHA1(arrayBufferToWordArray(e.target.result)).toString();
console.log('hash result in fileReader: ', hash);
callback(fileToHandle, hash);
};
}) (fileToHandle);
reader.onerror = function(e) {
console.error(e);
};
console.log('3');
reader.readAsArrayBuffer(fileToHandle);
}
I am working on MEAN stack application and I am using AWS SDK to upload multiple files to S3. I am using busboy and AWS SDK.
Code:
var inputObj = {};
var busboy = new Busboy({ headers: req.headers });
// The file upload has completed
busboy.on('finish', function() {
console.log('Upload finished.....');
var file = [];
const file1 = req.files.clogo;
const file2 = req.files.cbanner1;
const file3 = req.files.cbanner2;
const file4 = req.files.cbanner3;
const file5 = req.files.cbanner4;
const file6 = req.files.clongHeader;
file.push(file1);
file.push(file2);
file.push(file3);
file.push(file4);
file.push(file5);
file.push(file6);
multipleUploadToS3(req.body.cname, file, function(fileName) {
console.log("client file upload finished.....");
if(fileName.length == 6){
inputObj.clogo = fileName[0];
inputObj.cbanner1 = fileName[1];
inputObj.cbanner2 = fileName[2];
inputObj.cbanner3 = fileName[3];
inputObj.cbanner4 = fileName[4];
inputObj.clongHeader = fileName[5];
console.log(inputObj);
var clientObj = new client(inputObj);
clientObj.save(function(err, client) {
console.log("Client Saved.....");
if (err) {
return res.status(400).send({
message: errorHandler.getErrorMessage(err)
});
} else {
res.json(client);
}
});
}
});
});
req.pipe(busboy);
File Upload Method:
function multipleUploadToS3(client, file, callback) {
console.log("multipleUpload to S3");
console.log(client);
console.log(file);
let s3bucket = new AWS.S3({
accessKeyId: IAM_USER_KEY,
secretAccessKey: IAM_USER_SECRET,
Bucket: BUCKET_NAME,
});
var fileNames = [];
for(var i=0; i<file.length; i++){
s3bucket.createBucket(function () {
var params = {
Bucket: BUCKET_NAME,
Key: client+ '/' + file[i].name,
Body: file[i].data,
};
s3bucket.upload(params, function (err, data) {
if (err) {
console.log('error in callback');
console.log(err);
}
console.log('success');
//console.log(data.key);
fileNames.push(data.key);
if(i == file.length){ callback(fileNames);}
});
});
}
};
The issue: file upload is asynchronous so for example if file1 I am uploading is honest.jpg then I want multipleUploadToS3 method to return file name after its done uploading to S3. I am binding this in inputObj keys which will be saved to mongo db. so inputObj.logo should have logo.png in it not the banner image which is happening due to asynchronous call.
This is working for a single file but failing for multiple files.
The problem is because for loop is synchronous and file upload is asynchronous.
Take a look at this example below,
for(var i = 0; i<5;i++) {
setTimeout(function(){ console.log(i); }, 100);
}
The above loop will print 5 for 5 times i.e 55555.
This behaviour is because for loop gets executed immediately making i=5 and when timeout gets executed it prints "i" value 5 for 5 times. 5 times because setTimeout is pushed in the queue for five times.
There are two ways to solve the problem you are facing
You can use Recursion.
Use neo-async(async-parallel) lib to control the async flow of javascript(Nodejs). click here for neo-async lib
Hope this clears your doubt. Please comment for more info.
I am new to JS and trying to write a simple webserver. Here is my prj browser.
When I go to browser -http://localhost:4000/home.html
I get the error - Cannot call method 'toString' of null.
The problem is that UrlResLoader.httpcode and UrlResLoader.fileType isn't defined
var UrlResLoader = new UrlLoader();
UrlResLoader.requestUrl(fileResEngine);
res.writeHead(UrlResLoader.httpcode, UrlResLoader.fileType);
res.write(UrlResLoader.data);
res.end()
I am not sure what is the problem here, I have hooked a debugger and have found that the problem happens on the fs.readFile(fileResEngine.fullpath, function (err, data).
I am still unclear on why is it happening. After researching a bit, I found that inorder to invoke closure functions, I should save the "this" pointer to refer the member varaibles. Other wise, the instance will be different.
But, this hasn't fixed the problem.
Also, any design flaws or comments will be welcomed here.
Here is my code -
The main file -
const http = require('http');
const parseUrl = require('parseurl');
const fileEngine = require('./fileErrHandler');
const UrlLoader = require('./urlController');
http.createServer( function (req, res)
{
try
{
// this is a library function
var pathName = decodeURIComponent(parseUrl(req).pathname);
var fileResEngine= new fileEngine(pathName);
// create a literal validateFile to validate the path
fileResEngine.pathCheck();
if (fileResEngine.error === true )
{
res.statusCode = fileResEngine.statusCode;
res.end(fileResEngine.ErrorMsg);
return;
}
else
{
var UrlResLoader = new UrlLoader();
UrlResLoader.requestUrl(fileResEngine);
res.writeHead(UrlResLoader.httpcode, UrlResLoader.fileType);
res.write(UrlResLoader.data);
res.end();
}
}
catch(err)
{
res.statusCode = err.status || 500;
res.end(err.message);
}
}).listen(4000);
The file error handler
var resolvePath = require('resolve-path');
const path = require('path');
var pagesDir = path.join(__dirname, 'Pages');
function pathCheckerEngine(path)
{
this.error = true;
this.path = path;
this.statusCode = 500;
this.ErrorMsg = "Internal Server Error";
this.PageRequest = "home.html";
this.extname = "html";
this.fullpath = './';
var pcEngine = this;
this.pathCheck = function()
{
try {
if (!path) {
pcEngine.statusCode = 400;
pcEngine.ErrorMsg = 'path required';
pcEngine.error = true;
}
else {
//removes first '/' of the path
pcEngine.PageRequest = path.substr(1);
pcEngine.fullpath = resolvePath(pagesDir, this.PageRequest);
pcEngine.statusCode = 200;
pcEngine.ErrorMsg = null;
pcEngine.error = false;
pcEngine.extname = this.PageRequest.split('.').pop();
}
}
catch(err)
{
pcEngine.statusCode = err.status || 500;
pcEngine.ErrorMsg = 'Malicious Page Request';
pcEngine.error = true;
}
}
}
module.exports = pathCheckerEngine;
And the final file
const fileEngine = require('./fileErrHandler');
const fs = require('fs');
const mime = require('mime');
function UrlController(fileResEngine) {
this.httpcode = null;
this.fileType = null;
this.data = null;
var urctrl = this;
this.requestUrl = function (fileResEngine) {
switch (fileResEngine.extname) {
case 'html':
fs.readFile(fileResEngine.fullpath, function (err, data) {
if (err)
{
console.log(err);
urctrl.httpcode = 404;
urctrl.data = "Page not found";
return;
}
urctrl.httpcode = 200;
urctrl.fileType = "'Content-Type': 'text/html'";
urctrl.data = data;
});
break;
case 'png':
case 'jpeg':
case 'jpg':
case 'bmp':
fs.readFile(fileResEngine.fullpath, function (err, data) {
if (err) {
console.log(err);
urctrl.httpcode = 404;
urctrl.data = "File not found";
return;
}
urctrl.httpcode = 200;
urctrl.fileType = mime.lookup(mime.lookup('./images' + req.url));
urctrl.data = data;
});
break;
default:
urctrl.httpcode = 404;
urctrl.data = "Page not Found"
break;
}
}
return;
}
module.exports = UrlController;
For starters, this is wrong:
var pathName = decodeURIComponent(parseUrl(req).pathname);
which is probably why your pathName later on is wrong. req is a request object, it's not a URL you can parse. The URL path is in that request object. req.url will contain the request path (without domain. port and protocol - just the path).
So, change that above line to this:
var pathName = decodeURIComponent(req.url);
I can't promise there aren't other errors in that block of code (I don't know that fileEngine module at all), but this is at least one thing to clear up that affects your pathName being wrong which can lead to the error you get later on when trying to use the pathName.
Now that you've fixed that, it appears you also have asynchronous problems. You've coded UrlController.requestUrl, but not coded any way to know when it is done with its asynchronous work. It returns right away and then sometime later, the fs.readFile() inside of it finishes. So you end up trying to use the properties of your object long before they have been set.
You will need to use either promises or a callback to let your caller know when your asynchronous operations are actually done. I'd suggest you read this canonical answer on the topic of returning async data.