Overview
I am working in a react app using redux. I have an action that checks wether or not we have new image data. If so, it will upload it, receive the new URL and then use this data to update the object.
Problem
However, the order of operations within my function are working as expected, however code outside the function runs before it is completed.
Question
Why is my console log at the bottom executing before the contents of my async function are completed?
if(imageData) {
const imageGUID = guid();
const storageRef = projectStorage.ref(`${imageData.name}_${imageGUID}`);
// This function should complete before the console log at the bottom is called.
await storageRef.put(imageData).on('state_changed', snap => {
}, async (err) => {
console.log(err)
toastr.error("Uh Oh!", `Could not upload image`);
}, async () => {
imageURL = await storageRef.getDownloadURL();
console.log("NEW IMAGE URL: ", imageURL);
})
}
console.log("Done setting up new image: ", imageURL) // This is called before we get the IMAGE URL from Firestore.... why?
The .on function does not return a Promise so there is nothing to wait for the await. You have to convert the event base API of put to a Promise.
if (imageData) {
const imageGUID = guid();
const storageRef = projectStorage.ref(`${imageData.name}_${imageGUID}`);
// This function should complete before the console log at the bottom is called.
try {
await new Promise((resolve, reject) => {
storageRef.put(imageData)
.on('state_changed', snap => {},
reject, resolve)
})
imageURL = await storageRef.getDownloadURL();
console.log("NEW IMAGE URL: ", imageURL);
} catch (err) {
console.log(err)
toastr.error("Uh Oh!", `Could not upload image`);
}
}
console.log("Done setting up new image: ", imageURL)
storageRef.put(imageData).on - doesn't look like promise (you upload your image in callback), so await doesn't make sense
If you want to use promises, you should write it something like that
await new Promise((resolve, reject) => {
storageRef
.put(imageData)
.on('state_changed', snap => {
}, async (err) => {
console.log(err);
toastr.error('Uh Oh!', `Could not upload image`);
reject()
}, async () => {
imageURL = await storageRef.getDownloadURL();
console.log('NEW IMAGE URL: ', imageURL);
resolve()
});
})
Related
I'm working on a microcontroller that would either take docx files or html strings in input and would transform it into a singular pdf file and return its link as an ouput.
My code looks like this so far:
// 'files' is an array of uploaded docx files.
const uploaded = files.map((file) => {
return new Promise((resolve, reject) => {
pump(
file.toBuffer(),
fs.createWriteStream(join(__dirname, 'files', file.filename))
.on('finish', resolve)
)
})
})
Promise.all(uploaded)
// Is triggered
.then(async () => await convertFiles())
// Is not triggered
.then(async () => {
// concatStoreFiles() is an external function because I need it somewhere else too
test = await concatStoreFiles(join(__dirname, 'files'))
console.log({test})
res.send(test)
})
const convertFiles = () => {
return new Promise((resolve, reject) => {
const cmd = `soffice --headless --convert-to pdf --outdir ${join(__dirname, 'files')} ${join(__dirname, 'files', '*.*')}`
exec(cmd, (error, stdout, stderror) => {
if (error) console.warn(error)
resolve(stdout ?? stderror)
})
})
}
concatStoreFile.js
module.exports = async function concatFiles (dirPath, outPath) {
return new Promise ((resolve, reject) => {
const existingFiles = []
fs.readdir(dirPath, (e, files) => {
files.forEach((file) => {
// is added to the files list only if finishing with ".pdf"
if (/[\d\w_-]+.pdf/.matches(file)) {
existingFiles.push(file)
}
});
resolve(existingFiles)
})
})
}
I'm working with Insomnia for my development / test process, and it tells me that I get an empty response. However, I'm supposed to get an array of pdf files existing in a specific directory. I'm not even getting console.log({test}), so I don't think my second then() is triggered.
I'm really rusty with async / await and Promise syntaxes, what should I do in this situation?
Thank you in advance
The #fastify/multipart's toBuffer() API returns a Promise, not a buffer. Checkout this article
So you need to write something like:
const uploaded = files.map(processFile)
async function processFile (file) {
const buffer = await file.toBuffer()
const storedFileName = join(__dirname, 'files', file.filename)
const writeStream = fs.createWriteStream(storedFileName)
return new Promise((resolve, reject) => {
pump(buffer, writeStream, (err) => {
if(err) { return reject(err) }
resolve(storedFileName)
})
}
}
Moreover, to improve the code, I returned the storedFileName instead of recalculating it.
You can convert this:
.then(async () => await convertFiles())
to this:
.then(() => convertFiles())
Mixing async/await and promise then/catch leads to hidden bugs hard to find
I'm doing my first ever react website and I need help to write an asynchronous JavaScript function.
Here I'm uploading the user input files to a firebase storage and then making a post request to the API to store the data on the database. However, since the firebase upload takes some time to upload the data to its storage, the API request happens before the upload finishes, therefore the data does not get uploaded to the db. Now I know I should use promises of async await keywords here, but I can't figure out how to. I'd appreciate if someone could help. Thanks in advance!
Here's the relevant code snippet:
const save = (items) => {
items.forEach((item) => {
const fileName = new Date().getTime() + item.label + item.file.name;
const uploadTask = storage.ref(`/items/${fileName}`).put(item.file);
uploadTask.on(
"state_changed",
(snapshot) => {
const progress = (snapshot.bytesTransferred / snapshot.totalBytes) * 100;
console.log("Upload is" + progress + "% done.");
},
(err) => {
console.log(err)
},
() => {
storage.ref("items").child(fileName).getDownloadURL().then((url) => {
setSong((prev) => {
return { ...prev, [item.label]: url };
});
});
}
);
})
console.log(song)
axios.post("songs/create", song);
}
PS: Here, items is the array of input files from the user, each file is with a label and it is how the attributes are named on the json document.
setSong is a useState function. Here The JSON document already contains the other user inputs(which are not files), and the setSong method is used to append the firebase URLs of the files to it.
You have to wait for all files to get uploaded then you can call your API, in order to do that you should use Promise.all to wait to resolve all files :
const save = items => {
Promise.all(
items.map(item => {
return new Promise(resolve => {
const fileName = new Date().getTime() + item.label + item.file.name
const uploadTask = storage.ref(`/items/${fileName}`).put(item.file)
uploadTask.on(
'state_changed',
snapshot => {
const progress =
(snapshot.bytesTransferred / snapshot.totalBytes) * 100
console.log('Upload is' + progress + '% done.')
},
err => {
console.log(err)
},
() => {
storage
.ref('items')
.child(fileName)
.getDownloadURL()
.then(url => {
setSong(prev => {
return { ...prev, [item.label]: url }
})
resolve({[item.label]: url})
})
}
)
})
})
).then((res) => {
const song = {}
res.forEach(item => {
return {
...song,
...item
}
})
axios.post('songs/create', song)
})
}
Explanation
Functions and Async
Async/Await can be implemented wherever a function starts. Functions can be written in following forms:
function name(){};
function name() => {};
To write an async function, you would do the following:
async function name(){};
All of these functions are called functions though, to make functions run without calling them, we need to turn them into IIFE's, or Immediately Invoked Function Execution. If you want to create a function and execute it immediately you would surround the function in ()'s and end it with an ();.
(function () {})();
If we simplify this:
(() => {})();
Implementing async would go like this:
(async () => {})();
Await
The await operator is used to wait for a Promise, puting await in front of an expression that uses promises makes it wait for the promise. If it is used in front of an expression that doesn't use promises, it is redundant and your code editor/IDE will say so.
(async () => {
const str = await 'some string';
console.log(str);
})();
await is redundant here since the expression 'some string' does not relate to a promise, but a string.
(async () => {
const myPromise = new Promise((resolve, reject) =>
resolve('some string')
);
const str = await myPromise.then(x => console.log(x));
})();
await is properly used here since the expression myPromise is related to a promise that outputs a string.
Implementation
I'm not 100% sure how the api works within promises, I recommend you figure it out yourself, but this is my educated guess:
const save = (items) => {
Promise.all(
items.map((item) => {
return new Promise(async (resolve) => {
const fileName = new Date().getTime() + item.label + item.file.name;
const uploadTask = await storage
.ref(`/items/${fileName}`)
.put(item.file);
await uploadTask.on(
"state_changed",
(snapshot) => {
const progress =
(snapshot.bytesTransferred / snapshot.totalBytes) * 100;
console.log("Upload is" + progress + "% done.");
},
(err) => {
console.log(err);
},
async () => {
await storage
.ref("items")
.child(fileName)
.getDownloadURL()
.then((url) => {
setSong((prev) => {
return { ...prev, [item.label]: url };
});
resolve({ [item.label]: url });
});
}
);
});
})
).then(async (res) => {
const song = {};
res.forEach((item) => {
return {
...song,
...item,
};
});
await axios.post("songs/create", song);
});
};
I am new to node and stuck with this issue. Here' the file:
I am running 'startProcess' function and I want to run 'downloadFiles' and wait until it's completed and save the files before executing any code after it.
This code always ends up running 'runVideoUploadEngine' even before the download has been completed?
const downloadAndSaveFiles = async ({ url, dir }) => {
try {
https.get(url, (res) => {
// File will be stored at this path
console.log('dir: ', dir);
var filePath = fs.createWriteStream(dir);
res.pipe(filePath);
filePath.on('finish', () => {
filePath.close();
console.log('Download Completed');
});
});
return true;
} catch (e) {
console.log(e);
throw e;
}
};
const downloadFiles = async ({ data }) => {
try {
mediaUrl = data.mediaUrl;
thumbnailUrl = data.thumbnailUrl;
const mediaExt = path.extname(mediaUrl);
const thumbExt = path.extname(thumbnailUrl);
mediaDir = `${__dirname}/temp/${'media'}${mediaExt}`;
thumbDir = `${__dirname}/temp/${'thumb'}${thumbExt}`;
await downloadAndSaveFiles({ url: mediaUrl, dir: mediaDir });
await downloadAndSaveFiles({ url: thumbnailUrl, dir: thumbDir });
return { mediaDir, thumbDir };
} catch (e) {
console.log(e);
throw e;
}
};
module.exports = {
startProcess: async ({ message }) => {
//check if message is proper
data = JSON.parse(message.Body);
//download video and thumbnail and store in temp.
console.log('starting download..');
const { mediaDir, thumbDir } = await downloadFiles({ data });
console.log('dir:- ', mediaDir, thumbDir);
pageAccessToken =
'myRandomToken';
_pageId = 'myRandomPageID';
console.log('running engine');
await runVideoUploadEngine({ pageAccessToken, _pageId, mediaDir, thumbDir });
//start videoUploadEngine
//on success: delete video/thumbnail
},
};
What am I doing wrong?
downloadAndSaveFiles returns a promise (because the function is async) but that promise doesn't "wait" for https.get or fs.createWriteStream to finish, and therefore none of the code that calls downloadAndSaveFiles can properly "wait".
If you interact with callback APIs you cannot really use async/await. You have to create the promise manually. For example:
const downloadAndSaveFiles = ({ url, dir }) => {
return new Promise((resolve, reject) => {
// TODO: Error handling
https.get(url, (res) => {
// File will be stored at this path
console.log('dir: ', dir);
var filePath = fs.createWriteStream(dir);
filePath.on('finish', () => {
filePath.close();
console.log('Download Completed');
resolve(); // resolve promise once everything is done
});
res.pipe(filePath);
});
});
};
I'm trying to upload multiple images using cloudinary in node.js application.
Storing every image URL in an array. But my array is empty outside the loop. Can't understand why.
const postCreate = (req,res,next) => {
req.body.post.images = [];
const file_length = req.files.length;
let arr = [];
//console.log(req.files);
new Promise((resolve, reject) => {
req.files.forEach((file,index) => {
i = index;
cloudinary.v2.uploader.upload(file.path)
.then(image => {
//console.log(image);
req.body.post.images.push({
url: image.secure_url,
public_id: image.public_id
});
console.log("array", req.body.post.images);//there array is containing the element which is pushed.
});
console.log("arr", req.body.post.images);//but there it is showing empty array .Can't understand why array is empty.
});
resolve();
}).then(() => {
Post.create(req.body.post)
.then(post => {
//console.log(req.body.post.images);
res.redirect(`/posts/${post.id}`);
}).catch(err => {
console.log('Error will saving posts from db ', err);
return next(err);
});
});
Each of the uploads is asynchronous and returns a promise.
You need to have all those promises resolve before moving on to the final then()
You can map an array of those promises and use Promise.all() to return the full array to the final then()
Something like:
const doUpload = (file) => {
// return the upload promise
return cloudinary.v2.uploader.upload(file.path).then(image => {
return {
url: image.secure_url,
public_id: image.public_id
};
});
}
const postCreate = (req, res, next) => {
// map array of individual upload promises
const uploadPromises = req.files.map(doUpload);
Promise.all(uploadPromises).then(imagesArray => {
// assign new array to post body
req.body.post.images = imagesArray;
Post.create(req.body.post)
.then(post => {
//console.log(req.body.post.images);
res.redirect(`/posts/${post.id}`);
}).catch(err => {
console.log('Error will saving posts from db ', err);
return next(err);
});
}).catch(err=> console.log('One of the uploads failed'));
}
The 2nd log message actually gets called first while the array is empty because the code in the then block is waiting for something asynchronous to complete.
your problem is your print function fire before loop is complated so you have to use async-await for proper solution and learn more about this topic
please refer https://blog.risingstack.com/mastering-async-await-in-nodejs for your solution
it describe the async await for your proper output
I am trying to download txt and mp3 files and use the content of them in another node module.
How can I create a global variable with the piped contents from the downloaded txt (and MP3 at a later stage) file to use outside of the FTP function?
async function example() {
var finalData = '';
const client = new ftp.Client()
client.ftp.verbose = true
try {
await client.access({
host: "XXXX",
user: "XXXX",
password: "XXXX",
})
await client.upload(fs.createReadStream("README.txt"), myFileNameWithExtension)
//let writeStream = fs.createWriteStream('/tmp/' + myFileNameWithExtension);
//await client.download(writeStream, myFileNameWithExtension)
finalData = await (() => {
return new Promise((resolve, reject) => {
writeStream
.on('finish', () => {
// Create a global variable to be used outside of the FTP function scope to pipe the txt content into another node mogule
})
.on('error', (err) => {
console.log(err);
reject(err);
})
})
})();
}
catch (err) {
console.log(err)
}
client.close();
return finalData;
}
No, don't create any global variables. Just resolve the promise with the data:
var finalData = await new Promise((resolve, reject) => {
writeStream.on('finish', () => {
resolve(); // pass a value
}).on('error', (err) => {
reject(err);
});
});
The finalData will become whatever value you pass into resolve(…) - I don't know what result you want to pass there. In the end, just return the data from your example function (as you already do), so that the caller will be able to use it after waiting for the returned promise.