I have been using the code below (which I have now added await to) to send files to S3. It has worked fine with my lambda code but as I move to transfer larger files like MP4 I feel I need async/await.
How can I fully convert this to async/await?
exports.handler = async (event, context, callback) => {
...
// Copy data to a variable to enable write to S3 Bucket
var result = response.audioContent;
console.log('Result contents ', result);
// Set S3 bucket details and put MP3 file into S3 bucket from tmp
var s3 = new AWS.S3();
await var params = {
Bucket: 'bucketname',
Key: filename + ".txt",
ACL: 'public-read',
Body: result
};
await s3.putObject(params, function (err, result) {
if (err) console.log('TXT file not sent to S3 - FAILED'); // an error occurred
else console.log('TXT file sent to S3 - SUCCESS'); // successful response
context.succeed('TXT file has been sent to S3');
});
You only await functions that return a promise. s3.putObject does not return a promise (similar to most functions that take a callback). It returns a Request object. If you want to use async/await, you need to chain the .promise() method onto the end of your s3.putObject call and remove the callback (https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/Request.html#promise-property)
try { // You should always catch your errors when using async/await
const s3Response = await s3.putObject(params).promise();
callback(null, s3Response);
} catch (e) {
console.log(e);
callback(e);
}
As #djheru said, Async/Await only works with functions that return promises.
I would recommend creating a simple wrapper function to assist with this problem.
const putObjectWrapper = (params) => {
return new Promise((resolve, reject) => {
s3.putObject(params, function (err, result) {
if(err) reject(err);
if(result) resolve(result);
});
})
}
Then you could use it like this:
const result = await putObjectWrapper(params);
Here is a really great resource on Promises and Async/Await:
https://javascript.info/async
Related
I'm trying to grab thousands of files from Amazon S3 within a Promise but I can't seem to figure out how to include the ContinuationToken within if the list is truncated and gather it all together within the promise. I'm a novice with JS and could use some help. Here's what I have, so far:
getFiles()
.then(filterFiles)
.then(mapUrls)
;
function getFiles(token) {
var params = {
Bucket: bucket,
MaxKeys: 5000,
ContinuationToken: token
};
var allKeys = [];
var p = new Promise(function(resolve, reject){
s3.listObjectsV2(params, function(err, data) {
if (err) {
return reject(err);
}
allKeys.push(data.Contents)
if (data.IsTruncated) {
s3.listObjectsV2({Bucket: bucket, MaxKeys: 5000, ContinuationToken: data.NextContinuationToken})
console.log('Getting more images...');
allKeys.push(data.Contents)
}
resolve(data.Contents);
});
});
return p;
}
I need the function to continue to run until I've created a list of all objects in the bucket to return.
You need ContinuationToken the second time only.
var params = {
Bucket: bucket,
MaxKeys: 5000,
};
if (data.IsTruncated) {
s3.listObjectsV2({...params, ContinuationToken: data.NextContinuationToken})
IMO, this is just a s3 function called twice, more like a nested
call. Recursion is when a function keeps calling itself
until a specified condition is met.
Read more about recursion: https://medium.com/#vickdayaram/recursion-caad288bf621
I was able to list all objects in the bucket using async/await and the code below to populate an array.
async function getFiles(objects = []) {
const response = await s3.listObjectsV2(params).promise();
response.Contents.forEach(obj => filelist.push(obj.Key));
if (response.NextContinuationToken) {
params.ContinuationToken = response.NextContinuationToken;
await getFiles(params, objects);
}
console.log(filelist.length)
return filelist;
}
Thanks to all who helped!
I'm pretty sure I'm missing something very obvious here, but:
I'm uploading a file to an s3 bucket using aws-sdk as follows:
const awsURL = await s3.upload(params, (err, data) => {
if (err) {
console.log(err);
return null;
}
console.log(`File uploaded successfully. ${data.Location}`);
return data.Location;
});
return awsURL;
I'm able to log the upload url successfully, however the awsURL returned is an array, not the data.Location value - shouldn't the data.Location be returned from the callback?
Convert s3.upload to return a promise:
const data = await s3.upload(params).promise(); // this line
console.log(`File uploaded successfully. ${data.Location}`);
return data.Location;
I am refactoring some code that was using http module in Node to use got instead. I tried the following:
function get(url, filePath) {
return new Promise((resolve, reject) => {
got.stream(url).on
("response", response => {
const newFile = fs.createWriteStream(filePath);
response.pipe(newFile);
newFile.on("finish", () => {
newFile.close(resolve());
});
newFile.on("error", err => {
reject(err);
});
}).on
("error", err => {
reject(err);
});
});
}
The finish event never fired. The file (filePath) is created with 0 bytes.
The block of code using newFile was something that worked when I was using the Node http module.
What is the proper way to pipe got.stream to a file?
Per the got() documentation, you want to pipe the stream directly to your file and if you use pipeline() to do it, it will collect errors and report completion.
const pipeline = promisify(stream.pipeline);
const fsp = require('fs').promises;
function get(url, filePath) {
return pipeline(
got.stream(url),
fs.createWriteStream(filePath)
);
}
// usage
get(...).then(() => {
console.log("all done");
}).catch(err => {
console.log(err);
});
FYI, the point of got.stream() is to return a stream that you can directly use as a stream and since you want it to go to a file, you can pipe that stream to that file. I use pipeline() instead of .pipe() because pipeline has much more complete error handling that .pipe(), though in non-error conditions, .pipe() would also work.
Here's a version that cleans up the output file if there's an error:
function get(url, filePath) {
return pipeline(
got.stream(url),
fs.createWriteStream(filePath)
).catch(err => {
fsp.unlink(filePath).catch(err => {
if (err.code !== 'ENOENT') {
// trying to delete output file upon error
console.log('error trying to delete output file', err);
}
});
throw err;
});
}
I'm using the AWS Javascript SDK and I'm following the tutorial on how to send an SQS message. I'm basically following the AWS tutorial which has an example of the sendMessage as follows:
sqs.sendMessage(params, function(err, data) {
if (err) {
console.log("Error", err);
} else {
console.log("Success", data.MessageId);
}
});
So the sendMessage function uses a callback function to output whether the operation was successful or not. Instead of printing to the console I want to return a variable, but every value I set is only visible within the callback function, even global variables like window.result are not visible outside the callback function. Is there any way to return values outside the callback?
The only workaround I've found at the moment is to set a data attribute in an HTML element, but I don't think it's really elegant solution.
I would suggest to use Promises and the new async and await keywords in ES2016. It makes your code so much easier to read.
async function sendMessage(message) {
return new Promise((resolve, reject) => {
// TODO be sure SQS client is initialized
// TODO set your params correctly
const params = {
payload : message
};
sqs.sendMessage(params, (err, data) => {
if (err) {
console.log("Error when calling SQS");
console.log(err, err.stack); // an error occurred
reject(err);
} else {
resolve(data);
}
});
});
}
// calling the above and getting the result is now as simple as :
const result = await sendMessage("Hello World");
I have the following snippet of code below. It currently works, but I'm hoping to optimize/refactor it a bit.
Basically, it fetches JSON data, extracts the urls for a number of PDFs from the response, and then downloads those PDFs into a folder.
I'm hoping to refactor this code in order to process the PDFs once they are all downloaded. Currently, I'm not sure how to do that. There are a lot of nested asynchronous functions going on.
How might I refactor this to allow me to tack on another .then call before my error handler, so that I can then process the PDFs that are downloaded?
const axios = require("axios");
const moment = require("moment");
const fs = require("fs");
const download = require("download");
const mkdirp = require("mkdirp"); // Makes nested files...
const getDirName = require("path").dirname; // Current directory name...
const today = moment().format("YYYY-MM-DD");
function writeFile(path, contents, cb){
mkdirp(getDirName(path), function(err){
if (err) return cb(err)
fs.writeFile(path, contents, cb)
})
};
axios.get(`http://federalregister.gov/api/v1/public-inspection-documents.json?conditions%5Bavailable_on%5D=${today}`)
.then((res) => {
res.data.results.forEach((item) => {
download(item.pdf_url).then((data) => {
writeFile(`${__dirname}/${today}/${item.pdf_file_name}`, data, (err) => {
if(err){
console.log(err);
} else {
console.log("FILE WRITTEN: ", item.pdf_file_name);
}
})
})
})
})
.catch((err) => {
console.log("COULD NOT DOWNLOAD FILES: \n", err);
})
Thanks for any help you all can provide.
P.S. –– When I simply tack on the .then call right now, it fires immediately. This means that my forEach loop is non-blocking? I thought that forEach loops were blocking.
The current forEach will run synchronously, and will not wait for the asynchronous operations to complete. You should use .map instead of forEach so you can map each item to its Promise from download. Then, you can use Promise.all on the resulting array, which will resolve once all downloads are complete:
axios.get(`http://federalregister.gov/api/v1/public-inspection-documents.json?conditions%5Bavailable_on%5D=${today}`)
.then(processResults)
.catch((err) => {
console.log("COULD NOT DOWNLOAD FILES: \n", err);
});
function processResults(res) {
const downloadPromises = res.data.results.map((item) => (
download(item.pdf_url).then(data => new Promise((resolve, reject) => {
writeFile(`${__dirname}/${today}/${item.pdf_file_name}`, data, (err) => {
if(err) reject(err);
else resolve(console.log("FILE WRITTEN: ", item.pdf_file_name));
});
}))
));
return Promise.all(downloadPromises)
.then(() => {
console.log('all done');
});
}
If you wanted to essentially block the function on each iteration, you would want to use an async function in combination with await instead.