We Keep Coding

How to require directories using async / await

I am trying to load data which is in javascript files into an object.
All the files are like this:
module.exports = {
test: ‘qwerty’
}
I’m using require, but I have to load several directories so need to do the loads one at a time.
I tried wrapping it in a promise:
function load(fileOrDirPath) {
return new Promise(function(resolve, reject) {
let data;
debug(`Requiring: [${fileOrDirPath}]`);
try {
data = require(fileOrDirPath);
}
catch(e) {
return reject(e);
}
debug(`Loaded data: [${JSON.stringify(data, 0, 2)}]`);
return resolve(data);
});
}
I use the function:
const dirPath = ‘redacted’
const data = await load(dirPath);
debug(`Loaded: [${JSON.stringify(data, 0, 2)}]`);
And the logs show that the data is loaded inside the function. However outside the data is always null.
Why doesn’t the function await?
I tried looking on npm for a module but couldn’t find any.
How can I load a directory of javascript files recursively into an object?

Putting s3 object data into an array - node

I'm trying to fetch data from an S3 object and put it into an array. I plan to map through this array and display the data on a React front end in grid/list whatever. I'm struggling with nested functions though, so I'd appreciate some help.
const dataFromS3 = async (bucket, file) => {
let lines = [];
const options = {
Bucket: bucket,
Key: file
};
s3.getObject(options, (err, data) => {
if (err) {
console.log(err);
} else {
let objectData = data.Body.toString('utf-8');
lines.push(objectData);
console.log(lines);
return lines;
}
});
};
Formatting is a bit weird but this is my function to get data from s3. I want to take the output of this function in the form of an array and pass it to my '/' route which I'm testing:
app.get('/', async (req, res, next) => {
try {
let apolloKey = await dataFromS3(s3Bucket, apolloKeywords);
res.send(apolloKey);
} catch (err) {
console.log('Error: ', err);
}
});
It seems that the return value of lines in the s3.getObject function needs to be returned within the first function so that I can access it in app.get but I can't seem to do it after some attempts. The value in lines turns into an empty array if I return it at the end of datafromS3() and I can't find a way to return it. I've tried using promises also using a method found here - How to get response from S3 getObject in Node.js? but I get a TypeError: Converting Circular Structure to JSON...
Thank you

You need to make your dataFromS3 func like htis. You were not returning anything from that. AWS also provided promise based function.
const dataFromS3 = async (bucket, file) => {
const lines = [];
const options = {
"Bucket": bucket,
"Key": file
};
const data = await s3.getObject(options).promise();
const objectData = data.Body.toString("utf-8");
lines.push(objectData); // You might need to conversion here using JSON.parse(objectData);
console.log(lines);
return lines;
};

Write to S3 bucket using Async/Await in AWS Lambda

I have been using the code below (which I have now added await to) to send files to S3. It has worked fine with my lambda code but as I move to transfer larger files like MP4 I feel I need async/await.
How can I fully convert this to async/await?
exports.handler = async (event, context, callback) => {
...
// Copy data to a variable to enable write to S3 Bucket
var result = response.audioContent;
console.log('Result contents ', result);
// Set S3 bucket details and put MP3 file into S3 bucket from tmp
var s3 = new AWS.S3();
await var params = {
Bucket: 'bucketname',
Key: filename + ".txt",
ACL: 'public-read',
Body: result
};
await s3.putObject(params, function (err, result) {
if (err) console.log('TXT file not sent to S3 - FAILED'); // an error occurred
else console.log('TXT file sent to S3 - SUCCESS'); // successful response
context.succeed('TXT file has been sent to S3');
});

You only await functions that return a promise. s3.putObject does not return a promise (similar to most functions that take a callback). It returns a Request object. If you want to use async/await, you need to chain the .promise() method onto the end of your s3.putObject call and remove the callback (https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/Request.html#promise-property)
try { // You should always catch your errors when using async/await
const s3Response = await s3.putObject(params).promise();
callback(null, s3Response);
} catch (e) {
console.log(e);
callback(e);
}

As #djheru said, Async/Await only works with functions that return promises.
I would recommend creating a simple wrapper function to assist with this problem.
const putObjectWrapper = (params) => {
return new Promise((resolve, reject) => {
s3.putObject(params, function (err, result) {
if(err) reject(err);
if(result) resolve(result);
});
})
}
Then you could use it like this:
const result = await putObjectWrapper(params);
Here is a really great resource on Promises and Async/Await:
https://javascript.info/async

Why does object property not exist when typescript transpiles but it does at run time?

Typescript is failing to transpile my code as it believes the objects method doesn't exist.
At run time, when the promise has resolved, the method does exists. It just doesn't exist at the time the typescript is trying to transpile because the Promise is returned.
My function that returns the promise which resolves the oauthclient:
public generateOauthClient(){
return new Promise((resolve, reject) => {
// Load client secrets from a local file.
fs.readFile(appConfig.youtube.keyFilename, 'utf8', function processClientSecrets(err, content) {
if(err) {
return reject('Error loading client secret file: ' + err);
}
const credentials = JSON.parse(content);
const clientSecret = credentials.installed.client_secret;
const clientId = credentials.installed.client_id;
const redirectUrl = credentials.installed.redirect_uris[0];
const auth = new googleAuth();
const oauth2Client = new auth.OAuth2(clientId, clientSecret, redirectUrl);
resolve(oauth2Client);
})
});
}
And then I try call that elsewhere in my code
const oauth2Client = await this.generateOauthClient();
oauth2Client.setCredentials({
access_token: JSON.parse(tokens).access_token,
refresh_token: JSON.parse(tokens).refresh_token
});
oauth2Client.refreshAccessToken((err, tokens) => {
if(err) return reject(err)
this.storeToken(tokens).then(()=>{
resolve('Successfully updated tokens')
}).catch((err2)=> {
reject(err2)
})
});
Then I get the typescript error: Property 'setCredentials' does not exist on type '{}'
If I change my generateOauthClient function to return a callback as opposed to returning a promise, typescript transpiles fine.
How do I get around this so I can use promises instead of callbacks?

You need to tell Typescript what your Promise will return. This will allow Typescript to check what you do with the result in either syntax you use (async/await or .then):
new Promise<auth.OAuth2>((resolve, reject) => { .. } )

Retrieve paginated data recursively using promises

I'm using a function which returns data in a paginated form. So it'll return max 100 items and a key to retrieve the next 100 items. I want to retrieve all the items available.
How do I recursively achieve this? Is recursion a good choice here? Can I do it any other way without recursion?
I'm using Bluebird 3x as the promises library.
Here is a snippet of what I'm trying to achieve:
getEndpoints(null, platformApplication)
.then(function(allEndpoints) {
// process on allEndpoints
});
function getEndpoints(nextToken, platformApplication) {
var params = {
PlatformApplicationArn: platformApplication
};
if (nextToken) {
params.NextToken = nextToken;
}
return sns.listEndpointsByPlatformApplicationAsync(params)
.then(function(data) {
if (data.NextToken) {
// There is more data available that I want to retrieve.
// But the problem here is that getEndpoints return a promise
// and not the array. How do I chain this here so that
// in the end I get an array of all the endpoints concatenated.
var moreEndpoints = getEndpoints(data.NextToken, platformApplication);
moreEndpoints.push.apply(data.Endpoints, moreEndpoints);
}
return data.Endpoints;
});
}
But the problem is that if there is more data to be retrieved (see if (data.NextToken) { ... }), how do I chain the promises up so that in the end I get the list of all endpoints etc.

Recursion is probably the easiest way to get all the endpoints.
function getAllEndpoints(platformApplication) {
return getEndpoints(null, platformApplication);
}
function getEndpoints(nextToken, platformApplication, endpoints = []) {
var params = {
PlatformApplicationArn: platformApplication
};
if (nextToken) {
params.NextToken = nextToken;
}
return sns.listEndpointsByPlatformApplicationAsync(params)
.then(function(data) {
endpoints.push.apply(endpoints, data.Endpoints);
if (data.NextToken) {
return getEndpoints(data.NextToken, platformApplication, endpoints);
} else {
return endpoints;
}
});
}

For a more general purpose example of recursively getting and returning data from a paginated endpoint, here is what I came up with:
getData(page, dataArray) {
return new Promise((resolve, reject) => {
getEndpointHere(
{
page,
pageSize: 50,
},
(err, result) => {
if (err)
return console.error("there was a problem retrieving your data");
dataArray = dataArray.concat(result);
if (result.length < 50) {
resolve(dataArray);
} else {
resolve(getData(page + 1, dataArray));
}
}
);
});
}
getData(1, [])
.then((res) => {
console.log("SEVERAL PAGES OF DATA", res);
})
This example is employing a callback -- (err, result) -- however, that logic could be extracted out to handle the response from the endpoint. The resource I was consuming does not return a cursor or "next" token indicating if there are more records, so I used the logic of if the response has less than 50 records as the basis for continuing to request more data.