We Keep Coding

What is context.done doing in AWS Lambda? [duplicate]

I was following the guide here for setting up a presignup trigger.
However, when I used callback(null, event) my lambda function would never actually return and I would end up getting an error
{ code: 'UnexpectedLambdaException',
name: 'UnexpectedLambdaException',
message: 'arn:aws:lambda:us-east-2:642684845958:function:proj-dev-confirm-1OP5DB3KK5WTA failed with error Socket timeout while invoking Lambda function.' }
I found a similar link here that says to use context.done().
After switching it works perfectly fine.
What's the difference?
exports.confirm = (event, context, callback) => {
event.response.autoConfirmUser = true;
context.done(null, event);
//callback(null, event); does not work
}

Back in the original Lambda runtime environment for Node.js 0.10, Lambda provided helper functions in the context object: context.done(err, res) context.succeed(res) and context.fail(err).
This was formerly documented, but has been removed.
Using the Earlier Node.js Runtime v0.10.42 is an archived copy of a page that no longer exists in the Lambda documentation, that explains how these methods were used.
When the Node.js 4.3 runtime for Lambda was launched, these remained for backwards compatibility (and remain available but undocumented), and callback(err, res) was introduced.
Here's the nature of your problem, and why the two solutions you found actually seem to solve it.
Context.succeed, context.done, and context.fail however, are more than just bookkeeping – they cause the request to return after the current task completes and freeze the process immediately, even if other tasks remain in the Node.js event loop. Generally that’s not what you want if those tasks represent incomplete callbacks.
https://aws.amazon.com/blogs/compute/node-js-4-3-2-runtime-now-available-on-lambda/
So with callback, Lambda functions now behave in a more paradigmatically correct way, but this is a problem if you intend for certain objects to remain on the event loop during the freeze that occurs between invocations -- unlike the old (deprecated) done fail succeed methods, using the callback doesn't suspend things immediately. Instead, it waits for the event loop to be empty.
context.callbackWaitsForEmptyEventLoop -- default true -- was introduced so that you can set it to false for those cases where you want the Lambda function to return immediately after you call the callback, regardless of what's happening in the event loop. The default is true because false can mask bugs in your function and can cause very erratic/unexpected behavior if you fail to consider the implications of container reuse -- so you shouldn't set this to false unless and until you understand why it is needed.
A common reason false is needed would be a database connection made by your function. If you create a database connection object in a global variable, it will have an open socket, and potentially other things like timers, sitting on the event loop. This prevents the callback from causing Lambda to return a response, until these operations are also finished or the invocation timeout timer fires.
Identify why you need to set this to false, and if it's a valid reason, then it is correct to use it.
Otherwise, your code may have a bug that you need to understand and fix, such as leaving requests in flight or other work unfinished, when calling the callback.
So, how do we parse the Cognito error? At first, it seemed pretty unusual, but now it's clear that it is not.
When executing a function, Lambda will throw an error that the tasked timed out after the configured number of seconds. You should find this to be what happens when you test your function in the Lambda console.
Unfortunately, Cognito appears to have taken an internal design shortcut when invoking a Lambda function, and instead of waiting for Lambda to timeout the invocarion (which could tie up resources inside Cognito) or imposing its own explicit timer on the maximum duration Cognito will wait for a Lambda response, it's relying on a lower layer socket timer to constrain this wait... thus an "unexpected" error is thrown while invoking the timeout.
Further complicating interpreting the error message, there are missing quotes in the error, where the lower layer exception is interpolated.
To me, the problem would be much more clear if the error read like this:
'arn:aws:lambda:...' failed with error 'Socket timeout' while invoking Lambda function
This format would more clearly indicate that while Cognito was invoking the function, it threw an internal Socket timeout error (as opposed to Lambda encountering an unexpected internal error, which was my original -- and incorrect -- assumption).
It's quite reasonable for Cognito to impose some kind of response time limit on the Lambda function, but I don't see this documented. I suspect a short timeout on your Lambda function itself (making it fail more promptly) would cause Cognito to throw a somewhat more useful error, but in my mind, Cognito should have been designed to include logic to make this an expected, defined error, rather than categorizing it as "unexpected."

As an update the Runtime Node.js 10.x handler supports an async function that makes use of return and throw statements to return success or error responses, respectively. Additionally, if your function performs asynchronous tasks then you can return a Promise where you would then use resolve or reject to return a success or error, respectively. Either approach simplifies things by not requiring context or callback to signal completion to the invoker, so your lambda function could look something like this:
exports.handler = async (event) => {
// perform tasking...
const data = doStuffWith(event)
// later encounter an error situation
throw new Error('tell invoker you encountered an error')
// finished tasking with no errors
return { data }
}
Of course you can still use context but its not required to signal completion.

Jest unit testing - Async tests failing Timeout

I got the following error message "Timeout - Async callback was not invoked within timeout specified by jasmine.DEFAULT_TIMEOUT_INTERVAL." On top of that, I also received an error message stating that 0 assertions were executed while expecting 2 assertions.
I've tried extending the timeout to 10 seconds using jest.setTimeout(10000), which should be more than sufficient time to execute that code, but the problem persisted.
I know m.employeeGetAll() works because when I test my web app using the browser, I can see the list of employees in the view.
Here's how my test looks like
it('Lists all employees successfully', () => {
expect.assertions(2);
return m.employeeGetAll().then(result => { //m.employeeGetAll() returns a promise
expect(result).toBeDefined();
expect(result.length).toBe(3);
});
});

The problem I've discovered, was the way async code works.
What cannot be seen in the code snippet was the call to mongoose.connection.close(); at the very end of my test file.
This call must be done inside afterEach() or afterAll() function for Jest unit testing framework. Otherwise, the connection to the database will be closed before the tests could complete since all of the calls in my controller methods are asynchronous; This leads to no promise ever being returned and the code goes into timeout.
Since I'm using beforeAll() and afterAll(), to load data from database once before the start of all tests and to clear the database at the end of all tests, I've included the call to connect to DB using mongoose inside beforeAll() as well.
Hope this helps someone who's also stuck in my situation.

with async you have to call done
it('Lists all employees successfully', (done) => {
expect.assertions(2);
return m.employeeGetAll().then(result => { //m.employeeGetAll() returns a promise
expect(result).toBeDefined();
expect(result.length).toBe(3);
done();
});
});

mdg:validated-method _execute asynchronous issues

I'm running into problems with the validated method package in my app tests. I'm calling my methods through the _execute function in order to be able to pass a userId to simulate a logged-in user while testing. The problem is that my asserts right underneath that _execute are called before the method has a chance of completing. I know my test works though because it only happens sometimes, mostly because mongo isn't always returning results quite as fast.
I looked around and found a todos app that uses the _execute function in its tests. I can't get those tests to fail no matter how many times I rerun them, though.
This is an example of my test code.
describe('clients.add', function() {
it('should add an empty (draft) client', function() {
const res = clients_add._execute({ userId: 'CURRENTUSERID' }, { company_id: c1._id });
assert.instanceOf(res, Mongo.ObjectID, 'method returns the newly created clients ID');
const db_client = Clients.findOne(res);
assert.isTrue(db_client.draft, 'client is drafted');
assert.isDefined(db_client.created, 'there\'s a created date');
});
});
clients_add does quite a few permission checks and can therefor take a little while before completing. Rerunning this test 20 times will fail about 5 times and pass the other 15.
Shouldn't the _execute function be synchronous? How do I make it? What am I missing?

In server code, if you provide a callback to database modification functions like insert, it returns the created ID instantaneously, and runs the callback only once the database has acknowledged the write. If you don't provide a callback, the insert call is synchronous and throws an error if the operation fails. See more about this in Meteor docs.
It seems that you have provided an error-handling callback to the insert-function in your method code. This causes the inconsistent behavior, since the database might not actually have had time to do the write before you call findOne in your test. Also, this is redundant since if an error occurs in the insert, the method has already returned and the error is never shown to the user. It's better to simply omit the error-handling callback altogether:
return Clients.insert(new_client);

How to console.log an error with stack trace in node.js?

I've been trying to debug my node app to find the source of an error in my log that shows up only as "Error: Can't set headers after they are sent", with no trace information or any context.
As it happens, I think I've now fixed this... I am using connect-timeout and I was continuing processing a callback passed to an asynchronous network operation, which callback would eventually try to do a res.send(), despite req.timedout having been set to 'true' by connect-timeout during the network operation.
BUT I still can't understand why my log wasn't showing trace information for this error. Anywhere that an error is returned in my code I log it to the console with:
console.log(err);
If there is trace information available in the err object, and this seems to be placed in err.stack, shouldn't the above statement dump the whole content of err (including err.stack) to the console log? My understanding is that I wouldn't be losing any information by doing the above, compared e.g. to:
console.log(err.stack);
But posts like this one seem to suggest otherwise (though the linked post has now been updated).
I actually go further, and add some relevant text to help locate the error:
console.log('error in dodgyFunction:', err);
But despite this, I was still only getting "Error: Can't set headers after they are sent", without any of the context I'd put it. Would this be because this console error message is output within an external library (like express)? I thought that external libraries should send errors back to the main code to be dealt with accordingly?
Edit: here's an example of where I put my error and timeout checking, at the top of the callback function passed to the async operation:
var execFile = require('child_process').execFile;
execFile('dodgycommand', options, function(error, stdout, stderr) {
if (req.timedout) {
console.log('timeout detected whilst running dodgycommand, so aborting...');
return;
}
if (error) {
console.log('error running dodgycommand:', error);
res.sendStatus(400);
return;
}
// ... it's safe to continue ...
}
I basically follow this same pattern throughout.

I've just worked out what was going on, and I hope this will help others to avoid this beginner's error.
For some of my error logging I was using something like the following, using string concatenation to construct the error message:
console.log('error in function abc: ' + err + ' whilst doing xyz');
whereas elsewhere I was using something like the following, just passing the pieces of the error message as separate arguments to console.log:
console.log('error in function xyz:', err, 'whilst doing abc');
I now see that these give different results!
The former must stringify err so that it can be concatenated with the other parts of the message, and according to this, in doing so it just uses the message part.
However, in the latter form the err object must be processed by console.log unadulterated, and dumped as a whole.
This explains why I was sometimes not seeing the whole content of the error, as I was expecting, and other times I was.
As for console log messages put there by other libraries, something else to check is that you're not filtering out the 'stack' parts of the log messages in your log viewer... turns out that I was (in order to save on log quota... am using papertrail)... d'oh. I was doing so by filtering out any lines starting with ____at (four spaces followed by 'at'), for example ____at Request.self.callback.

I've installed n now and I can confirm the following:
Node 4.0.0
Using console.log(err) prints only the error message.
Node 7.7.0 (latest)
Using console.log(err) prints the error message and the full stack.
I've confirmed that this behavior changed on version 6.0.0. So, if you use an older version, I suggest that you update your Node.js or use console.log(err.stack) instead to print the full stack.

Your pattern looks generally common, though I'll say that as a rule I don't like it, more on that in a second.
As for your main question, it's really hard to answer it based on what you've provided. If you show the actual code rather than the "I generally follow this pattern", it might help. But it's equally possible that the error was being thrown somewhere that you weren't expecting, and so your console.log wasn't getting called at all.
Seems like you're looking for best practices, so I'll give you what I think is the best I have found so far.
First, don't use console.log for logging. It's not horrible, but you can do much, much better. My favorite is to use morgan as middleware for logging request information, and debug for application logging.
With debug you can set up custom log levels and listen to whatever level you want with whatever level of granularity you want. It's all controlled by setting the DEBUG environment variable, and in production you can redirect to file or whatever other destination you want. Further, many node modules (including Express and Connect) use Debug as their logger under the hood, so by tweaking your DEBUG variable you can see as much or little of their inner logging as you want. very useful for figuring out what went wrong where.
Second, as I said I don't use the pattern you have at all when it comes to routing. I've found it's easy to accidentally send headers more than once if I am not careful, so my middleware always return next() and responses are only sent in actual handlers that I can be sure fire only once. When it comes to error, I always pass next(e) which I can then handle in an error handler function. I also created the praeter library to provide standard errors based on web status codes and a generic error handler.
The pattern looks something like this:
// middleware function to put something on the request object
app.use((req, res, next) => {
MyModel.doSomething((e, thing) => {
if (e) return next(e);
if (!thing) return next(new NotFound()); // NotFound is an error in praeter that equates to a 404.
req.thing = thing;
return next();
});
});
Then later
// log in here is a reference to my debug configured log object
app.use((err, req, res, next) => {
log.error(err);
log.error(err.stack);
return res.status(err.statusCode || 500).send(err.message)
});
Note this is a simple example of a final error handler. I often have several of these where I might handle different error codes differently, depending on application needs.

Process timeout | Amazon Lambda to Firebase

i've written code in node.js and my data is on Firebase. The problem i'm facing is that my code never exits. I've done it like this one Link
The problem is that firebase referance/listener never become null and therefore my function never exits. I tried using firebase.database().goOffline() but it didn't work.
On my local machine i forcefully stopped the process using process.exit(0), but when i deployed my code on AWS lambda, it doesn't return any response/call back and exits (giving error message "Process exited before completing request")
I also added wait of 5-10 seconds after invoking callback in lambda and then forcefully exited the process, but it didn't help either.
How to fix this issue? Please help.

Your going through crisis that any new lambda user has gone.
As suggested, you can use context.done for stopping.
However, this is not recommended as this is only possible due to historic runtime versions of nodejs.
why this timeout happens?
Your lambda may get to the last line of your code and still keep running. Well, it is actually waiting for something - for the event loop to be empty.
what this means?
In nodejs, when you make an async operation and register a callback function to be executed once the operation is done, the registration sort of happens in the event loop.
In one line, it's the event loop that knows which callback function to execute when an async operation ends. But that's to another thread :)
back to Lambda
Given the above information, it follows that lambda should not halt before empty event loop is reached - as this means some follow-up procedure will not execute after some async operation returns.
What if you still need to halt the execution manually? regardless of the event loop status?
At the beginning of the function, execute:
context.callbackWaitsForEmptyEventLoop = false
And then use the third parameter you get in the handler signature. Which is the callback.
the callback parameter
It is a function which you call when you want to end the execution.
If you call it with no parameters, or with the first parameter as null and text as second parameter - it is considered as a successful invocation.
To fail the lambda execution, you can call the callback function with some non-null value as the first parameter.

Add this line at the beginning of your handler function and then you should be able to use the callback without issue:
function handler (event, context, callback) {
context.callbackWaitsForEmptyEventLoop = false // Add this line
}

Setting callbackWaitsForEmptyEventLoop to false should only be your last resort if nothing else works for you, as this might introduce worse bugs than the problem you're trying to solve here.
This is what I do instead to ensure every call has firebase initialized, and deleted before exiting.
// handlerWithFirebase.js
const admin = require("firebase-admin");
const config = require("./config.json");
function initialize() {
return admin.initializeApp({
credential: admin.credential.cert(config),
databaseURL: "https://<your_app>.firebaseio.com",
});
}
function handlerWithFirebase(func) {
return (event, context, callback) => {
const firebase = initialize();
let _callback = (error, result) => {
firebase.delete();
callback(error, result);
}
// passing firebase into your handler here is optional
func(event, context, _callback, firebase /*optional*/);
}
}
module.exports = handlerWithFirebase;
And then in my lambda handler code
// myHandler.js
const handlerWithFirebase = require("../firebase/handler");
module.exports.handler = handlerWithFirebase(
(event, context, callback, firebase) => {
...
});

Calling callbackfunciton and then process.exit(0) didn't help in my case. goOffline() method of firebase didn't help either.
I fixed the issue calling context.done(error, response) (instead of callback method). Now, my code is working.
Still, if any one have better solution, kindly post here. It may help some one else :)