We Keep Coding

Node.js 12.x AWS lambda does not return using firebase-admin realtime database

I've been stuck on this issue for days with little to no progress, please help if you can!
I have a Node.js (v12) AWS Lambda that needs to pull data from my Firebase realtime database and process each record into a Redis cache if it doesn't exist already. The function starts but never finishes and instead I receive Task timed out after 180.10 seconds from AWS.
Things I've tried:
Using exports.handler = async function(event) versus exports.handler = function(event, context, callback);
For the synchronous attempt above I've tried using context.callbackWaitsForEmptyEventLoop = false versus not;
Using promises versus cascading functions versus stitching a bunch of .then()'s together;
Using the firebase-admin versus the https module in-conjunction with the Firebase REST API;
Using settimeout to fire the callback later versus not;
Setting the GOOGLE_APPLICATION_CREDENTIALS environment variable to my service account credentials versus referencing the file directly in the code;
I've even beefed-up the memory and timeout of the Lambda itself to the maximum it can go, as well as cut down the data I want to pull from Firebase to just 1 record.
Responses I've had as-per the attempts above:
AWS (the most frequent): Task timed out after 180.10 seconds;
AWS (.then stitching approach): Function completed successfully (but no data was actually processed);
Node HTTPS (REST API approach): ETIMEDOUT or ECONNREFUSED;
Below is where I'm up to and still no luck. I have cut-out the caching code since I know that works fine. The settimeout's you see were my last resorts before reaching out here.
const admin = require("firebase-admin");
admin.initializeApp({
credential: admin.credential.applicationDefault(),
databaseURL: "https://{projectName}.firebaseio.com"
});
var result = [];
exports.handler = (event, context, callback) => {
context.callbackWaitsForEmptyEventLoop = false;
try {
admin.database().ref("data").orderByChild("timestamp").limitToLast(1).once("value", snapshot => {
if (snapshot.exists()) {
console.log('snapshot exists...');
let posts = snapshot.val();
result = Object.keys(posts);
}
setTimeout(() => {
admin.database().goOffline();
admin.app().delete();
callback(null, `Success! ${JSON.stringify(result)}`); // <-- NEVER RETURNS
}, 2000);
}, error => {
setTimeout(() => {
admin.database().goOffline();
admin.app().delete();
callback(error); // <-- NEVER RETURNS
}, 2000);
});
} catch (error) {
setTimeout(() => {
admin.database().goOffline();
admin.app().delete();
callback(error); // <-- NEVER RETURNS
}, 2000);
}
};

It doesn't appear you are storing or using the setTimeout on the root level of your function. You should store it so the call back function can keep running since it only exists while it's in scope. Doing this also requires you to bind the object so you have a self-reference if you decide to push it into an array for multiple callbacks
var result = [];
var timeOut;
//...
timeOut = setTimeout(() => {
admin.database().goOffline();
admin.app().delete();
callback(error);
}.bind(this), 2000);
Source: MSDN Function.prototype.bind()
If Binding is not a solution and you want a blocking method, you might be interested in a delay function, it behaves the same as a setTimeout but works with promises
function sleep(ms) {
return new Promise(resolve => setTimeout(resolve, ms));
}
// Async function solution
await sleep(2000);
console.log('Two seconds later, showing sleep in a loop...');
// non-Async solution
sleep(2000)
.then(()=> {
admin.database().goOffline();
admin.app().delete();
callback(error);
})
.catch(e => console.log(e));

Is creating a new promise with a async function call bad practice?

Snippets are from a node.js and mongoDB CRUD application.Github repo for full code. The code is working fine but unsure if my structure and use of promises and async await are bad practice.
handlers._newbies = {};
handlers._newbies.post = (parsedReq, res) => {
const newbie = JSON.parse(parsedReq.payload);
databaseCalls.create(newbie)
.then((result) => {
res.writeHead(200,{'Content-Type' : 'application/json'});
const resultToString = JSON.stringify(result.ops[0]);
res.write(resultToString);
res.end();
})
.catch(err => console.log(err));
};
const databaseCalls = {};
databaseCalls.create = (newbie) => {
return new Promise(async (resolve, reject) => {
try {
const client = await MongoClient.connect('mongodb://localhost:27017', { useNewUrlParser: true });
console.log("Connected correctly to server");
const db = client.db('Noob-List');
const result = await db.collection('newbies').insertOne(newbie);
client.close();
resolve(result);
} catch(err) {
console.log(err);
}
});
};
When the node server gets a POST request with the JSON payload, it calls the handlers._newbies.post handler which takes the payload and passed it to the
const newbie = JSON.parse(parsedReq.payload);
databaseCalls.create(newbie)
call. I want this database call to return a promise that holds the result of the db.collection('newbies').insertOne(newbie);
call. I was having trouble doing this with just returning the promise returned by the insertOne because after returning I cant call client.close();.
Again maybe what I have done here is fine but I haven't found anything online about creating promises with promises in them. Thank you for your time let me know what is unclear with my question.

It is considered an anti-pattern to be wrapping an existing promise in a manually created promise because there's just no reason to do so and it creates many an opportunities for error, particular in error handling.
And, in your case, you have several error handling issues.
If you get an error anywhere in your database code, you never resolve or reject the promise you are creating. This is a classic problem with the anti-pattern.
If you get an error after opening the DB, you don't close the DB
You don't communicate back an error to the caller.
Here's how you can do your .create() function without the anti-pattern and without the above problems:
databaseCalls.create = async function(newbie) {
let client;
try {
client = await MongoClient.connect('mongodb://localhost:27017', { useNewUrlParser: true });
console.log("Connected correctly to server");
const db = client.db('Noob-List');
return db.collection('newbies').insertOne(newbie);
} catch(err) {
// log error, but still reject the promise
console.log(err);
throw err;
} finally {
// clean up any open database
if (client) {
client.close();
}
}
}
Then, you would use this like:
databaseCalls.create(something).then(result => {
console.log("succeeded");'
}).catch(err => {
console.log(err);
});
FYI, I also modified some other things:
The database connection is closed, even in error conditions
The function returns a promise which is resolved with the result of .insertOne() (if there is a meaningful result there)
If there's an error, the returned promise is rejected with that error
Not particularly relevant to your issue with promises, but you will generally not want to open and close the DB connection on every operation. You can either use one lasting connection or create a pool of connections where you can fetch one from the pool and then put it back in the pool when done (most DBs have that type of feature for server-side work).

save() doesn't work on mongoose.Schema

const Data = require('./Models/Data');
...
let example = new Data( sample );
example.save( function ( err ){
console.log('test);
if ( err ) {
console.log('Error saving Data. 'Error: ', err);
}
});
Any ideas why save() callback function never runs? I mean, the "test" text doesn't show up, while "example" is created just like it should (I mean, when I print it, it looks ok).
Any ideas? TIA

Mongoose async save() function works with a function, which means you don't need to pass it a callback function, but rather use then/catch pattern:
const Data = require('./Models/Data');
...
let example = new Data( sample );
example.save()
.then(() => {
console.log('test);
})
.catch((err) => {
console.log('Error saving Data. 'Error: ', err);
});
see more here

As Nir Levy has already said: the then call can be an alternate way of saving the document.
You can also try supplying the second argument in the save's callback as:
example.save((err, doc) => {
console.log('test');
if (err) {
console.log('Error while saving data: ', err);
} else {
console.log('document is: ', doc);
}
});
Also note that you missed a closing quotemark on the console.log() inside example.save callback
Can you also make sure that your mongodb server is running and you are connected to the mongodb server? if that maybe causing the problem?

This typically occurs when you haven't connected to the database. You can create the model object but none of the Mongoose functions that operate on the database works. They fail silently instead. Make sure you connect with the connect function. You can also listen to the connection and error events to see that you actually get connected:
const dburl = `mongodb://localhost/testdb`;
mongoose.connect(dburl, { useMongoClient: true });
mongoose.connection.on('connected', () => { console.log(`Mongoose connected to ${dburl}`); });
mongoose.connection.on('error', (err) => { console.log(`Mongoose connection error: ${err}`); });
mongoose.connection.on('disconnected', () => { console.log('Mongoose disconnected'); });
Pay attention to the terminal when you start the app and look for Mongoose connected. Finally, you can turn on debug mode in Mongoose to see what's actually going on behind the scenes:
mongoose.set('debug', true);
You can add it right below the connect call, for example.
You don't have to use the promise method by the way. You can use the callback version that you're already using. Just make sure to correct the error with the missing ' in the console.log.

How to access data from function (node.js) [duplicate]

I’m new to node and mongo after 15 years of VB6 and MySql. I’m sure this is not what my final program will use but I need to get a basic understanding of how to call a function in another module and get results back.
I want a module to have a function to open a DB, find in a collection and return the results. I may want to add a couple more functions in that module for other collections too. For now I need it as simple as possible, I can add error handlers, etc later. I been on this for days trying different methods, module.exports={… around the function and with out it, .send, return all with no luck. I understand it’s async so the program may have passed the display point before the data is there.
Here’s what I’ve tried with Mongo running a database of db1 with a collection of col1.
Db1.js
var MongoClient = require('mongodb').MongoClient;
module.exports = {
FindinCol1 : function funk1(req, res) {
MongoClient.connect("mongodb://localhost:27017/db1", function (err,db) {
if (err) {
return console.dir(err);
}
var collection = db.collection('col1');
collection.find().toArray(function (err, items) {
console.log(items);
// res.send(items);
}
);
});
}
};
app.js
a=require('./db1');
b=a.FindinCol1();
console.log(b);
Console.log(items) works when the 'FindinCol1' calls but not console.log(b)(returns 'undefined') so I'm not getting the return or I'm pasted it by the time is returns. I’ve read dozens of post and watched dozens of videos but I'm still stuck at this point. Any help would be greatly appreciated.

As mentioned in another answer, this code is asynchronous, you can't simply return the value you want down the chain of callbacks (nested functions). You need to expose some interface that lets you signal the calling code once you have the value desired (hence, calling them back, or callback).
There is a callback example provided in another answer, but there is an alternative option definitely worth exploring: promises.
Instead of a callback function you call with the desired results, the module returns a promise that can enter two states, fulfilled or rejected. The calling code waits for the promise to enter one of these two states, the appropriate function being called when it does. The module triggers the state change by resolveing or rejecting. Anyways, here is an example using promises:
Db1.js:
// db1.js
var MongoClient = require('mongodb').MongoClient;
/*
node.js has native support for promises in recent versions.
If you are using an older version there are several libraries available:
bluebird, rsvp, Q. I'll use rsvp here as I'm familiar with it.
*/
var Promise = require('rsvp').Promise;
module.exports = {
FindinCol1: function() {
return new Promise(function(resolve, reject) {
MongoClient.connect('mongodb://localhost:27017/db1', function(err, db) {
if (err) {
reject(err);
} else {
resolve(db);
}
}
}).then(function(db) {
return new Promise(function(resolve, reject) {
var collection = db.collection('col1');
collection.find().toArray(function(err, items) {
if (err) {
reject(err);
} else {
console.log(items);
resolve(items);
}
});
});
});
}
};
// app.js
var db = require('./db1');
db.FindinCol1().then(function(items) {
console.info('The promise was fulfilled with items!', items);
}, function(err) {
console.error('The promise was rejected', err, err.stack);
});
Now, more up to date versions of the node.js mongodb driver have native support for promises, you don't have to do any work to wrap callbacks in promises like above. This is a much better example if you are using an up to date driver:
// db1.js
var MongoClient = require('mongodb').MongoClient;
module.exports = {
FindinCol1: function() {
return MongoClient.connect('mongodb://localhost:27017/db1').then(function(db) {
var collection = db.collection('col1');
return collection.find().toArray();
}).then(function(items) {
console.log(items);
return items;
});
}
};
// app.js
var db = require('./db1');
db.FindinCol1().then(function(items) {
console.info('The promise was fulfilled with items!', items);
}, function(err) {
console.error('The promise was rejected', err, err.stack);
});
Promises provide an excellent method for asynchronous control flow, I highly recommend spending some time familiarizing yourself with them.

Yes, this is an async code and with a return you will get the MongoClient object or nothing, based on where you put.
You should use a callback parameter:
module.exports = {
FindinCol1 : function funk1(callback) {
MongoClient.connect("mongodb://localhost:27017/db1", function (err,db) {
if (err) {
return console.dir(err);
}
var collection = db.collection('col1');
collection.find().toArray(function (err, items) {
console.log(items);
return callback(items);
});
});
}
};
Pass a callback function to FindinCol1:
a.FindinCol1(function(items) {
console.log(items);
});
I suggest you to check this article:
https://docs.nodejitsu.com/articles/getting-started/control-flow/what-are-callbacks

Storing data from MongoDB in Javascript Objects in Node.js [duplicate]

I’m new to node and mongo after 15 years of VB6 and MySql. I’m sure this is not what my final program will use but I need to get a basic understanding of how to call a function in another module and get results back.
I want a module to have a function to open a DB, find in a collection and return the results. I may want to add a couple more functions in that module for other collections too. For now I need it as simple as possible, I can add error handlers, etc later. I been on this for days trying different methods, module.exports={… around the function and with out it, .send, return all with no luck. I understand it’s async so the program may have passed the display point before the data is there.
Here’s what I’ve tried with Mongo running a database of db1 with a collection of col1.
Db1.js
var MongoClient = require('mongodb').MongoClient;
module.exports = {
FindinCol1 : function funk1(req, res) {
MongoClient.connect("mongodb://localhost:27017/db1", function (err,db) {
if (err) {
return console.dir(err);
}
var collection = db.collection('col1');
collection.find().toArray(function (err, items) {
console.log(items);
// res.send(items);
}
);
});
}
};
app.js
a=require('./db1');
b=a.FindinCol1();
console.log(b);
Console.log(items) works when the 'FindinCol1' calls but not console.log(b)(returns 'undefined') so I'm not getting the return or I'm pasted it by the time is returns. I’ve read dozens of post and watched dozens of videos but I'm still stuck at this point. Any help would be greatly appreciated.

As mentioned in another answer, this code is asynchronous, you can't simply return the value you want down the chain of callbacks (nested functions). You need to expose some interface that lets you signal the calling code once you have the value desired (hence, calling them back, or callback).
There is a callback example provided in another answer, but there is an alternative option definitely worth exploring: promises.
Instead of a callback function you call with the desired results, the module returns a promise that can enter two states, fulfilled or rejected. The calling code waits for the promise to enter one of these two states, the appropriate function being called when it does. The module triggers the state change by resolveing or rejecting. Anyways, here is an example using promises:
Db1.js:
// db1.js
var MongoClient = require('mongodb').MongoClient;
/*
node.js has native support for promises in recent versions.
If you are using an older version there are several libraries available:
bluebird, rsvp, Q. I'll use rsvp here as I'm familiar with it.
*/
var Promise = require('rsvp').Promise;
module.exports = {
FindinCol1: function() {
return new Promise(function(resolve, reject) {
MongoClient.connect('mongodb://localhost:27017/db1', function(err, db) {
if (err) {
reject(err);
} else {
resolve(db);
}
}
}).then(function(db) {
return new Promise(function(resolve, reject) {
var collection = db.collection('col1');
collection.find().toArray(function(err, items) {
if (err) {
reject(err);
} else {
console.log(items);
resolve(items);
}
});
});
});
}
};
// app.js
var db = require('./db1');
db.FindinCol1().then(function(items) {
console.info('The promise was fulfilled with items!', items);
}, function(err) {
console.error('The promise was rejected', err, err.stack);
});
Now, more up to date versions of the node.js mongodb driver have native support for promises, you don't have to do any work to wrap callbacks in promises like above. This is a much better example if you are using an up to date driver:
// db1.js
var MongoClient = require('mongodb').MongoClient;
module.exports = {
FindinCol1: function() {
return MongoClient.connect('mongodb://localhost:27017/db1').then(function(db) {
var collection = db.collection('col1');
return collection.find().toArray();
}).then(function(items) {
console.log(items);
return items;
});
}
};
// app.js
var db = require('./db1');
db.FindinCol1().then(function(items) {
console.info('The promise was fulfilled with items!', items);
}, function(err) {
console.error('The promise was rejected', err, err.stack);
});
Promises provide an excellent method for asynchronous control flow, I highly recommend spending some time familiarizing yourself with them.

Yes, this is an async code and with a return you will get the MongoClient object or nothing, based on where you put.
You should use a callback parameter:
module.exports = {
FindinCol1 : function funk1(callback) {
MongoClient.connect("mongodb://localhost:27017/db1", function (err,db) {
if (err) {
return console.dir(err);
}
var collection = db.collection('col1');
collection.find().toArray(function (err, items) {
console.log(items);
return callback(items);
});
});
}
};
Pass a callback function to FindinCol1:
a.FindinCol1(function(items) {
console.log(items);
});
I suggest you to check this article:
https://docs.nodejitsu.com/articles/getting-started/control-flow/what-are-callbacks