I recently started working with the sqlite3 package in Node.js. Primarily I'm using the run, all and get functions. Since I like async/await more and those functions are taking callbacks, I used promisify from the util package to make them use promises that I can await.
import sqlite from 'sqlite3'
import { promisify } from 'util'
const db = new sqlite.Database('./myDatabase.db', (err) => {
if (err) {
return console.error(err.message);
}
console.log('Connected to the database.');
});
db.run = promisify(db.run)
db.all = promisify(db.all)
db.get = promisify(db.get)
This works with async/await however a problem shows up when I try to insert or update data using run().
const changedRows = await db.run("UPDATE races SET is_open = false WHERE code = ?", [racecode])
When I want to fetch the inserted ID or the affected rows I get undefined. That makes sense because the run function usually has no argument for a result thus resolve yields undefined.
From the docs:
db.run(`INSERT INTO langs(name) VALUES(?)`, ['C'], function(err) {
if (err) {
return console.log(err.message);
}
// get the last insert id
console.log(`A row has been inserted with rowid ${this.lastID}`);
});
The question is: Can I even get this.lastID from a promisified run ? I know there are some wrapper packages for sqlite3 out there, but I wanna try to use as few packages as possible before adding new ones.
Related
I have a simple function which deletes a product entry from the database, and now I'm trying to delete the image file of that product as well. I checked the Node.js file system docs and found 2 functions which deal with that - fs.unlink(path, callback) and fs.unlinkSync(path). I understand that the first one is asynchronous and that the second one is synchronous, but I'm still not quite sure which one should I use and why.
module.exports.deleteProduct = async (req, res, next) => {
let productId = req.body.productId
try {
let product = await Product.destroy({
where: {
id: productId
}
})
res.status(200).json({
product: product
})
} catch (e) {
console.log(e)
res.status(500)
}
}
Some code and an idea for you:
As others have already said, async is better than sync, so you won't end up blocking, even though, unless your API volume is extremely high, it probably won't matter, as indicated in another answer.
You can use the fs promises API via
const fs = require('fs').promises; //es5 OR
import { promises as fs } from 'fs'; //es6
to use the async (non-blocking) API with a one-liner await.
Special note: you may not want your API request to fail if you failed to unlink the directory, as you did in fact delete the product from the database.
// make sure you are using the promise API from fs
const fs = require('fs').promises;
module.exports.deleteProduct = async (req, res, next) => {
let productId = req.body.productId
try {
let product = await Product.destroy({
where: {
id: productId
}
})
try {
await fs.unlink('the/path/to/the/product/image');
} catch {
// you may want to handle a failure to delete separately
}
res.status(200).json({product: product})
} catch (e) {
console.log(e)
res.status(500)
}
}
If your server OS is Linux or some other UNIX derivative with a local file system, both .unlinkSync() and .unlink() run quickly: the OS-level unlinking operation is designed to complete quickly and predictably. So, if you use the blocking .unlinkSync() version you won't do much harm, especially if your unlinking is infrequent.
That being said, if you can use the asynchronous version it's a good practice to do so.
It looks like you can; you can call res.status()... from within a callback or after an await.
Don't Block the event loop in Node Js
The synchronous methods blocks the event loop unnecessarily ,which affects your application performance .always use async methods ,wherever possible.
or if you want to use it with awaitoperation (pseudo sync) ,you can do something like below ,by wrapping it within promise
const fs=require("fs");
function unlinkPromise(file)
{
return new Promise((resolve,reject)=>{
fs.unlink(file,(err,data)=>{
if(err)
{
reject(err);
}
resolve(data);
})
})
}
async function data()
{
console.log(await unlinkPromise("file"));
}
I hope this is not a silly question. I have a homepage route that loads up a lot of mongo databases, and I originally had it loop through the mongo databases and add them to an array that was rendered to a webpage. However, the databases have become more complicated and they need to be populated, so I can no longer use a loop to accomplish this, and needed to refactor to getting the databases individually. However, I seem to be having problems with the variable's scopes, as they are always returned as empty outside of the .find function.
My original code was this:
const collections = [User, Ticket, Client, Job, Transaction];
let endCollections = [];
for (let i = 0; i < collections.length; i++){
await collections[i].find({}, function(err, foundCollection){
if (err) {
console.log(err);
} else {
endCollections[i] = foundCollection;
}
});
}
res.render("dashboard", {transactions: endCollections[4], clients: endCollections[2], tickets: endCollections[1], jobs: endCollections[3]});
And this worked fine. But I need to populate the individual databases, so this was no longer useful. I rewrote it out to populate, but I am having problems changing the global variables inside of the functions. Here is the new code I am trying:
let transactions = [],
clients = [],
jobs = [],
tickets = [];
await Transaction.find({}).populate("job").populate("client").populate("deposited_by_user").exec(function(err, foundTransactions){
if(err){
console.log(err)
} else {
for (let i = 0; i < foundTransactions.length; i++){
foundTransactions[i]["transaction_info"]["new_amount"] = numberWithCommas(foundTransactions[i]["transaction_info"]["amount"]);
}
}
transactions = foundTransactions;
});
await Client.find({}).populate("transactions").populate("jobs").exec(function(err, foundClients){
if(err){
console.log(err)
}
clients = foundClients;
});
await Ticket.find({}).populate("created_by").populate("assigned_user").populate("completed_by_user").exec(function(err, foundTickets){
if(err){
console.log(err)
}
tickets = foundTickets;
});
await Job.find({}).populate("created_by").populate("client").populate("transactions").exec(function(err, foundJobs){
if(err){
console.log(err)
}
jobs = foundJobs;
});
res.render("dashboard", {transactions: transactions, clients: clients, tickets: tickets, jobs: jobs});
For example, if I console.log "jobs" right after the line jobs = foundJobs;, it will show the jobs array being populated. However, if I console.log "jobs" right before the res.render, it shows it as empty. Considering the global variable endCollections in my original code seemed to be changed within the functions before, I am unsure why my new code does not do the same as everything is returned empty. I know that somehow the scope of the variable is what is wrong here, but I cannot see how. Is there something obvious I am missing? Thanks.
Here now the answer so it is not buried in the post's comments.
After reading the docs, I think you should either use await with an empty exec() or use exec(callback).
What happens when you use both is that exec(callback) sees u passed a callback, it asynchronously executes your query and adds the callback to the promise.then of the query promise to be called once the query promise is settled. Then it immediately returns but it does not return the query promise since you passed a callback. The await is simply awaiting the normal (probably void/undefined) return of the function which is why removing it does not change anything.
After awaiting the return of the function, res.render executes and some time after that, the promise that had been created in the exec(callback) call settles and the callback you passed is executed.
So what is the appropriate way of fixing this? I would encourage you to read deeper into async/awai, promises, and the docs I linked above and find it out yourself before you read on, but since the solution is quite simple I'll leave it here.
// your variable declarations
try {
const foundTransactions = await Transaction.find({}).populate("job").populate("client").populate("deposited_by_user").exec();
// your for loop
transactions = foundTransactions;
// same for the other calls
tickets: tickets, jobs: jobs});
catch (e) {console.log(e);}
res.render("dashboard", {transactions: transactions, clients: clients,...
I'm struggling a little bit with the concepts of asynchronous programming and I would appreciate some help/guidance from someone.
Basically, I'm developing a node.js web server that's connected to a mongodb database.
I'm using EJS to generate HTML files as you can see below.
app.get("/", function(req, res){
res.render('home', {date: getData.todaysDate(), n: getData.todaysIncidents(), nTot: getData.totalIncidents()});
});
Most of these values ('n' and 'nTot') are obtained by querying my database and then doing some other operations, as you can see in the following example code sample.
//------getData.js------//
exports.todaysIncidents = function() {
let server_n = 0;
Incident.find({dia: {$gt:y}}, function(err, result) {
if (err) {
console.log(err);
} else{
//do some stuff...
server_n = 12345
}
}
});
return server_n;
};
Here is the problem: The values printed in the HTML file are always those used for variable initialization, such as 0 for variable 'server_n'. After doing some research, I understand that this happens because .find(...) is an asynchronous function so the program executes right away the instruction "return server_n;", which means that on the HTML file the value showed will be 0 and not 12345.
I already looked at other questions here in StackOverflow, but I'm struggling to understand a possible fix to this issue, I mean I can't be the only one going through this, right?
Could you please provide some basic explanation of how I could go around this issue? I'm still learning and a lot of these concepts remain hard to understand.
Thank you very much.
Yes, you are right that the problem is as a result of improper handling of asynchronous operations like querying the DB. So how do you fix it?
Using async/await:
There are multiple options to handle asynchronous operations in NodeJS, however, I would strongly recommend using async/await, it's syntactically clean and easy to understand.
To put it simply, async/await is a way of specifying and dealing with asynchronous operations. You use the async keyword to specify that a function is asynchronous, and you use the await keyword to wait for an asynchronous operation. One key thing to note is that you can only use the await keyword inside an async function. You can read more about async/await here.
If your nodeJS version is 7.6 or higher, async/await is supported out of the box, however, if you are using a lower version and can't upgrade, you can set up build tools like Babel to be able to use javascript features supported in newer ECMAScript spec.
When using async/await, your code should be something like this.:
//------getData.js------//
// NOTE: the typeof todaysIncidents is not more the regular function,
// it's now an AsyncFunction because of the async keyword
exports.todaysIncidents = async function () {
let server_n = 0;
try {
// In simple terms, the await keyword here would ensure that the DB query
// resolves or reject before moving on to the next statement
const incident = await Incident.find({ dia: { $gt: y } });
// The incident variable stores the result of the query
server_n = 12345
} catch (err) {
// Handle error from the DB query
console.log(err);
}
return server_n;
};
.
//------The router------//
// NOTE: You also need to make the route handler an AsyncFunction
app.get("/", async function (req, res) {
// You can await the differeint DB queries in here to ensure they resolve(or reject) before rendering the view
const todaysDate = await getData.todaysDate();
const todaysIncidents = await getData.todaysIncidents();
const totalIncidents = await getData.totalIncidents();
res.render('home', { date: todaysDate, n: todaysIncidents, nTot: totalIncidents });
});
I am a little bit confused when I use mysql js. I would like to figure out the actual mechanism behind the code.
As I know , if I would like to do a lot of mysql insertion which can run in parallel , it would be more efficient to pack all insertion instructions and commit to mysql one time.
My question is , I am using promise-mysql.js. I would like to use transaction to wrap the whole instructions like this
conn.beginTransaction()
einvoiceList.map(e => {
conn.query("do some insertion")
})
conn.commit()
Am I correct to use this method ?
I have an example using nodejs and mysql2 lib to handle multiple queries to the DB in parallel inside a transaction: https://github.com/Talento90/organization-api/blob/master/organizations-api/src/organizations/manager.js#L30
Basically the idea is to open a transaction and then do all the db queries and at the end commit or rollback the changes.
Tips: Try to use async/await it makes the job much easier :)
async myAsyncMethod(root) {
let promises = [];
let conn = null;
try {
conn = await this.database.getConnection();
await conn.query('START TRANSACTION');
//Execute multiple queries and save them inside an array to wait for the results later
for(...){
promises.push();
}
//Wait until all queries are done!
let results = await Promise.all(promises);
await conn.query('COMMIT');
await conn.release();
return results;
} catch (error) {
if (conn != null) {
await conn.query('ROLLBACK');
}
return 0;
}
}
I have written node.js code for getting some number using mongodb database.this is my code for that
MongoClient.connect('mongodb://localhost:27017/mongomart', function(err, db) {
assert.equal(null, err);
var numItems=db.collection('item').find({"category":category}).count();
callback(numItems);
});
This mongodb query runs correct on mongo shell but it is giving error when using with node.js
Promise <Pending>
I don't know what is this "promise" ? Please help..
node.js code is asynchronous so that numItems won't contain count of items - it rather contains Promise that contains count of items when resolved. You defenetely have to master the basics of node.js and asynchronous programming. Try to modify your code like this
MongoClient.connect('mongodb://localhost:27017/mongomart', function(err, db) {
assert.equal(null, err);
db.collection('item').find({"category":category}).count()
.then(function(numItems) {
console.log(numItems); // Use this to debug
callback(numItems);
})
});
For native Promise check out documentation https://developer.mozilla.org/ru/docs/Web/JavaScript/Reference/Global_Objects/Promise
Also look at bluebird promises https://github.com/petkaantonov/bluebird
A promise is a substitute temporary value that is given while you wait on the real value. To get the real value do
numItems.then(function (value) { callback(value) });
Or better yet, return the promise from your function, and let they implement it using the Promises pattern, instead of the callback pattern.
Had the same problem. Don't know if it's still relevant to you, but this is what solved it for me:
var category = 'categoryToSearch';
var cursor = db.collection('item').find({'category':category});
cursor.count(function (err, num) {
if(err) {
return console.log(err);
}
return num;
});
I drove myself bananas trying to solve a similar problem, where the document.save() option just gave Promise{pending} no matter what I did. Here's what I did:
Change (req,res) to async(req,res).
Change var post = doc.save() to var post = await doc.save().
Finally, log in to MongoDB web, and change accessible IP addresses to 0.0.0.0 (all addresses). Not doing this can cause issues sometimes even when your IP is whitelisted.
try this:
MongoClient.connect('mongodb://localhost:27017/mongomart', async (err, db) => {
assert.equal(null, err);
var numItems= await db.collection('item').find({"category":category}).count();
callback(numItems);
});
(adding the await and turn this function to async function)