We Keep Coding

How can I insert objects in SQL Server running a function in a loop? ConnectionError: .connect can not be called on a Connection in `Connecting` state

I'm working in a NodeJS project, this project I decided to change the way I'm doing it because this way wasn't working, let me try to explain it.
I need to insert data into a SQL Server DB, so I did a function insertOffice() this function opens a connection using Tedious, then fetchs data to an url with data from an array data2 to load coords, and then with this coords creates an object, then inserts this object into a DB. When inserting only one part of my data2 array, it works, by only sendind data[0] it adds:
{
latjson: 1,
lonjson: 1,
idoficina: "1",
}
But I want to insert both of the parts of my array, changing data2[0] to data2[index], to be able to insert all my array, so I tried creating another function functionLooper()that loops insertOffice() to insert my data from my array data2.
I builded this little code to learn how to loop a function, this prints index that is the value I use for bringing idoficina.
As you can see functionLooper() runs the code twice, so it can read fully data2 array, I have this little code that works with the same logic, I builded my full code using this:
function insertOffice(index) {
console.log(index);
}
function functionLooper() {
for (let i = 0; i < 5; i++) {
let response = insertOffice(i);
}
}
functionLooper();
This prints:
0
1
2
3
4
So my code it's supposed to send index
I'm expecting my code to loop my insertOffice() and being able to insert my objects, the issue is that this doesn't seems to work as I am getting this error:
C:\...\node_modules\tedious\lib\connection.js:993
throw new _errors.ConnectionError('`.connect` can not be called on a Connection in `' + this.state.name + '` state.');
^
ConnectionError: `.connect` can not be called on a Connection in `Connecting` state.
this is my code:
var config = {
....
};
const data2 = [
...
];
var connection = new Connection(config);
function insertOffice(index) {
console.log(index)
connection.on("connect", function (err) {
console.log("Successful connection");
});
connection.connect();
const request = new Request(
"EXEC SPInsert #Data1, ... ",
function (err) {
if (err) {
console.log("Couldn't insert, " + err);
} else {
console.log("Inserted")
}
}
);
console.log(myObject.Id_Oficina)
request.addParameter("Data1", TYPES.SmallInt, myObject.Id_Oficina);
request.on("row", function (columns) {
columns.forEach(function (column) {
if (column.value === null) {
console.log("NULL");
} else {
console.log("Product id of inserted item is " + column.value);
}
});
});
request.on("requestCompleted", function () {
connection.close();
});
connection.execSql(request);
}
function functionLooper() {
for (let i = 0; i < 2; i++) {
let response = insertOffice(i);
}
}
functionLooper();
I do not know if this is the right way to do it (looping the inserting function insertOffice()twice), if you know a better way to do it and if you could show me how in an example using a similar code to mine, would really appreciate it.

You're approaching an asynchronous problem as if it's a synchronous one. You're also making your life a bit harder by mixing event based async tasks with promise based ones.
For example, connection.connect() is asynchronous (meaning that it doesn't finish all its work before the next lines of code is executed), it is only done when connection emits the connect event. So the trigger for starting the processing of your data should not be started until this event is fired.
For each of the events in your loop they are not running one at a time but all at the same time because the fetch() is a promise (asynchronous) it doesn't complete before the next iteration of the loop. In some cases it may have even finished before the database connection is ready, meaning the code execution has moved on to DB requests before the connection to the database is established.
To allow your code to be as manageable as possible you should aim to "promisify" the connection / requests so that you can then write an entirely promise based program, rather than mixing promises and events (which will be pretty tricky to manage - but is possible).
For example:
const connection = new Connection(config);
// turn the connection event into a promise
function connect() {
return new Promise((resolve, reject) => {
connection.once('connect', (err) => err ? reject(err) : resolve(connection));
connection.connect()
});
}
// insert your data once the connection is ready and then close it when all the work is done
function insertOffices() {
connect().then((conn) => {
// connection is ready I can do what I want
// NB: Make sure you return a promise here otherwise the connection.close() call will fire before it's done
}).then(() => {
connection.close();
});
}
The same approach can be taken to "promisify" the inserts.
// turn a DB request into a promise
function request(conn) {
return new Promise((resolve, reject) => {
const request = new Request(...);
request.once('error', reject);
request.once('requestCompleted', resolve);
conn.execSql(request);
});
}
This can then be combined to perform a loop where it's executed one at a time:
function doInserts() {
return connect().then((conn) => {
// create a "chain" of promises that execute one after the other
let inserts = Promise.resolve();
for (let i = 0; i < limit; i++) {
inserts = inserts.then(() => request(conn));
}
return inserts;
}).then(() => connection.close())
}
or in parallel:
function doInserts() {
return connect().then((conn) => {
// create an array of promises that all execute independently
// NB - this probably won't work currently because it would need
// multiple connections to work (rather than one)
let inserts = [];
for (let i = 0; i < limit; i++) {
inserts.push(request(conn));
}
return Promise.all(inserts);
}).then(() => connection.close())
}

Finally I could fix it, I'm sharing my code for everyone to could use it and do multiple inserts, thanks to Dan Hensby, I didn't do it his way but used part of what he said, thanks to RbarryYoung and MichaelSun90 who told me how, just what I did was changing my
var connection = new Connection(config);
to run inside my
function insertOffice(index) { ... }
Looking like this:
function insertOffice(index) {
var connection = new Connection(config);
....
}

Adapting a function that isnt chainable to return a value

I am trying to get all the pages of a pdf in one object using the pdfreader package. The function originally returns each page (as its own object) when it processes it. My goal is to write a wrapper that returns all pages as an array of page objects. Can someone explain why this didn't work?
I tried:
adding .then and a return condition - because I expected the parseFileItems method to return a value:
let pages = [];
new pdfreader.PdfReader()
.parseFileItems(pp, function(err, item) {
{
if (!item) {
return pages;
} else if (item.page) {
pages.push(lines);
rows = {};
} else if (item && item.text) {
// accumulate text items into rows object, per line
(rows[item.y] = rows[item.y] || []).push(item.text);
}
}
})
.then(() => {
console.log("done" + pages.length);
});
and got the error
TypeError: Cannot read property 'then' of undefined
The function I'm modifying (From the package documentation):
var pdfreader = require("pdfreader");
var rows = {}; // indexed by y-position
function printRows() {
Object.keys(rows) // => array of y-positions (type: float)
.sort((y1, y2) => parseFloat(y1) - parseFloat(y2)) // sort float positions
.forEach(y => console.log((rows[y] || []).join("")));
}
new pdfreader.PdfReader().parseFileItems("CV_ErhanYasar.pdf", function(
err,
item
) {
if (!item || item.page) {
// end of file, or page
printRows();
console.log("PAGE:", item.page);
rows = {}; // clear rows for next page
} else if (item.text) {
// accumulate text items into rows object, per line
(rows[item.y] = rows[item.y] || []).push(item.text);
}
});

There seem to be several issues/misconceptions at once here. Let's try to look at them once at a time.
Firstly, you seem to have thought that the outer function will return ("pass on") your callback's return value
This is not the case as you can see in the library source.
Also, it wouldn't even make sense, because the callback called once for each item. So, with 10 items, it will be invoked 10 times, and then how would parseFileItems know which of the 10 return values of your callback to pass to the outside?
It doesn't matter what you return from the callback function, as the parseFileItems function simply ignores it. Furthermore, the parseFileItems function itself doesn't return anything either. So, the result of new pdfreader.parseFileItems(...) will always evaluate to undefined (and undefined obviously has no property then).
Secondly, you seem to have thought that .then is some sort of universal chaining method for function calls.
In fact, .then is a way to chain promises, or to react on the fulfillment of a promise. In this case, there are no promises anywhere, and in particular parseFileItems doesn't returns a promise (it returns undefined as described above), so you cannot call .then on its result.
According to the docs, you are supposed to react on errors and the end of the stream yourself. So, your code would work like this:
let pages = [];
new pdfreader.PdfReader()
.parseFileItems(pp, function(err, item) {
{
if (!item) {
// ****** Here we are done! ******
console.log("done" + pages.length) // The code that was in the `then` goes here instead
} else if (item.page) {
pages.push(lines);
rows = {};
} else if (item && item.text) {
// accumulate text items into rows object, per line
(rows[item.y] = rows[item.y] || []).push(item.text);
}
}
})
However, I agree that it'd be nicer to have a promise wrapper so that you won't have to stuff all the following code inside the callback's if (!item) branch. You could achieve that like this, using new Promise:
const promisifiedParseFileItems = (pp, itemHandler) => new Promise((resolve, reject) => {
new pdfreader.PdfReader().parseFileItems(pp, (err, item) => {
if (err) {
reject(err)
} else if (!item) {
resolve()
} else {
itemHandler(item)
}
})
})
let pages = []
promisifiedParseFileItems(pp, item => {
if (item.page) {
pages.push(lines)
rows = {}
} else if (item && item.text) {
// accumulate text items into rows object, per line
(rows[item.y] = rows[item.y] || []).push(item.text)
}
}).then(() => {
console.log("done", pages.length)
}, e => {
console.error("error", e)
})
Note: You would get even nicer code with async generators but that is too much to explain here now, because the conversion from a callback to an async generator is less trivial than you may think.

If you want to chain a then, you need the callback function to return a Promise :
new pdfreader.PdfReader()
.parseFileItems(pp, function (err, item) {
return new Promise( (resolve, reject) => {
let pages = ...
// do stuff
resolve(pages);
}
})
.then( pages => {
console.log("done" + pages.length);
});

How to assign results of an http function within array.map() in javascript

I've read MANY articles, but I'm totally a newbie to this whole async thing and am having a very hard time wrapping my brain around how it all works. I want to map the filtered array of objects, and inside that, I'd like to return the result of a function (an amount) and set that as the value of pmtdue. I tried this a bunch of ways, but always get zoneawarepromise or observable when it's logged, or a ton of errors. This is probably the closest I've gotten, but it's still not right.
async today(day = null, status = null) {
this.logger.log(`show today's ${status} appts ${day}`);
// filter master list for today
const filtered = [...this.apptList].filter(appt => {
if (!status) {
return (
appt.scheduled >= this.helperService.dayStart(day) &&
appt.scheduled <= this.helperService.dayEnd(day) &&
appt.status.status !== 'Checked Out' &&
appt.status.status !== 'Scheduled'
);
} else {
return (
appt.scheduled >= this.helperService.dayStart(day) &&
appt.scheduled <= this.helperService.dayEnd(day) &&
appt.status.status === status
);
}
});
// calculate due amount and map it to pmtdue field
const dueappts = await this.getTotalDue(filtered).then(
res => {
// console.log(res);
this.ApptModels = res;
},
err => {
console.log(err);
}
);
// send the data to ng2-smart-table
console.log(`filtered ApptModels`, this.ApptModels);
}
This is the function that does the mapping and has the functions I want to work
// a.pmtduedate returns the correct value as there is no http call
// a.pmtdue returns a zoneawarepromise but I don't know how to get the VALUE
getTotalDue(appts: Array<any>): Promise<any> {
return Promise.all(
appts.map(async (a: any) => {
a.pmtduedate = await this.helperService.getDueDate(a);
a.pmtdue = await this.dataService.sendTotalDue(a);
console.log(a.pmtdue); // logs undefined
return a;
})
);
}
My data service function (I know sometimes code matters that I think is insignificant):
async sendTotalDue(appt) {
this.logger.log(`fetch amount ${appt.patientID.nickname} owes`);
return await this.http.post(`${SERVER_URL}/sendtotaldue`, appt);
}
And finally, the backend function(minus details on data). It logs the correct amount on the backend, I just can't get it to display on the frontend:
module.exports.sendTotalDue = (req, res) => {
const appt = req.body;
// callback function that handles returning data
function done(err, results) {
const totaldue = parseInt(results, 10);
console.log(`API sendTotalDue CALLBACK done...totaldue: ${totaldue}`);
if (err) {
console.log('ERROR getting total due: callback error', err);
res.sendStatus(500).json(err); // server error; it'd be good to be more specific if possible
} else {
// end the request, send totaldue to frontend
console.log(`SUCCESS send totaldue to frontend ${totaldue}`);
res.status(200).json(totaldue);
}
}
// run first function
console.log(`1. getAmtDue:`);
this.getAmtDue(appt, done);
};
module.exports.getAmtDue(appt, callback) {
... function finds past visits, past payment and due totals
}
module.exports.getCurrentDue(appt, pastdueamt, callback) {
... function finds current visits and payments. calculates current due and adds the past due
callback(null, totaldue);
}
Can someone please help me understand what I'm doing wrong? Feel free to dumb it down for me, cause that's how I feel at this point.
EDITED TO FIX ERRORS like missing await and return. It is now to the point where I can see the value returned in the data service, but I get undefined in the map function section.

Ooooh!!! I GOT IT! I still don't totally understand WHY it works, but I changed the data service as follows:
async sendTotalDue(appt): Promise<any> {
this.logger.log(`fetch amount ${appt.patientID.nickname} owes`);
try {
const result = await this.http
.post(`${SERVER_URL}/sendtotaldue`, appt)
.toPromise();
return result as any[];
} catch (error) {
console.log(error);
}
}
Changing my service to the above finally got my values to appear exactly where I wanted them in the data table! :)
I found this article, which helped figure out how to work with Observable
Angular Tutorial with Async and Await

How can I improve this nested asynchronous loop?

The problem is as follows. I have an array of objects like so:
let myObj = [
{'db1':['doc1','doc2','doc3']},
{'db2':['doc4','doc5']},
{'db3':['doc7','doc8','doc9','doc10']}
]
Note that this is a data structure I decided to use for the problem and can be changed if it can improve the overall implementation. The actual db and doc Ids are read from a text file formatted as below.
"db1","doc1"
"db1","doc2"
...
My app will iterate through the db list synchronously. Inside each db iteration, there will be an asynchronous iteration of the document list. Each document will be retrieved, processed and saved back to the db.
So basically at any given instance: one db, but multiple documents.
I have a working implementation of the above like so:
dbIterator: the synchronous outer loop to iterate dbs. The callback passed to docIterator will trigger the next iteration.
const dbIterator = function (x) {
if (x < myObj.length) {
let dbObj = myObj[x];
let dbId = Object.keys(dbObj)[0];
docIterator(dbId, dbObj[dbId], ()=>merchantIterator(x+1));
} else {
logger.info('All dbs processed');
}
};
docIterator: the asynchronous loop to iterate docs. The callback cb is called after all documents are processed. This is tracked via the docsProcessed and docsToBeProcessed variables
const docIterator = function(dbId, docIds, cb){
//create connection
targetConnection = //some config for connection to dbId
let docsProcessed = 0;
let docsToBeProcessed = docIds.length;
//asynchronous iteration of documents
docIds.forEach((docId)=>{
getDocument(docId, targetConnection).then((doc)=>{
//process document
processDoc(doc, targetConnection).then(()=>{
//if processing is successful
if (++docsProcessed >= docsToBeProcessed) {
cb();
}
})
//if processing fails
.catch((e) => {
logger.error('error when processing document');
if (++docsProcessed >= docsToBeProcessed) {
cb();
}
});
}).catch((e)=>{
logger.error('error when retrieving document: ');
if (++docsProcessed >= docsToBeProcessed) {
cb();
}
});
});
};
processDoc: used to process and save an individual document. This returns a promise that gets resolved when the document processing is done which in turn increments docsProcessed and conditionally (docsProcessed >= docsToBeProcessed) calls the call back passed into docIterator
const processDoc = function(doc, targetConnection) {
return new Promise(function(resolve, reject) {
if(shouldThisDocBeProcessed(doc){
let updatedDoc = logic(doc);
targetConnection.insert(updatedDoc, updatedDoc._id,
function (error, response) {
if (!error){
logger.info('updated successfully');
} else {
logger.error('error when saving doc');
}
resolve();
}
);
} else {
resolve();
}
})
};
This works as expected but for me this implementation is sub-optimal and messy. I'm pretty sure this can be improved upon and most importantly a chance to better understand and implement solutions to synchronous and asynchronous problems.
I'm open to constructive criticism. So how can this be improved?

Maybe something like this?
An example implementation of throttle can be found here.
//this should be available in both modules so you can filter
const Fail = function(details){this.details=details;};
// docIterator(dbId,docIds)
// .then(
// results =>{
// const failedResults = results.filter(
// result => (result&&result.constructor)===Failed
// );
// const successfullResults = results.filter(
// result => (result&&result.constructor)!==Failed
// );
// }
// )
const docIterator = function(dbId, docIds){
//create connection
// targetConnection = //some config for connection to dbId
let docsProcessed = 0;
let docsToBeProcessed = docIds.length;
//asynchronous iteration of documents
docIds.map(
docId =>
new Promise(
(resolve,reject) =>
//if you use throttled you can do:
// max10(
// ([docId,targetConnection])=>
// getDocument(docId,targetConnection)
// )([docId, targetConnection])
getDocument(docId, targetConnection)
)
.then(
doc =>
//if this returns nothing then maybe you'd like to return the document
processDoc(doc, targetConnection)
.then(
_ => doc
)
)
.catch(
err => new fail([err,docId])
)
)
};

Calling async promises inside async promise using .join()

I´m using node/ epxress, mysql and bluebird.
I´m currently doing an async database operation after the client request it. Inside the callback of the first database operation I have to perform some calculations first and afterwards doing two more database queries which are required to provide the client the correct result.
My Code is separated into a Controller class, which handles the get/ post request. In the middle a service class for business logic, which talks to a database class which queries in the database.
I´m currently able to do perform the first and second database request.
getVacation(departmentID) {
return departmentDatabase.getVacation(departmentID)
.then(result => [ result, result.map(entry => this.getDateRange(new Date(entry.dateFrom), new Date(entry.dateTo))) ])
.spread(function(result, dateRange){
var mergedDateRange = [].concat.apply([], dateRange);
var counts = {};
mergedDateRange.forEach(function(x) { counts[x] = (counts[x] || 0)+1; });
return [{"vacationRequest": result, "dateRange": dateRange, "countedDateRange": counts}];
})
.then( result => [result, departmentDatabase.countUser(departmentID)])
.spread(function (result, userOfDepartmentCount){
console.log(userOfDepartmentCount);
console.log(result);
//console.log(blocked);
return departmentID; //return just for not running into timeout
})
.catch(err => {
// ...do something with it...
// If you want to propagate it:
return Promise.reject(err);
// Or you can do:
// throw err;
});
}
But when trying to perform the third I´m running into trouble.
For a solution of this problem I read the Bluebird Doc´s, which pointed me to .all() or (even better) .join(). But trying to use either of them didn´t worked for me.
If I try it with .join() is always results in join is not a function, which I find confusing because I can use all other function. I also tried to require
var Promise = require("bluebird");
var join = Promise.join;
But not even this helped.
Currently I just require Bluebird as Promise in my database class.
So here now my entire service class.
'use strict';
var departmentDatabase = require('../database/department');
var moment = require('moment');
class DepartmentService {
constructor() {
}
getVacation(departmentID) {
return departmentDatabase.getVacation(departmentID)
.then(result => [ result, result.map(entry => this.getDateRange(new Date(entry.dateFrom), new Date(entry.dateTo))) ])
.spread(function(result, dateRange){
var mergedDateRange = [].concat.apply([], dateRange);
var counts = {};
mergedDateRange.forEach(function(x) { counts[x] = (counts[x] || 0)+1; });
return [{"vacationRequest": result, "dateRange": dateRange, "countedDateRange": counts}];
})
//THIS DOES NOT WORK
.join(result => [result, departmentDatabase.countUser(departmentID), departmentDatabase.blockedDaysOfResponsible(departmentID)])
.spread(function (result, userOfDepartmentCount, blocked){
console.log(userOfDepartmentCount);
console.log(result);
console.log(blocked);
return departmentID;
})
.catch(err => {
// ...do something with it...
// If you want to propagate it:
return Promise.reject(err);
// Or you can do:
// throw err;
});
}
getDateRange(startDate, stopDate) {
var dateArray = [];
var currentDate = moment(startDate);
while (currentDate <= stopDate) {
dateArray.push(moment(currentDate).format('YYYY-MM-DD'))
currentDate = moment(currentDate).add(1, 'days');
}
return dateArray;
}
}
module.exports = new DepartmentService();
Is someone able to give me an example how to do it right?
EDIT:
Here an example code I´m using inside my databaseCall, to return the db result and the promise
return Promise.using(dbConnection.getConnection(), function (conn) {
return conn.queryAsync(sql, [departmentID])
.then(function (result) {
return result;
})
.catch(function (err) {
return err;
});
});

Promise.join is nice, but it might not suit your situation the best. Promise.all will combine several promises like you have there into a single resolution:
.then(result => Promise.all([result, departmentDatabase.countUser(departmentID), departmentDatabase.blockedDaysOfResponsible(departmentID)])]‌​))
Then you could spread that result (an array) into a function call:
.spread(function(a, b, c) {
console.log("Results", a, b, c);
});
Promise.all takes an array of Promises, waits for all of them to resolve (or reject), and then continues with the results in an ordered array into the subsequent .then (or other promise) clause.

If your looking for a module that makes control flow with promises easier then you might like relign. Promise.all may do the trick here, but if you need the resolved results, then relign.parallel or relign.series might be better for you.