We Keep Coding

Insert multiple entries to SQL Server with Node.js

I am rewriting an old API for which I am trying to insert multiple values at once into a MSSQL-Server (2008) database using the node module mssql. Now, I am capable of doing this somehow, but I want to this following best practices. I've done my research and tried a lot of things to accomplish my target. However, I was not able to find a single solution which works just right.
Before
You may wonder:
Well, you are rewriting this API, so there must be a way this has been done before and that was working?
Sure, you're right, it was working before, but... not in a way I'd feel comfortable with using in the rewrite. Let me show you how it was done before (little bit of abstraction added of course):
const request = new sql.Request(connection);
let query = "INSERT INTO tbl (col1, col2, col3, col4) VALUES ";
for (/*basic for loop w/ counter variable i*/) {
query += "(1, #col2" + [i] + ", #col3" + [i] + ", (SELECT x FROM y WHERE z = #someParam" + [i] + "))";
// a check whether to add a comma or not
request.input("col2" + [i], sql.Int(), values[i]);
// ...
}
request.query(query, function(err, recordset) {
// ...
}
While this is working, again, I don't quite think this could be called anything like 'best practice'. Also this shows the biggest problem: a subselect is used to insert a value.
What I tried so far
The easy way
At first I tried the probably easiest thing:
// simplified
const sQuery = "INSERT INTO tbl (col1, col2, col3, col4) VALUES (1, #col2, #col3, (SELECT x FROM y WHERE z = #col4));";
oPool.request().then(oRequest => {
return oRequest
.input("col2", sql.Int(), aValues.map(oValue => oValue.col2))
.input("col3", sql.Int(), aValues.map(oValue => oValue.col3))
.input("col4", sql.Int(), aValues.map(oValue => oValue.col4))
.query(sQuery);
});
I'd say, this was a pretty good guess and actually working relative fine.
Except for the part, that ignores every item after the first one... which makes this pretty useless. So, I tried...
Request.multiple = true
...and I thought, it would do the job. But - surprise - it doesn't, still only the first item is inserted.
Using '?' for parameters
At this point I really started the search for a solution, as the second one was only a quick search in the modules documentation.
I stumbled upon this answer and tried it immediately.
Didn't take long for my terminal to spit out a
RequestError: Incorrect syntax near '?'.
So much for that.
Bulk inserting
Some further research led to bulk inserting.
Pretty interesting, cool feature and excellent updating of the question with the solution by the OP!
I had some struggle getting started here, but eventually it looked really good: Multiple records were inserted and the values seemed okay.
Until I added the subquery. Using it as value for a column declared didn't cause any error, however when checking the values of the table, it simply displayed a 0 as value for this column. Not a big surprise at all, but everybody can dream, right?
The lazy way
I don't really know what to think about this:
// simplified
Promise.all(aValues.map(oValue => {
return oPool.request().then(oRequest =>
oRequest
.input("col2", sql.Int, oValue.col2)
.input("col3", sql.Int, oValue.col3)
.input("col4", sql.Int, oValue.col4)
.query(sQuery);
});
});
It does the job, but if any of the request fails for whichever reason, the other, non-failing inserts, will still be executed, even though this should not be possible.
Lazy + Transaction
As continuing even if some fail was the major problem with the last method, I tried building a transaction around it. All querys are successful? Good, commit. Any query has an errpr? Well, just rollback than. So I build a transaction, moved my Promise.all construct into it and tried again.
Aaand the next error pops up in my terminal:
TransactionError: Can't acquire connection for the request. There is another request in progress.
If you came this far, I don't need to tell you what the problem is.
Summary
What I didn't try yet (and I don't think I will try this) is using the transaction way and calling the statements sequentially. I do not believe that this is be the way to go.
And I also don't think the lazy way is the one that should be used, as it uses single requests for every record to insert, when this could somehow be done using only one request. It's just that this somehow is, I don't know, not in my head right now. So please, if you have anything that could help me, tell me.
Also, if you see anything else that's wrong with my code, feel free to point it out. I am not considering myself as a beginner, but I also don't think that learning will ever end. :)

The way I solved this was using PQueue library with concurrency 1. Its slow due to concurrency of one but it works with thousands of queries:
const transaction = sql.transaction();
const request = transaction.request();
const queue = new PQueue({ concurrency: 1 });
// being transaction
await transaction.begin();
for (const query of queries) {
queue.add(async () => {
try {
await request.query(query);
} catch (err) {
// stop pending transactions
await queue.clear();
await queue.onIdle();
// rollback transaction
await transaction.rollback();
// throw error
throw err;
}
});
}
// await queue
await queue.onIdle();
// comit transaction
await transaction.commit();

JS expression not working as part of a string in ReactJS?

Could you please help with something?
I’m trying to do the following…
import {chosenIpAddress} from './socketEvents.js';
const socket = socketClient(`http:// ${chosenIpAddress} :8081`);
…but no matter what I try, it simply ignores my expression and sees it as http://:8081. I’ve tried with plus signs, commas, backticks, quotation marks, shouting, and threatening it, but to no avail.
I can log out the IP address, so I know it’s populated, but as part of the string it just gets ignored and it’s driving me crazy!
Thanks in advance xxx
P.S... I've seen some similar questions, but these do not help with mine, hence the new question.
Update: As requested, this is my export...
let chosenIpAddress = "";
function chosenTank(tank) {
socket.emit('chosen tank', tank);
console.log('Tank', tank);
chosenIpAddress = tank.IpAddress;
}
export {
chosenIpAddress,
};

You need to export a function that returns the IP address when called.
The file importing the chosenIpAddress has the original value (empty string), but it will never be updated even when chosenTake is called. Javascript strings are copied by value, so if you update the original variable, any other references to it will not be updated.
https://codeburst.io/explaining-value-vs-reference-in-javascript-647a975e12a0
Example of strings copied by value:
chosenIpAddress = "";
x = chosenIpAddress; // x is ""
chosenIpAddress = "localhost"; // chosenIpAddress is "localhost", x is still ""
// This same issues applies to imports/exports.
So do something like this in your ip address file:
let chosenIpAddress = "";
function chosenTank(tank) {
socket.emit('chosen tank', tank);
console.log('Tank', tank);
chosenIpAddress = tank.IpAddress;
}
function getChosenIpAddress() {
// This can be "" if chosenTank is not called first
return chosenIpAddress;
}
export {
getChosenIpAddress,
};
Also, as pointed out in the comments, you need to call chosenTank before you access the chosenIpAddress, or you're going to get an empty string every time.
Further, you'll also need to build the socket string as a function as well, so that it gets the most up-to-date value from getChosenIpAddress when it's called:
import {getChosenIpAddress} from './socketEvents.js';
function getChosenSocket() {
return socketClient(`http://${getChosenIpAddress()}:8081`);
}

So, for anyone who runs into this problem, I worked it out. The answer given is kind of getting there, but not right, I tried that (and variations of it), but it/they didn't work.
Basically in my case, the IP address was being set when the application started, so the chosenIpAddress would always be an empty string as it's being set before the connections had taken place, no variations of exporting, recalling, or function building would've done the job.
To get around this, I let the initial connection take place with a 'placeholder' socketClient to stop the application from crashing...
let socket = socketClient('http://:8081');
...and then when ready, I called to repopulate the IP address when the component mounted from the frontend...
Called from front end when ready...
componentDidMount() {
frontEndConnected();
}
Add IP address as required...
function frontEndConnected() {
socket = socketClient(`http://${chosenTankIpAddress}:8081`);
}
...which works a charm.

Cannot get the full json with request json

When I am actually entering the XXXX YYYY, then I am getting the players json code in my html page (around 150 values).
But when I am trying to use a function on the players list it somewhy does not contain all the 150 values and the try throws me into the catch error part, where I can see that players json has only 100 players inside there.
Any idea what could be the problem?
if(yourID === "XXXX" && targetID === "YYYY"){
return players;
}
try{
if(isUserAlive(yourID)){
if(targetID === ""){
return userTargetInfo(yourID);
}
var checkForMatch = getUserTarget(yourID);
if(checkForMatch === targetID){
killTarget(targetID);
getUser(yourID).targetID = getTargetTarget(targetID);
addScore(yourID);
return userTargetInfo(yourID);
//return getTargetTargetStats(targetID);
}else{
return "INVALID";
}
}else{
return "DEAD"
}
}catch(err){
console.log("Error",console.log(players))
return "INVALID"
}
Edit: Since I had no time, I created 2 websites and divided the database into 2 different databases, so it would work under 100 people on each. Did not have time to fix the error at this point. So I won't be choosing the solution to that since I won't be trying that any time soon.
Thank you for all your help!

Check the link api that you are using , it might have pagination integrated with it . in that case i will return certain number of object 1st and then you can re-request to get next batch . Most likely they might have a option to change the no of object returned (sometimes with max value)

I'm pretty sure body is returned as a string. Try changing it to an object so you can work with it easier.
Change:
players = body;
to:
players = JSON.parse(body);
I'm not sure the rest of your code, but you may want to add var on your players variable declaration because this looks like the first time you are setting it.
Research: namespace collisions
If you are still having issues, edit your question to include the response you are getting from console.log(JSON.parse(body));. You will be able to get more helpful answers. Personally, I am curious to see the keys such as:
{ query:
{ count: 1,
created: '2017-04-23T22:03:31Z',
lang: 'en-US',
results: { channel: [Object] } } }
If it's paginated, you should see some kind of cursor key in there, or prev and next along with some kind of totalCount.
Hope this helps.

What is more effective: every time make an ajax request or slice a result in func?

I have a JSON data of news like this:
{
"news": [
{"title": "some title #1","text": "text","date": "27.12.15 23:45"},
{"title": "some title #2","text": "text","date": "26.12.15 22:35"},
...
]
}
I need to get a certain number of this list, depended on an argument in a function. As I understand, its called pagination.
I can get the ajax response and slice it immediately. So that every time the function is called - every time it makes an ajax request.
Like this:
function showNews(page) {
var newsPerPage = 5,
firstArticle = newsPerPage*(page-1);
xhr.onreadystatechange = function() {
if(xhr.readyState == 4) {
var newsArr = JSON.parse(xhr.responseText),
;
newsArr.news = newsArr.news.slice(firstArticle, newsPerPage*(page));
addNews(newsArr);
}
};
xhr.open("GET", url, true);
xhr.send();
Or I can store all the result in newsArr and slice it in that additional function addNews, sorted by pages.
function addNews(newsArr, newsPerPage) {
var pages = Math.ceil(amount/newsPerPages), // counts number of pages
pagesData = {};
for(var i=0; i<=pages; i++) {
var min = i*newsPerPages, //min index of current page in loop
max = (i+1)*newsPerPages; // max index of current page in loop
newsArr.news.forEach(createPageData);
}
function createPageData(item, j) {
if(j+1 <= max && j >= min) {
if(!pagesData["page"+(i+1)]) {
pagesData["page"+(i+1)] = {news: []};
}
pagesData["page"+(i+1)].news.push(item);
}
}
So, simple question is which variant is more effective? The first one loads a server and the second loads users' memory. What would you choose in my situation? :)
Thanks for the answers. I understood what I wanted. But there is so much good answers that I can't choose the best

It is actually a primarily opinion-based question.
For me, pagination approach looks better because it will not produce "lag" before displaying the news. From user's POV the page will load faster.
As for me, I would do pagination + preload of the next page. I.e., always store the contents of the next page, so that you can show it without a delay. When a user moves to the last page - load another one.
Loading all the news is definitely a bad idea. If you have 1000 news records, then every user will have to load all of them...even if he isn't going to read a single one.
In my opinion, less requests == better rule doesn't apply here. It is not guaranteed that a user will read all the news. If StackOverflow loaded all the questions it has every time you open the main page, then both StackOverflow and users would have huge problems.

If the max number of records that your service returns is around 1000, then I don't think it is going to create a huge payload or memory issues (by looking at the nature of your data), so I think option-2 is better because
number of service calls will be less
since user will not see any lag while paginating, his experience of using the site will be better.

As a rule of thumb:
less requests == better
but that's not always possible. You may run out of memory/network if the data you store is huge, i.e. you may need pagination on the server side. Actually server side pagination should be the default approach and then you think about improvements (e.g. local caching) if you really need them.
So what you should do is try all scenarios and see how well they behave in your concrete situation.

I prefer fetch all data but showing on some certain condition like click on next button data is already there just do hide and show on condition using jquery.
Every time call ajax is bad idea.
but you also need to call ajax for new data if data is changed after some periodic time

Variable not updating in script string, yet it updates

Very confused here.
I have a search box which reads a list of school names from my database. When I select a school, the id (from the db) gets put in a hidden textbox.
I also have a search box which reads a list of courses from my database. However, I made the query so that it only reads the courses from the selected school.
It does that, in theory.
I was planning to pass the school id, which I grab from the hidden box, to the search script which in turn passes it to my database query. However, the variable I put my school id in doesn't seem to be updating.. yet it does. Let me explain.
I come on the page. The school for my test account has id 1. The id number in my hidden box is indeed 1. I search for a school which I know has some courses assigned to it: the id number in the box changes to 3.
I have a JS variable called school_id which I declared outside of my $(document).ready. I assume that means it's global (that's what I got taught even though SO told me once it isn't really the correct way to do this. Still have to look into that). I wrote a function which updates this variable when the school search box loses focus:
$("#school").blur(function() {
school_id = $("#school_id").val();
});
A quick javascript:alert(school_id); in my browser bar also shows the updated variable: it is now 3 instead of 1.
Onto the search script part of my page (excerpt of the script):
script:"/profiel/search_richting?json=true&limit=6&id=" + school_id + "&"
As you can see, I pass the school_id variable to the script here. However, what seems to be happening is that it always passes '1', the default variable when the page loads. It simply ignores the updated variable. Does this string get parsed when the page loads? In other words, as soon as the page loads, does it actually say &id=1? That's the only idea I can come up with why it would always pass '1'.
Is there a way to make this variable update in my script string? Or what would be the best way to solve this? I'm probably missing out on something very simple here again, as usual. Thanks a lot.
EDIT
Updated per request. I added a function getTheString as was suggest and I use the value of this function to get the URL. Still doesn't work though, it still seems to be concatenating before I get a chance to update the var. HOWEVER, with this code, my ajax log says id:[object HTMLInputElement], instead of id:1. Not sure what that means.
<script type="text/javascript">
var school_id;
$(document).ready(function() {
$("#school").blur(function() {
school_id = $("#school_id").val();
});
// zoekfunctie
var scholen = {
script:"/profiel/search_school?json=true&limit=6&",
varname:"input",
json:true,
shownoresults:false,
maxresults:6,
callback: function (obj) { document.getElementById('school_id').value = obj.id; }
};
var as_json = new bsn.AutoSuggest('school', scholen);
var richtingen = {
script: getTheString(),
varname:"input",
json:true,
shownoresults:true,
maxresults:6
};
var as_json2 = new bsn.AutoSuggest('studierichting', richtingen);
});
function getTheString() {
return "/profiel/search_richting?json=true&limit=6&id=" + school_id + "&";
}
</script>

This is because the URL is static, it is not updated as the ID changes.
You should update the URL as part of the code you wrote to get the ID:
$("#school").blur(function() {
school_id = $("#school_id").val();
// update URL here ...
});

Aren't you concatenating script:"/profiel/search_richting?json=true&limit=6&id=" + school_id + "&" before the event is fired and the var updated?

Okay. So the problem was my third party plug-in instead of the code I wrote. I fixed this by editing the code of the autoSuggest plugin so it now includes my id field in the AJAX request.
var url = this.oP.script+this.oP.varname+"="+encodeURIComponent(this.sInp)+"&id="+ $("#school_id").val();
Thanks to everyone who tried to help me out!