We Keep Coding

NodeJS HTTP response max size reached?

I stumbled on a very strange issue recently concerning my Node.JS REST API. It's aim is to convert HTTP requests to SQL requests and the SQL response to HTTP response so I can then transfer everything over SSL/TLS and it's encoded.
The problem is that when the response size is small, it all works fine but when it exceed a certain size (about 255.37 KB), the response body is cutted right in the middle of the JSON. After multiple tests it appears that the issue is related to the HTTP response total size (including the headers) because when some custom headers are removed, more of the body is sent. I wrote a similar code in PHP and the response in JSON from the PHP API is fine so I assumed that the issue was originating from a bug in the Node.JS web server. Also, the Content-Lenght header is fine and has the right value (like the PHP API response does).
I'm using Node.JS v6.11.0 with the last release of Express. All my dependencies are up to date thanks to npm.
Here is the code of the function that handle the GET HTTP requests and proceed in doing SELECT SQL requests, then parse to json string the answer
function SelectData(connection){
return function (req, res) {
let tableName = req.params.table;
let id = req.params.id;
console.log("GET request received with tableName = " + tableName + " and id = " + id);
// Should match this regex
if (!tableName.match(/^[\-\_\s0-9A-Za-z]+$/)){
console.log("Error: Invalid table name provided : " + tableName);
badRequestError(req, res);
return;
}
if (id !== undefined){
if (!id.match(/^[0-9]+$/)){
console.log("Error: Invalid id provided : " + id + " (table name was valid : " + tableName + ")");
badRequestError(req, res);
return;
}
}
// if id is empty, then don't use any condition, else, add SQL condition
idPart = (id == undefined) ? "" : ('WHERE id="' + id + '"');
// TODO: try to replace " + var + " by question marks and bind params, find a way to do it w/ id
connection.query("SELECT * FROM " + tableName + " " + idPart + ";", function(err, result){
res.setHeader('Content-Type', 'application/json');
console.log("Request executed successfully !");
// Works too but I prefer the .send() method
// res.status(200).write(JSON.stringify(result)).end();
res.status(200).send(JSON.stringify(result));
req.connection.destroy();
return;
});
}
}
function badRequestError(req, res){
res.setHeader('Content-Type', 'text/plain');
res.status(400).send("Bad Request");
req.connection.destroy();
}
exports.SelectData = SelectData;
Edit: After researching yesterday I found similar issue resulting of the use of the HTTPS module so I removed it but it still happens.

Can't send parameter containing "#" to dot net web service from ajax

Cant send parameter containing "#" to dot net web service from ajax.
var s = encodeURI(
"http://subdomain.mydomain.domain.asmx/getData?OUserId=" + UserId +
"&Token=" + Token +
"&OrgId=" + OrgId +
'&Message=' + Message +
'&Schoolid=' + SchoolId +
'&SessionId=" ' + SessionId +
'&UnicodeValue=' + UnicodeValue +
'&ClassID=' + ClassIdCommaSeparated.toString()
);
$.ajax({
url: s,
error: function(err) {
alert(err);
},
success: function(data) {....
}
});
Here classIdCommaSeparated is 1#1#1#1#1,1#1#1#1#1,1#1#1#1#1.

Use encodeURIComponent on the individual parts, rather than encodeURI on the whole:
var s = "http://subdomain.mydomain.domain.asmx/getData?OUserId=" + encodeURIComponent(UserId) +
"&Token=" + encodeURIComponent(Token) +
"&OrgId=" + encodeURIComponent(OrgId) +
'&Message=' + encodeURIComponent(Message) +
'&Schoolid=' + encodeURIComponent(SchoolId) +
'&SessionId=" ' + encodeURIComponent(SessionId) +
'&UnicodeValue=' + encodeURIComponent(UnicodeValue) +
'&ClassID=' + encodeURIComponent(ClassIdCommaSeparated.toString());
$.ajax({
url: s,
error: function(err) {
alert(err);
},
success: function(data) {....
}
});
Technically, both the name (before the =) and the value (after the =) need to be encoded, but when your names consist just of the letters A-Z (in upper or lower case) or digits, like yours do, encoding them doesn't change them at all. (If you didn't know what those names were, you'd definitely want to pass them through encodeURIComponent.)

several hours after i am not able to understand what is arising this problem.but i have worked around to have a temporary solution to the problem.i have used underscore in place of # and i got it working.thanks #T.J. Crowder for having a look upon.

node.js: HTML form hangs after submit when inserting data into PostgreSQL database

I have recently set up node.js using Express and I created a simple HTML form using Jade. The form is to insert the data in a PostgreSQL database. The problem is that when I press submit on the form, everything is inserted on the database, but the HTML form is just hanging/lingering, and at some point it stops with No data received, ERR_EMPTY_RESPONSE. Sometimes it also inserts the data twice. I guess this is because the server side does not return a response, but I cannot see how (I am new to node.js).
The form has action="add_device" which is routed to routes/add_device.js. add_device.js looks like this:
var express = require('express');
var router = express.Router();
router.get('/', function(request, response, next) {
res.send('Nothing to see here. Move along.');
});
router.post('/', function(request, response, next) {
var db = require('../public/javascripts/db/insert');
var result = db.insertDevice(request, response);
return result;
});
module.exports = router;
The insertDevice function in my db module looks like this (it is exported with module.exports):
// Insert new QA device. Data arriving as a request from a HTML form.
insertDevice: function (request, response) {
// Input that is verified in the HTML form.
// Convert to proper format for PostgreSQL query.
var name = '\'' + request.body.name + '\'';
var ip_address = '\'' + request.body.ip_address + '\'';
var os = '\'' + request.body.os + '\'';
// Input that needs to be verified. Prepare for PostgreSQL query.
var mac_address;
var os_version;
request.body.mac_address == "" ? mac_address = 'NULL' : mac_address = '\'' + request.body.mac_address + '\'';
request.body.os_version == "" ? os_version = 'NULL' : os_version = '\'' + request.body.os_version + '\'';
var pg = require('pg'); // PostgreSQL module.
var td = require('./table_data') // Database constants.
var client = new pg.Client(request.app.get('postgreConnection'));
client.connect(function(err) {
if (err) {
return console.error('Could not connect to postgres', err);
}
var QUERY = "INSERT INTO " + td.QA_DEVICES.TABLE_NAME + "(" +
td.QA_DEVICES.COLUMN_NAME + ", " +
td.QA_DEVICES.COLUMN_MAC_ADDRESS + ", " +
td.QA_DEVICES.COLUMN_IP_ADDRESS + ", " +
td.QA_DEVICES.COLUMN_OS + ", " +
td.QA_DEVICES.COLUMN_OS_VERSION + ") VALUES(" +
name + ", " +
mac_address + ", " +
ip_address + ", " +
os + ", " +
os_version + ");";
client.query(QUERY, function (err, result) {
if (err) {
return console.error('Error running query: ' + QUERY, err);
}
console.log('Query performed: ' + QUERY);
client.end();
});
});
}
The 'Query performed' is always logged to console and data inserted into the database, but the form is still hanging. My questions are:
Is it the lack of response from the server that makes the form hang?
How can I "send a response back" to the front end?
Is it possible to route the front end to another page after insertion into the database? What is the best practice?

Yes, your request is receiving no response, so it is hanging.
In order to send a response, you can either send a blind acknowledgement right when the request is received (that is not dependent upon the success of the query and may be bad practice), or you can send it in the callback.
client.query(QUERY, function (err, result) {
if (err) {
// response.json({status: 'error'});
response.write('Error');
return console.error('Error running query: ' + QUERY, err);
} else {
// You can send json here too
// response.json({status: 'success'});
response.write('Success');
}
console.log('Query performed: ' + QUERY);
client.end();
});
If you want to go to another page, simply parse the incoming response on the client side and do a redirect. Using json is a good way to carry this out. You can also do a response.redirect(url) on the server side too, instead of sending back data. Have fun

push function with javascript doesn't work in this place

var members = ['herpaderpus', 'turtles_head', 'nubstep_rs', 'ardens_fide', 'newending', 'pve_bros', 'rsphilippe', 'pureismwars', 'smap51', 'iprimal_rs', 'im_mr_bloo', 'mrknowles100', 'aikohero', 'cowsbelieve', 'dombo_12', 'diovista', 'mrpixels17'];
var memberData = [];
$.each(members, function(index, member) {
$.getJSON('https://api.twitch.tv/kraken/users/' + member + '?callback=?', function(d) {
if(d.status == 404) {}
else {
var data = [];
data[0] = member;
data[1] = d.display_name;
memberData.push(data[0]);
$.getJSON('https://api.twitch.tv/kraken/streams/' + data[0] + '?callback=?', function(d) {
if(d.stream != null) {
$( "#player" ).append( "<img src='http://pso-clan.com/twitch/lib/images/online.png'>" + "<a target='_blank' href='http://www.twitch.tv/" + data[0] + "'>" + data[1] + "</a>" + " - Viewers: " + d.stream.viewers + "<br>" );
}
else {
$( "#offline" ).append( "<img src='http://pso-clan.com/twitch/lib/images/offline.png'>" + "<a target='_blank' href='http://www.twitch.tv/" + data[0] + "'>" + data[1] + "</a> - Offline<br>" );
}
});
}
});
}); alert(memberData[0]);
I don't seem to be able to call
memberData.push(data[0]);
in the place where it's right now, the alert just show undefined. Why doesn't it properly push the member to the memberData array?

You're actually incorrect - where you're using .push() is just fine (in the callback of your first $.getJSON() request).
Where you're alerting it, however is not, because this is an asynchronous request, whereby the alert occurs before the first ajax request is complete.
Take a look at the console output here: http://jsfiddle.net/remus/spSxE/ -- you'll see that undefined appears before any of the array push logging.
Options:
Rewrite your method to use parallel ajax requests like the solution here
Write the necessary DOM update functions (or whatever you're doing with the results) into the callback of each ajax request so that it is updated after each call completes.

500 error from ajax request (jQuery) to Code Igniter controller

Was wondering if anyone had encountered this problem.
I am calling a CI controller that runs a model (on the server) and takes longer (about 5 minutes) for a specific scenario. The problem is that I am getting a 500 error after a long request but I do not get any errors when the request is shorter (about 1 and a half minutes).
Some things that I already checked:
CI's 'csrf_protection' is OFF
I've set a long timeout in my ajax call (900000)
I've set max_execution_time in PHP to 900
I've set Idle Time-Out in IIS to (20 minutes)
Here's my ajax call:
$.ajax({
type:'POST',
url:"run/runScenario/" + saveOrUpdate + "/" + scenarioname + "/" + units + "/" + stateid + "/" + climatestationid + "/" + soiltexture + "/" +
moisturecontent + "/" + modsoilflag + "/" + slopelength + "/" + slopeshape + "/" + slopesteepness + "/" + vegcommunity + "/" +
basalcover + "/" + canopycover + "/" + rockcover + "/" + littercover + "/" + cryptogamscover,
data: {scenarioDescription: scenariodescription},
timeout:900000,
statusCode: {
500: function() {
alert( "page not found" );
}
},
success: function(runData){
$('#progressBar').append('<p>Done running scenario.</p>');
$('#progressBar').append('<p>Saving scenario...</p>');
saveScenarioResultsTable();
$('#progressBar').append('<p>Creating output table...</p>');
printScenarioResultsTable(scenarioname);
},
error: function (xhr, ajaxOptions, thrownError) {
alert(xhr.responseText);
// stop timer
var end = new Date().getTime();
var execution_time = end - TIMER;
runTimeMessage = "<br/><b>Ran scenario in " + (parseInt(execution_time) * 0.001) + " seconds.</b>";
alert(runTimeMessage);
}
});
UPDATE I created a test function (as part of my run controller) and set a sleep(300) inside the function. I got the same 500 error.
But, when I change to sleep(299), the function runs successfully. Obviously, there is a 5 minute limit per request.
I have already changed the *max_execution_time* in php.ini. Any other suggestions?
UPDATE #2 I have found the solution to my problem. The problem was that because "safe_mode" was not enabled in my PHP settings, the PHP timeout was being overwritten in CodeIgniter.php (line 107). I am running CodeIgniter 2.1.4. I hope this helps someone else. :)

Just as an idea, but not a fixing of the code:
What if you will use two different ajax requests:
Request for processing and writing the result of processing in some buffer like file or sql record. This request may or may not return a random descriptor of future resource.
Request for getting results of processing by descriptor or just by availability of resources. This request have to be joined to the timer.
Timeline:
-------------------------------------------------------------------
Step Local server processing
-------------------------------------------------------------------
1. Request for processing >>> Server >>> Thread for processing
2. Descriptor of resource <<< Server >>> Descriptor of resource
3. Server <<< Result of processing
4. Request by descriptor >>> Server
5. Result <<< Server >>> Deletion of file or record
-------------------------------------------------------------------
There:
1, 2 - Ajax #1, fires by demand.
4, 5 - Ajax #2, fires every 10sec.