I have searched and searched and am just not finding the answer to this one.
I am using pg.js to run queries from my Node.js server to a Postgres database. I would like to use 1 function to run queries on two different tables. So I would like to do this:
database.query("SELECT * FROM $1 WHERE $2 = $3;",
[type, idName, parseInt(id, 10)],
(error, result) => {});
This results in a syntax error.
error: syntax error at or near "$1"
I found some SO articles that use either :name or a ~ to cast the variable as a "name". Like this:
database.query("SELECT * FROM $1~ WHERE $2~ = $3;",
[type, idName, parseInt(id, 10)],
(error, result) => {});
This results in the same error.
If I hard-code the table name and try to use the ~ on just the column name. I get the error:
error: operator does not exist: unknown ~
The only thing that seems to work is this very bad solution:
database.query("SELECT * FROM "+type+" WHERE "+idName+" = $1;",
[parseInt(id, 10)],
(error, result) => {});
Any help appreciated.
The problem you are facing is because of interpolation. Dynamic column and table names are not same as injecting dynamic values.
You might want to give this a try:
npm install pg-format
var format = require('pg-format');
var sql = format('SELECT * FROM %I WHERE my_col = %L %s', 'my_table', 34, 'LIMIT 10');
console.log(sql); // SELECT * FROM my_table WHERE my_col = '34' LIMIT 10
More details here - https://github.com/datalanche/node-pg-format
A dirty ES6 fix could be:
database.query(`SELECT * FROM ${type} WHERE ${idName} = $3;`,
[parseInt(id, 10)],
(error, result) => {});
If you plan on building a lot of dynamic queries, consider giving this a try - https://knexjs.org/
You cannot create prepared statements and cannot inject dynamic table or column names in the query because this would disallow preparing the statement completely. If you definitely have to do prepared statements, you need to know that in PreparedStatements the input datatypes (the dynamic variables) AND the return datatypes (the returned columns) MUST be defined. Since * will return differnet return types for different tables the above will never work. If you know for example that you always return an id and a name you might create a function in postgresql like this:
CREATE TYPE idname AS (id int4, name text);
CREATE OR REPLACE FUNCTION dynamicidnamequery(tablename text,field text,content text) RETURNS SETOF idname AS
$$
DECLARE
r idname;
BEGIN
FOR r IN EXECUTE 'SELECT id,name FROM '||tablename||' WHERE '||field||'='''||content||''''
LOOP
return next r;
END LOOP;
END$$ language 'plpgsql';
select * from dynamicidnamequery('company','name','Amazon');
The last select can be queried dynamic now.
I did find a partial solution. While I can not find any documentation about this on the PostGres website, I found some online articles that showed examples of using the shorthand notation for CAST, to cast a string as a column name.
As mentioned in the question, I found several stackoverflow articles that mentioned :name as a solution, but what works for me is ::name which is the shorthand for CAST. Postgres does not document name as a datatype, but this does work for me.
database.query("SELECT * FROM "+ type +" WHERE $1::name = $2;",
[idName, parseInt(id, 10)],
(error, result) => {});
The same thing does not work for the table name.
Related
So I have a Node.js query that should pass in these three values and update a specific column by a specific amount for a user with a specific id:
await client.query(sqlStatement, [columnName, changeBy, id])
The 'sqlStatement' looks like:
'UPDATE tableName SET ($1) = ($1) + ($2) WHERE id = ($3)'
As you can see, it should pass in the values such that the SQL statement looks like:
'UPDATE tableName SET columnName = columnName + changeBy WHERE id = id'
I've also tried to write sqlStatement as the below (and just pass in columnName twice into the query but this also results in an error:
'UPDATE tableName SET ($1) = ($2) + ($3) WHERE id = ($4)'
The error returned is error: syntax error at or near "$1".
Not sure how to fix this--any help is much appreciated.
The reason this is happening is that node-postgres doesn't support query parameters for identifiers and your column name after SET is an identifier. Also, even if this worked, unless node-postgres somehow substituted the entire ($1) (with parentheses) for your value, you'd get
ERROR: source for a multiple-column UPDATE item must be a sub-SELECT or ROW() expression.
If you wish to keep node-postgres, its documentation recommends to use pg-format in such cases, to build a dynamic SQL statement. If you consider alternatives, you can look into Knex.js that will build the queries for you, like so:
knex('tableName')
.update({
columnName: knex.raw('?? + ?',['columnName',changeBy])
})
.where('id', id)
And in the meanwhile, as a workaround, you should be able to set your sqlStatement to a dynamic SQL query on your own:
do language plpgsql $$ begin execute format('UPDATE test SET %I = %I + %s WHERE id = %s','$1','$1',$2,$3); end$$;
and try with that. Note that I removed the parentheses from around the update column to avoid the multiple-column UPDATE error. What should happen is that node-postgres evaluates the above to
do language plpgsql $$ begin execute format('UPDATE test SET %I = %I + %s WHERE id = %s','col','col',51,72); end$$;
and pass it to PostgreSQL, which should execute it as
UPDATE test SET "col" = "col" + 51 WHERE id = 72;
From the doc:
%I is for Identifiers, that PostgreSQL will automatically quote if needed.
%L is for string Literals
%s is for simple string substitution, in the sense that it will not be quoted, so it can be used for numeric literals
postgresql / node-postgres / javascript
I am trying to use a parameter $1 in a postrgesql query so I can sort a table based on a string variable (eventually the string variable will come from req.query, but for now it is just a variable called sort that I am creating and defining)
In my code I have:
let sort = 'user_name';
const allScores = await pool.query(
"SELECT * FROM typeTable ORDER BY $1 desc;", [sort]);
console.log(allScores);
but allScores does not sort the table by user_name (which is a valid column name)
The code below does sort the table by user_name. The only difference is that I am manually writing user_name instead of using a parameter.
const allScores = await pool.query(
"SELECT * FROM typeTable ORDER BY user_name desc;");
console.log(allScores);
In my head, I would have thought these two blocks of code would basically be equivalent but they are not. I am not sure what my mistake is. Any help to get the parameter working would be appreciated. Thanks!
From example where-col-in example and this answer, WHERE IN clauses should have query with parameters with following syntax
const response = await db.any('SELECT * FROM table WHERE id IN ($1:csv)', [data])
where data is an array.
Now, when data is an empty array, it produces the following query
SELECT * FROM users WHERE id IN ()
which is a syntax error.
Consider following statements:
this works
const x = await db.any('SELECT * FROM table WHERE id IN ($1:csv)', [[1, 2, 3]]);
this does not work
const y = await db.any('SELECT * FROM table WHERE id IN ($1:csv)', [[]]);
A similar error reported for squel library has answers on how knex and ruby's sequel behaves in such scenario.
Is this a bug or am I doing something wrong? Could there be an alternate syntax which works for both scenarios.
For instance, an alternate query using ANY works for both situations:
await db.any(`SELECT * FROM table WHERE id = ANY($1)`, [[1, 2, 3]]);
await db.any(`SELECT * FROM table WHERE id = ANY($1)`, [[]]);
What should be the best way to have WHERE col IN queries which could also handle empty arrays as params?
Library pg-promise gives you complete freedom in generating any SQL you want, it does not validate or control it in any way, as it is not an ORM.
CSV Filter is generic, it can be used in various context for generating queries. So when you are using it specifically for IN ($1:csv), it doesn't know it, and produces again generic output.
As per the comments I left in this issue, the right approach is to check whether you have any data in your array, and if not - do not execute the query at all. First, the query would be invalid, and even if you patch it with some empty logic, that means it won't generate any result, and executing such a query becomes a waste of IO.
let result = [];
if (data.length) {
result = await db.any('SELECT * FROM table WHERE id IN ($1:csv)', [data]);
}
/* else: do nothing */
Common Answer
Is this a bug or am I doing something wrong?
Not a bug, but a flaw for most SQL frameworks. It is very difficult to handle such parameters, so most frameworks just leave the empty list as it is to generate invalid SQL XXX in ().
Could there be an alternate syntax which works for both scenarios.
A simple approach is:
if(data is empty) data = [ -1 ] //fill a non-existing id
db.any('SELECT * FROM table WHERE id IN ($1:csv)', [data])
What about knex or sequel?
They are Query Builder frameworks, so they have chances to generate special SQL to handle empty lists. Popular methods used by Query Builder frameworks to handle WHERE id in () OR ...:
WHERE (id!=id) OR ...
WHERE (1=0) OR ...
WHERE (1!=1) OR ...
WHERE false OR ...
etc
Personally I do not like id!=id :)
For Some Framework
You may check its manual to see if there is some way to handle empty lists, eg: can the framework replace the empty list with a non-existing value?
I currently have a manually composed query in knex.js that uses knex.raw:
db.raw(`SELECT * FROM
(VALUES ${Array(ids.length).fill('(?)').join(',')}) v (__whereId)
LEFT JOIN LATERAL
(SELECT * FROM ?? WHERE ?? = v.__whereId ORDER BY ?? ASC LIMIT 2) t
ON true`,
[...ids, table, `${table}.${field}`, primaryKey])
I'd like to convert it to use as much knex.js built-in stuff as possible, but every time I try to come up with cleaner options using knex.wrap, knex.as etc. I run in to something that is unconvertible, or I just don't know how to write it.
I know knex.js does not support VALUES in anything except insert, so that has to be something raw, but the other things probably should be possible
I managed to find a solution to my own question:
const values = db.raw(`(VALUES ${Array(ids.length).fill('(?)').join(',')}) AS v (__whereId)`, ids);
const sub = db.table(table).whereRaw('??.?? = v.__whereId', [table, field]).orderBy(primaryKey, 'asc').limit(2).as('t');
const query = db.from(values).joinRaw('left join lateral ? on true', sub).select();
The last part is the interesting part - apparently using ? inside a raw input, and giving a sub query as the parameter for the placeholder triggers some internal knex logic to automatically wrap the query and the nested placeholders correctly.
I'm trying to perform a 'doesNotContainAllObjectsInArray' type operation on Azure Mobile Services. For example, let's say I have a table called Number and within that table are these records with these 'number' values: 11111, 22222, 33333, 44444.
I want to be able to write a query that will allow me to pass in an array of numbers that I specifically don't want, for example: [11111,44444] should yield me with [22222, 33333].
I've tried using JavaScript in my where operator, but I'm getting an error back stating that the expression isn't supported. This is what I've tried:
var numberTable = tables.getTable('Number');
var ignoreNumbers = ['11111', '44444'];
numberTable.where(function(numbers) {
return (numbers.indexOf(this.number) > -1);
}, ignoreNumbers).read({
success: function(foundNumbers) {
console.log('Found ' + foundNumbers.length + ' numbers!');
},
error: function(error) {
console.error('Error with query! ' + error);
}
});
Note: I can't hard code the ignoreNumbers values, since that array is produced from a previous query.
Can anyone recommend how I might go about executing a query like this? Would I need build a SQL statement and execute it with mssql? (...is that even possible with Table Storage?)
You are describing the SQL Except operator which isn't supported in Table Queries. The only way I've found to do this is to load the table into memory (often not feasible due to size) and then use LINQ to do an Except query.
I managed to solve this by creating a SQL query and executing it through the request.service.mssql object, something like this:
SELECT * FROM Number WHERE (number != '11111' && number != '22222')
The WHERE part of the query is built by iterating the ignoreNumbers array and building the SQL statement by string concatenation.
Not sure if it's the most efficient thing in the world, but in reality there are only going to be a couple of numbers (maybe 5-10) and so far it seems to work.