From example where-col-in example and this answer, WHERE IN clauses should have query with parameters with following syntax
const response = await db.any('SELECT * FROM table WHERE id IN ($1:csv)', [data])
where data is an array.
Now, when data is an empty array, it produces the following query
SELECT * FROM users WHERE id IN ()
which is a syntax error.
Consider following statements:
this works
const x = await db.any('SELECT * FROM table WHERE id IN ($1:csv)', [[1, 2, 3]]);
this does not work
const y = await db.any('SELECT * FROM table WHERE id IN ($1:csv)', [[]]);
A similar error reported for squel library has answers on how knex and ruby's sequel behaves in such scenario.
Is this a bug or am I doing something wrong? Could there be an alternate syntax which works for both scenarios.
For instance, an alternate query using ANY works for both situations:
await db.any(`SELECT * FROM table WHERE id = ANY($1)`, [[1, 2, 3]]);
await db.any(`SELECT * FROM table WHERE id = ANY($1)`, [[]]);
What should be the best way to have WHERE col IN queries which could also handle empty arrays as params?
Library pg-promise gives you complete freedom in generating any SQL you want, it does not validate or control it in any way, as it is not an ORM.
CSV Filter is generic, it can be used in various context for generating queries. So when you are using it specifically for IN ($1:csv), it doesn't know it, and produces again generic output.
As per the comments I left in this issue, the right approach is to check whether you have any data in your array, and if not - do not execute the query at all. First, the query would be invalid, and even if you patch it with some empty logic, that means it won't generate any result, and executing such a query becomes a waste of IO.
let result = [];
if (data.length) {
result = await db.any('SELECT * FROM table WHERE id IN ($1:csv)', [data]);
}
/* else: do nothing */
Common Answer
Is this a bug or am I doing something wrong?
Not a bug, but a flaw for most SQL frameworks. It is very difficult to handle such parameters, so most frameworks just leave the empty list as it is to generate invalid SQL XXX in ().
Could there be an alternate syntax which works for both scenarios.
A simple approach is:
if(data is empty) data = [ -1 ] //fill a non-existing id
db.any('SELECT * FROM table WHERE id IN ($1:csv)', [data])
What about knex or sequel?
They are Query Builder frameworks, so they have chances to generate special SQL to handle empty lists. Popular methods used by Query Builder frameworks to handle WHERE id in () OR ...:
WHERE (id!=id) OR ...
WHERE (1=0) OR ...
WHERE (1!=1) OR ...
WHERE false OR ...
etc
Personally I do not like id!=id :)
For Some Framework
You may check its manual to see if there is some way to handle empty lists, eg: can the framework replace the empty list with a non-existing value?
Related
TWO QUESTIONS IN ONE:
Number one: This returns Failing row contains for name column and some more columns when trying to do it in SQL. However I know for a fact that the id allready exsist in the table. So how can I prevent it from giving this error?
Number two: The code returns error: syntax error at or near "$1" when ran in node? Is it beacuse values is a string and how can i prevent this?
This gives error error: syntax error at or near "$1". I have a feeling
let values = ("("+req.body.oldid+","+false+"),("+req.body.newid+","+true+")")
console.log(values) // returns (70,false),(4,true)
const results = await db.query("INSERT INTO practiceSite (id,frontpage) VALUES $1 ON CONFLICT DO UPDATE frontpage = values(frontpage);",[values])
Based on your use case, it looks like you'll only ever be inserting 2 entries at a time, so this should work.
In your example values is seen as a string, so we need to be more granular and pass in each parameter, and create the rows.
const results = await db.query(`
INSERT INTO practiceSite (id, frontpage)
VALUES ($1, FALSE), ($2, TRUE)
ON CONFLICT
DO UPDATE SET frontpage = EXCLUDED.frontpage;
`,
[req.body.oldid, req.body.newid]
)
In my backend using nestjs + typeorm + postgresql I have a CustomRepository and want to repalce some plain sql queries.
This is what I have
const sqlQuery = `SELECT DISTINCT "myColumn" FROM "myTable"`
const sqlRes = await this.query(sqlQuery);
I am trying to get something like this
this.find({select:["myColumn"]}); // note the missing DISTINCT
But this is giving me the complete column but I want just the DISTINCT values.
I found a lot of weird createQueryBuilder().select(DISTINCT "myColumn" FROM "...... etc... solutions, which are not really giving my any benefit over my working solution.
You could do:
await getManager().createQueryBuilder('entity')
.select('column')
.distinct(true)
.getRawMany();
I am currently using the database class from http://vitaly-t.github.io/pg-promise/Database.html and trying to implement an Update statement using a PreparedStatment on my Postgres DB while having an Array passed to the WHERE IN clause ?
const updatePreparedStatment = new PS('prepared-statement', 'UPDATE mytable SET "MESSAGE"=$1 WHERE "ID" IN ($2)', ["dummy update", ["1","2","3"]]);
It is described in the FAQ of node-postgres https://github.com/brianc/node-postgres/wiki/FAQ#11-how-do-i-build-a-where-foo-in--query-to-find-rows-matching-an-array-of-values
How do I build a WHERE foo IN (...) query to find rows matching an array of values?
node-postgres supports mapping simple JavaScript arrays to PostgreSQL arrays, so in most cases you can just pass it like any other parameter.
client.query("SELECT * FROM stooges WHERE name = ANY ($1)", [ ['larry', 'curly', 'moe'] ], ...);
Note that = ANY is another way to write IN (...), but unlike IN (...) it will work how you'd expect when you pass an array as a query parameter.
If you know the length of the array in advance you can flatten it to an IN list:
// passing a flat array of values will work:
client.query("SELECT * FROM stooges WHERE name IN ($1, $2, $3)", ['larry', 'curly', 'moe'], ...);
... but there's little benefit when = ANY works with a JavaScript array.
If you're on an old version of node-postgres or you need to create more complex PostgreSQL arrays (arrays of composite types, etc) that node-postgres isn't coping with, you can generate an array literal with dynamic SQL, but be extremely careful of SQL injection when doing this. The following approach is safe because it generates a query string with query parameters and a flattened parameter list, so you're still using the driver's support for parameterised queries ("prepared statements") to protect against SQL injection:
var stooge_names = ['larry', 'curly', 'moe'];
var offset = 1;
var placeholders = stooge_names.map(function(name,i) {
return '$'+(i+offset);
}).join(',');
client.query("SELECT * FROM stooges WHERE name IN ("+placeholders+")", stooge_names, ...);
Hope that helps since google fails to find this
I have searched and searched and am just not finding the answer to this one.
I am using pg.js to run queries from my Node.js server to a Postgres database. I would like to use 1 function to run queries on two different tables. So I would like to do this:
database.query("SELECT * FROM $1 WHERE $2 = $3;",
[type, idName, parseInt(id, 10)],
(error, result) => {});
This results in a syntax error.
error: syntax error at or near "$1"
I found some SO articles that use either :name or a ~ to cast the variable as a "name". Like this:
database.query("SELECT * FROM $1~ WHERE $2~ = $3;",
[type, idName, parseInt(id, 10)],
(error, result) => {});
This results in the same error.
If I hard-code the table name and try to use the ~ on just the column name. I get the error:
error: operator does not exist: unknown ~
The only thing that seems to work is this very bad solution:
database.query("SELECT * FROM "+type+" WHERE "+idName+" = $1;",
[parseInt(id, 10)],
(error, result) => {});
Any help appreciated.
The problem you are facing is because of interpolation. Dynamic column and table names are not same as injecting dynamic values.
You might want to give this a try:
npm install pg-format
var format = require('pg-format');
var sql = format('SELECT * FROM %I WHERE my_col = %L %s', 'my_table', 34, 'LIMIT 10');
console.log(sql); // SELECT * FROM my_table WHERE my_col = '34' LIMIT 10
More details here - https://github.com/datalanche/node-pg-format
A dirty ES6 fix could be:
database.query(`SELECT * FROM ${type} WHERE ${idName} = $3;`,
[parseInt(id, 10)],
(error, result) => {});
If you plan on building a lot of dynamic queries, consider giving this a try - https://knexjs.org/
You cannot create prepared statements and cannot inject dynamic table or column names in the query because this would disallow preparing the statement completely. If you definitely have to do prepared statements, you need to know that in PreparedStatements the input datatypes (the dynamic variables) AND the return datatypes (the returned columns) MUST be defined. Since * will return differnet return types for different tables the above will never work. If you know for example that you always return an id and a name you might create a function in postgresql like this:
CREATE TYPE idname AS (id int4, name text);
CREATE OR REPLACE FUNCTION dynamicidnamequery(tablename text,field text,content text) RETURNS SETOF idname AS
$$
DECLARE
r idname;
BEGIN
FOR r IN EXECUTE 'SELECT id,name FROM '||tablename||' WHERE '||field||'='''||content||''''
LOOP
return next r;
END LOOP;
END$$ language 'plpgsql';
select * from dynamicidnamequery('company','name','Amazon');
The last select can be queried dynamic now.
I did find a partial solution. While I can not find any documentation about this on the PostGres website, I found some online articles that showed examples of using the shorthand notation for CAST, to cast a string as a column name.
As mentioned in the question, I found several stackoverflow articles that mentioned :name as a solution, but what works for me is ::name which is the shorthand for CAST. Postgres does not document name as a datatype, but this does work for me.
database.query("SELECT * FROM "+ type +" WHERE $1::name = $2;",
[idName, parseInt(id, 10)],
(error, result) => {});
The same thing does not work for the table name.
I currently have a manually composed query in knex.js that uses knex.raw:
db.raw(`SELECT * FROM
(VALUES ${Array(ids.length).fill('(?)').join(',')}) v (__whereId)
LEFT JOIN LATERAL
(SELECT * FROM ?? WHERE ?? = v.__whereId ORDER BY ?? ASC LIMIT 2) t
ON true`,
[...ids, table, `${table}.${field}`, primaryKey])
I'd like to convert it to use as much knex.js built-in stuff as possible, but every time I try to come up with cleaner options using knex.wrap, knex.as etc. I run in to something that is unconvertible, or I just don't know how to write it.
I know knex.js does not support VALUES in anything except insert, so that has to be something raw, but the other things probably should be possible
I managed to find a solution to my own question:
const values = db.raw(`(VALUES ${Array(ids.length).fill('(?)').join(',')}) AS v (__whereId)`, ids);
const sub = db.table(table).whereRaw('??.?? = v.__whereId', [table, field]).orderBy(primaryKey, 'asc').limit(2).as('t');
const query = db.from(values).joinRaw('left join lateral ? on true', sub).select();
The last part is the interesting part - apparently using ? inside a raw input, and giving a sub query as the parameter for the placeholder triggers some internal knex logic to automatically wrap the query and the nested placeholders correctly.