Im just new to knex and came across Transactions. I think it's useful to use since it has a rollback feature. Tried using it (see code below)
await knex.transaction(trx => {
knex.raw(delete from myTable where "id" = 1 )
.transacting(trx)
.then(trx.commit)
.catch(trx.rollback)
})
I just wanted to delete a row with a certain id nothing more and less.
Works fine, then i tried to remove 'trx.commit'. I was expecting that it wont apply the query but it did. From what I understand, if trx.commit is not called the query will not run and wont affect the database.
Is my understanding wrong? Did I use knex.raw improperly inside knex.transactions? I dont see examples of transactions that uses raw queries. I am connected to a database in my localhost(postgresql) btw.
knex.js has a modified promise interface.
Calling .then triggers the query to actually fire (including the BEGIN transaction if it is the first query). Note that in a single knex query you won't need to call rollback since a single query to the database should be transactional and abort/rollback if any errors are encountered.
Based on your usage (and the docs) if you remove trx.commit it should not commit at all. I recommend actually returning a promise to the transaction callback - then it will auto-commit on promise resolution, and auto-rollback on promise failure.
In the following usage if either query failed it would auto-rollback everything.
knex.transaction(trx => {
return Promise.all([
knex.raw(`update table x`).transacting(trx),
knex.raw(`update table y`).transacting(trx)
]);
})
Related
I am having a problem where I am making a bulk insert of multiple elements into a table, then I immediatly get the last X elements from that table that were recently inserted but when I do that it seems that the elements have not yet been inserted fully even thought I am using async await to wait for the async operations.
I am making a bulk insert like
const createElements = elementsArray => {
return knex
.insert(elementsArray)
.into('elements');
};
Then I have a method to immediately access those X elements that were inserted:
const getLastXInsertedElements = (userId, length, columns=['*']) => {
return knex.select(...columns)
.from('elements').where('userId', userId)
.orderBy('createdAt', 'desc')
.limit(length);
}
And finally after getting those elements I get their ids and save them into another table that makes use of element_id of those recently added elements.
so I have something like:
// A simple helper function that handles promises easily
const handleResponse = (promise, message) => {
return promise
.then(data => ([data, undefined]))
.catch(error => {
if (message) {
throw new Error(`${message}: ${error}`);
} else {
return Promise.resolve([undefined, `${message}: ${error}`])
}
}
);
};
async function service() {
await handleResponse(createElements(list), 'error text'); // insert x elements from the list
const [elements] = await handleResponse(getLastXInsertedElements(userId, list.length), 'error text') // get last x elements that were recently added
await handleResponse(useElementsIdAsForeignKey(listMakingUseOfElementsIds), 'error text'); // Here we use the ids of the elements we got from the last query, but we are not getting them properly for some reason
}
So the problem:
Some times when I execute getLastXInsertedElements it seems that the elements are not yet finished inserting, even thought I am waiting with async/await for it, any ideas why this is? maybe something related to bulk inserts that I don't know of? an important note, all the elements always properly inserted into the table at some point, it just seems like this point is not respected by the promise (async operation that returns success for the knex.insert).
Update 1:
I have tried putting the select after the insert inside a setTimeout of 5 seconds for testing purposes, but the problem seems to persist, that is really weird, seems one would think 5 seconds is enough between the insert and the select to get all the data.
I would like to have all X elements that were just inserted accessible in the select query from getLastXInsertedElements consistently.
Which DB are you using, how big list of data are you inserting? You could also test if you are inserting and getLastXInsertedElements in a transaction if that hides your problem.
Doing those operations in transaction also forces knex to use the same connection for both queries so it might lead to a tracks where is this coming from.
Another trick to force queries to use the same connection would be to set pool's min and max configuration to be 1 (just for testing is parallelism is indeed the problem here).
Also since you have not provided complete reproduction code for this, I'm suspecting there is something else here in the mix which causes this odd behavior. Usually (but not always) this kind of weird cases that shouldn't happen are caused by user error in elsewhere using the library.
I'll update the answer if there is more information provided. Complete reproduction code would be the most important piece of information.
I am not 100% sure but I guess the knex functions do not return promise by default (but a builder object for the query). That builder has a function called then that transforms the builder into a promise. So you may try to add a call to that:
...
limit(length)
.then(x => x); // required to transform to promise
Maybe try debugging the actual type of the returned value. It might happen that this still is not a promise. In this case you may not use async await but need to use the then Syntax because it might not be real js promises but their own implementation.
Also see this issue about standard js promise in knex https://github.com/knex/knex/issues/1588
In theory, it should work.
You say "it seems"... a more clear problem explanation could be helpful.
I can argue the problem is that you have elements.length = list.length - n where n > 0; in your code there are no details about userId property in your list; a possible source of the problem could be that some elements in your list has a no properly set userId property.
I'm writing a code for a chatbot and i'm having a problem with the database query on my firebase DB that I don't know how to fix, because it is the same as the other examples I found.
var ref = admin.database().ref();
var consultasRef = ref.child('consultas')
agent.add('Test 2')
consultasRef.on("value", function(snap){
agent.add('Test 3')
agent.add(snap.val());
});
As said the function that should print the snap.(val). It's not starting as it should. The right syntax is the same as the one I use. "Test 2" is being printed, unlike "Test 3", which is inside the function
For those unfamiliar with Dialogflow, agent.add() is the same as console.log() on JS, but for Dialogflow.
When you make an asynchronous call (such as a database call), you must return a Promise from your Intent Handler so it knows when the call has completed so it can send the result back to the bot.
Additionally, you probably want to use the once() function, since you don't care about the database updating (since the result will already have been sent).
You can probably do both of these with code looking something like this:
return consultasRef.once('value')
.then( snap => {
agent.add('Test 3');
agent.add(snap.val());
});
There may also be issues with adding more than one or two text replies, depending on the Integration you're using.
I'm running into problems with the validated method package in my app tests. I'm calling my methods through the _execute function in order to be able to pass a userId to simulate a logged-in user while testing. The problem is that my asserts right underneath that _execute are called before the method has a chance of completing. I know my test works though because it only happens sometimes, mostly because mongo isn't always returning results quite as fast.
I looked around and found a todos app that uses the _execute function in its tests. I can't get those tests to fail no matter how many times I rerun them, though.
This is an example of my test code.
describe('clients.add', function() {
it('should add an empty (draft) client', function() {
const res = clients_add._execute({ userId: 'CURRENTUSERID' }, { company_id: c1._id });
assert.instanceOf(res, Mongo.ObjectID, 'method returns the newly created clients ID');
const db_client = Clients.findOne(res);
assert.isTrue(db_client.draft, 'client is drafted');
assert.isDefined(db_client.created, 'there\'s a created date');
});
});
clients_add does quite a few permission checks and can therefor take a little while before completing. Rerunning this test 20 times will fail about 5 times and pass the other 15.
Shouldn't the _execute function be synchronous? How do I make it? What am I missing?
In server code, if you provide a callback to database modification functions like insert, it returns the created ID instantaneously, and runs the callback only once the database has acknowledged the write. If you don't provide a callback, the insert call is synchronous and throws an error if the operation fails. See more about this in Meteor docs.
It seems that you have provided an error-handling callback to the insert-function in your method code. This causes the inconsistent behavior, since the database might not actually have had time to do the write before you call findOne in your test. Also, this is redundant since if an error occurs in the insert, the method has already returned and the error is never shown to the user. It's better to simply omit the error-handling callback altogether:
return Clients.insert(new_client);
I got tripped up by how the Knex API for building a schema doesn't actually create a table until you call .then.
For example, this code won't actually affect the database:
knex.schema.createTable('users', table => {
table.string('name')
})
But this code will:
knex.schema.createTable('users', table => {
table.string('name')
}).then(console.log.bind(console))
Is this behavior (not doing anything until .then is called):
a) required by the Promises A+ spec
b) prohibited by the Promises A+ spec
c) unspecified
?
I read the spec, and it seems like the behavior is unspecified, but I'm not sure. This seems too important to be unspecified.
Update
see #Vohuman's answer: The then method on the Knex schema builder first performs a side effect, then returns a promise. So even if the answer to my question is (b) Knex wouldn't be in violation of the spec. Though the choice of then as a method name is very misleading in this case.
This isn't exactly "wrong", though it isn't common. Knex requires .then because it has to be able to tell when the query is done, as opposed to mid-build.
Example from the docs (commentary added):
knex.select('name').from('users')
.where('id', '>', 20) // Can I issue the query yet?
.andWhere('id', '<', 200) // Okay I'll...oh, you have more conditions.
.limit(10) // Hey, I'm here, still ready...
.offset(x) // Just let me know when you want me to--
.then(function(rows) { // OH OKAY I'll issue it now.
return _.pluck(rows, 'name');
})
Google's API.js helper follows this pattern too, specifically for immediate vs batched queries:
When you create a request with the intention of adding it to a batch, do not invoke its then method until after the request has been added to the batch. If the then method is invoked before the request is added, the request is sent immediately instead of as part of the batch.
As SLaks pointed out, this is not explicitly specified anywhere in the docs: In order for the object to be Promises/A+ compliant, it must have a method named then with the proper semantics and return value, and nothing specifies that then cannot have additional behavior. Naturally this prohibits these API libraries from renaming the then method to something more apt, like thenIssueRequest or thenGetResponse. (You can add an alias, but then is required to exist.)
As an API designer, the only alternative would be to separate the creation of the promise with the chaining of then, with the caveat that almost every call to then would have an extra method call that precedes or wraps it. Because then is the only way to access the result, I can understand how optimizing for the common case would result in removing the extra method.
fluentApi.createRequest().parameterA(1).parameterB(2).issue().then(...);
// or
fluentApi.issue(fluentApi.createRequest().parameterA(1).parameterB(2)).then(...);
Finally, bear in mind that you should always catch a Promise at some point, which would trigger the request:
knex.schema.createTable('users', table => {
table.string('name')
}).catch(console.log.bind(console));
...and that both arguments to then are optional, so if you insist on skipping error handling you can even have an empty then:
knex.schema.createTable('users', table => {
table.string('name')
}).then(null);
knex.schema.createTable('users', table => {
table.string('name')
}).then();
This behavior is wrong; calling then() should not have any side-effects beyond the promise itself (i.e., executing the callback).
However, the spec doesn't actually say anything about this.
knex is a query builder. then is just a helper function that executes the generated [SQL] statements and calls the then function of a promise object behind the scenes. If you replace the then with the .toString function you will get the generated string. then here is not a method of a promise object.
From knex source code:
Target.prototype.then = function(/* onFulfilled, onRejected */) {
const result = this.client.runner(this).run()
return result.then.apply(result, arguments);
};
I am using PhoneGap and jQuery Mobile. I am trying to get some JSON data from a remote location and then populate a local WebSQL database with it. Here is my JS function:
function getLocations() {
var tx = window.openDatabase('csdistroloc', '1.0', 'Distro DB', 1000000);
tx.transaction(function(tx) {
tx.executeSql('DROP TABLE IF EXISTS locations'); //this line works!
tx.executeSql('CREATE TABLE IF NOT EXISTS locations (id, name, address, postalcode, phone, category)'); //this line works!
$.ajax({
url: "http://mydomain.com/api.php",
dataType: 'json',
data: { action: "getlocations" },
success: function(data) {
tx.executeSql("INSERT INTO locations (id, name, address, postalcode, phone, category) VALUES (2,'cheese','232','seven',5,6)"); //this line produces an error
}});
}, dberror, dbsuccess);
}
Running the above function gives me an error "INVALID_STATE_ERR: DOM Exception 11" on the line noted above. It does the same thing when I am actually trying to use the returned JSON data to insert data. I have also tried the $.getJSON technique with the exact same result.
Any advice would be appreciated!
Although the accepted answer is correct, I would like to expand upon it because I encountered the same problem and that answer doesn't say why it doesn't work as the OP had it.
When you create a transaction in Web SQL, the transaction processing remains alive only so long as there are any statements queued up in the transaction. As soon as the pipeline of statements in the transaction dries up, the engine closes (commits) the transaction. The idea is that when the function(tx) { ... } callback runs,
It executes all of the statements it need to. executeSql is asynchronous, so it returns immediately even though the statement has not yet been executed.
It returns control back to the Web SQL engine.
At this point the engine notices that there are statements queued up and runs them to completion before closing the transaction. In your case, what happens is this:
You call executeSql twice to queue up two statements.
You request something through ajax.
You return
The engine runs the two statements that it has queued up. The ajax request is also running asynchronously but it must access the network which is slow so it likely has not completed yet. At this point, the statement queue is empty and the Web SQL engine decides that it's time to commit and close the transaction! It has no way of knowing that there is going to be another statement coming later! By the time the ajax call returns and it attempts to execute the INSERT INTO locations, it's too late, the transaction is already closed.
The solution suggested by the accepted answer works: don't use the same transaction inside the ajax callback but create a new one. Unfortunately, it has the pitfall you would expect from using 2 transactions instead of 1: the operation is no longer atomic. That may or may not be important for your application.
If atomicity of the transaction is important for you, your only 2 recourses are:
Do everything (all 3 statements) in one transaction inside the ajax callback.
This is what I recommend. I think it's very likely that waiting until after the ajax request completes before creating the table is compatible with your application requirements.
Perform the ajax request synchronously as explained here.
I don't recommend that. Asynchronous programming in JavaScript is a good thing.
By the way, I encountered the problem in the context of Promises, in code that looked something like this:
// XXX don't do this, it doesn't work!
db.transaction(function(tx) {
new Promise(function(resolve, reject) {
tx.executeSql(
"SELECT some stuff FROM table ....", [],
function(tx, result) {
// extract the data that are needed for
// the next step
var answer = result.rows.item( .... ).some_column;
resolve(answer);
}
)
}).then(function(answer) {
tx.executeSql(
"UPDATE something else",
// The answer from the previous query is a parameter to this one
[ ... , answer, .... ]
)
});
});
The problem is that, with promises, the chained .then() clause is not run immediately upon resolution of the original promise. It is only queued for later execution, much like the ajax request. The only difference is that, unlike the slow ajax request, the .then() clause runs almost immediately. But "almost" is not good enough: it may or may not run soon enough to slip in the next SQL statement into the queue before the transaction gets closed; accordingly, the code may or may not produce the invalid state error depending on timing and/or browser implementation.
Too bad: Promise would have been useful to use inside SQL transactions. The above pseudo-example can easily be rewritten without promises, but some use cases can greatly take advantage of chains of many .then()s as well as things like Promise.all that can make sure that an entire collection of SQL statements run in any order but all complete prior to some other statement.
I would first suggest not naming your database 'tx' but rather db or database. This could be a variable naming problem since both the function parameter and your database variables are called "tx"
EDIT: I had this same problem and solved it by making the query within the callback it's own transaction. Like so:
success: function(data) {
tx.transaction(function(transaction){
transaction.executeSql("INSERT INTO locations (id, name, address, postalcode, phone, category)
VALUES (2,'cheese','232','seven',5,6)"); //now more DOM exception!
}
}}
I think the problem is by the time the callback is fired the outer transaction has completed because webSQL's transactions are not synchronous.
We do have a way to lock the transaction, while you do any AJAX or other async operation. Basically before calling AJAX you need to start a dummy db operation and on success of that operation check if AJAX is done or not, and again call the same dummy operation till your AJAX is done. When AJAX is done you can now reuse the transaction object do next set of executeSQLs. This approach is thoroughly explained in this article here. (I hope someone will not delete this answer too, as someone did earlier on a similar question)
Try this approach