I'm dealing with a promblem for a couple of days, and I'm really hoping, you could help me.
It's a node.js based API using sequelize for MySQL.
On certain API calls the code starts SQL transactions which lock certain tables, and if I send multiple requests to the API simultaneously, I got LOCK_WAIT_TIMEOUT errors.
var SQLProcess = function () {
var self = this;
var _arguments = arguments;
return sequelize.transaction(function (transaction) {
return doSomething({transaction: transactioin});
})
.catch(function (error) {
if (error && error.original && error.original.code === 'ER_LOCK_WAIT_TIMEOUT') {
return Promise.delay(Math.random() * 1000)
.then(function () {
return SQLProcess.apply(self, _arguments);
});
} else {
throw error;
}
});
};
My problem is, the simultaneously running requests lock each other for a long time, and my request returns after a long-long time (~60 seconds).
I hope I could explain it clear and understandable, and you could offer me some solution.
This may not be a direct answer to your question, but maybe by looking at why you had this problem would also help.
1) What does that doSomething() do? Anyway we can do some improvements there?
First, a transaction that take 60 sec is suspicious.. If you lock a table for that long, chances are the design should be revisited. Given a typical db operation runs 10 - 100 ms.
Ideally, all the data preparation should be done outside of the transaction, including data read from database. And the transaction should be really for only transactional operations.
2) Is it possible to use mysql stored procedure?
True, the stored procedure for mysql is not compiled, as PL/SQL for Oracle. But it is still running on the database server. If your application is really complicated and contain a lot of back and force network traffic between database and your node application in that transaction, and considering there are so many layer of javascript calls, it could really slowing things down. If 1) doesn't save you a lot of time, consider using mysql stored procedure.
The drawback of this approach, obviously, is that it is harder to maintain the codes in both nodejs and mysql.
If 1) and 2) are definitely not possible, you may consider some kind of flow control or queuing tool. Either your app make sure the 2nd request doesn't go until the first one finishes, or your have some 3rd party queuing tools to handle that. Seems you don't need any parallelism in running those requests anyway.
The main reason for deadlocks is poor database design. Without further information about your database design and which exact queries might or might not lock each other it is impossible to give you a specific solution for your problem.
However I can give you a general advice/approach to solve this issue:
I would make sure that your database is normalized at least into Third Normal Form or, if that still isnt enough even further. There might be tools to automate this process for you.
Aside from reducing the likelihood of deadlocks this also helps keeping your data consistent, which is always a good thing.
Keep your transactions as slim as possible. If you are inserting new rows into your tables and update other tables accordingly you might want to use a Trigger rather than another SQL statement to do so. The same applies to reading rows and values. Such things can be done before or after your transaction.
Choose the correct Isolation Level. Possible isolation levels are:
READ_UNCOMMITTED
READ_COMMITTED
REPEATABLE_READ
SERIALIZABLE
Sequelize's official documentation describes how you can set the isolation level and lock/unlock transactions by yourself.
As I said, without further insight about your database and query design thats all I can do for you right now.
Hope this helps.
Related
I am building a billing web app where users update stock of an item frequently. So it will cost me much. Is there a way to only update the stock without causing a document write?
Short answer: no
Long answer: most likely not
Each document updated will count as a document write for billing on Firestore. Even if you try to bundle multiple updates into a single batch call.
One suggestion would be to throttle updates on the client side to slow down rapid updates. This will however affect the accuracy of the data your end users will get while using the app. Example of throttling: Small article on throtting/debouncing
Another suggestion is to avoid making updates that are meaningless.
You can check if a value isn't going to change or if the value change isn't important before trying to do an update:
function updateValue(newValue) {
// Check if value has changed
if (newValue === value) {
// pass
return;
}
// make an update request
// ...
}
Any other suggested solution would require your end users sharing the same network or introducing another service outside of Firestore.
Last thing to consider is that maybe it's too soon to consider billing. Firestore is pretty cheap for what it offers. Here's a good video by Firebase about billing:
Firebase video on Getting to know pricing
If none of these suggestions has been helpful, I'd recommend adding some code examples, etc for better suggestions :)
At the risk of getting roasted for not posting code, what is the best way for getting around the 6 concurrent call limitation for ajax requests?
So currently I'm working with an application that can have up to 40 or so ajax requests on page load. For background, most of these requests are for various graphs, hidden behind tabs. There are some requests that can be triggered by the user (such as updating the name of an entity without refreshing the page). This means that with the limitation on concurrent requests the user won't be able to change anything until there's only 5 other requests running, and that's an unacceptable user experience.
This may mean that the app is structured badly, but most of the things loading are not required right away.
Anyway, I've looked a bit into fetch() and webworkers but can't find any information on whether these would help get around the limitation.
One solution would be to put resources on different subdomains, but this makes the backend API unnecessarily complicated (and it's a browser issue, not a server issue).
I've considered these approaches:
delay requests until the user actively needs them (IMO this is a bad user experience because they will have to wait a little bit a lot, which is annoying)
create a queuing system that leaves open one spot for user initiated requests (I'm not sure how to implement this, but it should be doable)
restructure the API so that more data is returned per request (this again is mainly a backend solution that feels a little dirty and unRESTful. Also it won't necessarily improve the load time)
chaining calls such as with Multiple Async AJAX Calls Best Practice (however given there are an unpredictable number of calls on unrelated endpoints so I don't think this is all that practical here)
webworkers? (again, not sure if this could help, since this is used to multithread js)
fetch()? (I can't find info on whether this is subject to the same limitation)
This is very much opinion based.
40 requests is not unreasonable but depending on your server and site setup it can take quite a while.
With that many calls I would bundle some of them together in a initializePage=X call. This does involve some serverside work.
Delay requests is not necessarily bad, depending on your estimated time to deliver. If possible you could present some kind of animation or "expected result" until the response ticks in, to keep the user entertained. The same applies to Queing your requests.
Restructuring your code to return everything in a bundle could also greatly speed up your site if you run a lot of initialization on your server (like security checks).
If performance is still a concern you can look into connections that provide faster results such as EventSource or WebSocket. Such a faster connection also allows for a more flexible approach to chaining. EventSource, for instance, supports events, so you could set several events on a single, bundled request and fire them as the server returns data.
Webworkers is not the answer, as the problem here is connection speed and concurrent connection limits.
I don't think we can answer this question directly. Several of the solutions you have mentioned are viable but vary by level of effort. If you are willing to adjust architecture you can consider a GraphQL approach which can wrap the bundling for you. I would also say that you can maintain REST but have a special proxy service that bundles data for you. I'd also say, don't let RESTfullness dictate or force how you develop.
Also, delaying requests until the user needs them seems like the appropriate choice to me. It's the basis for why we have "above the fold" CSS styling and infinite scrolling. Load what is needed right now first and defer the stuff that might not actually matter when it needs to be.
Concurrency of AJAX calls would come into picture if these requests are called from one thread. If WebWorker is used with AJAX then no issues at all, reason being each instance of webworker will be isolated, in a thread that is not in the main thread.
I would call that as JaxWeb and I will be pushing a git repo in coming week where you may find pure JS code that takes care of it. This is being tested right now, but yeah it does solve the problem.
Example:
Add below code in JaxWeb.js
onmessage = function (e) {
var JaxWeb = function (e) {
return {
requestChannel: {},
get_csrf_token: function () {
return this._csrf_token;
},
set_csrf_token: function (_csrf_token = null) {
this._csrf_token = _csrf_token;
},
prepare: (function ( e ) {
this.requestChannel = new XMLHttpRequest();
this.requestChannel.onreadystatechange = function () {
if (this.readyState == 4 && this.status == 200) {
postMessage(JSON.parse(this.responseText));
}
};
this.requestChannel.open(e.data.method, e.data.callname, true);
this.requestChannel.setRequestHeader("X-CSRF-TOKEN", e.data.token);
var postData = '';
if (e.data.data)
postData = JSON.stringify(e.data.data);
this.requestChannel.send(postData);
})(e)
}
};
return JaxWeb(e);
}
Usage:
jaxWebGetServerResponse = function () {
var wk2 = new Worker('path_to_jaxweb_js/JaxWeb.js');
wk2.postMessage({
"callname": '<url end point>',
"method": '<your http method>',
"data": ''
});
wk2.onmessage = function (serverResponse) {
//
//process results
//with data that is received from server
}
};
//Invoke the function
jaxWebGetServerResponse();
Sometimes I'm having issues with firebase when the user is on a slow mobile connection. When the user saves an entry to firebase I actually have to write to 3 different locations. Sometimes, the first one works, but if the connection is slow the 2nd and 3rd may fail.
This leaves me with entries in the first location that I constantly need to clean up.
Is there a way to help prevent this from happening?
var newTikiID = ref.child("tikis").push(tiki, function(error){
if(!error){
console.log("new tiki created")
var tikiID = newTikiID.key()
saveToUser(tikiID)
saveToGeoFire(tikiID, tiki.tikiAddress)
} else {
console.log("an error occurred during tiki save")
}
});
There is no Firebase method to write to multiple paths at once. Some future tools planned by the team (e.g. Triggers) may resolve this in the future.
This topic has been explored before and the firebase-multi-write README contains a lot of discussion on the topic. The repo also has a partial solution to client-only atomic writes. However, there is no perfect solution without a server process.
It's important to evaluate your use case and see if this really matters. If the second and third writes failed to write to a geo query, chances are, there's really no consequence. Most likely, it's essentially the same as if the first write had failed, or if all writes had failed; it won't appear in searches by geo location. Thus, the complexity of resolving this issue is probably a time sink.
Of course, it does cost a few bytes of storage. If we're working with millions of records, that may matter. A simple solution for this scenario would be to run and audit report that detects broken links between the data and geofire tables and cleans up old data.
If an atomic operation is really necessary, such as gaming mechanics where fairness or cheating could be an issue, or where integrity is lost by having partial results, there are a couple options:
1) Master Record approach
Pick a master path (the one that must exist) and use security rules to ensure other records cannot be written, unless the master path exists.
".write": "root.child('maste_path').child(newData.child('master_record_id')).exists()"
2) Server-side script approach
Instead of writing the paths separately, use a queue strategy.
Create an single event by writing a single event to a queue
Have a server-side process monitor the queue and process events
The server-side process does the multiple writes and ensures they
all succeed
If any fail, the server-side process handles
rollbacks or retries
By using the server-side queue, you remove the risk of a client going offline between writes. The server can safely survive restarts and retry events or failures when using the queue model.
I have had the same problem and I ended up choosing to use condition Conditional Request with the Firebase REST API in order to write data transactionally. See my question and answer. Firebase: How to update multiple nodes transactionally? Swift 3 .
If you need to write concurrently (but not transactionally) to several paths, you can do that now as Firebase supports multi-path updates. https://firebase.google.com/docs/database/rest/save-data
https://firebase.googleblog.com/2015/09/introducing-multi-location-updates-and_86.html
I have the following problem:
Breeze fetches metadata (23.4KB)
Breeze fetches lookups (4.5MB)
Right after lookups are downloaded, the browser will become unresponsive for about 30 seconds.
After this, everything works like a charm.
Why does breeze not use timeouts to inform the UI?
Firefox complains about long script operation, unresponsive, etc. The task manager (shows Firefox/Chrome/etc) as unresponsive.
Am I doing something wrong, or this is by design?
If this is by design, can i use a 'Web Worker' to do all the heavy operations and then return the whole model or something?
I tried something like this:
var test = function (name) {
return Q.fcall(function () {
setTimeout(function () {
toastr.success(name); // Notify me
return EntityQuery.from(name)
.using(manager).execute()
}, 1000) // This should be zero
});
};
var primeData = function (name) {
return test('Languages')
.then(test('dummy1'))
.then(test('dummy2'))
.then(test('dummy3'))
.then(test('dummy4'))
};
However the notifications seem to be poping up all at the same time, indicating that
return EntityQuery.from(name)
.using(manager).execute()
does not return when entity constuction finishes but when the JSON data for this entity arrived.
EDIT
Answer with webWorker provided here : BreezeJs with dedicated web worker
I think I see your point. Breeze hogs the UI thread while processing those thousands of arriving entities. If Breeze could somehow realize how much work it was doing, and would be doing, it could throw a timeout in there to give the UI a chance to breathe.
I'm not sure how safe that would be as Breeze would have to pick a moment that didn't leave the cache in an unstable state from someone's perspective.
I believe you can make this easier on yourself by breaking the one giant Lookups call into several smaller ones. You could still async await completion of all the smaller lookup promises if that is critical to your app. The fact that they are independent promise callbacks should give you the relief you seek.
Please try that and let us know how it works for you.
P.S.: You also have a cool opportunity here to optionally cache these lookups in local storage (indexdb) so you don't have to download them everytime. You'd need a versioning scheme of course and some plumbing so this lies in your future once things are looking good.
I'm currently experimenting with node.js and WebSocket, building a simple MMO server and client (nothing complex, just to learn node.js and HTML5). Basic functionality is complete except data persistence. So far, all data has been kept in memory, but now I would like to add persistent storage with mongodb (or something similar). My question is about how to realize the communication between the node.js application and the data base.
The main activity of the server can be described as follows:
receive incoming message (via WebSocket)
-->
read data from DB
computation on data from step 1
read more data from DB (data required depends on outcome of step 2)
computation on data from steps 1 and 3
write data to DB (create or update)
-->
send response message (via WebSocket)
(steps 3-5 do not always occur)
I guess this is a very common use case.
Questions:
Since the data read in steps 1 and 3 must be consistent AND the write in step 5 might no longer be valid if data in DB has changed between steps 3 and 5, it seems to me that I cannot use async calls to the DB at all (because then data in the DB might be changed by other code in between the above steps). Is that correct?
When thinking of a deployment with multiple instances of node.js working on the same DB (I think this is what nodejitsu calls "drones", right?) then I even have to use data base transactions spanning steps 1 through 5. Is that correct?
It seems to me that using synchronous calls to the DB and having transactions in all these cases would be poor design and introduce performance issues. Is there a better way to do this?
Any hints would be greatly appreaciated! Thanks so much in advance!!
Question 1:
node.js is asynchronous. That is a fact. If you are not able to design your app under this pattern, you cannot use node.js
Of course, there are techniques and patterns, that allow you to do what you want within an asynchronous environment. Here are some keywords, functions, and links that might lead you to the right direction:
Queueing
Optimistic concurrency control
mongoDB/findAndModify
redis/transactions
Question 2:
It might be nice to use transactions, but you cannot, because mongoDB does not have any. You have to program them yourself nodejs-side. That`s one of the mongoDB fundamentals: All power to the programmer (and not the database). Weal and woe!