Trying to count how many child nodes a particular node contains on my database
I am planning on having many users, and want to the expierence to be as fast as possible, so I know I don't want to download the parent node and count
I've thought of simply having a counter field stored, and everytime a user does something to add to that parent also incriment that value.. however I am pretty inexpierenced with this and am worried
that somehow two users adding something at the same time or so will cause that value to be incorrect..which from my reading is whata transaction operation is create for
I remeber when I used to use Parse a while ago, there was something called CloudCode that would constantly run on the server and in particular I would use it for maintence operations on the database
Would running a transaction operation be the solution here? Currious to hear how others handle stuff like this.. do they have some sort of monitering server maintaining it
Related
I have a very large table with 100,000's of rows with a key and a timestamp.
I currently use batch of AWS servers that are using NodeJS and PG to query for the oldest last_updated timestamp, perform the function, and then update the row with the NOW().
My issue is I am trying to find a way scale with more processes and optimize my row checkouts. I started with each process selecting 1000 at a time and each one having a multiple of 1000 for OFFSET, doing the operation, and then updating
I started to read about SELECT using FOR UPDATE and SKIP LOCK but seems like this could have some performance impacts? Also can't find a clear way to do the SELECT/UPDATE in the same query or do I keep doing single updates at a time similar to this post but seems like this may not be good for larger operations like this?
implementing an UPDATE on SELECT in Postgres
Has anyone approached this type of setup? I also have been debating do I need to build my own middleware that manages a pool of work items and then the workers use that table to select/delete?
Does anyone know how to implement e.g. post counter to mongo db? I think I would do
accept /post with data
get mongo collection.count
add this custom id as {id: collection.count + 1}
but now I don't really know what will happen if 2 /posts will come at the same time. It will be queued in db? Or it will has 2 same fake id?
You can set the id field as unique to handle the case where 2 posts come at the same time...but, you will lose one document, since MongoDB won't allow another object for the same id.
You will have to write another piece of code to handle that case, which will be much more complex than a simple create operation.
Ideally, you shouldn't use a separate field in this way, unless it is required by your application and there is no alternative.
Here are a few caveats of using this approach:
For each post received, there are 2 DB operations being performed.
To prevent the loss of documents when the id field is set to unique, you will have to add another block of code, which might have to make a 3rd DB call to finally be stored in the database
Bottom line: Always use _id unless you have a reason not to do so.
I am currently creating a quiz app in meteor. One of the key variables Im using is called currentQuestions which keeps track of the questionnumber of each quiz the user has taken. It is currently an array which updates (in a Meteor session) everytime the user clicks the next or back button on a particular quiz. for example:
currentQuestion = Session.get('currentQuestion') || [0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]
This works fine for a small amount of quizzes. What would you suggest however when I want to have a large amount of quizzes? Currently when I create more quizzes than I have previously defined in the currenQuestion Array the app crashes.
Possible solution I have thought of
(Ugly solutions) Create a for loop that fills up the currentQuestion array with a large amount of zero's at startup.
Insert a object into the mongoDB database that contains the currentquestion with the id of the quiz that's taken (so getting rid of the array entirely)
Hope you can help.
Option 2 is definitely the right one:
If you don't need to keep track of scores between sessions, you can just use a client-side-only database and store objects as you suggest and it will be only a couple of lines of new code for a much more flexible outcome.
This would then be very easy to convert into a persistent store as well: either add sign-in with Meteor's accounts package, add the current userId to each quiz object, and then only publish the quiz objects with the same userId as the logged-in user. Alternatively, you could even use the amplify package to store results between sessions in a given browser if you wanted to avoid user login entirely (you'd have to pull existing results out again on startup, but this is very easy).
The result would have all the reactivity of a Session variable built in, and would avoid the need to populate an array with an arbitrary and changing number of zeros - just check whether the a document with the given quiz Id is in the collection or not, and if not then create it.
You'll need to get rid of the array entirely. If you are going to keep track between sessions then you'll need to go with something like mongoDB. You could created something like a progress or completion model which can hold info about the quizzes they have taken and progress for each. Once you have that stores those by some userId related key.
Your dirty solution is not the worst since you'll know at start how many quizzes you'll have and can allocate the array then.
I'm interested how does google docs store documents on server side because I need to create similar application.
Does it use pure RTF/ODF files or own database?
How do they make possible versioning and undo/redo feature?
If anybody have knowing according this question please share with me.
To answer you question specifically to how Google Docs works. They use a technology called
Operational Transformation
You may be able to use one of operational transformation engines listed on: https://en.wikipedia.org/wiki/Operational_transform#OT_software
The basic idea is that every operation has a context, e.g. "delete the fourth word in the fifth paragraph" or "add an input box after the button". The clients all send each other operations thru the server. The clients and server each keep their own version of the document and apply operations as they come.
When operations have overlapping contexts, there are a bunch of rules that kick in to resolve conflicts. Like you can't modify something that's been deleted, so the delete must come last in a sequence of concurrent operations on that context.
It's possible that the various clients and server will get out of sync, so you need a secondary algorithm to maintain consistency. One way would be to reload the data from the server whenever a conflict is detected.
--This is an answer I got from a professor when I asked the same thing a couple of years ago.
You should use a database. Perhaps a table storing each document revision. First, find a way to determine whether an update is significant or not. You can store minor changes client side for redo/undo, and then, either periodically or per some condition (e.g., user hits save), create a database entry per revision (you can store things like bytes changed, bytes added, bytes deleted, etc.).
Take a look at MediaWiki, which is open source, and essentially does what you're asking (i.e., take a look at their tables and code).
RTF/ODF would typically be generated, and served, when a user requests exporting the document.
Possibly, you should consider utilizing Google Drive's public API. See link for details.
I have a grid(employee grid) which has say 1000-2000 rows.
I display employee name and department in the grid.
When I get data for the grid, I get other detail for the employee too(Date of Birth, location,role,etc)
So the user has option to edit the employee details. when he clicks edit, I need to display other employee details in the pop up. since I have stored all the data in JavaScript, I search for the particular id and display all the details. so the code will be like
function getUserDetails(employeeId){
//i store all the employeedetails in a variable employeeInformation while getting //data for the grid.
for(var i=0;i<employeeInformation.length;i++){
if(employeeInformation[i].employeeID==employeeId){
//display employee details.
}
}
}
the second solution will be like pass employeeid to the database and get all the information for the employee. The code will be like
function getUserDetails(employeeId){
//make an ajax call to the controller which will call a procedure in the database
// to get the employee details
//then display employee details
}
So, which solution do you think will be optimal when I am handling 1000-2000 records.
I don't want to make the JavaScript heavy by storing a lot of data in the page.
UPDATED:
so one of my friend came up with a simple solution.
I am storing 4 columns for 500 rows(average). So I don't think there should not be rapid slowness in the webpage.
while loading the rows to the grid, under edit link, I give the data-rowId as an attribute so that it will be easy to retrieve the data.
say I store all the employee information in a variable called employeeInfo.
when someone clicks the edit link.. $(this).attr('data-rowId') will give the rowId and employeeInfo[$(this).attr('data-rowId')] should give all the information about the employee.
instead of storing the employeeid and looping over the employee table to find the matching employeeid, the rowid should do the trick. this is very simple. but did not strike me.
I would suggest you make an AJAX call to the controller. Because of two main reasons
It is not advisable to handle Database actiity in javascript due to security issues.
Javascript runs on client side machine it should have the least load and computation.
Javascript should be as light as possible. So i suggest you do it in the database itself.
Don't count on JavaScript performance, because it is heavily depend on computer that is running on. I suggest you to store and search on server-side rather than loading heavy payload of data in Browser which is quite restricted to resources of end-user.
Running long loops in JavaScript can lead to an unresponsive and irritating UI. Use Ajax calls to get needed data as a good practice.
Are you using HTML5? Will your users typically have relatively fast multicore computers? If so, a web-worker (http://www.w3schools.com/html/html5_webworkers.asp) might be a way to offload the search to the client while maintaining UI responsiveness.
Note, I've never used a Worker, so this advice may be way off base, but they certainly look interesting for something like this.
In terms of separation of concerns, and recommended best approach, you should be handling that domain-level data retrieval on your server, and relying on the client-side for processing and displaying only the records with which it is concerned.
By populating your client with several thousand records for it to then parse, sort, search, etc., you not only take a huge performance hit and diminish user experience, but you also create many potential security risks. Obviously this also depends on the nature of the data in the application, but for something such as employee records, you probably don't want to be storing that on the client-side. Anyone using the application will then have access to all of that.
The more pragmatic approach to this problem is to have your controller populate the client with only the specific data which pertains to it, eliminating the need for searching through many records. You can also retrieve a single object by making an ajax query to your server to retrieve the data. This has the dual benefit of guaranteeing that you're displaying the current state of the DB, as well as being far more optimized than anything you could ever hope to write in JS.