This question already has answers here:
Closed 10 years ago.
Possible Duplicate:
Sorting on the server or on the client?
As a part of a project for my university, I have to make a website like a forum let's say.
So there will be posts, many of them. Every Post has a like bar, comments, some text, some buttons etc. Also the user will be able to sort these posts that appear in a page, based on some criteria like date,name, popularity etc.
My question is how should I implement the sorting: 1. with javascript on the browser's side, or 2. with a form or something, and a new request to the server (in this way the server has to send me back the posts sorted) ?
Thank you in advance.
There are pro's and con's to both.
Generally speaking, if you already have all of the data available in the client anyhow, you will provide a more responsive user experience sorting on the client.
If you have to fetch extra records that you would otherwise not fetch to sort client-side, there's a great chance that you are bloating the download to the client beyond the optimal point, and a sort on the server-side via Ajax would be better.
That's a huge depends. Is there paging involved? What's the max size of the data set? It's only the records in the single page on client screen need to be sorted?
Server side sorting is better for:
Large data set
Faster initial page load
Accessibility for those not running JavaScript
Complex view business logic
Resilience to concurrent changes
Client side sorting is better for:
Small data set
Faster subsequent page loads
The sort criteria are user-selectable or numerous.
Once you have this feature, you can add filters, and pagination easily
Related question:
Sorting on the server or on the client?
Related answer:
The important thing to remember is that while balancing the load between powerful clients and the server may be a good idea in theory, only the server can maintain an index which is updated on every insert. Whatever the client does, it's starting with a non-indexed unsorted set of data.
if it is possible/practical to sort elements on the client side, that would bet the best solution (reduce server requests). however this is often not the case.
It happened to us. We sort the data in the client side. Then a new requirement arrived. We need to put the sorted data in a report. So, instead of translating the sorted data directly into report datasource (which will be achieved through server side sorting), we are then required to catch the sort details (table to sort, column to sort, sort order) from the client side activity (on sort column event) then send it to the server when print report button is pressed then do the sorting on the serverside. Lot of work.
Related
Update: I am getting the impression that this is not even the right website to post this. If someone can point me in the right direction, I'd be appreciative...
I have an existing PHP+MySQL application that wasn't built to render "real-time" or similarly live-style data. But now I need to build in a way to pull nearly real-time data into the application and keep the data on the page fresh. This live data is only for 1 page in the application.
Looked at things like socket.io and PHP-based websockets libraries, but it seemed like overkill because the data is basically coming from 1 source and being delivered to 1 person (the client). Multiple other users could have this process running, but each one would bring their own data endpoint. That's... like a year down the road. But good to think about. Would ideally have hundreds, or thousands of users on the system, pulling their live-ish data. So I want this to be as streamlined and low-impact as possible.
Users must be authenticated and authorized to consume the data. This is already baked into the current system.
The API to get the data (which has already been built by another vendor) is also NOT streaming. It's set on a 20-second cron, so the new data is available every 20 seconds, which satisfies the client's needs.
My current plan is to do something like this...
Data is pulled on a cron every 20 seconds, organized, and stored into the database (complete)
Adjust #1 so it also does any additional proprietary calculations on data AND compiles + writes a JSON file on the server (unique to the user) which is the exact data needed for the front end (DB data is needed for other pages)
Create small PHP-based service which validates a client-provided JWT and reads the JSON file out
Write AJAX front end to poll endpoint from #3 every X seconds using a JWT for authorization
This all seems sort of like I might be reinventing the wheel, or missing something. The fact that this is an existing PHP based application (LAMP) does have some limiting factors, but I feel like there's got to be a more efficient way to handle this... It's pretty new to me. Also, I'm open to other technologies that'll run on the LAMP stack, if it'll make things better.
I would say go for the API solution in the beginning :) Since it fits the architecture more and is for sure the least amount of work. Also if there will be problem with the "live" feeling of the data you can fix it by polling more often or introducing long polling, assuming you change the cron job time.
I mean in the end it is all about impact for the time spent, don't start implement features that customers don't care about :)
The biggest problem to solve is to implement it in a way that fits your requirements and is somewhat future extendable. You still have to deal with issues like resolution, time outs, reducing server processing when requesting data and so on!
For me, if you need to maintain a global service state because a single client(s) request could affect all other connected client request(s) then most all server-side scripting languages are not the best choice! Also to further add, if you plan on implementing something like this with PHP, you will be setting your self up for a living nightmare! Why, because simply put, PHP(s) socket(s) implementation is that bad!
Currently I am experiencing speed issues with parts of my application that load large amounts of data into a reporting table. The data in the reporting table is pulling from multiple tables and running some complex queries, but required quires.
Besides optimizing the code, my question is how do you personally handle large amounts of data that need to be displayed to the user, and what is the best practice?
Currently I am processing all the data before hand and then generating a table via the data table javascript library.
Things I know:
The user doesn't need to see all the data at once
The user needs to be able to search through all the data
The user needs to be able to filter through the data
Is the best way really to just use a loading spinner, and load only a small portion of the data when the page first loads? then the rest of the data retrieval is through Ajax?
I feel like there has to be a better way
Thanks,
I think you're answering your own question a bit. Yes, it is better to not deliver the whole database to the user at once, this is why any RDBMS supports things like LIMIT. Your three criteria exactly match what a database system can do for you – queries of small subsets of data (i.e. pages), optionally filtered or matched based on a search query.
For simplicity of the front end, you can make the first page load via AJAX as well, though having it pre-rendered does make the page feel more responsive. Having said that, there are many existing solutions to this problem; some template engines and JS front-end frameworks (Vue.js SSR) support server-side pre-render.
I've read a few StackOverflow posts related to this subject but I can't find anything specifically helps me in my scenario.
We have multiple monitoring instances within our network, monitoring different environments (Nagios, Icinga, more...). Currently I have a poller script written in PHP which runs every minute via cron, it asks the instance to return all of its problems in JSON, the script then interprets this and pushes it in to a MySQL database.
There is then an 'overview' page which simply reads the database and does some formatting. There's a bit of AJAX involved, every X seconds (currently use 30) it checks for changes (PHP script call) and if there are changes it requests them via AJAX and updates the page.
There's a few other little bits too (click a problem, another AJAX request goes off to fetch problem details to display in a modal etc).
I've always been a PHP/MySQL dev, so the above methodology seemed logical to me and was quick/easy to write, and it works 'ok'. However, the problems are: database constantly being polled by many users, mesh of javascript on the front end doing half the logic and PHP on the back doing the other half.
Would this use case benefit from switching to NodeJS? I've done a bit of Node.JS before but nothing like this. Can I subscribe to MySQL updates? Or trigger them when a 'data fetcher' pushes data in to the database? I've always been a bit confused as I use PHP to create data and javascript to 'draw' the page, is there still a split of NodeJS doing logic and front end javascript creating all the elements, or does NodeJS do all of this now? Sorry for the lack of knowledge in this area...
This is definitely an area where Node could offer improvements.
The short version: with websockets in the front-end and regular sockets or an API on the back-end you can eliminate the polling for new data across the board.
The long version:
Front-end:
You can remove all need for polling scripts by implementing websockets. That way, as soon as new data arrives on the server, you can broadcast it to all connected clients. I would advise Socket.io or the Primus websocket wrapper. Both are very easy to implement and incredibly powerful for what you want to achieve.
All data processing logic should happen on the server. The data is then sent to the client and should be rendered on the existing page, and that is basically the only logic the client should contain. There are some frameworks that do all of this for you (e.g. Sails) but I don't have experience with any of those frameworks, since they require you to write your entire app according to their rules, which I personally don't like (but I know a lot of developers do).
If you want to render the data in the client without a huge framework, I highly recommend the lightweight but incredibly useful Transparency rendering library. Using this, you can format a Javascript object on the server using Node, JSONify it, send it to the client, and then all the client would have to do is de-JSONify it and call Transparency's .render.
Back-end:
This one depends on how much control you have over the behaviour of the instances you need to check. I assume you have some control, since you can get all their data in a nice JSON format. So, there are multiple options.
You can keep polling every so often. This is the easiest solution since it requires no change to the external services. The Javascript setInterval function is very useful here. Depending on how you connect with the instances, you might be able to use a module like Request to do the actual request, so that takes out a bunch more of the heavy lifting.
The benefit of implementing the polling in your Node app as well, is that you will receive the data in your Node app and that way you can immediately broadcast it to the clients, even before inserting it into a database. This will greatly reduce the number of queries on your database.
An alternative to polling would be to set up a simple Express-based API where the applications can post their 'problems', as you call them. This way your application will get notified the moment a problem occurs, and combined with the websockets connection to the client this would result in practically real-time updates.
To be more redundant, you would have a polling timer alongside the API, so that you can check the instances in case there's something wrong that causes them to not send over any more data.
An alternative to the more high-level API would be to just use direct socket communication, which is basically the same approach only using a different set of functions.
Lastly, you could also keep the PHP-based polling script. This would be the most efficient solution since you wouldn't go and replace everything. Then from the Node app that's connected to the clients with websockets, you could set an interval to query the database every so often and broadcast the updates. This will still greatly reduce the number of queries, since no matter how many clients are connected there will only be one query, the response of which then gets sent to all connected clients.
I hope my post has give you some ideas of how you could implement your application using Node. Keep in mind though that I am just one developer, this is how I would approach building your application in Node. There will definitely be others who have different opinions.
In general is it better for performance to do lots of data calculations on the server side or on the javascript side?
I have a bunch of data that i'm displaying on a page - and I'm wondering if I should format/ parse/ make calculations on that data on the server side (in python) and return a template or if I should return the data as is and do all my calculating/ formatting on the javascript side?
Are there any general rules of thumb when making these decisions?
Examples of things i'm calculating - converting timestamps to dates.
This depends a lot on what you are trying to do. If the chart is dynamic and animated, doing it client side with js may be the only choice. It also depends on how much data you have. I would not recommend doing it in js if you have over 10mb of raw data.
First of all you should consider change you data format for storing a data for displaying. It should be stored already in the convenient way for displaying it - that does not requires any processing at all.
If you still need some recalculation of data consider doing it server-side as this gives you possibility of caching this result (for example in mamceched) as this make no sense to calculate them on every page refresh.
The worst option is recalculating them client side, as this can bring hard to detect noticeable performance issues on client browser (eg. on larger data) and you could be unable to see this kind of problem in your testing environment (you could see that your server is slowing down, but it is very hard to see that some clients renders your page very very slow - unless they write to you).
In addition to the facts stated by thedk, you should also keep in mind that calculations you do on client side are more likely to fail because the client may not fulfill certain preconditions. Think of disabled JavaScript or an unreliable internet connection. You generally have no control over your data as soon as it has left the server.
So, it would be highly advisable to move only unimportant calculations to the client side. Something like datetime formation might be okay, but don't try to parse your whole website with JavaScript. Your website should work (and look acceptable) even if JavaScript is disabled on the client.
I came across a site that does something very similar to Google Suggest. When you type in 2 characters in the search box (e.g. "ca" if you are searching for "canon" products), it makes 4 Ajax requests. Each request seems to get done in less than 125ms. I've casually observed Google Suggest taking 500ms or longer.
In either case, both sites are fast. What are the general concepts/strategies that should be followed in order to get super-fast requests/responses? Thanks.
EDIT 1: by the way, I plan to implement an autocomplete feature for an e-commerce site search where it 1.) provides search suggestion based on what is being typed and 2.) a list of potential products matches based on what has been typed so far. I'm trying for something similar to SLI Systems search (see http://www.bedbathstore.com/ for example).
This is a bit of a "how long is a piece of string" question and so I'm making this a community wiki answer — everyone feel free to jump in on it.
I'd say it's a matter of ensuring that:
The server / server farm / cloud you're querying is sized correctly according to the load you're throwing at it and/or can resize itself according to that load
The server /server farm / cloud is attached to a good quick network backbone
The data structures you're querying server-side (database tables or what-have-you) are tuned to respond to those precise requests as quickly as possible
You're not making unnecessary requests (HTTP requests can be expensive to set up; you want to avoid firing off four of them when one will do); you probably also want to throw in a bit of hysteresis management (delaying the request while people are typing, only sending it a couple of seconds after they stop, and resetting that timeout if they start again)
You're sending as little information across the wire as can reasonably be used to do the job
Your servers are configured to re-use connections (HTTP 1.1) rather than re-establishing them (this will be the default in most cases)
You're using the right kind of server; if a server has a large number of keep-alive requests, it needs to be designed to handle that gracefully (NodeJS is designed for this, as an example; Apache isn't, particularly, although it is of course an extremely capable server)
You can cache results for common queries so as to avoid going to the underlying data store unnecessarily
You will need a web server that is able to respond quickly, but that is usually not the problem. You will also need a database server that is fast, and can query very fast which popular search results start with 'ca'. Google doesn't use conventional database for this at all, but use large clusters of servers, a Cassandra-like database, and a most of that data is kept in memory as well for quicker access.
I'm not sure if you will need this, because you can probably get pretty good results using only a single server running PHP and MySQL, but you'll have to make some good choices about the way you store and retrieve the information. You won't get these fast results if you run a query like this:
select
q.search
from
previousqueries q
where
q.search LIKE 'ca%'
group by
q.search
order by
count(*) DESC
limit 1
This will probably work as long as fewer than 20 people have used your search, but will likely fail on you before you reach a 100.000.
This link explains how they made instant previews fast. The whole site highscalability.com is very informative.
Furthermore, you should store everything in memory and should avoid retrieving data from the disc (slow!). Redis for example is lightning fast!
You could start by doing a fast search engine for your products. Check out Lucene for full text searching. It is available for PHP, Java and .NET amongst other.