Real Time charts: what are the techniques ,dos and donts - javascript

am working on a school project. Final year BscIT. I used spring framework, jstl, spring-data-mongo and mongodb for storage. The project has an Restful API to which pharmacies and hospitals send some data to.namely their diagnostics and the number and type of medicines purchased.
Here I am trying to show the trends in a dashboard as and when anytime occurs in the database.I use jquery as javascript framework and after studying kendo dataviz, amchart and highchart , I will be going with amchart
Now It really doesn't matter with visualization tools I use. I am more interested in how to do it and prevent the browser from freezing.
Below is what am thinking:
option 1: I will be querying the database every 5 seconds.Does that means the javascript should know the "previous" 5 seconds ? As in I will pass the previous 5 seconds as start date and now as end data?
option 2 I will pull all data of the day every 5 seconds.
Am really not sure about how it is done in real life so, anyone feel free to show me the path.
Thanks in advance

Related

what should be the initial dataset from millions of data points for stock line highcharts

I want to prepare a line stock highchart like this example : https://www.highcharts.com/demo/stock/lazy-loading
In the given example, when you load the chart for the first time, it calls https://demo-live-data.highcharts.com/aapl-historical.json and fetches some points, to be precise 0-165 records (if you check the network tab and ajax call). At a same time All option is selected in the time range tool.
If you drill down further or go for any specific time range, it will bring more data always from the server.
Question: If you have millions of data points, consider from 2000 to 2022 years, then for All option, what are you going to display. what should be the initial data set or result or filter ?
NOTE: I will have millions of data points from 2000 to 2022 years going forward. When I load the chart for the first time, out of these millions points, what should come back from the server ?
Just for your reference, you can check example of time series data that I'm going to have in mock-data=>i.js folder/file which is NOT being used anywhere in below example as of now.
Highcharts 1.7 million points example : https://stackblitz.com/edit/js-wng4y6?file=index.js
P.S. : I'm new to stockhighcharts and I don't seem to find any proper explanation anywhere. Trying to reach out to the community for further help.
Server-side data grouping should be done based on the range for with you are trying to group the data, so All means nothing - however, in your case this will be 2 years.
For the data grouping you might also consider the chart size (this is done by default for a dataGrouping feature running client-side in Highcharts Stock). When relevant information is passed to the server it should return a set of grouped data points.
About grouping logic you can find more in the api options where is present method of approximation inside a group.
https://api.highcharts.com/highstock/series.area.dataGrouping.approximation
Sending this much data to highcharts to be processed is asking for issues. I highly recommend making a local highcharts server (something they support) and have this done within your system. See it here
This is very important when it comes to security as well (if your data is sensitive), having it race across the internets to highcharts and then sent back to you leaves it open to the world.
From here, you can also specify the start and end time of each render, and have that change based on user input. Personally I would generally display the last 5 days or something, and then if someone wanted to, they could pull the slider all the way back for the last significant amount of time.
But, to answer your question, when you send a data object to highcharts, either local server or the highcharts server, you will get a base64 image back that you can directly imbed in your UI.

architecture for get and store api request data

This is more of a architectural questions. An external platform had product and price information for let's say, books. There is an API available to get this information.
What I read is that it should be possible to create a function in Javascript and connect the Javascript to a page where you want to show the data on my own website. This would mean that for each page request an API-call is made. Since the requested information only changes once a day maximum this does not sound the most efficient solution.
Can someone advise a better solution? Something into the direction of a similar php or javascript function that does the request on the background, schedule an update and import the data into mysql? If so, what language would be most common.
I need the solution for a Joomla/php/mysql environment
Here's a simple idea - fetch and store results from the API (ones you think aren't gonna change in a day), either on disk, or in the database, and later use these stored results to retrieve what you otherwise would've fetched from the API.
Since storing anything in frontend JS across page reloads isn't easy, you need to make use of PHP for that. Based on what's given, you seem to have two ways of calling the API:
via the frontend JS (no-go)
via your PHP backend (good-to-go)
Now, you need to make sure your results are synced every (say) 24 hours.
Add a snippet to your PHP code that contains a variable $lastUpdated (or something similar), and assign it the "static" value of the current time (NOT using time()). Now, add a couple of statements to update the stored results if the current time is at least 24 hours greater than $lastUpdated, followed by updating $lastUpdated to current time.
This should give you what you need with one API call per day.
PS: I'm not an expert in PHP, but you can surely figure out the datetime stuff.
It sounds like you need a cache, and you're not the first person to run into that problem - so you probably don't need to reinvent the wheel and build your own.
Look into something like Redis. There's an article on it available here as well: https://www.compose.com/articles/api-caching-with-redis-and-nodejs/

Implementing highcharts for Coin currency market

I have coin currency market application and I would like to integrate a graph in it, however I don't know from where data should be taken. Is there some API that I can integrate into my highcharts in order to get graphical representation of values of the coin in the past 7,14,30 days? I have scrapped all data from http://coinmarketcap.com/ but I can't scrap graph because it's canvas. Can you help me with that? Where to search for coin data in the past?
The coinmarketcap site uses json data for the charts. If you look at the network activity in the console, you can see requests for items like:
http://coinmarketcap.com/static/generated_pages/currencies/datapoints/bitcoin-main.json
This contains the json data for the bitcoin chart. However, there does not seem to be any way to specify different time periods.
A quick look around produced this site for bitcoin info (https://bitcoinaverage.com/api). There are lots of links which may help here: able to get prices from here: http://bitcoincharts.com/charts
You can also check yahoo fincancem which allows to get data via API.

Where to put/generate large javascript data files in front end with cakephp

I have an app that uses a large database to fill in google maps and charts data. This ends up being about 5000 lines per column and about 20 columns. The issue I am running into is whether to put this data in the view template, which makes my source code several thousand lines long, or generating a javascript file for each instance and including them in the view. The issue I am running into with that method is that I am generating files with no way to delete them out of the webroot folder (without a cron job to go through and delete old ones). I was wondering what the solution is for this.
of course, but you as a developer are responsable for fast delivering websites. you cannot fetch all of the data. for example when using google maps it is common practice that you display a limited number of data according to the displayed area ( rectangle ). When using charts, then you should answer only with the already aggregated data.
there is almost never the reason to display all the data to the user. Ask yourself, what will you do when you see 5000 thousands lines of code on the website. do you need it all at once? No.
Use AJAX to fetch only the rows you need right now.

AngularJS - Large sets of data

I've been pondering moving our current admin system over to a JS framework for a while and I tested out AngularJS today. I really like how powerful it is. I created a demo application (source: https://github.com/andyhmltn/portfolio-viewer) that has a list of 'items' and displays them in a paginated list that you can order/search in realtime.
The problem that I'm having is figuring out how I would replicate this kind of behaviour with a larger data set. Ideally, I want to have a table of items that's sortable/searchable and paginated that's all in realtime.
The part that concerns me is that this table will have 10,000+ records at least. At current, that's no problem as it's a PHP file that limits the query to the current page and appends any search options to the end. The demo above only has about 15-20 records in. I'm wondering how hard it would be to do the same thing with such a large amount of records without pulling all of them into one JSON request at once as it'll be incredibly slow.
Does anyone have any ideas?
I'm used to handle large datasets in JavaScript, and I would suggest you to :
use pagination (either server-sided or client-sided, depending on the actual volume of your data, see below)
use Crossfilter.js to group your records and adopt a several-levels architecture in your GUI (records per month, double click, records per day for the clicked month, etc.)
An indicator I often use is the following :
rowsAmount x columnsAmount x dataManipulationsPerRow
Also, consider the fact that handling large datasets and displaying them are two very differents things.
Indeed pulling so many rows in one request would be a killer. Fortunately Angular has the ng-grid component that can do server-side paging (among many other things). Instructions are provided in the given link.

Categories