I'm using D3.js to plot a grouped bar chart where data for each series (bars at position N in each group) comes from a separate server request (via d3.json()). In the callback for each server call, I process the data and set x/y axis domains to newly calculated maxes. However, somewhere in here I have a race condition that causes the bars not always get rendered against the correct maxes and it can take a few page reloads to get the chart show up correctly.
I don't really understand how javascript, d3, and asynchronous callbacks work together and how parallel processing really works (I'm using Chrome for dev). Any pointers to relevant literature are appreciated.
Should I abandon one request per series strategy and combine the data on the server side? I was hoping to make this more dynamic so that in the future I could add/remove series on the fly.
Related
I want to prepare a line stock highchart like this example : https://www.highcharts.com/demo/stock/lazy-loading
In the given example, when you load the chart for the first time, it calls https://demo-live-data.highcharts.com/aapl-historical.json and fetches some points, to be precise 0-165 records (if you check the network tab and ajax call). At a same time All option is selected in the time range tool.
If you drill down further or go for any specific time range, it will bring more data always from the server.
Question: If you have millions of data points, consider from 2000 to 2022 years, then for All option, what are you going to display. what should be the initial data set or result or filter ?
NOTE: I will have millions of data points from 2000 to 2022 years going forward. When I load the chart for the first time, out of these millions points, what should come back from the server ?
Just for your reference, you can check example of time series data that I'm going to have in mock-data=>i.js folder/file which is NOT being used anywhere in below example as of now.
Highcharts 1.7 million points example : https://stackblitz.com/edit/js-wng4y6?file=index.js
P.S. : I'm new to stockhighcharts and I don't seem to find any proper explanation anywhere. Trying to reach out to the community for further help.
Server-side data grouping should be done based on the range for with you are trying to group the data, so All means nothing - however, in your case this will be 2 years.
For the data grouping you might also consider the chart size (this is done by default for a dataGrouping feature running client-side in Highcharts Stock). When relevant information is passed to the server it should return a set of grouped data points.
About grouping logic you can find more in the api options where is present method of approximation inside a group.
https://api.highcharts.com/highstock/series.area.dataGrouping.approximation
Sending this much data to highcharts to be processed is asking for issues. I highly recommend making a local highcharts server (something they support) and have this done within your system. See it here
This is very important when it comes to security as well (if your data is sensitive), having it race across the internets to highcharts and then sent back to you leaves it open to the world.
From here, you can also specify the start and end time of each render, and have that change based on user input. Personally I would generally display the last 5 days or something, and then if someone wanted to, they could pull the slider all the way back for the last significant amount of time.
But, to answer your question, when you send a data object to highcharts, either local server or the highcharts server, you will get a base64 image back that you can directly imbed in your UI.
I'm building a visualization for some industrial devices that produce large amounts of time-based data like temperatures, currents or voltages. All data is constantly written to a SQL Server database (can't control that part).
The HTML5 frontend consists of an interactive zoomable chart I made with d3.js. Data series can be added(loaded) to the chart on demand, in which case the frontend sends an ajax request, ASP.NET MVC and EF6 fetches the values from the DB and returns them as Json.
Each data element simply consists of a DateTime and a value. Please note that the values do not get written in regular intervals (like every 2 seconds or so), but in irregular intervals. This is because the device doesn't get polled regularly but sends data on specific events, like a raise/drop of a temperature by a given change of 0.1 °C, for example.
So far everything works really well and smooth, but the large amount of data becomes a problem. For example, when I want to show a line chart for a selected period of lets say 3 month, each data series already consists of appr. 500.000 values, so the Json response from the server also gets bigger and bigger and the request takes longer with growing time periods.
So I am looking for a way to reduce the amount of data without losing relevant information, such as peaks in temperature curves etc., but at the same time I want to smoothen out the noise in the signal.
Here's an example, please keep in mind that this is just a chosen period of some hours or days, usually the user would like to see data for several months or even years as well:
The green lines are temperatures, the red bars are representations of digital states (in this case a heater that makes one of the temp curves go up).
You can clearly see the noise in the signals, this is what I want to get rid of. At the same time, I want to keep characteristic features like the ones after the heater turns on and the temperature strongly rises and falls.
I already tried chopping the raw data into blocks of a given length and then aggregating the data in them, so that I have a min, max and average for that interval. This works, but by doing so I the characteristic features of the curve get lost and everything gets kind of flattened or averaged. Here's a picture of the same period as above zoomed out a bit, so that the aggregating kicks in:
The average of the upper series is shown as the green line, the extent (min/max) of each chop is represented by the green area around the average line.
Is there some kind of fancy algorithm that I can use to filter/smoothen/reduce my data right when it comes out of the DB and before it gets send to the frontend? What are the buzzwords here that I need to dig after? Any specific libraries, frameworks or techniques are highly appreciated, as well as general comments on this topic. I'm interested primarily in server-side solutions, but please feel free to mention client-side Javascript solutions as well as they might surely be of interest for other people facing the same problem.
"Is there some kind of fancy algorithm that I can use to filter/smoothen/reduce my data right when it comes out of the DB and before it gets send to the frontend? What are the buzzwords here that I need to dig after?"
I've asked a friend at the University where I work and she says Fourier Transforms can probably be used... but that looks like Dutch to me :)
Edit: looking at it a bit more myself, and because your data is time sampled, I'm guessing you'll be interested in Discrete Time Fourier Transforms
Further searching around this topic led me here - and that, to my (admittedly unexpert) eyes, looks like something that's useful...
Further Edit:
So, that link makes me think that you should be able to remove (for example) every second sample on the server-side: then on the client-side, you can use the interpolation technique described in that link (using the inverse fourier transform) to effectively "restore" the missing points on the client-side: you've transferred half of the points and yet the resulting graph will be exactly the same because on the client you've interpolated the missing samples.... or is that way off base? :)
I've got performance issue/question. I'm rendering about 80 area charts in single page using angular and highcharts-ng. When data is loaded and angular is binding it to charts my browser does not respond for about 2 seconds. It is maybe not great amount, but still...
What is the reason? It is angular-to-chart binding issue or just chart rendering by highcharts?
There is possibility to make it a little faster or just make it not hanging browser ?
EDIT:
Response comes from server really fast. Data size is quite small. When I turn off charts (ng-if="false"), rest data loads really fast without any performance issue.
Each area chart has max 12 datapoints.
Old post, but in case anyone else encounters something similar, watch out for what you put inside data objects in highcharts-ng. Everything gets $watched. In my case I put the arrays of objects into the chart data so I could then view details when a graph segment was clicked. However this caused horrendous performance issues as the digest cycle became huge. I solved it by putting a function that returned the array into the chart data instead of the array itself.
At API/Server end
I am also having angular app which had similar issues about performance(one diff is: I am using highcharts but without directive-highchartsNG). At angular and server call's end , Sometimes when data is too much it takes times to download data on a page. however your api response comes fast but actually the request is stalled and takes time to load data on page and provide it to highchart/or any other dom label markup things.
My response data had many fields timestamp ,value,average,etc etc. I optimized the query response to return me required fields only to make it faster.
To make chart rendering faster :
Recently highcharts relased boost.js module which enhance the performance of various highcharts charts. it is a module to allow quick loading of hundred thousands of data points in Highcharts.
See here Highcharts Boost.js module official page
I'm using dygraphs to produce a canvas-based stock chart in a web trading platform I'm developing -- source here. Data is delivered to the web client via a WebSocket connection.
If you look at the source, you'll see I'm appending data for the chart to an array, chartData, as data comes in over the socket (line 100), and then I'm passing the data to the dygraphs chart via updateOptions (line 111), causing the chart to redraw itself using the most current data.
This works fine, and it performs well. However, after about an hour, when maybe 10,000 data items are appended to the chart, the page (I'm using Chrome) crashes, probably due to memory usage. Data is stored in both the chart AND the array (chartData), so I imagine that's a good chunk of memory for one web page. Plus, I'm using ExtJS which is a hog :)
Does anyone have suggestions on how I can reduce the memory footprint for the chart?
Besides the obvious, "Don't use Ext", I can offer several guesses, but nothing definitive.
If, as I assume, much of the data being used is not in a currently viewable portion of the chart, then perhaps you can simply remove it. After you have enough data to fill the chart, every time you add n records to the end, splice off n records from the beginning.
If the data is coming in faster than you can comfortably render it (unlikely but possible), swap in and out several images: collect the data into a group. At a certain interval, clone that group into your rendering area, and use that to render a new chart. When rendering is complete, place it in the DOM, discarding the old one.
But simply removing older data might solve many of your problems...
... especially if you get rid of Ext.
I'm using VivaGraph to create a graph on my webpage as per some data. I used a websocket connection to receive the data(its an infinite data stream). I want to update my graph based on the data received but I'm not able to figure out how.
For example:
Suppose my graph currently that I can see on the screen is A----B
The next date received says that now there's a node C that's linked to B and the link from A to B has been deleted.
How do I do this on the same graph efficiently? That is: Now, I can see B----C
Thanks
The graph can be updated in real time after rendering without any problems or using graph.beginUpdate() and graph.endUpdate().
The problem and the errors seem to lie only in cases where an exception is risen, such as adding a node that already exists or removing a link that doesn't. Make sure that there are no exceptions using normal if conditions and the graph will work perfectly in real time.