I have a list of values over time (basically a list of integers).
Currently the NPM website is down so I can't really search for API's properly, therefore I am asking if any of you know an API for Javascript Node which takes an input of a list of integers (representing y coordinates on a graph) and can then make a prediction of how the graph will continue to flow.
Thank you very much for any responses!
It depends on how you want to make a prediction. You can calculate a linear regression, fit a time series model, train a neural network, etc.
The easiest is a linear regression model. It won't be very accurate; just give you a sense of direction as to where the data is going at the current time.
Time series modelling (autoregression) is probably what you are looking for. This usually involves using historical data to fit a model that can be used to predict (referred to as forecast) future values. This involves some fairly heavy lifting with statistics, but JavaScript libraries do exist, like timeseries-analysis.
If you want to create a fully-blown neural network, tensorflow is being developed for JavaScript, but I wouldn't recommend this approach unless you have a fairly good understanding of machine learning. In order to make an effective forecaster with a neural network, you need to be able to properly normalize data and write a recurrent network; not a basic topic by any stretch of the imagination.
Related
I am fairly new to NLP in general. My goal is to create some kind of parser that can easily find files on my various hard drives.
I have no idea how to properly parse the input to transform it into any managable representation that a program can easily use to create a given output.
For example, the following sentences should return a list of documents:
documents created 3 months ago
documents modified 2 weeks ago
photos taken in china (this one would then use the GPS data within the image file)
It can probably be easily done using some kind of Regex pattern (<filetype> <action> <time>) but I would love to make it more flexible.
I looked into compromise, a JS library that has some easy to use API to retrieve specific parts of the input. But I kind of doubt that calling methods like calculatedResult.nouns()[0] and calculatedResult.verbs()[0].stem() should be used to parse the commands as those require a fixed kind of syntax.
Any tips on how to achieve my goal? I am not sure if using ML and training a custom model is the way to go. I never use ML and, based on my low knowledge of it, it seems kind of hard to train it constructs like those (as I would need a LOT of example sentences but there is just a finite amount of realisically used combinations that make sense).
The NLP technique you need to explore is intent detection. You can either integrate a NLP library like RASA or Spacy into your program or work with a commercial API that conducts intent detection. You will need example sentences in both cases but probably not as many as you think. Intent detection is a key part of chatbots so there's quite a lot of tools out there. Low level, hands-ons ML development is not really needed these days with all the high level intent detection tools out there.
For my application, I am making a get request of thousands of data points.
When I use charts js to display the data, it takes a long time to render, and I experience lag. I also noticed that the x-axis labels for each data point don't appear properly, so they had to be omitted
I like the sleek design and ui of the graphs, but cannot get it to work well for my use case. Is charts js not meant to be used with large data sets? Is there another library like charts js that can handle large data sets? While also being free?
if you want to handle big data you should use Highcharts
it easy can handle some million data without a big delay
Another option to consider is ZingChart. It is free as a branded version, but renders large amounts of data quickly while still maintaining flexibility in customization. If you are looking for a sleek design and UI, ZingChart allows the user to change just about every size, shape, and color to match your taste.
Full disclosure, I am on the ZingChart team. However, we developed a speed test tool that I think you will find helpful in testing your number of data points, regardless of which library you end up selecting. Note that some of these libraries will use up all your browser memory, so proceed with caution in some cases.
I had the same problem, Charts js seems to be unable to handle large data sets. The best alternative I've found is https://github.com/danvk/dygraphs . Also you could try http://canvasjs.com/ although it is comercial.
Have a look at LightningChart JS... It is made with WebGL. It can render
1 million data points in ~80 ms in line chart
10 million data points in ~800 ms
that is for static data. Those I got from my PC (AMD Ryzen, NVidia GTX1060)
But for scrolling streaming data, the performance is yet more impressive. Dozens of millions of points, with some configurations with Firefox browser, over 100 million points.
There is a chart performance tester application
I work with the team making this chart...
I am using d3 to make some graphs but the constraint is the number of data. To be more specific, I have an average number of points of 500,000 on a graph.
It could go from 100,000 to 1,000,000 points on the graph.
Whenever there are 'zoom' possibilities or some other trick (that I would be pleased if someone would explain to me), I wonder if d3 (or any other lib) will really handle a 500,000 points graph.
Morever, I suppose it will depend on the client's computer, which mean they could have the latest i7 proc or a old intel pentium. For this last case, how would the browser react? I suppose it's going so slow that it will just crash?
Well, if people can give some feedback/advice please?
(Currently trying to display the data differently but my brain just imploded.)
I do not have numbers or trials but I can say that I have seen d3 go slow after certain data sizes especially for certain types of graphs. 500.000 seems to be quite huge in terms of data points so if you have no way to reduce that number by aggregation (such as representing 10 day data as 1 day) you might be right about worrying.
As in all performance related questions, the best way to know is to test it, so I can advice you to test it and see if it fits your requirements. If it does not you might want to try some of the non-free libraries such as HighCharts. Another free library which satisfies me with its performance is Chart.js (although it only has 6 charts supported).
I don't think that using d3 for representing a massive amounts of data is the right choice due to performance:
Vector graphics javascript library like d3 are quite heavy(for the
client) to run.
Complex visualization logic can easily hang the browser for multiple
seconds.
Large data manipulation using DOM could be slow.
Try something like Graphviz or Gephi that maybe could be more suitable for your requests.
I am looking to build a graphical frontend for some python functionality.
Since I am very proficient in HTML/CSS/JS I decided I might check this out as a possibility.
The main idea is to make it easier to explore data/variables, I am thinking of a datagrid for example.
I was also thinking of creating a GUI for things such as plotting with matplotlib (ie: Choose variable for X axis, choose variable for Y axis, pick a color from a dropdown, ...) in order to cut out some of the tediousness in exploring data with several variables.
I would then probably use something like Node-webkit to make a desktop application out of it if possible (and if not, just keep it as a browser project, perhaps).
Essentially what it would boil down to is kind of creating an IDE for python, specialized for data exploration/plotting and perhaps data manipulation.
An example I found for a web-based python console is repl.it
However I have spent a fair amount of time looking for viable ways to go about this, since sending and receiving data from the python interpreter wouldn't be too hard, but it might get a bit harder for exploring variables or plotting data.
So the question is: Are there any projects or libraries out there to facilitate this kind of functionality, or which provide a good starting point for interacting with Python through JavaScript? Are there any pointers you guys can give me to make the experience for users as seamless as possible?
Thanks!
I have several data "layers" over the same list of "samples". Some layers are continuous float data, some are multinomial/categorical data, and some are binomial/dichotomous/boolean data. In my Javascript web application, I want users to be able to select a set of samples and see which layers are significantly different between the selected set of samples and all other samples. The end result should be a p-value for each layer, from a two-tailed test (where applicable) of the null hypothesis that the distribution over selected samples is the same as the distribution over unselected samples.
I've done the mathematical reasoning and determined that I want to use a Mann-Whitney U test for the continuous data, a Pearson's Chi-Squared test for categorical data, and a Binomial test (exact, without the normal approximation) for dichotomous data. All of these tests are available in the excellent scipy.stats library for Python.
Is there a Javascript library available with implementations of these tests? Failing that, is there a Javascript library that provides PDFs and PMFs and CDFs of the distributions that would be required to implement these tests, like the Chi-squared distribution, or the (discrete) binomial distribution? Failing that, is there a resource available that explains how these tests work with an eye towards implementation? Failing that, is there a library of basic mathematical functions for probability, like erf or the gamma integral?
I am aware of jStat, which seems to provide only a few continuous distributions with no API documentation, and of OpenEpi, which is more of a monolithically integrated epidemiological statistics system than a usable library.
This is not a full answer, just evidence to the negative.
Apart from never encountering such a powerful library I have googled a little and found no results other than jStat. However I have found a page (http://home.ubalt.edu/ntsbarsh/stat-data/Javastat.htm) where some client-side statistical calculations can be performed. This is not a library, and I have had a look at some of their javascript and it seems like they have coded the calculations by hand (which I wouldn't expect if a library existed)
So at best I offer some evidence to the negative.
(Also note the rather inconclusive post Recommend a good javascript statistics library?)
I wouldn't want to do heavy numerical integration in javascript though, can't you do that on the server-side instead, especially if the user is merely selecting data, and not entering it?