I have several data "layers" over the same list of "samples". Some layers are continuous float data, some are multinomial/categorical data, and some are binomial/dichotomous/boolean data. In my Javascript web application, I want users to be able to select a set of samples and see which layers are significantly different between the selected set of samples and all other samples. The end result should be a p-value for each layer, from a two-tailed test (where applicable) of the null hypothesis that the distribution over selected samples is the same as the distribution over unselected samples.
I've done the mathematical reasoning and determined that I want to use a Mann-Whitney U test for the continuous data, a Pearson's Chi-Squared test for categorical data, and a Binomial test (exact, without the normal approximation) for dichotomous data. All of these tests are available in the excellent scipy.stats library for Python.
Is there a Javascript library available with implementations of these tests? Failing that, is there a Javascript library that provides PDFs and PMFs and CDFs of the distributions that would be required to implement these tests, like the Chi-squared distribution, or the (discrete) binomial distribution? Failing that, is there a resource available that explains how these tests work with an eye towards implementation? Failing that, is there a library of basic mathematical functions for probability, like erf or the gamma integral?
I am aware of jStat, which seems to provide only a few continuous distributions with no API documentation, and of OpenEpi, which is more of a monolithically integrated epidemiological statistics system than a usable library.
This is not a full answer, just evidence to the negative.
Apart from never encountering such a powerful library I have googled a little and found no results other than jStat. However I have found a page (http://home.ubalt.edu/ntsbarsh/stat-data/Javastat.htm) where some client-side statistical calculations can be performed. This is not a library, and I have had a look at some of their javascript and it seems like they have coded the calculations by hand (which I wouldn't expect if a library existed)
So at best I offer some evidence to the negative.
(Also note the rather inconclusive post Recommend a good javascript statistics library?)
I wouldn't want to do heavy numerical integration in javascript though, can't you do that on the server-side instead, especially if the user is merely selecting data, and not entering it?
Related
I am fairly new to NLP in general. My goal is to create some kind of parser that can easily find files on my various hard drives.
I have no idea how to properly parse the input to transform it into any managable representation that a program can easily use to create a given output.
For example, the following sentences should return a list of documents:
documents created 3 months ago
documents modified 2 weeks ago
photos taken in china (this one would then use the GPS data within the image file)
It can probably be easily done using some kind of Regex pattern (<filetype> <action> <time>) but I would love to make it more flexible.
I looked into compromise, a JS library that has some easy to use API to retrieve specific parts of the input. But I kind of doubt that calling methods like calculatedResult.nouns()[0] and calculatedResult.verbs()[0].stem() should be used to parse the commands as those require a fixed kind of syntax.
Any tips on how to achieve my goal? I am not sure if using ML and training a custom model is the way to go. I never use ML and, based on my low knowledge of it, it seems kind of hard to train it constructs like those (as I would need a LOT of example sentences but there is just a finite amount of realisically used combinations that make sense).
The NLP technique you need to explore is intent detection. You can either integrate a NLP library like RASA or Spacy into your program or work with a commercial API that conducts intent detection. You will need example sentences in both cases but probably not as many as you think. Intent detection is a key part of chatbots so there's quite a lot of tools out there. Low level, hands-ons ML development is not really needed these days with all the high level intent detection tools out there.
I am attempting to use iTowns2 (https://github.com/iTowns/itowns2) to visualize point cloud data in the browser. According to the README: "[iTowns'] first purpose was the visualisation of street view images and terrestrial lidar point cloud."
From this I glean that there should be instances of people using iTowns to visualize point cloud data somewhere online. I've been looking for days and I can't find an example of someone using iTowns2 to visualize point cloud data in the browser.
The example in the GH repo renders a globe in the browser but no point cloud. There is a iTowns/iTowns2-sample-data repo which has a bunch of point cloud data but no instructions on how to use the data or references to other resources.
Has anyone used this package to show point cloud data in the browser? Does anyone know an article or resource that demonstrates doing this with iTowns2? Does anyone know of a different library for rendering point cloud data with examples and/or better documentation?
Ideally I would be able to track down the source code for something like this: http://www.itowns-project.org/#demo
The documentation is quite ambiguous, and judgind by the Github issues it looks like the library is under heavy refactoring.
I took a quick look to this repo and realized that it is just using Potree for point cloud visualization:
http://potree.org/
So you can just use Potree directly. Wich is better documented.
In addition to this, it's quite trivial to set up your own point cloud visualizer using Three.js.
Just take a look at the Points object:
https://threejs.org/docs/#api/objects/Points
And this example:
https://github.com/mrdoob/three.js/blob/master/examples/webgl_buffergeometry_points.html
Three.js also includes some 3D format loaders, like ply:
https://github.com/mrdoob/three.js/blob/master/examples/webgl_loader_ply.html
If you are interested in using las files you might also want to look at:
https://github.com/verma/plasio
Let's provide a early 2018 update! (source: I'm a maintainer)
iTowns now supports visualizing pointclouds directly. You can test it here: http://www.itowns-project.org/itowns/examples/pointcloud.html
If you want to test your own data, please visit http://www.itowns-project.org/itowns/examples/pointcloud.html?selector=1
We currently support results from PotreeConverter, and lopocs. We plan to add 3dtiles pointcloud format soon.
We indeed used potree for pointclouds before, but that was not ideal, a bit because we diverge on some technology/design choices, but mainly because using potree prevented us to tightly integrate pointcloud visualization in iTowns. For instance, iTowns stops its rendering loop when it has nothing to do (saves a lot of cpu), and potree does not. It also allows us to implement our own culling/SSE/network priority... heuristics.
Potree has currently better graphic post-treatments of pointclouds, although we also plan to add EDL and other improvements (occlusions for instance) soon. And of course, the advantage of iTowns is that it's not limited to pointclouds, but can display a variety of data type, from rasters to vectors, see the examples page and especially this example of a pointcloud on a globe.
But the main difference between these 2 projects is that Potree aims at being a standalone viewer (AFAIK), whereas iTowns is more a framework to implement your own app! Potree remains a big source of inspiration for us concerning pointclouds, big kudos to their maintainer :-)
(Btw, the github has moved to https://github.com/iTowns/itowns)
I am new to SCORM and have been given an assignment to integrate SAP Workforce Performance Builder exported SCORM (can either be 1.2 or 2004) content into an existing PHP website.
To put it simple, I need to be able to display the exported SCORM material in the browser (I can already do this), and be able to get the statistics through the SCORM runtime API.
I understand that I will need to make use of an LMS to allow communication with the SCO through the SCORM runtime API. I have looked into several open source LMS's, but haven't found a good solution for my purpose. The problem is that a lot of these LMS's are designed to run on the domain of the provider, and have built in tools to follow up on users' progress and scoring.
What I'm looking for is a simple, lightweight solution to be able to interact with the SCORM runtime API, so I can fetch the time a user has spent on a course, his score, etc. I will insert the gathered data into my own database, and code the backend where results can be evaluated myself, all I need is a way to get to the SCORM data.
I feel like I'm missing something, as surely you don't need an entire LMS implementation to simply listen for the basic 8 SCORM API calls, and log the results? Any help or a nudge in the right direction is greatly appreciated!
If you just need to mimic an LMS, providing a pseudo SCORM API so the course can 'speak' to your PHP site, try Claude Ostyn's SCORM Test Wrapper. It's pure client-side JavaScript, as lightweight as you can get with SCORM.
In a nutshell, Claude's test wrapper provides a simple SCORM API for the course to connect to. It receives communication from the course, which you can handle however you like. No backend code is provided; if you want to incorporate with a database, you will need to modify the wrapper to push/pull data from your site's database (this is typically handled via AJAX).
Once you build out the data store, you can make your site behave as an LMS, enabling the site to launch SCORM courses, and enabling the courses to send/receive data to your site via the SCORM API. No LMS or 3rd-party server required.
Notes:
There is no support for unzipping packages or reading manifests. (I suspect you're not interested in going that far.)
SCORM also supports sequencing and navigation, which go way beyond simple JavaScript wrappers. If you need to support the sequencing and navigation features, you'll need to grab them from an existing open-source project (not easy) or pay a 3rd party like Rustici Software (SCORM Cloud). I suspect the content you create via SAP will not use any of SCORM's sequencing or navigation features, so you'll probably be OK.
Claude passed away a while ago, so he can't support you. Shout out to the guys at Rustici Software, who have preserved the site for the SCORM community.
From the courseware's point of view, it is just using javascript to call functions on an API or API_1484_11 object. If you can write the javascript code to sufficiently ape the interface, and store/return the necessary data model elements, then you don't need "an entire LMS implementation".
You need to carefully read the Run-Time Environment documentation though.
If you only ever plan to use it for running SAP Workforce Performance Builder produced courseware, then you can implement enough or the data-model to make that work correctly (although I've seen this done, then people surprised/confused/angry when other SCORM compliant courseware does not work, so beware.)
(Aside) You also need a reliable way to install/update your courseware packages from a PIF zip file. Again, for dealing with courseware from a specific content creator and not needing to write a full blown generic interface, you can just pick out the bits of the imsmanifest.xml file you need.
(Digression) Having written the courseware side of the interface a few times, I've seen interesting gotchas in various LMS implementations of the API, including things like returning the boolean true or false instead of a string "true" or "false" which can catch you off guard. May favourite so far is an LMS that truncates the cmi.suspend_data at the first newline character. (Actually, the implementation was that inept that there was a bug in their bug, and it also chopped off the character before the newline as well.)
You'll mainly want to capture, maintain and enforce the Student Attempt Object. I've used this in a JSON format now for a while, and you can take different approaches to how you store information collected by a Shareable Content Object. Normally people pluck the parts they need vs. trying to go 100% into full SCORM support so these types of questions are popular.
By creating the SCORM Runtime for either SCORM 1.2 or 2004 you'll mainly be providing those methods to build the data from the student session.
This can look like https://gist.github.com/cybercussion/4675334 (based on Unit test data for SCORM 2004)
You attempt to route your calls to your server side. Normally this results in a lot of lag. And I normally don't advocate it as an option.
You cache the student attempt, but you post the whole JSON object on a commit call. This normally results in a larger data post which can blimp on you if there are a lot of journaled interactions.
You take a hybrid approach and only post the data thats changed and merge that on your server limiting the data blimp issues that could occur.
I have a bunch of info up on the wiki here too https://github.com/cybercussion/SCOBot/wiki as well as a lot of sample code, tips etc...
I am looking to build a graphical frontend for some python functionality.
Since I am very proficient in HTML/CSS/JS I decided I might check this out as a possibility.
The main idea is to make it easier to explore data/variables, I am thinking of a datagrid for example.
I was also thinking of creating a GUI for things such as plotting with matplotlib (ie: Choose variable for X axis, choose variable for Y axis, pick a color from a dropdown, ...) in order to cut out some of the tediousness in exploring data with several variables.
I would then probably use something like Node-webkit to make a desktop application out of it if possible (and if not, just keep it as a browser project, perhaps).
Essentially what it would boil down to is kind of creating an IDE for python, specialized for data exploration/plotting and perhaps data manipulation.
An example I found for a web-based python console is repl.it
However I have spent a fair amount of time looking for viable ways to go about this, since sending and receiving data from the python interpreter wouldn't be too hard, but it might get a bit harder for exploring variables or plotting data.
So the question is: Are there any projects or libraries out there to facilitate this kind of functionality, or which provide a good starting point for interacting with Python through JavaScript? Are there any pointers you guys can give me to make the experience for users as seamless as possible?
Thanks!
Say I have an image blending program written in Python. I want to show the image blending process in a real-time fashion in the frontend while user adjusting the parameters (say several scroll bars), which might require Javascript.
I guess one way to do so is to use libraries mentioned in this similar question: first write a Python class capable of doing the backend job, compile it into JS code and call it from the frontend. Any better ways?
The image manipulation task might be heavier in the future so I write in Python instead of in JS directly.
Depending on the ops you are going to implement, perhaps you could write a parallel implementation of your algorithms using HTML5 Canvas. Then show that in a minimal resolution or provide some way to set up a viewfinder (basically a cropped part of the whole). Once the actual job's done, show the full result.
You might also want to consider using Node.js for something like this. Essentially this would allow you to use pretty much the same code for both backend and frontend reducing duplication of the algorithms.
You could also try to rethink the way you perform the manipulation. Aviary encourages the users to perform one operation at a time. In addition they provide undo (easy to implement). This kind of scheme would work really well with Canvas.