I'm sitting here with a huge geoJSON that I got from an Open Street Map shape-file. However, most of the polygons are unnecessary. These could, in theory, easily be singled out based on certain properties.
But how do I query the geoJSON file to remove certain elements (features)? Or would it be easier to save the shape-file in another format (working in QGIS)?
Link to sample of json-file: http://dl.dropbox.com/u/15955488/hki_test_sample.json (240 kB)
When you say "query the geoJSON," are you talking about having the source where you get the geoJSON give you a subset of data? There is no widely-implemented standard for "querying" JSON like this, but each site you retrieve from may have its own parameters to reduce the size of data you get.
If you're talking about paring down the data in client-side code, simply looping through the structure and removing properties (with delete) and array items is what you'd have to do.
Shapefile beats GeoJSON for large (not mega) data. It supports random access to features. To get at the GeoJSON features in a collection you have to read and deserialize the entire file.
Depending on how you want to edit it and what software is available you have a few options. If you have access to Safe FME this is by far the best geographic feature manipuluation software and will give you tons of options (it can read / write (and convert between) just about any geographic format). If you're just looking for a text editor that can handle the volume of data I would look at Notepad++ - it can hold a lot of text and you can do find / replace using regular expressions. Safe FME can be a little pricy, but you might be able to get a trial
As Jacob says, just iterate and remove the elements you don't want. I like http://documentcloud.github.com/underscore/#reject for convenience.
If you are going to permanently remove fields just convert it to a shapefile, remove the fields you don't want, and re-export it as GeoJSON.
I realize this question is old, but if anyone comes across this now, I'd recommend TopoJSON.
Convert it to TopoJSON.
By default TopoJSON removes all attributes, but you can flag those you'd like to keep like this:
topojson -o output.topojson -p fieldToKeep,anotherFieldToKeep input.geojson
More info in the TopoJSON command line reference
Related
I have already looked here, here, and here:
D3.js: How to combine 2 datasets in order to create a map and show values on.mouseover?
How to add properties to topojson file?
https://github.com/topojson/topojson-1.x-api-reference/blob/master/Command-Line-Reference.md
They were a great help in getting me as close as I am now. However, I'm attempting to recursively parse several datasets in .tsv format and join them together with the json geometry data. I use the word join loosely; I know that topojson has some command line abilities to join .tsv to the json file. I think that would simplify things a lot, but if I'm not mistaken that requires Node. I tried many times to get Node working on my computer, it is either a compatibility issue or I'm not adept enough.
So, simply put, my question is:
Bearing in mind that I don't have Node, what changes do I need to make (data file related, or other) to get my block working (parsing dynamically)
I'm assuming that it's not working because of my chaotic data structure (as suggested by the post title), but if you think its something else, let me know.
My block:
http://bl.ocks.org/diggetybo/42f4534b2168a00694219888ebb1a2fb
I have a Leaflet map and I want to include some data from a webservice that returns the data in a strange XML format.
Is leaflet able to handle XML somehow? (I'm afraid it isn't as I didn't find anything about that)
If not, what would be the best/easiest way to handle that XML? Should I write/use (I found a lot of them) a parser to convert the XML to a more leaflet-comptaible format like (geo)json? Should I parse the XML directly into a Leaflet-Layer (sounds like writing a plugin myself)?
Is there any other option I am missing?
Depending on what your XML data contains- If it is a lot of geographic information I would consider converting it to KML.
I'm using something similar where I use a php xml handler to edit an existing KML into a format that is more Leaflet friendly, then loading it into my map with this.
Otherwise I think geojson is the way to go.
Hope this helps.
The X in XML stands for "extensible". The link provided above (FDSN) is not a "strange format", but rather it's a particular schema. JSON is a very easy to use format because no schema information is required, although JSON schemas do exist. However, when information needs to be standardized, for any number of reasons, a schema is necessary in order for the information to be comprehensible. The FDSN schema apparently represents the standard metadata schema for working with seismographic data. One will find that various industries and organizations have decided on elements to be incorporated in documents for purposes of interoperability.
XML generally is not as straight-forward as JSON (or GeoJSON), but the reason it is used is precisely because of the extensibility, because in many applications and data collections a rich metadata schema is required and/or preferrable. Regardless, a quick conversion may not be possible if the schema itself is not well understood.
See the following link for an outstanding tutorial on using XML in Leaflet. NOTE: Anytime XML is used, one needs to at least open the file and look around in order to get an understanding of exactly what each element entails. Where is the coordinate information, if any?
http://erica.altschul.info/Tutorial_XML-to-Leaflet.pdf
ternJS have several. JSON files defs which contains the definition of librarys. Can someone explain to me how I can best generate my own to my javascript libraries / or only definition objects?
I can not see that there is no common procedure for this?
There's a tool for this included in Tern. See condense at http://ternjs.net/doc/manual.html#utils . It runs Tern on your file and tries to output the types that it finds. It's far from flawless, but for simple programs it works well. For files with a complicated structure or interface, you'll often have to hand-write the definitions.
There are three ways I have thought about to solve your problem:
Using Abstract Syntax Tree Parser and Visitor
One way to solve your problem would be to use abstract syntax tree parser and visitor in order to automate the task of scanning through the code and documenting it.
The resources here will be of help:
-http://ramkulkarni.com/blog/understanding-ast-created-by-mozilla-rhino-parser/
-What is JavaScript AST, how to play with it?
You usually use a parser to retrieve a tree, and then use a visitor to visit all the nodes and do your work within there.
You will essentially have a tree representing the specific library and then you must write the code to store this in the def format you link to.
Getting a Documentation Generator and Modifying
Another idea is to download the source code for a documentation generator, e.g. https://github.com/yui/yuidoc/
By modifying the styling/output format you can generate "documentation" in the appropriate json format.
Converting Existing Documentation (HTML doc) into JSON
You can make a parser that takes a standard documentation format (I'm sure as Javadoc is one for java there should be one for javascript), and write a converter that exctracts the relevant information and stores in a JSON definition.
I have a problem I'd like to solve to not have to spend a lot of manual work to analyze as an alternative.
I have 2 JSON objects (returned from different web service API or HTTP responses). There is intersecting data between the 2 JSON objects, and they share similar JSON structure, but not identical. One JSON (the smaller one) is like a subset of the bigger JSON object.
I want to find all the interesecting data between the two objects. Actually, I'm more interested in the shared parameters/properties within the object, not really the actual values of the parameters/properties of each object. Because I want to eventually use data from one JSON output to construct the other JSON as input to an API call. Unfortunately, I don't have the documentation that defines the JSON for each API. :(
What makes this tougher is the JSON objects are huge. One spans a page if you print it out via Windows Notepad. The other spans 37 pages. The APIs return the JSON output compressed as a single line. Normal text compare doesn't do much, I'd have to reformat manually or w/ script to break up object w/ newlines, etc. for a text compare to work well. Tried with Beyond Compare tool.
I could do manual search/grep but that's a pain to cycle through all the parameters inside the smaller JSON. Could write code to do it but I'd also have to spend time to do that, and test if the code works also. Or maybe there's some ready made code already for that...
Or can look for JSON diff type tools. Searched for some. Came across these:
https://github.com/samsonjs/json-diff or https://tlrobinson.net/projects/javascript-fun/jsondiff
https://github.com/andreyvit/json-diff
both failed to do what I wanted. Presumably the JSON is either too complex or too large to process.
Any thoughts on best solution? Or might the best solution for now be manual analysis w/ grep for each parameter/property?
In terms of a code solution, any language will do. I just need a parser or diff tool that will do what I want.
Sorry, can't share the JSON data structure with you either, it may be considered confidential.
Beyond Compare works well, if you set up a JSON file format in it to use Python to pretty-print the JSON. Sample setup for Windows:
Install Python 2.7.
In Beyond Compare, go under Tools, under File Formats.
Click New. Choose Text Format. Enter "JSON" as a name.
Under the General tab:
Mask: *.json
Under the Conversion tab:
Conversion: External program (Unicode filenames)
Loading: c:\Python27\python.exe -m json.tool %s %t
Note, that second parameter in the command line must be %t, if you enter two %ss you will suffer data loss.
Click Save.
Jeremy Simmons has created a better File Format package Posted on forum: "JsonFileFormat.bcpkg" for BEYOND COMPARE that does not require python or so to be installed.
Just download the file and open it with BC and you are good to go. So, its much more simpler.
JSON File Format
I needed a file format for JSON files.
I wanted to pretty-print & sort my JSON to make comparison easy.
I have attached my bcpackage with my completed JSON File Format.
The formatting is done via jq - http://stedolan.github.io/jq/
Props to
Stephen Dolan for the utility https://github.com/stedolan.
I have sent a message to the folks at Scooter Software asking them to
include it in the page with additional formats.
If you're interested in seeing it on there, I'm sure a quick reply to
the thread with an up-vote would help them see the value posting it.
Attached Files Attached Files File Type: bcpkg JsonFileFormat.bcpkg
(449.8 KB, 58 views)
I have a small GPL project that would do the trick for simple JSON. I have not added support for nested entities as it is more of a simple ObjectDB solution and not actually JSON (Despite the fact it was clearly inspired by it.
Long and short the API is pretty simple. Make a new group, populate it, and then pull a subset via whatever logical parameters you need.
https://github.com/danielbchapman/groups
The API is used basically like ->
SubGroup items = group
.notEqual("field", "value")
.lessThan("field2", 50); //...etc...
There's actually support for basic unions and joins which would do pretty much what you want.
Long and short you probably want a Set as your data-type. Considering your comparisons are probably complex you need a more complex set of methods.
My only caution is that it is GPL. If your data is confidential, odds are you may not be interested in that license.
I am looking for a javascript library which renders an arbitrary (but acyclic) JSON data blob into some sort of semi-interactive HTML view. James Padolsey's Prettyprint library comes close, but its output is very verbose ("object" and "array" headers on everything, for instance), is only marginally interactive (the ability to collapse and expand subtrees would be nice, especially) and not particularly customizable. I also found jstree, but it looks like that doesn't do arbitrary JSON data blobs, only ones specifically constructed to be fed to it. Also, a strict treeview is not really right for the data I have; I want more of a key/value presentation (but with some values being nested objects).
I do not need the ability to modify the data structure, just show it in a more-or-less human readable fashion.
Any suggestions?
I have a small project to display jsobjects.
Its not very pretty and could use some improvements but it might help out a little.
It is built on "jquery-1.4.2.min.js" but should work with older versions.
Files:
http://empirium.dnet.nu/js/object-browser.js
http://empirium.dnet.nu/js/object-browser.css
This is an example on how to use it:
http://empirium.dnet.nu/OBTest.html
Clicking on the bold black type will open and close complex datastructures that are not visible imediatly.
I hope you have some use for it and if you have any suggestions please just comment here.
Its not an active project, just something I wrote to do some debugging.