I downloaded a shape file from the U.S. Census and passed it through http://mapshaper.org/, which gave me a working topojson file to use in D3.
However, I realized the topojson file had all the features labels stripped. So there is no way identify features to link to CSV data.
I installed topojson, but topojson version 2 does not have the same commands as the version one to convert shape files to topojson and retain the features. Previously, topojson -o output.json input.shp This is the current topojson github reference https://github.com/topojson/topojson.
So my question is what is the best way to convert shapefiles to topojson and retain the features attributes.
I'll answer in relation to the mapshaper method you've used.
The problem as I understand it is that you are losing attributes/properties contained within the shapefile when converting to topojson, such as feature id or name which breaks links to data in other files.
When using mapshaper, you need to copy the .dbf, .prj and the .shp files that come with the shapefile into mapshaper. The .dbf contains all the attributes/properties of the features. This will ensure the topojson has the properties that the shapefile does.
If there are too many attributes per feature and you only want one or two of them, then you might need to use another piece of software for that action (or another to both remove attributes/properties and export to topojson).
Related
I have a CSV file that looks like this:
;A;B;C;DA;0;0;1;2;B;0;3;0;0;C;0;0;0;1;D;1;0;2;0
and I want to make a force directed graph out of it.
Do I need to convert it into a json file first, or can I immediately use the CSV file? Also, the actual CSV file is much larger (1053x1053), with values from 0-9, so the links have to be weighted as well.
I have never worked with d3.js before so I am a bit lost.
Thanks in advance!
D3 has a property that allow you to deal with csv files. usually you write d3.csv('path') to access the data in the csv file so you don't really need to convert them into a json object. I attached below some references for converting csv in d3. link to d3 wiki
Is it possible to get a proper JavaScript source file if I have the minified file and its corresponding source map?
I know how to prettify JS code (line-breaks and indents), but I would like to get original function / variable names in the file, to be able to better understand the source code.
I would like to get the un-minified JS file to work with, instead of using it to debug in a browser.
PS It is probably somewhere right under my nose, but I didn't manage to find it so far. Sorry if this was already asked!
To work sourcemaps requires both files, minified and original, often original is included in sourcemap file(it has optional sourcesContent for sources that can not be hosted).
Sourcemap is just JSON file, and you can found all needed information inside:
sources - list of source file names,
sourcesContent - optional list
of original sources, if source is not presented it would be null
here.
Utility script, I have written before for this purpose: https://gist.github.com/zxbodya/ca6fb758259f6a077de7
I suggest using the Source Map Visualization tool online to view the original code with both js file and js soucemap file.
https://sokra.github.io/source-map-visualization/
I think you won't be able to completely revert such code to its original state as a lot of information (for example comments or certain variable names) is simply lost in the process. When I understand it correctly for sourcemaps to do this you still need the original file.
If you only aim to prettify the code so its readable again you do not need source maps. Many advanced editors have such functions. For example if you are using Sublime text there is this plugin: https://packagecontrol.io/packages/HTML-CSS-JS%20Prettify which does a great job.
Also see this similar question: How can I debug a minified JS in firebug?
I've been attempting to create a TopoJson file with consolidated layer data containing, among other layers, U.S. States, Counties, and Congressional Districts.
Original .shp shapefiles come from the Census Bureau's Cartographic Boundary Files.
These were converted to GeoJson via ogr2ogr.
Then combined into TopoJson format via the node server side library, with quantization of 1e7 and retain-proportion of 0.15. Up to this point there is no indication of any problem.
I view the final topojson file using mapshaper and things seem to look OK:
But, when attempting to render with the topojson client library and D3.geo.path(), I encounter some strange paths in the congressionalDist layer: (notice the large rectangular paths around the continental US, AK, and HI)
A working version of the page can be found here: http://jsl6906.net/D3/topojson_problem/map/
A couple of relevant notes:
If I change my topojson generation script to remove simplification, the paths then seem to show correctly via the same d3.js page
If I only keep the congressionalDist layer when creating the topojson, the paths then seem to show correctly via the same d3.js page:
After attempting as much troubleshooting as I've been able to handle, I figured I would ask someone here to see if someone has experienced any similar issues. Thanks for any help.
As I mentioned in the comments, I had noticed that the three offending rectangles all were bound to data with an id property ending in ZZ, while all other paths had IDs ending with numbers.
After some Google searching, I came up with what I think is the answer.
According to this document on the census.gov website,
In Connecticut, Illinois, and Michigan the state participant did not assign the current (113th)
congressional districts to cover all of the state or equivalent area. The code “ZZ” has been assigned
to areas with no congressional district defined (usually large water bodies). These unassigned areas
are treated within state as a single congressional district for purposes of data presentation.
It seems that these three undefined districts would account for the three rectangles. It is unclear at what point in the process they cause the issue, but I believe there is a simple solution to your immediate problem. While searching for information about the ZZ code, I stumbled across this makefile in a project by mbostock called us-atlas.
It seems he had encountered a similar issue and had managed to filter out the undefined congressional districts when running ogr2ogr. Here is the relevant code from that file:
# remove undefined congressional districts
shp/us/congress-ungrouped.shp: shp/us/congress-unfiltered.shp
rm -f $#
ogr2ogr -f 'ESRI Shapefile' -where "GEOID NOT LIKE '%ZZ'" $# $<
I'm betting that if you run your ogr2ogr on your shapefile using the flags shown here it will solve the problem.
Am very new to alfresco and I am tasked to upload alot of images with their respective metadata. The metadata is in xml format. For example the first batch have 10,000 files with the following naming, for the image is a imagename.Tiff(employee1.tiff,employee2.tiff etc), the xml contains metadata of the employee e.g employee no,name, department, date employeed etc. I have created custom aspects that corresponds to the metadata but my problem is how to index the metadata into alfresco.
I have already created a listening folder where when i drop my files thy are been picked and placed in the right space. The remaining issue is to index the metadata for each uploaded image.
Kindly assist.
An easy way to ingest in bulk, with metadata, is to use Alfresco's Bulk Import Tool
If you are on Alfresco Enterprise, then I recommend that you use the In-place Bulk Import. However, if you do use the In-place Bulk Import, you'll want to distribute the files efficiently on the filesystem and also keep children limited to a few thousand as recommended by Alfresco.
Regarding the metadata in XML - you'll need to transform the XML into the format the tool is looking for. Details available here: http://docs.alfresco.com/4.2/topic/com.alfresco.enterprise.doc/concepts/bulk-import-prepare-filesystem.html
I have seen all the responses to a similar question, however, they are all either old, or no one has answered them.
I have been given the task to obtain zip codes and display their corresponding boundaries to the user on a Google map, like in this example.
I am writing this code in Javascript and using the Google Maps API. I want the user to input a zip code and a marker drops down on their destination with a border representing that zip code area. I see that Google Maps currently has something in their map code that allows one to see the boundaries if someone puts a zip code on maps.google.com. I have used polygons but this wouldn't help make a border around a certain zip code.
Any suggestions on how to obtain this?
Thanks in advance!
There is not an easy answer to this that I know of. But here is a high level design of how to do it.
All of the shape files for zip codes can be found at the census site and can be downloaded from this ftp server. However, that's a ton of data, so you need a place to store it. I recommend using the PostgreSQL database with the PostGIS add on. It is free and open source and generally awesome. It has a utility for converting .shp files (the type in the census shape files) into PostGIS geometry form. PostGIS let's you retrieve the shapes back out as KML.
You can either a) retrieve a shape from the database as KML when it is needed and display it on the map or b) pre-generate a kml file for every zip code ahead of time and retrieve a file as it is needed (this would take up quite a bit of space).
You need to become familiar with GeoJSON formatted FeatureCollections. You can render them on any set of map tiles with OpenLayers (or probably Google API as well)
This may seem pretty hard, but is totally approachable.
You can purchase GeoJSON files for groups of Zipcodes if you search around.
DOwnload the shapefile from here https://catalog.data.gov/dataset/tiger-line-shapefile-2019-2010-nation-u-s-2010-census-5-digit-zip-code-tabulation-area-zcta5-na
Simplifying using GDAL
We can use the ogr2ogr command from the GDAL library to convert the shapefile to geojson but even with only one field and simple coordinates the output file is over 1GB.
ogr2ogr -f GeoJSON -select ZCTA5CE10 -lco COORDINATE_PRECISION=6 zcta.geojson /vsizip/tl_2017_us_zcta510.zip
I tried to simplify this to topojson, but the topojson library chokes on this even on a very powerful 2017 MacBook Pro.
npx topojson -q 1e4 -o zcta_topo.json zcta.geojson >> JavaScript head out of memory
Another method I tried was using the -simplify option in ogr2ogr. The simplify argument is a unit of measure based on the spatial reference system of the shapefile. Since the srs for the ZCTAs is WGS84 the unit is a lat/lon measure.
ogr2ogr -f "GeoJSON" -lco COORDINATE_PRECISION=6 -select ZCTA5CE10 -simplify 0.006 zcta.geojson /vsizip/tl_2017_us_zcta510.zip
This creates a much smaller GeoJSON file (30MB) which the TopoJSON can easily handle and we end up with a more managable (but still too large) 13MB topojson file. Additionally, the topology of the dataset is very poor at medium to large scales.
npx topojson -q 1e5 -o zcta_topo.json zcta.geojson
Simplifying using Postgis
Create a docker volume to use for persistence
docker volume create postgresql
Run the postgis docker
docker run --name postgis -p 25432:5432 -it --mount source=postgresql,target=/var/lib/postgresql kartoza/postgis
Load the zcta shapefile into postgis
ogr2ogr -f "PostgreSQL" -progress -select "ZCTA5CE10" -overwrite -lco OVERWRITE=yes -nln zcta -nlt PROMOTE_TO_MULTI -t_srs "EPSG:4326" PG:"dbname='gis' host='localhost' port='25432' user='docker' password='docker'" ~/Downloads/tl_2017_us_zcta510/tl_2017_us_zcta510.shp
Sample query with st_simplifypreservetopology (New England). This takes a long time to run for the entire country and we still lose a lot of the topology.
select st_simplifypreservetopology(wkb_geometry, 0.025) as thegeom, zcta5ce10 from zcta where zcta5ce10 like '0%' OR zcta5ce10 like '1%'
Simplifying using Mapshaper (Best solution)
The Mapshaper library can output TopoJSON directly from the shapefile without JavaScript memory heap errors. This command creates a ~6MB topojson file that we can use. It also manages to keep topology very well by assuming that very close verticies and edges should be coincident.
npx -p mapshaper mapshaper-xl tl_2017_us_zcta510.shp snap -simplify 0.1% -filter-fields ZCTA5CE10 -rename-fields zip=ZCTA5CE10 -o format=topojson zcta_mapshaper.json
source:https://github.com/elastic/ems-file-service/issues/6