I have seen all the responses to a similar question, however, they are all either old, or no one has answered them.
I have been given the task to obtain zip codes and display their corresponding boundaries to the user on a Google map, like in this example.
I am writing this code in Javascript and using the Google Maps API. I want the user to input a zip code and a marker drops down on their destination with a border representing that zip code area. I see that Google Maps currently has something in their map code that allows one to see the boundaries if someone puts a zip code on maps.google.com. I have used polygons but this wouldn't help make a border around a certain zip code.
Any suggestions on how to obtain this?
Thanks in advance!
There is not an easy answer to this that I know of. But here is a high level design of how to do it.
All of the shape files for zip codes can be found at the census site and can be downloaded from this ftp server. However, that's a ton of data, so you need a place to store it. I recommend using the PostgreSQL database with the PostGIS add on. It is free and open source and generally awesome. It has a utility for converting .shp files (the type in the census shape files) into PostGIS geometry form. PostGIS let's you retrieve the shapes back out as KML.
You can either a) retrieve a shape from the database as KML when it is needed and display it on the map or b) pre-generate a kml file for every zip code ahead of time and retrieve a file as it is needed (this would take up quite a bit of space).
You need to become familiar with GeoJSON formatted FeatureCollections. You can render them on any set of map tiles with OpenLayers (or probably Google API as well)
This may seem pretty hard, but is totally approachable.
You can purchase GeoJSON files for groups of Zipcodes if you search around.
DOwnload the shapefile from here https://catalog.data.gov/dataset/tiger-line-shapefile-2019-2010-nation-u-s-2010-census-5-digit-zip-code-tabulation-area-zcta5-na
Simplifying using GDAL
We can use the ogr2ogr command from the GDAL library to convert the shapefile to geojson but even with only one field and simple coordinates the output file is over 1GB.
ogr2ogr -f GeoJSON -select ZCTA5CE10 -lco COORDINATE_PRECISION=6 zcta.geojson /vsizip/tl_2017_us_zcta510.zip
I tried to simplify this to topojson, but the topojson library chokes on this even on a very powerful 2017 MacBook Pro.
npx topojson -q 1e4 -o zcta_topo.json zcta.geojson >> JavaScript head out of memory
Another method I tried was using the -simplify option in ogr2ogr. The simplify argument is a unit of measure based on the spatial reference system of the shapefile. Since the srs for the ZCTAs is WGS84 the unit is a lat/lon measure.
ogr2ogr -f "GeoJSON" -lco COORDINATE_PRECISION=6 -select ZCTA5CE10 -simplify 0.006 zcta.geojson /vsizip/tl_2017_us_zcta510.zip
This creates a much smaller GeoJSON file (30MB) which the TopoJSON can easily handle and we end up with a more managable (but still too large) 13MB topojson file. Additionally, the topology of the dataset is very poor at medium to large scales.
npx topojson -q 1e5 -o zcta_topo.json zcta.geojson
Simplifying using Postgis
Create a docker volume to use for persistence
docker volume create postgresql
Run the postgis docker
docker run --name postgis -p 25432:5432 -it --mount source=postgresql,target=/var/lib/postgresql kartoza/postgis
Load the zcta shapefile into postgis
ogr2ogr -f "PostgreSQL" -progress -select "ZCTA5CE10" -overwrite -lco OVERWRITE=yes -nln zcta -nlt PROMOTE_TO_MULTI -t_srs "EPSG:4326" PG:"dbname='gis' host='localhost' port='25432' user='docker' password='docker'" ~/Downloads/tl_2017_us_zcta510/tl_2017_us_zcta510.shp
Sample query with st_simplifypreservetopology (New England). This takes a long time to run for the entire country and we still lose a lot of the topology.
select st_simplifypreservetopology(wkb_geometry, 0.025) as thegeom, zcta5ce10 from zcta where zcta5ce10 like '0%' OR zcta5ce10 like '1%'
Simplifying using Mapshaper (Best solution)
The Mapshaper library can output TopoJSON directly from the shapefile without JavaScript memory heap errors. This command creates a ~6MB topojson file that we can use. It also manages to keep topology very well by assuming that very close verticies and edges should be coincident.
npx -p mapshaper mapshaper-xl tl_2017_us_zcta510.shp snap -simplify 0.1% -filter-fields ZCTA5CE10 -rename-fields zip=ZCTA5CE10 -o format=topojson zcta_mapshaper.json
source:https://github.com/elastic/ems-file-service/issues/6
Related
I'm ingesting a large geoJSON file in leaflet with somewhere near 19,000 features (points). This results in a massive amount of clutter on the map and is unmanageable.
The goal is to use geolocation to mark my location, draw a 5nm circle around it, and only display geoJSON features that reside within that geometry.
A condensed version of my project is:
https://jsfiddle.net/blintster/d2ucock2/3/
I've worked out finding my location and drawing the circle, but I'm not able to parse the geoJSON features on location. Ideally the output would function like this: https://esri.github.io/esri-leaflet/examples/spatial-queries.html However, that method only seems to apply to L.esri.FeatureLayer and this is a locally imported geoJSON.
The geoJSON layer in question is below where [airports] is the 19,000 entries:
var allairportsLayer = L.geoJson([airports], {
filter: airportFilter,
onEachFeature: function(feature, layer) {
layer.bindPopup(feature.properties.Type + " - " + feature.properties.FacilityName + "<br>Contact Info: " + feature.properties.Manager + "<br> Phone: " + feature.properties.ManagerPhone);
}
}).addTo(map);
function airportFilter(feature) {
if (feature.properties.State === "MD") return true
};
I was able to pair down the results slightly by using the filter method by state but that only allowed me to determine if an attribute meets a specified criteria.
I'v also tried the following method: https://www.mapbox.com/mapbox.js/example/v1.0.0/marker-radius-search/
with no luck.
Does anyone know of any additional methods I could try to parse the data so it only shows points that reside within a geometry?
However, that method only seems to apply to L.esri.FeatureLayer and this is a locally imported geoJSON.
esri-leaflet piggybacks off ArcGIS Server and ArcGIS Online services which provide a backend hosted database that supports spatial queries.
obviously Esri isn't the only option, but your use-case is a perfect example of a situation when it is beneficial not to fetch an entire dataset that you don't plan on displaying.
you could create a arcgis developer account and then sign into arcgis.com to upload your .geojson file as a new hosted service for free.
you could find another hosted service that provides comparable functionality
you could run your own server, install your own PostGIS database and hook up spatial web queries yourself.
you could continue to download all 19,000 features on page load and either:
a) simplify your search and test whether the relevant L.latLngBounds.contains() each point.
b) use something like turf to test the relationship with an actual circle. (one caveat worth mentioning here is that leaflet doesn't include any built in methods for generating actual L.circle geometry so you'd need more custom logic for that too. i wrote something similar here that you are welcome to ripoff).
I downloaded a shape file from the U.S. Census and passed it through http://mapshaper.org/, which gave me a working topojson file to use in D3.
However, I realized the topojson file had all the features labels stripped. So there is no way identify features to link to CSV data.
I installed topojson, but topojson version 2 does not have the same commands as the version one to convert shape files to topojson and retain the features. Previously, topojson -o output.json input.shp This is the current topojson github reference https://github.com/topojson/topojson.
So my question is what is the best way to convert shapefiles to topojson and retain the features attributes.
I'll answer in relation to the mapshaper method you've used.
The problem as I understand it is that you are losing attributes/properties contained within the shapefile when converting to topojson, such as feature id or name which breaks links to data in other files.
When using mapshaper, you need to copy the .dbf, .prj and the .shp files that come with the shapefile into mapshaper. The .dbf contains all the attributes/properties of the features. This will ensure the topojson has the properties that the shapefile does.
If there are too many attributes per feature and you only want one or two of them, then you might need to use another piece of software for that action (or another to both remove attributes/properties and export to topojson).
Part of my web application will allow users to click on a certain territory of a map and have it highlight that section in the Google Maps that I have on the page. This is pretty standard on a lot of websites, but I'm having a hard time with the amount of data that's there.
I'm trying to generate data from http://global.mapit.mysociety.org/, which provides me several output options for actual coordinates that I want to turn into polygons, but exporting the WKT, GeoJSON, or the KML is HUGE - 700-900 KB for EACH instance, and I don't understand how this website is able to incorporate these polygons without loading this data, (Chrome doesn't show any item over 50 KB being loaded.)
What I'm doing NOW:
I downloaded a few of the WKT files, then used PHP to translate that into a javascript string so :
var coordinates = [
new google.maps.LatLng(1.1, 1.2),
...
new google.maps.LatLng(1.1, 1.2)
];
I did this with about 10 locations, saved my file, and it was 9 MB - all from these super long coordinates strings...
How do I do this more efficiently, as this is obviously not the way to incorporate polygon data into a dynamic map load?
Don't translate it on serverside, the string new google.maps.LatLng will blow up your file.
When you use e.g. geoJSON you may load the polygon via loadGeoJson.
The linked page uses geoJSON to draw the polygons(e.g. http://global.mapit.mysociety.org/area/686972.html loads these data: http://global.mapit.mysociety.org/area/686972.geojson?simplify_tolerance=0.0001 ~100KB)
Another option would be to encode each single polygon(polyline) on serverside and reassemble/decode the polygons via JS.
See https://developers.google.com/maps/documentation/utilities/polylinealgorithm and https://developers.google.com/maps/documentation/javascript/reference#encoding for more details.
I have 2 docx files that I am working with. One docx file contains text information of a product (start serial number, length, width, and height). The other docx file contains a sticker label with an image and all of the text information from the first file.
This is what I do currently:
I open the first docx file and copy all of the text information (serial, length, width, and height)
Then I paste each info into the second docx file that contains the formatted label.
If I need to make more than one label, I copy the label and increment the serial number by 1.
This takes a lot of time to make several labels for different products. My goal is to come up with an easier way to take data from one docx and inject it into the other. Also, generating more labels when needed.
My first thought was to extract the docx file to get it's xml contents. Then read the data using javascript, c++, or any other language. Then Ask user to input number of labels to generate, manipulate the xml, and repack it as a docx file.
Then I thought about trying to use the windows office "mail merge" feature, but I have never done this before.
I would like to know if anyone has any suggestions for an easy solution to import data from one docx file and generating labels into another.
I am open for any suggestion.
Also, I am not a professional programmer. I am an undergraduate computer engineering student with some experience in c, c++, java, javascript, python, MIPS assembly, and php.
The only open-source (and probably easier to come by) solution I know know is:
http://poi.apache.org/
http://poi.apache.org/document/quick-guide-xwpf.html
This is a good bet when it comes to speed and it is free software.
But if you open a file, alter it and save it again - the result can be flaky: The formatting can be slightly off. At least in my tests with the pptx counterpart.
I reckon when you have user interaction (web page?) in order to create the document, you can build a small HTTP Api around the library.
There is also: http://www.docx4java.org/trac/docx4j - which I have not tested yet.
You can also go the C#/Redmond way: How do I create the .docx document with Microsoft.Office.Interop.Word?
The Interop (2nd Example in the first answer of the question above) way gives the best result when it comes to the accuracy of the formatting. Basically when you open a file with Interop - it will look the same when you alter and save it. But you cannot use this when interacting with a user - because it starts a separate MS Office process - and I would not count on this from my own user experience. But if you want to generate these files as a batch in a single user session - it will deliver a good result.
I cannot comment on the "OpenXML SDK" library described in the above SO question.
Wath about the Open XML https://www.youtube.com/watch?v=rMnEl6JZ7I8 and website developer http://openxmldeveloper.org/ .
On the site you found sdk for:
Open XML SDK for JavaScript: http://openxmldeveloper.org/wiki/w/wiki/open-xml-sdk-for-javascript.aspx. Demo: http://openxmldeveloper.org/blog/b/openxmldeveloper/p/openxmlsdkjs_demo.aspx
Open XML and Java http://openxmldeveloper.org/blog/b/openxmldeveloper/archive/2006/11/21/openxmlandjava.aspx
.Net Resources http://openxmldeveloper.org/resources/dotnet/m/cc/default.aspx
I've been attempting to create a TopoJson file with consolidated layer data containing, among other layers, U.S. States, Counties, and Congressional Districts.
Original .shp shapefiles come from the Census Bureau's Cartographic Boundary Files.
These were converted to GeoJson via ogr2ogr.
Then combined into TopoJson format via the node server side library, with quantization of 1e7 and retain-proportion of 0.15. Up to this point there is no indication of any problem.
I view the final topojson file using mapshaper and things seem to look OK:
But, when attempting to render with the topojson client library and D3.geo.path(), I encounter some strange paths in the congressionalDist layer: (notice the large rectangular paths around the continental US, AK, and HI)
A working version of the page can be found here: http://jsl6906.net/D3/topojson_problem/map/
A couple of relevant notes:
If I change my topojson generation script to remove simplification, the paths then seem to show correctly via the same d3.js page
If I only keep the congressionalDist layer when creating the topojson, the paths then seem to show correctly via the same d3.js page:
After attempting as much troubleshooting as I've been able to handle, I figured I would ask someone here to see if someone has experienced any similar issues. Thanks for any help.
As I mentioned in the comments, I had noticed that the three offending rectangles all were bound to data with an id property ending in ZZ, while all other paths had IDs ending with numbers.
After some Google searching, I came up with what I think is the answer.
According to this document on the census.gov website,
In Connecticut, Illinois, and Michigan the state participant did not assign the current (113th)
congressional districts to cover all of the state or equivalent area. The code “ZZ” has been assigned
to areas with no congressional district defined (usually large water bodies). These unassigned areas
are treated within state as a single congressional district for purposes of data presentation.
It seems that these three undefined districts would account for the three rectangles. It is unclear at what point in the process they cause the issue, but I believe there is a simple solution to your immediate problem. While searching for information about the ZZ code, I stumbled across this makefile in a project by mbostock called us-atlas.
It seems he had encountered a similar issue and had managed to filter out the undefined congressional districts when running ogr2ogr. Here is the relevant code from that file:
# remove undefined congressional districts
shp/us/congress-ungrouped.shp: shp/us/congress-unfiltered.shp
rm -f $#
ogr2ogr -f 'ESRI Shapefile' -where "GEOID NOT LIKE '%ZZ'" $# $<
I'm betting that if you run your ogr2ogr on your shapefile using the flags shown here it will solve the problem.