Show HN: Simple Mbtiles Server – Self-host the entire planet of OpenStreetMaps
github.comDoes anyone know if there is a way to host just a few of the tiles in a static way? For example, if I wanted to build a web page which just shows a map at zoom level 6 for a lat/lng point, and then go to zoom level 13. That would require a tiny subset of tiles; is there a simple way to add the tiles plus the JS code in a static way so no external downloads were necessary?
Yes, that’s pretty easy. I did it for Half Moon Bay, California here: https://simonw.github.io/hmb-map/
Code here: https://github.com/simonw/hmb-map
I've used maptiler to do this. You can give it a boundary and zoom levels, along with a theme, and generate prerendered tiles that you can then server just using http. This has worked well with leaflet.
Look at https://protomaps.com/ - You can have a single file in a that represents the vector data for a region at multiple zoom levels. There used to be a really nice UI where you could just select a region. They've changed things around a bit since then so I'm not sure how it works now.
The replacement for the UI is to use the "pmtiles extract" command line tool:
https://docs.protomaps.com/guide/getting-started#_3-extract-...
You can simply pre-render them and store the png tiles in a folder. The folder structure and naming has to be such that e.g. leaflet understands it. Did this for a research demo here: https://map2seq.schumann.pub/nllni/demo/
Is there actually a simple way to pre-render? AFAIK it's a lot of work and learning at least if you're coming from scratch.
QGIS can make a tile pyramid of the current map/project at a specific set of bounds.
I've used TileMill to create tiles for a specific layer of data for a specific area. (Fire evacuation zones for a county.)
I had success with Protomaps (https://protomaps.com/).
Image, screenshot, Gifs, anything really so I can see what it looks like before allocating 90GBS would be nice.
90 is okay, when I did years ago I was on 300+ but it doesn't really state which zoom level is available.
Is there any explanation why it has decreased in size?
There are two ways to store map tile data like OSM. One is as pre-rendered raster tiles (256x256 png images) and the other is as tiles of vector data that get rendered either on the fly by dedicate rendering middleware or rendered client side by the browser. Vector tiles are anywhere between 20-50% smaller that corresponding raster tiles. Raster tiles was the standard way of doing web maps up until a few years ago, when client side rendering in the browser became feasible and now most people use vector tiles.
It seems to be 0-14 according to the GitHub repo.
Yeah going to full 20 requires about 2TB iirc, OSM doesn't host beyond level 19 anymore either to reduce traffic. 14 is around 10m per pixel, best you'll be able to make out on that is like a street. Not useful enough to bother with imho, even 19 leaves a lot to be desired up close.
Worth noting that these are vector tiles and not raster tiles, so zoom levels work slightly differently. Generally you don't need as high zoom level with vector tiles since it is much easier to 'zoom in' on vector data than on raster data. A vector zoom level of 14 stores the data with roughly 50cm precision which is good enough for most casual mapping uses.
Ah quite right, I see they're mvt. In that case it might actually be usable.
While useful, this still uses a server to process tile requests. Why not https://github.com/protomaps/PMTiles ?
For the record, I am currently using maptiler over at https://citybik.es to serve tiles, and I am evaluating moving to serving static pmtiles on range requests to cut the middle man
PMTiles is great, but there's a risk hosting multi-gb files on metered hosting -- a client might not respect range requests and you open yourself up to a denial of wallet attack.
I can't really see doing it on S3, unless it's behind a lightweight tile server. I'd do it on hetzner or linode or one of the lower cost bandwidth providers.
(edit -- citybik.es is pretty cool, are you backing up to GBFS or a different set of data feeds? e.g. https://github.com/MobilityData/gbfs/blob/master/systems.csv is a list of ~ 800 gbfs feeds, some dockless, some docked I've done a bit of work in this space, feel free to ping me)
To mitigate the "denial of wallet" attack on S3 there are a few options:
AWS Lambda: https://docs.protomaps.com/deploy/aws lightweight tile server: https://docs.protomaps.com/pmtiles/cli#serve
Will keep that in mind if I go that route.
GBFS is just one format of the many supported. This is where we keep all instance data https://github.com/eskerda/pybikes/tree/master/pybikes/data . This is the list of GBFS systems we have https://github.com/eskerda/pybikes/blob/master/pybikes/data/...
Hosting maps of the entire planet in a single file doesn't seem practical. How do you keep it updated? PMTiles is good for small local maps, but I don't think they're meant for this scale.
Here's an example of the entire planet as a PMTiles:
https://pmtiles.io/?url=https%3A%2F%2Fdata.source.coop%2Fpro...
This is zoom 0-15, or 1,431,655,765 addressed tiles. So it is designed for this scale - for a production internet site you can add a lambda, server or CDN as an additional layer for lower latency: https://docs.protomaps.com/deploy/
For updates you're probably better off just regenerating and replacing the mbtiles file from scratch every time you want a new version, rather than trying to incrementally update it.
I assumed you would keep it updated the same way you do with mbtiles, but I see that mbtiles are an sqlite so I guess you can do atomic updates with mbtiles.
Note though that there are many cases on which served tile information rarely changes. A lot of us are serving a once-downloaded mbtile file and are using this just to _not_ use a provider (mapbox, ...)
> Note though that there are many cases on which served tile information rarely changes.
It depends on what you're doing, but outdated maps can be worse than no maps. On low zoom values that probably works fine, but new buildings and streets are built everyday all over the world.
everyday all over the world
If you need updates more often than once a week, you're going to have to take a different approach, since OSM only publish their database exports once a week. In that case you're going to have to use the OSM API to query their database for changes directly and it is only reasonable to track and manage continuous updates on a relatively small surface area.
All of this is of course ignoring the fact that there can be a time lag of anything from hours to decades between a house being built and that house being added to OSM.
without a means to update the map, this isn't a breakthrough. I'd rather use protomaps[0](which has regular builds, from a better-known source) with caddy.
It's OpenStreetMap, without the "s" at the end, please fix the title.
Entire planet maps/topology I guess. But my mind had much much more ambitious first thought.
I remember when I worked on a project in 2010-2012 that relied on OSM it took days to set up a part of the system, the documentation was mostly in German or if you were lucky in an arcane bash script, and which part did what required a deep understanding of the entire thing. I don’t need this anymore but this is really nice.
Tangentially related, does anyone know of an open source 3D terrain system?
I’ve been thinking of making a native map and globe program similar to Google earth to embed inside a dashboard (visually similar to kerbal space program).
So far I was thinking about just using ArcGIS (or something else) and taking in free Lidar data and imposing the depth to satellite or OSM tiles then rendering them as OBJs in OpenGL (or maybe a full 3d engine like unity). This unfortunately wouldn’t be an automatic process as lidar data is taxing to process in real time, especially for a SBC.
I do see that there are some 3D tiles sets out there but I don’t see where I can pull them from nor how it compares to the lidar data.
However, the main goal is a 3D terrain viewer in an offline native program.
> Tangentially related, does anyone know of an open source 3D terrain system?
You can get the heightmap from NASA. It's 10x10 meter resolution, which is enough for static geography.
So I assume if I run this I can aim something like leaflet at it for the map tiles?
Any information you have on how production-ready this is would be nice. Things such as performance, stability, max expected clients with a small vps or something.
Some things you may want to consider when hosting map tiles in production:
- load balancing and caching. Some tiles are much more popular than others.
- how will you update the tiles (sync from OSM).
- if required, how will you manage customizations (and merges with OSM).
Where is my .exe file to run the entire planet in ~30 minutes?
To add some value to my joke comment: I would assume this is running "Open Street Map" locally on a webserver.
A lotta negativity in here... My immediate reaction was "fuck yeah I always wanted to host OSM but I couldn't figure out how last time". thanks dude :)
explanation of what this is would be nice
OpenStreetMap is a map ecosystem based on an open data set. There are numerous redundant ways of achieving the same thing with this data, whether it's hosting the "tiles" (the thing you see on the map), search, UI, location bookmarking, navigation, etc. This is great for flexibility, but it's a difficult thing to dive into and understand.
With that said, the "tile hosting" part is notoriously difficult to actually self-host. It's a lot of data, it traditionally took a lot of time to transform the raw data into the png files representing the tiles. Many if not most web apps actually call out to openstreetmap.org or similar to do the tiles for them. That's not fully self-hosting. But more recently, there have been new ways to do tiles, namely with vector formats instead of pngs. They take up less space, they're easier to share around.
So this submission appears to be a simple few lines to get your fully self-hosted tile server of the whole planet up and running. You should then be able to use this in whatever web app you're making to make it fully self hosted (though if you wanted self-hosted search, that's another hard one...).
Hmm, Im not really sure why MBTiles are popular? They seems to be cache unfriendly. Good old tiles via /{x}/{y}/{z}.png seems to be better. And you can slap web cache in front.
Hosting those is super simple. The problem I have is how to generate them efficently w/o big resources.
I'd like to find a cheap way to self-host a stripped version of Nominatim planet. There's lots of data that I don't really need to deal with and I don't want to have to use 32/64gb of ram.
Looks like every single "tile" image from OpenStreetMaps (OSM) a service that hosts them so that you can pass the same OSM parameters to the URL and get maps
Same feedback as everyone else: You fell into the trap of assuming that everyone else understands this topic/project/thing that you're deeply familiar with because you've been living it for so long.
Not everyone knows what "OSM" is.
Even a 2-3 sentence paragraph in your TL;DR that explains (a) what this is and (b) why people should care or want this, would be immensely helpful.
On the other hand if you don't know what "OSM" is, then you probably don't need this, and probably don't need to care.
Fair but I was referring to the whole write-up on the Gist page not just the headline here on HN.
> Not everyone knows what "OSM" is.
Obligatory: https://xkcd.com/1053/
I wish the readme explained what this project actually was instead of the incredibly vague and unhelpful "self host the entire planet".
> incredibly vague and unhelpful
As is very common with projects on Github. They should have a paragraph at the start explaining simply what they do.
And a screenshot for every project with a UI. I‘d love to see that more often.
They 'could' have a paragraph at the beginning to help potential users understand the purpose of the project.
It might seem inconsequential, but maintainers often deal with a lot of angry entitled users and it could be helpful to stay away from language that implies that the people giving away their code for free (and for fun) have any obligation to provide anyone anything.
True on principle, but if an explanation or a screenshot that costs 5 hours to make would prevent replying or even just reading support requests for 6 hours, I would call it a victory.
Copilot could easily scan and auto generate the read me.
Found little more info in homepage https://github.com/markuman/sms
Still barely more detailed.
The problem with the whole OSM ecosystem is that it expects a lot of domain knowledge of technologies, formats, tools, their relations and features, limitations, that is rarely explained well, or explanaitions are often terribly outdated. Feels much like the JavaScript Frontend frameworks for an outsider, it is a lot of effort to get to an overview and understanding, despite the concepts are not new.
Usually this is a sign of a project that isn't ready for a lot of attention. It's very brave and a little bit stupid to offer Open Source to the world (especially without a sustainability plan), if you put a lot of effort into 'marketing' at an early stage you've probably just overpromised to people not focused on that domain, won't be able to deliver, and will burn out, and it will become yet another abandoned project.
Perhaps in a few years, as the project picks up uses and contributors, it will get its own website and support network and be easy to understand and use by "anyone."
I can’t tell if your comment was meant to be ironic but OSM has been around for 20 years, and its data is used by billions of people.
The comment was about https://github.com/markuman/sms, not OSM.
I don't think this project is looking for a lot of attention.
It is more like a sample of cool stuff you can do.
I agree, but still 1-2 paragraph about why this was made, what is for laymans, and what it is in terms of OSM ecosystem. Possibly my extrovert self makes it natural to write similar intros to repos I make public. (I even used to blog!)
That’s true, I was super confused what “self-host” entailed. They should go into detail explaining that. Leaflet?
Because I understood about “the whole planet” in the context of OSM.
Anyone looking for this will already be in map/OpenStreetMap context. That people originating from StumbleUpon or a niche little place called ycombinator news are confused is not too surprising tbh: that very likely wasn't the audience they had in mind
Considering the title starts with "Show HN" and that the HN username is very closely tied to the GitHub username, I'm gonna go out on a limb here by saying HN is indeed the target audience.
OSM is OpenStreetMaps. This appears to be a server to locally mirror the ~90GB global tile set and serve them with your choice of style to OSM client applications.
It's not OpenStreetMaps, it's OpenStreetMap.
I've always wondered why people keep adding the s. Is it marketing because Google Map sounds like a spelling mistake (missing 's after Google) and we're noticing the effect of that other brand, or is there something about OpenStreetMap that makes it seem like a plural project? People don't say Linuxes, Signals, or Androids
I think it's most likely that most of the other commonly used alternatives (most significantly Google Maps but also Apple Maps, Bing Maps, Maps.me) use the plural.
Whereas when OSM started, they didn't (StreetMap and Multimap in the UK, MapQuest in the US).
Maybe because "Google Map" is actually called "Google Maps" (with the s)?
But where does the plural come from?
Maybe the old days when you needed multiple maps folded up on paper instead of one zoomable super map containing the entire world.
Also "Google Map" sounds like a map of Google.
Because various comments are upset that this isn't actually self-hosting a planet simulation[0] or that the readme isn't good enough - this is really cool. Premade tile server for OpenStreetMaps is lovely.
Why copy the README as a gist instead of posting a link to the repo? https://github.com/markuman/sms?tab=readme-ov-file
It would be great if it's an actual planet with its own ecosystem. This is just a tile map