OpenHistoricalMap in Marble

Hi, I’m a contributor and advisor to OpenHistoricalMap (openhistoricalmap.org), a spinoff of OpenStreetMap that focuses on collaboratively mapping the world throughout history – everything from borders to aqueducts to railways to restaurants. You can read more about it at wiki.openstreetmap.org/wiki/OpenHistoricalMap.

I’d love to see Marble integrate OHM as a map view alongside OSM. As you’d expect from such an open-ended project, coverage is still very incomplete and very uneven. However, it has the potential to grow into a serious resource for historical geography, just as OSM has for present-day geography. OHM’s hyperlocal resolution would complement Marble’s existing support for georectifying old world maps.

OHM runs on all the same software as OSM and uses the same data model and file format, except that the data is in the public domain. As far as I can tell, an OHM map view would be very similar to the Vector OSM map view. Adding a raster map view isn’t an option, because OHM doesn’t publish raster tiles. (We can’t publish a different set of raster tiles for every day in history!) Instead, we publish a planet .pbf every day, as well as vector tiles in MVT format that update within minutes of an edit. Although Marble uses a custom vector tile format, the existing tools should work just fine for converting OHM’s planet to that format.

The most interesting part would be dynamically filtering the data by time, probably using the existing Time Control dialog box. Digging around the source code, I couldn’t tell if there’s already a way to filter a vector layer dynamically, but an OHM map view would need that in order to be usable. For search, we also have our own Nominatim instance, though it has some rough edges.

As this is my first foray into the KDE community, please let me know if there’s someone in particular I should reach out to in order to explore this idea further. Thanks for creating this impressive map viewer!

3 Likes

I can only comment on the server infrastructure part of this, for the client-side part of Marble you probably want to talk to Torsten Rahn (rahn@kde.org), not sure whether he follows this forum.

IIUC the OHM data model is basically OSM plus temporal tags? If so the “raw data” tiles server approach should work as-is indeed, leaving the time-based filtering entirely to the client.

Besides an initial full planet pbf the server needs two things to work, incremental updates in the same format as they are provided for OSM (currently applied daily, but the system is capable of minutely updates as well if needed), and huge and fast SSDs (for OSM we recently crossed the 1TB mark for the database). Unless OHM is significantly smaller than OSM the latter would at least be a blocker on our current hardware I fear.

3 Likes

Thank you for your quick response!

Correct, the only differences from OSM are in a few tagging conventions, such as those temporal tags. Otherwise, the data formats are exactly the same.

Yes, incremental updates are also available from planet.openhistoricalmap.org in the same format that OSM provides. Assuming you don’t need the full revision history, the whole planet .pbf currently weighs 883.5 MB, compared to OSM’s 76 GB and comparable to an OSM extract of Austria. Over the last four years, it has grown by less than 3 MB per month, though it’s been ticking up slightly in the last year or so.

The database tables should therefore take up a fraction of what you’re currently using for OSM. However, depending how you’re processing the data, the size could be misleading. We have a lot more boundary data than OSM, including some ways that are members of hundreds of boundary relations each. So any complex postprocessing on boundary relations might take more resources, especially at admin_level=6 and beyond.

2 Likes

Hey, this looks nice indeed. Personally I really like historical maps. So from what I understand the issue is twofold: We’d have to figure out how to serve the data as vector tiles (Volker already mentioned the challenges involved). And on the other hand we have the adjustments to the client code. Regarding the latter if I understood you correctly you are source code savy and could look into the issue yourself :wink:
For Marble Vector tile rendering Marble uses the o5m format which resembles the official binary representation of OpenStreetMap’s OSM format. This OSM data structure is then imported into Marble’s unified data structure (where the data model adheres to the semantics and structure of the KML file format). So if you have a good idea about the OSM file format and the KML format (KML Reference  |  Keyhole Markup Language  |  Google for Developers) then you should be able to follow the code.

Time tags or filtering are not implemented. However adding this should be a relatively straight-forward task.
Thinking about it: Actually the tree symbols we use in Marble follow the seasons: In autumn they are displayed brownish and in winter they have a white top. Unfortunately I just realized when I tried it that this feature is tied to the system clock and not to the MarbleClock. Hence you need to change the system clock to see the effect (while it should instead honor the TimeControl’s Marble Clock). Also I needed to manually trigger a tile reload to see the effect after changing the system time. I guess I need to fix this …
@1ec5 if you should feel like adding support to Marble for this, then don’t hesitate to submit patches.

Some more thoughts regarding the client implementation:

As mentioned above Marble’s internal data structure tries to adhere to the KML datastructure. And if you check the KML reference, you will realize that KML has support for a TimeSpan element:

Judging by the example code shown in this section of the reference (which ties the placemark Colorado to its beginning of statehood in 1876) this should be the right way to add support for your planned feature to Marble.

And Marble’s source code seems to even have some initial support for this tag:

I don’t know whether this tag is honored by the rendering yet (I fear that at this point the support might be very limited at best - or non-existent).

So in order to make Marble support this you might want to follow this plan:

  • Step1: Adjust Marble’s OSM-Runner (found at src/plugin/runner/osm). Make sure that the temporal OSM tags get converted to GeoDataTimeSpan information and that the GeoDataDocument that is returned has all the GeoDataPlacemarks (e.g. the ones associated with your historical boundaries) tied to the correct GeoDataTimeSpan objects.

  • Step 2: Adjust the rendering of placemarks so that they only get shown if the GeoDataPlacemark’s TimeSpan matches the time provided by the MarbleClock. If you are lucky this might even be supported already. It could also be that a change of the TimeControl-Widget needs to trigger a reload or the re-rendering of map shown in the client.

  • Step 3: Submit patch :slight_smile:

Did this help? Do you still have any questions?

Have fun!

2 Likes

That’s clever! I should’ve checked out Marble some months earlier to see this. :grinning:

I guess a good starting point would be to try and generate some OHM vector tiles in the required format locally. Is techbase.kde.org/Marble/OSMVectorTileCreation the most current guide to generating vector tiles from an extract?

Not sure how well this still works (@vkrause any idea?). I haven’t tried this for quite some time.

Alternatively you could start off with some .osm file where you just add the temporary tag manually. As mentioned before, the difference between .OSM and .O5M is just that .osm is human-readable/writable while .O5M is the binary flavor. Inside OsmParser.cpp they both get converted to OsmNodes, OsmWays, OsmRelations and from there to a GeoDataDocument.
Hope this helps :slight_smile:

And some remark on the “gimmick-seasonal-tree” feature: I looked at the code and this gets handled in the StyleBuilder, line 326ff in a rather quick and dirty way: The code there has a TODO comment which points out that instead of using QDateTime this should rather use MarbleClock. However, the “central” MarbleClock which is located in the marbleModel is not easily reachable from this code section (and in fact should not be by design) .

However, for your case of “filtering” Placemarks by TimeSpan I’d assume that the marbleModel’s MarbleClock instance should be easier to reach. So this issue shouldn’t be a problem for the OHM feature.

1 Like

That does seem very outdated to me, tools/vectorosm-tilecreator/setup · master · Education / Marble · GitLab describes better what’s currently deployed on the server I think.

1 Like