Automated update system (Autoupdate)

This article provides a description of the Autoupdate feature that is available for customers on
MapTiler Server and Data.

Global Vector Dataset Updates

MapTiler Data subscription package allows downloading of data for an unlimited amount of times, which means that every time, there is a new global vector dataset available, (every week) you can download it and replace the dataset manually to keep the data updated.

By subscribing to MapTiler Business or Partner subscription tiers, you are eligible for the Autoupdate feature which saves you time and effort to keep the data as fresh as possible at all times.

About Autoupdate

Autoupdate enables you to automatically apply differential updates to global vector datasets.

The whole automated process is done by running a script on your machine. This is a console-based script, which needs to be run from the Terminal or Command prompt or as a service (crontab, Windows service, etc).

The script will update the "planet.mbtiles" file to the latest version visible on the OSM Dataset download page. In other words, the script will attempt to download all the diff-files available since the last release. This could cause temporary congestion. The auto-update script is designed to run continuously on the customer's machine without any longer interruption. After the update is finished, the old planet file will be named "planet-old.mbtiles" and stored in the temporary directory.

In case your MapTiler planet dataset is older than 1 month, it is better to download a new MapTiler Planet dataset. That will save you from downloading more diff files than 4 (the diff files are issued every week. Once you have the latest planet, you can run the auto-update script to make sure all the updates are downloaded to your machine. We recommend running the script once a week (WE/TH) when the new diff files are usually released.

Note: Autoupdate is available only for OpenStreetMap vector tiles of planet dataset.

Update Process

1. Run the script on your server with a planet package.
2. The script will download a new planet differential file.
3. The script will rename and move the original planet file - the web server will still read this old planet file.
4. The script will merge these differential files with your old planet.
5. The script will automatically loop to update the planet to the latest version possible.
6. You can restart your web server to finish the process and serve a new planet package.

Your maps are available through the whole update process without interruption, except for a short timeout during the restart of your web server.

Here is a short real-life example:

# ls -lh /var/www/html/map/*.mbtiles
-rw-r--r--. 1 root root 100G Jan  5 20:49 /var/.../map/maptiler-osm-2022-11-28-v3.14-planet.mbtiles
-rw-------. 1 root root  33G Jan  5 15:00 /var/.../map/maptiler-osm-2022-12-05-v3.14-planet-
-rw-------. 1 root root  32G Jan  5 19:44 /var/.../map/maptiler-osm-2022-12-19-v3.14-planet-diff_beov2f5z.mbtiles
-rw-r--r--. 1 root root  99G Jan  5 19:04 /var/.../map/planet-old.mbtiles

Note: In case the OpenStreetMap vector tiles planet schema had changed since your last update (your locally stored data have a schema that is different from the schema of the auto-updates you are trying to download), the auto-update will download the whole new file, and replace it, as merging between the two schemas is not possible. 

Note: When you copy the downloaded mbtiles into the right directory and click "RE-SCAN", you might occasionally face an issue that reads: "CANNOT BE SERVED" with the following reason:

This happens because the file is corrupted - this could happen due to limited space during the auto-update operation. To verify the file, you can run the following command: maptiler_autoupdate --info file.mbtiles
If the file is corrupted, it cannot be repaired (hence there are backups of the old planet before merging), but has to be downloaded freshly from the website. You can verify that the file was fully downloaded using MD5 hash sum.md5sum file.mbtiles

Technical Specification

The binary script is packed in a ZIP file and available for all platforms.

Merging differential MBTiles files with a whole planet MBTiles is intensive for I/O, which requires a fast Hard Drive disk (SSD). The average speed of merging of a single diff MBTiles (with approx 30 GB size) with a planet (with approx 100 GB size) is 1 hour using a locally attached SSD disk. This was measured on an Intel Xeon D-1520 - 4c/8t - 2.2GHz, 32GB DDR4, 880 GB SSD.