Advanced Topics

Indexdata

pbfhogg embeds additional metadata in BlobHeader fields: element type, ID range, spatial bounding box, and tag key sets. Standard PBF readers silently skip these fields per protobuf wire format rules.

This metadata enables commands to skip decompression of irrelevant blobs entirely. For example, apply-changes classifies blobs in O(1) without decompressing them, passing through ~92% of blobs as raw bytes. tags-filter skips blobs that provably lack required tag keys.

Generate an indexed PBF:

pbfhogg cat input.osm.pbf -o indexed.osm.pbf

Without a --type flag, cat adds indexdata via decompress+scan without re-compressing blobs. Memory usage is minimal. Planet (87 GB) completes in ~8 minutes with under 0.5% file size overhead.

Commands that benefit from indexdata: apply-changes, sort, add-locations-to-ways, extract (complete/smart), tags-filter, getid, cat --type, inspect tags --type, inspect --nodes, and build-geocode-index.

O_DIRECT for planet-scale I/O

Planet-scale operations read and write 80 GB+, polluting the entire page cache and evicting useful data from co-resident processes. The linux-direct-io feature bypasses the page cache entirely.

pbfhogg apply-changes base.osm.pbf changes.osc.gz -o output.osm.pbf --direct-io

O_DIRECT requires a real filesystem (not tmpfs). Wall time is typically unchanged at country scale (CPU-bound) - the benefit is cache hygiene at planet scale. For sequential single-file passthrough (cat), buffered I/O is actually faster because the page cache prefetch helps. --direct-io wins for concurrent read/write patterns like merge.

io_uring writes

The linux-io-uring feature replaces the synchronous writer thread with io_uring WriteFixed and pre-registered page-aligned buffers. Requires Linux 5.1+ and sufficient RLIMIT_MEMLOCK (16 MB for the default 64-buffer pool).

pbfhogg apply-changes base.osm.pbf changes.osc.gz -o output.osm.pbf --io-uring

At North America scale (18.8 GB), io_uring + --compression none is 20% faster than buffered writes (11.9s vs 14.9s). Below ~4 GB input size, buffered writes keep up - io_uring overhead dominates when the page cache absorbs everything.

Compression modes

All write commands accept --compression:

Value	Description
none	No compression. Fastest writes, largest files. Ideal for intermediate files or erofs storage.
zlib	Zlib level 6 (default). Standard PBF compression, compatible with all tools.
zlib:LEVEL	Zlib with explicit level (0-9). Higher = smaller + slower.
zstd	Zstandard level 3. Better ratio and faster decompression than zlib.
zstd:LEVEL	Zstandard with explicit level.

With pipelined writes (the production path), compression is dispatched to rayon and all modes converge to the decode + serialization floor. The choice mainly affects file size and downstream read speed.

Zlib uses zlib-rs (pure Rust). No C compiler needed.

Add-locations-to-ways index types

add-locations-to-ways embeds node coordinates in ways. It supports two index strategies (plus auto) via --index-type:

pbfhogg add-locations-to-ways input.osm.pbf -o output.osm.pbf --index-type external

Type	Memory	Disk	Sorted required	Best for
sparse (default)	~540 MB + IdSet/rank index	referenced_count * 8 bytes (japan 2 GB, europe ~29 GB)	no	Small to europe scale; survives europe at ~6 minutes on a 27 GB-RAM host
external	~8.7 GB	~256 GB (planet)	yes + indexdata	Planet-scale, the only mode that survives at planet on memory-constrained hosts
auto	varies	varies	scale-aware (see below)	Recommended default

sparse is a rank-indexed flat mmap array (~8 bytes per referenced node). Builds a referenced-id IdSet in pass 1, then writes locations in pass 2 indexed by rank within that set. Works on any PBF.

external uses a rank-bucketed counting sort with parallel stages and bounded memory. Requires sorted input with indexdata. ~256 GB temp disk at planet scale.

auto is scale-aware, not a fixed choice: it estimates the sparse store size from per-blob indexdata node counts (8 bytes per node) and routes to external only when the input is sorted + indexed AND the estimate exceeds ~80% of the host's available RAM - otherwise it picks sparse. The threshold is computed at runtime, so identical input can route differently on differently-sized hosts. Auto previously routed every sorted + indexed input to external, which misrouted small and medium inputs (denmark: sparse 5.8s vs external 12.3s) since external's fixed scratch round trips only pay off once the sparse store outgrows the page cache. When the estimate is unavailable (missing indexdata mid-file, no /proc/meminfo), sorted + indexed inputs fall back to external. The routing decision and both numbers are printed to stderr. Explicit --index-type sparse|external is unaffected.

A previous dense mode (a direct-mapped mmap array indexed by node ID) was removed: the rank-indexed flat sparse layout dominated dense at every measured scale (japan 4.3x faster) and worked in regimes dense did not (europe survives where dense OOMs).

Multi-extract

Extract multiple regions in a single pass using a JSON config file:

pbfhogg extract input.osm.pbf -c regions.json

The config file defines multiple extract regions, each with a name, output path, and bounding box or polygon. All regions are extracted in one pass over the input, which is much faster than running separate extracts.

Use -d to override the output directory for all extracts:

pbfhogg extract input.osm.pbf -c regions.json -d /data/extracts/

Tags-filter with and without -R

Without -R (default mode), tags-filter resolves matched relation members transitively: member ways, member nodes, nested member relations are included, and node refs of included ways are pulled in. This requires multiple passes but gives complete, usable output.

With -R (omit-referenced), only directly matched elements are emitted. This is a single pass and significantly faster, but the output may have dangling references.

# Full resolution (default) - complete output
pbfhogg tags-filter denmark.osm.pbf -o highways.osm.pbf "highway=primary"

# Direct matches only - faster but may have dangling refs
pbfhogg tags-filter denmark.osm.pbf -o highways.osm.pbf -R "highway=primary"

Tags-filter also supports OSC input (autodetected from extension, or override with --input-kind osc). In OSC mode, delete directives are always preserved.

Build-geocode-index

Build a reverse geocoding index from a PBF file:

pbfhogg build-geocode-index denmark.osm.pbf --output-dir geocode-index/

This runs a 4-pass pipeline: admin boundary relations, referenced node collection, node+way fused scan, and bucketed S2 cell assignment. The output is a set of 19 memory-mappable binary files for fast reverse geocoding queries.

Europe: ~10 minutes, 7.5 GB RSS. Planet: ~22 minutes, 18 GB RSS.

The index can be queried from Rust using the geocode-reader feature:

[dependencies]
pbfhogg = { version = "0.5", default-features = false, features = ["geocode-reader"] }