hyperparse

A pure-Python reader for Tableau .hyper files — no Tableau Hyper API, no third-party dependencies, no native extension.

.hyper is Tableau's on-disk extract format: a full multi-schema database image (superblocks, storage directory, per-column compressed data blocks), not a simple serialized table. hyperparse.py reads that container directly and reconstructs the actual rows, decoding every data type and compression scheme the format defines.

Features

• Reads the container end to end: root record / superblock, storage directory, catalog, and every compressed data block.
• Reconstructs real row values for all standard Hyper types — integers, floats, numeric/BigNumeric, date/time/timestamp (including dates past the Python datetime range), boolean, string/JSON, bytea, geography, interval.
• Handles multi-block relations, multi-relation files, and deleted rows (tombstoned rows are correctly excluded from the read-back result).
• Decodes LZ4-compressed objects with no external LZ4 dependency.
• Can cross-check its own output against the real Hyper API when tableauhyperapi is installed, for verification.

Installation

No installation step — it's a single script with no third-party runtime dependencies:

python3 hyperparse.py --help

Requires Python 3.10+.

Usage

python3 hyperparse.py FILE...              # superblocks + directory + objects + catalog
python3 hyperparse.py --blocks FILE        # every data block, resolved to its column
python3 hyperparse.py --rows 10 FILE       # reconstruct the first N rows
python3 hyperparse.py --samples 10 FILE    # each relation's built-in sample, self-checked
python3 hyperparse.py --json FILE          # machine-readable output
python3 hyperparse.py --dump 0x2000 FILE   # hexdump at an offset

If tableauhyperapi is installed, you can also verify a parse against the real engine:

python3 hyperparse.py --verify FILE

Limitations

• The 14 Huge* compression codes (a third dictionary-entry width, for values or string heaps past 4 GiB) are recognized but not decoded — the reader refuses those blocks rather than guessing at their layout.
• Encrypted files (encryptionSchemeId != 0) are not supported; none have been observed to test against.
• PSMA (a scan-accelerator structure) is parsed for offsets but not used — it's safe to skip when reading rows.

Other tools in this repo

File	Purpose
checkrows.py	Validates the row decoder against the real Hyper API.
kscheck.py	Compiles tableau_hyper.ksy (a Kaitai Struct definition of the format) and diffs it against hyperparse.py field by field.
tableau_hyper.ksy	Kaitai Struct spec for the format, kept in sync via kscheck.py.
lz4probe.py	Standalone tolerant LZ4 block scanner; forensic use only.
make_sample.py, probe*.py	Generate synthetic .hyper files for testing specific encodings, used together with tableauhyperapi as an oracle.

Reference

• Lang, Mühlbauer, Funke, Boncz, Neumann, Kemper — Data Blocks: Hybrid OLTP and OLAP on Compressed Storage using both Vectorization and Compilation, SIGMOD 2016. https://db.in.tum.de/downloads/publications/datablocks.pdf (describes the in-memory ancestor of this on-disk format; useful for vocabulary, not authoritative for byte offsets).
• Official Hyper API docs: https://tableau.github.io/hyper-db/docs/

Status

This is a research/reverse-engineering project, not an officially supported Tableau tool. It has been validated against the real Hyper API across a corpus of 100+ files covering hundreds of columns and relations, but should be treated as best-effort for files or type combinations outside that coverage — always prefer the official Hyper API where available.