Cuneiform Keys: How Ancient Near East Data Became a Point-Query Lakehouse

From Clay Tablets to Cloud Tables

For thousands of years, the ancient Near East ran on clay. Scribes in Uruk pressed cuneiform into wet tablets to track barley rations, sheep counts, and temple inventories. Those records weren't just archival—they were operational. A priest needed to know how many goats were owed for a festival without reading every tablet in the archive. The system worked because each tablet was a physical record you could pull by hand.

Fast forward to the modern data center, and we've built something strangely similar—but slower at the edges. Our data lakes hold exabytes of information, but grabbing a single record still feels like digging through a collapsed ziggurat. Distributed query engines like Trino and BigQuery are built to scan vast horizons, not to answer "what's the balance for user 417?" That's a point query, and it's the bread and butter of online services.

Spotify recently described a storage architecture called Random Access Parquet (RAP) that tries to fix this. It adds an external index on top of Apache Parquet files, so you can run low-latency point queries directly on data lake objects. No copying to a separate operational database. No duplicate storage. Just a smarter way to find the needle.

The Problem: Lakes Are Not Databases

Data lakes became the central repository for analytics and AI workloads. They're cheap, scalable, and open. But they were never designed for key-based lookups. When you run a query like "fetch the order row with order_id = 998877," the engine has to plan the query, traverse metadata, and discover which files might contain that key. Even with cloud object storage now offering millisecond access latency, the overhead of planning and metadata traversal can dwarf the actual read.

Spotify's numbers make the pain concrete. They store petabytes of online data in Bigtable and exabytes in a Google Cloud Storage data lake. Copying data at that scale to a serving database is expensive and wasteful. It's like re-carving every clay tablet into a new format just to answer a single question.

RAP: An Index Layer for the Lake

RAP doesn't rewrite your files. It adds an external index that maps query keys—like a user ID—directly to the Parquet file and the row position within that file. Instead of scanning thousands of files, the query engine resolves the key in the index and issues a targeted range read against object storage. That's the difference between reading a whole archive and pulling one tablet off the shelf.

The index is built incrementally. As new data lands in Apache Iceberg tables, the index builder appends new index fragments without touching the immutable Parquet files. That means the same dataset can serve analytics, machine learning pipelines, notebooks, AI agents, and latency-sensitive online applications—all without maintaining duplicate storage systems.

Why This Matters for the Ancient Near East

You might be wondering what this has to do with the ancient Near East. The connection is more than metaphorical. The region's administrative records—like the thousands of cuneiform tablets from the Ebla archive—were essentially a data lake. Scribes stored records for taxation, trade, and temple management. Retrieving a specific transaction required either a trained archivist who knew where to look or a laborious manual scan.

Modern systems face the same problem, just at a different scale. The ancient solution was to create catalogs and indexes—often on separate tablets—that listed tablet numbers and their contents. RAP is doing the same thing for Parquet files. It's an external index that makes the lake behave more like a database, without sacrificing the openness and flexibility that made the lake attractive in the first place.

Storage Layout: The Scribe's Art

Spotify also describes several storage layout optimizations that echo ancient scribal practices. One is sorting data by query key, which reduces the number of files you need to access. In Ebla, tablets were often grouped by year or by commodity, so a scribe could find the right batch quickly. Another is interleaving value columns, so that reading a single row fetches all related attributes in one go. Imagine a tablet that lists both the quantity of grain and the recipient's name on the same line—no need to consult a separate tablet.

These techniques come with a cost: files and indexes get slightly larger. But they drastically reduce the number of storage operations. Some point queries now require only a single range read of a few kilobytes. That's like pulling one tablet from a shelf instead of unrolling an entire scroll.

Secondary Indexes: Multiple Keys, No Rewrites

RAP also supports secondary indexes, so you can query by different dimensions—say, buyer ID or seller ID—without rewriting the Parquet files. Hash-based indexes handle exact-match queries, while sorted indexes support range queries. These indexes are managed at the serving layer, which means you can add new access paths without changing your data pipelines. The same Parquet dataset continues to serve both analytical scans and interactive point queries.

Further, techniques like Z-ordering and Hilbert curves can improve data locality for secondary query dimensions. These are essentially ways to map multi-dimensional data onto a linear storage order, so that points that are close in the key space are also close in storage. It's a bit like organizing tablets by geographic region first, then by date—so that a query about a specific city touches a minimal number of tablets.

The Broader Push Toward Operational Data Lakes

Spotify's announcement is part of a larger industry trend. Google Cloud recently described a lakehouse architecture based on Apache Iceberg aimed at AI applications, also trying to enable operational access without data duplication. RAP differs by adding a dedicated external index layer optimized for point queries, while staying compatible with existing Parquet and Iceberg files.

The data engineering community has taken note. Andrew Lamb sees RAP as an example of extending open data formats to support interactive workloads. Vikas Singh, in a LinkedIn discussion, pointed out that as cloud object storage gets faster, the bottleneck shifts to query planning and metadata access—exactly what RAP's precomputed indexes reduce.

Lessons from the Archive

There's something satisfying about seeing ancient Near Eastern record-keeping mirrored in modern data engineering. The scribes of Sumer and Akkad knew that you couldn't just pile up tablets and hope to find anything. They built indexes, sorted by key, and organized storage to minimize retrieval time. We're doing the same with bytes.

RAP isn't a silver bullet. It adds complexity and storage overhead. But it's a practical step toward making data lakes serve both analytical and operational workloads without duplicating everything. And that's a lesson from the ancient world: good indexing beats brute-force scanning, even if you have to chisel the index into clay.

What's Next for the Lake

As data lakes continue to grow, the need for fast point queries will only intensify. AI agents and real-time applications demand low latency. If we can get that from open formats like Parquet and Iceberg, we might not need separate operational databases at all. That would be a win for cost, consistency, and simplicity.

In the meantime, we can take a cue from the ancient Near East: organize your records so you can find them quickly, and don't be afraid to build an index separate from the data itself. The clay may have hardened, but the principle lives on.