# MultiChain Blockchain Data Provenance for Deterministic Stream Processing with Kafka Streams: A Weather Data Case Study

Crypto.world

> Stream processing engines such as Kafka Streams produce results that depend on runtime scheduling, window triggers, arrival order, and network jitter, so re-running the same pipeline can yield…

**arXiv**: 2601.18011

**Published**: 2026-01-25

**Authors**: Niaz Mohammad Ramaki, Florian Schintke

## AI summary

## Overview

**Research area:** Blockchain-backed data provenance and auditability for real-time stream processing systems (Kafka Streams + MultiChain), demonstrated on live weather-station data.

**Technical level:** Intermediate. Readers benefit from basic familiarity with stream processing concepts (topics, windows, offsets), hash functions, and Merkle trees, but the paper explains each building block.

**Scope (one sentence):** The paper designs and evaluates a scheme that anchors cryptographic commitments of each Kafka Streams window — Merkle root, record count, Kafka offset boundaries, and a payload hash — into MultiChain blockchain streams while keeping the actual window payloads off-chain.

## What This Paper Is About

Stream processing engines such as Kafka Streams produce results that depend on runtime scheduling, window triggers, arrival order, and network jitter, so re-running the same pipeline can yield different outputs and leaves no traceable evidence for an auditor. The authors' running example is two Berlin weather stations feeding a pipeline that computes a 5-minute temperature average: if a record arrives late on a second run, it lands in a different window and the result changes. The goal is to make each window's result independently verifiable — an auditor should be able to check that the payload is complete, was serialized canonically, and that derived statistics like min/max/average temperature are correct — without storing the data itself on the blockchain.

## Key Contributions

1. **A blockchain-anchored provenance architecture for windowed streams** that publishes only cryptographic checkpoints (Merkle root, payload SHA-256, record count, Kafka `offsetStart`/`offsetEnd`, window boundaries, window ID, off-chain payload path) to MultiChain streams, leaving window payloads off-chain.
2. **Deterministic windowing and canonical serialization** — epoch-aligned tumbling windows with index `w = floor(t/Δ)` and interval `I_w = [wΔ, (w+1)Δ)`, combined with canonical JSON (fields pruned, timestamps normalized to UTC ISO-8601 at second precision, keys lexicographically sorted), deduplication, and lexicographically sorted SHA-256 Merkle leaves, so window membership and hashes are independent of ingestion order.
3. **A concrete verification procedure** in which an auditor loads the off-chain payload, re-canonicalizes every record, recomputes the record count `N̂` and Merkle root `R̂`, and accepts the window only if `R̂ = R` and `N̂ = N` against the on-chain checkpoint.
4. **An empirical evaluation on real weather data** from the Berlin-Brandenburg and Berlin-Tempelhof stations (via the German Meteorological Service / BrightSky API), covering verification cost, cross-machine reproducibility, and MultiChain transaction throughput under varying payload sizes and publisher counts.

## Main Findings

- **Linear verification cost:** End-to-end verification time is modelled as `T(n) = αn + β`, where `n` is record count, `α` captures hashing cost per record and `β` fixed I/O overhead; observed `α` falls within linear bounds, consistent with a tree of depth `O(log n)` hashed in batches. Specific latency values are shown only in plots (Figure 8) and are not reported numerically in the text.
- **Stable, predictable verification latency:** The distribution of end-to-end verification latency across all evaluated windows is described as narrow and compact with no long-tail outliers; verification throughput (`recordCount / verificationTime`) stays high and does not collapse as window size grows.
- **Byte-identical reproducibility across environments:** Independent ingestion runs on MacOS M2 with Java 21 and MacOS M4 with Java 21 produced byte-identical canonical JSON, payload SHA-256, and Merkle roots, supporting the claim that canonicalization removes environment-induced nondeterminism.
- **Low real-world on-chain load:** Observed on-chain throughput across the five blockchain streams, aggregated over 5-minute windows, was 0–0.015 items/s, reflecting infrequent checkpointing relative to raw event rates; average TPS per stream was correspondingly low in normal operation.
- **Higher throughput under synthetic benchmarks:** Publishing payloads of 128 Bytes, 512 Bytes, and 2048 Bytes with varying numbers of workers reached up to approximately 2.1 tx/s for smaller payloads, with larger payloads reducing throughput due to serialization and validation overhead.
- **Consensus-bound saturation:** All publisher configurations exhibit a common saturation region beyond eight workers, and increasing worker count with a fixed 350-Byte payload raised API throughput while TPS stayed stable — attributed to MultiChain's round-robin consensus.
- **Merkle trees preferred over hash chains:** The authors argue hash chains bind provenance to ingestion order, cannot be replayed under re-keyed windows, and incur linear verification cost plus growing storage, whereas Merkle roots give logarithmic verification cost and compact commitments.

## Methodology in Plain English

The team built a three-part pipeline. First, a Kafka source connector pulls live weather records from two Berlin stations into separate Kafka topics. Second, Kafka Streams merges the streams and computes per-station and city-wide minimum, maximum, and average temperatures over fixed, epoch-aligned tumbling windows — chosen because their boundaries are absolute and global rather than relative to when the system started. A "Checkpointer" component buffers each incoming record as it passes.

When a window closes, the buffered records are cleaned up: duplicates removed, each record converted to canonical JSON, and the result written off-chain to append-only NDJSON files (chosen for memory efficiency, stream-friendliness, parallel splitting, and one-object-per-line parsing). Each canonical record is hashed with SHA-256 to form a leaf, leaves are sorted lexicographically, and pairs are hashed together up the tree (duplicating the last leaf when a level has an odd count) to produce the window's Merkle root.

That root, together with the record count, payload path, payload SHA-256, Kafka offset range, window start/end, and window ID, is written as a checkpoint into a per-station MultiChain stream; the same checkpoint is also logged off-chain annotated with its blockchain transaction ID. To audit, someone supplies a window ID, fetches the checkpoint from the blockchain, reloads the off-chain payload, re-canonicalizes and re-hashes it, rebuilds the Merkle root, and compares both root and count against the on-chain values — a check that does not require access to Kafka or the original processing cluster.

## Why This Matters

The work targets a gap the authors identify in the literature: substantial research exists on data provenance generally, but little on verifiable provenance for data *streams*, and existing stream checkpointing work (e.g. coordinated checkpointing for exactly-once recovery) provides no cryptographic evidence that a third party can check.

**Impact on research:** It offers a reusable pattern — commit windowed payloads with Merkle roots on a permissioned chain, keep bulk data off-chain — that other streaming-provenance efforts can benchmark against, and it frames auditability formally as `audit(y) = (X_{k,w}(F,C), ρ, α)` with `α` anchored in an immutable medium.

**Real-world applications:**

- Auditing published weather and environmental measurements, so downstream consumers can confirm that reported aggregates match the raw station records.
- Compliance and regulatory reporting over IoT sensor fleets, where an external regulator needs proof that a metric was computed from a specific, complete set of readings.
- Multi-organizational data sharing where parties do not trust each other's processing clusters and need an independent verification path.
- Tamper-evident telemetry pipelines in industrial or infrastructure monitoring, where derived alerts and statistics must be traceable to their inputs.

**Industry relevance:** The design uses off-the-shelf components — Kafka Streams, a JSON-RPC-accessible permissioned blockchain, and commodity JVM deployments — and the reported cost profile (linear verification, negligible overhead, checkpoint rates far below event rates) suggests provenance can be added to existing pipelines without the blockchain becoming the bottleneck, given MultiChain's observed saturation beyond eight concurrent publishers.

## Future Directions

- **Smart contracts or chaincodes on the blockchain side** to add programmability and finer-grained, step-by-step provenance tracking than a single per-window Merkle root provides.
- **A semantic layer (the authors name RDF)** to describe how results are derived in complex pipelines such as window joins, where a Merkle root alone proves integrity but not meaning.
- **Zero-knowledge proofs** so that auditors can verify windows without the payloads being exposed to them, motivated by multi-party verification and confidentiality requirements.
- **Scaling beyond the consensus ceiling:** the observed saturation past eight publishers and the roughly 2.1 tx/s ceiling for small payloads raise the open question of how the anchoring layer behaves under much higher checkpoint rates or larger payloads, which the paper leaves unresolved.

## Target Audience

Graduate students and researchers in distributed systems, stream processing, and blockchain security; data engineers and platform architects who need auditability or reproducibility guarantees for production Kafka pipelines; and compliance or data-governance practitioners evaluating whether cryptographic anchoring can substitute for, or supplement, conventional logging and replay-based audits. Readers chiefly interested in permissionless-chain or smart-contract-heavy designs will find the paper deliberately minimal on that side.

## Authors’ abstract

Auditability and reproducibility still are critical challenges for real-time data streams pipelines. Streaming engines are highly dependent on runtime scheduling, window triggers, arrival orders, and uncertainties such as network jitters. These all derive the streaming pipeline platforms to throw non-determinist outputs. In this work, we introduce a blockchain-backed provenance architecture for streaming platform (e.g Kafka Streams) the publishes cryptographic data of a…

[The full abstract on arXiv](https://arxiv.org/abs/2601.18011)

[Read the original paper](https://arxiv.org/abs/2601.18011)

---

Canonical: https://crypto.world/paper/multichain-blockchain-data-provenance-for-deterministic-stream-2601.18011
