Skip to content
Crypto.world

Research

push0: Scalable and Fault-Tolerant Orchestration for Zero-Knowledge Proof Generation

Generating zero-knowledge proofs is slow and failure-prone, and the software that decides which prover gets which task — the orchestration layer — has been treated as an afterthought.

push0: Scalable and Fault-Tolerant Orchestration for Zero-Knowledge Proof Generation
arXiv
2602.16338
Published
2026-02-18
Authors
Mohsen Ahmadvand, Rok Pajnič, Ching-Lun Chiu

AI summary

Overview

Research area: Distributed systems and blockchain infrastructure — specifically the orchestration (scheduling, routing, fault recovery) of zero-knowledge proof generation for ZK-rollups and Ethereum's L1 zkEVM.

Technical level: Advanced. The paper mixes production systems engineering (message buses, queue topologies, Kubernetes deployment) with formal correctness arguments (theorems on starvation freedom, barrier locality, routing continuity). Readers need familiarity with distributed-systems concepts such as barriers, at-least-once delivery, and partitioning.

Scope (one sentence): The paper presents push0, a cloud-native, prover-agnostic orchestration layer that enforces strict block-sequential ordering while allowing unlimited intra-block parallelism, backed by formal properties and production measurements from the Zircuit zkrollup.

What This Paper Is About

Generating zero-knowledge proofs is slow and failure-prone, and the software that decides which prover gets which task — the orchestration layer — has been treated as an afterthought. Yet a rollup cannot advance (a "chain halt") if a proof stalls, and Ethereum's planned L1 zkEVM needs proofs finished inside the 12-second slot window.

The authors argue that no existing system simultaneously delivers strict head-of-chain ordering, bounded latency, automatic failover, support for arbitrary prover binaries, horizontal scalability, and observability. push0 is their answer: a dispatcher–collector architecture built on persistent priority queues that separates the orchestration logic from the prover binaries, so the same scheduling software can drive heterogeneous zkVMs without code changes.

Key Contributions

  1. Requirements formalization. Six orchestration requirements (head-of-chain ordering, bounded latency, fault tolerance, prover agnosticism, horizontal scalability, observability) distilled from nearly a year of running a production ZK-rollup and from analyzing the Ethereum real-time proving specification. The authors highlight the core tension: ordering constrains parallelism, but real-time proving demands it.

  2. Dispatcher–collector architecture. An event-driven design on persistent priority queues that decouples inter-block ordering (strict) from intra-block parallelism (unlimited). The workflow DAG is implicit in queue topology and collection-strategy configuration rather than stored as an explicit graph object.

  3. Correctness properties. Theorems for starvation freedom (priority ordered by block number, retry count, enqueue time), barrier locality under partition-affine routing, and routing continuity during scale-down via partition takeover plus graceful drain and redelivery. The paper quantifies barrier fragmentation: with C collectors and k inputs per barrier under random assignment, completion probability is C^-(k-1), which for C=4 and k=8 is about 0.006%.

  4. Prover-agnostic integration and production validation. Two integration modes — a JSON executor (--input-path / --output-path, with shell-wrapper gluing for non-conforming CLIs) and a Rust Prover<Input, Output> trait "composer" executor that removes file I/O. The same orchestration software is deployed across three distinct proof systems, and push0 has run on Zircuit since March 4, 2025, processing over 14 million mainnet blocks.

Main Findings

  • Orchestration overhead is negligible. On a production Kubernetes cluster, push0 achieves a 5 ms median orchestration overhead — under 0.1% relative to proof computation times of 7+ seconds. The design target stated in the requirements section is sub-100 ms per task for dispatch, routing, and collection.

  • Scaling efficiency is near-linear. The cluster experiments report 99–100% scaling efficiency at 32 dispatchers for realistic workloads.

  • Controlled Docker runs corroborate the cluster results. With network variance removed, measured latency is 3–10 ms at P50, comparable to the Kubernetes numbers.

  • Naive load balancing is catastrophic for aggregation. Under random collector assignment with C=4 collectors and k=8 subproofs per barrier, the probability that all inputs for a block land on one collector is roughly 0.006%, so incomplete barriers accumulate in memory indefinitely. Partition-affine routing — partition(m) = h(g) mod C — fixes this.

  • Timeouts and heartbeats prevent both false failures and stuck tasks. Dispatchers emit heartbeats every 5 seconds by default to extend acknowledgment deadlines for legitimate multi-minute proofs, while a configurable acknowledgment timeout T_ack, a retry cap R_max, and a dead-letter queue handle genuinely stalled work.

  • Speculative k-of-n dispatch trades compute for tail latency. The same task can be sent to n provers with the first k valid results accepted; redundant tasks expire naturally through the acknowledgment timeout rather than explicit cancellation.

  • Production deployment provides ecological validity. The Zircuit zkrollup deployment (14+ million mainnet blocks since March 2025) is presented as evidence that the controlled experiments reflect real conditions. The paper states that orchestration failures, not prover bugs, account for the majority of production incidents in their ZK infrastructure experience.

  • The message bus can be swapped. Any system offering persistence, worker groups (exclusive delivery), and timeout-based redelivery suffices; NATS JetStream, RabbitMQ, Apache Kafka, and Redis Streams are named as candidates. Production uses NATS JetStream in a 3-node cluster across availability zones.

Methodology in Plain English

The authors first wrote down what a proof pipeline actually needs, drawing on their own production operations and on Ethereum's real-time proving spec. They then designed a system around three pieces: a durable message bus with priority queues, stateless workers called dispatchers that pull a task and run a prover program, and collectors that wait for groups of partial results before emitting the next task.

Ordering is enforced by priority: lower block numbers always come off the queue first, with retry count used to escalate repeatedly failing blocks and enqueue time breaking ties. Fault tolerance comes from a three-phase acknowledgment protocol — a dispatcher only acknowledges a message after the prover succeeds, the result is published, and publication is confirmed — so a crash means the task is redelivered rather than lost. Deduplication by message ID and TaskID keeps aggregation at-most-once despite redelivery.

To keep aggregation correct when several collectors are running, each aggregation group (typically a block number) is mapped by modulo hashing to exactly one collector; when the collector count shrinks, surviving collectors take over the orphaned partitions so that in-flight and new messages for the same group still meet in the same place.

For evaluation, they ran experiments on a 3-node Kubernetes cluster (AWS t3.medium, 2 vCPUs and 4 GB RAM per node) with a cloud-hosted NATS service, and supplementary Docker Compose tests on a single Apple M3 Max with 36 GB RAM. Because real proving is expensive, they used an echo executor that returns immediately or sleeps for a configurable 0–5 seconds, which isolates pure coordination cost. The truncated portion of the text is where the multi-second prover latency analysis begins; the paper's reported headline results (5 ms median overhead, 99–100% efficiency at 32 dispatchers, 3–10 ms P50 in Docker) are stated in the abstract and introduction.

Why This Matters

Impact on research: The paper reframes proof orchestration as a first-class systems problem rather than plumbing. Its formal treatment of barrier fragmentation and partition-affine routing gives other ZK infrastructure work a vocabulary and correctness baseline, and its requirement set is a checklist that later systems can be measured against.

Real-world applications:

  • ZK-rollup operators who need finality to keep pace with transaction demand and cannot afford chain halts from a stuck prover.
  • Ethereum L1 validators and client teams pursuing the 12-second-slot real-time proving target, where P99 latency determines slot compliance.
  • Multi-prover coordination efforts such as Ethproofs, which need strict ordering and failover across independent zkVM implementations from different vendors.
  • Decentralized prover marketplaces, which currently address capacity pricing but not task routing, barrier synchronization, or operator-controlled failover.

Industry relevance: The separation of orchestration from prover binaries means operators can upgrade a zkVM or swap proof systems without rebuilding scheduling infrastructure — a practical requirement in a field where zkVMs (SP1, RISC Zero), circuit-specific provers (Halo2, Plonky2), and proprietary systems coexist and evolve quickly. The message-bus abstraction also lets the same software run on bare metal, Kubernetes, cloud VMs, or hybrid setups.

Future Directions

  • Decentralized, multi-operator orchestration. push0 is validated in a single-operator production setting; the authors position it as usable beneath decentralized multi-prover networks, but ordering and failover guarantees when dispatchers and collectors belong to mutually distrusting parties remain an open problem.

  • Adaptive scaling policies. The paper specifies graceful drain, partition takeover, and routing continuity for scaling events but leaves the decision of when to scale collectors and dispatchers — and how to set T_ack, T_collect, and grace periods — to operators.

  • Scheduling under heterogeneous and dynamic workloads. Block complexity is described as varying by orders of magnitude, and speculative k-of-n redundancy converts compute into latency reduction; how to allocate redundancy and priority across provers with different hardware and cost profiles is not resolved.

  • Formal validation beyond proof sketches. The starvation-freedom, barrier-locality, and routing-continuity results are presented with proof sketches; machine-checked verification of the full protocol, including message redelivery during partition takeover, would strengthen the guarantees.

Target Audience

Infrastructure and platform engineers building proving pipelines for rollups or zkEVMs; distributed-systems researchers interested in ordering-plus-parallelism scheduling problems; and architects at teams evaluating whether to adopt a zkVM or a prover marketplace, who need to understand the scheduling layer that sits underneath either choice. It is less suited to readers seeking cryptographic or circuit-level results — the paper explicitly situates itself away from proof-system cryptography and proof acceleration.

Authors’ abstract

Zero-knowledge proof generation imposes stringent timing and reliability constraints on blockchain systems. For ZK-rollups, delayed proofs cause finality lag and economic loss; for Ethereum's emerging L1 zkEVM, proofs must complete within the 12-second slot window to enable stateless validation. The Ethereum Foundation's Ethproofs initiative coordinates multiple independent zkVMs across proving clusters to achieve real-time block proving, yet no principled orchestration…

The full abstract on arXiv
Read the original paper