A layered, fully-async .NET platform where each component has a single responsibility — from Raft consensus to memory-mapped segment storage.
Seven discrete components, zero external dependencies. Each layer exposes clean interfaces used by the layers above.
IonStream ships with a full Raft implementation. No external coordinators. Leader election, log replication, and quorum management happen entirely inside the cluster.
Nodes use randomized election timeouts to avoid split votes. Term numbers ensure stale candidates cannot win.
The leader appends entries and fans them out via AppendEntries RPC. Followers acknowledge; the leader advances the commit index once a quorum responds.
New nodes join via PROXY_MANAGERS. Even-count additions join as read-only mirrors; odd-count triggers automatic quorum promotion.
IonStream's core storage uses memory-mapped file segments for zero-copy reads and kernel-efficient writes. Each topic-partition maintains its own sequence of log segment files paired with sparse offset indexes for fast seek-by-offset.
Every message gets a time-ordered GUID v7. Timestamp extraction is free — no extra field needed for time-based seeks.
Segments roll at a configurable byte threshold. Time-based retention deletes sealed segments automatically. Log compaction merges duplicate keys.
In-Sync Replica tracking with configurable min.insync.replicas. Acks.All produce requests block until the required quorum acknowledges.
End-to-end message lifecycle — produce, replicate, consume — with ISR acknowledgment and consumer group coordination.
The broker node runtime. Exposes gRPC APIs, coordinates produce/consume, manages consumer groups, routes to partition leaders, and assembles all server components.
Full Raft consensus implementation. Handles leader election via RequestVote, log replication via AppendEntries, and dynamic cluster join. Exposes its own gRPC service.
The storage engine. Memory-mapped log segments, sparse offset indexes, segment rolling, time-based retention, log compaction, and ISR replica state tracking.
Separate web-facing proxy and admin dashboard. Uses YARP with a PartitionAwareLoadBalancingPolicy to route to partition leaders and serve follower reads.
Shared TLS utilities used by server, client, and broker. Supports deterministic CA derivation from a shared secret, or externally provided PKCS#12 certificate files.
The shared contract layer. Defines StreamMessage, IProducer, IConsumer, protocol enums, and interceptor interfaces. No runtime dependencies.
The full .NET SDK. Connection management, metadata refresh, leader-aware routing, connection pooling, Channel<T> APIs, and consumer group protocol.
Official clients for Go (REST), TypeScript/JavaScript (gRPC), Python (async gRPC), Rust (async gRPC), and Java (gRPC). All MIT-licensed.
| Feature | Kafka | Redpanda | NATS JetStream | Pulsar | IonStream |
|---|---|---|---|---|---|
| Runtime | JVM | C++ | Go | JVM | .NET 10 AOT |
| External Deps | ZooKeeper (legacy) | ✓ None | ✓ None | ZooKeeper + BookKeeper | ✓ None |
| Wire Protocol | Custom TCP | Kafka Protocol | NATS Protocol | Custom Binary | gRPC / HTTP/2 |
| Consensus | KRaft / Raft | Raft | Raft | ZooKeeper | Raft |
| Log Compaction | ✓ | ✓ | ✓ | ✓ | ✓ |
| Log Retention | ✓ | ✓ | ✓ | ✓ | ✓ |
| ISR Enforcement | ✓ | ✓ | N/A | ✓ | ✓ |
| Follower Reads | ✓ | ✓ | ✓ | ✓ | ✓ |
| Native .NET Client | Confluent SDK | Kafka-compat | ✓ | ✓ | ✓ Optimized |
| Channels API (.NET) | ✗ | ✗ | ✗ | ✗ | ✓ |
| Browser / gRPC-Web | ✗ | ✗ | Partial | ✗ | ✓ REST + gRPC-Web |
| TLS | ✓ | ✓ | ✓ | ✓ | ✓ TLS 1.3 |
| OpenTelemetry / Prometheus | Via plugin | ✓ | ✓ | ✓ | ✓ Built-in |
| Admin UI | External | ✓ Built-in | ✓ Built-in | Pulsar Manager | ✓ Included |
| Grafana Dashboards | Community | ✓ | Community | Community | ✓ Included |
| Cross-cluster Replication | MirrorMaker 2 | Replication | ✓ | ✓ | ✓ Built-in |
| Native AOT Binary | ✗ | N/A | ✓ Go binary | ✗ | ✓ win/linux |
| Retry Queues & DLQ | External plugin | ✗ | Partial (JetStream) | Partial | ✓ Built-in |
What each client SDK implements today. Preview clients ship as generated gRPC stubs — usable, but without the idiomatic helpers, retries, and high-level abstractions of the full SDKs.
| Feature | .NET | Go | TypeScript | JavaScript | Python | Rust | Java | Browser |
|---|---|---|---|---|---|---|---|---|
| Produce | ✓ | ✓ | ✓ | ✓ | Raw gRPC | Raw gRPC | Raw gRPC | ✓ |
| Consume | ✓ | ✓ | ✓ | ✓ | Raw gRPC | Raw gRPC | Raw gRPC | ✓ |
| Ack / Nack | ✓ | ✓ | ✓ | ✓ | Manual | Manual | Manual | ✓ |
| Consumer Groups | ✓ | ✓ | ✓ | ✓ | Manual | Manual | Manual | ✓ |
| Idempotent Producer | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ |
| Channels / Async Streaming API | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ |
| Auto-Reconnect | ✓ | ✓ | Basic | Basic | ✗ | ✗ | ✗ | Basic |
| TLS / mTLS | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | HTTPS only |
| Admin / Metadata | ✓ | REST | ✓ | ✓ | Raw gRPC | Raw gRPC | Raw gRPC | ✓ |
| Seek (offset / timestamp) | ✓ | ✗ | ✓ | ✓ | Raw gRPC | Raw gRPC | Raw gRPC | ✗ |
| DLQ Operations | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✓ Dashboard |
| Retry Config | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ |
| License | MIT | MIT | MIT | MIT | MIT | MIT | MIT | MIT |
Raw gRPC = preview clients expose generated protobuf stubs; the operation works but requires manual wiring. Idiomatic helpers land as each client matures.