Confluent Platform is Apache Kafka plus a commercial layer, priced by subscription/CKU that scales with throughput, with premium connectors and features on top. Because Confluent is Kafka at the core, moving to self-managed open-source Apache Kafka keeps the same wire protocol and client APIs, your producers and consumers barely change. The work is replacing Confluent’s value-add components and bridging the data without loss.
Know what Confluent adds before you drop it
The Kafka brokers and client API are open source and identical. What you give up by leaving Confluent Platform:
- Schema Registry, open-source Kafka has the Apache/community registry; the Confluent Schema Registry has its own licensing, so plan an open equivalent (e.g. Apicurio) if you use schemas heavily.
- Pre-built connectors, many Confluent connectors are commercial; you’ll use open Kafka Connect connectors or write your own.
- ksqlDB, Control Center, RBAC, tiered storage, Self-Balancing, these are Confluent extras. Replace with open tooling (Kafka Streams/Flink, CMAK/Redpanda Console for UI, standard ACLs) or do without.
If you mostly use core pub/sub and a few connectors, the migration is light. If you’re deep into ksqlDB and the commercial connector catalog, scope those replacements first.
Both sides are Kafka, so it’s an inventory not a translation
There is no paradigm shift here, both sides are Kafka, so the “mapping” is really an inventory of surface area rather than a translation of semantics. Topics keep their names, partition counts, and retention. Replication factor, cleanup policy (delete vs compact), and per-topic overrides all carry across, so capture the full topic config, not just the topic list. ACLs and any RBAC role bindings from Confluent need to be re-expressed as standard Kafka ACLs, which is where estates that leaned on Confluent RBAC do real work, the model is coarser and you may consolidate roles.
The pieces that are not one-for-one live above the broker. Schema references embedded in messages point at a registry by ID, so the registry migration and the topic migration are linked: a consumer on the new cluster must resolve the same schema ID it sees in the payload. Connectors are configuration plus running tasks; you recreate the config against open Kafka Connect and let the connector rebuild its state from its stored offsets. ksqlDB queries have no open drop-in, so those become Kafka Streams or Flink jobs, treat that as application work scheduled alongside, not part of, the broker cutover.
Keep both clusters live, then drain the lag
The clean path keeps both clusters live during transition:
- Inventory topics, partitions, ACLs, schemas, connectors, and, critically, delivery semantics (ordering, at-least-once vs exactly-once). Baseline throughput and consumer lag.
- Stand up open Kafka in KRaft mode (no ZooKeeper), recreate topics/configs and ACLs, and stand up your chosen registry/UI.
- Bridge with MirrorMaker 2 to replicate topics, consumer offsets, and configs from Confluent to the new cluster, keeping them in sync.
- Repoint producers first, consumers second, then let MirrorMaker drain until consumer lag reaches zero.
- Cut over once lag is drained and clients are confirmed on the new cluster; retire the Confluent cluster after a soak window.
Producers first, then consumers, and why the order matters
Producers move first for a reason. Once a producer writes to the new cluster, MirrorMaker keeps the new topic authoritative and mirrors it back if you need to, but you never have a consumer reading from the new cluster while its only producer still writes to the old one. Move producers topic family by topic family, confirm the new cluster is receiving, then move the matching consumers and watch their lag drain against the mirrored backlog. Keep MirrorMaker running in both directions during this window if any flow is bidirectional, and only stop it once every producer and consumer for a topic is confirmed on the target. A consumer that quietly stayed pointed at Confluent is the classic way to strand messages after you think you have cut over, so reconcile the client inventory before retiring anything.
Sizing and operations
Self-managed Kafka shifts cost from subscription to broker compute + operations, usually far lower at scale, but you now own broker tuning, rebalancing, upgrades, and monitoring. Size on broker cores and peak throughput, plan partition counts and replication factor for your durability needs, and stand up proper monitoring (JMX/Prometheus) before cutover, Confluent’s Control Center was doing more than you might realize.
Proving no message loss at the seam
The acceptance bar is “no message loss, and ordering/delivery semantics preserved under load.” Run a throughput/latency benchmark, delivery-semantics and ordering tests, a consumer-failover and replay test, and a backpressure/soak test. Validate on a non-critical topic first, and keep the bridge running until parity is confirmed.
Pay particular attention to the boundary that MirrorMaker introduces. Cross-cluster mirroring is at-least-once, so a topic that relied on Confluent exactly-once needs an explicit check that consumers on the target deduplicate or tolerate the occasional replayed record at the seam. Confirm offset translation actually lands consumers where you expect by moving a test group across and watching where it resumes. Verify partition ordering is preserved for keyed topics, since a misconfigured mirror can spread a key across partitions and quietly break per-key order. Treat any schema-dependent topic as its own test: produce, mirror, and consume through the new registry so a mismatched schema ID surfaces in staging rather than in production.
What you’re really signing up for
Confluent → Apache Kafka keeps the protocol and clients while dropping the subscription; the work is replacing Confluent’s commercial add-ons (Schema Registry, connectors, ksqlDB, Control Center) and bridging with MirrorMaker 2 so nothing is lost. Repoint producers then consumers, drain lag, and cut over after soak. Model your per-core savings in the calculator above, and treat them as illustrative until you’ve scoped the add-on replacements and operational ownership.