Kafka Metadata Bloat — Deep Dive

  The Architecture Problem

  Each Venice server creates a shared consumer pool per Kafka cluster:

  3 Kafka clusters (local + 2 remote for AA replication)
    × 5 pool types (AA-WC-current, AA-WC-non-current, sep-RT, non-AA-current, non-AA-non-current)
    = 15 PartitionWiseKafkaConsumerService instances
    × 11 consumers each
    = 165 SharedKafkaConsumers
    + 273 KafkaProducers (leader replicas producing to VT topics)
    = ~380 NetworkClients, each with its OWN MetadataCache

  The Waste

  Each Kafka consumer/producer sends a MetadataRequest to the broker, and the broker responds with full cluster metadata — every topic, every partition, every replica list. The client caches all of it.

  ┌───────────────────────────────────────┬──────────────────────────────────┐
  │                Metric                 │              Value               │
  ├───────────────────────────────────────┼──────────────────────────────────┤
  │ Total MetadataCache instances         │ 380                              │
  ├───────────────────────────────────────┼──────────────────────────────────┤
  │ Unique topics cached                  │ 1,491                            │
  ├───────────────────────────────────────┼──────────────────────────────────┤
  │ Total PartitionInfo across all caches │ 703,707                          │
  ├───────────────────────────────────────┼──────────────────────────────────┤
  │ Avg partitions per topic              │ 472                              │
  ├───────────────────────────────────────┼──────────────────────────────────┤
  │ Max partitions in single cache        │ 13,982                           │
  ├───────────────────────────────────────┼──────────────────────────────────┤
  │ 70 large caches (5001+ entries)       │ 540,178 entries (76.8% of total) │
  ├───────────────────────────────────────┼──────────────────────────────────┤
  │ 109 empty caches                      │ Unused/idle consumers            │
  └───────────────────────────────────────┴──────────────────────────────────┘

  Size Distribution of MetadataCaches

  ┌───────────────────┬────────┬─────────┬─────────┬────────────┐
  │       Tier        │ Caches │ Entries │ Est. MB │ % of Total │
  ├───────────────────┼────────┼─────────┼─────────┼────────────┤
  │ Empty (0)         │ 109    │ 0       │ 0.0     │ 0%         │
  ├───────────────────┼────────┼─────────┼─────────┼────────────┤
  │ Small (≤500)      │ 127    │ 40,856  │ 9.7     │ 5.8%       │
  ├───────────────────┼────────┼─────────┼─────────┼────────────┤
  │ Medium (501-5000) │ 74     │ 122,673 │ 29.0    │ 17.4%      │
  ├───────────────────┼────────┼─────────┼─────────┼────────────┤
  │ Large (5001+)     │ 70     │ 540,178 │ 127.8   │ 76.8%      │
  └───────────────────┴────────┴─────────┴─────────┴────────────┘

  Full Kafka Metadata Memory Bill: ~582 MB

  ┌─────────────────────────────────────────────────────────────────────────┬───────┬─────────┐
  │                                Component                                │ Count │ Est. MB │
  ├─────────────────────────────────────────────────────────────────────────┼───────┼─────────┤
  │ MetadataCache entries (PartitionInfo+Epoch+HashMap.Node+TopicPartition) │ 703K  │ 171.8   │
  ├─────────────────────────────────────────────────────────────────────────┼───────┼─────────┤
  │ Kafka Node[] arrays (replica/ISR lists)                                 │ 2.1M  │ 80.5    │
  ├─────────────────────────────────────────────────────────────────────────┼───────┼─────────┤
  │ KafkaNumberedShardMeta (xinfra wrapper)                                 │ 1.6M  │ 75.4    │
  ├─────────────────────────────────────────────────────────────────────────┼───────┼─────────┤
  │ KafkaOffsetPosition (xinfra wrapper)                                    │ 1.8M  │ 67.1    │
  ├─────────────────────────────────────────────────────────────────────────┼───────┼─────────┤
  │ TopicPartition objects (all uses)                                       │ 2.2M  │ 65.9    │
  ├─────────────────────────────────────────────────────────────────────────┼───────┼─────────┤
  │ PubSubNumberedShard (xinfra wrapper)                                    │ 1.6M  │ 62.9    │
  ├─────────────────────────────────────────────────────────────────────────┼───────┼─────────┤
  │ PartitionInfo objects                                                   │ 703K  │ 37.6    │
  ├─────────────────────────────────────────────────────────────────────────┼───────┼─────────┤
  │ Kafka MetricName + KafkaMetric                                          │ 278K  │ 21.2    │
  ├─────────────────────────────────────────────────────────────────────────┼───────┼─────────┤
  │ Total                                                                   │       │ ~582 MB │
  └─────────────────────────────────────────────────────────────────────────┴───────┴─────────┘

  Why This Is So Wasteful

  A Venice server hosting 490 ingestion tasks needs metadata for at most ~500-1000 topics (VT and RT topics for its assigned stores). But each consumer/producer caches metadata for all 1,491 topics on the cluster. The 70 largest
  caches hold ~13K partitions each but likely only subscribe to a fraction.

  Estimated waste: 93.5% in the large caches (540K entries cached vs ~35K actually needed).

  What metadata.max.age.ms Does and Doesn't Do

  This Kafka config controls how often the client refreshes its metadata. The default is 300,000ms (5 min). Increasing it would reduce metadata refresh frequency but does NOT reduce cache size — each refresh still fetches full
  cluster metadata.

  So tuning metadata.max.age.ms alone is not sufficient. The real fixes are:

  Actual Solutions (in order of impact)

  1. Topic-filtered MetadataRequests (Kafka 2.8+/KIP-708): Kafka supports sending only the topics the client is subscribed to in the MetadataRequest. If the xinfra PubSub layer doesn't pass topic filters, each client gets back
  everything. This is the biggest lever — each cache would shrink from ~13K entries to only the subscribed topics' partitions.
  2. Reduce consumer pool count: 165 consumers × 380 metadata caches is a lot. The pool sizes are configurable:
    - server.consumer.pool.size.per.kafka.cluster (default 5)
    - server.consumer.pool.size.for.current.version.aa.wc.leader (default 10)
    - Reducing from 11 per service to 5 would cut consumers nearly in half.
  3. Share MetadataCache across consumers in the same pool: All 11 consumers in a pool connect to the same Kafka cluster. They could share a single Metadata updater instead of each maintaining their own. This would require Kafka
  client changes or a shared metadata layer.
  4. Reduce producer count: 273 KafkaProducers (for leader replicas) — each has its own MetadataCache. A shared producer pool per cluster (similar to shared consumers) would significantly reduce producer-side metadata.
  5. PubSub wrapper deduplication: The xinfra PubSub abstraction creates KafkaOffsetPosition, PubSubNumberedShard, and KafkaNumberedShardMeta wrappers around Kafka's native TopicPartition, effectively doubling the metadata object
  count. Reducing wrapper allocation (pooling or flyweight pattern) would save ~205 MB.

  Want me to check the venice-backend config files to see what pool sizes are currently configured for venice-5, or search the codebase for where MetadataRequest topic filtering could be added?