❯ can you look at the code and try to match what parts of the code are causing this?

⏺ Let me dig into the heap data and Venice source code to correlate the waste with specific code paths. I'll run targeted queries on the parquet data and search the codebase in parallel.

⏺ Let me run targeted queries on the heap data to identify the actual types/owners behind the waste, and simultaneously search the Venice codebase for the patterns.

⏺ 2 agents finished (ctrl+o to expand)
   ├─ Agent (Query heap for waste owners) · 30 tool uses · 34.5k tokens
   │  ⎿  Done
   └─ Explore (Search Venice code for waste sources) · 43 tool uses · 124.5k tokens
      ⎿  Done

⏺ Venice Server Heap Waste → Code Correlation

  Heap: venice-server_prod-lva1_3672702 | 94M objects | 6.7 GB HPROF | 1.8 GB estimated waste

  ---
  1. All-Zero float[10000] — 93 MB waste (easiest fix)

  Root cause: Tehuti ExponentiallyDecayingReservoir pre-allocates a float[10000] sample array per sensor, but 2,448 of 3,682 reservoirs (66.5%) are never sampled — the array stays zeroed.

  Code path: Sensors are registered per-store, per-partition, per-region in:
  - HostLevelIngestionStats.java:265-344 — regional sensors (registerSensor(regionNamePrefix + "_rt_bytes_consumed", new Rate())) multiplied across regions × stores
  - ServerHttpRequestStats.java — per-store request stats with ratio sensors

  Fix: Lazy-allocate the reservoir array — only create float[10000] on first update() call.

  ---
  2. Bad Collections — 408 MB waste (biggest category)

  Empty HashMaps (672K, ~86 MB)

  Primary source: LeaderFollowerStoreIngestionTask.java:2932,2964,2982 — passes new HashMap<>() as empty headers to asyncSendControlMessage() on every control message (SOP, EOP, TOPIC_SWITCH) across every partition:
  .asyncSendControlMessage(
      controlMessage,
      versionTopicPartitionToBeProduced,
      new HashMap<>(),    // empty headers — created per control message per partition
      callback,
      leaderMetadataWrapper)

  Fix: Use Collections.emptyMap() — headers are rarely populated.

  Empty/Single ConcurrentHashMaps (272K, ~42 MB)

  Primary source: PartitionConsumptionState.java:363-366 — creates 4 VeniceConcurrentHashMap<>(3) per partition:
  latestConsumedRtPositions = new VeniceConcurrentHashMap<>(3);
  divRtCheckpointPositions = new VeniceConcurrentHashMap<>(3);
  latestProcessedRtPositions = new VeniceConcurrentHashMap<>(3);
  trackingIncrementalPushStatus = new VeniceConcurrentHashMap<>(3);
  Also: transientRecordMap (line 217), cachedHeartbeatKeys (line 317), globalRtDivKeyBytesCache in LeaderFollowerStoreIngestionTask.java:350.

  With ~5K partitions, that's 20K+ CHMs most of which stay empty (batch-only stores don't use RT positions).

  Fix: Lazy-initialize — only create CHMs when first entry is put. For batch-only partitions, these are never populated.

  Empty/Single ArrayLists (2M, ~91 MB)

  Primary source: StoreIngestionTask.java:1526-1541 — batch processing creates new ArrayList<>(batchSize) per batch in a loop:
  List<DefaultPubSubMessage> ongoingBatch = new ArrayList<>(batchSize);
  // ... in loop:
  ongoingBatch = new ArrayList<>(batchSize);  // Line 1541 — new list per batch
  Also: ServerReadMetadataRepository.java creates new lists per metadata request (lines 123, 133, 146, 204, 219, 231).

  ---
  3. Duplicate Strings — 275 MB waste

  Top duplicated values (from 10% sample, ×10 for true count):

  ┌─────────────────────────────────────────────────┬─────────────┬──────────────────────────────────────┐
  │                     String                      │ Est. Copies │                Source                │
  ├─────────────────────────────────────────────────┼─────────────┼──────────────────────────────────────┤
  │ "COMPLETED"                                     │ ~378K       │ Helix partition state per replica    │
  ├─────────────────────────────────────────────────┼─────────────┼──────────────────────────────────────┤
  │ "STANDBY"                                       │ ~248K       │ Helix partition state per replica    │
  ├─────────────────────────────────────────────────┼─────────────┼──────────────────────────────────────┤
  │ "venice-server"                                 │ ~200K       │ App name in metrics/Helix context    │
  ├─────────────────────────────────────────────────┼─────────────┼──────────────────────────────────────┤
  │ "LEADER"                                        │ ~135K       │ Helix partition state per replica    │
  ├─────────────────────────────────────────────────┼─────────────┼──────────────────────────────────────┤
  │ "IngestionStatsGauge"                           │ ~101K       │ Tehuti metric type string per sensor │
  ├─────────────────────────────────────────────────┼─────────────┼──────────────────────────────────────┤
  │ "Gauge"                                         │ ~46K        │ Tehuti metric type string            │
  ├─────────────────────────────────────────────────┼─────────────┼──────────────────────────────────────┤
  │ X.500 cert OIDs ("2.5.4.3", "o=linkedin", etc.) │ ~42K each   │ SSL cert parsing per connection      │
  └─────────────────────────────────────────────────┴─────────────┴──────────────────────────────────────┘

  Code paths:
  - Helix states: Each partition replica stores state as a fresh String. With 2.16M TopicPartition objects, each state transition creates new String instances.
  - Metric names: Tehuti sensor registration creates type strings per sensor (438K sensors × type string).
  - Cert strings: Every SSL handshake parses X.500 subject fields into new Strings.

  Fix: Intern enum-like strings ("COMPLETED", "STANDBY", "LEADER"). Use static constants for metric type strings and cert OIDs.

  ---
  4. Duplicate byte[] — 363 MB waste

  Key patterns from size distribution:

  ┌─────────────┬─────────────────────┬───────────────┬──────────────────────────────────────────────┐
  │ Array Size  │        Dups         │ Unique Values │                Likely Source                 │
  ├─────────────┼─────────────────────┼───────────────┼──────────────────────────────────────────────┤
  │ 1,191 bytes │ 3,776 of same value │ 1             │ Serialized Avro schema repeated per consumer │
  ├─────────────┼─────────────────────┼───────────────┼──────────────────────────────────────────────┤
  │ 32 bytes    │ 43K                 │ 1,918         │ SHA-256 digests of partition data            │
  ├─────────────┼─────────────────────┼───────────────┼──────────────────────────────────────────────┤
  │ 9 bytes     │ 98K                 │ 172           │ Short encoded topic/partition names          │
  ├─────────────┼─────────────────────┼───────────────┼──────────────────────────────────────────────┤
  │ 13 bytes    │ 78K                 │ 654           │ Short string .getBytes() calls               │
  ├─────────────┼─────────────────────┼───────────────┼──────────────────────────────────────────────┤
  │ 8,192 bytes │ 109 of same value   │ 1             │ Buffer template duplicated across consumers  │
  └─────────────┴─────────────────────┴───────────────┴──────────────────────────────────────────────┘

  Code path: PartitionConsumptionState.java:763 creates new PubSubTopicPartitionImpl(topic, getPartition()) which likely serializes topic names to byte[]. ServerReadMetadataRepository.java:308,315 calls
  schemaEntry.getSchema().toString() and .getBytes() per request.

  Fix: Cache serialized schema bytes by schema ID. Deduplicate short byte arrays for topic/partition identifiers.

  ---
  5. TopicPartition/Metadata Explosion — 2.16M objects

  Root cause: 380 MetadataCache instances — one per Kafka consumer. Each independently caches the full cluster metadata:

  380 consumers × ~1,850 partitions = 703K PartitionInfo (exact match with heap data)

  Each PartitionInfo creates a TopicPartition + Node[] array, explaining the 2.16M TopicPartition and 2.11M Node[] objects.

  Contributing code:
  - PubSubNumberedShard: 1.65M instances — wraps partition IDs
  - KafkaOffsetPosition: 1.76M instances — wraps offsets per partition per consumer
  - KafkaNumberedShardMeta: 1.65M instances — partition metadata wrappers

  Fix: Share a single MetadataCache across consumers connected to the same Kafka cluster. This alone would reduce TopicPartition objects from 2.16M to ~6K (one set per cluster, not per consumer).

  ---
  6. Lambda Class Proliferation — 5,819 distinct lambda types

  Primary files:
  - LeaderFollowerStoreIngestionTask.java — 15+ lambda sites in per-message paths (lines 292, 333, 1308, 1576, 1776, 3367, 3620, 4357-4394)
  - ActiveActiveStoreIngestionTask.java — additional lambda sites in AA processing

  Each lambda capturing different variables creates a new synthetic class. Combined with 42K total loaded classes, this pressures metaspace (~6-12 MB).

  ---
  Prioritized Fix List

  ┌──────────┬───────────────────────────────────────────────────────┬─────────┬──────────────────────────────┐
  │ Priority │                          Fix                          │ Savings │          Complexity          │
  ├──────────┼───────────────────────────────────────────────────────┼─────────┼──────────────────────────────┤
  │ P0       │ Lazy-allocate Tehuti reservoir float[10000]           │ 93 MB   │ Low — single class change    │
  ├──────────┼───────────────────────────────────────────────────────┼─────────┼──────────────────────────────┤
  │ P0       │ Use Collections.emptyMap() in asyncSendControlMessage │ ~50 MB  │ Trivial — 3 lines            │
  ├──────────┼───────────────────────────────────────────────────────┼─────────┼──────────────────────────────┤
  │ P1       │ Lazy-init CHMs in PartitionConsumptionState           │ ~42 MB  │ Medium — null checks needed  │
  ├──────────┼───────────────────────────────────────────────────────┼─────────┼──────────────────────────────┤
  │ P1       │ Share MetadataCache across same-cluster consumers     │ ~115 MB │ High — Kafka client refactor │
  ├──────────┼───────────────────────────────────────────────────────┼─────────┼──────────────────────────────┤
  │ P1       │ Intern Helix state strings + metric type strings      │ ~60 MB  │ Low — string constants       │
  ├──────────┼───────────────────────────────────────────────────────┼─────────┼──────────────────────────────┤
  │ P2       │ Cache serialized schema byte arrays                   │ ~50 MB  │ Medium                       │
  ├──────────┼───────────────────────────────────────────────────────┼─────────┼──────────────────────────────┤
  │ P2       │ Lazy-init empty ArrayLists, use singletonList         │ ~90 MB  │ Medium — many callsites      │
  └──────────┴───────────────────────────────────────────────────────┴─────────┴──────────────────────────────┘

  The top two P0 fixes alone would save ~143 MB with minimal code changes.