Opsmas 2025 Day 8: kraft & wtf is wrong with anthropic

Opsmas 2025 Day 8: kraft & wtf is wrong with anthropic

TOC:

kraft

kraft is my take on a self-contained and easy to verify raft implementation. By “raft implementation” I mean it treats raft purely as an abstract state machine for all operations (i.e. no networking or storage) so you can verify it logically and generate in-process synthetic test node data structures for rapid evaluation and smashing error conditions against. I started it maybe ten years ago as a new “cleanroom” raft implementation with the intention to integrate it into more live systems, but then other things happened and I abandoned it for a while, but picked it up again this year for “polish and public release” finally.

kraft implements the core raft state machine with optional callbacks you can register to get notified when raft state changes.

Basically, kraft is a system for modeling the raft state machine internally, but it doesn’t define any storage format or network protocol itself. It’s “bring your own client/server” for implementing an inter-server RAFT protocol language, your local guaranteed storage mechanism, and then any other client-side facing networking for interacting with user data. This is also where loopy and kvidxkit come into play because these were all designed to work together (specifically: kvidxkit is built around a (key, term, value) format for indexing and operatig on the concept of “Raft Terms” directly for data layout/storage/flushing/retrieval in a clean isolated and ideally correct way).

You can do fun things like run tests with almost any cluster sizes possible and they instantly launch (because the in-code clusters are just indexed structs tracking their own state against all other nodes (also just structs) sending and receiving internal updates). You want to test a RAFT cluster of 3 nodes? 27 nodes? 277 nodes? It’s all internal data structures at the fundamental testing level, so there’s zero network/setup/coordination problems when defining arbitrary testing or correctness checking conditions.

Some unique or complete features of kraft:

or in bigger format

Core Raft Protocol

  • Complete Raft Implementation: Leader election, log replication, safety
  • Pre-Vote Protocol: Prevents disruption from partitioned nodes rejoining
  • Leadership Transfer: Graceful leader handoff for maintenance operations
  • ReadIndex: Linearizable reads without log replication overhead
  • Log Compaction: Snapshot support with configurable thresholds
  • Membership Changes: Joint consensus for safe cluster reconfiguration

Advanced Features

  • Witness Nodes: Non-voting read replicas for scaling reads
    • Learner/observer mode for catching up without affecting elections
    • Promotion to full voting member when ready
    • Read serving for stale-tolerant queries
  • Async Commit Modes: Configurable consistency/latency tradeoff
    • STRONG: Wait for majority acknowledgment (default Raft)
    • LEADER_ONLY: Acknowledge after leader persistence (lowest latency)
    • ASYNC: Fire-and-forget with background replication
    • FLEXIBLE: Per-command consistency level selection
  • Intelligent Snapshots: Advanced log compaction
    • Automatic triggering based on log size, entry count, or time interval
    • Incremental snapshot support (delta from previous)
    • Streaming chunk-based transfer to followers
    • Parallel transfers to multiple followers
    • Configurable retention and compaction policies

also more weirdness like federated/multi-raft:

In a single-group Raft cluster, all data is replicated across all nodes using one Raft consensus instance. This works well for small datasets but becomes a bottleneck as data grows because:

  1. All writes go through one leader - single point of throughput limitation
  2. All data on all nodes - storage limited to smallest node
  3. Log grows unboundedly - snapshot/compaction for entire dataset

Multi-Raft solves this by partitioning data across multiple independent Raft consensus groups, each responsible for a subset of the data (typically a key range).

Single-Raft:                    Multi-Raft:
┌─────────────────────┐        ┌──────────┐ ┌──────────┐ ┌──────────┐
│  All Data (1 Group) │        │ Range A  │ │ Range B  │ │ Range C  │
│  Leader: Node 1     │   →    │ Group 1  │ │ Group 2  │ │ Group 3  │
│  Nodes: 1,2,3,4,5   │        │ L:N1     │ │ L:N2     │ │ L:N3     │
└─────────────────────┘        └──────────┘ └──────────┘ └──────────┘

Each group:

  • Has its own leader election
  • Maintains its own Raft log
  • Can be on different nodes (leadership spread across cluster)
  • Operates completely independently from other groups

tests

===============================================================
  Kraft Linearizability Test
===============================================================

Configuration:
  Nodes:      3
  Clients:    5
  Operations: 1000
  Max time:   30.0 seconds
  Failure:    none
  Seed:       1766107674

Test data: test-latest
Starting cluster...
Using kraft-loopy binary: loopy-demo/kraft-loopy
Resolved to: /Users/matt/repos/base/kraft/build/src/loopy-demo/kraft-loopy
All nodes forked. Waiting for client API ports...
[✓] Cluster ready (leader: node 1)

Starting 5 clients...
[✓] Clients started

Running workload...
Progress: 0.0% (0/1000 ops, 0.0s elapsed)   
[!] Max time reached


Stopping clients...
[✓] Clients stopped (58 operations completed)

Stopping cluster...
[✓] Cluster stopped

Verifying linearizability...

===============================================================
  RESULT: PASS
===============================================================

=== LINEARIZABILITY: PASS ===
All operations are linearizable.

Verification statistics:
  Time:          108 us
  Graph edges:   514
  States explored: 58

Statistics:
  Invocations:  58
  Completions:  23
  Timeouts:     35
  Success rate: 39.66%
$ ./kraft-fuzz -n 3

=======================================================
   Kraft Raft Consensus - Fuzzer                     
=======================================================

[FUZZ] Starting fuzzer with seed 1766105853
[FUZZ] Config: 3 nodes, 10000 ops, 60000000 us max time
[ELECTION] Node runId=1766105853968093: votes=1, need=2, cluster=3
[PROMOTE] Node 2416951005 becoming LEADER in term 1 with 2/2 votes (cluster=3)
[ELECTION] Node runId=1766105853968093: votes=1, need=2, cluster=3
[ELECTION] Node runId=1766105853967681: votes=1, need=2, cluster=3
[ELECTION] Node runId=1766105853968093: votes=1, need=2, cluster=3
[PROMOTE] Node 2416951005 becoming LEADER in term 4 with 2/2 votes (cluster=3)
[ELECTION] Node runId=1766105853967681: votes=1, need=2, cluster=3
[ELECTION] Node runId=1766105853967681: votes=1, need=2, cluster=3
[PROMOTE] Node 2416950593 becoming LEADER in term 6 with 2/2 votes (cluster=3)
[ELECTION] Node runId=1766105853967681: votes=1, need=2, cluster=3
[PROMOTE] Node 2416950593 becoming LEADER in term 7 with 2/2 votes (cluster=3)
[FUZZ] Operations: 1000 | Commands: 67 | Crashes: 21 | Recoveries: 21 | Elections: 43
       Partitions: 16 | Heals: 16 | Messages: 167/42 | Invariants: 10/10
[ELECTION] Node runId=1766105853968093: votes=1, need=2, cluster=3
[PROMOTE] Node 2416951005 becoming LEADER in term 8 with 2/2 votes (cluster=3)
[ELECTION] Node runId=1766105853968093: votes=1, need=2, cluster=3
[PROMOTE] Node 2416951005 becoming LEADER in term 9 with 2/2 votes (cluster=3)
[ELECTION] Node runId=1766105853967681: votes=1, need=2, cluster=3
[PROMOTE] Node 2416950593 becoming LEADER in term 10 with 2/2 votes (cluster=3)
[ELECTION] Node runId=1766105853968093: votes=1, need=2, cluster=3
[FUZZ] Operations: 2000 | Commands: 157 | Crashes: 43 | Recoveries: 42 | Elections: 78
       Partitions: 35 | Heals: 35 | Messages: 293/72 | Invariants: 20/20
[ELECTION] Node runId=1766105853968093: votes=1, need=2, cluster=3
[PROMOTE] Node 2416951005 becoming LEADER in term 12 with 2/2 votes (cluster=3)
[ELECTION] Node runId=1766105853968093: votes=1, need=2, cluster=3
[PROMOTE] Node 2416951005 becoming LEADER in term 13 with 2/2 votes (cluster=3)
[ELECTION] Node runId=1766105853967681: votes=1, need=2, cluster=3
[FUZZ] Operations: 3000 | Commands: 196 | Crashes: 69 | Recoveries: 67 | Elections: 116
       Partitions: 50 | Heals: 50 | Messages: 348/115 | Invariants: 30/30
[ELECTION] Node runId=1766105853967681: votes=1, need=2, cluster=3
[PROMOTE] Node 2416950593 becoming LEADER in term 15 with 2/2 votes (cluster=3)
[ELECTION] Node runId=1766105853968093: votes=1, need=2, cluster=3
[PROMOTE] Node 2416951005 becoming LEADER in term 16 with 2/2 votes (cluster=3)
[ELECTION] Node runId=1766105853968093: votes=1, need=2, cluster=3
[PROMOTE] Node 2416951005 becoming LEADER in term 17 with 2/2 votes (cluster=3)
[ELECTION] Node runId=1766105853968093: votes=1, need=2, cluster=3
[PROMOTE] Node 2416951005 becoming LEADER in term 18 with 2/2 votes (cluster=3)
[FUZZ] Operations: 4000 | Commands: 339 | Crashes: 91 | Recoveries: 90 | Elections: 146
       Partitions: 65 | Heals: 65 | Messages: 497/155 | Invariants: 40/40
[ELECTION] Node runId=1766105853968093: votes=1, need=2, cluster=3
[PROMOTE] Node 2416951005 becoming LEADER in term 19 with 2/2 votes (cluster=3)
[ELECTION] Node runId=1766105853967681: votes=1, need=2, cluster=3
[ELECTION] Node runId=1766105853967681: votes=1, need=2, cluster=3
[ELECTION] Node runId=1766105853968093: votes=1, need=2, cluster=3
[PROMOTE] Node 2416951005 becoming LEADER in term 22 with 2/2 votes (cluster=3)
[FUZZ] Operations: 5000 | Commands: 363 | Crashes: 115 | Recoveries: 114 | Elections: 192
       Partitions: 85 | Heals: 85 | Messages: 545/170 | Invariants: 50/50
[ELECTION] Node runId=1766105853967681: votes=1, need=2, cluster=3
[PROMOTE] Node 2416950593 becoming LEADER in term 23 with 2/2 votes (cluster=3)
[ELECTION] Node runId=1766105853968093: votes=1, need=2, cluster=3
[PROMOTE] Node 2416951005 becoming LEADER in term 24 with 2/2 votes (cluster=3)
[ELECTION] Node runId=1766105853968093: votes=1, need=2, cluster=3
[PROMOTE] Node 2416951005 becoming LEADER in term 25 with 2/2 votes (cluster=3)
[ELECTION] Node runId=1766105853967681: votes=1, need=2, cluster=3
[PROMOTE] Node 2416950593 becoming LEADER in term 26 with 2/2 votes (cluster=3)
[ELECTION] Node runId=1766105853968093: votes=1, need=2, cluster=3
[PROMOTE] Node 2416951005 becoming LEADER in term 27 with 2/2 votes (cluster=3)
[FUZZ] Operations: 6000 | Commands: 479 | Crashes: 137 | Recoveries: 135 | Elections: 222
       Partitions: 99 | Heals: 98 | Messages: 661/198 | Invariants: 60/60

[FUZZ] Fuzzing complete
[FUZZ] Operations: 6000 | Commands: 479 | Crashes: 137 | Recoveries: 135 | Elections: 222
       Partitions: 99 | Heals: 98 | Messages: 661/198 | Invariants: 61/61

=======================================================
   Fuzzing Result: PASSED (ran to time limit without issues)
   Seed: 1766105853  (reproduce with: -s 1766105853 -n 3)
=======================================================
=======================================================
   Kraft Raft Consensus - Test Suite                  
=======================================================

[TEST 1] Single node cluster (trivial case)...
  Starting election process...
  Node 0 role: 2
  Node 0 term: 0
  Election timeout: 214627
[ELECTION] Node runId=1766105923673625: votes=1, need=1, cluster=1
[ELECTION] PROMOTING TO LEADER: votes=1 >= needed=1
[PROMOTE] Node 2486656537 becoming LEADER in term 1 with 1/1 votes (cluster=1)
  After election request - role: 3, term: 1
  Leader: 0
  Single node correctly elected itself
  ✓ PASSED

[TEST 2] Initial leader election - 3 nodes...
[ELECTION] Node runId=1766105923692557: votes=1, need=2, cluster=3
[PROMOTE] Node 2486675469 becoming LEADER in term 1 with 2/2 votes (cluster=3)
  Leader: Node 0, Term: 1, Elections: 1
  ✓ PASSED

[TEST 3] Initial leader election - 5 nodes...
[ELECTION] Node runId=1766105923709564: votes=1, need=3, cluster=5
[PROMOTE] Node 2486692476 becoming LEADER in term 1 with 3/3 votes (cluster=5)
  5-node cluster leader: Node 3
  ✓ PASSED

[TEST 4] Election with one node failure...
[ELECTION] Node runId=1766105923724682: votes=1, need=3, cluster=5
[PROMOTE] Node 2486707594 becoming LEADER in term 1 with 3/3 votes (cluster=5)
  Leader despite node 2 crash: Node 3
  ✓ PASSED

[TEST 5] Re-election after leader crashes...
[ELECTION] Node runId=1766105923739664: votes=1, need=3, cluster=5
[PROMOTE] Node 2486722576 becoming LEADER in term 1 with 3/3 votes (cluster=5)
  Initial leader: Node 2
[ELECTION] Node runId=1766105923739753: votes=1, need=3, cluster=5
[PROMOTE] Node 2486722665 becoming LEADER in term 2 with 3/3 votes (cluster=5)
  New leader after crash: Node 3
  ✓ PASSED

[TEST 6] Election Safety - at most one leader per term...
[ELECTION] Node runId=1766105923756415: votes=1, need=3, cluster=5
[PROMOTE] Node 2486739327 becoming LEADER in term 1 with 3/3 votes (cluster=5)
[ELECTION] Node runId=1766105923756495: votes=1, need=3, cluster=5
[PROMOTE] Node 2486739407 becoming LEADER in term 2 with 3/3 votes (cluster=5)
[ELECTION] Node runId=1766105923756084: votes=1, need=3, cluster=5
[PROMOTE] Node 2486738996 becoming LEADER in term 3 with 3/3 votes (cluster=5)
[ELECTION] Node runId=1766105923756569: votes=1, need=3, cluster=5
[PROMOTE] Node 2486739481 becoming LEADER in term 4 with 3/3 votes (cluster=5)
[ELECTION] Node runId=1766105923756318: votes=1, need=3, cluster=5
[PROMOTE] Node 2486739230 becoming LEADER in term 5 with 3/3 votes (cluster=5)
[ELECTION] Node runId=1766105923756415: votes=1, need=3, cluster=5
[PROMOTE] Node 2486739327 becoming LEADER in term 6 with 3/3 votes (cluster=5)
  Election safety maintained across 7 elections
  ✓ PASSED

[TEST 7] Log Matching Property...
[ELECTION] Node runId=1766105923772368: votes=1, need=3, cluster=5
[PROMOTE] Node 2486755280 becoming LEADER in term 1 with 3/3 votes (cluster=5)
[LEADER-COMMIT] leader=1 term=1 committing idx=1 (remotelyLogged=2 +1leader = 3, need=3)
[LEADER-COMMIT] leader=1 term=1 committing idx=2 (remotelyLogged=2 +1leader = 3, need=3)
[LEADER-COMMIT] leader=1 term=1 committing idx=3 (remotelyLogged=2 +1leader = 3, need=3)
[LEADER-COMMIT] leader=1 term=1 committing idx=4 (remotelyLogged=2 +1leader = 3, need=3)
[LEADER-COMMIT] leader=1 term=1 committing idx=5 (remotelyLogged=2 +1leader = 3, need=3)
[LEADER-COMMIT] leader=1 term=1 committing idx=6 (remotelyLogged=2 +1leader = 3, need=3)
[LEADER-COMMIT] leader=1 term=1 committing idx=7 (remotelyLogged=2 +1leader = 3, need=3)
[LEADER-COMMIT] leader=1 term=1 committing idx=8 (remotelyLogged=2 +1leader = 3, need=3)
[LEADER-COMMIT] leader=1 term=1 committing idx=9 (remotelyLogged=2 +1leader = 3, need=3)
[LEADER-COMMIT] leader=1 term=1 committing idx=10 (remotelyLogged=2 +1leader = 3, need=3)
  Log matching verified across all nodes
  ✓ PASSED

[TEST 8] Cluster survives minority failures...
[ELECTION] Node runId=1766105923794564: votes=1, need=3, cluster=5
[PROMOTE] Node 2486777476 becoming LEADER in term 1 with 3/3 votes (cluster=5)
  Cluster operational with 2/5 nodes crashed
  ✓ PASSED

[TEST 9] Network partition - split brain prevention...
[ELECTION] Node runId=1766105923812727: votes=1, need=3, cluster=5
[PROMOTE] Node 2486795639 becoming LEADER in term 1 with 3/3 votes (cluster=5)
  Split brain prevented, cluster healed successfully
  ✓ PASSED

[TEST 10] Packet loss tolerance (10%)...
[ELECTION] Node runId=1766105923826818: votes=1, need=3, cluster=5
[PROMOTE] Node 2486809730 becoming LEADER in term 1 with 3/3 votes (cluster=5)
  Leader elected with 10% loss (dropped: 4/18)
  ✓ PASSED

[TEST 11] Leadership transfer (zero-downtime handoff)...
  Waiting for initial leader election...
[ELECTION] Node runId=1766105923843574: votes=1, need=2, cluster=3
[PROMOTE] Node 2486826486 becoming LEADER in term 1 with 2/2 votes (cluster=3)
  Initial leader: Node 2
  Found target node for transfer: clusterId=1, nodeId=0
  Initiating leadership transfer...
  Transfer initiated successfully
  Waiting for transfer to complete...
[ELECTION] Node runId=1766105923843178: votes=1, need=2, cluster=3
[PROMOTE] Node 2486826090 becoming LEADER in term 2 with 2/2 votes (cluster=3)
  New leader after transfer: Node 0
  Leadership successfully transferred from Node 2 to Node 0
  ✓ PASSED

[TEST 12] ReadIndex (linearizable reads without log writes)...
  Waiting for leader election...
[ELECTION] Node runId=1766105923859806: votes=1, need=2, cluster=3
[PROMOTE] Node 2486842718 becoming LEADER in term 1 with 2/2 votes (cluster=3)
  Leader: Node 2
  ReadIndex enabled
  Testing ReadIndex API...
  ReadIndex request returned: 0 (commitIndex=0)
  Initial ready state: no (expected: no - quorum not yet confirmed)
  After heartbeats ready state: yes
  ReadIndex completed successfully
  ReadIndex on follower returned: -1 (expected: negative error)
  ReadIndex API test completed successfully
  ✓ PASSED

[TEST 13] Witness Manager - Lifecycle (init, add, remove, free)...
  Witness lifecycle tests passed
  ✓ PASSED

[TEST 14] Witness Manager - Replication Tracking...
  Replication tracking tests passed
  ✓ PASSED

[TEST 15] Witness Manager - Read Serving...
  Read serving tests passed
  ✓ PASSED

[TEST 16] Witness Manager - Promotion to Voter...
  Promotion tests passed
  ✓ PASSED

[TEST 17] Witness Manager - Vote Prevention...
  Vote prevention tests passed
  ✓ PASSED

[TEST 18] Async Commit - Configuration...
  Config tests passed
  ✓ PASSED

[TEST 19] Async Commit - Lifecycle (init, submit, free)...
  Lifecycle tests passed
  ✓ PASSED

[TEST 20] Async Commit - STRONG Mode...
  STRONG mode tests passed
  ✓ PASSED

[TEST 21] Async Commit - LEADER_ONLY Mode...
  LEADER_ONLY mode tests passed
  ✓ PASSED

[TEST 22] Async Commit - ASYNC Mode...
  ASYNC mode tests passed
  ✓ PASSED

[TEST 23] Async Commit - FLEXIBLE Mode...
  FLEXIBLE mode tests passed
  ✓ PASSED

[TEST 24] Async Commit - Per-Request Mode Override...
  Mode override tests passed
  ✓ PASSED

[TEST 25] Async Commit - Blocking on Max Pending...
  Blocking tests passed
  ✓ PASSED

[TEST 26] Async Commit - Statistics...
  Stats:
Async Commit Stats:
  Default Mode: STRONG
  Total Writes: 6
    Strong: 2
    Leader-Only: 1
    Async: 3
    Flexible: 0
  Pending Writes: 2
  Pending Bytes: 200
  Avg Ack Latency: 0 us
  ✓ PASSED

[TEST 27] Snapshot - Configuration Defaults...
  Config defaults verified
  ✓ PASSED

[TEST 28] Snapshot - Manager Lifecycle...
  Lifecycle tests passed
  ✓ PASSED

[TEST 29] Snapshot - Trigger Policies...
  Trigger policy tests passed
  ✓ PASSED

[TEST 30] Snapshot - Creation and Writing...
  Creation tests passed
  ✓ PASSED

[TEST 31] Snapshot - Transfer to Follower...
  Transfer tests passed (3 chunks)
  ✓ PASSED

[TEST 32] Snapshot - Installation on Follower...
  Installation tests passed
  ✓ PASSED

[TEST 33] Snapshot - Statistics...
  Stats:
Snapshot Stats:
  State: IDLE
  Current Snapshot ID: 3662201801265543149
  Last Included Index: 20
  Total Snapshots: 2
  Total Bytes: 15
  Total Transfers: 0
  Failed Transfers: 0
  Active Transfers: 0
  Avg Snapshot Time: 0 us
  Avg Transfer Time: 0 us
  ✓ PASSED

[TEST 34] Multi-Raft - Configuration...
  Config tests passed
  ✓ PASSED

[TEST 35] Multi-Raft - Lifecycle (init, create, free)...
  Lifecycle tests passed
  ✓ PASSED

[TEST 36] Multi-Raft - Group Creation...
  Group creation tests passed
  ✓ PASSED

[TEST 37] Multi-Raft - Group Removal...
  Group removal tests passed
  ✓ PASSED

[TEST 38] Multi-Raft - Group Stop/Resume...
  Stop/Resume tests passed
  ✓ PASSED

[TEST 39] Multi-Raft - Range Key Lookup...
  Range lookup tests passed
  ✓ PASSED

[TEST 40] Multi-Raft - Ticking Groups...
  Ticking tests passed
  ✓ PASSED

[TEST 41] Multi-Raft - Statistics...
  Stats:
Multi-Raft Stats:
  Total groups:     2
  Active groups:    2
  Groups as leader: 0
  Total ticks:      5
  Total messages:   0
  Avg msgs/tick:    0
  ✓ PASSED

[TEST 42] Multi-Raft - Many Groups Stress Test...
  Created 200 groups, removed 100, verified consistency
  ✓ PASSED

[TEST 43] Multi-Raft - Hash Table Collision Handling...
  Hash collision handling verified
  ✓ PASSED

[TEST 44] Multi-Raft - Range Boundary Edge Cases...
  Range boundary edge cases passed
  ✓ PASSED

[TEST 45] Multi-Raft - Group Split Operation...
  Group split operation verified
  ✓ PASSED

[TEST 46] Multi-Raft - Group Merge Operation...
  Group merge operation verified
  ✓ PASSED

[TEST 47] Multi-Raft - Group ID Enumeration...
  Group ID enumeration verified
  ✓ PASSED

[TEST 48] Multi-Raft - Group Iterator...
  Group iterator verified
  ✓ PASSED

[TEST 49] Multi-Raft - Maximum Capacity Limits...
  Maximum capacity limits verified
  ✓ PASSED

[TEST 50] Multi-Raft - Null/Edge Case Handling...
  Null and edge case handling verified
  ✓ PASSED

[TEST 51] Multi-Raft - Stop/Resume Many Groups Concurrently...
  Concurrent stop/resume verified
  ✓ PASSED

[TEST 52] Multi-Raft - Overlapping Range Detection...
  Overlapping/gap range handling verified
  ✓ PASSED

[TEST 53] Multi-Raft - Long Key Handling...
  Long key handling verified
  ✓ PASSED

[TEST 54] Multi-Raft - Rapid Create/Remove Cycles...
  Rapid create/remove cycles verified
  ✓ PASSED

[TEST 55] Profile - Default Configuration...
  Default profile validated
  ✓ PASSED

[TEST 56] Profile - All Profiles Valid...
  All 7 profiles validated
  ✓ PASSED

[TEST 57] Profile - Validation Logic...
  Validation logic verified
  ✓ PASSED

[TEST 58] Profile - Override Helpers...
  Override helpers verified
  ✓ PASSED

[TEST 59] Profile - Timing Constraints Per Profile...
  Timing constraints verified per profile
  ✓ PASSED

[TEST 60] Profile - Name Strings...
  Profile names verified
  ✓ PASSED

[TEST 61] Chaos - Engine Lifecycle...
  Engine lifecycle verified
  ✓ PASSED

[TEST 62] Chaos - Network Fault Configuration...
  Network fault configuration verified
  ✓ PASSED

[TEST 63] Chaos - Partition Management...
  Partition management verified
  ✓ PASSED

[TEST 64] Chaos - Node Fault Injection...
  Node fault injection verified
  ✓ PASSED

[TEST 65] Chaos - Byzantine Behavior...
  Byzantine behavior verified
  ✓ PASSED

[TEST 66] Chaos - Clock Manipulation...
  Clock manipulation verified
  ✓ PASSED

[TEST 67] Chaos - Statistics...
  Statistics verified
  ✓ PASSED

[TEST 68] Chaos - Presets...
  Presets verified
  ✓ PASSED

[TEST 69] Chaos - String Helpers...
  String helpers verified
  ✓ PASSED

[TEST 70] CRDT - Manager Lifecycle...
  Manager lifecycle verified
  ✓ PASSED

[TEST 71] CRDT - Key Declaration...
  Key declaration verified
  ✓ PASSED

[TEST 72] CRDT - LWW Register...
  LWW Register verified
  ✓ PASSED

[TEST 73] CRDT - G-Counter...
  G-Counter verified
  ✓ PASSED

[TEST 74] CRDT - PN-Counter...
  PN-Counter verified
  ✓ PASSED

[TEST 75] CRDT - G-Set...
  G-Set verified
  ✓ PASSED

[TEST 76] CRDT - OR-Set...
  OR-Set verified
  ✓ PASSED

[TEST 77] CRDT - String Helpers...
  String helpers verified
  ✓ PASSED

=======================================================
   Test Results:                                       
   Total:  77                                          
   Passed: 77                                          
   Failed: 0                                          
=======================================================
=======================================================
   Kraft Feature Combination Fuzzer                   
=======================================================

[FUZZ-COMBO] Starting Feature Combination Fuzzer
[FUZZ-COMBO] Config: 5 nodes, 1000 ops/combo, 16 combinations
[FUZZ-COMBO] Base seed: 1766105956

[FUZZ-COMBO] Combination 1/16: [LT:Y RI:Y BAT:N BP:Y SES:Y WIT:Y AC:N SNP:N]
[ELECTION] Node runId=1766105956784991: votes=1, need=3, cluster=5
[PROMOTE] Node 2519767903 becoming LEADER in term 1 with 3/3 votes (cluster=5)
[LEADER-COMMIT] leader=0 term=1 committing idx=1 (remotelyLogged=2 +1leader = 3, need=3)
[LEADER-COMMIT] leader=0 term=1 committing idx=2 (remotelyLogged=2 +1leader = 3, need=3)
[ELECTION] Node runId=1766105956786121: votes=1, need=3, cluster=5
[PROMOTE] Node 2519769033 becoming LEADER in term 2 with 3/3 votes (cluster=5)
[LEADER-COMMIT] leader=3 term=2 committing idx=3 (remotelyLogged=2 +1leader = 3, need=3)
[ELECTION] Node runId=1766105956785946: votes=1, need=3, cluster=5
[PROMOTE] Node 2519768858 becoming LEADER in term 3 with 3/3 votes (cluster=5)
[ELECTION] Node runId=1766105956786121: votes=1, need=3, cluster=5
[PROMOTE] Node 2519769033 becoming LEADER in term 4 with 3/3 votes (cluster=5)
[ELECTION] Node runId=1766105956786196: votes=1, need=3, cluster=5
[PROMOTE] Node 2519769108 becoming LEADER in term 5 with 3/3 votes (cluster=5)
[TRUNCATE] node=1 term=5 truncating from idx=4 (commitIdx=1) newTerm=5 from leader
[TRUNCATE] node=3 term=5 truncating from idx=4 (commitIdx=3) newTerm=5 from leader
[LEADER-COMMIT] leader=4 term=5 committing idx=4 (remotelyLogged=2 +1leader = 3, need=3)
[ELECTION] Node runId=1766105956786121: votes=1, need=3, cluster=5
[PROMOTE] Node 2519769033 becoming LEADER in term 6 with 3/3 votes (cluster=5)
[TRUNCATE] node=4 term=6 truncating from idx=5 (commitIdx=4) newTerm=6 from leader
[LEADER-COMMIT] leader=3 term=6 committing idx=5 (remotelyLogged=2 +1leader = 3, need=3)
[ELECTION] Node runId=1766105956785946: votes=1, need=3, cluster=5
[PROMOTE] Node 2519768858 becoming LEADER in term 7 with 3/3 votes (cluster=5)
[TRUNCATE] node=3 term=7 truncating from idx=6 (commitIdx=5) newTerm=7 from leader
[LEADER-COMMIT] leader=1 term=7 committing idx=6 (remotelyLogged=2 +1leader = 3, need=3)
[LEADER-COMMIT] leader=1 term=7 committing idx=7 (remotelyLogged=2 +1leader = 3, need=3)
[ELECTION] Node runId=1766105956785946: votes=1, need=3, cluster=5
[PROMOTE] Node 2519768858 becoming LEADER in term 8 with 3/3 votes (cluster=5)
  -> PASSED (1000 ops, 21 invariant checks)
[FUZZ-COMBO] Combination 2/16: [LT:Y RI:N BAT:Y BP:Y SES:Y WIT:N AC:Y SNP:Y]
[ELECTION] Node runId=1766105956832535: votes=1, need=3, cluster=5
[PROMOTE] Node 2519815447 becoming LEADER in term 1 with 3/3 votes (cluster=5)
[LEADER-COMMIT] leader=4 term=1 committing idx=1 (remotelyLogged=2 +1leader = 3, need=3)
[LEADER-COMMIT] leader=4 term=1 committing idx=2 (remotelyLogged=2 +1leader = 3, need=3)
[ELECTION] Node runId=1766105956832446: votes=1, need=3, cluster=5
[PROMOTE] Node 2519815358 becoming LEADER in term 2 with 3/3 votes (cluster=5)
[TRUNCATE] node=4 term=2 truncating from idx=3 (commitIdx=2) newTerm=2 from leader
[LEADER-COMMIT] leader=3 term=2 committing idx=3 (remotelyLogged=2 +1leader = 3, need=3)
[ELECTION] Node runId=1766105956832348: votes=1, need=3, cluster=5
[PROMOTE] Node 2519815260 becoming LEADER in term 3 with 3/3 votes (cluster=5)
[LEADER-COMMIT] leader=2 term=3 committing idx=4 (remotelyLogged=2 +1leader = 3, need=3)
[LEADER-COMMIT] leader=2 term=3 committing idx=5 (remotelyLogged=2 +1leader = 3, need=3)
[LEADER-COMMIT] leader=2 term=3 committing idx=6 (remotelyLogged=2 +1leader = 3, need=3)
[ELECTION] Node runId=1766105956832535: votes=1, need=3, cluster=5
[PROMOTE] Node 2519815447 becoming LEADER in term 4 with 3/3 votes (cluster=5)
[TRUNCATE] node=2 term=4 truncating from idx=7 (commitIdx=6) newTerm=4 from leader
[ELECTION] Node runId=1766105956832348: votes=1, need=3, cluster=5
[PROMOTE] Node 2519815260 becoming LEADER in term 5 with 3/3 votes (cluster=5)
[TRUNCATE] node=4 term=5 truncating from idx=8 (commitIdx=5) newTerm=5 from leader
[ELECTION] Node runId=1766105956832535: votes=1, need=3, cluster=5
[PROMOTE] Node 2519815447 becoming LEADER in term 6 with 3/3 votes (cluster=5)
[ELECTION] Node runId=1766105956831986: votes=1, need=3, cluster=5
[PROMOTE] Node 2519814898 becoming LEADER in term 7 with 3/3 votes (cluster=5)
[TRUNCATE] node=2 term=7 truncating from idx=7 (commitIdx=6) newTerm=7 from leader
  -> PASSED (1000 ops, 21 invariant checks)
[FUZZ-COMBO] Combination 3/16: [LT:N RI:Y BAT:Y BP:Y SES:Y WIT:Y AC:N SNP:N]
[ELECTION] Node runId=1766105956875244: votes=1, need=3, cluster=5
[PROMOTE] Node 2519858156 becoming LEADER in term 1 with 3/3 votes (cluster=5)
[ELECTION] Node runId=1766105956875154: votes=1, need=3, cluster=5
[PROMOTE] Node 2519858066 becoming LEADER in term 2 with 3/3 votes (cluster=5)
[LEADER-COMMIT] leader=1 term=2 committing idx=1 (remotelyLogged=2 +1leader = 3, need=3)
[ELECTION] Node runId=1766105956874958: votes=1, need=3, cluster=5
[PROMOTE] Node 2519857870 becoming LEADER in term 3 with 3/3 votes (cluster=5)
[ELECTION] Node runId=1766105956875244: votes=1, need=3, cluster=5
[PROMOTE] Node 2519858156 becoming LEADER in term 4 with 3/3 votes (cluster=5)
[LEADER-COMMIT] leader=2 term=4 committing idx=2 (remotelyLogged=2 +1leader = 3, need=3)
[LEADER-COMMIT] leader=2 term=4 committing idx=3 (remotelyLogged=2 +1leader = 3, need=3)
[LEADER-COMMIT] leader=2 term=4 committing idx=4 (remotelyLogged=2 +1leader = 3, need=3)
[ELECTION] Node runId=1766105956875392: votes=1, need=3, cluster=5
[PROMOTE] Node 2519858304 becoming LEADER in term 5 with 3/3 votes (cluster=5)
[TRUNCATE] node=2 term=5 truncating from idx=5 (commitIdx=4) newTerm=5 from leader
[LEADER-COMMIT] leader=4 term=5 committing idx=5 (remotelyLogged=2 +1leader = 3, need=3)
[LEADER-COMMIT] leader=4 term=5 committing idx=6 (remotelyLogged=2 +1leader = 3, need=3)
[ELECTION] Node runId=1766105956875322: votes=1, need=3, cluster=5
[PROMOTE] Node 2519858234 becoming LEADER in term 6 with 3/3 votes (cluster=5)
[TRUNCATE] node=4 term=6 truncating from idx=7 (commitIdx=6) newTerm=6 from leader
[LEADER-COMMIT] leader=3 term=6 committing idx=7 (remotelyLogged=2 +1leader = 3, need=3)
[ELECTION] Node runId=1766105956875244: votes=1, need=3, cluster=5
[PROMOTE] Node 2519858156 becoming LEADER in term 7 with 3/3 votes (cluster=5)
[ELECTION] Node runId=1766105956875392: votes=1, need=3, cluster=5
[PROMOTE] Node 2519858304 becoming LEADER in term 8 with 3/3 votes (cluster=5)
[TRUNCATE] node=2 term=8 truncating from idx=9 (commitIdx=5) newTerm=8 from leader
[ELECTION] Node runId=1766105956875244: votes=1, need=3, cluster=5
[PROMOTE] Node 2519858156 becoming LEADER in term 9 with 3/3 votes (cluster=5)
[ELECTION] Node runId=1766105956875392: votes=1, need=3, cluster=5
[PROMOTE] Node 2519858304 becoming LEADER in term 10 with 3/3 votes (cluster=5)
[ELECTION] Node runId=1766105956875322: votes=1, need=3, cluster=5
[PROMOTE] Node 2519858234 becoming LEADER in term 11 with 3/3 votes (cluster=5)
[ELECTION] Node runId=1766105956875244: votes=1, need=3, cluster=5
[PROMOTE] Node 2519858156 becoming LEADER in term 12 with 3/3 votes (cluster=5)
[ELECTION] Node runId=1766105956875244: votes=1, need=3, cluster=5
[PROMOTE] Node 2519858156 becoming LEADER in term 13 with 3/3 votes (cluster=5)
  -> PASSED (1000 ops, 21 invariant checks)
[FUZZ-COMBO] Combination 4/16: [LT:Y RI:Y BAT:Y BP:Y SES:N WIT:N AC:Y SNP:N]
[ELECTION] Node runId=1766105956916226: votes=1, need=3, cluster=5
[PROMOTE] Node 2519899138 becoming LEADER in term 1 with 3/3 votes (cluster=5)
[LEADER-COMMIT] leader=1 term=1 committing idx=1 (remotelyLogged=2 +1leader = 3, need=3)
[LEADER-COMMIT] leader=1 term=1 committing idx=2 (remotelyLogged=2 +1leader = 3, need=3)
[ELECTION] Node runId=1766105956916051: votes=1, need=3, cluster=5
[PROMOTE] Node 2519898963 becoming LEADER in term 2 with 3/3 votes (cluster=5)
[ELECTION] Node runId=1766105956916051: votes=1, need=3, cluster=5
[PROMOTE] Node 2519898963 becoming LEADER in term 3 with 3/3 votes (cluster=5)
[ELECTION] Node runId=1766105956916316: votes=1, need=3, cluster=5
[PROMOTE] Node 2519899228 becoming LEADER in term 4 with 3/3 votes (cluster=5)
[TRUNCATE] node=1 term=4 truncating from idx=3 (commitIdx=2) newTerm=4 from leader
[ELECTION] Node runId=1766105956916226: votes=1, need=3, cluster=5
[PROMOTE] Node 2519899138 becoming LEADER in term 5 with 3/3 votes (cluster=5)
[ELECTION] Node runId=1766105956916316: votes=1, need=3, cluster=5
[PROMOTE] Node 2519899228 becoming LEADER in term 6 with 3/3 votes (cluster=5)
[ELECTION] Node runId=1766105956916226: votes=1, need=3, cluster=5
[PROMOTE] Node 2519899138 becoming LEADER in term 7 with 3/3 votes (cluster=5)
[ELECTION] Node runId=1766105956916316: votes=1, need=3, cluster=5
[PROMOTE] Node 2519899228 becoming LEADER in term 8 with 3/3 votes (cluster=5)
[ELECTION] Node runId=1766105956916226: votes=1, need=3, cluster=5
[PROMOTE] Node 2519899138 becoming LEADER in term 9 with 3/3 votes (cluster=5)
  -> PASSED (1000 ops, 21 invariant checks)
[FUZZ-COMBO] Combination 5/16: [LT:Y RI:Y BAT:N BP:Y SES:N WIT:N AC:Y SNP:Y]
[ELECTION] Node runId=1766105956964118: votes=1, need=3, cluster=5
[PROMOTE] Node 2519947030 becoming LEADER in term 1 with 3/3 votes (cluster=5)
[LEADER-COMMIT] leader=1 term=1 committing idx=1 (remotelyLogged=2 +1leader = 3, need=3)
[LEADER-COMMIT] leader=1 term=1 committing idx=2 (remotelyLogged=2 +1leader = 3, need=3)
[ELECTION] Node runId=1766105956964200: votes=1, need=3, cluster=5
[PROMOTE] Node 2519947112 becoming LEADER in term 2 with 3/3 votes (cluster=5)
[LEADER-COMMIT] leader=2 term=2 committing idx=4 (remotelyLogged=2 +1leader = 3, need=3)
[ELECTION] Node runId=1766105956964267: votes=1, need=3, cluster=5
[PROMOTE] Node 2519947179 becoming LEADER in term 3 with 3/3 votes (cluster=5)
[ELECTION] Node runId=1766105956964200: votes=1, need=3, cluster=5
[PROMOTE] Node 2519947112 becoming LEADER in term 4 with 3/3 votes (cluster=5)
[ELECTION] Node runId=1766105956964200: votes=1, need=3, cluster=5
[PROMOTE] Node 2519947112 becoming LEADER in term 5 with 3/3 votes (cluster=5)
[ELECTION] Node runId=1766105956964267: votes=1, need=3, cluster=5
[PROMOTE] Node 2519947179 becoming LEADER in term 6 with 3/3 votes (cluster=5)
[ELECTION] Node runId=1766105956963950: votes=1, need=3, cluster=5
[PROMOTE] Node 2519946862 becoming LEADER in term 7 with 3/3 votes (cluster=5)
[TRUNCATE] node=2 term=7 truncating from idx=5 (commitIdx=4) newTerm=7 from leader
[TRUNCATE] node=3 term=7 truncating from idx=5 (commitIdx=4) newTerm=7 from leader
[LEADER-COMMIT] leader=0 term=7 committing idx=5 (remotelyLogged=2 +1leader = 3, need=3)
[ELECTION] Node runId=1766105956964267: votes=1, need=3, cluster=5
[PROMOTE] Node 2519947179 becoming LEADER in term 8 with 3/3 votes (cluster=5)
[TRUNCATE] node=0 term=8 truncating from idx=6 (commitIdx=5) newTerm=8 from leader
[LEADER-COMMIT] leader=3 term=8 committing idx=6 (remotelyLogged=2 +1leader = 3, need=3)
[LEADER-COMMIT] leader=3 term=8 committing idx=7 (remotelyLogged=2 +1leader = 3, need=3)
[ELECTION] Node runId=1766105956963950: votes=1, need=3, cluster=5
[PROMOTE] Node 2519946862 becoming LEADER in term 9 with 3/3 votes (cluster=5)
[ELECTION] Node runId=1766105956964267: votes=1, need=3, cluster=5
[PROMOTE] Node 2519947179 becoming LEADER in term 10 with 3/3 votes (cluster=5)
[ELECTION] Node runId=1766105956964200: votes=1, need=3, cluster=5
[PROMOTE] Node 2519947112 becoming LEADER in term 11 with 3/3 votes (cluster=5)
[TRUNCATE] node=0 term=11 truncating from idx=8 (commitIdx=6) newTerm=11 from leader
[TRUNCATE] node=3 term=11 truncating from idx=8 (commitIdx=7) newTerm=11 from leader
[LEADER-COMMIT] leader=2 term=11 committing idx=8 (remotelyLogged=2 +1leader = 3, need=3)
  -> PASSED (1000 ops, 21 invariant checks)
[FUZZ-COMBO] Combination 6/16: [LT:Y RI:Y BAT:Y BP:Y SES:Y WIT:N AC:N SNP:Y]
[ELECTION] Node runId=1766105956998281: votes=1, need=3, cluster=5
[PROMOTE] Node 2519981193 becoming LEADER in term 1 with 3/3 votes (cluster=5)
[LEADER-COMMIT] leader=0 term=1 committing idx=1 (remotelyLogged=2 +1leader = 3, need=3)
[ELECTION] Node runId=1766105956998584: votes=1, need=3, cluster=5
[PROMOTE] Node 2519981496 becoming LEADER in term 2 with 3/3 votes (cluster=5)
[LEADER-COMMIT] leader=2 term=2 committing idx=2 (remotelyLogged=2 +1leader = 3, need=3)
[ELECTION] Node runId=1766105956998662: votes=1, need=3, cluster=5
[PROMOTE] Node 2519981574 becoming LEADER in term 3 with 3/3 votes (cluster=5)
[ELECTION] Node runId=1766105956998734: votes=1, need=3, cluster=5
[PROMOTE] Node 2519981646 becoming LEADER in term 4 with 3/3 votes (cluster=5)
[ELECTION] Node runId=1766105956998662: votes=1, need=3, cluster=5
[PROMOTE] Node 2519981574 becoming LEADER in term 5 with 3/3 votes (cluster=5)
[ELECTION] Node runId=1766105956998584: votes=1, need=3, cluster=5
[PROMOTE] Node 2519981496 becoming LEADER in term 6 with 3/3 votes (cluster=5)
[ELECTION] Node runId=1766105956998662: votes=1, need=3, cluster=5
[PROMOTE] Node 2519981574 becoming LEADER in term 7 with 3/3 votes (cluster=5)
  -> PASSED (1000 ops, 21 invariant checks)
[FUZZ-COMBO] Combination 7/16: [LT:N RI:Y BAT:Y BP:Y SES:Y WIT:N AC:Y SNP:N]
[ELECTION] Node runId=1766105957049936: votes=1, need=3, cluster=5
[PROMOTE] Node 2520032848 becoming LEADER in term 1 with 3/3 votes (cluster=5)
[LEADER-COMMIT] leader=1 term=1 committing idx=1 (remotelyLogged=2 +1leader = 3, need=3)
[LEADER-COMMIT] leader=1 term=1 committing idx=2 (remotelyLogged=2 +1leader = 3, need=3)
[LEADER-COMMIT] leader=1 term=1 committing idx=3 (remotelyLogged=2 +1leader = 3, need=3)
[LEADER-COMMIT] leader=1 term=1 committing idx=4 (remotelyLogged=2 +1leader = 3, need=3)
[LEADER-COMMIT] leader=1 term=1 committing idx=5 (remotelyLogged=2 +1leader = 3, need=3)
[ELECTION] Node runId=1766105957050157: votes=1, need=3, cluster=5
[PROMOTE] Node 2520033069 becoming LEADER in term 2 with 3/3 votes (cluster=5)
[TRUNCATE] node=1 term=2 truncating from idx=6 (commitIdx=5) newTerm=2 from leader
[LEADER-COMMIT] leader=4 term=2 committing idx=6 (remotelyLogged=2 +1leader = 3, need=3)
[LEADER-COMMIT] leader=4 term=2 committing idx=7 (remotelyLogged=2 +1leader = 3, need=3)
[LEADER-COMMIT] leader=4 term=2 committing idx=8 (remotelyLogged=2 +1leader = 3, need=3)
[ELECTION] Node runId=1766105957050022: votes=1, need=3, cluster=5
[PROMOTE] Node 2520032934 becoming LEADER in term 3 with 3/3 votes (cluster=5)
[LEADER-COMMIT] leader=2 term=3 committing idx=9 (remotelyLogged=2 +1leader = 3, need=3)
[ELECTION] Node runId=1766105957049936: votes=1, need=3, cluster=5
[PROMOTE] Node 2520032848 becoming LEADER in term 4 with 3/3 votes (cluster=5)
[TRUNCATE] node=2 term=4 truncating from idx=10 (commitIdx=9) newTerm=4 from leader
[LEADER-COMMIT] leader=1 term=4 committing idx=10 (remotelyLogged=2 +1leader = 3, need=3)
[LEADER-COMMIT] leader=1 term=4 committing idx=11 (remotelyLogged=2 +1leader = 3, need=3)
[ELECTION] Node runId=1766105957050092: votes=1, need=3, cluster=5
[PROMOTE] Node 2520033004 becoming LEADER in term 5 with 3/3 votes (cluster=5)
[ELECTION] Node runId=1766105957050092: votes=1, need=3, cluster=5
[PROMOTE] Node 2520033004 becoming LEADER in term 6 with 3/3 votes (cluster=5)
[TRUNCATE] node=1 term=6 truncating from idx=12 (commitIdx=11) newTerm=6 from leader
[LEADER-COMMIT] leader=3 term=6 committing idx=12 (remotelyLogged=2 +1leader = 3, need=3)
[ELECTION] Node runId=1766105957050022: votes=1, need=3, cluster=5
[PROMOTE] Node 2520032934 becoming LEADER in term 7 with 3/3 votes (cluster=5)
[TRUNCATE] node=3 term=7 truncating from idx=13 (commitIdx=12) newTerm=7 from leader
[LEADER-COMMIT] leader=2 term=7 committing idx=13 (remotelyLogged=2 +1leader = 3, need=3)
[ELECTION] Node runId=1766105957049936: votes=1, need=3, cluster=5
[PROMOTE] Node 2520032848 becoming LEADER in term 8 with 3/3 votes (cluster=5)
[TRUNCATE] node=2 term=8 truncating from idx=14 (commitIdx=13) newTerm=8 from leader
  -> PASSED (1000 ops, 21 invariant checks)
[FUZZ-COMBO] Combination 8/16: [LT:N RI:N BAT:N BP:Y SES:Y WIT:N AC:Y SNP:N]
[ELECTION] Node runId=1766105957092783: votes=1, need=3, cluster=5
[PROMOTE] Node 2520075695 becoming LEADER in term 1 with 3/3 votes (cluster=5)
[LEADER-COMMIT] leader=3 term=1 committing idx=1 (remotelyLogged=2 +1leader = 3, need=3)
[ELECTION] Node runId=1766105957092857: votes=1, need=3, cluster=5
[PROMOTE] Node 2520075769 becoming LEADER in term 2 with 3/3 votes (cluster=5)
[TRUNCATE] node=3 term=2 truncating from idx=2 (commitIdx=1) newTerm=2 from leader
[LEADER-COMMIT] leader=4 term=2 committing idx=2 (remotelyLogged=2 +1leader = 3, need=3)
[LEADER-COMMIT] leader=4 term=2 committing idx=3 (remotelyLogged=2 +1leader = 3, need=3)
[LEADER-COMMIT] leader=4 term=2 committing idx=4 (remotelyLogged=2 +1leader = 3, need=3)
[LEADER-COMMIT] leader=4 term=2 committing idx=5 (remotelyLogged=2 +1leader = 3, need=3)
[LEADER-COMMIT] leader=4 term=2 committing idx=6 (remotelyLogged=2 +1leader = 3, need=3)
[LEADER-COMMIT] leader=4 term=2 committing idx=7 (remotelyLogged=2 +1leader = 3, need=3)
[LEADER-COMMIT] leader=4 term=2 committing idx=8 (remotelyLogged=2 +1leader = 3, need=3)
[LEADER-COMMIT] leader=4 term=2 committing idx=9 (remotelyLogged=2 +1leader = 3, need=3)
[LEADER-COMMIT] leader=4 term=2 committing idx=10 (remotelyLogged=2 +1leader = 3, need=3)
[ELECTION] Node runId=1766105957092425: votes=1, need=3, cluster=5
[PROMOTE] Node 2520075337 becoming LEADER in term 3 with 3/3 votes (cluster=5)
[TRUNCATE] node=4 term=3 truncating from idx=11 (commitIdx=10) newTerm=3 from leader
[LEADER-COMMIT] leader=0 term=3 committing idx=11 (remotelyLogged=2 +1leader = 3, need=3)
[LEADER-COMMIT] leader=0 term=3 committing idx=12 (remotelyLogged=2 +1leader = 3, need=3)
[ELECTION] Node runId=1766105957092783: votes=1, need=3, cluster=5
[PROMOTE] Node 2520075695 becoming LEADER in term 4 with 3/3 votes (cluster=5)
[LEADER-COMMIT] leader=3 term=4 committing idx=13 (remotelyLogged=2 +1leader = 3, need=3)
[LEADER-COMMIT] leader=3 term=4 committing idx=14 (remotelyLogged=2 +1leader = 3, need=3)
[LEADER-COMMIT] leader=3 term=4 committing idx=15 (remotelyLogged=2 +1leader = 3, need=3)
[ELECTION] Node runId=1766105957092705: votes=1, need=3, cluster=5
[PROMOTE] Node 2520075617 becoming LEADER in term 5 with 3/3 votes (cluster=5)
[TRUNCATE] node=3 term=5 truncating from idx=16 (commitIdx=15) newTerm=5 from leader
[ELECTION] Node runId=1766105957092783: votes=1, need=3, cluster=5
[PROMOTE] Node 2520075695 becoming LEADER in term 6 with 3/3 votes (cluster=5)
[ELECTION] Node runId=1766105957092857: votes=1, need=3, cluster=5
[PROMOTE] Node 2520075769 becoming LEADER in term 7 with 3/3 votes (cluster=5)
[ELECTION] Node runId=1766105957092783: votes=1, need=3, cluster=5
[PROMOTE] Node 2520075695 becoming LEADER in term 8 with 3/3 votes (cluster=5)
[ELECTION] Node runId=1766105957092857: votes=1, need=3, cluster=5
[PROMOTE] Node 2520075769 becoming LEADER in term 9 with 3/3 votes (cluster=5)
  -> PASSED (1000 ops, 21 invariant checks)
[FUZZ-COMBO] Combination 9/16: [LT:Y RI:Y BAT:Y BP:N SES:Y WIT:N AC:Y SNP:N]
[ELECTION] Node runId=1766105957144093: votes=1, need=3, cluster=5
[PROMOTE] Node 2520127005 becoming LEADER in term 1 with 3/3 votes (cluster=5)
[LEADER-COMMIT] leader=1 term=1 committing idx=1 (remotelyLogged=2 +1leader = 3, need=3)
[LEADER-COMMIT] leader=1 term=1 committing idx=2 (remotelyLogged=2 +1leader = 3, need=3)
[LEADER-COMMIT] leader=1 term=1 committing idx=3 (remotelyLogged=2 +1leader = 3, need=3)
[LEADER-COMMIT] leader=1 term=1 committing idx=4 (remotelyLogged=2 +1leader = 3, need=3)
[ELECTION] Node runId=1766105957144184: votes=1, need=3, cluster=5
[PROMOTE] Node 2520127096 becoming LEADER in term 2 with 3/3 votes (cluster=5)
[TRUNCATE] node=1 term=2 truncating from idx=5 (commitIdx=4) newTerm=2 from leader
[LEADER-COMMIT] leader=2 term=2 committing idx=5 (remotelyLogged=2 +1leader = 3, need=3)
[ELECTION] Node runId=1766105957144259: votes=1, need=3, cluster=5
[PROMOTE] Node 2520127171 becoming LEADER in term 3 with 3/3 votes (cluster=5)
[TRUNCATE] node=2 term=3 truncating from idx=6 (commitIdx=5) newTerm=3 from leader
[LEADER-COMMIT] leader=3 term=3 committing idx=6 (remotelyLogged=2 +1leader = 3, need=3)
[ELECTION] Node runId=1766105957144093: votes=1, need=3, cluster=5
[PROMOTE] Node 2520127005 becoming LEADER in term 4 with 3/3 votes (cluster=5)
[TRUNCATE] node=3 term=4 truncating from idx=7 (commitIdx=6) newTerm=4 from leader
[LEADER-COMMIT] leader=1 term=4 committing idx=7 (remotelyLogged=2 +1leader = 3, need=3)
[ELECTION] Node runId=1766105957144259: votes=1, need=3, cluster=5
[PROMOTE] Node 2520127171 becoming LEADER in term 5 with 3/3 votes (cluster=5)
[TRUNCATE] node=1 term=5 truncating from idx=9 (commitIdx=7) newTerm=5 from leader
[ELECTION] Node runId=1766105957144184: votes=1, need=3, cluster=5
[PROMOTE] Node 2520127096 becoming LEADER in term 6 with 3/3 votes (cluster=5)
[TRUNCATE] node=1 term=6 truncating from idx=8 (commitIdx=7) newTerm=6 from leader
[TRUNCATE] node=3 term=6 truncating from idx=8 (commitIdx=7) newTerm=6 from leader
[LEADER-COMMIT] leader=2 term=6 committing idx=8 (remotelyLogged=2 +1leader = 3, need=3)
[LEADER-COMMIT] leader=2 term=6 committing idx=9 (remotelyLogged=2 +1leader = 3, need=3)
[LEADER-COMMIT] leader=2 term=6 committing idx=10 (remotelyLogged=2 +1leader = 3, need=3)
[LEADER-COMMIT] leader=2 term=6 committing idx=11 (remotelyLogged=2 +1leader = 3, need=3)
[ELECTION] Node runId=1766105957144093: votes=1, need=3, cluster=5
[PROMOTE] Node 2520127005 becoming LEADER in term 7 with 3/3 votes (cluster=5)
[TRUNCATE] node=2 term=7 truncating from idx=12 (commitIdx=11) newTerm=7 from leader
[ELECTION] Node runId=1766105957144184: votes=1, need=3, cluster=5
[PROMOTE] Node 2520127096 becoming LEADER in term 8 with 3/3 votes (cluster=5)
[TRUNCATE] node=1 term=8 truncating from idx=13 (commitIdx=10) newTerm=8 from leader
  -> PASSED (1000 ops, 21 invariant checks)
[FUZZ-COMBO] Combination 10/16: [LT:Y RI:Y BAT:N BP:Y SES:Y WIT:Y AC:Y SNP:N]
[ELECTION] Node runId=1766105957183880: votes=1, need=3, cluster=5
[PROMOTE] Node 2520166792 becoming LEADER in term 1 with 3/3 votes (cluster=5)
[LEADER-COMMIT] leader=3 term=1 committing idx=1 (remotelyLogged=2 +1leader = 3, need=3)
[ELECTION] Node runId=1766105957183948: votes=1, need=3, cluster=5
[PROMOTE] Node 2520166860 becoming LEADER in term 2 with 3/3 votes (cluster=5)
[TRUNCATE] node=3 term=2 truncating from idx=2 (commitIdx=1) newTerm=2 from leader
[LEADER-COMMIT] leader=4 term=2 committing idx=2 (remotelyLogged=2 +1leader = 3, need=3)
[ELECTION] Node runId=1766105957183540: votes=1, need=3, cluster=5
[PROMOTE] Node 2520166452 becoming LEADER in term 3 with 3/3 votes (cluster=5)
[TRUNCATE] node=4 term=3 truncating from idx=3 (commitIdx=2) newTerm=3 from leader
[LEADER-COMMIT] leader=0 term=3 committing idx=3 (remotelyLogged=2 +1leader = 3, need=3)
[ELECTION] Node runId=1766105957183880: votes=1, need=3, cluster=5
[PROMOTE] Node 2520166792 becoming LEADER in term 4 with 3/3 votes (cluster=5)
[ELECTION] Node runId=1766105957183540: votes=1, need=3, cluster=5
[PROMOTE] Node 2520166452 becoming LEADER in term 5 with 3/3 votes (cluster=5)
[ELECTION] Node runId=1766105957183540: votes=1, need=3, cluster=5
[PROMOTE] Node 2520166452 becoming LEADER in term 6 with 3/3 votes (cluster=5)
[ELECTION] Node runId=1766105957183880: votes=1, need=3, cluster=5
[PROMOTE] Node 2520166792 becoming LEADER in term 7 with 3/3 votes (cluster=5)
[ELECTION] Node runId=1766105957183540: votes=1, need=3, cluster=5
[PROMOTE] Node 2520166452 becoming LEADER in term 8 with 3/3 votes (cluster=5)
[ELECTION] Node runId=1766105957183880: votes=1, need=3, cluster=5
[PROMOTE] Node 2520166792 becoming LEADER in term 9 with 3/3 votes (cluster=5)
[ELECTION] Node runId=1766105957183540: votes=1, need=3, cluster=5
[PROMOTE] Node 2520166452 becoming LEADER in term 10 with 3/3 votes (cluster=5)
[ELECTION] Node runId=1766105957183880: votes=1, need=3, cluster=5
[PROMOTE] Node 2520166792 becoming LEADER in term 11 with 3/3 votes (cluster=5)
  -> PASSED (1000 ops, 21 invariant checks)
[FUZZ-COMBO] Combination 11/16: [LT:Y RI:Y BAT:Y BP:Y SES:N WIT:N AC:Y SNP:Y]
[ELECTION] Node runId=1766105957226706: votes=1, need=3, cluster=5
[PROMOTE] Node 2520209618 becoming LEADER in term 1 with 3/3 votes (cluster=5)
[LEADER-COMMIT] leader=0 term=1 committing idx=1 (remotelyLogged=2 +1leader = 3, need=3)
[LEADER-COMMIT] leader=0 term=1 committing idx=2 (remotelyLogged=2 +1leader = 3, need=3)
[LEADER-COMMIT] leader=0 term=1 committing idx=3 (remotelyLogged=2 +1leader = 3, need=3)
[ELECTION] Node runId=1766105957226706: votes=1, need=3, cluster=5
[PROMOTE] Node 2520209618 becoming LEADER in term 2 with 3/3 votes (cluster=5)
[ELECTION] Node runId=1766105957226911: votes=1, need=3, cluster=5
[PROMOTE] Node 2520209823 becoming LEADER in term 3 with 3/3 votes (cluster=5)
[LEADER-COMMIT] leader=3 term=3 committing idx=4 (remotelyLogged=2 +1leader = 3, need=3)
[LEADER-COMMIT] leader=3 term=3 committing idx=5 (remotelyLogged=2 +1leader = 3, need=3)
[LEADER-COMMIT] leader=3 term=3 committing idx=6 (remotelyLogged=2 +1leader = 3, need=3)
[ELECTION] Node runId=1766105957226911: votes=1, need=3, cluster=5
[PROMOTE] Node 2520209823 becoming LEADER in term 4 with 3/3 votes (cluster=5)
[ELECTION] Node runId=1766105957226971: votes=1, need=3, cluster=5
[PROMOTE] Node 2520209883 becoming LEADER in term 5 with 3/3 votes (cluster=5)
[ELECTION] Node runId=1766105957226785: votes=1, need=3, cluster=5
[PROMOTE] Node 2520209697 becoming LEADER in term 6 with 3/3 votes (cluster=5)
[TRUNCATE] node=4 term=6 truncating from idx=8 (commitIdx=5) newTerm=6 from leader
  -> PASSED (1000 ops, 21 invariant checks)
[FUZZ-COMBO] Combination 12/16: [LT:Y RI:N BAT:Y BP:Y SES:N WIT:Y AC:Y SNP:Y]
[ELECTION] Node runId=1766105957291042: votes=1, need=3, cluster=5
[PROMOTE] Node 2520273954 becoming LEADER in term 1 with 3/3 votes (cluster=5)
[LEADER-COMMIT] leader=3 term=1 committing idx=1 (remotelyLogged=2 +1leader = 3, need=3)
[ELECTION] Node runId=1766105957291042: votes=1, need=3, cluster=5
[PROMOTE] Node 2520273954 becoming LEADER in term 2 with 3/3 votes (cluster=5)
[LEADER-COMMIT] leader=3 term=2 committing idx=2 (remotelyLogged=2 +1leader = 3, need=3)
[ELECTION] Node runId=1766105957290862: votes=1, need=3, cluster=5
[PROMOTE] Node 2520273774 becoming LEADER in term 3 with 3/3 votes (cluster=5)
[LEADER-COMMIT] leader=1 term=3 committing idx=3 (remotelyLogged=2 +1leader = 3, need=3)
[LEADER-COMMIT] leader=1 term=3 committing idx=4 (remotelyLogged=2 +1leader = 3, need=3)
[ELECTION] Node runId=1766105957290960: votes=1, need=3, cluster=5
[PROMOTE] Node 2520273872 becoming LEADER in term 4 with 3/3 votes (cluster=5)
[TRUNCATE] node=1 term=4 truncating from idx=6 (commitIdx=4) newTerm=4 from leader
[ELECTION] Node runId=1766105957290862: votes=1, need=3, cluster=5
[PROMOTE] Node 2520273774 becoming LEADER in term 5 with 3/3 votes (cluster=5)
[ELECTION] Node runId=1766105957290862: votes=1, need=3, cluster=5
[PROMOTE] Node 2520273774 becoming LEADER in term 6 with 3/3 votes (cluster=5)
[ELECTION] Node runId=1766105957290960: votes=1, need=3, cluster=5
[PROMOTE] Node 2520273872 becoming LEADER in term 7 with 3/3 votes (cluster=5)
  -> PASSED (1000 ops, 21 invariant checks)
[FUZZ-COMBO] Combination 13/16: [LT:Y RI:Y BAT:N BP:Y SES:N WIT:N AC:Y SNP:Y]
[ELECTION] Node runId=1766105957341068: votes=1, need=3, cluster=5
[PROMOTE] Node 2520323980 becoming LEADER in term 1 with 3/3 votes (cluster=5)
[LEADER-COMMIT] leader=3 term=1 committing idx=1 (remotelyLogged=2 +1leader = 3, need=3)
[LEADER-COMMIT] leader=3 term=1 committing idx=2 (remotelyLogged=2 +1leader = 3, need=3)
[ELECTION] Node runId=1766105957341141: votes=1, need=3, cluster=5
[PROMOTE] Node 2520324053 becoming LEADER in term 2 with 3/3 votes (cluster=5)
[LEADER-COMMIT] leader=4 term=2 committing idx=3 (remotelyLogged=2 +1leader = 3, need=3)
[ELECTION] Node runId=1766105957340990: votes=1, need=3, cluster=5
[PROMOTE] Node 2520323902 becoming LEADER in term 3 with 3/3 votes (cluster=5)
[ELECTION] Node runId=1766105957340900: votes=1, need=3, cluster=5
[PROMOTE] Node 2520323812 becoming LEADER in term 4 with 3/3 votes (cluster=5)
[TRUNCATE] node=4 term=4 truncating from idx=4 (commitIdx=3) newTerm=4 from leader
[ELECTION] Node runId=1766105957340900: votes=1, need=3, cluster=5
[PROMOTE] Node 2520323812 becoming LEADER in term 5 with 3/3 votes (cluster=5)
[ELECTION] Node runId=1766105957340990: votes=1, need=3, cluster=5
[PROMOTE] Node 2520323902 becoming LEADER in term 6 with 3/3 votes (cluster=5)
[ELECTION] Node runId=1766105957340900: votes=1, need=3, cluster=5
[PROMOTE] Node 2520323812 becoming LEADER in term 7 with 3/3 votes (cluster=5)
[ELECTION] Node runId=1766105957340990: votes=1, need=3, cluster=5
[PROMOTE] Node 2520323902 becoming LEADER in term 8 with 3/3 votes (cluster=5)
  -> PASSED (1000 ops, 21 invariant checks)
[FUZZ-COMBO] Combination 14/16: [LT:N RI:Y BAT:Y BP:Y SES:Y WIT:N AC:Y SNP:N]
[ELECTION] Node runId=1766105957375674: votes=1, need=3, cluster=5
[PROMOTE] Node 2520358586 becoming LEADER in term 1 with 3/3 votes (cluster=5)
[LEADER-COMMIT] leader=4 term=1 committing idx=1 (remotelyLogged=2 +1leader = 3, need=3)
[LEADER-COMMIT] leader=4 term=1 committing idx=2 (remotelyLogged=2 +1leader = 3, need=3)
[ELECTION] Node runId=1766105957375269: votes=1, need=3, cluster=5
[PROMOTE] Node 2520358181 becoming LEADER in term 2 with 3/3 votes (cluster=5)
[TRUNCATE] node=4 term=2 truncating from idx=3 (commitIdx=2) newTerm=2 from leader
[LEADER-COMMIT] leader=1 term=2 committing idx=3 (remotelyLogged=2 +1leader = 3, need=3)
[ELECTION] Node runId=1766105957375674: votes=1, need=3, cluster=5
[PROMOTE] Node 2520358586 becoming LEADER in term 3 with 3/3 votes (cluster=5)
[ELECTION] Node runId=1766105957375471: votes=1, need=3, cluster=5
[PROMOTE] Node 2520358383 becoming LEADER in term 4 with 3/3 votes (cluster=5)
[TRUNCATE] node=4 term=4 truncating from idx=5 (commitIdx=2) newTerm=4 from leader
[ELECTION] Node runId=1766105957375674: votes=1, need=3, cluster=5
[PROMOTE] Node 2520358586 becoming LEADER in term 5 with 3/3 votes (cluster=5)
[TRUNCATE] node=2 term=5 truncating from idx=6 (commitIdx=2) newTerm=5 from leader
[ELECTION] Node runId=1766105957375471: votes=1, need=3, cluster=5
[PROMOTE] Node 2520358383 becoming LEADER in term 6 with 3/3 votes (cluster=5)
[ELECTION] Node runId=1766105957375269: votes=1, need=3, cluster=5
[PROMOTE] Node 2520358181 becoming LEADER in term 7 with 3/3 votes (cluster=5)
[ELECTION] Node runId=1766105957375674: votes=1, need=3, cluster=5
[PROMOTE] Node 2520358586 becoming LEADER in term 8 with 3/3 votes (cluster=5)
[ELECTION] Node runId=1766105957375269: votes=1, need=3, cluster=5
[PROMOTE] Node 2520358181 becoming LEADER in term 9 with 3/3 votes (cluster=5)
[ELECTION] Node runId=1766105957375674: votes=1, need=3, cluster=5
[PROMOTE] Node 2520358586 becoming LEADER in term 10 with 3/3 votes (cluster=5)
[ELECTION] Node runId=1766105957375269: votes=1, need=3, cluster=5
[PROMOTE] Node 2520358181 becoming LEADER in term 11 with 3/3 votes (cluster=5)
  -> PASSED (1000 ops, 21 invariant checks)
[FUZZ-COMBO] Combination 15/16: [LT:Y RI:Y BAT:N BP:Y SES:N WIT:N AC:N SNP:Y]
[ELECTION] Node runId=1766105957411389: votes=1, need=3, cluster=5
[PROMOTE] Node 2520394301 becoming LEADER in term 1 with 3/3 votes (cluster=5)
[LEADER-COMMIT] leader=0 term=1 committing idx=1 (remotelyLogged=2 +1leader = 3, need=3)
[LEADER-COMMIT] leader=0 term=1 committing idx=2 (remotelyLogged=2 +1leader = 3, need=3)
[LEADER-COMMIT] leader=0 term=1 committing idx=3 (remotelyLogged=2 +1leader = 3, need=3)
[ELECTION] Node runId=1766105957411798: votes=1, need=3, cluster=5
[PROMOTE] Node 2520394710 becoming LEADER in term 2 with 3/3 votes (cluster=5)
[LEADER-COMMIT] leader=4 term=2 committing idx=4 (remotelyLogged=2 +1leader = 3, need=3)
[LEADER-COMMIT] leader=4 term=2 committing idx=5 (remotelyLogged=2 +1leader = 3, need=3)
[LEADER-COMMIT] leader=4 term=2 committing idx=6 (remotelyLogged=2 +1leader = 3, need=3)
[LEADER-COMMIT] leader=4 term=2 committing idx=7 (remotelyLogged=2 +1leader = 3, need=3)
[LEADER-COMMIT] leader=4 term=2 committing idx=8 (remotelyLogged=2 +1leader = 3, need=3)
[ELECTION] Node runId=1766105957411559: votes=1, need=3, cluster=5
[PROMOTE] Node 2520394471 becoming LEADER in term 3 with 3/3 votes (cluster=5)
[TRUNCATE] node=3 term=3 truncating from idx=9 (commitIdx=8) newTerm=3 from leader
[TRUNCATE] node=4 term=3 truncating from idx=9 (commitIdx=8) newTerm=3 from leader
[LEADER-COMMIT] leader=1 term=3 committing idx=9 (remotelyLogged=2 +1leader = 3, need=3)
[LEADER-COMMIT] leader=1 term=3 committing idx=10 (remotelyLogged=2 +1leader = 3, need=3)
[LEADER-COMMIT] leader=1 term=3 committing idx=11 (remotelyLogged=2 +1leader = 3, need=3)
[ELECTION] Node runId=1766105957411718: votes=1, need=3, cluster=5
[PROMOTE] Node 2520394630 becoming LEADER in term 4 with 3/3 votes (cluster=5)
[TRUNCATE] node=1 term=4 truncating from idx=12 (commitIdx=11) newTerm=4 from leader
[LEADER-COMMIT] leader=3 term=4 committing idx=12 (remotelyLogged=2 +1leader = 3, need=3)
[ELECTION] Node runId=1766105957411559: votes=1, need=3, cluster=5
[PROMOTE] Node 2520394471 becoming LEADER in term 5 with 3/3 votes (cluster=5)
[TRUNCATE] node=3 term=5 truncating from idx=13 (commitIdx=12) newTerm=5 from leader
[TRUNCATE] node=4 term=5 truncating from idx=13 (commitIdx=12) newTerm=5 from leader
[LEADER-COMMIT] leader=1 term=5 committing idx=13 (remotelyLogged=2 +1leader = 3, need=3)
[LEADER-COMMIT] leader=1 term=5 committing idx=14 (remotelyLogged=2 +1leader = 3, need=3)
[ELECTION] Node runId=1766105957411798: votes=1, need=3, cluster=5
[PROMOTE] Node 2520394710 becoming LEADER in term 6 with 3/3 votes (cluster=5)
[TRUNCATE] node=1 term=6 truncating from idx=15 (commitIdx=14) newTerm=6 from leader
[ELECTION] Node runId=1766105957411718: votes=1, need=3, cluster=5
[PROMOTE] Node 2520394630 becoming LEADER in term 7 with 3/3 votes (cluster=5)
[TRUNCATE] node=1 term=7 truncating from idx=15 (commitIdx=14) newTerm=7 from leader
[TRUNCATE] node=4 term=7 truncating from idx=15 (commitIdx=13) newTerm=7 from leader
[LEADER-COMMIT] leader=3 term=7 committing idx=15 (remotelyLogged=2 +1leader = 3, need=3)
[LEADER-COMMIT] leader=3 term=7 committing idx=16 (remotelyLogged=2 +1leader = 3, need=3)
[LEADER-COMMIT] leader=3 term=7 committing idx=17 (remotelyLogged=2 +1leader = 3, need=3)
[ELECTION] Node runId=1766105957411798: votes=1, need=3, cluster=5
[PROMOTE] Node 2520394710 becoming LEADER in term 8 with 3/3 votes (cluster=5)
[TRUNCATE] node=3 term=8 truncating from idx=18 (commitIdx=17) newTerm=8 from leader
[LEADER-COMMIT] leader=4 term=8 committing idx=18 (remotelyLogged=2 +1leader = 3, need=3)
[ELECTION] Node runId=1766105957411559: votes=1, need=3, cluster=5
[PROMOTE] Node 2520394471 becoming LEADER in term 9 with 3/3 votes (cluster=5)
[TRUNCATE] node=4 term=9 truncating from idx=19 (commitIdx=18) newTerm=9 from leader
[LEADER-COMMIT] leader=1 term=9 committing idx=19 (remotelyLogged=2 +1leader = 3, need=3)
[LEADER-COMMIT] leader=1 term=9 committing idx=20 (remotelyLogged=2 +1leader = 3, need=3)
[ELECTION] Node runId=1766105957411798: votes=1, need=3, cluster=5
[PROMOTE] Node 2520394710 becoming LEADER in term 10 with 3/3 votes (cluster=5)
[ELECTION] Node runId=1766105957411718: votes=1, need=3, cluster=5
[PROMOTE] Node 2520394630 becoming LEADER in term 11 with 3/3 votes (cluster=5)
  -> PASSED (1000 ops, 21 invariant checks)
[FUZZ-COMBO] Combination 16/16: [LT:N RI:Y BAT:Y BP:N SES:N WIT:N AC:Y SNP:N]
[ELECTION] Node runId=1766105957455453: votes=1, need=3, cluster=5
[PROMOTE] Node 2520438365 becoming LEADER in term 1 with 3/3 votes (cluster=5)
[LEADER-COMMIT] leader=1 term=1 committing idx=1 (remotelyLogged=2 +1leader = 3, need=3)
[ELECTION] Node runId=1766105957455533: votes=1, need=3, cluster=5
[PROMOTE] Node 2520438445 becoming LEADER in term 2 with 3/3 votes (cluster=5)
[TRUNCATE] node=1 term=2 truncating from idx=2 (commitIdx=1) newTerm=2 from leader
[LEADER-COMMIT] leader=2 term=2 committing idx=2 (remotelyLogged=2 +1leader = 3, need=3)
[ELECTION] Node runId=1766105957455453: votes=1, need=3, cluster=5
[PROMOTE] Node 2520438365 becoming LEADER in term 3 with 3/3 votes (cluster=5)
[ELECTION] Node runId=1766105957455605: votes=1, need=3, cluster=5
[PROMOTE] Node 2520438517 becoming LEADER in term 4 with 3/3 votes (cluster=5)
[LEADER-COMMIT] leader=3 term=4 committing idx=3 (remotelyLogged=2 +1leader = 3, need=3)
[ELECTION] Node runId=1766105957455298: votes=1, need=3, cluster=5
[PROMOTE] Node 2520438210 becoming LEADER in term 5 with 3/3 votes (cluster=5)
[ELECTION] Node runId=1766105957455670: votes=1, need=3, cluster=5
[PROMOTE] Node 2520438582 becoming LEADER in term 6 with 3/3 votes (cluster=5)
[TRUNCATE] node=0 term=6 truncating from idx=4 (commitIdx=0) newTerm=6 from leader
[ELECTION] Node runId=1766105957455298: votes=1, need=3, cluster=5
[PROMOTE] Node 2520438210 becoming LEADER in term 7 with 3/3 votes (cluster=5)
[TRUNCATE] node=4 term=7 truncating from idx=7 (commitIdx=0) newTerm=7 from leader
[ELECTION] Node runId=1766105957455670: votes=1, need=3, cluster=5
[PROMOTE] Node 2520438582 becoming LEADER in term 8 with 3/3 votes (cluster=5)
[ELECTION] Node runId=1766105957455605: votes=1, need=3, cluster=5
[PROMOTE] Node 2520438517 becoming LEADER in term 9 with 3/3 votes (cluster=5)
[ELECTION] Node runId=1766105957455605: votes=1, need=3, cluster=5
[PROMOTE] Node 2520438517 becoming LEADER in term 10 with 3/3 votes (cluster=5)
[ELECTION] Node runId=1766105957455298: votes=1, need=3, cluster=5
[PROMOTE] Node 2520438210 becoming LEADER in term 11 with 3/3 votes (cluster=5)
[ELECTION] Node runId=1766105957455670: votes=1, need=3, cluster=5
[PROMOTE] Node 2520438582 becoming LEADER in term 12 with 3/3 votes (cluster=5)
[ELECTION] Node runId=1766105957455298: votes=1, need=3, cluster=5
[PROMOTE] Node 2520438210 becoming LEADER in term 13 with 3/3 votes (cluster=5)
  -> PASSED (1000 ops, 21 invariant checks)

[FUZZ-COMBO] ==========================================
[FUZZ-COMBO] SUMMARY
[FUZZ-COMBO] ==========================================
  Combinations tested:  16
  Combinations passed:  16
  Combinations failed:  0
  Total operations:     16000
  Total invariants:     336
  Violations:           0
[FUZZ-COMBO] ==========================================


=======================================================
   Overall Result: PASSED (all combinations passed)
   Seed: 1766105956  (reproduce with: -s 1766105956 -n 5 -c 16)
=======================================================
=======================================================
   Kraft Chaos-Enhanced Fuzzer                        
=======================================================

Configuration:
  Nodes:        27
  Scenario:     mixed
  Max ops:      4000
  Max time:     300 s
  Seed:         1766106261
  Dynamic:      yes (escalating)

[ELECTION] Node runId=1766106261011813: votes=1, need=14, cluster=27
[PROMOTE] Node 2823994725 becoming LEADER in term 1 with 14/14 votes (cluster=27)
[LEADER-COMMIT] leader=22 term=1 committing idx=1 (remotelyLogged=13 +1leader = 14, need=14)
[LEADER-COMMIT] leader=22 term=1 committing idx=2 (remotelyLogged=13 +1leader = 14, need=14)
[LEADER-COMMIT] leader=22 term=1 committing idx=3 (remotelyLogged=13 +1leader = 14, need=14)
[LEADER-COMMIT] leader=22 term=1 committing idx=4 (remotelyLogged=13 +1leader = 14, need=14)
[LEADER-COMMIT] leader=22 term=1 committing idx=5 (remotelyLogged=13 +1leader = 14, need=14)
[LEADER-COMMIT] leader=22 term=1 committing idx=6 (remotelyLogged=13 +1leader = 14, need=14)
[LEADER-COMMIT] leader=22 term=1 committing idx=7 (remotelyLogged=13 +1leader = 14, need=14)
[LEADER-COMMIT] leader=22 term=1 committing idx=8 (remotelyLogged=13 +1leader = 14, need=14)
[LEADER-COMMIT] leader=22 term=1 committing idx=9 (remotelyLogged=13 +1leader = 14, need=14)
[LEADER-COMMIT] leader=22 term=1 committing idx=10 (remotelyLogged=13 +1leader = 14, need=14)
[LEADER-COMMIT] leader=22 term=1 committing idx=11 (remotelyLogged=13 +1leader = 14, need=14)
[LEADER-COMMIT] leader=22 term=1 committing idx=12 (remotelyLogged=13 +1leader = 14, need=14)
[LEADER-COMMIT] leader=22 term=1 committing idx=13 (remotelyLogged=13 +1leader = 14, need=14)
[LEADER-COMMIT] leader=22 term=1 committing idx=14 (remotelyLogged=13 +1leader = 14, need=14)
[LEADER-COMMIT] leader=22 term=1 committing idx=15 (remotelyLogged=13 +1leader = 14, need=14)
[LEADER-COMMIT] leader=22 term=1 committing idx=16 (remotelyLogged=13 +1leader = 14, need=14)
[LEADER-COMMIT] leader=22 term=1 committing idx=17 (remotelyLogged=13 +1leader = 14, need=14)
[LEADER-COMMIT] leader=22 term=1 committing idx=18 (remotelyLogged=13 +1leader = 14, need=14)
[LEADER-COMMIT] leader=22 term=1 committing idx=19 (remotelyLogged=13 +1leader = 14, need=14)
[LEADER-COMMIT] leader=22 term=1 committing idx=20 (remotelyLogged=13 +1leader = 14, need=14)
[LEADER-COMMIT] leader=22 term=1 committing idx=21 (remotelyLogged=13 +1leader = 14, need=14)
[LEADER-COMMIT] leader=22 term=1 committing idx=22 (remotelyLogged=13 +1leader = 14, need=14)
[LEADER-COMMIT] leader=22 term=1 committing idx=23 (remotelyLogged=13 +1leader = 14, need=14)
[LEADER-COMMIT] leader=22 term=1 committing idx=24 (remotelyLogged=13 +1leader = 14, need=14)
[LEADER-COMMIT] leader=22 term=1 committing idx=25 (remotelyLogged=13 +1leader = 14, need=14)
[LEADER-COMMIT] leader=22 term=1 committing idx=26 (remotelyLogged=13 +1leader = 14, need=14)
[LEADER-COMMIT] leader=22 term=1 committing idx=27 (remotelyLogged=13 +1leader = 14, need=14)
[LEADER-COMMIT] leader=22 term=1 committing idx=28 (remotelyLogged=13 +1leader = 14, need=14)
[LEADER-COMMIT] leader=22 term=1 committing idx=29 (remotelyLogged=13 +1leader = 14, need=14)
[LEADER-COMMIT] leader=22 term=1 committing idx=30 (remotelyLogged=13 +1leader = 14, need=14)
[LEADER-COMMIT] leader=22 term=1 committing idx=31 (remotelyLogged=13 +1leader = 14, need=14)
[LEADER-COMMIT] leader=22 term=1 committing idx=32 (remotelyLogged=13 +1leader = 14, need=14)
[LEADER-COMMIT] leader=22 term=1 committing idx=33 (remotelyLogged=13 +1leader = 14, need=14)
[LEADER-COMMIT] leader=22 term=1 committing idx=34 (remotelyLogged=13 +1leader = 14, need=14)
[LEADER-COMMIT] leader=22 term=1 committing idx=35 (remotelyLogged=13 +1leader = 14, need=14)
[LEADER-COMMIT] leader=22 term=1 committing idx=36 (remotelyLogged=13 +1leader = 14, need=14)
[LEADER-COMMIT] leader=22 term=1 committing idx=37 (remotelyLogged=13 +1leader = 14, need=14)
[LEADER-COMMIT] leader=22 term=1 committing idx=38 (remotelyLogged=13 +1leader = 14, need=14)
[LEADER-COMMIT] leader=22 term=1 committing idx=39 (remotelyLogged=13 +1leader = 14, need=14)
[LEADER-COMMIT] leader=22 term=1 committing idx=40 (remotelyLogged=13 +1leader = 14, need=14)
[LEADER-COMMIT] leader=22 term=1 committing idx=41 (remotelyLogged=13 +1leader = 14, need=14)
[LEADER-COMMIT] leader=22 term=1 committing idx=42 (remotelyLogged=13 +1leader = 14, need=14)
[LEADER-COMMIT] leader=22 term=1 committing idx=43 (remotelyLogged=13 +1leader = 14, need=14)
[LEADER-COMMIT] leader=22 term=1 committing idx=44 (remotelyLogged=13 +1leader = 14, need=14)
[LEADER-COMMIT] leader=22 term=1 committing idx=45 (remotelyLogged=13 +1leader = 14, need=14)
[LEADER-COMMIT] leader=22 term=1 committing idx=46 (remotelyLogged=13 +1leader = 14, need=14)
[LEADER-COMMIT] leader=22 term=1 committing idx=47 (remotelyLogged=13 +1leader = 14, need=14)
[LEADER-COMMIT] leader=22 term=1 committing idx=48 (remotelyLogged=13 +1leader = 14, need=14)
[LEADER-COMMIT] leader=22 term=1 committing idx=49 (remotelyLogged=13 +1leader = 14, need=14)
[LEADER-COMMIT] leader=22 term=1 committing idx=50 (remotelyLogged=13 +1leader = 14, need=14)
[LEADER-COMMIT] leader=22 term=1 committing idx=51 (remotelyLogged=13 +1leader = 14, need=14)
[LEADER-COMMIT] leader=22 term=1 committing idx=52 (remotelyLogged=13 +1leader = 14, need=14)
[LEADER-COMMIT] leader=22 term=1 committing idx=53 (remotelyLogged=13 +1leader = 14, need=14)
[LEADER-COMMIT] leader=22 term=1 committing idx=54 (remotelyLogged=13 +1leader = 14, need=14)
[LEADER-COMMIT] leader=22 term=1 committing idx=55 (remotelyLogged=13 +1leader = 14, need=14)
[LEADER-COMMIT] leader=22 term=1 committing idx=56 (remotelyLogged=13 +1leader = 14, need=14)
[LEADER-COMMIT] leader=22 term=1 committing idx=57 (remotelyLogged=13 +1leader = 14, need=14)
[LEADER-COMMIT] leader=22 term=1 committing idx=58 (remotelyLogged=13 +1leader = 14, need=14)
[LEADER-COMMIT] leader=22 term=1 committing idx=59 (remotelyLogged=13 +1leader = 14, need=14)
[LEADER-COMMIT] leader=22 term=1 committing idx=60 (remotelyLogged=13 +1leader = 14, need=14)
[LEADER-COMMIT] leader=22 term=1 committing idx=61 (remotelyLogged=13 +1leader = 14, need=14)
[LEADER-COMMIT] leader=22 term=1 committing idx=62 (remotelyLogged=13 +1leader = 14, need=14)
[LEADER-COMMIT] leader=22 term=1 committing idx=63 (remotelyLogged=13 +1leader = 14, need=14)
[LEADER-COMMIT] leader=22 term=1 committing idx=64 (remotelyLogged=13 +1leader = 14, need=14)
[LEADER-COMMIT] leader=22 term=1 committing idx=65 (remotelyLogged=13 +1leader = 14, need=14)
[LEADER-COMMIT] leader=22 term=1 committing idx=66 (remotelyLogged=13 +1leader = 14, need=14)
[LEADER-COMMIT] leader=22 term=1 committing idx=67 (remotelyLogged=13 +1leader = 14, need=14)
[LEADER-COMMIT] leader=22 term=1 committing idx=68 (remotelyLogged=13 +1leader = 14, need=14)
[LEADER-COMMIT] leader=22 term=1 committing idx=69 (remotelyLogged=13 +1leader = 14, need=14)
[LEADER-COMMIT] leader=22 term=1 committing idx=70 (remotelyLogged=13 +1leader = 14, need=14)
[LEADER-COMMIT] leader=22 term=1 committing idx=71 (remotelyLogged=13 +1leader = 14, need=14)
[LEADER-COMMIT] leader=22 term=1 committing idx=72 (remotelyLogged=13 +1leader = 14, need=14)
[LEADER-COMMIT] leader=22 term=1 committing idx=73 (remotelyLogged=13 +1leader = 14, need=14)
[LEADER-COMMIT] leader=22 term=1 committing idx=74 (remotelyLogged=13 +1leader = 14, need=14)
[LEADER-COMMIT] leader=22 term=1 committing idx=75 (remotelyLogged=13 +1leader = 14, need=14)
[LEADER-COMMIT] leader=22 term=1 committing idx=76 (remotelyLogged=13 +1leader = 14, need=14)
[LEADER-COMMIT] leader=22 term=1 committing idx=77 (remotelyLogged=13 +1leader = 14, need=14)
[LEADER-COMMIT] leader=22 term=1 committing idx=78 (remotelyLogged=13 +1leader = 14, need=14)
[LEADER-COMMIT] leader=22 term=1 committing idx=79 (remotelyLogged=13 +1leader = 14, need=14)
[LEADER-COMMIT] leader=22 term=1 committing idx=80 (remotelyLogged=13 +1leader = 14, need=14)
[LEADER-COMMIT] leader=22 term=1 committing idx=81 (remotelyLogged=13 +1leader = 14, need=14)
[LEADER-COMMIT] leader=22 term=1 committing idx=82 (remotelyLogged=13 +1leader = 14, need=14)
[LEADER-COMMIT] leader=22 term=1 committing idx=83 (remotelyLogged=13 +1leader = 14, need=14)
[LEADER-COMMIT] leader=22 term=1 committing idx=84 (remotelyLogged=13 +1leader = 14, need=14)
[LEADER-COMMIT] leader=22 term=1 committing idx=85 (remotelyLogged=13 +1leader = 14, need=14)
[LEADER-COMMIT] leader=22 term=1 committing idx=86 (remotelyLogged=13 +1leader = 14, need=14)
[LEADER-COMMIT] leader=22 term=1 committing idx=87 (remotelyLogged=13 +1leader = 14, need=14)
[LEADER-COMMIT] leader=22 term=1 committing idx=88 (remotelyLogged=13 +1leader = 14, need=14)
[LEADER-COMMIT] leader=22 term=1 committing idx=89 (remotelyLogged=13 +1leader = 14, need=14)
[LEADER-COMMIT] leader=22 term=1 committing idx=90 (remotelyLogged=13 +1leader = 14, need=14)
[LEADER-COMMIT] leader=22 term=1 committing idx=91 (remotelyLogged=13 +1leader = 14, need=14)
[LEADER-COMMIT] leader=22 term=1 committing idx=92 (remotelyLogged=13 +1leader = 14, need=14)
[LEADER-COMMIT] leader=22 term=1 committing idx=93 (remotelyLogged=13 +1leader = 14, need=14)
[LEADER-COMMIT] leader=22 term=1 committing idx=94 (remotelyLogged=13 +1leader = 14, need=14)
[LEADER-COMMIT] leader=22 term=1 committing idx=95 (remotelyLogged=13 +1leader = 14, need=14)
[LEADER-COMMIT] leader=22 term=1 committing idx=96 (remotelyLogged=13 +1leader = 14, need=14)
[LEADER-COMMIT] leader=22 term=1 committing idx=97 (remotelyLogged=13 +1leader = 14, need=14)
[LEADER-COMMIT] leader=22 term=1 committing idx=98 (remotelyLogged=13 +1leader = 14, need=14)
[LEADER-COMMIT] leader=22 term=1 committing idx=99 (remotelyLogged=13 +1leader = 14, need=14)
[LEADER-COMMIT] leader=22 term=1 committing idx=100 (remotelyLogged=13 +1leader = 14, need=14)
[LEADER-COMMIT] leader=22 term=1 committing idx=101 (remotelyLogged=13 +1leader = 14, need=14)
[LEADER-COMMIT] leader=22 term=1 committing idx=102 (remotelyLogged=13 +1leader = 14, need=14)
[ELECTION] Node runId=1766106261009500: votes=1, need=14, cluster=27
[PROMOTE] Node 2823992412 becoming LEADER in term 2 with 14/14 votes (cluster=27)
[LEADER-COMMIT] leader=1 term=2 committing idx=103 (remotelyLogged=13 +1leader = 14, need=14)
[LEADER-COMMIT] leader=1 term=2 committing idx=104 (remotelyLogged=13 +1leader = 14, need=14)
[LEADER-COMMIT] leader=1 term=2 committing idx=105 (remotelyLogged=13 +1leader = 14, need=14)
[LEADER-COMMIT] leader=1 term=2 committing idx=106 (remotelyLogged=13 +1leader = 14, need=14)
[LEADER-COMMIT] leader=1 term=2 committing idx=107 (remotelyLogged=13 +1leader = 14, need=14)
[LEADER-COMMIT] leader=1 term=2 committing idx=108 (remotelyLogged=13 +1leader = 14, need=14)
[LEADER-COMMIT] leader=1 term=2 committing idx=109 (remotelyLogged=13 +1leader = 14, need=14)
[LEADER-COMMIT] leader=1 term=2 committing idx=110 (remotelyLogged=13 +1leader = 14, need=14)
[LEADER-COMMIT] leader=1 term=2 committing idx=111 (remotelyLogged=13 +1leader = 14, need=14)
[LEADER-COMMIT] leader=1 term=2 committing idx=112 (remotelyLogged=13 +1leader = 14, need=14)
[LEADER-COMMIT] leader=1 term=2 committing idx=113 (remotelyLogged=13 +1leader = 14, need=14)
[LEADER-COMMIT] leader=1 term=2 committing idx=114 (remotelyLogged=13 +1leader = 14, need=14)
[LEADER-COMMIT] leader=1 term=2 committing idx=115 (remotelyLogged=13 +1leader = 14, need=14)
[ELECTION] Node runId=1766106261009066: votes=1, need=14, cluster=27
[PROMOTE] Node 2823991978 becoming LEADER in term 3 with 14/14 votes (cluster=27)
[ELECTION] Node runId=1766106261011284: votes=1, need=14, cluster=27
[PROMOTE] Node 2823994196 becoming LEADER in term 4 with 14/14 votes (cluster=27)
[TRUNCATE] node=0 term=4 truncating from idx=196 (commitIdx=115) newTerm=4 from leader
[TRUNCATE] node=1 term=4 truncating from idx=196 (commitIdx=115) newTerm=4 from leader
[TRUNCATE] node=9 term=4 truncating from idx=196 (commitIdx=115) newTerm=4 from leader
[TRUNCATE] node=24 term=4 truncating from idx=196 (commitIdx=115) newTerm=4 from leader
[ELECTION] Node runId=1766106261011372: votes=1, need=14, cluster=27
[PROMOTE] Node 2823994284 becoming LEADER in term 5 with 14/14 votes (cluster=27)
[TRUNCATE] node=0 term=5 truncating from idx=155 (commitIdx=115) newTerm=5 from leader
[TRUNCATE] node=1 term=5 truncating from idx=155 (commitIdx=115) newTerm=5 from leader
[TRUNCATE] node=8 term=5 truncating from idx=155 (commitIdx=115) newTerm=5 from leader
[TRUNCATE] node=9 term=5 truncating from idx=155 (commitIdx=115) newTerm=5 from leader
[TRUNCATE] node=16 term=5 truncating from idx=155 (commitIdx=115) newTerm=5 from leader
[TRUNCATE] node=24 term=5 truncating from idx=155 (commitIdx=115) newTerm=5 from leader
[ELECTION] Node runId=1766106261011461: votes=1, need=14, cluster=27
[PROMOTE] Node 2823994373 becoming LEADER in term 6 with 14/14 votes (cluster=27)
[ELECTION] Node runId=1766106261012166: votes=1, need=14, cluster=27
[PROMOTE] Node 2823995078 becoming LEADER in term 7 with 14/14 votes (cluster=27)
[TRUNCATE] node=18 term=7 truncating from idx=159 (commitIdx=115) newTerm=7 from leader
[ELECTION] Node runId=1766106261010635: votes=1, need=14, cluster=27
[PROMOTE] Node 2823993547 becoming LEADER in term 8 with 14/14 votes (cluster=27)
[ELECTION] Node runId=1766106261011641: votes=1, need=14, cluster=27
[PROMOTE] Node 2823994553 becoming LEADER in term 9 with 14/14 votes (cluster=27)
[TRUNCATE] node=0 term=9 truncating from idx=125 (commitIdx=115) newTerm=9 from leader
[TRUNCATE] node=1 term=9 truncating from idx=125 (commitIdx=115) newTerm=9 from leader
[TRUNCATE] node=2 term=9 truncating from idx=125 (commitIdx=115) newTerm=9 from leader
[TRUNCATE] node=5 term=9 truncating from idx=125 (commitIdx=115) newTerm=9 from leader
[TRUNCATE] node=8 term=9 truncating from idx=125 (commitIdx=115) newTerm=9 from leader
[TRUNCATE] node=9 term=9 truncating from idx=125 (commitIdx=115) newTerm=9 from leader
[TRUNCATE] node=16 term=9 truncating from idx=125 (commitIdx=115) newTerm=9 from leader
[TRUNCATE] node=17 term=9 truncating from idx=125 (commitIdx=115) newTerm=9 from leader
[TRUNCATE] node=18 term=9 truncating from idx=125 (commitIdx=115) newTerm=9 from leader
[TRUNCATE] node=24 term=9 truncating from idx=125 (commitIdx=115) newTerm=9 from leader
[ELECTION] Node runId=1766106261011016: votes=1, need=14, cluster=27
[PROMOTE] Node 2823993928 becoming LEADER in term 10 with 14/14 votes (cluster=27)
[TRUNCATE] node=0 term=10 truncating from idx=118 (commitIdx=115) newTerm=10 from leader
[TRUNCATE] node=1 term=10 truncating from idx=118 (commitIdx=115) newTerm=10 from leader
[TRUNCATE] node=2 term=10 truncating from idx=118 (commitIdx=115) newTerm=10 from leader
[TRUNCATE] node=5 term=10 truncating from idx=118 (commitIdx=115) newTerm=10 from leader
[TRUNCATE] node=8 term=10 truncating from idx=118 (commitIdx=115) newTerm=10 from leader
[TRUNCATE] node=9 term=10 truncating from idx=118 (commitIdx=115) newTerm=10 from leader
[TRUNCATE] node=16 term=10 truncating from idx=118 (commitIdx=115) newTerm=10 from leader
[TRUNCATE] node=17 term=10 truncating from idx=118 (commitIdx=115) newTerm=10 from leader
[TRUNCATE] node=18 term=10 truncating from idx=118 (commitIdx=115) newTerm=10 from leader
[TRUNCATE] node=20 term=10 truncating from idx=118 (commitIdx=115) newTerm=10 from leader
[TRUNCATE] node=24 term=10 truncating from idx=118 (commitIdx=115) newTerm=10 from leader
[ELECTION] Node runId=1766106261009500: votes=1, need=14, cluster=27
[PROMOTE] Node 2823992412 becoming LEADER in term 11 with 14/14 votes (cluster=27)
[TRUNCATE] node=0 term=11 truncating from idx=126 (commitIdx=115) newTerm=11 from leader
[TRUNCATE] node=2 term=11 truncating from idx=126 (commitIdx=115) newTerm=11 from leader
[TRUNCATE] node=5 term=11 truncating from idx=126 (commitIdx=115) newTerm=11 from leader
[TRUNCATE] node=8 term=11 truncating from idx=126 (commitIdx=115) newTerm=11 from leader
[TRUNCATE] node=9 term=11 truncating from idx=126 (commitIdx=115) newTerm=11 from leader
[TRUNCATE] node=13 term=11 truncating from idx=126 (commitIdx=115) newTerm=11 from leader
[TRUNCATE] node=16 term=11 truncating from idx=126 (commitIdx=115) newTerm=11 from leader
[TRUNCATE] node=17 term=11 truncating from idx=126 (commitIdx=115) newTerm=11 from leader
[TRUNCATE] node=18 term=11 truncating from idx=126 (commitIdx=115) newTerm=11 from leader
[TRUNCATE] node=20 term=11 truncating from idx=126 (commitIdx=115) newTerm=11 from leader
[TRUNCATE] node=24 term=11 truncating from idx=126 (commitIdx=115) newTerm=11 from leader
[ELECTION] Node runId=1766106261011898: votes=1, need=14, cluster=27
[PROMOTE] Node 2823994810 becoming LEADER in term 12 with 14/14 votes (cluster=27)
[TRUNCATE] node=0 term=12 truncating from idx=116 (commitIdx=115) newTerm=12 from leader
[TRUNCATE] node=1 term=12 truncating from idx=116 (commitIdx=115) newTerm=12 from leader
[TRUNCATE] node=2 term=12 truncating from idx=116 (commitIdx=115) newTerm=12 from leader
[TRUNCATE] node=5 term=12 truncating from idx=116 (commitIdx=115) newTerm=12 from leader
[TRUNCATE] node=8 term=12 truncating from idx=116 (commitIdx=115) newTerm=12 from leader
[TRUNCATE] node=9 term=12 truncating from idx=116 (commitIdx=115) newTerm=12 from leader
[TRUNCATE] node=13 term=12 truncating from idx=116 (commitIdx=115) newTerm=12 from leader
[TRUNCATE] node=16 term=12 truncating from idx=116 (commitIdx=115) newTerm=12 from leader
[TRUNCATE] node=17 term=12 truncating from idx=116 (commitIdx=115) newTerm=12 from leader
[TRUNCATE] node=18 term=12 truncating from idx=116 (commitIdx=115) newTerm=12 from leader
[TRUNCATE] node=20 term=12 truncating from idx=116 (commitIdx=115) newTerm=12 from leader
[TRUNCATE] node=24 term=12 truncating from idx=116 (commitIdx=115) newTerm=12 from leader
[TRUNCATE] node=26 term=12 truncating from idx=116 (commitIdx=115) newTerm=12 from leader
[LEADER-COMMIT] leader=23 term=12 committing idx=116 (remotelyLogged=13 +1leader = 14, need=14)
[LEADER-COMMIT] leader=23 term=12 committing idx=117 (remotelyLogged=13 +1leader = 14, need=14)
[LEADER-COMMIT] leader=23 term=12 committing idx=118 (remotelyLogged=13 +1leader = 14, need=14)
[LEADER-COMMIT] leader=23 term=12 committing idx=119 (remotelyLogged=13 +1leader = 14, need=14)
[LEADER-COMMIT] leader=23 term=12 committing idx=120 (remotelyLogged=13 +1leader = 14, need=14)
[LEADER-COMMIT] leader=23 term=12 committing idx=121 (remotelyLogged=13 +1leader = 14, need=14)
[LEADER-COMMIT] leader=23 term=12 committing idx=122 (remotelyLogged=13 +1leader = 14, need=14)
[LEADER-COMMIT] leader=23 term=12 committing idx=123 (remotelyLogged=13 +1leader = 14, need=14)
[LEADER-COMMIT] leader=23 term=12 committing idx=124 (remotelyLogged=13 +1leader = 14, need=14)
[LEADER-COMMIT] leader=23 term=12 committing idx=125 (remotelyLogged=13 +1leader = 14, need=14)
[LEADER-COMMIT] leader=23 term=12 committing idx=126 (remotelyLogged=13 +1leader = 14, need=14)
[LEADER-COMMIT] leader=23 term=12 committing idx=127 (remotelyLogged=13 +1leader = 14, need=14)
[LEADER-COMMIT] leader=23 term=12 committing idx=128 (remotelyLogged=13 +1leader = 14, need=14)
[ELECTION] Node runId=1766106261010533: votes=1, need=14, cluster=27
[PROMOTE] Node 2823993445 becoming LEADER in term 13 with 14/14 votes (cluster=27)
[TRUNCATE] node=0 term=13 truncating from idx=137 (commitIdx=128) newTerm=13 from leader
[TRUNCATE] node=1 term=13 truncating from idx=137 (commitIdx=128) newTerm=13 from leader
[TRUNCATE] node=2 term=13 truncating from idx=137 (commitIdx=128) newTerm=13 from leader
[TRUNCATE] node=5 term=13 truncating from idx=137 (commitIdx=128) newTerm=13 from leader
[TRUNCATE] node=9 term=13 truncating from idx=137 (commitIdx=128) newTerm=13 from leader
[TRUNCATE] node=13 term=13 truncating from idx=137 (commitIdx=128) newTerm=13 from leader
[TRUNCATE] node=17 term=13 truncating from idx=137 (commitIdx=128) newTerm=13 from leader
[TRUNCATE] node=18 term=13 truncating from idx=137 (commitIdx=128) newTerm=13 from leader
[TRUNCATE] node=20 term=13 truncating from idx=137 (commitIdx=128) newTerm=13 from leader
[TRUNCATE] node=23 term=13 truncating from idx=137 (commitIdx=128) newTerm=13 from leader
[TRUNCATE] node=24 term=13 truncating from idx=137 (commitIdx=128) newTerm=13 from leader
[TRUNCATE] node=26 term=13 truncating from idx=137 (commitIdx=128) newTerm=13 from leader
[ELECTION] Node runId=1766106261011898: votes=1, need=14, cluster=27
[PROMOTE] Node 2823994810 becoming LEADER in term 14 with 14/14 votes (cluster=27)
[TRUNCATE] node=0 term=14 truncating from idx=196 (commitIdx=128) newTerm=14 from leader
[TRUNCATE] node=1 term=14 truncating from idx=196 (commitIdx=128) newTerm=14 from leader
[TRUNCATE] node=5 term=14 truncating from idx=196 (commitIdx=128) newTerm=14 from leader
[TRUNCATE] node=8 term=14 truncating from idx=196 (commitIdx=128) newTerm=14 from leader
[TRUNCATE] node=9 term=14 truncating from idx=196 (commitIdx=128) newTerm=14 from leader
[TRUNCATE] node=17 term=14 truncating from idx=196 (commitIdx=128) newTerm=14 from leader
[TRUNCATE] node=18 term=14 truncating from idx=196 (commitIdx=128) newTerm=14 from leader
[TRUNCATE] node=24 term=14 truncating from idx=196 (commitIdx=128) newTerm=14 from leader
[ELECTION] Node runId=1766106261011016: votes=1, need=14, cluster=27
[PROMOTE] Node 2823993928 becoming LEADER in term 15 with 14/14 votes (cluster=27)
[TRUNCATE] node=2 term=15 truncating from idx=204 (commitIdx=128) newTerm=15 from leader
[ELECTION] Node runId=1766106261012166: votes=1, need=14, cluster=27
[PROMOTE] Node 2823995078 becoming LEADER in term 16 with 14/14 votes (cluster=27)
[TRUNCATE] node=0 term=16 truncating from idx=175 (commitIdx=128) newTerm=16 from leader
[TRUNCATE] node=1 term=16 truncating from idx=175 (commitIdx=128) newTerm=16 from leader
[TRUNCATE] node=2 term=16 truncating from idx=175 (commitIdx=128) newTerm=16 from leader
[TRUNCATE] node=5 term=16 truncating from idx=175 (commitIdx=128) newTerm=16 from leader
[TRUNCATE] node=8 term=16 truncating from idx=175 (commitIdx=128) newTerm=16 from leader
[TRUNCATE] node=9 term=16 truncating from idx=175 (commitIdx=128) newTerm=16 from leader
[TRUNCATE] node=13 term=16 truncating from idx=175 (commitIdx=128) newTerm=16 from leader
[TRUNCATE] node=17 term=16 truncating from idx=175 (commitIdx=128) newTerm=16 from leader
[TRUNCATE] node=18 term=16 truncating from idx=175 (commitIdx=128) newTerm=16 from leader
[TRUNCATE] node=23 term=16 truncating from idx=175 (commitIdx=128) newTerm=16 from leader
[TRUNCATE] node=24 term=16 truncating from idx=175 (commitIdx=128) newTerm=16 from leader
[ELECTION] Node runId=1766106261011641: votes=1, need=14, cluster=27
[PROMOTE] Node 2823994553 becoming LEADER in term 17 with 14/14 votes (cluster=27)
[TRUNCATE] node=0 term=17 truncating from idx=172 (commitIdx=128) newTerm=17 from leader
[TRUNCATE] node=1 term=17 truncating from idx=172 (commitIdx=128) newTerm=17 from leader
[TRUNCATE] node=2 term=17 truncating from idx=172 (commitIdx=128) newTerm=17 from leader
[TRUNCATE] node=5 term=17 truncating from idx=172 (commitIdx=128) newTerm=17 from leader
[TRUNCATE] node=8 term=17 truncating from idx=172 (commitIdx=128) newTerm=17 from leader
[TRUNCATE] node=9 term=17 truncating from idx=172 (commitIdx=128) newTerm=17 from leader
[TRUNCATE] node=13 term=17 truncating from idx=172 (commitIdx=128) newTerm=17 from leader
[TRUNCATE] node=17 term=17 truncating from idx=172 (commitIdx=128) newTerm=17 from leader
[TRUNCATE] node=18 term=17 truncating from idx=172 (commitIdx=128) newTerm=17 from leader
[TRUNCATE] node=23 term=17 truncating from idx=172 (commitIdx=128) newTerm=17 from leader
[TRUNCATE] node=24 term=17 truncating from idx=172 (commitIdx=128) newTerm=17 from leader
[TRUNCATE] node=26 term=17 truncating from idx=172 (commitIdx=128) newTerm=17 from leader

--- Chaos Fuzzer Statistics ---
Operations:     4000
Simulated time: 4805000 us (4.80 s)
Commands:       712 submitted, 128 committed

Elections:      185 total, 7 successful
Leader changes: 21

Messages:       27734 total
  Delivered:    26476
  Dropped:      0
  Delayed:      0
  Duplicated:   0
  Corrupted:    0

Partitions:     0 created, 95 healed
Node crashes:   109 crashed, 108 recovered
Node pauses:    109 paused, 0 resumed

Clock drifts:   61
Clock jumps:    0

Invariants:     81 checks, 0 violations
Chaos phases:   200
-------------------------------

=======================================================
   Result: PASSED (completed all operations)
   Seed: 1766106261  (reproduce with: -s 1766106261 -n 27 --scenario mixed)

stats


anthropic

yo. what is up with the ’throps these days?

another case of “we’re so big and rich and smart and have ultra brain wrinkles we don’t need or respect any real world experience in actual software or platform or architecture or infrastructure development at all!!! bow before the mighty AI pocket protector ethics club!!!! we don’t need no computer knowledge because we javascript our terminals!!!”

here’s my braindump log of anthropic problems over the past couple months:

general complaints

the persistent claude web ui problem over a year: the more you type, the slower it gets. the longer your session, the slower it gets, because they are using some bullshit SPA interface they don’t understand and generating unlimited anti-pattern design practices instead of using native high performance data-driven systems the web was built around and all browsers support natively (somebody pointed out like a year ago, for every character you type in the claude web ui, it completely re-draws and re-renders every character in the entire chat history and eventually your session just collapses)

in a long chat session with dozens of turns accumulating in the history, claude web ui makes my entire 10 core laptop with 64 GB RAM take 5+ seconds to print a single character after typing? what are you people even doing anymore? have you given up on understanding your platforms end-to-end with actual experience and knowledge from the ground up? or are you all “we only write js js can be fast and do no wrong duuuuuuuuuuuuurrrrrrrr js go bbbbbur<OOM Crash on a 1.5 TB RAM server only having your single chat page loaded>”

your “we use javascript to make react to make console UIs” thing can’t stop using infinite memory and showing seizure-inducing thousand of lines per second for minutes scroll back strobe flashes.

your “genius best in the world developers” for a year haven’t thought to block all / commands from being submitted as queued chat messages when a session is in progress. if you don’t have an active conversation/turn in progress -> /usage is CLI command (as expected) -> Active conversation turn running -> /usage gets forwarded AS QUEUED TO THE MODEL FOR MODEL RESPONSE? WTF ARE YOU DOING. THIS IS DAY ONE ENGINEERING USABLE EDGE CASE DESIGN WORK! DAY ONE. DAY ONE!

the dreaded claude cli “infinte compaction” loop where it compacts, outputs about 20 more tokens, then immediately compacts again, outputs another 20 tokens, compacts again, and will never continue, it just burns your rate limit with an inifinite compaction sequence and you have to start an entire new “Claude Context Brain” to continue working.

claude code input data structure incompetence: why does it take 3-5 seconds to “up-arrow-key for previous entry” prompt? what the hells bells are you doing behind the scenes? O(n^k) iterating over every character in the full chat buffer? trying to render pixels instead of characters? somehow searching, targeting, then redrawing every byte in the scrollback history to check if it should redraw to fit/syntax/style/permission check on every keypress again? first-principles reinventing interfaces from 40 years ago except without even standard curses/ncurses experience, optimization insight, performance targets, or any really care for usability or user latency experience at all? a goldfish living in a tuba who has never seen a computer before could make a better front-end-backend design.

claude code for web complaints

constant “disconnects” and you have to reload the page a thousand times a week

shows previous session snapshot on reload, then waits 5 seconds and “jumps to live”

random session hard crashes where it refuses to work and you have to start a new session and give it detailed manual instrutions about how to merge your previous branch into the new session branch

the classic “claude compacts history then forgets how to use your project” (a manual CLAUDE.md can largely help improve/fix this, but it should also be auto-maintained in the same session)

the UI bugs around billing where it flashed (current spend) -> You have the full available of $1,000 -> (current spend) -> flashing expiration dates back and forth (looks like shitty CDN caching issue or SSR-vs-SPA-vs-cache-vs-fetch/xhr issue because you don’t know how to make web platforms? ESPECIALLY AROUND BILLING DETAILS???? you know what’s also mondo ethical? not lying to people about credits in their account!)

differences between the app header showing details in-page (“Credits Expire Nov 23”) and the actual billing docs (“Credits Expire Nov 22 at 11:59 PT”) because “Nov 23” is still 1 full day, so “Expires Nov 23” imples “Expires when the DAY NOV 23 NO LONGER EXISTS” but you actually meant “expires when nov 23 begins existing”.

After the credits “expired” at Nov 22 midnight, the web UI WENT BACK TO SAYING “YOU HAVE 1,000 CREDITS EXPIRING NOV 23”

basic UI problems from people who have never used systems or platforms or used software before? The chat messages don’t have timestamps or metadata. You can’t tell if something happened 3 minutes ago or 3 days ago.

It has like 1/20th the features of the claude cli. You can’t even “recall” or edit a message you queued to send before it is “Read” but the remote model. whose idea was it to run a $1,000 credit promotion for a broken crashing failing platform to get more exposure? it makes me never want to waste time trying “claude for code for web for containers for wasting your time and locking up your browser and breaking your branches” ever again. first impressions matter. brand experience is a const identifier. how do you earn it back?

random false UI toast errors “you have exceeded your concurrency limit” — THE ENTIRE POINT OF YOUR “CLAUDE CODE FOR WEB ALL REPOS ALL THE TIME ANY TIME ANYWHERE ANY PLACE YOU WANT” PLATFORM IS YOU ALLOW UNLIMITED PROJECTS TO BE WORKED ON AT ONCE IN YOUR ASYNC CONTAINER BULLSHIT ENVIRONMENTS

can’t “up arrow” to recall previous prompts

one nice thing: you can click to expand hidden content when it’s too large for the default display on individual sections, where instead with claude code cli it is “expand your entire history and risk permanently corrupting your console session requiring a full application restart” instead

congrats on all being super geniuses and giving out “the hardest take home hiring tests in the industry we like hafta keep makin so much harder because our models can solve all our hiring tests (spoiler: then you aren’t measuring valid metrics at all for experience or ability)” and stuff, but like, what the hell are you even doing? and don’t get me started on your entire “we’re targeting $100 billion/year revenue soon” company just going 100% offline when cloudflare goes down because, you know, it’s not like you should be competent at running your own networks and infrastructure, right? nah, just outsource captcha bullshit to cloudfail so nobody can login and use your services and it’s fully out of your control for remediation or responsiblity or SLA or DR or DMT or BCP or MDMA or anything. great job, super ethics geniuses. the computer god appreciates your ambition even when faced with lack of experience to truly achieve your goals at scale.