Opsmas 2025 Day 6: loopyring
loopyring
What is a loopyring?
Well, what if you combine loopy and ring?
You get loopyRing!
loopyRing combines the loopy event loop with the ring producer/consumer ring buffer allowing lock-free multi-core hierarchical task delegation like:
Level 0 (Input) Level 1 (Parse) Level 2 (Process) Level 3 (Output)
┌─────────────┐ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐
│ Listener │────▶│ Parser 0 │────▶│ Worker 0 │──────▶│ Replier 0 │
│ (accept) │ │ Parser 1 │ │ Worker 1 │ │ Replier 1 │
│ │ │ │ │ Worker 2 │ │ │
└─────────────┘ └─────────────┘ │ Worker 3 │ └─────────────┘
└─────────────┘Stages
A stage is a single worker instance within a level. Multiple stages at the same level process work in parallel. Each stage has:
- Its own thread
- Its own event loop (
loopyLoop) - Its own ring buffer (
selfRing) - Connection to input rings from upstream levels
Topology
The topology defines how levels connect:
- Level-wide attachment: All stages in level N read from all stages in level M
- Self attachment: Stages read from their own ring (external writers publish directly)
- Barrier routing: Entries flow through barriers for multi-phase processing
Architecture
we have a funky little architecture document with all levels of detail about how everything piles on together.
Threading Model
┌──────────────────────┐
│ Main Thread │
│ (loopyRingStart) │
└──────────┬───────────┘
│
┌──────────────────────────┼──────────────────────────┐
│ │ │
┌────────▼────────┐ ┌────────▼────────┐ ┌────────▼────────┐
│ Level 0 │ │ Level 1 │ │ Level 2 │
│ Thread 0 │ │ Thread 0 │ │ Thread 0 │
│ │ │ Thread 1 │ │ Thread 1 │
│ ┌─────────────┐ │ │ ┌─────────────┐ │ │ ┌─────────────┐ │
│ │ Event Loop │ │ │ │ Event Loop │ │ │ │ Event Loop │ │
│ │ (loopyLoop) │ │ │ │ (loopyLoop) │ │ │ │ (loopyLoop) │ │
│ └─────────────┘ │ │ └─────────────┘ │ │ └─────────────┘ │
│ ┌─────────────┐ │ │ ┌─────────────┐ │ │ ┌─────────────┐ │
│ │ Self Ring │ │ │ │ Self Ring │ │ │ │ Self Ring │ │
│ │ (buffer) │ │ │ │ (buffer) │ │ │ │ (buffer) │ │
│ └─────────────┘ │ │ └─────────────┘ │ │ └─────────────┘ │
└─────────────────┘ └─────────────────┘ └─────────────────┘Notification Mechanism
Workers sleep on file descriptors (pipes or eventfd on Linux) and wake when:
- Per-stage notification: Direct pipe write to specific stage
- Level-wide notification: Single write wakes all stages in level
// Notify single stage
pushNotification(stage); // Writes 1 byte to stage->pipe.writeFd
// Notify entire level
pushNotificationLevel(&lr->level[targetLevel]); // Wakes all workersData Flow
Producer Ring Buffer Consumer
──────── ─────────── ────────
│ │
│ 1. Get next slot ┌─────────────────────┐ │
├─────────────────────▶ │ ringPublisherMulti- │ │
│ │ EntryGetNext() │ │
│ └─────────────────────┘ │
│ │
│ 2. Write entry ┌─────────────────────┐ │
├─────────────────────▶ │ entry->content = x │ │
│ └─────────────────────┘ │
│ │
│ 3. Commit to barrier ┌─────────────────────┐ │
├─────────────────────▶ │ ringPublisherMulti- │ │
│ │ EntryCommit() │ │
│ └─────────────────────┘ │
│ │
│ 4. Notify consumer ┌─────────────────────┐ │
├─────────────────────▶ │ write(pipe, 1) │──────────▶│
│ └─────────────────────┘ │
│ │
│ 5. Wake & read │
│ ┌─────────────────────┐◀──────────┤
│ │ ringConsumerBarrier │ │
│ │ GetNext() │ │
│ └─────────────────────┘ │
│ │
│ 6. Process │
│ ┌─────────────────────┐◀──────────┤
│ │ processFn(entry) │ │
│ └─────────────────────┘ │
│ │
│ 7. Release │
│ ┌─────────────────────┐◀──────────┤
│ │ ringConsumerBarrier │ │
│ │ ReleaseEntry() │ │
│ └─────────────────────┘ │what’s it useful for?
great, but so what?
well, this is useful if you want to accelerate your magic 96-core or 288-core 3 TB RAM server into achieving perfect distributed concurrency data processing operating at the speed of contention-free memory reads and writes with no bulky syscall-triggering locks in the way.
pretty cool if you can swing it.
rando exampos here: https://github.com/mattsta/loopyRing/tree/main/examples
stats
===============================================================================
Language Files Lines Code Comments Blanks
===============================================================================
C 12 5103 3241 869 993
C Header 7 1902 1317 315 270
CMake 2 190 119 35 36
-------------------------------------------------------------------------------
Markdown 3 735 0 510 225
|- BASH 2 28 20 4 4
|- C 3 709 488 105 116
(Total) 1472 508 619 345
===============================================================================
Total 24 7930 4677 1729 1524
===============================================================================