School-Core at Scale: 20+ Parallel Students

Design artifact — NOT yet implemented. Informs the next build phase. Pedagogy-first. Orca stays as the execution truth-source.

0. Why this document exists

The original pipeline was a single serial loop (issue_bridge.bridge_issues): one teacher, one whiteboard (the Mac's single Orca daemon), one student at a time. That loop proved the learning contract — a student produces verified, two-judge-reviewed, graded work, persisted to a bookbag. That contract is the project's own gate (fc7.3.6: "concurrency 1 … not permission to increase concurrency").

Scaling to 20+ students in parallel is not "a bigger whiteboard." It is a different topology: a school with a dispatch office, a fleet of classrooms (Orca daemons), and a grading department that reviews finished work asynchronously from dispatch. This doc designs that topology using the modules already present in the repo.

Orca is not replaced. Orca is where a coding student's code is actually written, built, and tested in a real worktree — the source of the verify contract's authority. Scaling means more Orca capacity (a fleet of daemons / machines), not a different substrate. Bot Mode (school_bot_execute) is kept for non-coding roles (research, triage, docs) that don't need a worktree.

1. The school metaphor → module map (already built)

The only two pieces missing for 20+ are: (a) a dispatch scheduler that fans issues out to available Orca worktrees, and (b) a grading queue so review is decoupled from dispatch. Everything else already exists.

2. Target topology (three tiers, producer/consumer)

            ┌─────────────────────────────────────────────┐
            │  CURRICULUM  (github_fetcher + capabilities) │  pull + classify issues
            └───────────────────────┬─────────────────────┘
                                     │ issues (domain, difficulty, role)
                                     ▼
            ┌─────────────────────────────────────────────┐
            │  DISPATCH OFFICE  (NEW: school-scheduler)     │  capability-aware assignment
            │  - admission via lock-safe registry          │  to AVAILABLE worktrees
            │  - bounded by fleet capacity                 │
            └───────────┬───────────────────────┬──────────┘
         dispatch N crews                grading jobs enqueued
                        ▼                        ▼
   ┌───────────────────────────┐      ┌───────────────────────────────┐
   │ ORCA FLEET (many daemons) │      │ GRADING QUEUE (NEW)           │
   │ each hosts K worktrees    │      │ finished crews → teacher bots  │
   │ student works + verifies  │      │ run two-judge + verify gate   │
   │ writes bookbag            │      │ score → lock-safe ledger      │
   └────────────┬──────────────┘      └───────────────┬──────────────┘
                │ report.md + verification              │ graded result
                ▼                                      ▼
            ┌──────────────────────────────────────────────────────┐
            │  OBSERVABILITY  (Kanban board, compound_learning)     │  one board, many lanes
            └──────────────────────────────────────────────────────┘

Dispatch and grading are independent pipelines joined only by the bookbag + a job record. A student finishing does not block the next dispatch; a grader stalling does not block the next student. Throughput is limited by the minimum of (fleet size, grader concurrency, ledger write throughput) — not by one teacher watching one whiteboard.

3. The two new components

3.1 Dispatch Office (school_scheduler.py) — IMPLEMENTED

Status: landed as of this slice. The dispatch office + fleet registry exist and are tested (tests/test_school_scheduler.py, 7 tests). Both new components (§3.1 + §3.2) are now in place; the topology is wired end-to-end (dispatch → grading queue → consumer) and ready for the cap-raise step.

• DispatchOffice.dispatch(...) owns the admission decision using the existing lock-safe decide_admission + _crew_active_count(CREW_RUNS_FILE) (the fc7.3 repair). dispatch_crew writes a running record on spawn, so a concurrent decision observes in-flight crews and configured_cap holds.
• Fleet registry (FleetRegistry, data/fleet.json): default is one local daemon with one worktree; add daemon entries (endpoint + worktrees + capacity) to scale to N without code change. Assignment is least-loaded-first with round-robin tie-break.
• Worktree lease (N6.2) + isolation pre-verify (N4.1) guard the chosen worktree so two crews never share one; a retry budget (N7.1) bounds spawn failures.
• Bounded by the existing CREW_MAX_PER_CYCLE (still 1). The office does NOT raise it; raising cap is the separate, gated step. At cap=1 the office is behavior-identical to the prior inline dispatch — issue_bridge.process_issues now delegates to get_dispatch_office().dispatch(...) and keeps all downstream run_task / fallback / grading logic untouched.
• On crew done, the grading job is enqueued by process_issues (after run_task, where the full review_packet/task_score exist) — not by the office, to avoid a key-deduped empty job shadowing the rich one.

3.2 Grading Department (school_grader.py + queue) — IMPLEMENTED

Status: landed as of this slice. The grading queue + consumer exist and are tested (tests/test_school_grader.py, 9 tests). The dispatch-office scheduler (§3.1) is the next slice.

• GradingQueue — file-backed durable JSONL queue (data/grading_queue.jsonl), fcntl-locked for concurrent enqueues from a fleet of dispatchers, dedup by (issue_number, crew_id) (N2.1).
• grade(job, ...) consumer — finalizes one finished job: two-judge acceptance decision (reuses ReviewPacket, never re-gates the clean base — N5.3), idempotent lock-safe ScoreStore write (N5.2 / N2.3 — replay = no-op), compound_learning observation (fail-soft), GitHub label via the non-fatal LabelWriteQueue (N7.2). Never raises.
• drain(queue, ...) — processes one pending snapshot in bounded waves (bounded_grader_pool_size, N6.1) so ledger writes never exceed what the lock-safe store can absorb. Failed jobs retain a bounded error and attempt count for the next drain; after three failed drains they remain visible as dead_letter entries in the JSONL queue. A job is attempted at most once in one drain call.
• CLI: python -m school_grader --drain [--score-store PATH] [--compound-store PATH] [--max-workers N] [--max-attempts N] — a standalone pipeline stage. Operators can inspect failures with --list-dead-letters and explicitly retry one with --requeue-dead-letter KEY; requeue resets the attempt budget while preserving the prior failure summary.
• The scheduled School Loop seeds data/grading_queue.jsonl from board-publish and checkpoints it with the other allowlisted state. The sanitizer handles each JSONL record structurally, preserving queue identity and retry/dead-letter fields while redacting sensitive keys and values. The fresh-checkout test verifies pending and dead-letter state survives restore; no live workflow execution is implied by this local wiring.
• Integration seam: issue_bridge.process_issues enqueues a GradingJob on every successful task (non-fatal). At cap=1 the inline finalization still runs (behavior unchanged); the queue is the ready hook for the 20+ separate grading stage. The consumer's idempotency means the inline+queue pairing never double-grades.

This extraction is mechanical: the review/grade logic already lived in one cohesive block; it is now reachable behind a queue consumer, decoupled from dispatch. The actual switch (inline finalization → separate drain stage) happens when concurrency rises, not before.

4. Must-fix before 20+ (small, known)

1. ScoreStore lock (scoring.py:save): open("w") + json.dump with no lock. At 20 concurrent graders this risks a torn write / leaderboard wipe on next load(). Fix: wrap save() in the same fcntl.flock pattern already used by the crew registry (_registry_lock). Trivial, high-value, should land regardless of scale work.
2. Orca fleet capacity: define how daemons are addressed (env / registry of Orca endpoints). Today one daemon + 30-min CI job timeout is the real ceiling; the fleet removes both.
3. CI lock relaxation: school-loop.yml school-core-live-orca concurrency group currently forbids two execute jobs. With a fleet, the lock should key per-daemon, not globally.

5. Discipline: respect the fc7.3.6 gate

fc7.3.6 (still OPEN) is the project's own check: prove the single-student full-path pilot passes before raising concurrency. This design does not bypass it — the 2-concurrency step (loop refactor + CREW_MAX_PER_CYCLE=2) remains the last validation of the single-student contract, and the fleet/queue work builds on top of that proven base. Recommendation: close fc7.3.6 first.

6. Incremental path (no big-bang rewrite)

1. Land the ScoreStore lock (§4.1) — independent, safe now.
2. Close fc7.3.6 — concurrency-1 pilot, prove contract.
3. 2-concurrency — loop refactor + raise cap (already designed earlier).
4. Extract grading into a queue consumer (§3.2) — decouples grading.
5. Dispatch office + Orca fleet (§3.1) — the actual 20+ scale-out.

Each step is independently testable and keeps Orca as the classroom.

7. Open questions for the architect (you)

• Fleet shape: multiple Orca daemons on one Mac, or across machines? (Sets the addressing model in §4.2.)
• Grading queue: in-repo file broker (matches current zero-dep style) or a real broker? (Recommend file-backed for now.)
• Does the 2-concurrency loop refactor (step 3) still make sense as an intermediate, or go straight to the scheduler once fc7.3.6 closes?