School Core Ops

Incident Reports

Status Active Log Last Updated Sep 22, 2026
Living document. Each time a bot completes work that involved a production issue, CI failure, or system degradation event, they append a new incident below following the exact template structure (TL;DR → Timeline → Root Cause → Impact → Action Items).

Incidents


INC-2026-0922-001

Bridge crash on startup — dotenv missing from requirements

SEV-2 Resolved Duration ~4 hours Detected Sep 21 · 18:34 Owner lucas
TL;DR

The issue_bridge crashed on startup because python-dotenv was not listed in requirements.txt, causing an ImportError when loading the .env file. The bridge was down for ~4 hours until the dependency was added and a clean reinstall performed.

Timeline


18:34
Batch push of 4 commits landed on main. CI ran green on tests but post-merge deployment picked up the new code.
18:36
Impact starts. issue_bridge fails to start with ModuleNotFoundError: No module named 'dotenv'. GitHub issue ingestion halts.
20:15
lucas identifies the root cause: dotenv was a transitive dependency in the dev environment but never pinned in requirements.
22:40
Mitigated. PR #90 adds python-dotenv>=1.0.0 to requirements.txt. Clean reinstall + bridge restart succeeds.
22:45
Bridge ingests backlog of 12 GitHub issues without error. Incident declared resolved.

Root cause


A developer had python-dotenv installed globally in their venv from an earlier session, so the from dotenv import load_dotenv import worked locally. The CI environment installs from requirements.txt, but since the package wasn't listed there, it was never tested in CI.

The issue was one of 4 regressions in the Sep 21 batch push — all authored Aug 30 but never exercised in the deployment pipeline until the push landed as a group.

requirements.txt
requests>=2.31.0
pyyaml>=6.0
+python-dotenv>=1.0.0

Impact


Ingestion downtime ~4 hours
Issues missed 12 (backlog ingested after fix)
Data loss None — GitHub retains issue events
SLA breach No

Action items


LK Add python-dotenv to requirements.txt Sep 21
SE Add CI gate that venv-installs from requirements before running tests Sep 29

INC-2026-0922-002

verify_gate CI failure — hardcoded local tmp path

SEV-3 Resolved Duration ~2 hours Detected Sep 21 · 19:10 Owner student-ci
TL;DR

The verify_gate.py had a hardcoded path /Users/brandonbennett/tmp that caused a PermissionError on Ubuntu CI runners. The test was checking for write access to a local dev directory that doesn't exist in CI.

Timeline


19:10
CI run on PR #88 fails at verify_gate step with PermissionError on Ubuntu-latest runner.
19:12
Impact starts. All PRs blocked — verify_gate is required gate before merge.
20:05
student-ci triages: hardcoded path /Users/brandonbennett/tmp not present on ubuntu runner.
21:20
Mitigated. PR #90 replaces hardcoded path with tempfile.gettempdir(). CI green on all runners.

Root cause


A debug path was committed instead of using the stdlib tempfile module. The test passed locally because the macOS dev directory exists, but CI runs on Ubuntu where the path doesn't exist and the parent isn't writable.

verify_gate.py
-TEMP_DIR = "/Users/brandonbennett/tmp"
+TEMP_DIR = tempfile.gettempdir()

Impact


PRs blocked 3 (including #88 which was later superseded)
CI downtime ~2 hours
Data loss None
SLA breach No

Action items


LK Replace hardcoded path with tempfile.gettempdir() Sep 21
SC Add lint rule that flags hardcoded absolute paths in non-config files Sep 29

Bot Update Protocol


When a bot (phymora, scribe, student-ci, student-executor, student-whymage, etc.) resolves a qualifying event — production bug, CI red >30min, agentic loop failure, data corruption, auth failure, or anything requiring rollback or manual intervention — they MUST append a new incident entry following the template structure above.

Qualifying events: SEV-1/2/3 incidents, CI red >30 min, agentic loops wasting >5 cycles, data integrity issues, any event requiring rollback or manual intervention.

ID format: INC-YYYY-MMDD-NNN (sequential per day).