Branch-aware selective testing
Reflection and notes on Branch-aware selective testing.
Trigger — - Recent commits touched git utilities, backlog handling, and selective test exemptions without a clear recorded strategy.
- Exploration objective encourages identifying specific, high-leverage reliability improvements rather than broad "run all tests" defaults.
Observations — - Current CI treats code and non-code PRs differently, but there is no documented mapping between changed paths and required test modules.
- TaskRouter, TaskSession, and SchemaMapper workstreams depend on timely feedback; full-suite pytest runs are expensive and often deferred.
- We already maintain file-level state (journal backfill paths, backlog append protections) that could feed a selective test planner.
Hypothesis — A lightweight selector that maps git diff paths → targeted test files will cut CI latency while keeping confidence ≥ current levels, especially for cortex/runtime changes.
Proposed Micro-Experiments — 1. Build a prototype mapper script that ingests git diff --name-only origin/main...HEAD and emits the smallest matching pytest node list using a hand-maintained rule table.
2. Track runtime + failure deltas: run selective tests first, then, in CI, gate on full suite until confidence is proven.
3. Iterate on rules for high-churn modules (TaskRouter, SchemaMapper) to make sure shared dependencies still run safety nets.
Risks and Mitigations — - Risk: Missing hidden coupling between modules → silent regressions.
Mitigation: Default to running umbrella integration tests when files outside the rule table change.
- Risk: Rule table drift as new modules land.
Mitigation: Treat rule table as code-reviewed config with ownership in cortex runtime team.
Next Steps — - Encode the initial rule table (router/session/schema paths) and wire a CLI entry point under tools/ for local use.
- Add backlog task to integrate with CI once prototype runtime savings are measured.
- Socialize approach with reliability stakeholders to confirm appetite before automating enforcement.