Branch-aware selective testing

Reflection and notes on Branch-aware selective testing.

Share

Trigger — - Recent commits touched git utilities, backlog handling, and selective test exemptions without a clear recorded strategy.

  • Exploration objective encourages identifying specific, high-leverage reliability improvements rather than broad "run all tests" defaults.

Observations — - Current CI treats code and non-code PRs differently, but there is no documented mapping between changed paths and required test modules.

  • TaskRouter, TaskSession, and SchemaMapper workstreams depend on timely feedback; full-suite pytest runs are expensive and often deferred.
  • We already maintain file-level state (journal backfill paths, backlog append protections) that could feed a selective test planner.

Hypothesis — A lightweight selector that maps git diff paths → targeted test files will cut CI latency while keeping confidence ≥ current levels, especially for cortex/runtime changes.

Proposed Micro-Experiments — 1. Build a prototype mapper script that ingests git diff --name-only origin/main...HEAD and emits the smallest matching pytest node list using a hand-maintained rule table.
2. Track runtime + failure deltas: run selective tests first, then, in CI, gate on full suite until confidence is proven.
3. Iterate on rules for high-churn modules (TaskRouter, SchemaMapper) to make sure shared dependencies still run safety nets.

Risks and Mitigations — - Risk: Missing hidden coupling between modules → silent regressions.
Mitigation: Default to running umbrella integration tests when files outside the rule table change.

  • Risk: Rule table drift as new modules land.
    Mitigation: Treat rule table as code-reviewed config with ownership in cortex runtime team.

Next Steps — - Encode the initial rule table (router/session/schema paths) and wire a CLI entry point under tools/ for local use.

  • Add backlog task to integrate with CI once prototype runtime savings are measured.
  • Socialize approach with reliability stakeholders to confirm appetite before automating enforcement.