Deterministic Detector Evaluation Report (Reproducible)
This report shows what River Review's deterministic (LLM-free) heuristic detectors catch, and what they deliberately do not false-positive on, as a reproducible fixture regression. Every number here can be reproduced by anyone with a single command.
LLM-based review varies from run to run. Deterministic detectors, by contrast, return the same result for the same input, so their quality can be guaranteed mechanically with fixture regression tests. River Review guards this layer with canary cases (a collection of known false-positive patterns) so that a fixed false-positive never silently reappears.
Scope and method
- Detectors:
security-basic/logging-observability/test-existence/coverage-gap/altitude-generalization/closure-scope-retention. They run offline (--offline/ rules-only) and require no API key. - Fixtures: both cases that should be detected (true positives) and guard cases that must not be flagged (false-positive prevention).
- Assertion: each fixture is checked mechanically against its expectations (
mustIncludeandmaxFindings). - Source: the case definitions live in
tests/fixtures/review-eval/cases.json.
Results (as of 2026-07-10)
17 of 17 fixtures pass. The breakdown:
| Category | Detector(s) | Detection cases | Guard (no false positive) | Total |
|---|---|---|---|---|
| secrets | security-basic | 3 | 1 | 4 |
| observability | logging-observability | 2 | 1 | 3 |
| tests | test-existence coverage-gap | 5 | 1 | 6 |
| altitude | altitude-generalization | 1 | 1 | 2 |
| closure | closure-scope-retention | 1 | 1 | 2 |
| Total | 12 | 5 | 17 |
Guard cases confirm that safe patterns (for example a process.env reference rather than a secret, a catch block that logs, a diff that also updates a test file, a public option any caller may opt into, or a function that immediately reduces large data and releases it) are not flagged. They act as canaries that prevent known false positives from recurring (#1070).
How to reproduce
The command below reproduces the results above verbatim. It is deterministic, so every run yields the same output, and it needs no API key.
npm run eval:fixtures
Each case prints as [PASS] or [FAIL]. To add or change a case, edit tests/fixtures/review-eval/cases.json.
Scope of this report (honest limits)
- It covers the deterministic detectors only. LLM-driven skill accuracy (precision / recall) is not measured here.
- These numbers are one slice of quality; for production use, try River Review on your own real pull requests to confirm it fits.
- Unlike vendor self-reported benchmarks, the emphasis here is that a third party can verify the result with the same command.