Bluefin QA — Methodology
This page documents the statistical methods used to compute every derived metric on factory.projectbluefin.io. Every panel on the site that displays a number with a confidence interval references a row in the table below. Click the notebook link to audit the methodology and re-execute it against the raw inputs.
Derived metrics
| Metric ID prefix | What it measures | Estimator | Window | Notebook | Citation |
|---|---|---|---|---|---|
trust_window_pass_rate | Per (variant, branch, suite) pass rate over an audience window. | Wilson score interval, 95% | 24h / 7d / 30d | 01_pass_rate_wilson.html | Wilson, E. B. (1927). JASA 22(158):209–212. |
reproducibility_rate | P(image_digest(rebuild) == image_digest(prior) given identical inputs). | Wilson score interval, 95% | rolling 30 nights | planned: 03_reproducibility | Reproducible Builds Project methodology |
provenance_completeness | Fraction of SLSA edges that verify under cosign verify-attestation. | Wilson score interval, 95% on edge counts | per-image | planned: 04_provenance_walk | SLSA v1.0; in-toto Attestation Framework |
regression_localization_latency | Median wall-time from regression onset (BOCPD change-point) to identified commit. | BOCPD posterior + git-blame attribution | per-detection | planned: 02_changepoint | Adams & MacKay (2007). Bayesian Online Changepoint Detection. arXiv:0710.3742 |
Failure Mode and Effects Analysis (FMEA)
Every derived metric record on this site carries a failure_modes array — a structured list of declared ways the metric's value could be wrong. Each failure mode has an id, a description, an active boolean (computed by the notebook from data heuristics), and where applicable a threshold andevidence_url.
Panels in the dashboard surface a warning chip whose count equals the number of currently active failure modes. Click the chip to expand the list. FMEA is reliability engineering's standard tool (IEC 60812); applying it to dashboard metrics makes the dashboard's own failure modes a first-class signal.
State vocabulary
The two state enums used on the site are locked:
Station status:partialdegradedunknownnot_attestednot_publishedblockedpublished
Snapshot / metric state:freshstaleunknownpartialdegraded
New state values cannot be introduced without amending bothtests/unit/test_factory_dashboard_contract.py andschemas/v2/derived-metric.schema.json in the same change.