Reporting & Verification (V&V)

Every BaseCalibration/BaseIdentification run produces the same underlying metrics — condition number, per-DOF/per-joint residuals, parameter uncertainty, held-out validation. This guide covers the four ways those metrics get surfaced, and when to reach for each one.

Artifact	Call	Audience
Terminal quality report	automatic, after solve()	you, right now, in the terminal
Self-contained HTML report	solve(html_report=True) / export_html_report()	you, or anyone you send the file to
Machine-readable verdict (JSON)	verify() / export_verification_report()	CI, scripts, anything that needs a pass/fail
Static two-run compare page	figaroh.tools.compare_report.generate_compare_page()	comparing two exported verdicts, offline

All four read from data solve() already computed — none of them re-run the calibration/identification, and none of them require network access or a backend.

Interpreting a verdict

Use Plan, Fit and Validate to establish input correctness, parameter scope and independent validation before interpreting these reports. Inspect computed and skipped checks: a passing verdict with unavailable validation metrics does not establish held-out accuracy. Keep solver termination, physical feasibility, prediction quality and export/reload parity as separate conclusions. The compare page's compatibility checks do not establish matching raw inputs, processing, objectives or absence of leakage; retain that provenance.

Stages, splits and revisions

Each run records what each step did (figaroh.tools.stages, #55/#63). The terminal report prints them on a Stages: line, and the HTML report shows a Stages and data section.

Each step gets its own verdict in verdict.stages: - the steps are data, fit, validation, physical and export; - each is pass, fail, fallback or not_run, or not_evaluated when the run recorded nothing for it; - the existing entries (numerical_execution, prediction, solver) are kept beside them.

A validation fallback means the metrics were computed on the training data. It is never held-out evidence, and prediction acceptance cannot pass on it.

Field	Meaning
verdict.stage_records	every step's status, reason and metrics with units
verdict.selected_stage	the step whose parameters the result reports (fit; none if the fit failed)
verdict.splits	which files, sessions and sample counts trained and validated the run, and whether validation was held out
provenance software.git_commit	the working directory's commit (an example run: the examples repository)
provenance software.figaroh_revision	the figaroh checkout's own commit and dirty state

Together, the two commits name the paired core/examples revisions. The exported JSON carries the records under stage_records and a schema_version. The run archive writes stages.json and adds each run's stage verdicts to index.jsonl.

Terminal quality reports

print_quality_report() runs automatically at the end of solve(); call it again any time after a successful solve to reprint it:

calibrator.solve()
calibrator.print_quality_report()   # same output, printed again

For calibration this prints convergence status, outlier count, condition number, per-DOF residual statistics (mean/std/RMSE/max/R²), held-out validation (if validation_data_file is configured), and any parameter pairs with |ρ| > 0.8. For identification it prints base-parameter count, condition number, RMSE, correlation, per-joint torque residuals, the top-5 worst-identified base parameters, and (if enabled) physical-consistency / reconstruction status.

Self-contained HTML reports

calibrator.solve(html_report=True)
# or, after the fact:
calibrator.export_html_report(output_path="results/calibration_report.html")

The identification side is the same shape:

identifier.solve(html_report=True)
identifier.export_html_report(output_path="results/identification_report.html")

Each report is a single self-contained HTML file — no external requests, no CDN dependency, inline CSS/JS only, light/dark theme aware — so it opens correctly straight from disk and is safe to attach to an email or a ticket. It contains:

• Summary — the same headline numbers as the terminal report.
• Insights — auto-flagged issues (ill-conditioning, poorly-identified parameters, weak validation improvement, no validation data configured).
• Per-DOF / per-joint residuals and parameter uncertainty tables, each parameter's relative uncertainty rendered as a confidence-tier bar.
• Validation — nominal vs. fitted vs. measured, when a genuinely separate held-out dataset was configured.
• Before / after — an interactive overlay chart of the same nominal/fitted/measured series, with wheel-to-zoom and hover-to-inspect, so you don't have to read the numbers out of a table to see the fit improve. This needs no extra call — it's populated automatically whenever validation data is available.

Machine-readable verification

verify() checks the run's metrics against a set of pass/fail thresholds and returns a VerificationVerdict — this is the piece that turns a report (for a human to read) into something a script can branch on:

verdict = identifier.verify(scope="execution")
print(verdict.passed)          # bool
for check in verdict.checks:
    print(check.name, check.value, check.comparison, check.threshold, check.passed)

A bare library verify() defaults to prediction acceptance, which stays incomplete without a validation set and explicit limits. Routine example CLIs explicitly request numerical execution:

verdict = identifier.verify(scope="execution")
print(verdict.scope, verdict.status, verdict.stages)

This checks finite, nonempty fitted parameters, predictions, measurements and RMSE, matching effort dimensions, and whether explicitly requested validation loaded. Calibration additionally checks its reported optimization success and finite parameter/residual outputs. It does not certify data provenance, a full physical model or export/reload parity. Those stages remain not_evaluated.

There are no universal improvement, correlation, raw condition-number or calibration-error gates. These remain report diagnostics. A good nominal model may have little improvement; high correlation can coexist with biased torque; raw conditioning changes with parameter units and basis scaling.

For prediction acceptance, provide independent validation data and explicit application limits. For identification, supply an absolute maximum RMSE in Nm for every active validation joint:

profile = {
    f"validation_rmse:{joint}": {
        "threshold": allowed_error_nm[joint],
        "comparison": "max",
        "rationale": "measurement uncertainty and the application requirement",
    }
    for joint in active_joints
}
verdict = identifier.verify(scope="prediction", thresholds=profile)

Identification also exposes validation_abs_bias:<joint> and validation_peak_error:<joint> in Nm (or training_… for fallback). Add explicit limits where the application needs them; the per-joint RMSE limits remain required.

For calibration, use position_rmse_mm and, when rotational DOFs are measured, orientation_rmse_deg, with application-specific maximum limits. No replacement universal error value is supplied. Record units, the requirement behind each limit and the validation split before evaluating acceptance. Loading a separate file is necessary but does not prove experimental independence: split integrity, coverage and acquisition provenance remain the experiment owner's responsibility.

passed remains available for existing consumers and is true only if all required checks in the named scope pass. Additive status, scope, stages and applied policy fields make the meaning explicit:

• pass: all required checks in scope succeeded.
• fail: a required evaluated check failed, including computed NaN/Inf evidence.
• not_evaluated: required evidence or a prediction profile is unavailable.

Missing checks are retained with their reason, rather than silently skipped. An empty threshold-only policy is not_evaluated, never PASS. A spec may set required=False for an advisory check; it is still reported. A failed explicitly requested validation load prevents execution acceptance. Training fallback can support diagnostics but cannot pass prediction acceptance. Physical and export acceptance require their own evidence; a scoped prediction PASS is not a full physical-model approval. Compare historical verdicts with care: their unscoped boolean used the earlier universal gates, not this policy.

Write the verdict to disk as JSON (numpy-safe, includes git commit / config hash / timestamp provenance):

path = identifier.export_verification_report(
    output_path=None,       # defaults to {output_dir}/identification_verification.json
    output_dir="results",
    thresholds=None,        # explicit limits, when evaluating prediction
    scope="execution",
)

The same call exists on BaseCalibration (writes {output_dir}/calibration_verification.json).

Wiring --verify into a CLI script

This is the pattern already used by the identification.py entry points in figaroh-examples:

verdict = identifier.verify(scope="execution")
identifier.export_verification_report(output_path=str(run_dir / "verdict.json"), scope="execution")

for check in verdict.checks:
    status = check.status.upper()
    print(f"  [{status}] {check.name}: {check.value} ({check.comparison} {check.threshold:.4g})")

if not verdict.passed:
    sys.exit(1)

Run it as python identification.py --verify. The scope must be printed with the verdict: zero means numerical execution passed by default, not that an independent prediction or physical model was accepted. Prediction-scoped incomplete evidence produces a nonzero exit.

Comparing two runs

figaroh.tools.compare_report.generate_compare_page() renders a static, self-contained HTML page that loads two export_verification_report() JSON files client-side (drag-and-drop or a file picker) and, once they pass a compatibility check, shows a per-metric diff table and an overlaid before/after series chart with a per-run visibility toggle:

from figaroh.tools.compare_report import generate_compare_page

generate_compare_page(output_path="results/compare.html")

Open results/compare.html in a browser and load two verdict JSON files — everything after that runs entirely client-side; there's no server, no network request, and nothing is uploaded anywhere.

The compatibility check runs before anything renders. Two verdicts are compared only if they agree on:

• domain (calibration vs. identification — inferred from compat.dof_names vs. compat.active_joints)
• the same DOF/joint names
• the same decimate setting (identification only)
• comparable sample counts (within ~20%)

On a mismatch the page blocks the comparison and explains why (e.g. "Different decimate setting: false vs. true") rather than silently overlaying two runs that aren't actually comparable — you can still force the comparison via an explicit "compare anyway" checkbox, with a persistent warning banner while forced.

This is deliberately a static, no-backend artifact, not a run library or dashboard — see the design rationale in docs/decisions/external-tool-comparisons.md Part C if you're curious why (short version: no validated need yet for a run history/backend, and comparing incompatible runs silently was judged actively dangerous).

CI integration example

A minimal build-gate recipe combining the pieces above:

identifier.solve(html_report=True)
verdict = identifier.verify()
identifier.export_verification_report(output_dir="results")

if not verdict.passed:
    failed = [c.name for c in verdict.checks if not c.passed]
    print(f"Verification failed: {', '.join(failed)}")
    sys.exit(1)

Point your CI at results/*_verification.json as a build artifact, and at results/*_report.html if you want a human-readable report attached to the same run.