0.3.0 evidence ledger
Acceptance status and reproducible evidence requirements for the 0.3.0 workbench scope.
0.3.0 evidence ledger
This page is the acceptance ledger for the 0.3.0 workbench scope. It separates implemented code from behavior that has been exercised at the public boundary. A capability is not marked complete because a type, binding, or handle exists. The ledger must be updated with a reproducible command, fixture, and observed result when a row changes.
Status vocabulary
- Exercised — the consumer-visible path was run and its expected result was observed. The evidence column identifies the test, fixture, or UI workflow.
- Partial — some implementation or lower-level coverage exists, but an acceptance boundary, error path, or integration path remains unverified.
- Missing — the required behavior is not implemented or no usable evidence exists.
- Blocked — implementation is present, but an external prerequisite (for example unavailable hardware or a provider) prevents the specified scenario. A blocked row is not a pass.
The status applies to the requirement, not to the apparent maturity of its
implementation. cargo check, a binding smoke test, or a non-empty result is
not evidence for an end-to-end row.
MolFrame acceptance matrix
| Area | Acceptance boundary | Status | Evidence / remaining proof |
|---|---|---|---|
| Structure identity | Author and label namespaces, insertion codes, absent fields, and ownership-preserving hierarchy accessors agree across Rust, Python, and source formats. | Partial | Native hierarchy and format changes are present in crates/molframe-py/src/hierarchy.rs and CIF/PDB paths. Exercise equal, unequal, absent, and insertion-coded identifiers through CIF, BCIF, PDB, and Python. |
| Canonical sequence | Canonical positions preserve gaps, observed substitutions, optional residue indices, atom ranges, and all applicable reference-alignment intervals. | Partial | The sequence model is an existing native boundary. Add evidence for explicit gaps, disjoint intervals, ambiguous alignments, and unmapped observed residues; ordinal 1..N inference is not acceptance evidence. |
| Chemistry and analysis | Component chemistry, provenance, coverage diagnostics, declared-SSE precedence, cached fallback, and the five interaction classes are exposed through public APIs. | Partial | Chemistry and DSSP-like native paths exist in crates/molframe-chem/ and crates/molframe-analysis/. Exercise missing-role diagnostics, provenance, declared-SSE precedence, and genuine interaction outputs. |
| Query language | Every supported query example parses, evaluates, prints canonically, and reports invalid spans and suggestions consistently across Rust, Python, and workbench. | Partial | Parser, AST, evaluator, and completion modules are present under crates/molframe-query/. Run the original example matrix, W4003 context checks, non-ASCII cursor checks, $selection dependency invalidation, and concurrent result ownership scenarios. |
| Verified acquisition | Fetch/cache hit, miss, offline, corruption, digest mismatch, bounded transfer, and concurrent materialization are distinguishable and atomic. | Partial | The verified adapter and CLI paths are present in crates/molframe-adapters/. Exercise a controlled provider and a real 1HHO acquisition; record provider/version, HTTPS origin, format validation, and whether an authoritative checksum exists. |
| Semantic selection | A pick resolves against structure/topology identity and instance provenance; click modifiers, granularity, stale events, ordering, and acknowledgement converge on one Session transaction. | Partial | Selection/session code and viewer interaction boundaries exist, but binding and UI behavior must be exercised with real canvas picks, rapid Shift/Alt operations, replacement during readback, duplicate events, and identical rows from different structures. |
| Sequence and focus UI | The mounted sequence model displays canonical gaps and numbering, emits compact native intervals, and focus creates editable retained layers with requested radius and controls. | Partial | Run the actual shared viewer in Chromium and AnyWidget with 4HHB and 1HHO. Verify tooltip, drag selection, layer edit/delete/undo, ligand preset radius, and acknowledged-selection highlighting. |
| Measurements and density | Exact atom references, units, degeneracy errors, retained measurement lifecycle, and real map sigma/provenance survive serialization, patching, undo, and round trips. | Partial | Native scene/science operations exist. Exercise two/three/four-atom gestures, stale cancellation, coordinate updates, map acquisition, sigma controls, and resource retirement; density picks must remain separate from atom selection. |
| Trajectory | Real frame/topology data, coalesced playback, pause drain, stable topology selections, and coordinate-revision invalidation are visible at the public boundary. | Partial | Run play/pause/scrub against an actual trajectory and verify frame/time, errors, stable selections, and measurement/analysis invalidation. A time setter alone is insufficient evidence. |
| Crystallography and comparison | Assembly/symmetry/unit-cell instance identity, native correspondence/superposition diagnostics, validation coloring, and provenance are exposed in the workbench. | Partial | Native domains exist. Exercise assembly picks, insufficient-correspondence errors, legends, missing data, and source/instance identity through Python/WASM and the viewer. |
| Snapshot and movie export | Retained layers, selections, measurements, camera, and resource references restore; movie output contains deterministic trajectory/camera samples and supports cancellation/cleanup. | Partial | Snapshot mechanisms are present under crates/molgfx-api/src/interop/. Decode a real movie and verify frame/time values; a screenshot sequence is not movie evidence. |
| Renderer lifecycle | Settled scenes stop RAF/submissions, completion metrics distinguish CPU, queue, callback, and GPU timing, in-flight ownership is bounded, and final completed frames are sampled. | Partial | The known Python run reported 65 tests with one established failure: test_a_settled_scene_does_not_schedule_more_browser_frames observed two frames instead of zero (python/tests/test_workbench_viewer.py:325). Fix at the producer and rerun the targeted regression and full suite. Unsupported GPU timestamps remain unavailable, not zero. |
| Cross-host release | Authoritative viewer source builds website and AnyWidget artifacts; actual Chromium, Firefox, WebKit, Jupyter, and available notebook hosts are exercised with capability skips recorded. | Partial | Use molgfx-viewer/package.json, CI, and manifests to determine exact commands before recording results. Generated widget output must be regenerated by the supported build, never hand-edited. |
Required evidence record
For each exercised row, record:
- the exact command or UI workflow and repository revision;
- fixture identity, source format, viewport/DPR/device, and relevant options;
- observed output, including expected failures and diagnostics;
- test discovery count and process exit status where a test runner is used;
- external limitations, unsupported capabilities, and raw artifact paths.
A skipped WebGPU backend, unavailable notebook host, provider outage, or missing redistributable chemistry resource must be recorded as a limitation; none converts a partial row into an exercised row. Keep implementation notes in the owning module and keep this page limited to acceptance evidence and status.
Measured performance against the specification targets
Release build on an Apple M5 Pro, Criterion with warm-up, revision 787a0a7 plus
the selection scans described below. Inputs are the embedded 1AON (58,870 atoms),
4HHB and smaller structures. Linear scaling to 100,000 atoms is stated as
extrapolation, not as a measurement.
| Target | Measured | Verdict |
|---|---|---|
mmCIF read to a Structure above 200 MB/s (program target 300 MiB/s) | cif_read: 231 MiB/s 1CRN, 216 MiB/s 1UBQ, 228 MiB/s 4HHB (3.20 ms), from 198–211 MiB/s | Spec target met; the 300 MiB/s program target is not. Top remaining frame: Lexer::next_token (about a third of samples), then the projection sink and number parsing |
| BinaryCIF container parse above 1 GB/s | 0.37–4.4 GiB/s (1CRN to 1AON) | Met for the larger inputs |
BinaryCIF read to a Structure above 1 GB/s | bcif_read/1aon: 7.53 ms for 2.77 MiB, 369 MiB/s end to end (13.96 ms, 199 MiB/s before); 4HHB 462 MiB/s. Split: bcif_decode_columns/1aon 2.40 ms, bcif_lower/1aon 5.13 ms (11.29 ms before) | Not met. Top remaining frames: per-row symbol lookups (symbol_of), decode_stage, ChunkBuilder::push |
Document to Structure for 100,000 atoms, under 20 ms | cif_lower/100k: 58.5 ms on 100,000 tiled 1AON atoms (69.5 ms before) | Not met |
| Time to first atom, 5 MB mmCIF | cif_first_atom/5mb: 12.8 µs (bounded-window batch source, first batch) | Met |
Full facade read (read_bytes) of 58,870 atoms | facade_enrich/1aon (default perception alone): 10.97 ms, from 31.3 ms; bond perception now runs in parallel cell blocks | Reported |
| Topological selection, 100,000 atoms, under 1 ms | chain A 0.014 ms, bfactor > 20 0.31 ms on 58,870 atoms; query_prune/element_zn_100k: 105 ns with chunk-statistics pruning (0.97 ms before) | Met |
| SASA, 100,000 atoms, 960 points, under 2 s | shrake_rupley_100k_960: 122 ms on 100,000 atoms (232.5 ms before), one streaming engine with lane-parallel occlusion | Met |
| Radius query, 100,000 atoms, 5 Å, under 100 µs | spatial_cell_list/radius_100k_5A: 294 ns against a prebuilt index | Met |
| Cell-list build, 100,000 atoms, under 5 ms | spatial_cell_list/build_100k: 1.37 ms (eager CellList::build_in) | Met |
| RMSD, 10,000 atoms | geom_coordinates/rmsd_10k: 6.68 µs (1.50 Gelem/s) | Reported |
| Coordinate hand-off to NumPy without a copy | facade_handoff/positions_slice: 0.83 ns; the slice pointer is asserted equal across calls | Met |
| Python import under 50 ms | 1.2 ms median over five subprocess imports of a release abi3 build; python/tests/test_import_time.py asserts the 50 ms budget | Met |
Column predicates used to cost one hierarchy walk and a binary search per atom.
Residue- and chain-level predicates now decide once per residue and return runs,
and the thresholded bfactor and occupancy columns read each chunk directly.
On 58,870 atoms chain A fell from 1.05 ms to 0.014 ms and bfactor > 20 from
0.72 ms to 0.31 ms; on 4HHB the typed and textual selection benchmarks fell from
44 µs and 93 µs to 17 µs. bfactor > 20 keeps most atoms, so the rest of its
cost is building the selection rather than comparing values.
Superposition and streamed block results
Same machine and method. QCP (3cd4e9c) was measured against the revision
before it (3cd4e9c^) and against its own rotation-free form on the 10,000-atom
fixture; the block-streaming changes were measured against the revision before
each.
| Change | Benchmark | Before | After |
|---|---|---|---|
| QCP superposition | geom_coordinates/superpose (small sample) | 18.515 µs | 18.245 µs |
| QCP superposition | geom_coordinates/rmsd, rmsd_10k | 3.206 µs, 6.698 µs | 3.198 µs, 6.717 µs |
| QCP, rotation built vs not | superpose_10k vs rmsd_after_fit_10k | — | 38.0 µs vs 37.9 µs |
| Stream accessible-area blocks | surface_geometry/lee_richards, surface_extended/surface_points | 31.79 ms, 377.6 µs | 31.56 ms, 365.6 µs |
| Stream cell-list pair blocks | spatial_pairs_within/CellList, spatial_scaling/cell_list/1aon | 625.5 µs, 8.999 ms | 590.9 µs, 8.745 ms |
| Stream bond-perception blocks | facade_enrich/1aon | 13.58 ms | 13.51–13.61 ms |
| Stream contact-potential blocks | analysis_potential/contact_potential_64_cubed | 9.4 ms | 7.9–11.9 ms (noise) |
QCP is performance-neutral: both fits are dominated by the single O(n)
covariance pass, so the 4×4 eigenproblem it replaces is not on the critical
path at these sizes. It is kept for its specified behaviour (a rotation-free
minimum deviation for streams, a Jacobi fallback on repeated roots), not as a
speed-up. The streaming changes are memory-bound: the peak is the result plus
a bounded block window rather than the result twice, and the bond and
contact-potential paths are equal within noise. The accumulator reductions in
molframe-spatial and the frame-block merge in molframe-traj keep
map_blocks_in: their contracts hand the caller one partial per block to merge
in one step, so there is no per-block output to stream without changing them.
Optimisations tried and reverted
Each was measured before and after on an Apple M5 Pro and reverted because the named benchmark did not improve.
| Change | Benchmark | Before | After |
|---|---|---|---|
| Parallel BinaryCIF column decode above 32,768 rows | bcif_read/1aon | 7.37 ms | 7.19 ms decode-only gain, bcif_lower +10% |
| Per-column class table in the mmCIF projection sink | cif_read/4hhb | 3.12 ms | 3.12 ms |
First-byte number prefilter in CifValueRef::parse | cif_read/4hhb | 3.12 ms | 3.11 ms |
| Whitespace lookup table in the lexer | cif_read/4hhb | 3.12 ms | 3.25 ms |
Packed-word bitmap in spatial within | spatial_pairs_within/within | 491–541 µs | 518–552 µs |
Experimental research pilot (2026-10-04)
This exploratory evidence does not promote a parity row or establish biological
prediction superiority. Portable workflows and reproduction commands are in
studies/semantic-stress/README.md; frozen inputs/reports live in the local
sibling ligare/data/molframe-research/experimental, not in a standalone clone.
The publisher SKEMPI 2.0 supplement contains 7,085 experimental records; a pinned
CSV mirror is checked against original scientific fields. The deterministic
fourteen-complex pilot retains 631 measures of 545 residues after excluding
censored/ambiguous measurements, discordant source rows, mismatched reference
and mutant temperatures, and an explicitly contradictory experimental WT identity.
The assay-condition amendment followed preliminary analysis, whose protocol and
results remain archived. It is not external preregistration.
| Boundary | Observed evidence |
|---|---|
| Experimental residue identity | 545 author-chain/residue/insertion/WT joins; independently checked with Gemmi |
| Admissible geometry pilot | 168 configurations: fourteen structures × twelve heavy-atom vdW policies |
| Domain kernel reference | 468 residue-degree comparisons against ProLIF over 1BRS, 1GUA and 4PWX; no differences |
| Wrong identifier namespace | 371 measured residues changed or lost across eight systems |
| Baseline / uniform policy median | Equal-weight AUROC 0.7223 / 0.7197; held-out balanced accuracy 0.6755 / 0.6871 across eleven two-class systems |
| Outcome dependence on policy | Contact degree changes for 204 of 545 measured residues |
| Sequence leakage screen | 473 cross-group chain comparisons; no pair has both >=30% identity and >=80% coverage under the recorded alignment |
Seven residues have conflicting measured hotspot labels and stay ambiguous. Three systems have a single class and retain unavailable AUROC/accuracy rather than being silently removed from the corpus. Bootstrap summaries condition on fitted predictions and do not establish confirmatory significance. Grouping uses source holdout metadata/shared names; the sequence screen is not exhaustive family validation. Experimental construct/state matching, external frozen validation and an additional biological task remain pending. A checked generic runner can perform the same identity safeguards; enumeration itself is not novelty.
A follow-up construct/state inventory covers all fourteen structures, 545
mapped residues and 631 retained measurements with all sixteen frozen input
hashes verified. Seven structures explicitly annotate mutation backgrounds;
twelve have selected-partner reference-sequence difference records. These
records can describe tags, variants or conflicts and require primary-methods
adjudication. Missing annotations remain unknown. For example, 1BRS has unknown
entity mutation fields but records ALA versus reference CYS at chain D positions
40 and 82. The inventory is experimental/construct-state-audit.json in the
external corpus; it leaves the pilot's sampling and estimates unchanged.
Corrected scientific definitions (post-0.6.0)
Four definitions were corrected against their published sources, each reproduced before the change and observed at the public boundary after it.
QS-score normalisation. The numerator is
sum w(min(d_ref, d_mod)) * (1 - |d_ref - d_mod| / 12) over shared residue
pairs, and the denominator adds the weight of every contact present on one side
only, exactly the weighted_scores / (weight_sum + weight_extra_mapped) of the
published definition. The previous kernel divided by the mean of the two
squared-weight masses. A single shared contact moved from 4 Å to 10 Å scored
0.129951 where the published score is 0.5; one shared plus one extra model
contact scored 2/3 where the published score is 0.5. Both are unit tests in
crates/molframe-compare/src/qs_tests.rs. On 4HHB chains A and B with the model
chain displaced 8 Å, the published 12 Å domain and the narrow 5 Å variant now
read 0.486286 and 0.045150 through Rust, Python and the compare --metrics qs
CLI.
Python QS default cutoff. 5 Å to the published 12 Å in qs_score and
mapped_qs_score, with the .pyi stubs updated. DockQ's 5 Å default is a
different metric and is unchanged.
Deposited connection classification. BondRecord carries the source's
_struct_conn.conn_type_id, and the mmCIF and BinaryCIF writers emit each
bond's own class. 4HHB's four deposited metalc Fe–histidine connections were
written as covale before the fix and are written as metalc now, confirmed by
the_four_deposited_metal_sites_keep_their_own_connection_class against the
shipped 4hhb.bcif and by molframe convert 4hhb.cif out.cif --generate-cif-connection-ids, whose four rows carry metalc.
TM-score superposition. The published score maximises the TM sum over superpositions; the code took it at a least-squares fit. On 1UBQ with one region displaced, that is 0.671257 instead of 0.691193; with a displaced tail, 0.886029 instead of 0.923243. The fit is now the deterministic reweighted search whose weights are the squared TM kernel.
Differential validation against reference libraries (2026-10-10)
A scratch workspace (/tmp, not committed) installed the reference
implementations and ran 92 checks over the domains that had no pinned evidence.
Every check names the library, its version, and the property compared; none was
tuned to the observed difference.
| Reference | Version | Domains | Result |
|---|---|---|---|
| gemmi | 0.7.5 | cell transforms, all 530 Hall settings, mmCIF atom/residue/element sets | 27 checks, all exact |
| biotite | 1.7.1 | distances, angles, dihedrals, Rg, centre of mass, SASA | 10 checks; Rg to 2e-14 Å, SASA total within 0.01% and 602/602 atoms inside the 960-point sampling bound |
| tmtools (TM-align) | 0.3.0 | TM-score on the 1UBQ Cα trace | exact for rigid motion; 0.04 lower on a hinge, recorded as a limitation |
| mdtraj | 1.11.2 | XTC, TRR, DCD, NetCDF read-back | coordinates within 2.4e-07 Å, atom and frame counts exact |
| CIAAW (IUPAC) | 2021 | atomic weights | 14/14 exact; gemmi 0.7.5 is the one carrying pre-2021 Ge and Se |
| biopython | 1.88 | BLOSUM62/50, PAM250, global and local affine alignment, FASTA, UPGMA, neighbour joining | 24 checks; every matrix entry and every alignment score exact |
| NumPy definitions | — | lDDT (Mariani), QS (Bertoni/OpenStructure), DockQ (Basu & Wallner), GDT (Zemla), contacts, HSE, minimum image, affine alignment | all exact |
The differentials are reproducible: a scratch run_all.py prints one line per
check with the reference it used, and CONSOLIDATED: 92/92 checks passed.
Three findings were not molframe defects, and are recorded here so the comparison is not misread later:
- mdtraj reports coordinates in nanometres while the files store ångström, which reads as a 10× difference until the unit is accounted for.
- gemmi regroups atoms by chain, discarding mmCIF file order; molframe preserves file order, so a positional comparison needs to be order-independent.
- mdtraj writes a present-but-zero 36-byte unit-cell block when it has no cell.
molframe refuses a degenerate cell, as GROMACS's own
do_trr_frame_datawould.
DockQ was checked against the published Basu & Wallner combination and its three components separately (fnat, LRMSD, iRMSD), each recomputed from the definition, at three displacements; all agree to 1e-15. GDT-TS and GDT-HA match the Zemla definition exactly, and the superposition RMSD matches TM-align's own reported value to 6.5e-08 Å.
The five results that are now pinned in the repository are the GW-041 to GW-045
workflows over three new files under crates/molframe/tests/reference/.
Python/Rust release-boundary checks (0.5.0, 2026-10-04)
The Python package remains a curated interface over native kernels, not a
one-for-one export of every low-level Rust type. The local Python suite passed
337 tests. The same locally built ABI3 wheel passed all 337 tests in isolated
environments on Python 3.12, 3.13 and 3.14; strict Pyright reported no errors
and --verifytypes reported 100% type completeness across 965 public symbols.
Type completeness alone is not numerical parity.
molframe-py tests directly compare Python and Rust centroid, distance matrix,
RMSD, TM-score and GDT on the same 1UBQ coordinates, plus internal-coordinate
seeds, Cartesian rebuilding, BAT measurement and BAT rebuilding. The numerical
array test runs in the Python CI/release job with NumPy on the embedded
interpreter's search path. Lengths retain Å; IC/BAT angles retain Rust radians,
while the established Python angle/torsion array operations return degrees.
The optional external-corpus test compared all 5,316 Ligare refined-set SDFs:
Python and Rust atom counts and every coordinate were identical. Reproduce it
with MOLFRAME_PARITY_CORPUS pointing to that acquired corpus, PYO3_PYTHON
pointing to the project interpreter, and PYTHONPATH containing its site-packages:
cargo test -p molframe-py python_and_rust_read_the_same_external_ligand_corpus -- --ignored --nocapture.
The corpus is not bundled and this check does not validate chemical truth.
The release workflow installs each built wheel into an isolated environment, asserts that import resolves inside that environment, and runs the Python contract suite on Linux x86-64/arm64, macOS x86-64/arm64 and Windows x86-64 before publication. All five platform builds completed successfully for 0.5.0, with 337 tests passing on each installed wheel. The Python release run records those gates and the successful PyPI upload.
The Rust 0.5.0 release completed its native gates and published all 25 crates.
Independent downloads verified their registry SHA-256 checksums and found
1,481 source/data files equal to the checkout. The manifests and local Python
version-test logs are retained in ligare/data/molframe-research/release-0.5.0.
The installed Windows wheel gate also exposed automatic CRLF conversion of
the embedded NCBI matrix bundle; its Git attributes now preserve canonical LF
bytes on every platform.