RISC-V parameter extraction

What this is

Work on extracting architectural parameters from RISC-V specification text into riscv-unified-db, done through an LFX mentorship with the Parameter SIG. The interesting part was not building another extractor. It was checking whether the existing published numbers meant what they were being read to mean.

The correction

The prompt used to produce the published recall figures builds every request through a function that unconditionally injects the complete list of gold parameter names — 185 of them, set-identical to the pinned ground truth — along with an instruction to use the exact name on a match.

So those figures measure grounding: given the catalogue, find which entries apply to a passage and cite the evidence for it. They do not measure whether a model can discover architectural parameters without being handed the answer key, which is the thing the number is naturally read as. The figures themselves were correctly measured and are unchanged. What changed is the claim attached to them.

I found this while building a variance experiment, corrected it in public on the project’s issue tracker, and every recall figure in the repository now carries the condition.

Two related findings came out of the same audit. Single-run recall is unstable — the same model, a byte-identical prompt and temperature=0 produced 33.9% and then 44.6%, so any single figure is one sample rather than a point estimate. And the gold labels for one parameter category are mostly not decidable from syntax at all: of 26 labels, 4 were decidable, 4 stale, and 18 undecidable. Two independent audits, one over schema shapes and one over the IDL, converged on the same four dissenting parameter names — an agreement now gated as an invariant in the verifier rather than left as a coincidence noticed once.

The same discipline retired a claim of my own. A first run suggested one behaviour collapsed when the catalogue was removed; the second run reversed direction, so what stands is the narrower statement that the contrast sits inside run-to-run noise at n=1.

The verifier

./verify.sh re-derives every registered figure from the artifact it came from and reports anything it cannot check rather than passing it. It runs offline, needs two pure data-handling dependencies, no credentials and no model calls, and CI runs the identical command on every push.

Six gates run. Five check what is published. The sixth checks what escaped being published as a claim — because a number that was never registered cannot be re-derived and so cannot fail. That gap is held at zero, which means a new figure added to a governed document without a claim behind it breaks the build.

What came of it

Sixteen merged pull requests upstream, none rejected, thirteen of them closing an issue that this measurement work had filed. Review comments from it were adopted into other contributors’ merged code, including one case where my own earlier advice was wrong, was corrected after someone had already built on it, and the merged code follows the correction.

One of them started as the mentorship’s own coding challenge: the challenge text described a cache block size as a power of two, and the database’s parameter did not actually enforce that. It does now.

Stack

Python · YAML / JSON Schema · RISC-V UDB + IDL · GitHub API (census) · offline claim verification