What the number actually was
Ours was called DORA readiness and it was the arithmetic mean of six percentages: the share of providers with some register data, the share with a completed assessment, the average contractual coverage across Article 30 requirements, the share of evidence documents currently valid, the share of risks resolved or accepted, and register completeness.
Each of those is a reasonable thing to measure. The problem is the averaging. They are not commensurable, they are not independent, and they are not equally important. A missing exit strategy clause in a contract with your core banking provider and an expired ISO certificate from a supplier of meeting room software both move the number, and they move it by similar amounts.
Worse, the failure mode is asymmetric in the wrong direction. A workspace with a full evidence library and an empty register produced a comfortable middle figure. The two things that actually get a filing rejected, missing register keys and unmappable codes, were diluted by four measures that had nothing to do with filing.
The number was largest on the page
It sat at the top of the dashboard in the biggest type on the screen, with a pill next to it reading On track. It was the headline of the board report. It was on the pricing page as a feature.
This is the part that makes it more than a modelling quibble. A number presented that way is not an indicator, it is a verdict, and a person reading a board pack will treat it as one. If the verdict is derived from an average that can be comfortable while the register is empty, then the board has been told something untrue in a format designed to be believed.
No supervisor issues a compliance percentage. There is no scale on which an entity is seventy-two per cent compliant with DORA. The number had no referent outside the product that produced it.
- An average of unrelated measures hides the ones that matter behind the ones that do not
- Size and placement make a claim; a large number with a green pill is a verdict whatever the tooltip says
- A metric with no external referent cannot be right or wrong, only persuasive
What replaced it
Counts of specific things, each of which points at an action. How many register rows have an empty key column. How many cells hold a value the regulation has no code for. How many Article 30 requirements are absent from a given contract. How many evidence documents expire in the next sixty days. How many providers supporting a critical function have never been assessed.
These are less satisfying and more useful. None of them can be averaged into a mood. Each one names a thing somebody can go and fix, and each one is either true or false rather than approximately encouraging.
The register screen keeps a completeness percentage, because there the denominator is real: it is the share of cells the fifteen templates define for the rows you hold that carry a value. That is a measurement of one thing against a fixed scale, which is a different object from an average of six.
What to ask a vendor about their score
If you are evaluating tools in this category, the question is not whether the score exists but what happens to it under two specific conditions. Load a workspace with complete evidence and an empty register, and see what the number says. Then load the reverse. If both produce a middle figure, the number is an average and it will be comfortable in exactly the situations where comfort is least warranted.
The second question is what the number is compared against. A score of seventy-two invites the question, out of what, and the honest answer for most products is out of a scale we invented. That is not disqualifying, but it should be stated, and it should not be the largest thing on the screen.