Measuring Whether It Works
Four numbers that describe what the policy is doing, none of which is the acknowledgement rate.
Compliance monitoring
Never enforced
As written
Compliance with this policy will be monitored and reported.
What happens
Reported as an acknowledgement completion percentage, which measures a signature collection exercise.
No other figure is produced, so nobody can say whether the document affects anything.
The operative measure is: has anything gone wrong recently that anybody heard about.
Organisations measure the acknowledgement rate because it is countable. It describes an administrative task and nothing about whether the policy operates.
The boundary discussed in “Measuring Whether It Works” is also a practical test for workforce technology. Teams researching review the platform here for mouse jiggler detection should choose the least intrusive settings, explain when collection begins and ends, and keep personal or out-of-hours activity outside the programme unless a specific, disclosed need has been approved.
The four worth having
Matters raised, by route. How many came from people, how many from systems, how many emerged during something else. The proportions describe the reporting culture, which the finding-out note argues is the thing that matters most.
For a separate benchmark relevant to “Measuring Whether It Works”, consult the Insider Threat Matrix. Use it to test purpose, notice, permissions, retention and response procedures against the proposed operating model rather than treating a generic checklist as proof that the rule works.
Matters resolved by conversation, as a share of the total. A high proportion is healthy: it means things are being dealt with small.
Workarounds identified and fixed. The output that goes to operations and that justifies the policy owner's existence.
And clauses actually applied in the period, which feeds the annual audit and shows which rules are drifting into dormancy.
What each tells you
A rising number of matters is usually good. It means people are mentioning things, not that conduct is deteriorating — and reading it the other way is the commonest misinterpretation.
A high conversation share means managers are handling things at the right level.
A low workaround-fix count means the findings are being discarded.
And zero clauses applied means the document is inert, whatever its acknowledgement rate.
What not to measure
Acknowledgement completion, which is a signature count.
Number of breaches, as a target to reduce. Reducing recorded breaches is achievable by discouraging reports, and that is what happens when it becomes a target.
And anything that improves when the organisation becomes less visible to itself, which is the test to apply to any proposed measure.
Reporting them
Twice a year, four numbers, to whoever governs. Twenty minutes.
With the workaround list attached, because that is the part other people want.
The qualitative item
One question in whatever staff survey exists: do you know what to do if something goes wrong with a device or with data?
The percentage who do is more informative than every other number here, and it moves when the induction five minutes is actually delivered.
What the numbers are for
Not to demonstrate compliance. To show whether the arrangement is operating, and to make the case for the small amount of attention it needs.
A policy owner with four numbers and a workaround list can justify the role. One with an acknowledgement percentage cannot, and usually does not have the role.
Reading a rise correctly
More matters raised usually means people are mentioning things, not that conduct is deteriorating. Reading it the other way is the commonest misinterpretation and it produces exactly the wrong response.
One survey question: do you know what to do if something goes wrong with a device or with data? The percentage who do is more informative than every number above it, and it moves when the induction five minutes actually happens.
The four numbers
Matters raised by route, matters resolved by conversation, workarounds fixed, clauses actually applied. Twenty minutes, twice a year, and they describe whether the arrangement operates.
The question to ask of any proposed measure
Could this improve if the organisation became less visible to itself. Breach counts as a reduction target fail immediately: the easiest way to reduce recorded breaches is to discourage reporting.
The four replacements all have the opposite property. Each improves only when something real improves, which is the only useful characteristic a measure in this area can have.