Compliance automation is mostly an evidence problem

I have spent a lot of the last year automating control assessments against Azure subscriptions. Going in, I assumed the difficulty would be in the evaluation logic — deciding whether a control passes. It is not. The evaluation is usually a query.

The hard part is evidence.

Pass is not a finding

A tool that reports AC-3: PASS has told an assessor nothing they can use. What they need is the answer to three questions:

1. What did you check? Not the control text — the actual resource, property and value.
2. When? Posture is a point-in-time claim and it decays.
3. How would I reproduce this? If they cannot re-run it, it is your word.

A finding that carries the subscription scope, the resource identifier, the property evaluated, the value found, the expected value, and a timestamp is evidence. A green checkmark is a feeling.

Not-assessed is a first-class result

The state that taught me the most is the one that is neither pass nor fail. A control can be un-assessable because the data source is not enabled — a Defender for Cloud plan that is off, a diagnostic setting never configured, a log that has no retention.

Early on I collapsed these into failures. That was wrong twice over: it inflated the failure count with things that were not control weaknesses, and it hid a pattern. When several unrelated controls all come back not-assessed, they are usually pointing at one missing telemetry source. That is a single, cheap fix that lights up a whole family of controls at once — and it only becomes visible if not-assessed is its own state with its own reason.

Exemptions need to be in the tool

Real environments have legitimate exceptions. A break-glass account will hold role assignments that a least-privilege control is right to flag and that you are not going to remove.

If the tool has no way to record that, one of two things happens: the finding gets ignored every cycle until nobody reads the report, or somebody edits the output. Both are worse than an exemption with an owner, a rationale, an expiry and a link to the approval. Azure Policy already models this well; assessment tooling should borrow the shape rather than invent one.

What the deliverable actually is

The output that gets used is not a JSON blob or a dashboard. It is a document someone can put in front of an auditor: scope, method, results grouped by control family, evidence per finding, exceptions with justification, and a signature block.

That is a boring thing to build and it is the entire product. The assessment logic is a weekend. The report is the year.