Stop Asking the Black Box to Explain Itself
Explainability was never a property of the model. It is a property of the system you build around it.
On 29 June 2026, the Council of the European Union gave its final approval to the Digital Omnibus, and a date fixed for four years quietly moved. The heaviest part of the EU AI Act — the high-risk regime for the stand-alone systems listed in Annex III, which in Article 13 requires that a system be “sufficiently transparent to enable deployers to interpret a system’s output and use it appropriately” — now applies from 2 December 2027 rather than 2 August 2026. Sixteen months of runway, granted not because the problem was solved but because the machinery to comply with it — harmonized standards, notified bodies, national supervisors — was not built in time.
One requirement did not move. Article 50, the duty to tell a person they are dealing with a machine and to mark synthetic content, still lands on 2 August 2026. Set the two side by side and the Union has said something precise, if by accident: you must disclose that the machine is present, but you have another eighteen months before you must make it interpretable. The easy half kept its date. The hard half slipped. Oops!!
The reaction in most steering committees has been relief — which is the wrong response, and not because the delay is unwelcome. The deadline was never the reason to do the work; the delay has merely exposed how many firms believed it was.
The consensus is reading a reprieve where there is only a runway
The consensus take is that the explainability pressure has eased; the interpretability line on the roadmap slides down a few quarters, and the vendor briefings say “delayed to 2027.” That treats explainability as a compliance artifact — something you produce because a regulator asks, on the regulator’s calendar.
That is the mistake. Explainability is not, at root, a regulatory deliverable; it is an operating necessity you also happen to owe a regulator. You need it the afternoon a model declines a good customer and someone has to say why, and the morning an output drifts and the on-call engineer has to find the cause before the customers do, not the auditors. December 2027 changes none of that. We have spent a decade asking the wrong entity to do the explaining.
The trade-off is a category error
The received wisdom is a trade-off: more performance, less explainability. The powerful models are opaque, the transparent ones weak, and governance is the melancholy art of choosing a point on that line. It is drawn as a straight line because everyone has agreed, without quite noticing, on one hidden premise — that the explanation must come from inside the model.
Drop that premise and the line dissolves. Here is the inversion it hides, and the thesis of this essay: the most explainable AI systems are usually built on the least explainable models. Explainability is not extracted from a model by crippling it; it is engineered into the system around it.
We even named the problem after its own cure. We call an opaque model a “black box” — which is precisely what aviation calls the one device on the aircraft designed to make an unexplainable event explainable. The flight recorder is not the engine made transparent; it sits outside the engine, reconstructing what the engine will never confess. The explanation was never in the component. It was in the system around it.
What follows:
- The TRIZ move that dissolves the performance-versus-explainability trade-off instead of splitting the difference
- The three altitudes of explanation — model, system, institution — and why most programs never leave the ground floor
- What each altitude owes you, quoted from MAS, the HKMA and the EU AI Act
- The two audiences of every explanation — the reason a regulator needs, the diagnosis an engineer needs — and why one mechanism cannot serve both
- A diagnostic to run before Friday on your highest-stakes model, ending in a single uncomfortable number
The framework: explainability is an altitude, not a property
TRIZ — the inventive-problem-solving tradition built from studying how thousands of hard engineering contradictions were resolved — has a name for this move. When two requirements fight inside one system and no compromise satisfies both, you stop hunting for the compromise and *transition to the supersystem*: you let the larger system carry the property the component cannot. An engine cannot both burn hot for power and stay cool for longevity, so the cooling requirement moves outward, into a system wrapped around it. Nobody demands a cooler flame; they build a better surround.
Model performance and explainability are that contradiction exactly. The resolution is not a more interpretable model, bought with accuracy you cannot spare; it is a more intelligent system around the model you already run. Three altitudes, each with two moves — and most programs never leave the first, which is why they feel explainability as a tax on performance rather than a property of design.
Altitude One — The model: decide what the model does not owe you
1. Let the decision set the standard, not the architecture
The universal pattern. How much explanation a decision requires is a function of its stakes, not the model’s internals. A recommendation you can reverse needs almost none; a decision that denies a livelihood needs a great deal. The standard is set by the consequence, and can never be read off the architecture.
The regulated-industries manifestation. This is how MAS found mature banks operating. Its December 2024 information paper on AI Model Risk Management, from a thematic review of selected banks, observed that they applied global and local explainability methods and, in stronger cases, defined *the minimum level of explainability required for different use cases*. The same network can be fine behind a marketing ranker and unacceptable behind a credit refusal — nothing about the weights changed, only the altitude of the decision.
Your translation. You stop asking “is this model explainable?” and start asking “what does this decision require, and can our system produce it?” — set by model risk, the business owner and compliance together, before a model is chosen.
What to build by Friday. A one-page use-case explanation matrix: every live AI decision, its stakes tier, the standard that tier demands. The mismatches are your finding.
2. Separate the two audiences of every explanation
The universal pattern. An explanation for a customer or regulator is a different object from one for an engineer. The first is a reason — why this outcome, in language a person can act on or contest. The second is a diagnosis — which feature moved the output, in numbers a builder can debug. A feature-attribution chart is a fine diagnosis and a useless reason.
The regulated-industries manifestation. The regulators are unambiguous about which they want, and it is never the weights. MAS’s FEAT principles require that data subjects receive, on request, “clear explanations on what data is used to make AIDA-driven decisions about the data subject and how the data affects the decision,” and on “the consequences that AIDA-driven decisions may have on them.” The EU AI Act’s Article 86 grants a person subject to a high-risk decision the right to “clear and meaningful explanations of the role of the AI system in the decision-making procedure and the main elements of the decision taken.” Both ask for a reason a human can hold.
Your translation. Every high-stakes use case gets two explanation templates — a reason-shaped artifact for the customer and regulator, a diagnosis-shaped one for the engineer. Different mechanisms, different owners.
What to build by Friday. For your highest-stakes model, draft the two side by side. If you cannot fill the reason-shaped one without gesturing at a feature-importance plot, you have found the gap.
### Altitude Two — The system: build the thing that produces the explanation
3. Instrument the model; do not interrogate it
The universal pattern. You cannot make an opaque process honest by staring harder at it. You wrap it in instruments. Post-hoc explanation methods, counterfactual probes, a challenger model in shadow, complete input-output logging — none of these open the box, and none need to. They are sensors bolted to an engine no one can see into, and together they produce the account the engine never could.
The regulated-industries manifestation. The EU AI Act, tellingly, does not demand a high-risk model be inherently interpretable. It demands the *system* ship with the means to explain it: Article 13 requires the instructions for use to state “the technical capabilities and characteristics of the high-risk AI system to provide information that is relevant to explain its output” — the supersystem move, written into statute.
Your translation. Model risk owns an instrumentation specification, not a hope that the vendor’s model is legible: which explanation method attaches to which model, what is logged and for how long, whether a challenger runs alongside to catch the day the box and a transparent approximation disagree.
What to build by Friday. An instrumentation sheet for your highest-materiality model — explanation method, logging retained, any challenger. Blank cells are the build backlog.
4. Make the human-in-the-loop an explanation surface, not a rubber stamp
The universal pattern. Somewhere, a model’s output becomes an accountable human decision. That handover is the richest place in the whole system to capture a reason. A human who approves or overrides, and records why, converts a silent output into an explained decision — but only if the system is built to catch the sentence.
The regulated-industries manifestation. Hong Kong’s regulator has made this structural. The HKMA’s guiding principles on generative AI in customer-facing applications, issued 19 August 2024, require that customers be able to request human intervention and that institutions continuously monitor generative outputs; its Deputy Chief Executive clarified in April 2025 that the human-in-the-loop is a safeguard scaled to the risk of the use case.
Your translation. The human-review step is redesigned to capture the reason at approval or override — a structured field, not a signature — so review produces a logged explanation instead of mere delay. Done well, your reviewers become your largest source of genuine reasons.
What to build by Friday. A revised review protocol for one high-stakes workflow, in which every human decision records a one-line reason in a retrievable field.
Altitude Three — The institution: make the supersystem the unit of accountability
5. Audit the system, not the model
The universal pattern. Return to the aircraft. When a flight ends badly, the board does not interrogate the turbine and wait for a confession; it reads the recorder, the maintenance history and the crew testimony, and assembles an account the engine could never give. An institution that can only explain its AI by pointing at the model has no account at all.
The regulated-industries manifestation. This is exactly the shape of explanation Article 86 requires — “the main elements of the decision taken,” assembled from the record, not extracted from the network. MAS makes the stakes explicit in its AI Model Risk Management paper: explainability is vital for accountability, inside the institution and toward the customer.
Your translation. You can assemble, for any challenged decision, the documents that constitute its explanation — the input record, the model version, the instrumentation output, the reviewer’s reason, the governing policy. Not a diagram of the model. A dossier from the system, owned by governance.
What to build by Friday. A decision-reconstruction pack for one live high-stakes decision: the exact artifacts you would assemble, and where each lives today. The gaps are the parts of your supersystem that do not yet exist.
6. Put the explanation on a clock
The universal pattern. An explanation you cannot produce in time is not an explanation; it is an intention. Resilience engineering measures what matters by recovery time. Explanation is the same: not “could we, eventually, reconstruct why?” but “how many hours until we can?”
The regulated-industries manifestation. Article 86 gives an affected person the *right* to an explanation, and a right with no latency behind it is a right in name only. No regulator has yet named a number — this is the frontier — but the direction is one way.
Your translation. Time-to-explain becomes a tracked metric for your highest-stakes decisions: the measured hours to reconstruct, from the system, why one specific decision was made. Architecture and governance own the number.
What to build by Friday. An explanation runbook for your most consequential AI decision, ending in one number: hours-to-reconstruct. The first time you run it, that number will embarrass someone. That is the point.
None of this is unique to finance. A pharmaceutical company whose model flags an adverse-event signal, a hospital deploying a diagnostic aid — each will one day be asked not “how does your model work?” but “why did it decide *this*, for *this* person, on this day?” That is answered by the supersystem or not at all, under whichever regulator holds jurisdiction. Regulated intelligence is wider than financial services, and the altitude problem is identical across it.
A note on confidence
The reader deserves the line between what is documented and what is argued. Confirmed, and linked below: the Digital Omnibus, approved by the Council on 29 June 2026, postponing the stand-alone Annex III high-risk regime — Article 13 included — to 2 December 2027, while Article 50 transparency holds at 2 August 2026; the verbatim texts of Articles 13 and 86; the MAS FEAT transparency principles; the MAS AI Model Risk Management paper of December 2024; the HKMA’s August 2024 generative-AI principles. One caveat: the Omnibus still awaits publication in the Official Journal, and until it appears the original dates remain, technically, the law plan against the new timeline, document against the old.
Strong inference, from repeated observation rather than measurement: that most firms treat explainability as a model property, and are surprised by how little their instrumentation yields when a real explanation is demanded. I have seen this across jurisdictions; I have not seen it counted, and will not dress a pattern as a statistic. Design hypothesis, for argument: that time-to-explain is the metric this decade converges on, and that three altitudes is the right decomposition — fewer collapses the system into the model, more rebuilds the bureaucracy the supersystem was meant to replace. Disagree with that last one; I would like to be wrong cheaply.
What a heat-map cannot say
A diagnostic that takes an afternoon and tends to end the argument.
Ask your model-risk committee to produce, for one live high-stakes decision, the explanation a customer would actually receive. What comes back, reliably, is a feature-attribution chart — the third variable contributed 0.18, the seventh minus 0.04. A competent diagnosis and a worthless reason; no customer was ever consoled by a ranked list of partial derivatives. That gap is the whole distance between the model and the system you failed to build around it — visible in ten minutes, which is why so few teams look.
What the kit does
Knowing explainability lives in the supersystem is the easy half. Building it — turning “score the decision, instrument the model, reconstruct the account” into artefacts your risk and engineering teams can produce next week — is the work, and it is exactly the structured, repetitive work a well-made prompt does quickly.
This week’s Prompt Kit is four sequenced prompts: the first builds the use-case explanation matrix that sets each decision’s standard, the second designs the instrumentation layer around a chosen model, the third turns the human-in-the-loop into a logged explanation surface, the fourth assembles the decision-reconstruction pack and its time-to-explain number. It is this essay’s framework rendered as something you run against a real model, not admire as a diagram.
Run this in seventy-two hours
Take your highest-stakes AI decision — the one that most affects a real person. Imagine that tomorrow a customer, or a regulator behind them, demands a clear and meaningful explanation of one specific instance of it. Ask four questions. Who produces it — a person, or a vendor you have to email? From what — a dossier the system assembled, or a model no one can read? In what form — a reason the person can act on, or a chart only your engineers understand? And in how many hours? Most readers reach the recognition by the third question: they spent their effort trying to make the model talk, when they should have been building the system that speaks for it. The black box was never going to explain itself. It was never supposed to. That was always the supersystem’s job — and Brussels just gave you until December of next year to build it, which is not the same as permission to wait.
If you are building AI governance for high-stakes decisions — and want a structured outside read on your explanation standards, your instrumentation layer, or your institution’s ability to reconstruct a decision on demand — mention it in your reply.
I take one such conversation per month. Specific beats general: name the decision that would be hardest to explain under pressure, and we will start there.
Replies to this post reach me directly. I read all of them.