Every article in my agentic lakehouse series has quietly assumed a plane that sees everything: the catalog’s audit stream, the semantic layer’s request log, the gateway’s traces, the agents’ case histories. That observability plane was built for operations, debugging, and cost, and in 2026 it acquired a second constituency with subpoena power. The EU AI Act’s high-risk obligations, arriving on the statute’s August 2026 clock with record-keeping at their operational core, make automatic, durable, inspectable logging a legal property of AI systems, not an engineering preference, and the organizations deploying agents against enterprise data are discovering, some by planning and some by counsel’s memo, that their telemetry is no longer just how they understand their systems. It is how they prove them, to assessors, to authorities, and eventually to courts.
This article is about building that telemetry properly, and the thesis is that the lakehouse the series already built is the right evidence substrate: AI decision records as governed Apache Iceberg tables, immutable by snapshot, retained by declared policy, addressable by tag for any auditor’s as-of question, queryable at analytical scale, and, the property the title names, policy-aware in both directions, recording the policy decisions the governed stack made while being themselves objects of policy, access-scoped, retention-governed, and erasure-capable. Compliance telemetry engineered this way is not a parallel system bolted on for the regulator. It is the estate’s existing observability plane, hardened to evidentiary grade, serving operations, economics, and law from one set of tables.
The plan: the regulatory moment stated honestly, timeline, scope, and the moving parts, what the Act’s record-keeping provisions actually demand translated into engineering, why Iceberg specifically fits the evidence problem, the telemetry schema and table design, the oversight ledger the assessors weight, the both-directions policy awareness, the capture pipeline through the layered architecture, the auditor-facing evidence workflows, the retention-versus-erasure tension every privacy lawyer will raise, and the worked example of an estate assembling it. A disclosure as always: I work at Dremio, whose platform operates the layers this telemetry instruments, and a second disclaimer this topic requires: I am an engineer writing about building systems, not a lawyer writing about complying with law, and the regulatory reading here is the widely-shared engineering interpretation, to be confirmed with counsel for any specific obligation.
The Regulatory Moment, Stated Honestly
The EU AI Act, in force since August 2024, phases in by risk tier, and the phase that matters for this article is the high-risk tier, whose obligations were set by statute to apply from August 2, 2026, covering the Annex III system categories, credit scoring, employment screening, essential-services eligibility, and their siblings, with provider requirements across risk management, data governance, technical documentation, logging, transparency, human oversight, and accuracy and resilience, deployer obligations alongside, conformity assessment and registration machinery around it all, and penalties that reach into the tens of millions of euros or mid-single-digit percentages of global turnover.
The honest complication is the calendar’s fine print. The Commission’s Digital Omnibus package, proposed in late 2025, moved to defer many high-risk obligations, with backstop dates stretching toward the end of 2027 for standalone Annex III systems, and the proposal spent 2026 in the legislative machinery, negotiating positions and trilogues, while the statutory date approached and, as this article publishes, arrives. Adjacent obligations move on their own clocks, the general-purpose model duties applying since 2025 with enforcement teeth from 2026, and the transparency tier’s content-marking requirements landing on the August 2026 boundary with their own grace periods for systems already in market.
The engineering posture that survives the calendar ambiguity is the one the serious compliance guidance converged on: build to the substantive obligations, treat any deferral as schedule relief rather than scope relief, because the requirements themselves are not softening in any proposal on the table, and record-keeping specifically is the obligation with the longest lead time to retrofit, since evidence not captured at decision time cannot be captured later. Organizations waiting for the dust are betting that the systems they deploy this year will never need to prove what they did this year, which is a bet the retention clocks make explicit: the logs the Act contemplates are the logs of the period regulators will eventually examine, and that period includes now.
Scope deserves its honest paragraph too, because “we don’t build high-risk AI” is the sentence most enterprises say one inventory too early. Annex III’s categories are use-based, not technology-based: the agentic estate’s assistant becomes relevant not by being an agent but by what its outputs feed, and the classification exercise, mapping each deployed system’s actual uses against the Annex, is the program’s real first step, typically finding a small high-risk core, a larger adjacent tier where the classification is arguable, and a long tail that is clearly out of scope. The telemetry architecture this article builds serves the whole inventory deliberately: evidentiary-grade for the high-risk core, and cheap enough to run estate-wide, because the adjacent tier’s classification can change with a use case, a guidance document, or an enforcement action, and the estate that logged everything governedly gets to respond with a scope change instead of a time machine.
What the Act Actually Asks of Logging
The record-keeping provisions read short and land heavy, and translating them into engineering requirements is this section’s job.
The core is Article 12’s automatic logging mandate: high-risk systems must be technically capable of automatically recording events over their lifetime, with the recording aimed at three named purposes, identifying situations where the system presents risk or undergoes substantial modification, facilitating post-market monitoring, and supporting the deployer’s operational monitoring. Read as engineering, the mandate has four load-bearing words. Automatic: capture is built into the system’s operation, not reconstructed from application logs after the fact, which is an instrumentation-architecture requirement. Events: the unit is the operationally meaningful occurrence, the decision, the input, the intervention, not the debug line. Lifetime: capture spans the system’s whole deployed existence, across versions and re-deployments, which makes version threading a schema requirement. And capable: the capability must exist by design, demonstrable at conformity assessment, which means the logging is itself a documented, tested component.
Around the core sit the custody clocks: providers keep the logs under their control per their documented periods, deployers retain the automatically generated logs for a floor of six months, longer where sectoral or national law says so, and financial-sector deployers inherit their industry’s sterner retention habits on top. The engineering reading: retention is declared, per record family, with the declaration traceable to the obligation that set it, and six months is a floor that real programs, mindful of multi-year investigation windows, routinely exceed by design.
And around the clocks sits the demonstration machinery: post-market monitoring plans that consume the logs, serious-incident reporting that reconstructs from them, technical documentation that describes the logging design, and market-surveillance authorities entitled to access, which together define the logs’ consumer profile. The records are not written to be stored. They are written to be queried, by internal monitoring on a cadence, by incident response under pressure, and by authorities with statutory patience, which is a query-workload specification wearing legal language, and the substrate section takes it from there.
One boundary keeps the section honest: the Act specifies objectives, not formats. No article names the fields, the storage, or the schema, the harmonized standards filling those gaps are still maturing, and the practical consequence favors the estates this series describes: organizations get to satisfy the objectives with well-engineered general-purpose infrastructure rather than waiting for a compliance-product category to congeal, provided the engineering can demonstrate, at assessment time, how the design meets each named purpose. That demonstration is a mapping document this article’s schema section is effectively drafting.
Why the Lakehouse Is the Evidence Substrate
The requirements profile, automatic capture, tamper-evident durability, long declared retention, as-of addressability, analytical queryability, governed access, is a profile Apache Iceberg tables on the governed lakehouse match property for property, and the match is worth walking because it decides the build.
Immutability arrives as architecture: Iceberg’s snapshots are immutable, every write is a new snapshot with full lineage in the table’s metadata, and the record of what the telemetry said as of any moment is addressable forever within retention, which is the tamper-evidence foundation, completed by the catalog’s commit arbitration putting every write under an authenticated principal in an audit stream of its own. The evidentiary chain, who wrote which records when, under what authority, is the format and catalog doing their ordinary jobs.
As-of answers arrive as tags: the auditor’s defining question, show me what the system’s records held on the date in question, answers by reference, the tagged states my WAP article’s tag discipline lays down, retention-matched to the compliance clocks, with time travel covering the untagged in-betweens. No restoration project, no backup archaeology, a name and a query.
Scale and cost arrive as the lakehouse’s basic economics: decision-grade telemetry at agent cadence is a high-volume append workload with long retention, exactly the shape object-storage tables price best, partitioned by time, compacted by the maintenance tier, cheap enough that the estate-wide capture the scope section recommended is affordable rather than aspirational.
Queryability arrives as the whole point: the records are ordinary tables, so post-market monitoring is scheduled SQL, incident reconstruction is a session, the regulator’s extract is a governed export, and the same analytical machinery the estate runs on its business data runs on its evidence, including, with deliberate symmetry, the quality contracts and WAP gates that make the telemetry tables themselves validated data.
Two substrate honesty notes complete the match. Immutable-by-snapshot is not immutable-by-law: an administrator with sufficient rights can expire snapshots or rewrite tables, which is true of every storage system ever fielded, and the evidentiary posture comes from the composition, catalog-audited writes under scoped principals, retention enforced by declared policy, separation between the roles that write telemetry and the roles that maintain it, and the anchoring digests the pipeline section adds for the records whose assessments warrant independent proof. And the format’s own metadata is part of the evidence: the snapshot history, the commit principals, the reference lifecycle live in the tables’ metadata and answer procedural questions, when was this written, by what, under whose authority, with the same queryability as the records themselves, which assessors familiar with opaque log stores tend to receive as a pleasant shock.
And governance arrives already built: the telemetry tables live under the same catalog, semantic policies, and access machinery as everything else, which the policy-aware section is about to need, because evidence about decisions concerning people is itself data about people, and the substrate that treats compliance records as ordinary governed tables gets their protection for free where a bolted-on log store starts from zero.
The Records: Schema for Decision-Grade Telemetry
What to record is where the Act’s objectives meet the estate’s layers, and the schema that works organizes as an event family around a spine of shared threading.
The spine is the correlation the observability plane already carries: every record bears its timestamps, its system identity and version set, model, prompt or policy versions, semantic contract versions, its principal, and its workflow identifier, the thread that joins a decision to its steps across every layer. The version set is the field family compliance adds to ordinary observability, because “which system” for the Act means which exact configuration, and substantial-modification detection, one of Article 12’s named purposes, is a query over version-set changes.
The event families, by the layer that emits them. Decision events are the headline records: the system’s consequential outputs, the score, the recommendation, the classification, the generated assessment, each with its inputs by reference, source records by stable identifier and snapshot, not by value, a design choice the privacy section defends, its outputs, its confidence where the system produces one, and the version spine. Data-access events come from the catalog and semantic tiers: what governed data the decision path touched, at which snapshots, under which policies, which is lineage doing compliance duty. Policy events record governance in action: the access granted and scoped, the row policies applied, the requests declined, the throttles and budget actions, the record that the control environment operated, which assessments increasingly ask to see demonstrated rather than described. Oversight events capture the humans: approvals, overrides, escalations, the interventions the Act’s human-oversight article contemplates, threaded to the decisions they governed, which turns “effective human oversight” from an org-chart claim into a queryable rate. And lifecycle events mark the system itself changing: deployments, version promotions, evaluation results, incident declarations, the substantial-modification trail.
The design compresses into tables shaped like this:
CREATE TABLE lake.compliance.decision_events (
event_id STRING,
event_ts TIMESTAMP,
system_id STRING,
version_set STRUCT<model: STRING, prompts: STRING,
contracts: STRING, policy_rev: STRING>,
principal STRING,
workflow_id STRING,
decision_type STRING,
input_refs ARRAY<STRUCT<dataset: STRING,
snapshot_id: BIGINT,
record_key: STRING>>,
output_summary STRING,
confidence DOUBLE,
oversight_ref STRING
)
USING iceberg
PARTITIONED BY (days(event_ts), system_id)
TBLPROPERTIES (
'history.expire.min-snapshots-to-keep' = '30',
'compliance.retention' = 'P24M',
'compliance.basis' = 'ai_act_art12_art26'
)Two schema disciplines carry the design’s weight. References over values: inputs and evidence point at governed datasets by identifier and snapshot rather than copying content into the log, which keeps the telemetry lean, keeps personal data in its governed home where its policies and erasure machinery already operate, and still reconstructs exactly, because the pointed-at snapshots are addressable for as long as their own retention says, a coordination the retention section returns to. And declared retention in the metadata: the compliance clock and its legal basis ride on the table as properties, machine-readable by the maintenance tier that enforces expiration and by the audit that verifies the estate’s clocks match its obligations, which turns retention from a runbook promise into an inspectable setting.
The Human Oversight Ledger, Specifically
One event family deserves its own section because the Act weights it and estates under-build it: the oversight record, the evidence that humans supervised effectively, which the layered architecture is unusually well positioned to produce and most deployments leave scattered.
The obligations’ shape: high-risk systems must be designed for effective human oversight, deployers must assign it to competent people with the authority to act, and the demonstration burden runs through records, that oversight existed, operated, and mattered. The agentic estate’s autonomy machinery, the approval tiers, the override channels, the escalation ladders my quality and agents writing builds, is exactly this oversight, already structured, and the telemetry discipline’s job is capturing it at evidentiary grain: every approval with its reviewer, evidence presented, and decision, every override with its reason, every escalation with its path and resolution, every tier promotion and demotion with its case-history citation, threaded to the decisions and systems they governed.
What the captured ledger buys, beyond the checkbox: the effectiveness question becomes answerable, override rates by system and reviewer, approval latencies, the correlation between oversight interventions and outcome quality, which is the difference between asserting supervision and measuring it, and the measurement runs both directions, the reviewer who approves everything in four seconds being a finding the program wants to make before an assessor does. The rubber-stamp detection my quality writing prescribes for gates applies verbatim to human oversight, and the ledger is where it runs.
And the design note that saves later grief: oversight records name humans, which makes this the family where the both-directions policy discipline works hardest, reviewer identities scoped to the roles that need them, aggregate effectiveness metrics computed for the broader consumers, and the retention basis for personnel-linked records reviewed by counsel separately from the decision records they thread to, because the evidence that protects the organization and the privacy of the people supervising are both real interests, and the schema is where they get reconciled rather than discovered in conflict.
Policy-Aware in Both Directions
The title’s phrase names the design’s distinguishing discipline, and it runs both ways through the telemetry.
Direction one: the telemetry records policy. The governed architecture this series builds makes every consequential decision pass through enforcement points, the catalog’s grants, the compiler’s row and column folding, the gateway’s scopes and budgets, the autonomy tiers’ authority checks, and the telemetry captures each enforcement as an event, which is what raises the record from “what the system output” to “what the control environment did.” The difference matters at assessment: a log of decisions demonstrates activity, and a log of decisions threaded to their policy events demonstrates governance, the access that was scoped, the request that was declined, the override that was recorded, which is the evidence shape the risk-management and oversight obligations actually contemplate. The layered estate gets this direction nearly free, because the enforcement points already exist and already emit: the compliance build is mostly persistence discipline, not new interception.
Direction two: the telemetry is governed by policy. Compliance records are among the estate’s most sensitive tables, decisions about people, threaded to the people who made and overrode them, and the design treats them as first-class governed data. Access scopes by role through the same semantic machinery as everything else, the compliance officers’ view, the incident responders’ view, the per-system views for the teams that own each system, with the regulator-facing extract as a governed, logged export rather than a bucket handoff. Row and column policies apply, the oversight events’ human identifiers masked outside the roles that need them, the cross-system views scoped so one product team’s investigation does not browse another’s decisions. And the telemetry’s own access is itself telemetered, reads of the compliance tables landing in the audit stream, because “who looked at the evidence” is a question investigations ask, and the substrate answers it the same way it answers everything.
The both-directions discipline resolves what bolt-on compliance logging never resolves cleanly: the log store that sits outside the governance perimeter becomes either a sensitive-data leak with excellent retention or a locked box nobody can query, and the estates that put the evidence inside the governed lakehouse get protection and usability from the same machinery, which is the architectural argument of this whole article in one sentence.
The Capture Pipeline: From Layers to Ledger
The records exist because the layers emit them, and the pipeline that lands them deserves the same engineering as any tier-one data flow, because evidence with gaps is evidence with an asterisk.
Emission rides the existing convergence points: the gateway’s per-request telemetry, the semantic layer’s compilation log, the catalog’s audit stream, the agent surface’s oversight and evaluation records, each already structured, each already carrying the spine identifiers, published to the estate’s event transport as they occur. The compliance build’s emission work is completeness auditing rather than new instrumentation: the event-family checklist against each layer’s emissions, the gaps closed where a policy decision or oversight action was enforced but not persisted, and the version-set threading verified end to end, which is typically the largest single task, since operational telemetry historically recorded “what” more diligently than “which version of everything.”
Landing is a streaming ingestion into the telemetry tables with the disciplines my ingestion and WAP writing prescribe: micro-batches onto staging branches, an audit gate that checks the evidence itself, spine completeness, referential resolution of the input references, sequence continuity per emitting system, and gated fast-forwards on the audit cadence, because telemetry that will face an assessor should demonstrably pass its own quality contract. The gate’s sequence-continuity check earns special mention: gaps in an evidence stream are findings-in-waiting, so the pipeline detects them at landing, alerts on them as incidents, and records the gap analysis itself, which converts “the logs are missing an hour” from a discovered embarrassment into a documented, explained event.
The WAP inheritance runs deeper than the landing gate, and naming it saves a build: the publish tags and audit-result records my branching article’s discipline lays down are themselves compliance telemetry, the evidence that every published state of every governed table passed its declared checks, keyed to the batches and versions that shipped it. Estates that adopted the gated-publish pattern for quality reasons discover, at compliance inventory time, that their data-governance evidence, the Act’s data-quality obligations translated to demonstrable practice, has been accumulating in the estate since the pattern’s first table, which is one more instance of this series’ standing arithmetic: the disciplines compound, and the estate that built them in order keeps finding the next requirement half-met.
Integrity hardening tops the pipeline off for the estates whose assessments demand it: the telemetry tables’ snapshot histories are the tamper-evidence baseline, and programs add periodic anchoring, digests of the tables’ states at tagged checkpoints, recorded to a separate custody, sometimes an external timestamping service, so that the claim “these records are as written” carries proof independent of the platform’s own administrators. The technique is old evidence engineering, cheap at the lakehouse’s cadence, and disproportionately reassuring in the assessment room.
Answering the Auditor: Evidence Workflows
The telemetry earns its keep in the workflows it makes routine, and three define the program.
Post-market monitoring becomes scheduled analytics: the monitoring plan’s questions, decision volumes and distributions by system and version, override and decline rates, drift indicators, incident correlations, compile into the standing queries and dashboards the estate already knows how to run, on a documented cadence, with the runs themselves logged as lifecycle events, which is the plan demonstrably operating rather than existing.
Incident reconstruction becomes a session instead of a project: a challenged decision resolves by workflow identifier to its complete thread, the decision event, its input references resolved at their recorded snapshots, the policies that applied, the oversight that did or did not intervene, the version set that produced it, and the reconstruction, which the direct-publish era measured in days of grep, prices in minutes of SQL, with the output shaped for the serious-incident reporting clock where the incident clears that bar.
The workflows get their consumption surface through the estate’s own semantic tier, which is the design closing its loop: the compliance record families publish as governed entities with described, versioned views, the monitoring metrics defined once as contracts, override_rate and decision_volume carrying their definitions the way every governed metric does, and the compliance team’s questions, increasingly, arriving through the same assistant surface the rest of the business uses, an agent answering “show me the scoring system’s override trend by version this quarter” from the evidence tables under the compliance role’s scopes. The recursion is worth savoring and then engineering soberly: agents querying the records of agent governance, under the governance being recorded, which works precisely because every layer of it is the same layered machinery, and which gets the same evaluation-suite treatment as any consequential agent surface before anyone relies on it in an assessment season.
And the authority’s request becomes a governed export: the scoped extract, this system, this period, these event families, produced by query, reviewed under the compliance role, delivered with its provenance, the tags and snapshot identifiers that let the extract be re-verified against the estate later, which is the difference between handing over files and handing over evidence. The workflows share a rehearsal discipline the incident world already knows: run them before they are demanded, the mock reconstruction, the practice extract, on the calendar, because the first execution of an evidence workflow should never be the one with a deadline attached.
Retention Meets Erasure: The Tension, Engineered
The privacy lawyer’s question arrives on schedule in every design review: the AI Act says keep, the GDPR says delete on request, and the telemetry design has to serve both masters, which it does with three coordinated moves.
References over values, again, doing its heaviest work: because the telemetry points at personal data in its governed home rather than copying it, erasure operates where it always operated, on the source datasets with their existing machinery, and the telemetry’s pointers to erased records dereference to the documented erasure event, the log remaining complete as a record of what was decided while the personal content honors its deletion. The design keeps identifiers in the log where a lawful basis supports them, retention under legal obligation being its own basis, a determination that belongs to counsel per record family, and the schema’s per-family retention and basis properties are exactly where that determination gets recorded.
Where identifying content must live in the log itself, the oversight events’ human actors, the decision summaries that quote inputs, the column policies and masking from the both-directions section carry the load in the reading path, and crypto-shredding patterns, per-subject keys whose destruction renders content unreadable within the immutable history, serve the estates whose obligations require erasure to reach even the snapshots, a technique to deploy deliberately and sparingly, because it trades queryability for erasability and most record families, designed with references, never need it.
And the clocks coordinate: the telemetry’s retention, the referenced datasets’ retention, and the tags’ retention align per record family so that evidence remains reconstructable for its full compliance window, the audit that verifies the alignment running as one of the estate’s standing policy tests, because the failure mode here is quiet, a source table’s ordinary expiration silently amputating the evidence’s references three years before the investigation that needed them.
Legal hold rounds out the clock machinery: when an investigation or dispute opens, the affected record families and their referenced snapshots freeze against expiration, which in this substrate is a reference operation, hold tags laid across the relevant tables with expiration suspended for the tagged lineages, applied by the compliance role, logged like everything, and released the same way. The estates that wire the hold path before needing it, one more rehearsal on the calendar, convert the general counsel’s most urgent email from a scramble across systems into a tagging operation with a confirmation query, which is the kind of sentence that wins the compliance program its next budget cycle.
One Telemetry Estate, Three Masters
The closing argument before the worked example is economic, because compliance builds fail by becoming parallel universes. The telemetry this article designs is the observability plane the series already runs, hardened, not duplicated: the same events serve the operations dashboards, the cost attribution my TCO writing meters, and the compliance evidence, one capture pipeline, one set of governed tables, three consumer workloads with three access scopes and three retention declarations where the clocks differ. The estates that build it this way report the compounding this series keeps finding: the compliance program’s budget hardens the telemetry that operations wanted anyway, the operational discipline keeps the compliance records healthy between assessments, and the cost meter rides along free, which is how a legal obligation ends up funding the estate’s best infrastructure rather than its most resented.
Getting Started: The Compliance-Telemetry Quarter
The program’s sequence, for the estate that has the layers and needs the ledger.
Weeks one through four are inventory and mapping: the system classification against the Annex with counsel, the event-family checklist against each layer’s current emissions, the version-threading audit, and the mapping document, obligations to mechanisms, that will anchor both the build and the eventual conformity documentation, drafted now because writing it exposes the gaps while they are cheap.
Weeks five through nine build the ledger: the compliance namespace’s tables with their declared retentions and bases, the emission gaps closed, the WAP-gated landing pipeline with its continuity alarms, the both-directions policies applied, and the evidentiary hardening, tags, anchoring where warranted, on the high-risk core’s records, with the estate-wide capture running at operational grade behind it.
Weeks ten through thirteen prove it: the monitoring plan’s queries scheduled, the mock reconstruction drilled and timed, the practice extract produced and re-verified, the retention-alignment audit added to the standing policy tests, and the quarter’s close-out review walking the mapping document with legal and the system owners, obligations to mechanisms to demonstrations, which is the artifact the assessment season will thank the program for.
The anti-pattern to refuse at kickoff is the vendor-first instinct’s compliance edition: a quarter evaluating AI-governance suites while the estate’s own telemetry runs unhardened buys a dashboard for records that do not yet exist. The suites have their place, workflow, registry, and reporting layers over the evidence, and the evidence itself is the estate’s to build, on the substrate it already trusts, which is the build this article specified.
The convergence extends across regulations, which the multi-regime estates should hear explicitly: the record families this article designs for the AI Act are the same evidence the adjacent regimes keep asking for in their own vocabularies, the financial-sector model-risk frameworks, the sectoral audit regimes, the operational-resilience rules, and the privacy accountability principle itself, one governed ledger answering many questionnaires, with the per-regime differences living in the retention bases, the access scopes, and the extract templates rather than in parallel capture systems. The estates that internalize this stop building a compliance system per regulation and start maintaining an evidence estate with regulatory views, which is the sustainable posture as the AI-regulation wave continues internationally, jurisdictions diverging in their texts and converging, with remarkable consistency, on the same underlying demand: prove what your systems did.
A Worked Example: The Estate That Was Mostly Ready
The composite, in this series’ pattern, with no invented benchmark numbers, and with the legal specifics illustrative rather than advisory.
The company runs the governed stack these articles build, and its exposure walks in through a product: the customer-risk scoring that one business line ships into credit-adjacent decisions, squarely Annex III territory, plus an agent estate whose classification review finds two systems adjacent and arguable. The compliance program’s kickoff inventory lands the finding this article predicts: the control environment largely exists, the layers enforce and emit, and the evidence discipline does not, telemetry retained on operational clocks, version threading partial, oversight actions living in a workflow tool outside the estate, and nobody able to answer “reconstruct this decision from March” inside a week.
The first quarter builds the ledger: the compliance namespace and its event-family tables, the emission audit across the layers closing the gaps, version-set threading completed through the gateway and semantic tiers, the oversight workflow’s records landed into the estate, and the WAP-gated capture pipeline running with its sequence-continuity alarms. The scoring system’s records get the evidentiary hardening, tags on the compliance clocks, the anchoring digests, the counsel-reviewed retention bases in the table properties, and the estate-wide capture runs at the cheaper operational grade behind it, the two-tier design the scope section recommended.
The second quarter builds the workflows and finds the payoffs: the post-market monitoring queries scheduled against the monitoring plan’s questions, the mock reconstruction drill resolving a six-month-old scoring decision in forty minutes end to end, the practice extract produced and re-verified, and the first genuine incident, an override-rate anomaly on one scoring version, caught by the monitoring queries and handled inside the serious-incident analysis with the evidence assembled by lunch. The drill’s forty minutes become the program’s internal legend, because the veterans in the room remember the last comparable reconstruction, pre-lakehouse, as a three-week archaeology.
The program’s stress test arrives in the following quarter, unscheduled, as stress tests do: one of the arguable-tier agent systems picks up a new use case that tips its classification, and the two-tier capture design pays out exactly as intended, the system’s operational-grade records already complete and threaded, the promotion to evidentiary grade a properties-and-tags change plus the counsel review, no historical gap, no retrofit, the scope change absorbed in a sprint. The compliance lead’s note in the review is the design’s vindication in one line: reclassification cost us a checklist, not a time machine.
The assessment season lands the design’s argument: the conformity documentation’s logging section describes the architecture in this article’s terms, objectives to mechanisms, Article by table family, the assessors’ sampling requests answer by query in the room, and the finding the program frames and keeps is the one worth quoting: the evidence system is the production system, not a copy of it, which collapsed an entire category of consistency questions. The retrospective’s transferable line belongs to the compliance lead: the estate did not build a compliance system, it promoted its telemetry to evidence, and the distance between those two projects was one quarter instead of the vendor category’s promised year.
Failure Modes
The evidence estate’s failures cluster where legal intent and engineering habit diverge, and each defense below is cheaper installed than discovered.
The parallel universe. Compliance builds its own capture, store, and pipeline beside the observability plane, the two drift, and the assessment discovers the evidence disagreeing with the operations dashboards. The defense is the one-estate discipline: same events, same tables, scoped consumers, with the compliance-specific hardening as properties of the shared substrate rather than a second one.
Values where references belonged. Convenience copies personal content into the log, the telemetry becomes a shadow dataset of the estate’s most sensitive data, and the erasure request arrives to find it un-erasable. The defenses: the reference discipline enforced at schema review, the landing gate’s referential checks, and the periodic scan for content-shaped columns in the compliance namespace.
Threading rot. Version sets and workflow identifiers degrade as systems evolve, and the reconstruction workflow returns threads with missing links exactly when it matters. The defenses: spine completeness as a landing-gate check, the threading verified in the mock drills, and the deployment templates carrying the version-set emission so new systems inherit it.
Retention by vibes. Tables carry retention nobody traced to an obligation, some evidence expires early, some sensitive records persist past any basis, and both directions are findings. The defense is the declared-basis discipline: retention and basis as table properties, the alignment audit as a standing policy test, and counsel’s sign-off recorded per family.
The gap discovered late. An emitter fails quietly, weeks of one system’s events never land, and the absence surfaces during an investigation. The defenses: sequence-continuity alarms at landing, per-system emission heartbeats on the operations dashboard, and the documented-gap protocol that turns detected holes into explained events rather than silent ones.
Evidence nobody rehearsed. The tables exist, the workflows were never drilled, and the first real request meets a team learning its own system under a statutory clock. The defense is the rehearsal calendar: the mock reconstruction, the practice extract, the monitoring plan’s dry run, before the deadline owns the schedule.
Conclusion
The EU AI Act made a specific engineering demand of every organization deploying consequential AI: be able to prove, durably and on demand, what your systems did, with what data, under what controls, with what human oversight, across every version you ever ran. Bolt-on compliance logging answers the demand with a parallel system that drifts from the truth it claims to record, and the governed lakehouse answers it structurally: decision-grade telemetry as Iceberg tables, immutable by snapshot, addressable by tag, retained by declared basis, validated by the same gates as any tier-one data, recording the policy decisions the layered estate already enforces, and governed by those same policies in return.
The deeper conclusion extends the series’ refrain one more layer: the estate that built its architecture for trust discovers the regulator was asking for the same properties all along. Governance as code produced the reviewable control environment, the observability plane produced the automatic records, the quality discipline produced evidence that passes its own gates, and the compliance program’s job collapses from construction to hardening, threading, clocks, and rehearsal. Auditability was never a tax the architecture pays. It is the architecture, described in a different vocabulary, and 2026 is the year the two vocabularies finally met with a deadline attached, in Brussels first and everywhere else soon enough.
Keep Going
If this piece was useful, the estate it hardens is the one my other writing builds: the observability and governance planes across the agentic lakehouse series, the WAP and quality gates that validate the evidence, and the platform foundations in Architecting an Apache Iceberg Lakehouse from Manning alongside Apache Iceberg: The Definitive Guide from O’Reilly. You can find every book I have written, across lakehouse architecture, Apache Iceberg, Apache Polaris, and AI, at books.alexmerced.com.