Failure Modes: What Happens When the AI Is Wrong
By Anushka Appala and Dr. Janio Rosales
Most pitches for government AI are written as if the system always works. Ours is designed around the certainty that sometimes it will not. This is not pessimism; it is the only honest way to build public infrastructure. A citizen owed a pension does not care about your average accuracy. She cares what happens on the day the model is wrong, the network is down, or the data about her is missing. So we want to walk, plainly, through the ways NaciluzIA can fail — and show that in every one of them, the system degrades to human handling, never to a guess. Candor here is not a confession. It is the credibility strategy.
The single property that makes this possible is the commitment gate: the point between any AI suggestion and any real-world effect that cannot produce a binding action without both an authorized result from the deterministic rules engine and a verified signature from a named human official. Every failure below is really a story about that gate refusing to move.
The five ways it breaks
The AI is wrong. This is the failure people fear most, and structurally it is the least dangerous. The model is confined to the advisory layer — it understands, translates, drafts, and flags. It has no authority to approve anything. When it drafts a bad acuerdo or misreads a request, the error lands in front of a human official who is reviewing before signing, and it is grounded by retrieval over cited law so the mistake is checkable against the statute it claims to rest on. A wrong suggestion is just a suggestion. It cannot become a payment, because the gate still needs a rule result and a signature the model cannot supply.
The AI is unavailable. Foundation models are consumed through secure APIs, and APIs go down. Because NaciluzIA is provider-agnostic, one option is to fail over to another vendor. But the deeper answer is that an outage cannot corrupt a decision — it can only slow one. With no draft to review, the case does not resolve itself on a guess; it waits for a human, or moves to a fully manual path. The absence of AI returns the state to how it worked before AI, which is inconvenient, not unjust.
The rule is wrong or outdated. The deterministic engine executes published, versioned law-as-code, which means a bad rule is a real risk — a threshold set incorrectly, a statute changed but not yet reflected. Two things contain it. First, every rule evaluation is recorded in the ledger with its exact version, so an erroneous rule is discoverable and every decision it touched can be found and reviewed. Second, a human official still signs; a rule result alone never acts. A wrong rule produces a wrong recommendation to a person, not an automatic wrong outcome — and because the rule is published and versioned, it is corrected in the open rather than buried in a model's weights.
Data is missing or conflicting. Real administrative data is incomplete and contradictory — the same person under three spellings, a record that says living and a record that says deceased. The deterministic engine does not paper over this. When inputs are missing or fail a consistency check, the rule cannot return an authorized "approve," so the gate has nothing to act on. The case routes to a human with the conflict surfaced, not resolved by assumption. The system's default in the face of bad data is to stop and ask, never to fill the gap with a plausible-looking guess.
The ledger is tampered with. The record itself is a target. The ledger is hash-chained: each entry carries the hash of the one before it, so altering any past entry breaks every link that follows and the tampering announces itself. It is digitally signed and valid under Decreto 47-2008. Tampering does not produce a false-but-trusted history; it produces a visibly broken chain that an auditor at the Contraloría can locate to the exact block. The attack does not get you a clean lie. It gets you caught.
The one thing none of them can do
Put the failures side by side and the pattern is the point.
| Failure mode | What the system does | What it never does |
|---|---|---|
| AI is wrong | Bad draft caught in human review; checkable against cited law | Auto-approve on a guess |
| AI is unavailable | Fail over or drop to manual handling; case waits | Resolve itself without review |
| Rule wrong / outdated | Recorded by version; discoverable; human still signs | Silently bind a bad outcome |
| Data missing / conflicting | Rule withholds "approve"; routes to a human | Fill the gap by assumption |
| Ledger tampered | Hash chain breaks; auditor locates the exact block | Present a false history as true |
Every row ends in the same place: the failure degrades the system toward human handling and slower service, and none of them can manufacture a binding action out of an error. That is the guarantee worth making in public. We are not promising a system that never fails. We are promising one whose failures cost time, not rights.
This is why we lead with the failure modes instead of hiding them. A state earning back trust — in a region where 62% of people said corruption had risen in the prior year (Transparency International, Global Corruption Barometer–LAC) — cannot afford a technology that asks to be believed. It needs one that invites the hard question, what happens when you are wrong?, and has a straight answer. Ours is: nothing irreversible, everything recorded, and a human still standing behind the result.
Cada decisión, a la luz.
— Anushka Appala and Dr. Janio Rosales