Silent success is the failure mode

A common problem in model-assisted work starts with a reasonable assumption. The model does not tell us that it made the assumption. It keeps going, produces a clean result and marks the job as complete. In regulatory work, this can be harder to spot than an answer that is clearly wrong.

The quiet failure

Suppose we are extracting an obligation from a Regulation. The citation is real, and the source supports the sentence. But the model missed a condition a few lines earlier that decides who the rule applies to. The obligation still reads well. Every field is filled, and the run reports success.

People often focus on hallucination, where a model invents information. This failure is different. The model used real text, but it continued when it should have raised a question. I call this unchecked completion. Asking the model to raise questions can help, but an instruction alone is not a reliable control.

Planner → generator ⇄ evaluator

Tracfox splits the work into three roles. The role names are less important than keeping their responsibilities separate.

The planner defines what a correct result must contain. It lists the evidence needed, the questions that cannot be guessed and the conditions that require SME review. The generator creates a draft using this definition.

The evaluator does not simply ask whether the draft looks good. It checks each claim against the source and the agreed checks. It confirms that the cited text supports the claim. It can also challenge the plan. If it finds a missing check, the plan is revised before generation continues.

The loop cannot continue forever. It stops when the checks pass or when the system cannot resolve an important question. Unresolved questions go to an SME with the findings attached. The loop makes the work easier to inspect. It does not guarantee that the result is correct.

Quality has to be gradable

The evaluator needs clear checks. A general instruction such as “check the quality” is not enough. For each obligation, I would ask:

• Grounding: Does the authoritative text support the obligation? Does the citation point to that support?
• Correctness: Does the obligation have the right meaning when read in context?
• Completeness: Does it keep who the rule applies to, along with its conditions, exceptions, thresholds and timing?
• Cross-references: Have the relevant definitions and referenced sections been followed far enough to understand the requirement?

A result can be grounded but incomplete. It may quote the right sentence and still miss an exception that changes who the rule applies to. It may follow one cross-reference but miss the definition that controls a key term. A citation and a completed form show that work happened. They do not prove that the result is right.

This is similar to a familiar compliance process: maker, checker and escalation. One role creates the draft. Another checks it. Unresolved questions go to an SME. This separation gives someone the responsibility to challenge a plausible answer before anyone relies on it.

The loop still needs tested checks, versioned sources, an audit trail and accountable approval. Without them, it is only a series of model calls.

Where the SME stays in the loop

Human review should be risk-based, not line by line. The system should complete routine checks automatically and send only the important exceptions to an SME. These may include weak evidence, conflicting sources, failed checks, major regulatory changes or questions the system could not resolve. Teams can also review samples to confirm that the process continues to work as expected.

Tracfox uses this pattern to reduce the review queue, not move every obligation into it. A reviewer can open an item and see its source, the checks performed and the reason it was accepted or escalated. Customers receive grounded obligations and related control best practices with evidence attached, while SMEs focus on the questions that need judgement.

Walk the chain on your own regulation →