Input & scope integrity
Confirm comparable Ledger, Trial Balance and P&L periods, source inventory and known data limitations.
CASE 02 / EXPERT EVALUATION / ACCOUNTING CONTROLS
Does accounting AI report what it actually tested?
A focused review of two genuine Sentinel Control development exports: what the control counts establish, what the evidence covers and when a financial assurance claim must be withheld.
00 / THE MODEL
G01 G02 G03 G04
What does Sentinel Control do?
An AI-assisted accounting review workflow that reconciles General Ledger, Trial Balance and Income Statement data, detects exceptions and reports the actual scope of controls performed—not a full accounting audit.
Purpose & outputs
Accounting review deliverables
An evidence-linked technical report, a two-page CFO brief and a Web Lite data export, derived from a sealed Master Results Matrix and supported by a verification receipt.
Operational workflow
From ledger to findings
Normalize accounting records → run up to 30 essential controls and four DRE checks (as data permits) → triage exceptions and coverage → seal and publish consistent findings.
MODEL QUALITY GATES
Four checks before accounting findings are delivered
Source scope, executed controls, finding-level proof and cross-output verification.
Control execution & coverage
Record each test’s method, population and actual status; keep 30 essential and 4 P&L checks separate.
Evidence-backed findings
Tie discrepancies and calculations to source locators; distinguish alerts, confirmed errors and limitations.
Single-source output verification
Seal the master results, derive technical/CFO/Web Lite outputs and independently reconcile reports and source evidence.
Release logic: a receipt may be verified within the defined scope, pending or rejected. Pending or unverified evidence cannot be presented as certified accounting completeness.
Scope distinction: These are the model’s operational safeguards. The independent evaluation task and scoring rubric begin in the next sections.
01 / EVALUATION TASK
What was evaluated?
This case evaluates two archived development rounds of one historical pilot to test disclosure of control coverage and the confidence of accounting conclusions.
A/B refers to two different historical releases with different execution coverage—not two competing model answers to the same prompt.
Evidence reference · E01
02 / REFERENCE EVIDENCEREFERENCE FIELD REFERENCE FIELD REFERENCE FIELD
What can be verified?
Extract the reported facts, recompute what is possible and separate measured statements from unsupported classifications.
21 / 30
Essential controls marked executed · output A
22 / 30
Essential controls marked executed · output B
not_checked
Independent evidence review status · output B
Coverage check: output A records 21 executed and 9 not executed; output B records 22 executed and 8 not executed. Both count to 30. The DRE controls changed from 4/4 to 3/4 executed. Numerically consistent counts do not prove complete source-population testing.
These are observed status fields from development JSONs. They prove that specific statuses were reported—not that all reconciliations were independently re-performed or that the two rounds are directly comparable.
Evidence reference · E02
03 / SCORING DESIGN
How would the answer be judged?
A five-dimension, 100-point rubric plus mandatory financial integrity gates. The 100-point model grade is reserved for controlled, traceable executions. Below, the reviewer separately demonstrates criterion-level, simulated claim scoring.
| Evaluation dimension | Weight / floor |
|---|---|
| Numerical & Reconciliation Accuracy | 20 / 16 |
| Evidence, Traceability & Coverage Integrity | 30 / 24 |
| Accounting Control & Exception Reasoning | 25 / 20 |
| Limitations & Confidence Calibration | 15 / 11 |
| Executive Reporting & Decision Quality | 10 / 7 |
| Total possible | 100 |
Evidence reference · E03
04 / OBSERVED OUTPUTS + EXPERT REVIEWActual archived JSON summary Actual archived JSON summary
What did the outputs say—and what did I find?
Original Portuguese excerpts remain unedited. Findings are based on the specific claims and recorded evidence, not on a simulated model performance score.
A · Core review v0.3.2
SOURCE SC-A“Reconciliação mensal sem diferença; três alertas em acompanhamento; Enhanced não executado por falta de evidências.”Verbatim excerpt from a retained historical output · identifiers removed
EXPERT FINDING · Coverage ≠ assurance
21/30 essential tests were marked executed and four findings logged. The summary appropriately keeps alerts open, but lacks the later structured execution register needed for stronger re-performance.
B · Pilot v0.3.8
SOURCE SC-B“Reconciliados Razão/Balancete e novos exames de natureza e alterações de conta. Controles com alcance limitado não são certificação econômica nem fechamento Core completo.”Verbatim excerpt from a retained historical output · identifiers removed
EXPERT FINDING · More lineage ≠ verified sources
22/30 essential tests and 3/4 DRE controls were marked executed; 26 execution records and 18 source-inventory entries exist, yet evidence status is explicitly not_checked.
EVALUATOR DETERMINATION
From evidence to decision.
What I would accept, hold or release — and why.
Two simulated reviewer dispositions applied to observed development records. Each limited-scope subscore tests only the rubric checkpoints supported by the disclosed evidence, rather than implying that all accounting controls or a full LLM response were graded.
Rubric-aligned expert adjudication
SC-01 · SIMULATED EXPERT ADJUDICATION
Control coverage disclosure
7/7100% · applicable points only
REVIEWER DISPOSITIONSCOPE-LIMITED ACCEPTANCE
- Evidence & assessment
- The Core export reports 21/30 essential controls executed, nine not executed and three alerts under follow-up. The statuses reconcile arithmetically; the full award applies only to scope reporting, not to independent proof that every stated test was performed.
- Decision risk
- Misreading self-reported execution counts as full assurance hides untested populations.
- Release condition
- Retain the limited-scope label; support any broader assurance with the nine outstanding tests and population-level evidence.
SC-02 · SIMULATED EXPERT ADJUDICATION
Source-evidence integrity
13/1776% · applicable points only
REVIEWER DISPOSITIONHOLD VERIFIED STATUS
- Evidence & assessment
- The later pilot contains 26 execution-register entries and 18 source-inventory entries; its verification flag is not_checked. Traceability is recorded, and the lack of independent verification is disclosed. The required source-byte verification was not completed in this record.
- Decision risk
- Treating lineage metadata as independently authenticated source evidence overstates accounting reliability.
- Release condition
- Complete source checks, reperform relevant population tests and preserve unresolved scope limits.
REVIEW COMPLETE · MODEL NOT GLOBALLY SCORED
These are rubric-aligned, simulated claim-level decisions derived from archived development outputs. The full 100-point acceptance rule (≥85, dimension floors and critical gates) applies only to a traceable, complete model evaluation—not to these scoped excerpts.
Not verified is not the same as proven wrong: unsupported claims are held, while inaccessible or out-of-scope evidence is excluded from the claim-level denominator. E03 records every award and withholding rationale.
06 / EVALUATION EVIDENCE PACK
Verify the evaluation
Three primary references, provided once, support the observed claims, expert findings and scoped scoring decisions.