Designing UI for AI Agents: Why Black-Box Models Fail in Commercial Insurance

There's a dangerous idea floating around insurtech: that you can take an LLM, point it at a pile of ACORD forms, and let it spit out quotes. No UI. No transparency. Just "AI-powered" magic behind the curtain.

This approach fails in insurance for one fundamental reason: underwriters need to know why.

The Black Box Problem

When a regulator asks how a premium was calculated, "the AI did it" isn't an answer. When a broker disputes a declination, "the model said no" doesn't hold up. Insurance is built on auditable decisions. Every number has a source. Every judgment has a trail.

Most AI tools treat the model's output as the final word. The underwriter sees the result but not the reasoning. They can accept or reject, but they can't understand or refine.

This is backwards. The AI should show its work — not because humans don't trust AI, but because showing work is how insurance operates.

AI Agents Need Their Own Interface

At Oyyo, we don't treat AI as a background process. Each AI agent has its own visual interface that communicates what it found, how confident it is, and where it looked.

When the Extraction Agent processes a 14-page ACORD 125, it doesn't just dump a JSON object. It:

• Highlights the exact location in the PDF where it found each data point
• Shows a confidence score for every field (green for high, yellow for medium, red for low)
• Flags fields where it's uncertain and needs human review
• Provides a one-click interface for the underwriter to approve, correct, or override

The underwriter sees the AI's reasoning in real-time. They don't wait for a final output and then reverse-engineer what happened.

Why This Matters for Adoption

The biggest barrier to AI adoption in insurance isn't technology — it's trust. Underwriters with 20 years of experience won't blindly accept a number from a model. They need to see the work.

When you design UI for AI agents — not just for humans — you solve the trust problem. The underwriter can see that the AI found "Total Insured Value: $4.25M" on page 7 of the SOV with 94% confidence. They glance at it, confirm it's right, and move on. Eight seconds instead of eight minutes of manual lookup.

That's not replacing the underwriter. That's giving them supervision tools worthy of the work.

The Oyyo Approach

Every agent in Oyyo has its own UI surface:

• The Extraction Agent shows bounding boxes on source documents, field-level confidence, and source attribution
• The Rater Agent shows premium calculations step by step — base rate, loadings, credits — with the ability to override any component
• The Quoter Agent shows the assembled quote document with highlighted dynamic sections the AI generated
• The Organizer Agent shows the folder tree it built, with explanations for why each document was categorized the way it was

This is what separates an AI workspace from an AI script. The workspace is designed for collaboration between humans and agents. The script just runs and hopes for the best.