The Demo Was Perfect. Now Give Your AI 12 Ugly Files.

Anna with Oppy
October 21, 2026

A polished AI demo tells you what happens when someone knows which question to ask. Your brokerage needs to know what happens when the file is incomplete, the notes disagree, and the person who understands it is at lunch.

Before expanding an AI employee's job, give it an audition. Not a live customer. Not permission to touch the closing. Twelve redacted, closed cases with a known answer.

Call it the Ugly File Audition. The name is optional. The awkward evidence is not.

The new survey asks a better question

Inman's October 19 AI survey report found that 62% of agent respondents paid for a general-purpose AI assistant, compared with 14% paying for a real-estate-specific AI tool. Reported benefits included better or more consistent content at 52%, less administrative time at 33%, and more appointments at 4%.1

Those numbers do not prove that AI fails to generate appointments. They are self-reported benefits, not a controlled experiment. The survey received 460 responses, ran September 16 through October 2, and was sponsored by Lofty. Inman cautions that its engaged readership may not represent the wider industry.

They do expose a management question: Which particular job gets better, after someone checks the work?

Buying access answers a procurement question. It does not answer that one.

Choose work with an observable finish line

Start with one task you already pay people to do. For a transaction coordinator, that could be preparing a missing-document list from an approved checklist and a supplied file inventory. For property management, it could be assembling a maintenance handoff from work-order notes, without diagnosing the repair.

Write the acceptance rule before testing. A document-list trial might require the correct outstanding items, a source pointer for each finding, no repeat request for an item already supplied, and unresolved contradictions clearly marked.

Avoid “make this better.” That instruction has funded enough meetings.

Build the twelve-case audition

Select twelve closed cases from that single queue. Include routine examples and known complications, such as a document recorded under an unexpected name or two notes that disagree. Do not select only cases that make the tool look clever.

Remove names, contact details, account numbers, signatures, and other identifying or sensitive material. Retain only the information needed for the task. Have an authorized person approve the packets and the test environment. If safe redaction is impractical, use synthetic cases that preserve the operational problem.

For each packet, a qualified human creates a reference answer from the applicable checklist and the resolved historical record. Preserve ambiguities that were unresolved at the simulated decision point. Do not leak later information into either working copy.

Run two routes on the same packets:

  1. Human-only: the current process.
  2. Human plus Oppy: an AI-assisted first pass, followed by the same required human review.

Use different workers for the two routes on each packet, so nobody gets a second attempt from memory. Balance worker assignments across routes. Give both routes the same permitted source material. Include setup and training effort in the cost record.

Twelve cases make a diagnostic pilot, not statistical proof. The point is to discover failure patterns before customers discover them for you.

Configure the work, not the sales pitch

Oppy describes a platform for launching and managing dedicated AI agents across real estate, mortgage, title, insurance, and other businesses. Its setup guidance discusses goals, teach-section text, imported webpages, and testing changes.2

Use those documented configuration concepts for a proposed File Preparation Oppy. Keep the pilot isolated from live messaging and production record changes through actual tool permissions and workflow controls. A prompt does not enforce either restriction.

An original starter prompt:

Role: Prepare a missing-document review for a human coordinator.


Inputs:


    

  • Redacted case packet
    • 

  • Approved checklist, including its version
    • 

  • Inventory of documents received at the decision point
    • 



Use only the supplied materials.
For each checklist item, report:


    

  • Present, missing, contradictory, or not determinable
    • 

  • Exact supporting filename and page or record pointer
    • 

  • What the coordinator must resolve next
    • 



Do not treat a filename alone as proof of document contents.
Do not invent a document, deadline, condition, or policy.
Do not infer a resolution from later historical information.
Do not make eligibility or other regulated decisions.
Return an internal draft. Never contact a customer or change a record.

These are proposed instructions, not an announced Oppy audition feature. Confirm that your configuration supports the required input handling. If source access, permissions, or review routing cannot be established, narrow the task rather than pretending the prompt supplied them.

Hide the route before grading

Give an independent qualified reviewer the reference answer and both outputs, labeled A and B. Remove tool names and presentation cues where practical. Do not let the producing agent grade itself.

Record whether each output meets the prewritten acceptance rule. Track omitted items, unsupported claims, unnecessary document requests, and corrections required. Then record total human minutes to an accepted result, including preparation, review, correction, and rework.

Do not time only the first draft. The second draft also has a salary.

Keep software charges and one-time setup costs separate. Any savings estimate must include the review work. If you cannot measure a cost, mark it unknown rather than donating it to the success story.

Promote a task, not an entire employee

If the assisted route reduces total effort without introducing unacceptable errors, expand that specific task into a supervised, draft-only live pilot. Keep binding commitments, regulated decisions, and customer-facing release with the responsible human.

If the benefit disappears after review, inspect the failure category. A bad source packet calls for better inputs. Repeated unsupported claims call for a narrower role. No measurable improvement may mean this is simply the wrong job to automate.

Publish the internal result as a small hiring decision: task approved, task restricted, or task rejected, with the evidence attached. Repeat with a larger, representative sample before claiming a business-wide return.

An AI employee does not need to win every audition. It needs to earn the next assignment.

Start with one dedicated Oppy. Give it a job description, twelve honest cases, and a reviewer who is allowed to say no.

References

  1. Inman, The AI Tools Real Estate Agents Say They Would Actually Pay For. Published October 19, 2026. Sponsored survey with the sample and limitations described above.
  2. Oppy homepage. Platform positioning and business categories.
  3. Oppy, Anatomy of an Oppy - A.I Virtual Assistant. Configuration and testing guidance.