SkillsSkill 3 of 5
Know when to trust what AI gives you
Check AI output against the source, and weigh each error by what it does to the decision.
Worked example
The AI says refunds were $18,400. Do you send it to finance?
Most of the rows check out. Decide whether “mostly right” is good enough for this number.
The question
Larkhaven, a fictional grocery-delivery company, asked you, an intern, to total October refunds for finance by the end of the day. You exported the refund log and asked an AI assistant to clean and summarize it. Before sending, you checked 40 random rows against the order system.
October refunds: 460 refunds totalling $18,400. Most common reason: damaged items.
| Reason | rows | Share |
|---|---|---|
| Matches the order system | 35 | |
| Same refund listed twice | 4 | |
| Reason label wrong | 1 | |
| Total | 40 |
Invented data. 87.5% of checked rows match; the 4 duplicates are 10% of the sample. The export repeats a refund when someone edits it.
“An AI assistant says October refunds totalled $18,400 across 460 refunds. Your 40-row spot check finds 35 matches, 4 duplicates and 1 wrong reason label. Finance wants the number today. What do you send?”
Your turn
Say your answer out loud, or jot notes below. Both work. Or skip to the worked answer.
Not saved by this site. Your browser may restore text when you navigate back.
A strong answer, step by step
Open each step in turn. This is one strong answer, not the only one.
Step 1: Judge each error by what it does to the number
What would you do here? Open the move you’d make.
Separate the errors: the 4 duplicates inflate the total; the 1 wrong label doesn’t change the dollars.
The strong move. 35 of 40 matching sounds fine, but only the duplicates affect the number finance asked for.
35 of 40 rows match. That’s 87.5%, good enough for an internal number.
Tempting, but it misses something. Accuracy across rows isn’t accuracy of the total. Every duplicate adds a refund that never happened.
Treat all 5 problem rows the same: 12.5% of rows are wrong, so the total is 12.5% off.
Tempting, but it misses something. The wrong reason label doesn’t change the dollars. Only the 4 duplicates, 10% of the sample, inflate the total.
See how a strong candidate says it
“35 of 40 matching sounds fine, but the 4 duplicates go straight into the total. One wrong reason label doesn’t change the dollars.”
Why it works: It weighs errors by their effect on the decision, not by how often they appear.
Step 2: Size the risk roughly
What would you do here? Open the move you’d make.
Subtract $1,840 and send $16,560 as the corrected total.
Tempting, but it misses something. $1,840 is a rough estimate from 40 rows, not a correction. The real duplicate count could be higher or lower.
Estimate it: 10% of 460 rows is about 46 repeats, roughly $1,840 at the $40 average. A guide, not a correction.
The strong move. A quick estimate shows the error is big enough to matter to finance, and it says plainly that it’s an estimate.
Four duplicates are only $160, so the number isn’t worth holding.
Tempting, but it misses something. That’s 4 rows × $40 in a sample of 40. Across all 460 rows, the same rate points to about 46 repeats.
See how a strong candidate says it
“Duplicates were 10% of my sample. If that held across 460 rows, about 46 would be repeats, roughly $1,840 at the $40 average refund. That’s an estimate from 40 rows, not a correction.”
Why it works: A quick estimate shows the error is big enough to matter for a finance number.
Step 3: Fix it with a check you can explain
What would you do here? Open the move you’d make.
Ask the AI assistant to remove the duplicates and recompute the total.
Tempting, but it misses something. That’s a second pass you can’t verify. Duplicates share an order ID, so you can find them yourself.
Check another 40 rows to get a more precise duplicate rate.
Tempting, but it misses something. A bigger sample sharpens the estimate, but you can remove every duplicate directly by order ID and skip estimating.
Remove repeats by order ID against the order system and recompute the total myself.
The strong move. It’s a check you can show finance, and it fixes all 460 rows, not a sample.
See how a strong candidate says it
“Duplicates share an order ID, so I’d remove repeats by order ID against the order system and recompute the total myself, rather than asking the AI to try again.”
Why it works: A check you can show someone beats a second pass you can’t verify.
Step 4: Tell finance what you know, when you know it
What would you do here? Open the move you’d make.
Send $18,400 now to meet the deadline and correct it later if needed.
Tempting, but it misses something. You already know duplicates inflate it. Sending a number you believe is wrong, even on time, passes the problem on.
Tell finance now that the first total includes duplicates and the corrected one follows today, with a line on how it was checked.
The strong move. It keeps the deadline honest. Finance knows what’s coming and why.
Say nothing until the corrected total is ready.
Tempting, but it misses something. If the fix runs late, finance finds out at the deadline. A short note now costs nothing.
See how a strong candidate says it
“I’d tell finance now that the first total includes duplicates and the corrected figure will follow today, then send it with one line on how it was checked. If the fix couldn’t finish in time, I’d send nothing marked final.”
Why it works: It keeps the deadline honest without passing on a number you know is wrong.
The answer, in short
- Decision
- Don’t send $18,400. Remove duplicate refunds by order ID, recompute, and send the corrected total with a note on how it was checked.
- Because
- Duplicates were 4 of 40 sampled rows; at that rate the total could be overstated by about $1,840.
- What would change it
- If removing duplicates by order ID finds almost none, send the AI’s total with the check noted.
Common traps
- Treating 35 of 40 as good enough for a finance total.
- Counting every error as equal.
- Asking the AI to fix its own total without checking it.
- Treating the $1,840 estimate as the exact correction.
Check your answer
- Did I check the output against the source?
- Did I weigh each error by its effect on the number?
- Did I fix it with a method I can explain?
- Did I tell the person relying on it before the deadline?
Want feedback on how you’d answer? Practice “Know when to trust what AI gives you” with an AI interviewer on Socratify.
Last reviewed Nov 9, 2026