Does the AI summary describe the support backlog?
The generated summary says half of all messages concern refunds.
The question
Harbor Market reviews a sample to plan refund support staffing. Within each channel, the 50 reviewed messages were selected at random; the AI refund labels have not yet been checked by people.
| Backlog messages | Reviewed | AI-labeled refund | |
|---|---|---|---|
| Phone | 100 | 50 | 40 |
| 900 | 50 | 10 |
Authored data. Channels partition a backlog of 1,000 messages; one message is the unit.
50 of 100 reviewed messages mention refunds, so 50% of the backlog needs refund support.
“Audit the estimate and say what must happen before using it for staffing.”
Not saved by this site. Your browser may restore text when you navigate back.
A strong answer, step by step
Open each step in turn. This is one strong answer, not the only one.
Step 1: Find the sampling error
What would you do here? Open the move you’d make.
Inspect the sampling design
The strong move. A balanced review sample is not a proportional sample.
Apply the pooled sample share to the full backlog
Tempting, but it misses something. The rare high-refund channel is overrepresented in the review.
See how a strong candidate says it
“The sample gives each channel equal weight; the backlog is 10% phone and 90% email. The pooled 50% is a sample share, not the backlog share.”
Why it works: A balanced review sample is not a proportional sample.
Step 2: Reweight the labels
What would you do here? Open the move you’d make.
Average both channel rates
Tempting, but it misses something. Equal channel weights repeat the original sampling error.
Weight channel rates by backlog
The strong move. Population weights match the declared unit.
See how a strong candidate says it
“Phone’s labeled rate is 40/50 = 80%; email’s is 10/50 = 20%. Weighting by backlog gives 0.1×80% + 0.9×20% = 26%, or an estimated 260 messages, conditional on valid labels and representative within-channel samples.”
Why it works: Population weights match the declared unit.
Step 3: Inspect the uncertain labels
What would you do here? Open the move you’d make.
Validate the labels and their misses
The strong move. Reweighting cannot fix a wrong label definition.
Use the weighted AI labels to set staffing
Tempting, but it misses something. An exact arithmetic result can still be based on classification error.
See how a strong candidate says it
“Before staffing, manually audit a channel-stratified set, including ambiguous cases and non-refund labels. Define refund intent and check misses as well as false alarms. The 26% estimate still has sampling and classification error.”
Why it works: Reweighting cannot fix a wrong label definition.
Step 4: Recommend a reversible move
What would you do here? Open the move you’d make.
Allocate staff in proportion to labeled messages
Tempting, but it misses something. The unit is messages; one issue can generate multiple messages.
Check handling time and repeat contacts
The strong move. The resource decision requires a workload measure.
See how a strong candidate says it
“Use the validated, weighted estimate as one input to a short staffing trial. Messages are not tickets or minutes: measure handling time and repeat contacts before converting counts to headcount.”
Why it works: The resource decision requires a workload measure.
The answer, in short
- Decision
- Validate labels, then estimate weighted workload.
- Because
- The pooled review share overweights phone messages.
- What would change it
- A non-random within-channel review, label errors or very different handling times would require a new estimate.
Common traps
- Reporting sampling output as population truth.
- Converting messages directly to staffing.
Check your answer
- Did you identify the 10/90 population weights?
- Did you calculate 26% rather than 50%?
- Did you state sampling and label uncertainty?
- Did you check message counts before staffing assumptions?
Last reviewed Nov 10, 2026