Should a discount with more orders scale?
A randomized pilot generated ten more orders, but the subsidy applies to every treated order.
The question
Harbor Market randomly assigns 200 eligible users equally to one-use discount or control. Groups are disjoint. The packet is an illustrative pilot, not a precise effect estimate.
| Assigned users | Completed orders | Contribution per order before discount (£) | Discount per order (£) | |
|---|---|---|---|---|
| Control | 100 | 60 | 8 | 0 |
| Discount | 100 | 70 | 8 | 2 |
Authored data. At most one order per user. Contribution already includes variable fulfillment costs; no spillovers or other measured costs in this packet.
The commercial lead can extend the pilot next week. No uncertainty interval, long-run retention or repeat-purchase data is supplied.
“Compare contribution, then write a bounded recommendation.”
Not saved by this site. Your browser may restore text when you navigate back.
A strong answer, step by step
Open each step in turn. This is one strong answer, not the only one.
Step 1: Use the assigned denominator
What would you do here? Open the move you’d make.
Compare conversion among discount redeemers
Tempting, but it misses something. Redeemers select themselves after assignment.
Compare all assigned eligible users
The strong move. Assignment preserves the pilot comparison.
See how a strong candidate says it
“Observed conversion is 60% in control and 70% with the discount: a ten-point difference. Use assigned users, not only redeemers or buyers.”
Why it works: Assignment preserves the pilot comparison.
Step 2: Price the full subsidy
What would you do here? Open the move you’d make.
Charge all 70 treated orders
The strong move. Every treated buyer receives the discount.
Subtract subsidy from incremental-order margin
Tempting, but it misses something. The discount also subsidizes 60 orders that appear in the control baseline.
See how a strong candidate says it
“Control contribution is 60×£8 = £480. Treated contribution is 70×(£8−£2) = £420. Observed contribution is £60 lower despite ten more orders. Charging the discount only to the extra ten orders would omit £120 of subsidy.”
Why it works: Every treated buyer receives the discount.
Step 3: State what remains uncertain
What would you do here? Open the move you’d make.
Budget next quarter from the pilot point estimate
Tempting, but it misses something. A point difference in one small trial is not the population effect.
Distinguish the observed result from a precise effect
The strong move. Design and precision are separate questions.
See how a strong candidate says it
“Random assignment supports a causal design, but this small observed difference is uncertain. Check assignment, confidence intervals and guardrails. We cannot infer a durable negative effect or future retention from this table alone.”
Why it works: Design and precision are separate questions.
Step 4: Give the decision owner a brief
What would you do here? Open the move you’d make.
Hold broad rollout and test the economics
The strong move. The next test targets the reason the current offer loses contribution.
Roll out based on the conversion lift
Tempting, but it misses something. Order lift alone does not pay for the full subsidy.
See how a strong candidate says it
“Do not broadly roll out on order volume. Validate the pilot, estimate contribution uncertainty and test a lower subsidy or targeted eligible group with a predeclared stopping rule. Retention value could change the economics if measured credibly.”
Why it works: The next test targets the reason the current offer loses contribution.
The answer, in short
- Decision
- Hold broad rollout; test contribution and uncertainty.
- Because
- The observed pilot earns £420 against £480 in control.
- What would change it
- Credible incremental retention value, a lower subsidy or a precise profitable contribution effect could support scaling.
Common traps
- Subsidizing only the incremental orders in the calculation.
- Treating a randomized design as a guarantee of precise results.
Check your answer
- Did you compare all 100 assigned users per group?
- Did you compute £480 versus £420?
- Did you include all 70 discounts?
- Did you separate uncertainty from the observed result?
Last reviewed Nov 10, 2026