Synthetic Population Testing for Recommendation Systems

Offline evaluation is necessary for recommender systems. It is also not a full test of recommender quality. The missing layer is not only better aggregate metrics, but better ways to test how a model behaves for different kinds of users before launch.

TL;DR

• In the last post, I argued that offline evaluation is useful but incomplete for recommendation systems.

• After that, I built a small public artifact to make the gap concrete.

• In the canonical MovieLens comparison, the popularity baseline wins Recall@10 and NDCG@10, but the candidate model does much better for Explorer and Niche-interest users and creates a very different behavioral profile.

• I do not think this means “offline evaluation is wrong.”

• I think it means a better pre-launch evaluation stack should include some form of synthetic population testing: explicit behavioral lenses, trajectory-aware diagnostics, and tests that make hidden tradeoffs visible before launch.

What Comes After “Offline Evaluation Is Not Enough”?

In the first post, I made a narrow claim:

offline evaluation is useful, but incomplete, because recommendation systems are interactive systems.

That argument matters, but by itself it leaves an obvious next question:

if aggregate offline metrics are not enough, what should be added to the evaluation stack?

I do not think the answer starts with a giant platform or a perfect user simulator.

I think the more practical place to start is smaller:

take the same baseline-vs-candidate comparison and test it through multiple behavioral lenses, not just one aggregate average.

That is what I built next.

The Artifact

The current artifact is a small public recommender behavior QA harness.

It compares:

• one baseline recommender

• one candidate recommender

• one fixed evaluation setup

And it produces:

• standard offline ranking metrics

• bucket-level utility

• behavioral diagnostics such as novelty, repetition, and catalog concentration

• short trajectory traces that make model behavior easier to inspect

The canonical public run is intentionally narrow:

• MovieLens 100K

• Model A: popularity baseline

• Model B: genre-profile recommender with a popularity prior

• 4 fixed buckets

• one frozen report bundle

The point is not to claim that these two models define recommender evaluation. The point is to create one clean, reproducible proof that aggregate offline metrics can hide useful pre-launch information.

The Canonical Result

The canonical MovieLens run shows the core value in one comparison.

On aggregate offline ranking metrics, the popularity baseline wins:

Model Recall@10 NDCG@10

Model A 0.088 0.057

Model B 0.058 0.036

If we stopped there, the conclusion would be straightforward: Model A looks better.

But the bucketed view tells a different story:

Bucket Model A Model B Delta (B-A)

Conservative mainstream 0.519 0.532 0.012

Explorer / novelty-seeking 0.339 0.523 0.184

Niche-interest 0.443 0.722 0.279

Low-patience 0.321 0.364 0.043

That is the point.

Aggregate offline metrics say one thing. The segment-aware view says something more useful:

• the baseline is better at recovering held-out positives

• the candidate is much stronger for important user lenses

• the behavioral profile of the system changes in ways the aggregate view compresses away

The behavioral diagnostics make that even clearer:

Model Novelty Repetition Catalog concentration

Model A 0.395 0.279 1.000

Model B 0.678 0.664 0.717

This is worth pausing on, because not every behavioral metric moves in the same direction.

Model B is:

• more novel

• less catalog-concentrated

• but also more repetitive in this diagnostic

That is not a bug in the framework. It is part of the point. Different recommendation strategies produce different behavioral signatures, and pre-launch evaluation should help make those signatures visible instead of collapsing everything into one average.

What “Synthetic Population Testing” Means Here

It is important to be precise about this phrase.

What I have today is not a rich simulation of realistic synthetic humans. There are no agent conversations, no generated personas with biographies, and no claim that the current system faithfully reproduces real user psychology.

What the artifact does have is a simpler and more controlled version of the same idea:

• fixed behavioral lenses

• explicit utility assumptions

• short trajectory simulation under those assumptions

The four v1 buckets are:

• Conservative mainstream

• Explorer / novelty-seeking

• Niche-interest

• Low-patience

Each bucket values recommendation behavior differently. The evaluation then asks how the same two models behave when the user lens changes.

So when I say synthetic population testing here, I mean:

an early, lightweight form of synthetic population testing built from fixed behavioral lenses, not full synthetic-user simulation.

I think that still matters. It turns vague product intuition like “some users may prefer this model more than others” into an explicit, reproducible pre-launch test.

Why This Is Better Than Another Aggregate Metric

A natural response to the first post is to ask whether we simply need better aggregate metrics.

I do not think that is enough.

The problem is not only that a metric is imperfect. The deeper problem is that recommender quality is heterogeneous.

Different users are helped by different behaviors:

• some want safer, familiar, high-exposure items

• some benefit from more novelty and more variety

• some have narrower tastes that require stronger matching to long-tail pockets

• some degrade faster when sequences become stale

A single global score cannot represent all of that well.

That is why I think the next useful layer should look more like testing against a small synthetic population than inventing one more scalar.

Instead of asking only:

which model wins on average?

we should also ask:

which model wins for which behavioral lens?

where do the models differ most?

what kind of trajectory does each model produce?

This does not mean the current bucket lenses are perfect. It means they are often more informative than one collapsed aggregate average.

One Short Trajectory Example

The trajectory view matters because recommendation quality is not only one-step.

Here is one Explorer / novelty-seeking comparison from the canonical run:

Model A

Raiders of the Lost Ark -> Fargo -> Toy Story -> Return of the Jedi

Enter fullscreen mode

Exit fullscreen mode

Model B

Prophecy, The -> Cat People -> Wes Craven's New Nightmare -> Relic, The

Enter fullscreen mode

Exit fullscreen mode

The first sequence stays much closer to familiar, high-exposure titles. The second is much more tailored to a narrower taste profile and much more novel.

This is exactly the kind of difference that disappears when evaluation is reduced to one aggregate ranking score.

Why This Matters Before Launch

Pre-launch evaluation is about decisions, not just measurements.

If a team is deciding whether to ship a new recommender, the real question is usually not:

did one mean score go up?

It is closer to this:

• who gets a better experience?

• who gets a worse one?

• does the candidate become more repetitive?

• does it collapse toward head items?

• does it create a healthier exploration profile?

Those are product and system questions, not only ranking-metric questions.

That is why I like this framing. It stays honest about what the artifact is doing. It is not trying to predict the full online future. It is trying to make hidden tradeoffs visible earlier, with a tool that is still small enough to run, inspect, and reason about.

What This Is, And What It Is Not

I think the strongest version of this argument is the honest one.

This artifact is:

• a small public proof

• a recommender-specific evaluation layer

• a way to make segment-level and trajectory-level tradeoffs visible

• a first wedge into broader testing for interactive systems

This artifact is not:

• a proof that the candidate model is globally better

• a replacement for offline evaluation

• a replacement for online experiments

• a full synthetic-human simulation framework

That distinction matters. If this work is useful, it will be useful because it is clear about what it adds, not because it overclaims.

A Better Evaluation Stack

The long-term picture I have in mind looks something like this:

• Standard offline evaluation remains the first layer.

• Segment-aware and trajectory-aware diagnostics become the second layer.

• Richer synthetic population testing may become the next layer after that.

• Online experiments still remain necessary for final validation.

That is a much more realistic stack than pretending a single aggregate metric can do the whole job.

In that stack, the current artifact sits at layer two. It adds explicit behavioral lenses and short trajectory diagnostics to the familiar offline comparison workflow.

That is why I think it matters, even in its current limited form.

It is not the final answer.

It is the first concrete artifact of the missing layer.

Conclusion

The first post argued that offline evaluation is not enough for recommendation systems.

This artifact is my first practical answer to what should come next.

Not a giant platform. Not a perfect simulation. Not a replacement for offline evaluation.

Just a small, reproducible evaluation harness that compares a baseline and a candidate through multiple behavioral lenses and shows tradeoffs that aggregate metrics compress away.

If offline evaluation is the first screen, then synthetic population testing, in some form, may be one of the next useful layers.

This v1 is a lightweight version of that idea.

If you want to see the public artifact, the canonical MovieLens demo lives in the limitation repo as a report, JSON result bundle, and supporting visuals.