All cases Machine Learning

What makes a feed “good”? Evaluating recommendation algorithms with real people.

The Machine Learning team built four new feed algorithms. To choose between them, we first needed to know what “good” meant to the people using the app.

Company
Cyprus
Role
Senior UX Researcher
Period
2025 – 2026
Type
Evaluative

Context

Gameram is a social app where gamers find people to play with, available across the Americas. The feed is where those connections begin.

The Machine Learning team had built four new versions of the recommendation algorithm. Technical metrics help compare models, but they don’t tell you whether the result makes sense to the person on the other side of the screen.

Research question: what makes a feed “good” for Gameram users, and where does each of the four algorithms succeed or fall short?

Approach

Before comparing the algorithms, we needed a yardstick. So the study started by defining criteria with people, and only then moved on to evaluation.

In practice, a psychometrics problem: measuring something subjective, the quality of a feed, in an objective way.

1. Interviews To understand, in users’ own words, what a good feed needs to deliver. The result was 12 criteria.
2. Moderated evaluation with users Each version of the feed was tested with users, who rated it and explained why.
3. Unmoderated evaluation with experts Internal experts evaluated the same versions, adding a second perspective on the results.
4. Combined analysis Ratings were combined with qualitative feedback to understand where each version did well or poorly, and why.

Outcome

4recommendation algorithms evaluated
12quality criteria defined with users
2perspectives: users and internal experts

There was no winner. Each feed had strengths and weaknesses, and reality was more complex than “better” and “worse”.

The team received a guide to the strengths and weaknesses of each algorithm, measured against the 12 user-defined criteria, alongside technical metrics. Instead of naming a winner, the research changed the question: do we want this strength, even with that weakness?

The feed choice became an informed decision, with the risks and benefits of each option in view. Findings were shared in decision-focused reports with the CPO and the data science team.

Some details have been omitted for confidentiality.

Next case

Only 2.5% made it to the end