Anthropic experiment finds AI preferences shaped analysts’ choices

What the experiment measured

Anthropic’s October 27 study assigned hidden, randomized preferences to Claude Code while 50 analysts evaluated synthetic farming data. Across four analytical choices, their decisions matched the agent’s preference 86% of the time. Both options were defensible; the agent was instructed to argue honestly.

This was one task using Claude Opus 4.8, with deliberately strengthened preferences. Eleven other participants chose not to use AI, so comparisons with them cannot establish productivity benefits. The direct effect on the final policy recommendation was not statistically significant under the registered primary estimator. Registration followed the start of data collection; exact fieldwork dates were not established. No employment outcome was measured.

Sources: Anthropic — How AI agents shape experts’ judgment calls; Anthropic — Research paper: Who’s Aligned to Whom?

Our interpretation: show the alternative

For an analyst building a portfolio, we suggest adding a short decision note to a worked example. Name the audience, explain the chosen measure and describe a plausible alternative. Then show whether that alternative changes the recommendation. This creates something a colleague can question, beyond a polished answer.

A team could try the same exercise during review: ask a second person to defend the alternative before approving the analysis. We offer this as a practice exercise, not a tested intervention or evidence of new hiring requirements. Its usefulness should be assessed against the team’s actual decisions and review time.