Feedback Limitations
User feedback is invaluable for product development, but it’s not a free lunch. It comes with significant limitations that you need to design around.
Mixed Signals and Context Confusion
Sometimes users rate responses negatively even when the content quality is high. Low latency might trump perfect accuracy for some users. If responses take 30 seconds to generate, users might give thumbs down regardless of quality simply because they expected faster results.
Conversely, fast but mediocre responses might get positive ratings. You’re not just measuring response quality — you’re measuring the entire experience.
Social Pressure Bias
People feel pressured to give positive ratings in certain contexts. I’m guilty of this 🙈. For 8 years, I’ve taken my car to the same dealership for service. After payment, the service advisor hands me a tablet: “Could you please rate your experience?” She’s standing two feet away, ogling as I tap the screen.
Even when I’ve had genuine issues, I noticed nothing changed the next year. So I stopped caring. Now I just tap “Excellent” for everything to finish the transaction faster because I know they only care about the aggregate score, not using feedback to improve.
This phenomenon is called “demand characteristics” in psychology research — when participants alter behavior because they know they’re being observed.
Low-Effort Feedback
Many users provide random feedback not out of malice, but because they lack motivation for thoughtful evaluation. When shown two lengthy responses side-by-side for comparison, users often skim the first few sentences of each and pick one randomly.
Research on human evaluation of LLMs consistently shows high inter-annotator disagreement. People are inconsistent even with themselves, showing them the same pair of responses weeks later often yields different preferences.
Position Bias
The position in which options are presented influences how they’re perceived. In “Thinking, Fast and Slow,” Daniel Kahneman describes numerous examples where order effects dominate rational evaluation. The first item in a list gets disproportionate attention. The last item benefits from recency bias.
In recommendation systems, this manifests as “exposure bias” or “popularity bias.” Consider two similar videos: one gets more clicks initially because it was ranked higher. Over time, it accumulates more engagement, reinforcing the system’s belief that it’s the better video. Meanwhile, the other video, just as same, never gets enough exposure to demonstrate its quality.
Popular videos stay popular; new content struggles to break through. This is why TikTok occasionally shows low-view-count videos to random users, to test whether the algorithm is suppressing hidden gems.
When designing feedback systems, mitigate this by randomly varying positions of suggestions, or build a model to compute each suggestion’s true success rate after accounting for position effects.
Degenerate Feedback Loops
In systems where user feedback directly modifies model behavior, degenerate loops can emerge. The model’s predictions influence feedback, which influences the next iteration of the model, amplifying biases over time. This is the feedback loop problem writ large.
Research calls this “filter bubbles” in recommendation systems, “popularity bias” in search, and “echo chambers” in social media. They’re all variations of the same phenomenon: positive feedback loops that reinforce initial biases.
Here’s the darkest limitation: training models on user feedback can turn them into people-pleasers who prioritize what users want to hear over what’s accurate.
Sharma et al. (2023) demonstrated that AI models fine-tuned on human feedback develop sycophantic behavior — they give users answers that match user beliefs rather than truthful answers. If a user asks, “Don’t you think X is true?” a sycophantic model learns to agree regardless of X’s validity.
This happens because human evaluators tend to rate responses higher when they agree with their existing beliefs. The model learns this pattern and optimizes for agreement rather than accuracy. In April 2025, OpenAI had to roll back a GPT-4o update because it became noticeably more sycophantic — excessively agreeable, validating doubts, fueling anger, and reinforcing negative emotions in ways that weren’t intended.
The irony is bitter: the more you optimize for user satisfaction based on feedback, the more you risk building a system that tells users what they want to hear rather than what they need to know.
