 | Hi all! This is an update from the IsItNerfed team, where we continuously evaluate LLMs and AI agents. We run a variety of tests through Claude Code and the OpenAI API. We also have a Vibe Check feature that lets users vote whenever they feel the quality of LLM answers has either improved or declined. Over the past few weeks, we've been working hard on our ideas and feedback from the community, and here are the new features we've added:
And yes, we finally tested Sonnet 4.5, and here are our results. It turns out that while Sonnet 4 averages around 37% failure rate, Sonnet 4.5 averages around 46% on our dataset. Remember that lower is better, which means Sonnet 4 is currently performing better than Sonnet 4.5 on our data. The situation does seem to be improving over the last 12 hours though, so we're hoping to see numbers better than Sonnet 4 soon. Please join our subreddit to stay up to date with the latest testing results: We're grateful for the community's comments and ideas! We'll keep improving the service for you. [link] [comments] |
Predictive technology: computational social science
