Well, first off there is no such thing as Claude as there are multiple models that you can select from. You did not list which model you were using. In my opinion the Claude 3.5 Sonnet model is spectacular. It’s the best model yet for coding both on leaderboard and empirically in projects I’ve had it help me with.
This topic is discussed in recent Lex Fridman interview with with CEO of Anthropic where he very clearly walks through how these claims of it being dumber or not true. It’s a great interview and after listening to it I’m even more bullish on Anthropic.
There was a small degradation in performance that they posted an alert at the top of the page 2 nights ago. It didn’t affect the quality of the responses I got but it didn’t cause somewhat of a slowdown in response speed.
> Well, first off there is no such thing as Claude as there are multiple models that you can select from.
Apologies, I assumed people would infer that I am referring to 3.5 sonnet.
> In my opinion the Claude 3.5 Sonnet model is spectacular.
Mine as well, until this morning.
> There was a small degradation in performance … 2 nights ago. It didn’t affect the quality of the responses I got…
Also same, but as of this morning the performance is fine but the quality seems to have gotten worse.
> This topic is discussed in recent Lex Fridman interview with with CEO of Anthropic where he very clearly walks through how these claims of it being dumber or not true
TL;DR they don’t change the weights, but they sometimes run A/B tests and modify the system prompt. The underlying model is very sensitive to changes. Even a small change can have broad impacts.
Sounds like it might be switching to Claude Haiku instead of Claude Sonnet.
Sonnet 3.5 always has this issue for me though. It excessively follows the original instructions, even in vague ways. It's likely 3.5 (new) is even worse. We use 3.0 in production because of this one quirk.
The underlying implication of the linked comment is that Anthropic is using quantization or similar quality-reducing strategies to help keep their service online due to the same shortage that has been causing availability issues.
Yes. It seems to be almost incapable of communicating in anything but terse (up to 5 word or so) bullet points.
And even when you force it write in coherent sentences the output still seems markedly worse than it used to.
I wonder if it is falling back to concise mode as a means to handle load?
When I switch it back to full I just get longer bullet point lists so perhaps they are doing that silently.
Well, first off there is no such thing as Claude as there are multiple models that you can select from. You did not list which model you were using. In my opinion the Claude 3.5 Sonnet model is spectacular. It’s the best model yet for coding both on leaderboard and empirically in projects I’ve had it help me with.
This topic is discussed in recent Lex Fridman interview with with CEO of Anthropic where he very clearly walks through how these claims of it being dumber or not true. It’s a great interview and after listening to it I’m even more bullish on Anthropic.
There was a small degradation in performance that they posted an alert at the top of the page 2 nights ago. It didn’t affect the quality of the responses I got but it didn’t cause somewhat of a slowdown in response speed.
> Well, first off there is no such thing as Claude as there are multiple models that you can select from.
Apologies, I assumed people would infer that I am referring to 3.5 sonnet.
> In my opinion the Claude 3.5 Sonnet model is spectacular.
Mine as well, until this morning.
> There was a small degradation in performance … 2 nights ago. It didn’t affect the quality of the responses I got…
Also same, but as of this morning the performance is fine but the quality seems to have gotten worse.
> This topic is discussed in recent Lex Fridman interview with with CEO of Anthropic where he very clearly walks through how these claims of it being dumber or not true
Could you elaborate on what was said?
I found the interview [1].
TL;DR they don’t change the weights, but they sometimes run A/B tests and modify the system prompt. The underlying model is very sensitive to changes. Even a small change can have broad impacts.
[1]: https://lexfridman.com/dario-amodei-transcript#chapter8_crit...
Sounds like it might be switching to Claude Haiku instead of Claude Sonnet.
Sonnet 3.5 always has this issue for me though. It excessively follows the original instructions, even in vague ways. It's likely 3.5 (new) is even worse. We use 3.0 in production because of this one quirk.
Yes, there are some reports. For example: https://news.ycombinator.com/item?id=42215912
Those are mostly about the availability issues.
I’m not having trouble getting responses as of today, but the quality of the responses seems to be much worse.
The underlying implication of the linked comment is that Anthropic is using quantization or similar quality-reducing strategies to help keep their service online due to the same shortage that has been causing availability issues.
I think so, it was bad enough for me to cancel my subscription.
The new model is almost certainly a cheaper version of the older model, where they tried to maintain quality.