1.5 months ago I showed here my tool for detecting code duplication with embedding models: https://news.ycombinator.com/item?id=48762038
I got valuable feedback and some questions I was not able to answer precisely. Now I have data.
The code-specialized model I recommended based on the provider's benchmark turned out to be one of the worst-performing in this use case. But their general-purpose model showed very good results.
Data confirmed that small specialized models can outperform big providers. It showed that the same model variant with a size of 3x more parameters has no practical difference. The dimension of a vector can indeed be reduced several times without any noticeable loss of quality.
1.5 months ago I showed here my tool for detecting code duplication with embedding models: https://news.ycombinator.com/item?id=48762038 I got valuable feedback and some questions I was not able to answer precisely. Now I have data.
The code-specialized model I recommended based on the provider's benchmark turned out to be one of the worst-performing in this use case. But their general-purpose model showed very good results.
Data confirmed that small specialized models can outperform big providers. It showed that the same model variant with a size of 3x more parameters has no practical difference. The dimension of a vector can indeed be reduced several times without any noticeable loss of quality.
All code snippets used in the benchmark are available on https://github.com/rafal-qa/embedding-benchmark