I'm increasingly thinking that the only solution may be to have a search index with human editorial control. As much as I think Google deserves scorn for how badly they have made the search experience on the web, we have to remember that they're fighting a battle against SEO and spam. It feels like they have mostly given up. Any automatic solution will always be an arms race where they will be one step behind. The only way to get ahead is to ensure that all content has to pass through human moderation before it can be included -- that's the only way I can think of to break the cat and mouse game dynamics at play here.
Google, Facebook and others will never do this because it's expensive. Their cost savings is our toxic waste.
So what does a viable solution look like? I'm not sure. Maybe a system with distributed, volunteer moderators like Wikipedia. It's not perfect, but that model has held up remarkably well over its lifespan and has seemed to mostly hold its own against increasingly sophisticated attacks.
Figuring out how to scale it may be a challenge, but it may be worth considering something along the lines of the directory sites that used to be the main method of discovery on the web prior to the rise of search engines, which itself can be searched.
I would personally find this useful at least, particularly when diving into unfamiliar topics. In a lot of those situations, a handful of high quality links can preempt a litany of more granular searches.
This might be an area where a traditional, respectful search engine style crawler backed by a small cheap LLM paired with human curation would work well. The crawler+LLM finds sites, attempts to compute some quality score, figures out where they'd fit in the directory's hierarchy, and then submits them for human review and approval. If a poor quality link makes its way through, users can flag it for review and if curation agrees it gets booted.
While the article is more about specialized search (videos, shopping) for general web search, Kagi has to be mentioned. It's been so long since I've used another search engine (years), I'm generally shocked to see what they look like and how they behave now. It makes web search just work, including things like exact phrases with quotes, logical operators, etc. which are mentioned as generally rotting capabilities.
Honestly I haven't found kagi that useful in the last 12 months. I used to be a paying member but then dropped it because the results didn't feel more useful than just asking the free versions of Gemini, claude, etc
Tbh I too have experienced the same but I think that is just what we are moving towards. I mean everything is now shifting to semantic search and it is better for natural language yeah it sucks that literals dont work as good now but "Optimization".
I'm increasingly thinking that the only solution may be to have a search index with human editorial control. As much as I think Google deserves scorn for how badly they have made the search experience on the web, we have to remember that they're fighting a battle against SEO and spam. It feels like they have mostly given up. Any automatic solution will always be an arms race where they will be one step behind. The only way to get ahead is to ensure that all content has to pass through human moderation before it can be included -- that's the only way I can think of to break the cat and mouse game dynamics at play here.
Google, Facebook and others will never do this because it's expensive. Their cost savings is our toxic waste.
So what does a viable solution look like? I'm not sure. Maybe a system with distributed, volunteer moderators like Wikipedia. It's not perfect, but that model has held up remarkably well over its lifespan and has seemed to mostly hold its own against increasingly sophisticated attacks.
Figuring out how to scale it may be a challenge, but it may be worth considering something along the lines of the directory sites that used to be the main method of discovery on the web prior to the rise of search engines, which itself can be searched.
I would personally find this useful at least, particularly when diving into unfamiliar topics. In a lot of those situations, a handful of high quality links can preempt a litany of more granular searches.
This might be an area where a traditional, respectful search engine style crawler backed by a small cheap LLM paired with human curation would work well. The crawler+LLM finds sites, attempts to compute some quality score, figures out where they'd fit in the directory's hierarchy, and then submits them for human review and approval. If a poor quality link makes its way through, users can flag it for review and if curation agrees it gets booted.
On one hand they fight SEO and spam, on the other they take advertising money to manipulate the order of search results...
While the article is more about specialized search (videos, shopping) for general web search, Kagi has to be mentioned. It's been so long since I've used another search engine (years), I'm generally shocked to see what they look like and how they behave now. It makes web search just work, including things like exact phrases with quotes, logical operators, etc. which are mentioned as generally rotting capabilities.
Honestly I haven't found kagi that useful in the last 12 months. I used to be a paying member but then dropped it because the results didn't feel more useful than just asking the free versions of Gemini, claude, etc
Tbh I too have experienced the same but I think that is just what we are moving towards. I mean everything is now shifting to semantic search and it is better for natural language yeah it sucks that literals dont work as good now but "Optimization".
If I quote a search terms, every returned result must contain that string. Even if that means showing zero results. Anything else is madness.