I have no specific information here, never seen/used Shepherd until I saw a list on there a few days ago. Disclaimer, I work for Google, but not on anything related to this space, and this is based on my previous job where we did some SEO for an ecommerce site.
The example list given just looks a lot like spam when you squint. It's a list of affiliate links to buy products, and there are many HN threads talking about the abundance of affiliate link aggregators being a blight on the web. The commentary does look useful, but distinguishing between good commentary and bad commentary is hard, whereas distinguishing between a site designed to extract affiliate commission vs one more about the content is easier.
The comparison given to the other results here is frustrating, I know, but probably not a valid experiment. All the major search engines change results based on the user using them, or the IP address, or the region, or whatever, so it's impossible to know what others see. The developer of a book-focused shopping site is likely to get very skewed results for a book related query. My results were noticeably better.
The author says that a Bookshop.org list they created that links back to Shepherd is ranking #2, and this kinda makes sense to me. Bookshop.org sells the books, it makes sense that would rank above a site that only links to (and makes money from) sites that sell books.
SEO, and people getting annoyed at not ranking, has been a thing for 25+ years, I don't think this instance is any different.
> The example list given just looks a lot like spam when you squint.
I had to do some deep breathing before responding (creator of Shepherd here) :)
When I squint while reading a Pulitzer Prize-winning book, it looks like every other book with words everywhere.
When my wife really squints, she says I "look" like Brad Pitt.
I say these things with humor... as the core of what Google is supposed to do is determine what is good content and what is spam.
Is that hard?
Yes, but you have a lot of smart people and are swimming in money.
The point here, and with all the other posts by indie publishers, is that Google has destroyed them while elevating much worse results (they used to rank just fine along with other accurate results). I am just one example of many, and sadly better than most because books are a passion and not that commercial. For indie websites scientifically reviewing air purification systems, they are destroyed, and now you just get Forbes spam where nobody reviewed it.
> with all the other posts by indie publishers, is that Google has destroyed them while elevating much worse results
While I agree with your overall premise, some of this seems very subjective, re: "much worse results". For example, in your post when you list the pages outranking your site for "best books on Battle of Midway", you say this about being outranked by a forum thread on BoardGameGeek:
"A 2010 post on a Board Game forum asking for good books about the Battle of Midway. Really Google?"
BoardGameGeek is pretty much the most important site for the entire board game / tabletop gaming hobby. It's been around for 25 years, and it is most definitely part of the independent web. The thread in question is on BGG's main wargaming forum; wargaming has been a thing for many decades and has significant crossover among officers of the military / ex-military. The thread includes a lot of organic book recommendations from real people, with no affiliate links.
Despite the thread being from 2010, I would expect this page absolutely should rank pretty well for this search query! So why the "Really Google?" snark, especially when describing a fellow independent web site?
Yeah, it's hard to take OP seriously when it just comes across as sour grapes. Especially when their substack just seems to be a means to serve affiliate links.
> SEO, and people getting annoyed at not ranking, has been a thing for 25+ years, I don't think this instance is any different.
"People have always complained about X" is always a bad argument: it's entirely possible that X has gotten much worse, and yes, there were complaints before and after.
In this case, there is a very loud chorus of people saying Google has gotten much worse in the last few years, and it certainly matches my experience.
> The author says that a Bookshop.org list they created that links back to Shepherd is ranking #2, and this kinda makes sense to me. Bookshop.org sells the books, it makes sense that would rank above a site that only links to (and makes money from) sites that sell books.
In other words google is giving preference to a site that probably pays them for ads to a site that is effectively a competitor (since the site would make revenue from affiliate links that google might have gotten from ads, especially if the user liked the site and went directly there for future book recommendations).
Also, if I am searching for the "best" of something, I want something with commentary, not a list from a vendor without relevant information for choosing the right product.
Do you use Google anymore? It has gotten so bad, I regularly can't find things on it anymore, no matter how I search for things it will keep showing the same results. You seem to have blinders on. How do you explain the site getting less and less traffic over time?
I have not used Google for a couple of years (with the exception of Streetview where they do not have any competition at all). So I cannot comment on the quality of their search results. They are a close to monopolist / duopolist / oligopolist in too many areas. Ethically such a bad company that I don't want to touch them with a stick. Unfortunately masses don't understand that and contine to use them. And engineers are greedy enough to work for the evil.
I personally hardly ever use any version of streetview because I don’t feel the need to so forgive me if I am missing what’s so great about Google’s version.
One day I tried a Google search and noticed that everything "above the fold" on the first page was either a Google property, an advertisement, or both.
I stopped using Google for web search, but I still use it for Google search.
Part of me wonders if the ad market is just radically different on Google in the US or other parts of the world, or if I just make very non-commercial searches. I don't see many ads, unless I'm actively going to Google Shopping essentially in search of them.
I do see the Google info box style results, but I find these to be one of the most useful parts and it's one of the reasons I like using Google for things that are basic facts, media fact finding (like "who was that person in that show" style queries), etc.
In my experience (American), the top 1-4 results are ads/sponsored links. Sometimes, very relevant to the search. Most of the time, not what I'm looking for.
I only use google search on my phone because I have a pixel, and the google searchbar is so well integrated into the launcher. If I don't like the results, I can just open a new tab and the default search is DuckDuckGo. Besides the ads, I'd say the searches are usually the same quality.
If they wanted to fix it they would. Most likely someone came up with the great idea of "Hey if search sucks, some losers will pay for AI hoping that fixes it, we are so smart"
One idea I've had is that it would be interesting if Google created a webmaster partner program to help build the web they want to see. Webmasters could join and embed Google's code into their website, and Google could get access to engagement stats and anything else to help them determine positive reactions by users (instead of pretending they are not pulling it from Chrome - see DOJ trial).
At YouTube, Google does a great job of helping creators, telling them specifically what they want, and promoting high-quality content that people love.
I'd love to see Google get the data they need so they can do that.
Is this not literally what Google Analytics is? I'm not sure if they're using it for everything you're hoping to have, but it seems like it basically performs that function.
They they are not allowed to use the data... which seems weird given Chrome and DOJ details. But I've never seen any details of them using it, and every statement is they don't use it.
> It's a list of affiliate links to buy products, and there are many HN threads talking about the abundance of affiliate link aggregators being a blight on the web.
This is a pattern that was made hugely popular specifically by google, by the way
It looks like shepherd.com is a book review site that doesn't have any reviews, just ratings. It links to Amazon.
Is this really the sort of content Google should be returning?
Edit: it seems I missed the link to the actual book reviews because the link text is uninformative: "Chosen by 1 person - see why." (Sometimes it goes to reviews, sometimes not.)
And the word "review" never appears on the pages that have reviews. Seems like bad SEO?
If you're looking for book reviews, here's a website with some pretty great reviews: https://www.thepsmiths.com/ (Content warning: the authors are conservative.)
Hi Brian, we don't have reviews at this stage; we have purely positive recommendations. And we push readers toward a very specific format for why the book resonated with them (Gestalt psychology so you see it through their eyes).
Given the nightmare that is Goodread's unmoderated review sludge, I wanted to focus on the positivity and why someone loved a book, so we are uniquely focused on highlighting only books that readers love.
In 2025 I will be adding "reviews" of any books using our Book DNA review format (which tries to narrow in on why someone liked/disliked a book so we can match you with people who share your Book DNA). A big part of my mission is to focus only on books that people love and positivity.
One thing I find especially bizarre about the current situation: the pages that are making it to the top aren’t even pretending to be real content. If they were difficult-to-detect LLM-generated pages, that would be one thing. But they’re generally extremely low-effort affiliate spam, mostly claiming that they researched something “so I don’t have to”, followed by a bunch of Amazon links and explicitly acknowledged scraped reviews, and finally an obviously uninformative summary. They don’t even pretend to have real content!
A lot of us believe that Google trained their AI on spam websites, but in the process, the AI now identifies a lot of things that independent websites have on them versus big brands like Forbes, which spam and license out huge sections of their websites to spammers. So we got pulled into this and Google doesn't care about the false positives.
At this point, a lot of websites are not coming back. So get used to Forbes and other large brands being the only results along with AI bots on Quora.
8-10 years google search was amazing. A well crafted query would hit informational gold most of the time. I've been noticing and commenting on the decline privately for most of the period, but it's only in the oast year this seems to have come to broader awareness. There's an argument that in an information economy, searching for information should be treated like a public utility necessary for the functioning of society. I'm not making that argument but when you experience the long slippery slope of degradation of a service that was near ideal for the technology of the time, it does xome to mind.
That's the thing. It DID work. Really well for a while. But it was always atomic and context-less. We now have the opportunity to make it even better by refining results through dialogue. I hope someone does.. soon.
The slippery slope began in 2012, when they pledged to start downgrading piracy websites. After the rollout, the followup questions were "well what if we downgrade other topics that [random government] doesnt like" and "what if we sell the ability to boost enterprise company results for certain topics". eventually the number of results that get filtered out or reordered, exceed the results that actually get displayed. creating a new search platform is not a viable solution - any private company will be incentivized in this direction until some kind of "search neutrality" law is introduced.
I'm not convinced this is the explanation. It's true that for some product-centric queries, you get mostly paid results. But for information-seeking queries, Google tries to give you organic results.
The problem is simply that there's too much money to be made by capturing these. For years, you had content farms and fake "review" sites stepping up their game. Now, LLMs essentially make it a losing proposition to try and surface the small web. The least-bad option for Google would be to send you to moderated communities, such as Reddit, Quora, Stack Exchange, Wikipedia, and so on. But not all queries can be handled that way.
If you look at the article that these guys are complaining about... how do you distinguish it from content-farmed spam? You can't.
Now, I think Google is throwing in the towel and just want an LLM to answer info queries instead. That has a ton of problems, but to the average user, probably feels more helpful. At least until the spammers start gaming that.
Sometimes I lose track of a bookmark and google a phrase from it verbatim and in quotes. Guess what I get most of the times? Not even a verbatim search spam, just nonsense with almost no original words. It doesn’t even try and I doubt it even has a database where a direct path exists from a phrase to it with a link. Google is not “I found this for you” anymore, it is “I think you meant this”.
I thought this was just me! Even if I use double quotes to search for a 1:1 phrase I often get bad results and nothing linking back to the source I was looking for. If you happen to remember the site something was on, adding "site:example.com" still produces decent results fortunately.
Not really? The vast majority of helpful websites monetizes the same way the spammers do.
Plus, even without affiliate links, purchase attribution is already scary good. Did you know that credit card issuers and ad tech companies collaborate to attribute brick-and-mortar purchases to online ad views? You have untold billions of dollars at stake. The industry is not banging rocks together.
You could make a search engine for non-commercial content only, and I would actually love to have that, but (a) it would be a woefully tiny sliver of the internet; and (b) Google is obviously not the right company to do it.
At its scale google could just hire a big group of diverse internet-aware professionals (two of each kind) who would do random searches and simply manually ban sites that restyle or spam content. It’s absolutely easy for a searcher to tell if it’s spam.
This is actually an interesting rabbit hole. Their other constraint is that they are trying to be transparent and fair, for some definition of that word. They need that for Section 230 protections, they need it to fend off antitrust lawsuits, etc.
So, there is a huge problem with hiring experts and telling them to make subjective decisions. Instead, Google publishes guidelines for webmasters and then enforces them rigidly. This usually ends up penalizing some good sites, while spammers swiftly discover workarounds.
then make it a gov position, like the "Beer and Malt Beverage Labeling and Formulation Approval" position, which consists of a single guy approving or rejecting the graphic designs of beer bottles/cans. there was an NPR segment a while back about the one guy that held that position for 40 years.
We are approaching a digital Kessler syndrome, or perhaps a Deepwater Horizon info-oilspill event, where there is so much useless SEO-driven slop (soon to be taken webscale with the advent of genAI) to sift through on the internet that it's growing increasingly difficult to find the signal amidst the noise. Google, once a prestigious company which prided themselves on "organizing the world's information" and "not being evil," eventually became a target for those wanting to peddle their wares and make a quick buck - a departure from the days of the early Internet which was mostly computer geeks, hobbyists, and forward thinkers sharing organic content they thought was interesting or useful. Because search was Google's entire business, they needed to develop countermeasures to combat spammers and pages gaming Google's algorithm with questionable SEO techniques in order to preserve the signal-to-noise ratio of the search engine results page.
This is now a bygone era - after discarding their original motto of "don't be evil," search and "organizing the world's information" are no longer Google's business, it's hawking advertisements [0]:
When Gomes pushed back on the multiple requests for growth, Fox added that all three
of them were responsible for search, that search was “the revenue engine of the
company,” and that bartering with the ads and finance teams was potentially “the new
reality of their jobs.”
On February 6th 2019, Gomes said that he believed that search was “getting too close
to the money,” and ended his email by saying that he was “concerned that growth is all
that Google was thinking about.”
Hence questionable grey UX patterns like blurring the distinction between ads and organic content, and sometimes cramming the page so full of ads that all the actual results are "below the fold." Remember the old Internet adage - if you're not paying for the product, you are the product - and like cattle we are all just herded into digital pens to be served marketing slop to serve the real customer - the advertisers.
If you want to be treated as a customer instead of cattle, you ought then to pay for your services, including search, to align the financial incentives with your own. Advertising needs to die, for it is a root cause of most of the ills of the modern internet [1]. If you can pay for streaming services or music, you can certainly pay for access to high quality organic information that actually aligns with your interests - not that of the advertisers. I've been using Kagi for a few years now and it really does hearken back to Google SERP quality maybe not at its peak, but rounding near to it.
At the risk of sounding elitist (and so what), this is just another consequence of the recurring Eternal September phenomenon - highly focused communities with a strong concentration of geeks, hobbyists, and experts were the norm back then, when computers were still new and arcane devices that were difficult to operate. The bar to entry was much higher, and one had to do a little bit of "reading the fucking manual" simply to get online and understand how to navigate the net effectively. Now that all the balls have been poured into the Galton board we have regressed to the mediocrity of content that exists on the contemporary Web, absent those pressures that once selected for high quality content online.
>If you want to be treated as a customer instead of cattle, you ought then to pay for your services, including search, to align the financial incentives with your own. Advertising needs to die, for it is a root cause of most of the ills of the modern internet. If you can pay for streaming services or music,
People pay for their Windows license, yet Windows now has ads baked into the start menu. People pay for Youtube Premium, but most videos now have "sponsor segments" -- yet more ads (though admittedly not controlled by or directly profiting YT). People pay for streaming services, but last I heard, Netflix was adding ads. Ages ago, people paid for cable television, and it wasn't long before it had ads too.
These companies are going to treat you like cattle whether you're paying them directly or not.
Then vote with your feet and your wallet by leaving. You need to send monetary signals that the service is undesirable by withdrawing your subscription. I don't understand why people continue to use tools that don't serve them and assume, by default, that because everyone else is on some platform, that I must be there too, or don't go into an actual cost-benefit analysis of whether or not the utility of the platform outweighs the drawbacks. Cal Newport comments on this default herd mentality in Deep Work:
The Any-Benefit Approach to Network Tool Selection: You’re justified in
using a network tool if you can identify any possible benefit to its use,
or anything you might possibly miss out on if you don’t use it.
In the case of Google the diminishing utility of search began to be outweighed by the increasing morass of SEO slop and advertisements, so I degoogled. After W7 when it was clear Windows was going to progress into a cloud-based ad delivery platform, I installed Linux on all my devices as a daily driver. If Netflix ever serves me ads, I am immediately terminating my subscription.
>Then vote with your feet and your wallet by leaving.
Yep, that's what I keep saying too, but I always either get ignored, downvoted to oblivion, or responses about how X is "necessary" because of Y (which isn't really very critical anyway, but I guess to them is, like "I must be able to play game Z!!!"). You can lead a horse...
bingo, this is why i mentioned (in the skip level comment) that all private companies will go through this sequence until a "search neutrality" law is introduced.
after MBA's start to get diminishing returns on new subscriptions per month, the focus shifts to advertisements.
> The least-bad option for Google would be to send you to moderated communities, such as Reddit, Quora, Stack Exchange, Wikipedia, and so on. But not all queries can be handled that way.
There's a wide gulf between "path of least resistance" and "least-bad". I don't think it's productive to conflate the first path with the second.
> "But for information-seeking queries, Google tries to give you organic results."
No, often it absolutely doesn't, and I posted two epic fails here previously. Worse still, Google organic results no longer understand(/distinguish) the difference between information-seeking queries vs product searches (or else, SEO people have been gaming it for years, and Google search rankings have made this worse):
Googling for "Africa longitude" should return a range of longitudes, like: "17.5°W - 51.5°E" or
"["17°31′13″W - 51°27′52″E"]" [1], or for the A+ answer:
*"Africa lies between 17°33'22" W, (Cape Verde, westernmost point) and 51°27'52" E, (Ras Hafun, Somalia, easternmost projection).
But it doesn't. Moreover these coordinates haven't substantially changed in 10,000 years (other than political/territorial disputes about islands, but the coords for the mainland certainyl haven't).
Googling returns the grossly misleading "Africa/Coordinates 8.7832° S, 34.5085° E". When you dig into why this so, it seems to be "optimized" for the SEO activities of a digital map storefront, MapsofWorld.com, acquired by MapSherpa Inc., based in Ottawa.
And those mystery nonsense coordinates ("8.7832° S, 34.5085° E") bizarrely point not even to the geographical centre of Africa but to a random rural location 2400km ESE away, in southern Tanzania, which appears to have been deceptively mislabeled, in Cyrillic, as a Russian store (by Russian SEO?). For a pin dropped in rural Tanzania. No QA!
(Update: the AI overview factbox has at least since been corrected to give Two: "Chang'e 4 (January 3, 2019) and Chang'e 6 (2024), instead of repeating what jagranjosh.com says).
However the #1 organic search hit is still the woefully inaccurate unauthoritative page:
https://www.jagranjosh.com/general-knowledge/list-of-all-suc...
which cheerfully claims "Aug 23, 2023 — There are over 21[!!] moon missions that have been launched successfully on the dark side of the moon by 4 countries."
This is hopelessly wrong, even if we utterly misunderstood the key word "landing" and also count any mission which merely photographed the dark side (Luna 3, 1959) or human overflight over it (Apollo 8, 1968). But not "landing".
And why on earth did Google decide jagranjosh.com was more authoritative than any reference website or wiki?
Well to be fair there is no "dark side" to the Moon, so it's not really an answerable query.
And searching google.com for "longitudes continent africa" returns "Africa's geographical coordinates span from Latitude 37˚21'N to 34˚51'15"S and Longitude 51˚27'52"E to 17˚33'22"W"
> what if we sell the ability to boost enterprise company results for certain topics
Maybe I'm naive but... has Google even considered doing that? Re the first one, I know top management had a project to reenter China back in the teens.
Exactly, the organic nature of search ceased to exist that very moment when they started human intervention in a purely machine based (crawler/indexing logic) algorithm.
> any private company will be incentivized in this direction
From the earliest papers on Google PageRank, Brin and Page warned, "we expect that advertising funded search engines will be inherently biased towards the advertisers and away from the needs of the consumers." <http://infolab.stanford.edu/~backrub/google.html>
> We now have the opportunity to make it even better by refining results through dialogue. I hope someone does.. soon.
TBH, I don't want dialog (too long and slow, leave dialogs for human to human comms), I just need my query words to exist in the result pages. Nothing more. Google messed this up long time ago.
> I just need my query words to exist in the result pages. Nothing more. Google messed this up long time ago.
Google didn't mess that up, SEO did.
You don't "just need" that. You need the page to be authored by a human with good intentions and end up containing those words because those are meaningfully part of what the human was trying to convey to you without ulterior motives.
But those kinds of pages are dwarfed by the millions of pages of content farm nonsense jammed full of every possible keyword and contaning absolutely nothing of value.
A search engine like you describe would be like walking in Chernobyl without a radiation suit on. A pleasant stroll in 1985, but not today.
Search only works if there exists something valuable to search for. The Google dominated internet has not incentivized a library of quality content. Most quality information exists on long dead domains. The rest is pay walled or unindexed. Google is a library search service but for a gas station convenience store of information.
8-10 years ago the competition for getting stuff before your eyes wasn't as tight. It's not only on google, the quality stuff is objectively harder to find in the ocean of slop today.
If you weren't born yesterday, or remember that "enshittification" describes a process (great for users, then great for businesses, then great for the middleman), then you already know that it's just a matter of time before Substack becomes Shitstack.
That is a private blog, so I don't care if anyone sees it. I just wanted something I updated every 2 to 3 weeks with my frustrations, wins, and so on as I build Shepherd.com.
I was on Twitter's free email thing but they shut down and imported to Substack. I just didn't want to pay for anything as the only person who reads it is me and my mom. It has 100 readers on it, as its just for me really.
Technically, yes. If the thesis is "Google search is inaccurate and overrun with SEO optimized slop and ads", them using a blog utilizing that same SEO optimization feels a bit ironic. Feeding the beast you want to change.
Because every additional article posted on the non ‘indie web’ is literally reducing the relative credibility/importance/etc. of the ‘indie web’ by a tiny bit… which is unnoticeable for just one, but when you add up millions…
1) It has already been proven by research that Google quality of results slips from year to year [1]
2) Google has many incentives to make the search more difficult for you
3) Google has proven that it prefers money over quality of results with allowing "malvertising"
4) It is true that the landscape is more difficult. There are more walled gardens, to which even Google might not have access. There are more scams, casinos. More AI slop. The game was always hard on the other hand, so these are just 'excuses'
5) Why so often I see in Google results leading to major news sites instead of normal links?
6) If I write "Warhammer" I would expect thousands and thousands of pages in results. I think that Google prefers "content" over "quality". "newer" is "better". I would expect thousands of fan pages, which do exists, but are not crawled, or forgotten. Why can't I browse older pages? Why is there a limit to 10 pages?
7) For "Emulation" first page leads to "wikipedia", "cambridge dictionary", "vocabulary", it is so f boring
The problem is people don’t “just”. We’re complicated critters with our own needs, capabilities, and goals. At an individual level, a person may not be capable of the “just” due to factors outside of their control. With a society, the resistance to change grows exponentially. You can’t scale one person at a time.
This doesn’t mean you shouldn’t try. Sometimes you can make a big difference to the people around you. But you’re gonna have a difficult time changing in the world.
In this case it really is easy to try a different search engine and switch to it to the extent you prefer it. At least if you're not too uncomfortable using software.
Are you saying friends and family using Google are an obstacle to you using something else? Or maybe if you use something else, that makes you so uncool that your friends and family are not more likely to follow you, but actually less? I guess what you're actually getting at is that my switching doesn't automatically make my friends and family switch, but why insist that change only happen in blocks of people all at once? How does that help?
The original comment was pointing out that in order to break the monopoly of Google, the wide society will need to switch to something else, not just a few tech-interested people.
The thing is, that kind of logic claiming the futility of individual action does not weaken the grandparent's advice.
If you're an advertiser, you can't just switch from Google and hope to change anything. You'd be hurting yourself to no benefit, probably. Collective action is your main hope.
It's not like this for users of search. Google can lose its search leadership one user at a time. When the user switches, "the problem is gone" for that user, and Google has that much less revenue. Gradient descent is powerful.
Of course it’s easy for anybody on this forum to change their search engine.
That’s the “thing”. The thing is always trivial to the person making the statement. The thing is completely irrelevant to the point I’m trying to make.
> With a society, the resistance to change grows exponentially
Yes and no. There's certainly the potential for a more flexible and adapt-minded culture for the same reason there's not... Human behavior.
Yes, once norms are established they're slow to change but that's how the culture is nudged and "managed". It's not necessarily how humans arw hardwired.
Eons ago, slow to adapt would mean certain death. Currently, fast to adapt labels you a rebel, a freak, an odd-ball, etc.
And I'm saying that the current system is "optimized" by the powers that be to not change too fast. Humans have and can adept if they're allowed to do so. The current sociopolitical system doesn't allow this. The bottleneck to change isn't humans, it's the system.
Me subscribing to it is. Getting everyone I know to subscribe to it is at least a step more complex than getting people to move to Firefox, and I haven't quite worked out a strategy for that yet.
Search results being absolutely shit is probably more of a problem for people than their choice of browser.
When someone complains to you directly about Google search results being shit, maybe point them to Kagi and mention it has a free trial account (100 searches still?) that's not time limited.
That way they'll have a solution they can turn to the next time Google pisses them off significantly. :)
The above marginalia.nu just surfaced the following very nice article for me, which neither Google nor Kagi returned in their top 10 search results for the same terms ('thaad aegis hwasong'):
Are you really comparing typing a dozen characters into a browser's URL bar and hitting Enter with "stop war"?
Using a different search engine, is just about the easiest kind of switch you could do. It's easier than, say, eating a different type of snack than you usually eat, or switching out your usual mug/plate/utensil.
Using a different search engine is easy. Voting for a good politician is also easy. But getting other people to switch to a different search engine is hard, just like getting other people to vote.
Long ago I wrote a blog post[1] about intermittent hardware, and not letting it suck you in. This was posted in a Blogger/blogspot blog, which is owned by Google.
This morning Google couldn't find it, neither could Bing.
(granted, I had to perform search with quotes on, but both Google and Bing still fail)
Hopefully demonstrates the difference between legacy and a modern search engine whose entire purpose of existence is to surface what you want to find, not what other people want you to see. (wave from Kagi founder here)
Visits from Google to one of my independent sites have been about the same in 2022, 2023, and 2024. Visits to another have roughly doubled from year to year. I don't use Google Search or surveillance ads but I don't have the issue that they report.
AyyEye's observation that their site is loaded with trackers and what look a lot like affiliate marketing links is one reason why Google Search might not like it any more.
Stats are fine, as sometimes people mistake a single data point for a truth. Might take a look at my post as there are links as this is a huge issue affecting many many many websites.
I don't have access to your "many friends and acquaintances who are going bankrupt and shutting down fantastic independently run websites." I absolutely don't have access to the internals of their businesses to decide whether I agree that their problem is Google Search being hostile to independent sites. But I do have access to the traffic stats for the sites which I run.
Many of the sites with similar complaints seem to focus on product reviews and recommendations and have been hit by a specific Google policy which favors trusted and well-known domains for those types of search, even if that domain is not trusted for product reviews in particular. (So $glossyMagazineInNYC can open a line of product reviews and get lots of Google traffic, while $smallProductReviewBusiness loses traffic).
I decided to remove my browser condoms and check Shepherd's network traffic. Updated my post to better reflect how "independent" the website is.
That's less than half of the malware it tried to run on my computer but I got bored checking all the domains it tried to run code from. It's repetitive. I get it, you put on every tracker and ad network you could find, and put your stuff on a bunch of CDNs.
Everything that can be outsourced to big tech was outsourced to big tech. How very independent.
Independent means that we are a small team and trying to create something cool for people. It means we are not part of Forbes or some giant publication that is focused purely on money and spamming the shit out of you :). You want a diverse and indie web so that the only search results and voices you here are Forbes, Disney, and other giant media conglomerates.
Independent does not mean we don't eat.
Independent does not mean we don't have server bills.
Independent does not mean our part-time developer works for free.
Independent does not mean our designer works for free.
There is no malware on the website crazy; it's just an ad network that pays for our servers. I will remove all those ugly display ads as soon as we have enough income from readers and authors directly.
> It means we are not... focused purely on money and spamming the shit out of you
Sure looks like you are. In fact I can't even find a single datapoint to the contrary. My browser made over 900 requests to other entities who's raison d'être is spamming me for money.
Even your default affiliate link is Amazon instead of Bookshop for crying out loud. You didn't even put them side by side -- you hid bookshop behind a drop down menu. Seems like you don't care one bit about being independent.
> You want a diverse and indie web so that the only search results and voices you here are Forbes, Disney, and other giant media conglomerates.
I can't tell the difference between shepherd and the rest of the dead internet. There's only one person or a small team behind most blogspam and SEO-spam but it's still ruining the web.
Independent web does not mean "We're not rich (yet)". Also 'web' implies links to other indie websites. A (hidden, even. Lol.) affiliate link to bookshop doesn't count.
> Independent does not mean we don't eat.
Independent means... Not dependent. You are dependent on Medium for Shepherd's blog for crying out loud. You could host your own blog on your server in minutes for zero marginal cost, and yet there you are paying big tech to put some words under a header.
> Independent does not mean we don't have server bills.
Use less (as-a-service). Even a small and cheap dedicated server or two should be able to handle this just fine. Maybe not as-built but you're doing nothing that needs expensive hosting.
> Independent does not mean our part-time developer works for free.
You have nearly 1000 subscribers and the minimum tier is $25. That's far from "works for free".
> There is no malware on the website crazy.
You don't even know what runs on your own website. It attempted to run code from hundreds of places who's whole purpose is to dox your users and run targeted ads to take their money. (That's money that might have been going to shepherd fwiw.)
> just an ad network that pays for our servers
"just" dozens of them. "Just" dozens of tracking and analytics.
There are ways to do ethical advertising without tracking. You could even sell ads yourself (maybe you are, I haven't looked that closely at the website).
How many servers do you have that those are a significant cost center when you have multiple employees?
> I will remove all those ugly display ads as soon as we have enough income from readers and authors directly.
Going into this I thought you were making shepherd yourself, but reading between the lines, you already hired 1-2 (or more) people before you made money. That's fine if this is a business venture -- but speculatively hiring multiple people in the hopes of becoming profitable is far from the "indie web". Light years away. It's standard operating procedure for the hyperscaler corporate web though.
I'm not trying to be harsh here, I promise. My original post had a little bit of snark because of the medium blog. But every time I look at this it gets uglier.
If you truly care to be independent you can. Get rid of the garbage you've accumulated (ad networks, trackers, minimize APIs, bring everything in-house). Slim down the network requests and JavaScript. Rethink how you do ads. Lastly, focus on being part of the indie web instead of another cog in the corporate web machine.
Keep doing what you love. I am not critiquing the functional bits of the website. Just the 'independent' bits. You can do a blog in-house easy. Maybe kill the analytics and try to figure out how to do some in-house ads? Also put bookshop ahead of Amazon.
If you have any questions I'm happy to help you break the chains that bind you! Come on in, the water's fine.
Any idea how many Googlers still have the pre-dotcom-boom Internet technologist-citizen mindset?
I know a bunch flocked to Google early on. But the landscape has changed a lot since then, and many people weren't even born until after the Internet and society were very different.
This problem is age old and always the same. Optimizing for Google is a fools errand. You can win in the short term, but when their priorities change you will find your website excluded for the very things that boosted it previously.
Google are unreliable and untrustworthy. Their focus is ad revenue for themselves and nothing else. Build sites for humans and let Google do what it wants to.
Only somewhat related, but I was just complaining about search in Gmail. I normally use Thunderbird and didn't have it, so I used the web. Basic searches on the subject were so bad! I even put quotes around it and tried all of my Google-foo.
It felt like I was fighting some AI that was sure it knew what I wanted despite my "exact phrase" search.
Google got so big it swallowed the internet. It now has digested it and what is left is... this.
I used to love crafting websites and cared about SEO. What's the point now, no one is going to find your content. It won't even be on the third page. Google will answer questions by regurgitating whatever it swallowed on your websites and presume no one will click through, it won't even bother marking the authors.
Instead it appears to be prioritising whichever website is going to give it revenue first, e.g. the click farms.
The regular folks don't care, they google for stuff like "am I dying if I have a pimple?" (to which the answer is always yes, apparently). No one does actual meaningful research using Google anymore, if you do, good luck, get your gloves out <picture of dinosaur poop in Jurassic Park>.
The global internet as it stands is close to dead. Discoverability of "cool" things is down to social media, tricked by "influencers", who are tricked by marketing themselves.
We need a hard reset button, it needs to start from the ground up with site rings, and good content. Ah... that last part, "good content", is now stuffed with AI Samey McSamey sounding text. I really don't see a way out of here.
The funny part is we used to think that the internet was going to change the world. We thought all idiots needed was information. Access to information would fix the world! Instead, it only has given the village idiots a global voice: if you can think of some dumb crazy thing, you'll find dumb crazy people agreeing with you, so you must be right!
I've been on the internet since 1997 and I think it's the worst it's ever been.
> The funny part is we used to think that the internet was going to change the world. We thought all idiots needed was information. Access to information would fix the world!
I was 9 in '97 and wasn't really aware of the internet until maybe 3 or 4 years later. The internet seemed like a magical place, where reason and common sense existed (unlike the messy meat world). I bought into the same ideals that information was power and once everyone had access to information, we'd collectively get smarter and wiser. Less wars!
I remember being excited for Google Search and then Gmail. FINALLY, a company that gets it!
It seems that any public or VC-backed company is destined for enshittification.
I wonder how feasible that is without leveraging one of the two existing search engines in the backend. I always pitted a general search engine as a top 5 difficult tech project to go for.
Ctrl+F'd for Perplexity. I knew Google was cooked the minute Perplexity worked better for questions about an obscure embedded systems SDK. It has little documentation, but a lot of mailing list and github issues. Google spits out the front page of the project and shrugs; Perplexity actually answers the question. The usual caveats for LLM hallucination apply.
Same. Try the "books on the Battle of Midway" query on Perplexity. The results are great and include the book mentioned in the article (authored by the Naval Aviator).
Always use google in incogniti so they dont give personalised results, thats the least we can do before google search completely dies,
Or just prevent google any cookie permission, keep it session only
As far as query trackers on google urls, they are necessary
One good case of necessity of google tracking parameters is
--> you search for movie or related, but muktiple movies by same name have been made , you dont remember year, you do remember some actor or story description
Searxg and google will in show with media carda foe movies you ckick iy and yoi get exact same film with year yoi wantes
Wait ->I am telling above because when that mefia image of film i inspecred itsurl it was just
Googke ?search=filmname&teackers
The search ket wasnt modified with year same as i searched first yet it gave me the desires year film
may be because of trackers &ved amd all paramers link tags to search teaukts its loke ctoss maching in databases
Google can't please all of the sites all of the time, or all the visitors.
It is too big to evem worry about that 4k a day clicks for one site. It is like us optimizing the expense of 0.01c. It makes a difference when that 0.01c is an API call that you call a million times. But it only surfaces if you do aggregate it.
Therefore this problem can only even be seem by Google if it can be surfaced in aggregate overy say a billion queries.
I wonder how that can be done.
Probably only can be done using data. Which means spying on people in various ways. And making assumptions about length of time on site equals quality.
They probably use machine learning too. There may be no reason for the lost rankings other than a wind change caused by some updated parameters in an OKR chasing model.
I realize "did you actually read the article" is against HN guidelines, but what else am I supposed to say when I see this at the top of the comment thread?
I mean when you say "Google can't please all of the sites all of the time, or all the visitors.", I wholeheartedly agree, but this blog post was excellently sourced with data that shows exactly how Google raising sites that any reasonable human would say are considerably shittier than this site that is getting down ranked. It also seems pretty clear that the things that have changed are Google's ranking algorithm at specific points.
> They probably use machine learning too. There may be no reason for the lost rankings other than a wind change caused by some updated parameters in an OKR chasing model.
That is literally what TFA says in the very first section: "Some people believe they have lost control of their AI ranking systems, ..."
Why don't people just stop using Google? And by people, I mean everyone here.
Whenever this point comes up, I see people claim they ONLY see the results they want in Google. How would you know if you don't actually use anything else? Kagi is excellent search. Neeva was pretty great when it was active. DuckDuckGo is passable. Idk how Qwant gets money but it's been around a bit.
Complaining about the same thing forever and expecting a change doesn't make any sense. Y'all are in abusive relationships with Google and refuse to leave. Sure, your job may use Google Suite and you need to make money. What about the rest of your life? Stop hitting yourself.
Once I have stable income I do plan to use Kagi. Though I do recognize that even Kagi partially has Google Search underneath. Even if you fully de-googlefy it's hard to not be indirectly supporting Google.
Been working on it for 2 years now and even then I'm more or less locked into 3 services
1. Gmail, 20 year old account I use for pretty much everything business. Even if I move I'll need to forward Gmail stuff to a new email for years.
2. YouTube. Pretty self explanatory (and the go to for why monopolies are always bad). Trying to avoid it entirely is like trying to avoid dang Twitter. Too many other companies use it as a go to for any video, no matter how inefficient (nothing better for graphical showcases than nitrate compression ruining all the details)
3. Play store. Used android all my life and while I can mostly move out I will be missing some critical apps from that result (financial apps are a big example)
It's a network effect like any other for 2 of those, and a lock in part of my online identity for another.
I can't help you with points 1 and 3 but for 2: do you comment on YouTube videos or upload your own? If you just consume content (like I do), you can subscribe to your favorite channels with RSS. Go to a channel and inspect source for "rss" to find the link. I'm partial to NetNewsWire for my feed reader.
I will boost this. I am very happy with Kagi. I'm on ultimate and am glad to have my money taken for reliable and high-quality search.
> Complaining about the same thing forever and expecting a change doesn't make any sense. Y'all are in abusive relationships with Google and refuse to leave.
This would be a more solid stance if they were complaining about this as a consumer and not as a business operator. They can't control what search engines other people use. The best they can do is optimise for other engines like Bing and DuckDuckGo.
The other alternative is finding other avenues to advertise away from search engines, which may be what you were alluding to.
> The other alternative is finding other avenues to advertise away from search engines, which may be what you were alluding to.
I was actually speaking from the consumer perspective, thanks for making the distinction. The OP has valid reasons for being upset with the most popular search engine ruining visibility.
To be fair, I have no idea what business could/should do. Bigger businesses can afford to hire social media folks to keep them relevant on Facebook or whatever. SMBs/mom & pops are screwed.
I used DDG as my default search engine for several years, but I kept having to go back to Google because it was consistently more reliable even though it has more crap. I hear Kagi is great but I am not signing up just to find out, nor do I feel confident that 100 searches are enough to make a decision about paying for it. I think Google is garbage but I have yet to se anything that is consistently qualitatively better.
100 searches last you a lot longer than you think, especially if you delegate “postal code Wall Street New York”-type searches to Google via “!g” prefixing.
It’ll give you time to get used to Kagi and set up some personal up- and downranks and blocks. Especially together with the Summarizer and Lenses your search result quality will dramatically improve.
No doubt this is true for you, but my search usage patterns are quite different. 100 searches in a day is nbd for me, sometimes I go through that in a few hours.
That's why I emphasized "everyone here." My wife isn't gonna stop using Google, and Kagi isn't a default option choice on iOS. Nerds will go out of their way to use a product.
> Complaining about the same thing forever and expecting a change doesn't make any sense.
Do you know that this forum isn't one person and thousands of people access it in different days and time, right? The notion that the same people are complaining while still using Google is a projection of your mind.
The author talks about how long users spend on his page, but can Google track that? Has ubiquitous tracking blocking, and Google's failure to adapt, eroded the quality of search?
There have been links from the DOJ case showing Google is pulling in Chrome data.
But its just one data point and nobody knows.
It seems more likely at this stage that Google has lost control of their AI system. they trained it on x websites, and indie websites got swept into that. And they don't care.
We don't even have ubiquitously used ad blocking right now. Do you think some normie is even going to know what a tracker is and why they should block it?
Checking my Search Console Tools recently, the number of "Crawled - currently not indexed" pages across all my sites has risen sharply recently, especially on sites where the age of the content is more than a few years.
I put a fair bit of time into trying to improve the sites with JSON-LD and breadcrumbs and what not. It seems to have helped just a little bit.
I don't make any money off any of it, but it's still kind of irritating that no one can find it.
The indie web was around before google and it will be around long after google is gone. I would argue that the indie web has incurred a much larger loss from people thinking seo/engagement metrics are something worth optimizing. Many of the best examples of the indie/small web don’t have js tracking and little to no css.
Google didn’t shit the bed, they just did whatever was profitable to them and they’ll continue doing so until they either fail or there’s no more profit to be squeezed. They dropped the “don’t be evil” motto a long tome ago.
I've had my lifetime quota of hearing the rants of guys who think their search quality metric is objectively superior, oh and by a total coincidence their preferred ranking gives them a direct financial benefit. Just a complete failure to imagine the larger issues.
Kagi is a viable choice for individual people, but it's not an answer for the concern of how to get traffic to visit your site and click on your affiliate links (which is what TFA is about). Kagi works well for users precisely because it's small enough that no one benefits from gaming its algorithms.
Um, I don't know when that would have kicked in because I haven't noticed a difference. Reddit still shows up whenever it's relevant, and the Forums lens turns up even more Reddit if you're worried you're missing any on the initial search.
I feel too that we're already past the point where they could fix things. They must be aware of issues and do the exact opposite of fixing them. It's a question how fast we can convince general public to move away, rather than when Google will improve.
I wouldn't give much regarding the general public. If you have a small blog they're usually not your target audience. Accept that the days of "making a few bucks off of advertising" on hobby publishing are over, focus on writing about things you care about without it reeking of being written by ChatGPT and chances are people will come at some point.
If small publishers don't change their behavior in radical ways, nothing will ever change.
I don't get what you're trying to say. I'm not arguing in favor of doing nothing in an ever-changing market, which is what Myspace and Friendster did most of the time. I'm saying the old world of "have good content/SEO and you'll see some traction thanks to Google" is dying and that you can either adapt or drown in a flood of LLM-created SEO blackhat nonsense. Google does not care about you and they've been in the "too big to fail" corner for too long.
If you want content made by humans to continue to exist and want to see some of them be able to make a buck off of it, support smaller search engines and creative people and their publications.
I mean that services will come and go. Some will stay for longer than others, but even without a coordinated action, Google may go away. It's also losing to Perplexity and ChatGPT with many people. And many geeks are very vocal about their love for Kagi. (Including me) Things may change over time and may not be too big to fail.
It took me about two weeks until I got consistently good results with DDG, it often just takes about one more search term because it doesn't try to (often incorrectly) guess the context based on search history like Google does. That was years ago, since then DDG has gotten better and Google a lot worse.
I could understand your complaint if it was about image search, but I don't see how anyone can look at today's Google results and think they're still superior.
Google is so odd these days, its like there are two "streams" of results. One is the kind of useless slop that the article in question is rightfully complaining about, but other results are still useful and high quality. The second stream is what I seem to get when I'm forced to "!g" when I'm using DuckDuckGo, and I can't work out why theres this stratification, this separation. Years ago, it was very rare to get crap results from Google. Now its far more common.
Haven't needed to use Google at all since using Kagi several months ago, whereas when using Duck Duck Go for a few years (prior to Kagi) I'd (very occasionally) need to.
There are people inside Google who can put a stop to this, if they merely stand up and say "this is wrong" including their CEO. It simply requires bravery and faith in Google's institutional ability to continue to profit doing different things.
Big companies have pivoted before on the heels of brave executives. This applies as much to search as it does to privacy, AI, their transparency, their support of open source, and their weaponization of their browser.
It's a crying shame to see how enshittified a company that could be changing the world has become.
So it’s the shareholders and the board who are at fault then. I’d posit that anyone holding Google stock is part of the problem and responsible for their continued pattern of anti-user and anti-consumer behavior.
Has nobody heard of paying dividends? It is a perfectly acceptable way to have a company with stable profits. But no, anti-capitalism cliches combined with warmed-over rapture logic and Marx's idiotic teleological view of history.
I have no specific information here, never seen/used Shepherd until I saw a list on there a few days ago. Disclaimer, I work for Google, but not on anything related to this space, and this is based on my previous job where we did some SEO for an ecommerce site.
The example list given just looks a lot like spam when you squint. It's a list of affiliate links to buy products, and there are many HN threads talking about the abundance of affiliate link aggregators being a blight on the web. The commentary does look useful, but distinguishing between good commentary and bad commentary is hard, whereas distinguishing between a site designed to extract affiliate commission vs one more about the content is easier.
The comparison given to the other results here is frustrating, I know, but probably not a valid experiment. All the major search engines change results based on the user using them, or the IP address, or the region, or whatever, so it's impossible to know what others see. The developer of a book-focused shopping site is likely to get very skewed results for a book related query. My results were noticeably better.
The author says that a Bookshop.org list they created that links back to Shepherd is ranking #2, and this kinda makes sense to me. Bookshop.org sells the books, it makes sense that would rank above a site that only links to (and makes money from) sites that sell books.
SEO, and people getting annoyed at not ranking, has been a thing for 25+ years, I don't think this instance is any different.
> The example list given just looks a lot like spam when you squint.
I had to do some deep breathing before responding (creator of Shepherd here) :)
When I squint while reading a Pulitzer Prize-winning book, it looks like every other book with words everywhere.
When my wife really squints, she says I "look" like Brad Pitt.
I say these things with humor... as the core of what Google is supposed to do is determine what is good content and what is spam.
Is that hard?
Yes, but you have a lot of smart people and are swimming in money.
The point here, and with all the other posts by indie publishers, is that Google has destroyed them while elevating much worse results (they used to rank just fine along with other accurate results). I am just one example of many, and sadly better than most because books are a passion and not that commercial. For indie websites scientifically reviewing air purification systems, they are destroyed, and now you just get Forbes spam where nobody reviewed it.
My suggestion for Google is to stop squinting :)
> with all the other posts by indie publishers, is that Google has destroyed them while elevating much worse results
While I agree with your overall premise, some of this seems very subjective, re: "much worse results". For example, in your post when you list the pages outranking your site for "best books on Battle of Midway", you say this about being outranked by a forum thread on BoardGameGeek:
"A 2010 post on a Board Game forum asking for good books about the Battle of Midway. Really Google?"
BoardGameGeek is pretty much the most important site for the entire board game / tabletop gaming hobby. It's been around for 25 years, and it is most definitely part of the independent web. The thread in question is on BGG's main wargaming forum; wargaming has been a thing for many decades and has significant crossover among officers of the military / ex-military. The thread includes a lot of organic book recommendations from real people, with no affiliate links.
Despite the thread being from 2010, I would expect this page absolutely should rank pretty well for this search query! So why the "Really Google?" snark, especially when describing a fellow independent web site?
Yeah, it's hard to take OP seriously when it just comes across as sour grapes. Especially when their substack just seems to be a means to serve affiliate links.
> SEO, and people getting annoyed at not ranking, has been a thing for 25+ years, I don't think this instance is any different.
"People have always complained about X" is always a bad argument: it's entirely possible that X has gotten much worse, and yes, there were complaints before and after.
In this case, there is a very loud chorus of people saying Google has gotten much worse in the last few years, and it certainly matches my experience.
> The author says that a Bookshop.org list they created that links back to Shepherd is ranking #2, and this kinda makes sense to me. Bookshop.org sells the books, it makes sense that would rank above a site that only links to (and makes money from) sites that sell books.
In other words google is giving preference to a site that probably pays them for ads to a site that is effectively a competitor (since the site would make revenue from affiliate links that google might have gotten from ads, especially if the user liked the site and went directly there for future book recommendations).
Also, if I am searching for the "best" of something, I want something with commentary, not a list from a vendor without relevant information for choosing the right product.
Do you use Google anymore? It has gotten so bad, I regularly can't find things on it anymore, no matter how I search for things it will keep showing the same results. You seem to have blinders on. How do you explain the site getting less and less traffic over time?
I have not used Google for a couple of years (with the exception of Streetview where they do not have any competition at all). So I cannot comment on the quality of their search results. They are a close to monopolist / duopolist / oligopolist in too many areas. Ethically such a bad company that I don't want to touch them with a stick. Unfortunately masses don't understand that and contine to use them. And engineers are greedy enough to work for the evil.
iOS has a streetview alternative.
I personally hardly ever use any version of streetview because I don’t feel the need to so forgive me if I am missing what’s so great about Google’s version.
I do, and I don't find it too bad. I switched from DDG back to Google recently because I realised I had to use !g on most of my searches.
One day I tried a Google search and noticed that everything "above the fold" on the first page was either a Google property, an advertisement, or both.
I stopped using Google for web search, but I still use it for Google search.
Part of me wonders if the ad market is just radically different on Google in the US or other parts of the world, or if I just make very non-commercial searches. I don't see many ads, unless I'm actively going to Google Shopping essentially in search of them.
I do see the Google info box style results, but I find these to be one of the most useful parts and it's one of the reasons I like using Google for things that are basic facts, media fact finding (like "who was that person in that show" style queries), etc.
In my experience (American), the top 1-4 results are ads/sponsored links. Sometimes, very relevant to the search. Most of the time, not what I'm looking for.
I only use google search on my phone because I have a pixel, and the google searchbar is so well integrated into the launcher. If I don't like the results, I can just open a new tab and the default search is DuckDuckGo. Besides the ads, I'd say the searches are usually the same quality.
I switched to Kagi about 3 months ago, and to ChatGPT for a lot of info searches.
If they wanted to fix it they would. Most likely someone came up with the great idea of "Hey if search sucks, some losers will pay for AI hoping that fixes it, we are so smart"
I don't now if the bad results are because no one has written about the topic I want to learn or google search results are just terrible..
Also, to add to this...
One idea I've had is that it would be interesting if Google created a webmaster partner program to help build the web they want to see. Webmasters could join and embed Google's code into their website, and Google could get access to engagement stats and anything else to help them determine positive reactions by users (instead of pretending they are not pulling it from Chrome - see DOJ trial).
At YouTube, Google does a great job of helping creators, telling them specifically what they want, and promoting high-quality content that people love.
I'd love to see Google get the data they need so they can do that.
Is this not literally what Google Analytics is? I'm not sure if they're using it for everything you're hoping to have, but it seems like it basically performs that function.
They they are not allowed to use the data... which seems weird given Chrome and DOJ details. But I've never seen any details of them using it, and every statement is they don't use it.
(That's not what dearth means.)
Whoops, thanks.
portmanteau of "dithering" and "blight" and "earth"
Honest question: Would you say Google Search is getting better and better?
your comment made me remember about this post:
- Google made me ruin a perfectly good website (2023) / https://theluddite.org/#!post/google-ads
- https://news.ycombinator.com/item?id=40184673 (HN comments)
I'm specifically thinking about this line:
> It's a list of affiliate links to buy products, and there are many HN threads talking about the abundance of affiliate link aggregators being a blight on the web.
This is a pattern that was made hugely popular specifically by google, by the way
It looks like shepherd.com is a book review site that doesn't have any reviews, just ratings. It links to Amazon.
Is this really the sort of content Google should be returning?
Edit: it seems I missed the link to the actual book reviews because the link text is uninformative: "Chosen by 1 person - see why." (Sometimes it goes to reviews, sometimes not.)
And the word "review" never appears on the pages that have reviews. Seems like bad SEO?
If you're looking for book reviews, here's a website with some pretty great reviews: https://www.thepsmiths.com/ (Content warning: the authors are conservative.)
Just went over there and yes they have reviews from readers
https://shepherd.com/book/exit-strategy
Thanks, I missed that.
Hi Brian, we don't have reviews at this stage; we have purely positive recommendations. And we push readers toward a very specific format for why the book resonated with them (Gestalt psychology so you see it through their eyes).
Given the nightmare that is Goodread's unmoderated review sludge, I wanted to focus on the positivity and why someone loved a book, so we are uniquely focused on highlighting only books that readers love.
In 2025 I will be adding "reviews" of any books using our Book DNA review format (which tries to narrow in on why someone liked/disliked a book so we can match you with people who share your Book DNA). A big part of my mission is to focus only on books that people love and positivity.
Was that content warning really necessary, or do we also tend to use them for liberal websites (for whatever definition of liberal) ?
One thing I find especially bizarre about the current situation: the pages that are making it to the top aren’t even pretending to be real content. If they were difficult-to-detect LLM-generated pages, that would be one thing. But they’re generally extremely low-effort affiliate spam, mostly claiming that they researched something “so I don’t have to”, followed by a bunch of Amazon links and explicitly acknowledged scraped reviews, and finally an obviously uninformative summary. They don’t even pretend to have real content!
What is Google doing?
Ya, that is what is frustrating here.
A lot of us believe that Google trained their AI on spam websites, but in the process, the AI now identifies a lot of things that independent websites have on them versus big brands like Forbes, which spam and license out huge sections of their websites to spammers. So we got pulled into this and Google doesn't care about the false positives.
At this point, a lot of websites are not coming back. So get used to Forbes and other large brands being the only results along with AI bots on Quora.
8-10 years google search was amazing. A well crafted query would hit informational gold most of the time. I've been noticing and commenting on the decline privately for most of the period, but it's only in the oast year this seems to have come to broader awareness. There's an argument that in an information economy, searching for information should be treated like a public utility necessary for the functioning of society. I'm not making that argument but when you experience the long slippery slope of degradation of a service that was near ideal for the technology of the time, it does xome to mind.
That's the thing. It DID work. Really well for a while. But it was always atomic and context-less. We now have the opportunity to make it even better by refining results through dialogue. I hope someone does.. soon.
The slippery slope began in 2012, when they pledged to start downgrading piracy websites. After the rollout, the followup questions were "well what if we downgrade other topics that [random government] doesnt like" and "what if we sell the ability to boost enterprise company results for certain topics". eventually the number of results that get filtered out or reordered, exceed the results that actually get displayed. creating a new search platform is not a viable solution - any private company will be incentivized in this direction until some kind of "search neutrality" law is introduced.
I'm not convinced this is the explanation. It's true that for some product-centric queries, you get mostly paid results. But for information-seeking queries, Google tries to give you organic results.
The problem is simply that there's too much money to be made by capturing these. For years, you had content farms and fake "review" sites stepping up their game. Now, LLMs essentially make it a losing proposition to try and surface the small web. The least-bad option for Google would be to send you to moderated communities, such as Reddit, Quora, Stack Exchange, Wikipedia, and so on. But not all queries can be handled that way.
If you look at the article that these guys are complaining about... how do you distinguish it from content-farmed spam? You can't.
Now, I think Google is throwing in the towel and just want an LLM to answer info queries instead. That has a ton of problems, but to the average user, probably feels more helpful. At least until the spammers start gaming that.
Sometimes I lose track of a bookmark and google a phrase from it verbatim and in quotes. Guess what I get most of the times? Not even a verbatim search spam, just nonsense with almost no original words. It doesn’t even try and I doubt it even has a database where a direct path exists from a phrase to it with a link. Google is not “I found this for you” anymore, it is “I think you meant this”.
I thought this was just me! Even if I use double quotes to search for a 1:1 phrase I often get bad results and nothing linking back to the source I was looking for. If you happen to remember the site something was on, adding "site:example.com" still produces decent results fortunately.
You have to go into search options and select verbatim. But since they stopped indexing many sites it might not even be there anyways.
>If you look at the article that these guys are complaining about... how do you distinguish it from content-farmed spam? You can't
You can, Google just hasn't bothered to even try. As an example it's pretty easy to detect affiliate marketing links.
Google: We've done nothing and we're all out of ideas, man!
Not really? The vast majority of helpful websites monetizes the same way the spammers do.
Plus, even without affiliate links, purchase attribution is already scary good. Did you know that credit card issuers and ad tech companies collaborate to attribute brick-and-mortar purchases to online ad views? You have untold billions of dollars at stake. The industry is not banging rocks together.
You could make a search engine for non-commercial content only, and I would actually love to have that, but (a) it would be a woefully tiny sliver of the internet; and (b) Google is obviously not the right company to do it.
At its scale google could just hire a big group of diverse internet-aware professionals (two of each kind) who would do random searches and simply manually ban sites that restyle or spam content. It’s absolutely easy for a searcher to tell if it’s spam.
This is actually an interesting rabbit hole. Their other constraint is that they are trying to be transparent and fair, for some definition of that word. They need that for Section 230 protections, they need it to fend off antitrust lawsuits, etc.
So, there is a huge problem with hiring experts and telling them to make subjective decisions. Instead, Google publishes guidelines for webmasters and then enforces them rigidly. This usually ends up penalizing some good sites, while spammers swiftly discover workarounds.
then make it a gov position, like the "Beer and Malt Beverage Labeling and Formulation Approval" position, which consists of a single guy approving or rejecting the graphic designs of beer bottles/cans. there was an NPR segment a while back about the one guy that held that position for 40 years.
We are approaching a digital Kessler syndrome, or perhaps a Deepwater Horizon info-oilspill event, where there is so much useless SEO-driven slop (soon to be taken webscale with the advent of genAI) to sift through on the internet that it's growing increasingly difficult to find the signal amidst the noise. Google, once a prestigious company which prided themselves on "organizing the world's information" and "not being evil," eventually became a target for those wanting to peddle their wares and make a quick buck - a departure from the days of the early Internet which was mostly computer geeks, hobbyists, and forward thinkers sharing organic content they thought was interesting or useful. Because search was Google's entire business, they needed to develop countermeasures to combat spammers and pages gaming Google's algorithm with questionable SEO techniques in order to preserve the signal-to-noise ratio of the search engine results page.
This is now a bygone era - after discarding their original motto of "don't be evil," search and "organizing the world's information" are no longer Google's business, it's hawking advertisements [0]:
Hence questionable grey UX patterns like blurring the distinction between ads and organic content, and sometimes cramming the page so full of ads that all the actual results are "below the fold." Remember the old Internet adage - if you're not paying for the product, you are the product - and like cattle we are all just herded into digital pens to be served marketing slop to serve the real customer - the advertisers.If you want to be treated as a customer instead of cattle, you ought then to pay for your services, including search, to align the financial incentives with your own. Advertising needs to die, for it is a root cause of most of the ills of the modern internet [1]. If you can pay for streaming services or music, you can certainly pay for access to high quality organic information that actually aligns with your interests - not that of the advertisers. I've been using Kagi for a few years now and it really does hearken back to Google SERP quality maybe not at its peak, but rounding near to it.
At the risk of sounding elitist (and so what), this is just another consequence of the recurring Eternal September phenomenon - highly focused communities with a strong concentration of geeks, hobbyists, and experts were the norm back then, when computers were still new and arcane devices that were difficult to operate. The bar to entry was much higher, and one had to do a little bit of "reading the fucking manual" simply to get online and understand how to navigate the net effectively. Now that all the balls have been poured into the Galton board we have regressed to the mediocrity of content that exists on the contemporary Web, absent those pressures that once selected for high quality content online.
[0] https://www.wheresyoured.at/the-men-who-killed-google/
[1] https://news.ycombinator.com/item?id=41940718
>If you want to be treated as a customer instead of cattle, you ought then to pay for your services, including search, to align the financial incentives with your own. Advertising needs to die, for it is a root cause of most of the ills of the modern internet. If you can pay for streaming services or music,
People pay for their Windows license, yet Windows now has ads baked into the start menu. People pay for Youtube Premium, but most videos now have "sponsor segments" -- yet more ads (though admittedly not controlled by or directly profiting YT). People pay for streaming services, but last I heard, Netflix was adding ads. Ages ago, people paid for cable television, and it wasn't long before it had ads too.
These companies are going to treat you like cattle whether you're paying them directly or not.
Then vote with your feet and your wallet by leaving. You need to send monetary signals that the service is undesirable by withdrawing your subscription. I don't understand why people continue to use tools that don't serve them and assume, by default, that because everyone else is on some platform, that I must be there too, or don't go into an actual cost-benefit analysis of whether or not the utility of the platform outweighs the drawbacks. Cal Newport comments on this default herd mentality in Deep Work:
In the case of Google the diminishing utility of search began to be outweighed by the increasing morass of SEO slop and advertisements, so I degoogled. After W7 when it was clear Windows was going to progress into a cloud-based ad delivery platform, I installed Linux on all my devices as a daily driver. If Netflix ever serves me ads, I am immediately terminating my subscription.Maybe it's too hard for some people.
>Then vote with your feet and your wallet by leaving.
Yep, that's what I keep saying too, but I always either get ignored, downvoted to oblivion, or responses about how X is "necessary" because of Y (which isn't really very critical anyway, but I guess to them is, like "I must be able to play game Z!!!"). You can lead a horse...
bingo, this is why i mentioned (in the skip level comment) that all private companies will go through this sequence until a "search neutrality" law is introduced.
after MBA's start to get diminishing returns on new subscriptions per month, the focus shifts to advertisements.
> The least-bad option for Google would be to send you to moderated communities, such as Reddit, Quora, Stack Exchange, Wikipedia, and so on. But not all queries can be handled that way.
There's a wide gulf between "path of least resistance" and "least-bad". I don't think it's productive to conflate the first path with the second.
> "But for information-seeking queries, Google tries to give you organic results."
No, often it absolutely doesn't, and I posted two epic fails here previously. Worse still, Google organic results no longer understand(/distinguish) the difference between information-seeking queries vs product searches (or else, SEO people have been gaming it for years, and Google search rankings have made this worse):
____________________________________________________________
1) Google search relevance fail: result for “Africa longitude” https://news.ycombinator.com/item?id=30337563
Googling for "Africa longitude" should return a range of longitudes, like: "17.5°W - 51.5°E" or "["17°31′13″W - 51°27′52″E"]" [1], or for the A+ answer: *"Africa lies between 17°33'22" W, (Cape Verde, westernmost point) and 51°27'52" E, (Ras Hafun, Somalia, easternmost projection).
But it doesn't. Moreover these coordinates haven't substantially changed in 10,000 years (other than political/territorial disputes about islands, but the coords for the mainland certainyl haven't).
Googling returns the grossly misleading "Africa/Coordinates 8.7832° S, 34.5085° E". When you dig into why this so, it seems to be "optimized" for the SEO activities of a digital map storefront, MapsofWorld.com, acquired by MapSherpa Inc., based in Ottawa. And those mystery nonsense coordinates ("8.7832° S, 34.5085° E") bizarrely point not even to the geographical centre of Africa but to a random rural location 2400km ESE away, in southern Tanzania, which appears to have been deceptively mislabeled, in Cyrillic, as a Russian store (by Russian SEO?). For a pin dropped in rural Tanzania. No QA!
https://news.ycombinator.com/item?id=30337563
____________________________________________________________
Similarly:
2) Google search for "How many landings on dark side of moon" is grossly incorrect https://news.ycombinator.com/item?id=40862146
(Update: the AI overview factbox has at least since been corrected to give Two: "Chang'e 4 (January 3, 2019) and Chang'e 6 (2024), instead of repeating what jagranjosh.com says).
However the #1 organic search hit is still the woefully inaccurate unauthoritative page: https://www.jagranjosh.com/general-knowledge/list-of-all-suc... which cheerfully claims "Aug 23, 2023 — There are over 21[!!] moon missions that have been launched successfully on the dark side of the moon by 4 countries."
This is hopelessly wrong, even if we utterly misunderstood the key word "landing" and also count any mission which merely photographed the dark side (Luna 3, 1959) or human overflight over it (Apollo 8, 1968). But not "landing".
And why on earth did Google decide jagranjosh.com was more authoritative than any reference website or wiki?
____________________________________________________________
[1] https://en.wikipedia.org/wiki/Africa#Geology,_geography,_eco...
Well to be fair there is no "dark side" to the Moon, so it's not really an answerable query.
And searching google.com for "longitudes continent africa" returns "Africa's geographical coordinates span from Latitude 37˚21'N to 34˚51'15"S and Longitude 51˚27'52"E to 17˚33'22"W"
It seems that GIGO still holds true.
> what if we sell the ability to boost enterprise company results for certain topics
Maybe I'm naive but... has Google even considered doing that? Re the first one, I know top management had a project to reenter China back in the teens.
You are really naive if you think that first one is about China.
That was a concrete example of eagerness to submit to an especially censorious government. I did not say it was the only case. Save your insults.
Exactly, the organic nature of search ceased to exist that very moment when they started human intervention in a purely machine based (crawler/indexing logic) algorithm.
> any private company will be incentivized in this direction
From the earliest papers on Google PageRank, Brin and Page warned, "we expect that advertising funded search engines will be inherently biased towards the advertisers and away from the needs of the consumers." <http://infolab.stanford.edu/~backrub/google.html>
once the new subs per month metric starts to decline, even subscription funded engines will move in that direction.
> We now have the opportunity to make it even better by refining results through dialogue. I hope someone does.. soon.
TBH, I don't want dialog (too long and slow, leave dialogs for human to human comms), I just need my query words to exist in the result pages. Nothing more. Google messed this up long time ago.
> I just need my query words to exist in the result pages. Nothing more. Google messed this up long time ago.
Google didn't mess that up, SEO did.
You don't "just need" that. You need the page to be authored by a human with good intentions and end up containing those words because those are meaningfully part of what the human was trying to convey to you without ulterior motives.
But those kinds of pages are dwarfed by the millions of pages of content farm nonsense jammed full of every possible keyword and contaning absolutely nothing of value.
A search engine like you describe would be like walking in Chernobyl without a radiation suit on. A pleasant stroll in 1985, but not today.
I wouldn't mind a dialog that helps me build an advanced query.
Have you tried advanced search?
https://www.google.com/advanced_search
I'm aware of advanced search. I'd like it to be more user friendly and widely used. I think a chat bot would help.
Search only works if there exists something valuable to search for. The Google dominated internet has not incentivized a library of quality content. Most quality information exists on long dead domains. The rest is pay walled or unindexed. Google is a library search service but for a gas station convenience store of information.
kagi is still far from it.
8-10 years ago the competition for getting stuff before your eyes wasn't as tight. It's not only on google, the quality stuff is objectively harder to find in the ocean of slop today.
I find it ironic that there’s a desperate plea for the indie web from a Substack blog.
Why?
Substack is a great platform and I love them. I also use Beehiiv for our mailing lists with authors/readers.
The “indie web” is something you create, control, and own like a personal site or WordPress blog.
There is nothing indie about using a third-party platform to host content.
Remember how indie Medium used to be until they paywalled it to death?
Substack is great. Right now.
If you weren't born yesterday, or remember that "enshittification" describes a process (great for users, then great for businesses, then great for the middleman), then you already know that it's just a matter of time before Substack becomes Shitstack.
To get more visibility?
That is a private blog, so I don't care if anyone sees it. I just wanted something I updated every 2 to 3 weeks with my frustrations, wins, and so on as I build Shepherd.com.
I was on Twitter's free email thing but they shut down and imported to Substack. I just didn't want to pay for anything as the only person who reads it is me and my mom. It has 100 readers on it, as its just for me really.
I think the irony works either way.
If OP was going for visibility by posting it HN, Reddit, etc. then does it matter if it was Substack or his own blog or even the Shepherd site?
Technically, yes. If the thesis is "Google search is inaccurate and overrun with SEO optimized slop and ads", them using a blog utilizing that same SEO optimization feels a bit ironic. Feeding the beast you want to change.
I don't quite understand, I still want Google traffic, they have a monopoly. Nothing has changed there.
It’s more than ironic, it’s self-undermining…
How so? It is where I keep a small blog for myself about building Shepherd.com. It has barely any readers and keeps me sane and having fun.
Because every additional article posted on the non ‘indie web’ is literally reducing the relative credibility/importance/etc. of the ‘indie web’ by a tiny bit… which is unnoticeable for just one, but when you add up millions…
I don't understand, Substack is the indie web as well. It is a fantastic platform for the indie web to use.
Can you explain in more detail what you are saying?
The shepherd.com link is the second one that shows up on Kagi, after a Quora link: https://kagi.com/search?q=best+books+on+Battle+of+Midway
Same search for those without Kagi account (using 'share search' option):
https://kagi.com/search?q=best+books+on+Battle+of+Midway&r=u...
I would assume every other kagi user has blacklisted quora already
Ohhhh... I didn't know you could blacklist, thanks!
Ya I have, Quora is so bad now, I used to love them long ago.
Top result in AI overview and 4th in the general search on Brave Search
https://search.brave.com/search?q=best+books+on+the+battle+o...
Top result on duckduckgo: https://duckduckgo.com/?t=ffab&q=best+books+on+the+battle+of...
1) It has already been proven by research that Google quality of results slips from year to year [1]
2) Google has many incentives to make the search more difficult for you
3) Google has proven that it prefers money over quality of results with allowing "malvertising"
4) It is true that the landscape is more difficult. There are more walled gardens, to which even Google might not have access. There are more scams, casinos. More AI slop. The game was always hard on the other hand, so these are just 'excuses'
5) Why so often I see in Google results leading to major news sites instead of normal links?
6) If I write "Warhammer" I would expect thousands and thousands of pages in results. I think that Google prefers "content" over "quality". "newer" is "better". I would expect thousands of fan pages, which do exists, but are not crawled, or forgotten. Why can't I browse older pages? Why is there a limit to 10 pages?
7) For "Emulation" first page leads to "wikipedia", "cambridge dictionary", "vocabulary", it is so f boring
[1] https://www.404media.co/google-search-really-has-gotten-wors...
Simple answer --- stop using Google.
Get your friends and family and other internet users to do the same.
The only thing likely to get their attention is if enough people follow suit.
This is what I call a “load-bearing just”.
Just do the thing and the problem is gone.
Very similar to “Draw the rest of the owl.”
The problem is people don’t “just”. We’re complicated critters with our own needs, capabilities, and goals. At an individual level, a person may not be capable of the “just” due to factors outside of their control. With a society, the resistance to change grows exponentially. You can’t scale one person at a time.
This doesn’t mean you shouldn’t try. Sometimes you can make a big difference to the people around you. But you’re gonna have a difficult time changing in the world.
In this case it really is easy to try a different search engine and switch to it to the extent you prefer it. At least if you're not too uncomfortable using software.
Actually, that’s exactly my point. The “thing” is always trivial to the person making the statement.
The “just” refers to ignoring that other factors are involved.
If your personal goal is to convert X people to do something, you are accounting for it.
If your goal is a specific call to action to have people voluntarily join you, you are accounting for it.
If you’re stating a lot of people need to change, you’re acknowledging the scope of the problem.
If you’re doing these things, you can make a difference. Go do the thing! Have others join you!
If you say everyone should just do it and walk away thinking you have the solution to the problem, you don’t. :)
Did you miss the “friends and family” part?
Are you saying friends and family using Google are an obstacle to you using something else? Or maybe if you use something else, that makes you so uncool that your friends and family are not more likely to follow you, but actually less? I guess what you're actually getting at is that my switching doesn't automatically make my friends and family switch, but why insist that change only happen in blocks of people all at once? How does that help?
The original comment was pointing out that in order to break the monopoly of Google, the wide society will need to switch to something else, not just a few tech-interested people.
The thing is, that kind of logic claiming the futility of individual action does not weaken the grandparent's advice.
If you're an advertiser, you can't just switch from Google and hope to change anything. You'd be hurting yourself to no benefit, probably. Collective action is your main hope.
It's not like this for users of search. Google can lose its search leadership one user at a time. When the user switches, "the problem is gone" for that user, and Google has that much less revenue. Gradient descent is powerful.
Settings | Search | Choose anything but Google
Quicker and easier than "Draw the rest of the owl" and not outside anyone's control.
Of course it’s easy for anybody on this forum to change their search engine.
That’s the “thing”. The thing is always trivial to the person making the statement. The thing is completely irrelevant to the point I’m trying to make.
The “just” is other people.
"load-bearing just;" I love this term.
> With a society, the resistance to change grows exponentially
Yes and no. There's certainly the potential for a more flexible and adapt-minded culture for the same reason there's not... Human behavior.
Yes, once norms are established they're slow to change but that's how the culture is nudged and "managed". It's not necessarily how humans arw hardwired.
Eons ago, slow to adapt would mean certain death. Currently, fast to adapt labels you a rebel, a freak, an odd-ball, etc.
> norms are established they're slow to change
That is all I meant when I said society.
And I'm saying that the current system is "optimized" by the powers that be to not change too fast. Humans have and can adept if they're allowed to do so. The current sociopolitical system doesn't allow this. The bottleneck to change isn't humans, it's the system.
An even better solution is to apply the rule of law and break them up. They never should have been allowed to get this big in the first place
https://www.thebignewsletter.com/p/the-rage-of-google
How would breaking up Google fix this?
"Simple answer, just get everyone to stay home for three weeks and the virus will die out"
"Simple answer, just stop electing corrupt politicans"
"Simple answer, just stop war"
k
Kagi is good enough. Haven’t used a Google product for 12+ months expect as required by my employer (or YouTube but with an ad blocker).
Not saying calling for a boycott is enough but it’s easy to avoid Google nowadays.
"... it’s easy to avoid Google nowadays."
If it says "Google", you really don't want it.
Some may be a little slow to come around to this realization but it's growing.
For example, more than half of internet users are now taking active steps to block "personalized advertising" which is Google's bread and butter.
Subscribing to Kagi is definitely easier (and less costly!) than handling the pandemic, ensuring world peace and utopian government.
Edit to add: and since we're talking about indie web: https://search.marginalia.nu/
Me subscribing to it is. Getting everyone I know to subscribe to it is at least a step more complex than getting people to move to Firefox, and I haven't quite worked out a strategy for that yet.
Search results being absolutely shit is probably more of a problem for people than their choice of browser.
When someone complains to you directly about Google search results being shit, maybe point them to Kagi and mention it has a free trial account (100 searches still?) that's not time limited.
That way they'll have a solution they can turn to the next time Google pisses them off significantly. :)
I don’t know about you, but I’ve never had someone complain to me about Google search results, other than people who already don’t use Google.
Interesting. I definitely have, though I do tend to hang out with IT-ish people a lot.
The above marginalia.nu just surfaced the following very nice article for me, which neither Google nor Kagi returned in their top 10 search results for the same terms ('thaad aegis hwasong'):
https://nautilus.org/napsnet/napsnet-special-reports/the-rol...
(duckduckgo.com returned it as the 5th returned result (not bad)),
Thanks!
I’ll be honest, I don’t want my searches necessarily always tied to my identity.
Are you really comparing typing a dozen characters into a browser's URL bar and hitting Enter with "stop war"?
Using a different search engine, is just about the easiest kind of switch you could do. It's easier than, say, eating a different type of snack than you usually eat, or switching out your usual mug/plate/utensil.
Using a different search engine is easy. Voting for a good politician is also easy. But getting other people to switch to a different search engine is hard, just like getting other people to vote.
You missed the point.
Have a great day.
+1
Ok another note, your karma is very close to 2381. Make sure you get a screenshot :)
Long ago I wrote a blog post[1] about intermittent hardware, and not letting it suck you in. This was posted in a Blogger/blogspot blog, which is owned by Google.
This morning Google couldn't find it, neither could Bing.
[1] https://mikewarot.blogspot.com/2007/10/mikes-law-of-intermit...
Kagi can.
https://kagi.com/search?q=%22Mike%27s+law+of+intermittent+ha...
(granted, I had to perform search with quotes on, but both Google and Bing still fail)
Hopefully demonstrates the difference between legacy and a modern search engine whose entire purpose of existence is to surface what you want to find, not what other people want you to see. (wave from Kagi founder here)
Visits from Google to one of my independent sites have been about the same in 2022, 2023, and 2024. Visits to another have roughly doubled from year to year. I don't use Google Search or surveillance ads but I don't have the issue that they report.
AyyEye's observation that their site is loaded with trackers and what look a lot like affiliate marketing links is one reason why Google Search might not like it any more.
I am glad you got lucky, I hope it holds up :)
Stats are fine, as sometimes people mistake a single data point for a truth. Might take a look at my post as there are links as this is a huge issue affecting many many many websites.
I don't have access to your "many friends and acquaintances who are going bankrupt and shutting down fantastic independently run websites." I absolutely don't have access to the internals of their businesses to decide whether I agree that their problem is Google Search being hostile to independent sites. But I do have access to the traffic stats for the sites which I run.
Many of the sites with similar complaints seem to focus on product reviews and recommendations and have been hit by a specific Google policy which favors trusted and well-known domains for those types of search, even if that domain is not trusted for product reviews in particular. (So $glossyMagazineInNYC can open a line of product reviews and get lots of Google traffic, while $smallProductReviewBusiness loses traffic).
Google recently changed who is running search. Basically, Prabhakar Raghavan was moved from running Search and ads to being Chief Technologist. Nick Fox is the new person running Google Search. This happened on 10/17/2024. For more information, please see https://searchengineland.com/google-shakes-up-leadership-rag... and https://www.theverge.com/2024/10/17/24272786/google-search-p... .
I wonder if Prabhakar Raghavan was moved out of Google Search because of Search's decline in quality?
I woke up this morning.
Then checked my mailchimp subscriptions
and my grow.me subscriptions
And my substack subscriptions
Made sure my Cloudinary was properly configured
And my newrelic analytics
And my sentry analytics
And my rlcdn analytics
And my growplow analytics
And my 33across analytics
And my Scorecard research analytics
And my openxcdn analytics
And my trustarc analytics
And my creativecdn analytics
And my Google Tag Manager
And my Google Analytics
Then I checked my Mediavine ads
And my adsrvr ads
And my adform ads
And my adnxs ads
And my yieldmo ads
And my criteo ads
And my mediavine ads
And my pubmatic ads
And my id5-sync ads
And my rubicon-project ads
And my triplelift ads
And my pghub ads
And my zemanta ads
And my cognitivlabs ads
And my doubleverify ads
And my media.net ads
And my kargo ads
And my Amazon ads
And my Google Adsense
When I finally got to WeWork
I booted up my Macbook
And checked my Google Workspace email
Our Amazon Affiliate account was approved
And I checked our private Github issues for tasks
Then I let Microsoft CoPilot write Stripe integration
I write a new post for my Substack
Finally using my Chrome browser and ycombinator's platform I posted:
'Google is killing the independent web'
What does any of what you cited have to do with the independent web?
Should he have clarified "properties not backed by tens, hundreds, thousands, or millions of millions of dollars?"
I decided to remove my browser condoms and check Shepherd's network traffic. Updated my post to better reflect how "independent" the website is.
That's less than half of the malware it tried to run on my computer but I got bored checking all the domains it tried to run code from. It's repetitive. I get it, you put on every tracker and ad network you could find, and put your stuff on a bunch of CDNs.
Everything that can be outsourced to big tech was outsourced to big tech. How very independent.
Great question :)
Independent means that we are a small team and trying to create something cool for people. It means we are not part of Forbes or some giant publication that is focused purely on money and spamming the shit out of you :). You want a diverse and indie web so that the only search results and voices you here are Forbes, Disney, and other giant media conglomerates.
Independent does not mean we don't eat.
Independent does not mean we don't have server bills.
Independent does not mean our part-time developer works for free.
Independent does not mean our designer works for free.
There is no malware on the website crazy; it's just an ad network that pays for our servers. I will remove all those ugly display ads as soon as we have enough income from readers and authors directly.
> There is no malware on the website crazy
Reasonable minds may differ. No need for name-calling.
> I will remove all those ugly display ads as soon as we have enough income from readers and authors directly.
Doubtful.
> It means we are not... focused purely on money and spamming the shit out of you
Sure looks like you are. In fact I can't even find a single datapoint to the contrary. My browser made over 900 requests to other entities who's raison d'être is spamming me for money.
Even your default affiliate link is Amazon instead of Bookshop for crying out loud. You didn't even put them side by side -- you hid bookshop behind a drop down menu. Seems like you don't care one bit about being independent.
> You want a diverse and indie web so that the only search results and voices you here are Forbes, Disney, and other giant media conglomerates.
I can't tell the difference between shepherd and the rest of the dead internet. There's only one person or a small team behind most blogspam and SEO-spam but it's still ruining the web.
Independent web does not mean "We're not rich (yet)". Also 'web' implies links to other indie websites. A (hidden, even. Lol.) affiliate link to bookshop doesn't count.
> Independent does not mean we don't eat.
Independent means... Not dependent. You are dependent on Medium for Shepherd's blog for crying out loud. You could host your own blog on your server in minutes for zero marginal cost, and yet there you are paying big tech to put some words under a header.
> Independent does not mean we don't have server bills.
Use less (as-a-service). Even a small and cheap dedicated server or two should be able to handle this just fine. Maybe not as-built but you're doing nothing that needs expensive hosting.
> Independent does not mean our part-time developer works for free.
You have nearly 1000 subscribers and the minimum tier is $25. That's far from "works for free".
> There is no malware on the website crazy.
You don't even know what runs on your own website. It attempted to run code from hundreds of places who's whole purpose is to dox your users and run targeted ads to take their money. (That's money that might have been going to shepherd fwiw.)
> just an ad network that pays for our servers
"just" dozens of them. "Just" dozens of tracking and analytics.
There are ways to do ethical advertising without tracking. You could even sell ads yourself (maybe you are, I haven't looked that closely at the website).
How many servers do you have that those are a significant cost center when you have multiple employees?
> I will remove all those ugly display ads as soon as we have enough income from readers and authors directly.
Going into this I thought you were making shepherd yourself, but reading between the lines, you already hired 1-2 (or more) people before you made money. That's fine if this is a business venture -- but speculatively hiring multiple people in the hopes of becoming profitable is far from the "indie web". Light years away. It's standard operating procedure for the hyperscaler corporate web though.
I'm not trying to be harsh here, I promise. My original post had a little bit of snark because of the medium blog. But every time I look at this it gets uglier.
If you truly care to be independent you can. Get rid of the garbage you've accumulated (ad networks, trackers, minimize APIs, bring everything in-house). Slim down the network requests and JavaScript. Rethink how you do ads. Lastly, focus on being part of the indie web instead of another cog in the corporate web machine.
Hah, I love HN :)
I've worked on this full-time for almost 4 years for zero pay. I love what I am building and hope you are doing ok, my friend :).
Keep doing what you love. I am not critiquing the functional bits of the website. Just the 'independent' bits. You can do a blog in-house easy. Maybe kill the analytics and try to figure out how to do some in-house ads? Also put bookshop ahead of Amazon.
If you have any questions I'm happy to help you break the chains that bind you! Come on in, the water's fine.
Finally time to leave.
I got in my friends Telsa
Waiting at the traffic lights started looking at Twitter
I got home and asked Siri
For food from UberEats
Only to get bored and open a tab of Reddit
Followed by television series from Netflix
With food from KFC
I then decided to go for a pee
Looked in the mirror
And questioned who is me.
Any idea how many Googlers still have the pre-dotcom-boom Internet technologist-citizen mindset?
I know a bunch flocked to Google early on. But the landscape has changed a lot since then, and many people weren't even born until after the Internet and society were very different.
All my friends at Google left so they could get things done...
I worry about that company.
This problem is age old and always the same. Optimizing for Google is a fools errand. You can win in the short term, but when their priorities change you will find your website excluded for the very things that boosted it previously.
Google are unreliable and untrustworthy. Their focus is ad revenue for themselves and nothing else. Build sites for humans and let Google do what it wants to.
Only somewhat related, but I was just complaining about search in Gmail. I normally use Thunderbird and didn't have it, so I used the web. Basic searches on the subject were so bad! I even put quotes around it and tried all of my Google-foo.
It felt like I was fighting some AI that was sure it knew what I wanted despite my "exact phrase" search.
These days I usually only search google for an exact phrase or title I got from chatgpt to check if it hallucinated.
... And google ignores your exact phrase and returns something its semantics and ads engine determine is close enough.
Google got so big it swallowed the internet. It now has digested it and what is left is... this.
I used to love crafting websites and cared about SEO. What's the point now, no one is going to find your content. It won't even be on the third page. Google will answer questions by regurgitating whatever it swallowed on your websites and presume no one will click through, it won't even bother marking the authors.
Instead it appears to be prioritising whichever website is going to give it revenue first, e.g. the click farms.
The regular folks don't care, they google for stuff like "am I dying if I have a pimple?" (to which the answer is always yes, apparently). No one does actual meaningful research using Google anymore, if you do, good luck, get your gloves out <picture of dinosaur poop in Jurassic Park>.
The global internet as it stands is close to dead. Discoverability of "cool" things is down to social media, tricked by "influencers", who are tricked by marketing themselves.
We need a hard reset button, it needs to start from the ground up with site rings, and good content. Ah... that last part, "good content", is now stuffed with AI Samey McSamey sounding text. I really don't see a way out of here.
The funny part is we used to think that the internet was going to change the world. We thought all idiots needed was information. Access to information would fix the world! Instead, it only has given the village idiots a global voice: if you can think of some dumb crazy thing, you'll find dumb crazy people agreeing with you, so you must be right!
I've been on the internet since 1997 and I think it's the worst it's ever been.
> The funny part is we used to think that the internet was going to change the world. We thought all idiots needed was information. Access to information would fix the world!
I was 9 in '97 and wasn't really aware of the internet until maybe 3 or 4 years later. The internet seemed like a magical place, where reason and common sense existed (unlike the messy meat world). I bought into the same ideals that information was power and once everyone had access to information, we'd collectively get smarter and wiser. Less wars!
I remember being excited for Google Search and then Gmail. FINALLY, a company that gets it!
It seems that any public or VC-backed company is destined for enshittification.
Opportunity for a new discovery engine. Not Kagi (I buy it) but a search focused on the small web, find stuff not in the top 100k sites.
https://search.marginalia.nu/ check out Marginalia
I love this search but I feel it's still very specialised.
Enjoy being absolutely crushed by Google and having to fight to get users to even know you exist, let alone use you
https://millionshort.com/
I wonder how feasible that is without leveraging one of the two existing search engines in the backend. I always pitted a general search engine as a top 5 difficult tech project to go for.
https://wiby.me/
Switched to Perplexity Pro and haven't looked back
Ctrl+F'd for Perplexity. I knew Google was cooked the minute Perplexity worked better for questions about an obscure embedded systems SDK. It has little documentation, but a lot of mailing list and github issues. Google spits out the front page of the project and shrugs; Perplexity actually answers the question. The usual caveats for LLM hallucination apply.
I'm already paying for Kagi and Claude. Is Perplexity Pro worth the extra subscription?
Same. Try the "books on the Battle of Midway" query on Perplexity. The results are great and include the book mentioned in the article (authored by the Naval Aviator).
Always use google in incogniti so they dont give personalised results, thats the least we can do before google search completely dies, Or just prevent google any cookie permission, keep it session only
As far as query trackers on google urls, they are necessary
One good case of necessity of google tracking parameters is --> you search for movie or related, but muktiple movies by same name have been made , you dont remember year, you do remember some actor or story description Searxg and google will in show with media carda foe movies you ckick iy and yoi get exact same film with year yoi wantes
Wait ->I am telling above because when that mefia image of film i inspecred itsurl it was just Googke ?search=filmname&teackers The search ket wasnt modified with year same as i searched first yet it gave me the desires year film
may be because of trackers &ved amd all paramers link tags to search teaukts its loke ctoss maching in databases
Google can't please all of the sites all of the time, or all the visitors.
It is too big to evem worry about that 4k a day clicks for one site. It is like us optimizing the expense of 0.01c. It makes a difference when that 0.01c is an API call that you call a million times. But it only surfaces if you do aggregate it.
Therefore this problem can only even be seem by Google if it can be surfaced in aggregate overy say a billion queries.
I wonder how that can be done.
Probably only can be done using data. Which means spying on people in various ways. And making assumptions about length of time on site equals quality.
They probably use machine learning too. There may be no reason for the lost rankings other than a wind change caused by some updated parameters in an OKR chasing model.
I realize "did you actually read the article" is against HN guidelines, but what else am I supposed to say when I see this at the top of the comment thread?
I mean when you say "Google can't please all of the sites all of the time, or all the visitors.", I wholeheartedly agree, but this blog post was excellently sourced with data that shows exactly how Google raising sites that any reasonable human would say are considerably shittier than this site that is getting down ranked. It also seems pretty clear that the things that have changed are Google's ranking algorithm at specific points.
> They probably use machine learning too. There may be no reason for the lost rankings other than a wind change caused by some updated parameters in an OKR chasing model.
That is literally what TFA says in the very first section: "Some people believe they have lost control of their AI ranking systems, ..."
I think I said a lot more than TFA on those points.
Why don't people just stop using Google? And by people, I mean everyone here.
Whenever this point comes up, I see people claim they ONLY see the results they want in Google. How would you know if you don't actually use anything else? Kagi is excellent search. Neeva was pretty great when it was active. DuckDuckGo is passable. Idk how Qwant gets money but it's been around a bit.
Complaining about the same thing forever and expecting a change doesn't make any sense. Y'all are in abusive relationships with Google and refuse to leave. Sure, your job may use Google Suite and you need to make money. What about the rest of your life? Stop hitting yourself.
Once I have stable income I do plan to use Kagi. Though I do recognize that even Kagi partially has Google Search underneath. Even if you fully de-googlefy it's hard to not be indirectly supporting Google.
Been working on it for 2 years now and even then I'm more or less locked into 3 services
1. Gmail, 20 year old account I use for pretty much everything business. Even if I move I'll need to forward Gmail stuff to a new email for years.
2. YouTube. Pretty self explanatory (and the go to for why monopolies are always bad). Trying to avoid it entirely is like trying to avoid dang Twitter. Too many other companies use it as a go to for any video, no matter how inefficient (nothing better for graphical showcases than nitrate compression ruining all the details)
3. Play store. Used android all my life and while I can mostly move out I will be missing some critical apps from that result (financial apps are a big example)
It's a network effect like any other for 2 of those, and a lock in part of my online identity for another.
I can't help you with points 1 and 3 but for 2: do you comment on YouTube videos or upload your own? If you just consume content (like I do), you can subscribe to your favorite channels with RSS. Go to a channel and inspect source for "rss" to find the link. I'm partial to NetNewsWire for my feed reader.
> Kagi is excellent search.
I will boost this. I am very happy with Kagi. I'm on ultimate and am glad to have my money taken for reliable and high-quality search.
> Complaining about the same thing forever and expecting a change doesn't make any sense. Y'all are in abusive relationships with Google and refuse to leave.
This would be a more solid stance if they were complaining about this as a consumer and not as a business operator. They can't control what search engines other people use. The best they can do is optimise for other engines like Bing and DuckDuckGo.
The other alternative is finding other avenues to advertise away from search engines, which may be what you were alluding to.
> The other alternative is finding other avenues to advertise away from search engines, which may be what you were alluding to.
I was actually speaking from the consumer perspective, thanks for making the distinction. The OP has valid reasons for being upset with the most popular search engine ruining visibility.
To be fair, I have no idea what business could/should do. Bigger businesses can afford to hire social media folks to keep them relevant on Facebook or whatever. SMBs/mom & pops are screwed.
I used DDG as my default search engine for several years, but I kept having to go back to Google because it was consistently more reliable even though it has more crap. I hear Kagi is great but I am not signing up just to find out, nor do I feel confident that 100 searches are enough to make a decision about paying for it. I think Google is garbage but I have yet to se anything that is consistently qualitatively better.
100 searches last you a lot longer than you think, especially if you delegate “postal code Wall Street New York”-type searches to Google via “!g” prefixing.
It’ll give you time to get used to Kagi and set up some personal up- and downranks and blocks. Especially together with the Summarizer and Lenses your search result quality will dramatically improve.
No doubt this is true for you, but my search usage patterns are quite different. 100 searches in a day is nbd for me, sometimes I go through that in a few hours.
> or do I feel confident that 100 searches are enough to make a decision about paying for it
I didn't think so either, coming from Neeva. Of course, YMMV.
> I have yet to se anything that is consistently qualitatively better.
And you won't, until you try.
Because the problem isn't just Google. The internet is a garden whose quality content was left to wither.
It stars with "m" and ends with "onopoly." You can't escape them.
I use Kagi, but I am a nerd.
> I use Kagi, but I am a nerd.
That's why I emphasized "everyone here." My wife isn't gonna stop using Google, and Kagi isn't a default option choice on iOS. Nerds will go out of their way to use a product.
> Complaining about the same thing forever and expecting a change doesn't make any sense.
Do you know that this forum isn't one person and thousands of people access it in different days and time, right? The notion that the same people are complaining while still using Google is a projection of your mind.
The author talks about how long users spend on his page, but can Google track that? Has ubiquitous tracking blocking, and Google's failure to adapt, eroded the quality of search?
There have been links from the DOJ case showing Google is pulling in Chrome data.
But its just one data point and nobody knows.
It seems more likely at this stage that Google has lost control of their AI system. they trained it on x websites, and indie websites got swept into that. And they don't care.
We don't even have ubiquitously used ad blocking right now. Do you think some normie is even going to know what a tracker is and why they should block it?
Ubiquitous tracking blocking? Where?
Checking my Search Console Tools recently, the number of "Crawled - currently not indexed" pages across all my sites has risen sharply recently, especially on sites where the age of the content is more than a few years.
I put a fair bit of time into trying to improve the sites with JSON-LD and breadcrumbs and what not. It seems to have helped just a little bit.
I don't make any money off any of it, but it's still kind of irritating that no one can find it.
The indie web was around before google and it will be around long after google is gone. I would argue that the indie web has incurred a much larger loss from people thinking seo/engagement metrics are something worth optimizing. Many of the best examples of the indie/small web don’t have js tracking and little to no css.
Google didn’t shit the bed, they just did whatever was profitable to them and they’ll continue doing so until they either fail or there’s no more profit to be squeezed. They dropped the “don’t be evil” motto a long tome ago.
Great work as always Danny Sullivan!
I've had my lifetime quota of hearing the rants of guys who think their search quality metric is objectively superior, oh and by a total coincidence their preferred ranking gives them a direct financial benefit. Just a complete failure to imagine the larger issues.
"It is difficult to get a man to understand something when his salary depends upon his not understanding it." - Upton Sinclair
but to google you're just a desperate flea from the indie web.
exa.ai good for finding these sites? but yes, many people will still be googling
thought this was going to be about the 'log in with google' popup
you should check out exa.ai/search if you miss indie content!
If you're still barking in the general direction of google and similarly sized search engines, good luck. I'd rather look towards kagi and brave.
Kagi is a viable choice for individual people, but it's not an answer for the concern of how to get traffic to visit your site and click on your affiliate links (which is what TFA is about). Kagi works well for users precisely because it's small enough that no one benefits from gaming its algorithms.
Why is it Google's job, who TFA isn't paying AFAIK, to get traffic to visit their site?
I don't think it is, but recommending Kagi as a solution for their complaint misses their point.
How is Kagi faring after Google's exclusivity deal with Reddit?
I think Kagi uses Google's APIs as a part of their mix of search results.
Um, I don't know when that would have kicked in because I haven't noticed a difference. Reddit still shows up whenever it's relevant, and the Forums lens turns up even more Reddit if you're worried you're missing any on the initial search.
I feel too that we're already past the point where they could fix things. They must be aware of issues and do the exact opposite of fixing them. It's a question how fast we can convince general public to move away, rather than when Google will improve.
I wouldn't give much regarding the general public. If you have a small blog they're usually not your target audience. Accept that the days of "making a few bucks off of advertising" on hobby publishing are over, focus on writing about things you care about without it reeking of being written by ChatGPT and chances are people will come at some point.
If small publishers don't change their behavior in radical ways, nothing will ever change.
> nothing will ever change.
That's probably what MySpace people thought. And Friendster. And Altavista. And ...
I don't get what you're trying to say. I'm not arguing in favor of doing nothing in an ever-changing market, which is what Myspace and Friendster did most of the time. I'm saying the old world of "have good content/SEO and you'll see some traction thanks to Google" is dying and that you can either adapt or drown in a flood of LLM-created SEO blackhat nonsense. Google does not care about you and they've been in the "too big to fail" corner for too long.
If you want content made by humans to continue to exist and want to see some of them be able to make a buck off of it, support smaller search engines and creative people and their publications.
I mean that services will come and go. Some will stay for longer than others, but even without a coordinated action, Google may go away. It's also losing to Perplexity and ChatGPT with many people. And many geeks are very vocal about their love for Kagi. (Including me) Things may change over time and may not be too big to fail.
And how is the experience? Even with Bing I find I have to switch to Google to get useful results for my queries.
Kagi is simply better in every way I can think of and I've been a paying user for over a year now.
It took me about two weeks until I got consistently good results with DDG, it often just takes about one more search term because it doesn't try to (often incorrectly) guess the context based on search history like Google does. That was years ago, since then DDG has gotten better and Google a lot worse.
I could understand your complaint if it was about image search, but I don't see how anyone can look at today's Google results and think they're still superior.
Google is so odd these days, its like there are two "streams" of results. One is the kind of useless slop that the article in question is rightfully complaining about, but other results are still useful and high quality. The second stream is what I seem to get when I'm forced to "!g" when I'm using DuckDuckGo, and I can't work out why theres this stratification, this separation. Years ago, it was very rare to get crap results from Google. Now its far more common.
Haven't needed to use Google at all since using Kagi several months ago, whereas when using Duck Duck Go for a few years (prior to Kagi) I'd (very occasionally) need to.
Kagi user here, I almost never need to !g. Used DDG a few years back and stopped after I had to fallback to Google for most qs.
is there a way to detect and block substack?
Maybe we should just start looking at other search engines seriously and without irony.
How’s shepherd.com (for example) doing on bing or duckduckgo?
Maybe it’s time for google to die.
Big adtech cares not as long as the ads keep flowing
As much as I share the frustration, what does Google gain from providing a better service?
Its only serious competitor is Bing, which isn't even a search engine anymore but a billboard for Microsoft to advertise ChatGPT.
Pride.
Happiness.
Contentment.
A workforce that is motivated and happy.
People don't want just money.
The real problem is that Google solves for the needs of the lowest common denominator of content consumers, which leads inevitably to enshittification
Hi stranger, please stop messing with the scrollbar
Hi (company that benefits from hurting me), please stop hurting me.
Company: No.
There are people inside Google who can put a stop to this, if they merely stand up and say "this is wrong" including their CEO. It simply requires bravery and faith in Google's institutional ability to continue to profit doing different things.
Big companies have pivoted before on the heels of brave executives. This applies as much to search as it does to privacy, AI, their transparency, their support of open source, and their weaponization of their browser.
It's a crying shame to see how enshittified a company that could be changing the world has become.
And a willingness to be fired the next day. Even the CEO can be fired.
So it’s the shareholders and the board who are at fault then. I’d posit that anyone holding Google stock is part of the problem and responsible for their continued pattern of anti-user and anti-consumer behavior.
Shareholders who don't want to maximize their share value are very strongly incentivized to fire themselves.
It's actually late stage capitalism that is at fault.
Line must always be going up, and if it doesn't go up as fast as someone thought it would then we make it go down.
If line goes sideways we sell and make it go down.
It must go up. Always up. Or it will go down.
You can’t blame the system and pretend none of that lands at the feet of people willingly engaging with it for their own benefit.
Has nobody heard of paying dividends? It is a perfectly acceptable way to have a company with stable profits. But no, anti-capitalism cliches combined with warmed-over rapture logic and Marx's idiotic teleological view of history.
I'm a fairly libertarian moderate but tyranny-by-proxy is still tyranny.
Yes. That's the bravery part.
Well they replaced the Yahoo alum accused of tanking search quality with a McKinsey guy....I'm sure things will get better now /s.
https://www.wheresyoured.at/requiem-for-raghavan/