> there's typically a 5-7 day gap between updating the robots.txt file and crawlers processing it
You could try moving your favicon to another dir, or root dir, for the time being, and update your HTML to match. That way it would be allowed according to the version that Google still has cached. Also, I think browsers look for a favicon at /favicon.ico regardless, so it might be worth making a copy there too.
/favicon.ico is the default and it will be loaded if your page does not specify a different path in the metadata but in my experience most clients respect the metadata and won't try to fetch the default path until after the <head> section of the page loads for HTML content.
But non-HTML content has no choice but to use the default so it's generally a good idea to make sure the default path resolves.
I think it is from the Internet Explorer days. .ico is an actual icon file format on Windows and IIRC originally the icons were in that format, with support for GIF coming when Netscape supported the feature.
1. What does it look like for a page to be indexed when googlebot is not allowed to crawl it? What is shown in search results (since googlebot has not seen its content)?
2. The linked page says to avoid Disallow in robots.txt and to rely on the noindex tag. But how can I prevent googlebot from crawling all user profiles to avoid database hits, bandwidth, etc. without an entry in robots.txt? With noindex, googlebot must visit each user profile page to see that it is not supposed to be indexed.
"Important: For the noindex rule to be effective, the page or resource must not be blocked by a robots.txt file, and it has to be otherwise accessible to the crawler. If the page is blocked by a robots.txt file or the crawler can't access the page, the crawler will never see the noindex rule, and the page can still appear in search results, for example if other pages link to it."
It's counterintuitive but if you want a page to never appear on Google search, you need to flag it as noindex, and not block it via robots.txt.
> 1. What does it look like for a page to be indexed when googlebot is not allowed to crawl it? What is shown in search results (since googlebot has not seen its content)?
It'll usually list the URL with a description like "No information is available for this page". This can happen for example if the page has a lot of backlinks, it's blocked via robots.txt, and it's missing the noindex flag.
1. Why is your favicon in the uploads directory? Usually, those would be at the root of your site or in an image directory?
2. Why is there an uploads directory for a static site hosted on GitHub? I don't believe that is useful on GitHub, is it? You cannot have visitors upload files to it, right?
1. I want nginx to serve static files, and everything else to be reverse proxied to the webapp
2. The configuration file that allows /favicon.ico (and others) to be a file but / and other paths to be passed to the webapp is kind of ugly. Here's mine:
In my own case I've so far decided to accept the ugly config file, but as you can see, I haven't gotten around to adding even a robots.txt or any of the other files the modern web ecosystem expects; and adding them involves adding them one-by-one. I can see why someone would say, "Why make an ugly hack of an nginx config, when I can just define the favicon location in the metadata to a path easily configured to be files-only?"
Thinking about SEO too hard sucks, but having a site that nobody can find (even when they already know about it and are specifically trying to find it again!) sucks even more.
You dont want to exclude your images. That can very much affect your results as it will remove you from image tab, but also content of articles that contain them might be affected.
I thought that Google Search Console had tools to test robots.txt and sitemap.xml files, but it has been a while since I have needed to do that.
For those wondering why favicon is in a directory, nowadays there are half a dozen different favicon files for different devices in different situations and there are online tools such as The Real Favicon Generator that will take a source image and make the variants for you. These come with a code snippet for head and the option to use a sub directory so that you don't clutter the root.
Maybe they should offer a robots.txt snippet too.
Fun fact, for a single page, you can base64 encode the favicon and shove it in the page, thereby not needing a separate file. Why would you want to do that? If you base64 encode all the images and add the scripts and stylesheets in, then you can have a HTML page that you don't have to upload, you can email it to someone. This is useful if wanting to share a design mockup.
I'm a complete ignorant when it comes to SEOs so what are the consequences of not having a robots.txt nor a sitemap.xml at all? Will that be detrimental in a big way?
My understanding is that a lack of robots.txt should be fine and the lack of a sitemap.xml shouldn't be too troublesome as long as there is something linking to your site and all pages are linked from somewhere on your site (the sitemap helps search engines find all the links in one place but you have a nav bar or an article list that should work similarly, but you can give a suggestion to search engines about how often they should recrawl in your sitemap, which I don't believe you can influence any other way).
There are many metrics and facets of a website considered for SEO, with varying weights. The absence of either a robots.txt or sitemap.xml has a non-negligible but relatively minor weight compared to some other metrics.
The files should be present and accurate when optimizing for seo. Ruby on Rails for example creates an empty robots.txt file with new projects, it does not create any sitemap.xml however.
I’m also ignorant of this, and to add a question on top of yours: is it worth worrying about robots.txt for personal portfolio websites built from scratch?
That's a valid robots.txt, but "proper" is entirely dependent on what you want to achieve. If you aren't looking to treat different bots differently, and are looking allow all of your site to be indexed, then that is exactly what you want.
If you don’t have millions of pages the crawl budget limitations most likely will have zero impact.
Make sure your basic technical SEO factors are all good. Search console is looking good and then don’t continue to worry unless you are a huge site that’s living off SEO traffic.
SEO can feel like such a balancing act—one tweak, and it’s a waiting game to see the impact! Sounds like you’ve learned a lot about the sensitivity of robots.txt.
If you’re into exploring new tech, you might like Rig. It’s a Rust library for building scalable, modular apps with LLMs, ideal if you’re branching out into AI or complex workflows. Keeps things type-safe and flexible.
> there's typically a 5-7 day gap between updating the robots.txt file and crawlers processing it
You could try moving your favicon to another dir, or root dir, for the time being, and update your HTML to match. That way it would be allowed according to the version that Google still has cached. Also, I think browsers look for a favicon at /favicon.ico regardless, so it might be worth making a copy there too.
/favicon.ico is the default and it will be loaded if your page does not specify a different path in the metadata but in my experience most clients respect the metadata and won't try to fetch the default path until after the <head> section of the page loads for HTML content.
But non-HTML content has no choice but to use the default so it's generally a good idea to make sure the default path resolves.
> won't try to fetch the default path until after the <head> section of the page loads for HTML content
That's a really interesting optimization. How did you discover this? Reading source?
Thanks for sharing, I wasn’t knowing that browsers look for a favicon at /favicon.ico. Thanks again.
I think it is from the Internet Explorer days. .ico is an actual icon file format on Windows and IIRC originally the icons were in that format, with support for GIF coming when Netscape supported the feature.
Many browsers will accept a favicon.ico that's actually a PNG file with no issues.
USE X-Robots-Tag: noindex to prevent files being indexed and let google determine how they crawl your site for themselves.
A nightmare scenario can result, otherwise, where you have content indexed but don't allow googlebot to crawl it. This does not end well.
https://developers.google.com/search/docs/crawling-indexing/...
I’ve got two questions:
1. What does it look like for a page to be indexed when googlebot is not allowed to crawl it? What is shown in search results (since googlebot has not seen its content)?
2. The linked page says to avoid Disallow in robots.txt and to rely on the noindex tag. But how can I prevent googlebot from crawling all user profiles to avoid database hits, bandwidth, etc. without an entry in robots.txt? With noindex, googlebot must visit each user profile page to see that it is not supposed to be indexed.
https://developers.google.com/search/docs/crawling-indexing/...
It's counterintuitive but if you want a page to never appear on Google search, you need to flag it as noindex, and not block it via robots.txt.> 1. What does it look like for a page to be indexed when googlebot is not allowed to crawl it? What is shown in search results (since googlebot has not seen its content)?
It'll usually list the URL with a description like "No information is available for this page". This can happen for example if the page has a lot of backlinks, it's blocked via robots.txt, and it's missing the noindex flag.
'But how can I prevent googlebot from crawling all user profiles to avoid database hits..'
If user profiles are noindexed then why should you care if google are crawling, when almost every other crawler out there does not obey robots.txt?
It's not in google's interest to waste resources on non-indexable content, you are worrying far too much about it.
It's good information but...
1. Why is your favicon in the uploads directory? Usually, those would be at the root of your site or in an image directory?
2. Why is there an uploads directory for a static site hosted on GitHub? I don't believe that is useful on GitHub, is it? You cannot have visitors upload files to it, right?
Speaking for myself:
1. I want nginx to serve static files, and everything else to be reverse proxied to the webapp
2. The configuration file that allows /favicon.ico (and others) to be a file but / and other paths to be passed to the webapp is kind of ugly. Here's mine:
In my own case I've so far decided to accept the ugly config file, but as you can see, I haven't gotten around to adding even a robots.txt or any of the other files the modern web ecosystem expects; and adding them involves adding them one-by-one. I can see why someone would say, "Why make an ugly hack of an nginx config, when I can just define the favicon location in the metadata to a path easily configured to be files-only?"How big is your site? Crawl budget is likely only relevant for huge sites, not personal blogs.
Exactly - this SEOveroptimisation
Crawl budget is relevant to every site in Common Crawl.
The best SEO advice is to not focus on SEO and make a site that people will like.
Technical SEO still is a very valid optimization. Making sure you have all the relevant tags, a good structure, fast loading pages etc.
Just focus on accessibility and standards.
Hmm. I never would have added Schema.org support unless there was an SEO reason.
Yeah. If you solve accessibility you generally solve SEO as a byproduct. Mainly because accessible sites are easy for machines to parse.
Thinking about SEO too hard sucks, but having a site that nobody can find (even when they already know about it and are specifically trying to find it again!) sucks even more.
You dont want to exclude your images. That can very much affect your results as it will remove you from image tab, but also content of articles that contain them might be affected.
I thought that Google Search Console had tools to test robots.txt and sitemap.xml files, but it has been a while since I have needed to do that.
For those wondering why favicon is in a directory, nowadays there are half a dozen different favicon files for different devices in different situations and there are online tools such as The Real Favicon Generator that will take a source image and make the variants for you. These come with a code snippet for head and the option to use a sub directory so that you don't clutter the root.
Maybe they should offer a robots.txt snippet too.
Fun fact, for a single page, you can base64 encode the favicon and shove it in the page, thereby not needing a separate file. Why would you want to do that? If you base64 encode all the images and add the scripts and stylesheets in, then you can have a HTML page that you don't have to upload, you can email it to someone. This is useful if wanting to share a design mockup.
I'm a complete ignorant when it comes to SEOs so what are the consequences of not having a robots.txt nor a sitemap.xml at all? Will that be detrimental in a big way?
My understanding is that a lack of robots.txt should be fine and the lack of a sitemap.xml shouldn't be too troublesome as long as there is something linking to your site and all pages are linked from somewhere on your site (the sitemap helps search engines find all the links in one place but you have a nav bar or an article list that should work similarly, but you can give a suggestion to search engines about how often they should recrawl in your sitemap, which I don't believe you can influence any other way).
There are many metrics and facets of a website considered for SEO, with varying weights. The absence of either a robots.txt or sitemap.xml has a non-negligible but relatively minor weight compared to some other metrics. The files should be present and accurate when optimizing for seo. Ruby on Rails for example creates an empty robots.txt file with new projects, it does not create any sitemap.xml however.
I’m also ignorant of this, and to add a question on top of yours: is it worth worrying about robots.txt for personal portfolio websites built from scratch?
Depends on your hosting platform.
Nope! A site without robots.txt is defaulted to saying, please do the default — crawl my site in its entirety.
Does anyone have suggestions on what a proper robots.txt would be?
How about:
The recommendation is to use an empty "Disallow:" rule rather than a catch all "Allow:" rule.
Otherwise that is the canonical minimal example.
Like this?
Precisely.
Why not just this:
Won't crawlers crawl by default?That's a valid robots.txt, but "proper" is entirely dependent on what you want to achieve. If you aren't looking to treat different bots differently, and are looking allow all of your site to be indexed, then that is exactly what you want.
If you don’t have millions of pages the crawl budget limitations most likely will have zero impact.
Make sure your basic technical SEO factors are all good. Search console is looking good and then don’t continue to worry unless you are a huge site that’s living off SEO traffic.
[dead]
[dead]
TLDR; I shoot myself in a foot thinking I'm shooting elsewhere. Don't do this!
Thanks for a useful info! /s
SEO can feel like such a balancing act—one tweak, and it’s a waiting game to see the impact! Sounds like you’ve learned a lot about the sensitivity of robots.txt.
If you’re into exploring new tech, you might like Rig. It’s a Rust library for building scalable, modular apps with LLMs, ideal if you’re branching out into AI or complex workflows. Keeps things type-safe and flexible.