I saw a discussion on Twitter this morning about the idea of having a feature in Robots.txt that would block both crawling AND indexing. It started with this tweet by Christian Thurston (@ct_oz):
“Hi John [Mueller], has Google considered making it so that the robots.txt file doesn’t just block crawling, but also blocks indexation? To quote @willcritchlow: “I can’t see many situations where I want to block crawling but don’t want to block indexing”.
“That would be a significant change in expectations (and yes, we do think about these things regardless). Do you have some examples where this would cause a visible improvement in search?” Mueller responded. “I’d like to avoid adding more directives. I’m still not aware of common issues caused by this documented functionality … SEOs worry about indexing, but usually these URLs only rank for site:-queries (or if there isn’t other, better content on the site), so it feels artificial?”
With over 20 years of experience in SEO, “I have never encountered a situation where a publisher wanted to have a page indexed that they block for crawling in robots.txt. Not even once have I seen that,” I tweeted in response. “It’s common practice for me to educate people that they have a choice: (1) block crawling, or: (2) prevent indexation, when what they want to do is both. Note: definitely more of an issue for larger sites where crawl budget is an issue.”
Will Critchlow of Search Pilot agreed, “100% agree. I can see the conservatism of not...
Read Full Story: https://searchengineland.com/should-robots-txt-support-a-feature-for-no-indexation-take-the-survey-374757
Your content is great. However, if any of the content contained herein violates any rights of yours, including those of copyright, please contact us immediately by e-mail at media[@]kissrpr.com.
