Google's Insights On Robots.txt, Semantic HTML, & Rel-Canonical - Search Engine Journal

Replace (not provided) with ALL of your organic keywords inside of Adobe & Google Analytics. Analyze performance by 400+ dimensions and metrics.

TRY FOR FREE

Google Analyst Gary Illyes offers guidance on large robots.txt files, the SEO impact of website redesigns, and the correct use of rel-canonical tags.

Illyes is taking questions sent to him via LinkedIn direct message and answering them publicly, offering valuable insights for those in the SEO community.

It’s already newsworthy for a Google employee to share SEO advice. This is especially so given it’s Illyes, who isn’t as active on social media as colleagues like Search Advocate John Mueller and Developer Advocate Martin Splitt.

Throughout the past week, Illyes has shared advice and offered guidance on the following subjects:

• Large robots.txt files
• The SEO impact of website redesigns
• The correct use of rel-canonical tags

Considering the engagement his posts are getting, there’s likely more to come. Here’s a summary of what you missed if you’re not following him on LinkedIn.

Keep Robots.Txt Files Under 500KB

Regarding a previously published poll on the size of robots.txt files, Illyes shares a PSA for those with a file size larger than 500kb.

Screenshot from: linkedin.com/in/garyillyes/, January 2023.

Illyes advises paying attention to the size of your website’s robots.txt file, especially if it’s larger than 500kb.

Google’s crawlers only process the first 500kb of the file, so it’s crucial to ensure that the...

Your content is great. However, if any of the content contained herein violates any rights of yours, including those of copyright, please contact us immediately by e-mail at media[@]kissrpr.com.