.Google.com's John Mueller addressed an inquiry about why Google.com indexes web pages that are actually prohibited from crawling through robots.txt and why the it is actually safe to overlook the related Browse Console files concerning those crawls.Bot Web Traffic To Question Criterion URLs.The individual asking the concern recorded that robots were developing web links to non-existent question criterion URLs (? q= xyz) to pages along with noindex meta tags that are actually likewise blocked out in robots.txt. What caused the inquiry is actually that Google.com is crawling the web links to those web pages, receiving blocked out through robots.txt (without envisioning a noindex robots meta tag) at that point getting reported in Google Browse Console as "Indexed, though obstructed through robots.txt.".The individual talked to the complying with concern:." However here's the significant concern: why would certainly Google index webpages when they can not even view the material? What's the advantage because?".Google's John Mueller validated that if they can't creep the page they can not see the noindex meta tag. He likewise creates an appealing acknowledgment of the website: search operator, advising to overlook the end results because the "typical" customers will not view those end results.He wrote:." Yes, you're correct: if our team can't creep the web page, our team can't find the noindex. That mentioned, if we can not crawl the web pages, then there is actually certainly not a great deal for us to mark. Therefore while you may see some of those web pages with a targeted web site:- query, the common user will not find all of them, so I would not bother it. Noindex is actually additionally alright (without robots.txt disallow), it merely implies the URLs will end up being crept (and find yourself in the Search Console file for crawled/not recorded-- neither of these standings result in concerns to the remainder of the internet site). The fundamental part is actually that you don't create them crawlable + indexable.".Takeaways:.1. Mueller's response confirms the limitations in using the Web site: search accelerated hunt operator for diagnostic explanations. Among those causes is since it's not hooked up to the frequent hunt mark, it is actually a separate point altogether.Google.com's John Mueller discussed the internet site hunt operator in 2021:." The brief response is actually that a site: concern is actually certainly not indicated to be full, nor used for diagnostics functions.A web site question is a certain kind of search that limits the results to a particular internet site. It's primarily simply words internet site, a bowel, and afterwards the site's domain name.This inquiry limits the outcomes to a certain site. It is actually not implied to be an extensive compilation of all the webpages from that web site.".2. Noindex tag without making use of a robots.txt is great for these type of conditions where a bot is linking to non-existent webpages that are actually obtaining found through Googlebot.3. URLs along with the noindex tag are going to produce a "crawled/not listed" entry in Browse Console which those won't have a damaging impact on the remainder of the web site.Review the inquiry as well as answer on LinkedIn:.Why would certainly Google.com index pages when they can't also see the information?Included Picture through Shutterstock/Krakenimages. com.