1
Hello, I found a file robots.txt with the following directives inside the file
User-agent: Googlebot
Allow: *
User-agent: Bingbot
Allow: *
User-agent: facebot
Allow: *
User-agent: *
Disallow: /
My question is, the last disallow command will not cancel all the above?
3
Will not block, the rules are not necessarily "in order", the rules with the "name" quoted have "preference" on a wildcard/wildcard (*), as in the google support example itself:
Allow access to a single tracker
User-agent: Googlebot-news Allow: / User-agent: * Disallow: /
Allow access to all trackers except one of them
User-agent: Unnecessarybot Disallow: / User-agent: * Allow: /
Another situation you should note is that the use of an asterisk (*), will correspond to all trackers, except the various trackers Google Adsbot, that need to be named explicitly
Browser other questions tagged html seo robots
You are not signed in. Login or sign up in order to post.
I thought that p
Disallow: /denied access to all internal directories, publicHTML type still with access, but publicHTML/Configuration the bot would not have access. Kind that the bot would only see what’s at the root, and not the folders inside the directory... But I guess I was finding rss wrong– hugocsl
@hugocsl this yes, blocks all, but in the "order", the "allow 'explicit'" apparently gives preference to the "named" BOT, but it depends on how you write, in case I reviewed the answer is
um wildcardand noto wildcard, soon maybe soon I need to reformulate the answer better to detail examples that would write, but I need to be sure of everything, because I only had to base the texts of google support, and maybe I "lost" something, beyond what I need to complement.– Guilherme Nascimento