4
When it comes to WEB systems What would be the safest and most used way for it to become invisible to search engines? I know there are the "Goals" but I think there should be more reliable way
1 answer
4
Before indexing a website search engines in general search for a file called robots txt. and respect the guidelines contained in this file.
Example: before visiting www.meusistema.with Crawler will search www.meusistema.with/robots.txt
You can say that the site should not be indexed by placing the following content in robots.txt:
User-agent: *
Disallow: /
You may also allow the site to be indexed with exceptions:
User-agent: *
Disallow: /protegido/
Disallow: /secreto/
A care in the latter case: While you are preventing these folders from being indexed you are also making it "public" that they exist.
Malicious robots will not respect these directives and will probably send a User-Agent string as if it were a common browser, in this case only Captchas may prevent them, but this is a more intrusive solution as real users are also affected.
Browser other questions tagged security-guard quest erp robots
You are not signed in. Login or sign up in order to post.
+1 because it is all right. Just as a complement I would add that there are on the web robot generators, that can help those who are unfamiliar with it.
– eightShirt