Block indexing by robots in controller

Question

I’m making use of Codeigniter 2 to set up a website.
The admin area is in the url http://meudominio.com/mod, to access you need to login.
To avoid indexing of search engines, I configured the Routes file

$route['mod/(:any)'] = "$1";
$route['mod_upload'] = '';
$route['mod_config'] = '';
$route['mod_logo_upload'] = '';

The controller access was like this
http://meudominio.com/mod/mod_upload
http://meudominio.com/mod/mod_config
http://meudominio.com/mod/mod_logo_upload

To avoid page indexing, and end up appearing in google, just "block the /mod" directory in the file roobts.txt ? or only login will be enough ?

Answer 1

If these routes have authentication, probably not authenticated users are redirected should view the login page, in case a BOT should also view the login page (because in my opinion the bots should see the pages in the same way as the user)if this is the case you can use http 401 (Not authorized):

• php 5.4+:

  if (false === $condicao_necessaria_para_acesso) {
      http_response_code(401);
      //View para login
  } else {
      //Condição normal, views, models, etc
  }
  

• php less than 5.4:

  if (false === $condicao_necessaria_para_acesso) {
      header('X-PHP-Response-Code: 401', true, 401);
      //View para login
  } else {
      //Condição normal, views, models, etc
  }
  

If the page does not have authentication, but the route is accessible by an address or by a user-agent specified, you can use http code 403 (Prohibited) with exit;, so no need to send a custom page, in case the browser when it has no content usually shows a default error page of it and the searchers do not index this page.

• Use a condition to inform access:

  if (false === $condicao_necessaria_para_acesso) {
      http_response_code(403);
      exit;
  }
  

• php < 5.4:

  if (false === $condicao_necessaria_para_acesso) {
      header('X-PHP-Response-Code: 403', true, 403);
      exit;
  }