Adding robots.txt to my moodle web root

After upgrading some of my moodle sites to 2.4, I have been monitoring the error logs for each.  Today I noticed that they were reporting errors of a missing robots.txt file.  To be honest, I had forgot about the robots.txt files after my upgrades.  I am not sure what is referencing this in my site, but I decided to create the robots.txt file again and place in the web root of each of my upgrades sites.

An excerpt from my error log

[Mon Feb 11 08:22:06 2013] [error] [client 208.115.111.73] File does not exist: C:/wamp/www/wayne/robots.txt

I do not know who is calling or looking for robots.txt on my site?

I used this page to help recreate the content of the file. I also used this Microsoft education file to help inform me about web crawlers and spiders and the robots.txt file.

I am allowing only the index.php file to be indexed...my robots file

User-agent: *
Disallow: /
Allow: /index.php

Moral(s) of the story?

1. Use your error logs, especially after an upgrade.  
2. You should probably create a robots.txt file to direct the web spiders that come to your site.