View Full Version : robots.txt

04-03-2006, 12:35 PM
HI all, I have recently read some stuff about the robots.txt file.

I am currently trying to optimise my site and was just wondering if I need this robots.txt file to be indexed correctly.

My site only has two .html pages, one is the index.htm and the flash version detector and the other holds my main-site page.

Is this robots.txt necessary ??

Any advice on optimisation would be most helpful to me at the moment.

the site is at www.waringcollins.com incase anyone wants to see the source/tags etc.

Thanks for any replys in advance, & apologies if i posted in the wrong forum !!!


04-03-2006, 01:17 PM
you only need that file if you want to give special instrusctions. If not the spider will just crawl and index your site normally, which is probably what you want.

04-03-2006, 01:33 PM
cheers mate, understood.