PDA

View Full Version : Blocl Site from Spiders


aargus07
07-13-2009, 11:20 AM
I have a site that needs to be "by invitation
only", meaning I only want people that I tell
about the site to get to it. Accordingly, I
would like to block the site from being spidered
by the search engines. How can I do this?

Dan

Corrosive
07-13-2009, 11:47 AM
I have a site that needs to be "by invitation
only", meaning I only want people that I tell
about the site to get to it. Accordingly, I
would like to block the site from being spidered
by the search engines. How can I do this?

Dan

Hi Dan

Read this...

http://www.convonix.com/articles/uncategorized/no-index-and-no-follow-tags-to-guide-search-engine-spiders/

for some guidance. There is loads on the web about the concept. If you make your first page a log-in using something like PHP, then make sure there is a session enabled on every other page for it to be entered, Spiders will not be able to access the other pages anyway.

If you don't intend to have a log-in then I reckon your pages will get found somehow and won't be as secure as you like. The web is public so 'invitation only' must be enforced somehow.

edbr
07-14-2009, 01:54 AM
look at htaccess, you could use it to block sections or password protect your site or block robots though not completely