PDA

View Full Version : Robots blocking folders


vinnyvangogh
07-07-2013, 02:47 PM
Hi,

I have a website which has folders which are not needed and folders that are needed - but include dead pages.
I have "disallowed" using robots.txt to stop Google indexing them.
Question is having a site map - which does not show the folders - when is it safe - from a GOOGLE indexing and busted links point of view -to just delete them and change the robots.txt file

edbr
07-08-2013, 07:17 AM
if they are not needed why keep them there just clear the site of the unneeded stuff

vinnyvangogh
07-08-2013, 03:04 PM
if they are not needed why keep them there just clear the site of the unneeded stuff

The blurb I read suggests that just deleting pages and folder - which are indexed with Google et al.. are seen as bad links /404s/301s etc and rack up bad points which can affect Googles listing status.

I presumed that the robots blocking will lead to the SEs indexing only whats allowed and eventually the blocked stuff can be deleted.. without adverse affect.

edbr
07-09-2013, 02:28 AM
i read that to but am not convinced its a huge factor. 404 stay on google records but you can apply to get them delisted. i dont see the blocking will make that much difference. sites change naturally so if the page is important set a redirect permanent

vinnyvangogh
07-15-2013, 01:21 AM
i read that to but am not convinced its a huge factor. 404 stay on google records but you can apply to get them delisted. i dont see the blocking will make that much difference. sites change naturally so if the page is important set a redirect permanent

I decided to delete everything not wanted on the site in the rebuild.
No more need for robots.txt file... renewed the site map and re-submitted for a re-index.
I do not expect much from Google for a few weeks... but so far its looking OK apart from busted links looking for the content I deleted - which presumably will go away when the site is actually fully reindexed.

edbr
07-15-2013, 02:17 AM
that may take sometiime which is part of my possibly twisted logic that google dont give it a of of credence . they do say somewhere that they expect 404 's