PDA

View Full Version : Robots.txt problems


vinnyvangogh
03-02-2013, 03:14 AM
I have updated a website and in doing so ( I want to have ) removed
folders/directories and pages/URLS

the tutorials on this forums pages say one thing, examples and Google seem at odds!

Sitemap: http://www.xxxxx.com/sitemap.xml
User-Agent:*
Disallow: /XXXXXX (a folder!)
Disallow: /XXXXX.html
is one example


The next is
User-Agent:*
Disallow: /XXXXXX/
Disallow: /~XXXXX.html

this adds / and ~


As it is after 4 hours of re-writing different txt files and running the Google check in Webtools - each time it reports...Syntax not understood.

Also I have seen User-Agent* and User-Agent: am I missing something??

What do I need to do to block folders entirely and pages within folders

TIA

vinnyvangogh
08-20-2013, 02:03 AM
I have a notification that fredpeterson543 made a reply to my post yesterday.. but there is no reply here on the forum and no indication the email was a PM...

For information on the saga of replacing a website with a new version.

Google is still reporting nearly 400 "404" errors from pages blocked by robots text and some that were deleted from the site almost 5 years ago!

Webtools has an option to re-index the home page and all other pages - which has been done a few times now! Google may re-index the site but they do not apparently update their archive and hang on to dead pages.

As a possible warning to others.. renaming or deleting old PDf files and such can and does enable Google to advice a bucket full of "errors" - dead links and getting them removed or re-named is almost impossible. Checking some of the dead links shows there are directory site who spyder the net and add a website without asking the owner/company concerned, which is possibly part of the reasons Googles discounting backlinks from such places.

edbr
08-20-2013, 02:26 AM
i deleted the post as it was a pointless warning to be careful using robot.txt. in reality it was a thinly disguised way to add a spam link. I am getting tired of these and dealt with it. All we ask is that people join this forum to be part of it , then after that it is within the rules to add a signature. its not that much to ask!!

yes google moves slow, im not convinced dead links are penalised and in fact google realises 404 warnings happen . ust an opinion though

vinnyvangogh
08-20-2013, 02:43 AM
i deleted the post as it was a pointless warning to be careful using robot.txt. in reality it was a thinly disguised way to add a spam link. I am getting tired of these and dealt with it. All we ask is that people join this forum to be part of it , then after that it is within the rules to add a signature. its not that much to ask!!

yes google moves slow, im not convinced dead links are penalised and in fact google realises 404 warnings happen . ust an opinion though

OK, I got the email on a PC - but the forum is on my Mac...moving over I was confused as to no post! Actually its happened before so I understand now.

Thanks for your vigilance.....

devkinddt
09-06-2013, 10:35 AM
That's good description about robots.txt file.

vinnyvangogh
03-11-2016, 07:10 AM
You can add your robot file using web master tool,there is search option is there you have to search robot file and add your robot plugin for your website....
Not sure if your reply is intended in response to my original posting - but it was posted in 2013 and I resolved all these issue then......

anikk302
04-28-2016, 06:04 AM
same day ago i face same problem .and one day i find robots.txt problem solution the tutorial very help full

vinnyvangogh
04-28-2016, 06:26 AM
same day ago i face same problem .and one day i find robots.txt problem solution the tutorial very help full

Not much point giving a link that is in arabic or other mid east language.

Robots txt files are the easiest files to create. And to make it easier - search robots text checker..and you get a load of free tools online to check your file.