Dreamweaver Club Forums

Dreamweaver Club Forums (http://www.dreamweaverclub.com/forum//index.php)
-   SEO (http://www.dreamweaverclub.com/forum//forumdisplay.php?f=192)
-   -   Robots.txt problems (http://www.dreamweaverclub.com/forum//showthread.php?t=45116)

vinnyvangogh 03-02-2013 02:14 AM

Robots.txt problems
 
I have updated a website and in doing so ( I want to have ) removed
folders/directories and pages/URLS

the tutorials on this forums pages say one thing, examples and Google seem at odds!

Sitemap: http://www.xxxxx.com/sitemap.xml
User-Agent:*
Disallow: /XXXXXX (a folder!)
Disallow: /XXXXX.html
is one example


The next is
User-Agent:*
Disallow: /XXXXXX/
Disallow: /~XXXXX.html

this adds / and ~


As it is after 4 hours of re-writing different txt files and running the Google check in Webtools - each time it reports...Syntax not understood.

Also I have seen User-Agent* and User-Agent: am I missing something??

What do I need to do to block folders entirely and pages within folders

TIA

vinnyvangogh 08-20-2013 01:03 AM

Is there a problem with notifications?
 
I have a notification that fredpeterson543 made a reply to my post yesterday.. but there is no reply here on the forum and no indication the email was a PM...

For information on the saga of replacing a website with a new version.

Google is still reporting nearly 400 "404" errors from pages blocked by robots text and some that were deleted from the site almost 5 years ago!

Webtools has an option to re-index the home page and all other pages - which has been done a few times now! Google may re-index the site but they do not apparently update their archive and hang on to dead pages.

As a possible warning to others.. renaming or deleting old PDf files and such can and does enable Google to advice a bucket full of "errors" - dead links and getting them removed or re-named is almost impossible. Checking some of the dead links shows there are directory site who spyder the net and add a website without asking the owner/company concerned, which is possibly part of the reasons Googles discounting backlinks from such places.

edbr 08-20-2013 01:26 AM

i deleted the post as it was a pointless warning to be careful using robot.txt. in reality it was a thinly disguised way to add a spam link. I am getting tired of these and dealt with it. All we ask is that people join this forum to be part of it , then after that it is within the rules to add a signature. its not that much to ask!!

yes google moves slow, im not convinced dead links are penalised and in fact google realises 404 warnings happen . ust an opinion though

vinnyvangogh 08-20-2013 01:43 AM

Quote:

Originally Posted by edbr (Post 211805)
i deleted the post as it was a pointless warning to be careful using robot.txt. in reality it was a thinly disguised way to add a spam link. I am getting tired of these and dealt with it. All we ask is that people join this forum to be part of it , then after that it is within the rules to add a signature. its not that much to ask!!

yes google moves slow, im not convinced dead links are penalised and in fact google realises 404 warnings happen . ust an opinion though

OK, I got the email on a PC - but the forum is on my Mac...moving over I was confused as to no post! Actually its happened before so I understand now.

Thanks for your vigilance.....

devkinddt 09-06-2013 09:35 AM

That's good description about robots.txt file.

vinnyvangogh 03-11-2016 06:10 AM

Quote:

Originally Posted by wdcbangalore (Post 217597)
You can add your robot file using web master tool,there is search option is there you have to search robot file and add your robot plugin for your website....

Not sure if your reply is intended in response to my original posting - but it was posted in 2013 and I resolved all these issue then......

anikk302 04-28-2016 05:04 AM

robots.txt problem solution
 
same day ago i face same problem .and one day i find robots.txt problem solution the tutorial very help full

vinnyvangogh 04-28-2016 05:26 AM

Quote:

Originally Posted by anikk302 (Post 217713)
same day ago i face same problem .and one day i find robots.txt problem solution the tutorial very help full

Not much point giving a link that is in arabic or other mid east language.

Robots txt files are the easiest files to create. And to make it easier - search robots text checker..and you get a load of free tools online to check your file.


All times are GMT. The time now is 10:34 PM.

Powered by vBulletin® Version 3.8.7
Copyright ©2000 - 2017, vBulletin Solutions, Inc.
Copyright 2006 DreamweaverClub.com