logo-dw

Go Back   Dreamweaver Club Forums > Dreamweaver forums > SEO
Register FAQ Members List Search Today's Posts Mark Forums Read

Closed Thread
 
Thread Tools Display Modes
Old 03-02-2013, 03:14 AM   #1
vinnyvangogh
 
Join Date: Aug 2009
Posts: 84
Default Robots.txt problems

I have updated a website and in doing so ( I want to have ) removed
folders/directories and pages/URLS

the tutorials on this forums pages say one thing, examples and Google seem at odds!

Sitemap: http://www.xxxxx.com/sitemap.xml
User-Agent:*
Disallow: /XXXXXX (a folder!)
Disallow: /XXXXX.html
is one example


The next is
User-Agent:*
Disallow: /XXXXXX/
Disallow: /~XXXXX.html

this adds / and ~


As it is after 4 hours of re-writing different txt files and running the Google check in Webtools - each time it reports...Syntax not understood.

Also I have seen User-Agent* and User-Agent: am I missing something??

What do I need to do to block folders entirely and pages within folders

TIA
vinnyvangogh is offline  
Old 08-20-2013, 02:03 AM   #2
vinnyvangogh
 
Join Date: Aug 2009
Posts: 84
Default Is there a problem with notifications?

I have a notification that fredpeterson543 made a reply to my post yesterday.. but there is no reply here on the forum and no indication the email was a PM...

For information on the saga of replacing a website with a new version.

Google is still reporting nearly 400 "404" errors from pages blocked by robots text and some that were deleted from the site almost 5 years ago!

Webtools has an option to re-index the home page and all other pages - which has been done a few times now! Google may re-index the site but they do not apparently update their archive and hang on to dead pages.

As a possible warning to others.. renaming or deleting old PDf files and such can and does enable Google to advice a bucket full of "errors" - dead links and getting them removed or re-named is almost impossible. Checking some of the dead links shows there are directory site who spyder the net and add a website without asking the owner/company concerned, which is possibly part of the reasons Googles discounting backlinks from such places.
vinnyvangogh is offline  
Old 08-20-2013, 02:26 AM   #3
edbr
edbr's Avatar
 
Join Date: Aug 2005
Location: Bali
Posts: 11,176
Default

i deleted the post as it was a pointless warning to be careful using robot.txt. in reality it was a thinly disguised way to add a spam link. I am getting tired of these and dealt with it. All we ask is that people join this forum to be part of it , then after that it is within the rules to add a signature. its not that much to ask!!

yes google moves slow, im not convinced dead links are penalised and in fact google realises 404 warnings happen . ust an opinion though
__________________
If you're happy and you know it shake your meds!
different style links examples

Flight / Hotel search
Free script download
Bali Villas
edbr is online now  
Old 08-20-2013, 02:43 AM   #4
vinnyvangogh
 
Join Date: Aug 2009
Posts: 84
Default

Quote:
Originally Posted by edbr View Post
i deleted the post as it was a pointless warning to be careful using robot.txt. in reality it was a thinly disguised way to add a spam link. I am getting tired of these and dealt with it. All we ask is that people join this forum to be part of it , then after that it is within the rules to add a signature. its not that much to ask!!

yes google moves slow, im not convinced dead links are penalised and in fact google realises 404 warnings happen . ust an opinion though
OK, I got the email on a PC - but the forum is on my Mac...moving over I was confused as to no post! Actually its happened before so I understand now.

Thanks for your vigilance.....
vinnyvangogh is offline  
Old 09-06-2013, 10:35 AM   #5
devkinddt
 
Join Date: Sep 2013
Location: Delhi
Posts: 1
Default

That's good description about robots.txt file.
devkinddt is offline  
Old 03-11-2016, 07:10 AM   #6
vinnyvangogh
 
Join Date: Aug 2009
Posts: 84
Default

Quote:
Originally Posted by wdcbangalore View Post
You can add your robot file using web master tool,there is search option is there you have to search robot file and add your robot plugin for your website....
Not sure if your reply is intended in response to my original posting - but it was posted in 2013 and I resolved all these issue then......
vinnyvangogh is offline  
Old 04-28-2016, 06:04 AM   #7
anikk302
Banned
 
Join Date: Apr 2016
Posts: 3
Default robots.txt problem solution

same day ago i face same problem .and one day i find robots.txt problem solution the tutorial very help full

Last edited by edbr; 05-03-2016 at 02:33 AM.. Reason: removed link
anikk302 is offline  
Old 04-28-2016, 06:26 AM   #8
vinnyvangogh
 
Join Date: Aug 2009
Posts: 84
Default

Quote:
Originally Posted by anikk302 View Post
same day ago i face same problem .and one day i find robots.txt problem solution the tutorial very help full
Not much point giving a link that is in arabic or other mid east language.

Robots txt files are the easiest files to create. And to make it easier - search robots text checker..and you get a load of free tools online to check your file.

Last edited by edbr; 05-03-2016 at 02:33 AM.. Reason: removed OP link
vinnyvangogh is offline  
Closed Thread


Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT. The time now is 05:50 PM.


Powered by vBulletin® Version 3.8.7
Copyright ©2000 - 2017, vBulletin Solutions, Inc.
Copyright 2006 DreamweaverClub.com