logo-dw

Go Back   Dreamweaver Club Forums > Dreamweaver forums > SEO
Register FAQ Members List Search Today's Posts Mark Forums Read

Closed Thread
 
Thread Tools Display Modes
Old 03-23-2011, 11:10 AM   #1
Corrosive
Corrosive's Avatar
 
Join Date: Feb 2008
Location: Bristol, England
Posts: 7,103
Default Google Crawling IP URLs

HELP!!

Very annoyed today . I discovered a few days ago that Google was crawling the https version of my site. Hands up that I should have applied the SSL to the required folder only but what is done is done.

Anyhoo, I found a .htaccess rewrite that served a different robots.txt when the https version was used. Then went through and requested removal of all https results. This has worked well and now I no longer show duplicate pages as http/https.

Having done all that I find that Google is now intent on crawling my site by IP! I mean WTF is Google doing crawling and serving IP results anyway?? Absolutely bizarre that I block https and then a matter of days later this crops up. If I were more paranoid I'd say something sinister was happening.

Anyhoo, any ideas on blocking access to IP URLs? Should I use the same method as I did with the https? Any help to preserve my sanity would be awesome.
Corrosive is offline  
Old 03-23-2011, 01:09 PM   #2
Corrosive
Corrosive's Avatar
 
Join Date: Feb 2008
Location: Bristol, England
Posts: 7,103
Default

Have settled on;

Code:
#Serves alternative robots file when requested through IP
RewriteCond %{HTTP_HOST} ^xxx\.xxx\.xxx\.xx
RewriteCond %{REQUEST_URI} ^/robots.txt$
RewriteRule ^(.*)$ /robots_alternative.txt [L]
to serve alternative robots.txt to try and prevent Google from crawling. Will see how it goes unless anyone has any better ideas?
Corrosive is offline  
Old 03-23-2011, 01:18 PM   #3
DWcourse
DWcourse's Avatar
 
Join Date: Apr 2009
Posts: 3,276
Default

You can also use the canonical meta tag in the header of your pages http://www.google.com/support/webmas...?answer=139394
__________________
If my answer helped, check out DWcourse.com for Dreamweaver tips and tutorials.
DWcourse is offline  
Old 03-23-2011, 01:34 PM   #4
Corrosive
Corrosive's Avatar
 
Join Date: Feb 2008
Location: Bristol, England
Posts: 7,103
Default

Have thought about that Jim but our site is dynamically generated. I guess I could do something clever with the page id.
Corrosive is offline  
Old 03-23-2011, 04:02 PM   #5
Corrosive
Corrosive's Avatar
 
Join Date: Feb 2008
Location: Bristol, England
Posts: 7,103
Default

Well, I have learnt more about .htaccess than I ever card to today. Still, hopefully fixed
Corrosive is offline  
Old 04-03-2011, 07:20 AM   #6
sbglobal
Banned
 
Join Date: Mar 2011
Posts: 8
Default

we are also getting same issue.
sbglobal is offline  
Closed Thread


Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT. The time now is 06:25 AM.


Powered by vBulletin® Version 3.8.7
Copyright ©2000 - 2017, vBulletin Solutions, Inc.
Copyright 2006 DreamweaverClub.com