NYCPHP Meetup

[inforequest]

Given the interest in search engine work I'm cross-posting this here f.y.i.

Reports are coming in that SonicWall firewalls are now delivered 
preconfigured to block any IP that requests robots.txt. SonicWall 
allegedy included this via two of the thousands of recommended default 
security policies. What this means is that search engine spiders will be 
blocked from your site, and that will lead to being dropped from the 
indexes. In fact, some webmasters only found out about this when 
multiple websites were dropped following a firewall upgrade.

This appears to be a major blunder, given how difficult it can be to get 
back into some SEs these days. The justification is ok, but having it ON 
by default or as part of a set of recommended policies is a blunder. 
(because robots.txt holds info on where you do not want spiders to go, 
it may highlight areas of your web server that contain privileged 
information. Given that, the robots.txt may provide sensitive info, and 
that is why Sonicwall claims it is a security risk)

Since many webmasters have no involvement with their network firewalls 
it is probably very wise to give a heads-up to the netadmin people, or 
to otherwise be aware of this issue.

see
http://software.sonicwall.com/applications/ips/index.asp?ev=sig&sigid=1600
http://software.sonicwall.com/applications/ips/index.asp?ev=sig&sigid=1601
http://www.webmasterworld.com/forum93/631.htm


-=john andrews
http://www.seo-fun.com