About eight months ago I
moved my blog
to a new domain and set
up 301 redirects
to point everyone to it. 301 means "moved
permanently": all bots should eventually stop hassling the old URL.
Does it work? Mostly; here's a list of hits in last week's traffic
from various bots.
- 808 hits, Rojobot
- A feed aggregator with several old RSS URLs they apparently have
no way of updating.
- 424 hits, TailRank robot
- A service that claims to "find hot stories", in my case by looking
at eight-month-old URLs and not following the "hot lead" that the site
has moved. Grabs both RSS feeds and some HTML.
- 390 hits, Feedfetcher-Google
- RSS crawler for Google Reader. I'm pretty
sure I know the guy responsible for this bot; tsk tsk, B.
- 254 hits, Yahoo!
- Basic crawlers trying to grab some HTML. Arguably legitimate:
they're verifying the 301 redirect is still live.
- 136 hits, BecomeBot
- Some shopping site bot. Guys, I got nothing to sell.
- 108 hits, AppleSyndication
- There's one person who uses the Safari RSS reader! Too bad about the
None of this matters much; the 301 is cheap to serve and other than
the Rojo and Safari examples I think no one would care that I stopped
serving it. At least the big consumers like BlogLines have all