Spurred on by Mark Pilgrim's and Tim Bray's postings, I looked at the access log for my four week old blog to see what robots were visiting me.

No fewer than 71 different hosts grabbed my robots.txt in eight days. Two creepy ones: NameProtect (trademark enforcement) and TurnItIn (anti-plagiarism). This doesn't bug me too much. I do wonder why some of these crawlers felt it necessary to fetch robots.txt forty times in eight days.

  2003-03-02 18:43 Z