| ||
Fri 2025-01-10
Thu 2025-01-09
Wed 2025-01-08
Tue 2025-01-07
Mon 2025-01-06
Sun 2025-01-05
Sat 2025-01-04
Fri 2025-01-03
Thu 2025-01-02
Wed 2025-01-01
Search
Archives
2023
12 11 10 09 08 07 06 05 04 03 02 01
2022
2021
2020
2019
2018
2017
2016
2015
2014
2013
2012
2011
2010
2009
2008
2007
2006
2005
2004
2003
One good site
MDN
Nelson Minar
Blog licensed under a Creative Commons License
|
Spurred on by
Mark
Pilgrim's
and
Tim Bray's
postings, I looked at the access log for my four week old blog to see
what robots were visiting me.
No fewer than 71 different hosts grabbed my robots.txt in eight days. Two creepy ones: NameProtect (trademark enforcement) and TurnItIn (anti-plagiarism). This doesn't bug me too much. I do wonder why some of these crawlers felt it necessary to fetch robots.txt forty times in eight days. |
|
