I can see your reasoning, I just think you have made a “brave” call that will likely cause trouble for some people. But it’s great to see that you guys are being so responsive about any problems and hopefully nothing too bad will happen.

Aside from that, The National Web Harvest sounds like a fantastic project. Rock on NatLib!

Thanks for the feedback (I know people usually say that snarkily, but I’m really sincere here).

We’ve just just blogged about the Web Harvest & the pain it’s causing (and the comments we’ve seen). Our intentions really are good – to collect & preserve & make accessible NZ’s digital heritage for people in the future, the same way we do already for books & newspapers & photographs – and we’re trying to respond and fix people’s problems as quickly as we can.

You can send comments & requests for us to change the crawler’s behaviour to web-harvest-2008@natlib.govt.nz.

Thanks heaps,

Courtney

I feel I have honored my end of the bargain in warning bots away from parts of my site that it would be useless to spider; NatLib is not upholding their responsibilities when running their bot.