tenpastmidnight blog

I've been adding code to read robots.txt files to my new website - SpiderTest. It's been one of those classic coding days where you're tapping away for hours and at the end of the day there's only a hundred lines of code, whereas other times the code's jumping out of your fingertips like it's no effort at all.

Still, the robots.txt support is in there for all robots, so it can tell if the page is spiderable at all, which is great. I just have to go and soothe my melting brain now.