tenpastmidnight blog

Making hay while the sun shines

Robots reading

I've been adding code to read robots.txt files to my new website - SpiderTest. It's been one of those classic coding days where you're tapping away for hours and at the end of the day there's only a hundred lines of code, whereas other times the code's jumping out of your fingertips like it's no effort at all.

Still, the robots.txt support is in there for all robots, so it can tell if the page is spiderable at all, which is great. I just have to go and soothe my melting brain now.

This page is powered by Blogger. Isn't yours?