Google announces open-source robots.txt parser
Google claims that the Robots Exclusion Protocol (REP) has been a standard for the past 25 years, bringing a lot of uncertainty to webmasters and crawler tool developers. Google now announced that it will...
