Google open sources its robots.txt parser as part of this announcement.

Please visit Search Engine Land for the full article.

Search Engine Land: News & Info About SEO, PPC, SEM, Search Engines & Search Marketing https://searchengineland.com/google-posts-draft-to-formalize-robots-exclusion-protocol-specification-318998