Google has proposed an official internet standard for the rules included in robots.txt files.
The post Google Wants to Establish an Official Standard for Using Robots.txt via @MattGSouthern appeared first on Search Engine Journal.
Search Engine Journal http://tracking.feedpress.it/link/13962/12403213