extra credit Search 85,062 technologies ⌘K Download for Mac

github.com/temoto/robotstxt

Package robotstxt implements the robots.txt Exclusion Protocol as specified in with various…

go get github.com/temoto/robotstxt
↑↓ move ↵ open