Extra Credit

robots-parser

A specification compliant robots.txt parser with wildcard (*) matching support.

$npm install robots-parser

Official docs outline

Extra Credit builds an outline of robots-parser's docs the first time your work uses it. Until then, start at the official documentation.

All 9,484