Determine if a page may be crawled from robots.txt and robots meta tags
composer require spatie/robots-txt