This plugin searches for the robots.txt file, and parses it. This file is used to as an ACL that defines what URL’s a search engine can access. By parsing this file, you can get more information about the target web application.
This plugin doesn’t have any user configured options.
This plugin has no dependencies.