Description

This plugin finds new URL’s using google. It will search for “site:domain.com” and do GET requests all the URL’s found in the result. One configurable parameter exists:

  • result_limit

Plugin type

Crawl

Options

Name Type Default Value Description Help
result_limit integer 300 Fetch the first “result_limit” results from the Google search No detailed help available

Source

For more information about this plugin and the associated tests, there’s always the source code to understand exactly what’s under the hood:
github-logoPlugin source code
Unittest source code

Dependencies

This plugin has no dependencies.