URLFinder: No More Active Scanning for URL Gathering

by ·

URLFinder

URLFinder is a high-speed, passive URL discovery tool designed to simplify and accelerate web asset discovery, ideal for penetration testers, security researchers, and developers looking to gather URLs without active scanning.

Feature

  • Curated Passive Sources to maximize comprehensive URL discovery
  • Supports multiple output formats (JSON, file, stdout)
  • Optimized for Speed and resource efficiency
  • STDIN/OUT support for easy integration into existing workflows

Use

A streamlined tool for discovering associated URLs.

Usage:
  ./urlfinder [flags]

Flags:
INPUT:
   -d, -list string[]  target domain or list of domains

SOURCE:
   -s, -sources string[]           specific sources for discovery (e.g., -s alienvault,commoncrawl)
   -es, -exclude-sources string[]   sources to exclude (e.g., -es alienvault,commoncrawl)
   -all                             use all sources (may be slower)

FILTER:
   -m, -match string[]       URLs or list to match (file or comma-separated)
   -f, -filter string[]      URLs or list to filter (file or comma-separated)

RATE-LIMIT:
   -rl, -rate-limit int      max HTTP requests per second (global)
   -rls, -rate-limits value  per-provider HTTP request limits (e.g., -rls waybackarchive=15/m)

UPDATE:
   -up, -update              update URLFinder to the latest version
   -duc, -disable-update-check  disable automatic update checks

OUTPUT:
   -o, -output string       specify output file
   -j, -jsonl               JSONL output format
   -od, -output-dir string  specify output directory
   -cs, -collect-sources    include all sources in JSON output

CONFIGURATION:
   -config string           config file (default "$CONFIG/urlfinder/config.yaml")
   -pc, -provider-config string  provider config file (default "$CONFIG/urlfinder/provider-config.yaml")
   -proxy string            HTTP proxy

DEBUG:
   -silent                  show only URLs in output
   -version                 display URLFinder version
   -v                       verbose output
   -nc, -no-color           disable colored output
   -ls, -list-sources       list all available sources
   -stats                   display source statistics

OPTIMIZATION:
   -timeout int   timeout in seconds (default 30)
   -max-time int  max time in minutes for enumeration (default 10)

Basic Usage

urlfinder -d tesla.com

This command enumerates URLs for the target domain tesla.com.

Filtering Options

 

Use the -m (match) and -f (filter) options to refine results based on URL patterns.

Examples

 

  1. Include URLs Matching Specific Patterns

    To include only URLs containing “shop” or “model”:

    urlfinder -d tesla.com -m shop,model

  2. Exclude URLs Matching Specific Patterns

    To exclude URLs containing “privacy” or “terms”:

    urlfinder -d tesla.com -f privacy,terms

  3. Combined Match and Filter

    To find URLs containing “support” but exclude those with “faq”:

    urlfinder -d tesla.com -m support -f faq

Using Files for Matching and Filtering

 

Provide patterns in files:

urlfinder -d tesla.com -m include-patterns.txt -f exclude-patterns.txt

JSONL Output Example

 

Use the -j or --jsonl flag to output results in JSONL (JSON Lines) format, where each line is a separate JSON object. This format is useful for processing large outputs in a structured way.

Command Example

 

urlfinder -d tesla.com -j

Example JSONL Output

 

{"url":"https://shop.tesla.com/product/model-s-plaid","input":"tesla.com","source":"waybackarchive"} {"url":"https://www.tesla.com/inventory/used/ms","input":"tesla.com","source":"waybackarchive"} {"url":"https://forums.tesla.com/discussion/101112/model-3-updates","input":"tesla.com","source":"waybackarchive"}
Each JSON object contains:
  • url: The discovered URL.
  • input: The target domain (e.g., tesla.com).
  • source: The data source for the URL discovery (e.g., waybackarchive).

Download

Tags: