Recon Tool: Hakrawler

by | Oct 21, 2022 | Tools

Premium Content

Patreon
Subscribe to Patreon to watch this episode.
Reading Time: 2 Minutes

Hakrawler

Hakrawler by hakluke is a fast Golang web crawler for gathering URLs and JavaScript file locations. This is basically a simple implementation of the awesome Gocolly library, designed for easy, quick discovery of endpoints and assets within a web application.

Basically as a Bug Bounty Hunter, Pentester, you always want a tool that can extract all URL endpoints from an application and simply dump them to the command line. So this is what hakrawler does.

The URLs are extracted by spidering the application, querying the Wayback machine, parsing robots.txt files, and parsing sitemap.xml files. The tool also collects any subdomains it finds along the way. The way it does its subdomain enumeration method is not currently used by any other popular subdomain enumeration tools, so it may help to uncover some additional targets.

 

See Also: So you want to be a hacker?
Complete Offensive Security and Ethical Hacking Course

 

Example usages

Single URL:

echo https://google.com | hakrawler

Multiple URLs:

cat urls.txt | hakrawler

Timeout for each line of stdin after 5 seconds:

cat urls.txt | hakrawler -timeout 5

Send all requests through a proxy:

cat urls.txt | hakrawler -proxy http://localhost:8080

Include subdomains:

echo https://google.com | hakrawler -subs

 

Note: a common issue is that the tool returns no URLs. This usually happens when a domain is specified (https://example.com), but it redirects to a subdomain (https://www.example.com). The subdomain is not included in the scope, so the no URLs are printed. In order to overcome this, either specify the final URL in the redirect chain or use the -subs option to include subdomains.

 

 

Example tool chain

Get all subdomains of google, find the ones that respond to http(s), crawl them all.

echo google.com | haktrails subdomains | httpx | hakrawler

 

 

Installation

Normal Install

First, you’ll need to install go.

Then run this command to download + compile hakrawler:

go install github.com/hakluke/hakrawler@latest

You can now run ~/go/bin/hakrawler. If you’d like to just run hakrawler without the full path, you’ll need to export PATH=”~/go/bin/:$PATH”. You can also add this line to your ~/.bashrc file if you’d like this to persist.

Docker Install (from dockerhub)

echo https://www.google.com | docker run --rm -i hakluke/hakrawler:v2 -subs

 

Local Docker Install

It’s much easier to use the dockerhub method above, but if you’d prefer to run it locally:

git clone https://github.com/hakluke/hakrawler

cd hakrawler

sudo docker build -t hakluke/hakrawler .

sudo docker run --rm -i hakluke/hakrawler --help

 

Kali Linux: Using apt

Note: This will install an older version of hakrawler without all the features, and it may be buggy. I recommend using one of the other methods.

sudo apt install hakrawler

Then, to run hakrawler:

echo https://www.google.com | docker run --rm -i hakluke/hakrawler -subs

 

 

Command-line options

SS

Clone the repo from here: GitHub Link

 

Merch

Recent Tools

Offensive Security & Ethical Hacking Course

Begin the learning curve of hacking now!


Information Security Solutions

Find out how Pentesting Services can help you.


Join our Community

Share This