Recon Tool: WitnessMe

Reading Time: 5 Minutes

Reconnaissance Tool: WitnessMe 

GitHub Link

WitnessMe

WitnessMe by byt3bl33d3r, is primarily a Web Inventory tool inspired by Eyewitness, its also written to be extensible allowing you to create custom functionality that can take advantage of the headless browser it drives in the back-end.

WitnessMe uses the Pyppeteer library to drive Headless Chromium.

When you are performing Bug Bounty or Pentesting, especially for thousands of URLs out of your recon process, such tools are designed to take screenshots of websites provide some server header info, and identify default credentials if known. This significantly speed up the process of engaging in Critical Thinking and easily identify weaknesses and planning your next steps, visually.

Features

Here are some of the main features that make WitnessMe “stand out”:

• Written in Python 3.7+
• Ability to parse extremely large Nessus and NMap XML files
• Docker compatible
• No installation/dependency hell
• Full test suite! Everything is less prone to bugs
• CSV & HTML reporting
• HTTP Proxy Support
• Provides a RESTful API! Scan stuff remotely!
• CLI interface to view and search scan results without having to view the reports.
• Signature scanning (Signatures use YAML files)
• Preview screenshots directly in the terminal (On MacOSX/ITerm2 and some Nix terminals)
• Extensible written, allowing you to add functionality that can take advantage of headless chromium.
• Built to be deployed to the Clouds (e.g. GCP Cloud Run , AWS ElasticBeanstalk etc…)

See Also: Apple fixes new zero-day exploited to hack macOS, iOS devices

Installation

Docker

Running WitnessMe from a Docker container is fully supported and is the easiest/recommended way of using the tool.

Note: It is highly recommended to give the Docker container at least 4GB of RAM during large scans as Chromium can be a resource hog. If you keep running into “Page Crash” errors, it’s because your container does not have enough memory. On Mac/Windows you can change this by clicking the Docker Task Bar Icon ➡ Preferences ➡ Resources. For Linux, refer to Docker’s documentation.

Pull the image from Docker Hub:

docker pull byt3bl33d3r/witnessme

You can then spin up a docker container, run it like the main witnessme script and pass it the same arguments:

docker run –rm -ti $IMAGE_ID screenshot https://google.com 192.168.0.1/24

Alternatively, you can drop into a shell within the container and run the tools that way. This also allows you to execute the wmdb and wmapi scripts.

docker run –rm -ti –entrypoint=/bin/sh $IMAGE_ID

Python Package

WitnessMe is also available as a Python package (Python 3.7 or above is required). If you do install it this way it is extremely recommended to use pipx as it takes care of installing everything in isolated environments for you in a seamless manner.

Run the following commands:

python3 -m pip install –user pipx
pipx install witnessme

All of the WitnessMe scripts should now be in your PATH and ready to go.

Development Install

You really should only install WitnessMe this way if you intend to hack on the source code. You’re going to Python 3.7+ and Poetry: please refer to the Poetry installation documentation in order to install it.

git clone https://github.com/byt3bl33d3r/WitnessMe && cd WitnessMe
poetry install

See Also: Complete Offensive Security and Ethical Hacking Course

Quick Starts

Finding F5 Load Balancers Vulnerable to CVE-2020-5902

Install WitnessMe using Docker:

docker pull byt3bl33d3r/witnessme

Get the $IMAGE_ID from the docker images command output, then run the following command to drop into a shell inside the container. Additionally, specify the -v flag to mount the current directory inside the container at the path /transfer in order to copy the scan results back to your host machine (if so desired):

docker run -it –entrypoint=/bin/sh -v $(pwd):/transfer $IMAGE_ID

Scan your network using WitnessMe, it can accept multiple .Nessus files, Nmap XMLs, IP ranges/CIDRs. Example:

witnessme screenshot 10.0.1.0/24 192.168.0.1-20 ~/my_nessus_scan.nessus ~/my_nmap_scan.xml

After the scan is finished, a folder will have been created in the current directory with the results. Access the results using the wmdb command line utility:

wmdb scan_2020_$TIME/

To quickly identify F5 load balancers, first perform a signature scan using the scan command. Then search for “BIG-IP” or “F5” using the servers command (this will search for the “BIG-IP” and “F5” string in the signature name, page title and server header):

Additionally, you can generate an HTML or CSV report using the following commands:

WMDB ≫ generate_report html

WMDB ≫ generate_report csv

You can then copy the entire scan folder which will contain all of the reports and results to your host machine by copying it to the /transfer folder.

See Also: Stuxnet – A weapon made out of code that almost started WW3

Scraping Javascript Heavy Webpages

As of v1.5.0, WitnessMe has a grab command which allows you to quickly scrape Javascript heavy webpages by rendering the page first with Headless Chromium and then parsing the resulting HTML using the specified XPath (see here for an XPath cheatsheet).

Below are a few examples to get your started.

This grabs a list of all advertised domains on the 144.161.160.0/23 subnet from Hurricane Electric’s BGP Toolkit:

witnessme -d grab -x ‘//div[@id=”dns”]/table//tr/td[2]/a/text()’ https://bgp.he.net/net/144.161.160.0/23#_dns

RESTful API

As of version 1.0, WitnessMe has a RESTful API which allows you to interact with the tool remotely.

Note: Currently, the API does not implement any authentication mechanisms. Make sure to allow/deny access at the transport level.

To start the RESTful API for testing/development purposes run:

wmapi

The API documentation will then be available at http://127.0.0.1:8000/docs

Uvicorn should be used to enable SSL and run the API in production. See this dockerfile for an example.

Deploying to the Cloud (™)

Since WitnessMe has a RESTful API now, you can deploy it to the magical cloud and perform scanning from there. This would have a number of benefits, including giving you a fresh external IP on every scan (More OPSEC safe when assessing attack surface on Red Teams).

There are a number of ways of doing this, you can obviously do it the traditional way (e.g. spin up a machine, install docker etc..).

Recently cloud service providers started offering ways of running Docker containers directly in a fully managed environment. Think of it as serverless functions (e.g. AWS Lambdas) only with Docker containers.

This would technically allow you to really quickly deploy and run WitnessMe (or really anything in a Docker container) without having to worry about underlying infrastructure and removes a lot of the security concerns that come with that.

Below are some of the ones that the creator have tried along with the steps necessary to get it going and any issues he encountered.

• GCP Cloud Run
• AWS ElasticBeanstalk

Usage

There are 3 main utilities:

• witnessme: is the main CLI interface.
• wmdb: allows you to browse the database (created on each scan) to view results and generate reports.
• wmapi: provides a RESTful API to schedule, start, stop and monitor scans.

• Modes of Operation
  • Screenshot Mode
  • Grab Mode
  • Interacting with the Scan Database

Generating Reports

You can use the generate_report command in the wmdb cli to generate reports in HTML or CSV format. To generate a HTML report simply run generate_report without any arguments. Here’s an example of what it’ll look like:

To generate a CSV report:

WMDB ≫ generate_report csv

The reports will then be available in the scan folder.

Proxying

As of v1.5 WitnessMe supports proxying all of its traffic through an HTTP proxy. Specify a HTTP_PROXY environment variable to force the underlying headless browser to proxy its traffic through the desired host:

HTTP_PROXY=http://127.0.0.1:8080 witnessme screenshot ~/my_targets.txt

HTTP_PROXY=http://127.0.0.1:8080 witnessme grab https://www.google.com

Preview Screenshots Directly in the Terminal

Note: This feature will only work if you’re on MacOSX and using ITerm2.

You can preview screenshots directly in the terminal using the show command:

Writing Signatures

If you run into a new webapp write a signature for it! It’s beyond simple and they’re all in YAML!

Here’s the AirOS signature (you can find them all in the signatures directory):

credentials:
- password: ubnt
  username: ubnt
name: AirOS
signatures:
- airos_logo.png
- form enctype="multipart/form-data" id="loginform" method="post"
- align="center" class="loginsubtable"
- function onLangChange()
# AirOS ubnt/ubnt