Open Mobile Menu

Blog

Filed In: Network Security, Security Testing

Web Application Enumeration at Scale

Views: 1471

Written By: Stephen Haywood December 07, 2018

When we conduct network penetration tests, we often find that the majority of accessible services are web applications. Often times, these web applications have administrative interfaces with weak/default passwords or have vulnerabilities that allow us to access sensitive data or even allow arbitrary execution of code. Trying to find interesting web applications among a list of hundreds or thousands of servers can be a daunting task. For each web server, we typically take a screenshot, brute force common directory names, and look for vulnerable configuration settings. The only sane way to do this at scale is through automation.

Today, we are releasing a set of basic scripts to the AppSec Consulting Github repo (https://github.com/AppSecConsulting/Pentest-Tools) that can be used to help with enumerating web applications at scale.

  • msfweb.sh – Extracts web servers from Metasploit console using the services command
  • gobuster.sh – Bruteforce common files on a web server using Gobuster
  • nikto.sh – Scan each web server for common vulnerabilities using Nikto
  • wkhtml.sh – Take a screenshot of the web server using wkhtmltoimage

Each of the scanning scripts (gobuster.sh, nikto.sh, and wkhtml.sh) expect a newline delimited file with each line describing a web server in the format of:

     scheme host port

 

msfweb.sh

During engagements, we typically use msfconsole to import Nmap and Nessus results into the Metasploit database so we can then search for targets that match specific criteria using the services command. The msfweb.sh script uses the services command to extract a list of web servers from the Metasploit database and then puts the list of servers into the “scheme host port” format required by the scanning scripts.

     # ./msfweb.sh
     Saving extracted web servers to servers.txt

     # cat servers.txt
     http 192.168.0.1 80
     http 192.168.0.2 443
     http 192.168.0.2 80
     http 192.168.0.3 80
     http 192.168.0.4 443
     http 192.168.0.4 80

 

gobuster.sh

The gobuster.sh script will run Gobuster against each web server in the input file. The script includes a WORDLIST variable and an OPTS variable, which can be modified to meet your needs. The gobuster.sh script expects an input file and an output directory, where all of the output files will be stored. The output for each web server will be stored in a separate file with a naming convention of “gobuster-[HOST]-[PORT].txt”.

To bruteforce common filenames on each web server with an output folder of “enum” and an input file of “servers.txt”, execute the following command:

     # ./gobuster.sh enum servers.txt

 

nikto.sh and wkhtml.sh

The nikto.sh and wkhtml.sh scripts work in the same way as the gobuster.sh script. Each script has an OPTS variable that can be modified to meet your needs and each one stores its output in a separate file within the given directory. The naming convention for the nikto.sh script is “nikto-[HOST]-[PORT].txt” and the naming convention for the wkhtml.sh script is “wkhtml-[HOST]-[PORT].png”.

To run a Nikto scan against each server with an output folder of “enum” and an input file of “servers.txt”, execute the following command:

     # ./nikto.sh enum servers.txt

To take a screenshot of each server with an output folder of “enum” and an input file of “servers.txt”, execute the following command:

     # ./wkhtml.sh enum servers.txt

Each script has a set of default options that have worked well for us. You can see the default options for each script in the screenshot below:


Figure 1 – Screenshot of default options in each scanning script.

These web enumeration scripts were designed to run in Kali Linux and should include most, if not all, of the prerequisites. If they aren’t installed for some reason, you can install the prerequisites with apt:

     # apt install gobuster, secliststs, nikto, csvkit, wkhtmltopdf

 

We’d love to get your feedback on web enumeration. What other scripts would you add? Are there better tools to use? Better default options? Let us know about it in our Github repo, https://github.com/AppSecConsulting/Pentest-Tools.

Stephen Haywood

Stephen Haywood, aka AverageSecurityGuy, is a Senior Penetration Tester with AppSec Consulting with 14 years of experience in the Information Technology field working as a programmer, technical trainer, network operations manager, and information security consultant. He holds a Bachelor of Science in Math, the Certified Information Systems Security Professional (CISSP) certification, the Offensive Security Certified Expert (OSCE) certification, and the Offensive Security Certified Professional (OSCP) certification. Over the last eight years, he has helped improve the network security of many small businesses ranging in size from ten employees to hundreds of employees by offering practical, time-tested information security advice.

In his off hours, Stephen created a number of security tools including the Prometheus firewall analysis tool and a set of penetration testing scripts used by testers worldwide. In addition, Stephen has made multiple contributions to the Metasploit exploitation framework including, auxiliary, exploitation, and post exploitation modules. Finally, Stephen created and delivered high-quality security training, spoke at multiple security conferences, and self-published an introduction to penetration testing book.

read more articles by Stephen Haywood