Skip to content

Instantly share code, notes, and snippets.

@calmhavoc

calmhavoc/scanning.md

Last active January 22, 2026 01:18
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save calmhavoc/3dbb11b1a80bf778a45f411b07a7395c to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save calmhavoc/3dbb11b1a80bf778a45f411b07a7395c to your computer and use it in GitHub Desktop.
Download ZIP
Raw
scanning.md
Note 1
# Note, to speed scans up, set a scan variable here and add it to nmap lines where desired, eg:

nmap_defaults="--max-rtt-timeout 100ms --min-hostgroup 64 --min-rate 1000 --max-retries 2"

# nmap -Pn -n -sS -p 21-23,111,137,445,80,443 -iL target_scope.txt -oG $projectdir/nmap/$custid.quickscan-tcp.gnmap $nmap_defaults
Note 2
# Note: for testy networks causing corrupted scans, we can scan each IP in a seperate thread to ensure we don't lose data on canceled scans. -P 10 creates 10 threads

# cat target_ips.txt | xargs -P 10 -I{} nmap --top-ports 1000 -sS -Pn --open -n -T4 --oA threaded_scan --append-output {}

Pasteables

Setup

Install resources and packages

git clone https://github.com/ChrisTruncer/gnmap-parser  
cd gnmap-parser  
mv Gnmap-Parser.sh /usr/local/bin/gnmap-parser.sh  
chmod +x /usr/local/bin/gnmap-parser.sh  
cd .. && rm -rf gnmap-parser

apt install eyewitness -y  
apt install dirsearch -y  
apt install prips -y  
apt install wapiti -y

# todo: project discovery toolkit

Modify configs

# vi /etc/nikto.conf and change UPDATES=yes to no; this disables Nikto calls to update its external central db for new found signatures, we should update when not connected to target networks

Set project variables; if using multiple sessions, set in each

project_id=TESTNET_101
user=kali
projectdir=/home/$user/ops/$project_id

mkdir -p $projectdir
chown $user:$user $projectdir

runuser $user -c "mkdir -p $projectdir/{nmap,smb,ftp,eyewitness,dirbusting,nikto,wapiti,nuclei}"

Create initial target resource files

cat << _EOF > $projectdir/target_scope.txt
192.168.1.0/24 
_EOF

##### Create list of IP addresses from the scope so we can remove IPs that have been completed.
nmap -sL -n -iL $projectdir/target_scope.txt | awk '/Nmap scan report/{print $NF}' > $projectdir/target_ips.txt

##### Or use prips to convert CIDR to IPs; or use nmap:  
# prips 10.10.110.0/25> $projectdir/target_ips.txt
Rework this, nmap can't take a port list this long; can just rescan top1000 for final scans
#### Create port list continaining all ports minus the top 1000
sort -r -k3 /usr/share/nmap/nmap-services | grep -i tcp | cut -f 2 | cut -d '/' -f1 | awk 'NR<=1000' > /tmp/nmap_top_1000.txt
for i in $(seq 1 65535); do echo $i>>/tmp/all_ports.txt; done

# set remaining ports variable for use in final scan
rem_ports = $(cat /tmp/all_ports.txt |grep -vwf /tmp/nmap_top_1000.txt| sed 's/^\|$//g'|paste -sd,)

Scanning

Phase 1 - Discovery

Conduct initial nmap quick discovery scans
nmap_defaults="--max-rtt-timeout 300ms --min-hostgroup 64 --min-rate 1000 --max-retries 1"

#### Nmap default discovery scan
nmap -sn -n -iL $projectdir/target_scope.txt -oG $projectdir/nmap/$project_id-sweep.gnmap

#### Create temp list of hosts discovered so far
cat $projectdir/nmap/$project_id-sweep.gnmap | grep Up | cut -d ":" -f2 | cut -d "(" -f1 | tr -d " " > $projectdir/nmap/$project_id.tmp_livehosts.txt

#### Discovery based on common open TCP ports
nmap -Pn -n -sS -p 21-23,111,137,445,80,443 -iL $projectdir/target_scope.txt -oG $projectdir/nmap/$project_id.quickscan-tcp.gnmap '$nmap_defaults'

#### Discovery based on common UDP ports (this one is a sped up via our nmap_defaults
nmap -Pn -sU -n -p 53,161,2049 -iL $projectdir/target_scope.txt -oG $projectdir/nmap/$project_id.quickscan-udp.gnmap '$nmap_defaults'

#### Create our final livehost file for deeper scanning
cat $projectdir/nmap/*.gnmap | grep -v "#" | grep -vE "filtered|Up" |awk '{print $2}' >> $projectdir/nmap/$project_id.tmp_livehosts.txt

cat $projectdir/nmap/$project_id.tmp_livehosts.txt | sort | uniq > $projectdir/nmap/$project_id.livehosts.txt

Phase 2 - Top Service Scan

Conduct a top 1000 TCP services scan of the discovered live hosts from above so we have something to work with while conducting final scans
nmap -Pn -n -sS -sV -O -T4 --script "default and safe" -oA $projectdir/nmap/$project_id.livehosts-top1000-TCP --open -iL $projectdir/nmap/$project_id.livehosts.txt '$nmap_defaults'
Conduct a top 20 UDP services scan of the discovered live hosts from above so we have something to work with while conducting final scans
nmap -Pn -sUV -n --top-ports=20 -iL $projectdir/target_scope.txt -oG $projectdir/nmap/$project_id.quickscan-udp.gnmap '$nmap_defaults'
Parse results so we can start looking for common items and low hanging fruit while the deeper scans run
cd $projectdir/nmap  
gnmap-parser.sh -p *.gnmap  
xsltproc $projectdir/nmap/$project_id.livehosts-top1000-TCP.xml -o $projectdir/nmap/$project_id.livehosts-top1000-TCP/vulns-nmap-top1000.html  
cd $projectdir

(If --scripts weren't executed, can use these deeper service and web scans to do a quick grab of commonly found items; can be skipped if desired )

SMB Checks

Optional - Attempt to list shares on discovered smb hosts
for i in $(cat $projectdir/nmap/Parsed-Results/Port-Matrix/TCP-Services-Matrix.csv|grep ",445" |cut -d "," -f1  
);do ip=$(echo $i |cut -d '/' -f 3| cut -d ':' -f1);nmap -n -v -sV -p 445 --script=smb-enum-domains,smb-enum-groups,smb-enum-processes,smb-enum-services,smb-enum-sessions,smb-enum-shares,smb-enum-users,smb-mbenum,smb-psexec,smb-system-info,smb-vuln-cve-2017-7494,smb-vuln-ms17-010 --script-args=unsafe=1 $i -oA $projectdir/smb/$project_id.$ip-smb_445-nmap; done

for i in $(cat $projectdir/nmap/Parsed-Results/Port-Matrix/TCP-Services-Matrix.csv|grep ",445" |cut -d "," -f1  
);do ip=$(echo $i |cut -d '/' -f 3| cut -d ':' -f1);echo $ip>$projectdir/smb/$project_id.$ip-smb-ls;smbclient -N -L $ip >>$projectdir/smb/$project_id.$ip-smb-ls;done


for i in $(cat $projectdir/nmap/Parsed-Results/Port-Matrix/UDP-Services-Matrix.csv|grep ",137" |cut -d "," -f1  
);do ip=$(echo $i |cut -d '/' -f 3| cut -d ':' -f1);nmap -sU -n -v -sV -p 137 --script=smb-enum-domains,smb-enum-groups,smb-enum-processes,smb-enum-services,smb-enum-sessions,smb-enum-shares,smb-enum-users,smb-mbenum,smb-psexec,smb-system-info,smb-vuln-cve-2017-7494,smb-vuln-ms17-010 --script-args=unsafe=1 $i -oA $projectdir/smb/$project_id.$ip-smb_137-nmap; done

FTP Checks

for i in $(cat $projectdir/nmap/Parsed-Results/Port-Matrix/TCP-Services-Matrix.csv|grep ",21" |cut -d "," -f1);do ip=$(echo $i |cut -d '/' -f 3| cut -d ':' -f1);nmap -n -Pn -sV -v -p 20,21 --script=ftp-anon,ftp-syst,ftp-vsftpd-backdoor,ftp-proftpd-backdoor,ftp-libopie $i -oA $projectdir/ftp/$project_id.$ip-ftp_nmap; done

Exexcute additional web scans for webhosts; using for loops to prevent rescans

Create web host urls from nmap

Using bash and gnmap file

output_file=/tmp/tmp_webhosts
# Use find to get .gnmap files and process them
find "$projectdir/nmap" -name "*.gnmap" | while IFS= read -r gnmap_file; do
  # Extract IP addresses from the .gnmap files
  grep -oE 'Host: [0-9.]+' "$gnmap_file" | awk '{print $2}' | while IFS= read -r ip; do
    # Look for open TCP services related to the IP
    grep "$ip" "$gnmap_file" | grep -oE '[0-9]+/open/tcp//[^/]+/[^ ]+' | while IFS=/ read -r port _ _ service; do
      # Check for HTTP or HTTPS services
      if echo "$service" | grep -q 'https\|ssl\|rtsp'; then
        echo "https://$ip:$port" >> "$output_file"
      elif echo "$service" | grep -q 'http'; then
        echo "http://$ip:$port" >> "$output_file"
      fi
    done
  done
done

# Sort and deduplicate the results
sort -u "$output_file" | sort -t. -k1,1n -k2,2n -k3,3n -k4,4n > "$projectdir/web_hosts.txt"

Run eyewitness - Review to help direct dirbusting

runuser $user -c "eyewitness --no-dns --no-prompt --timeout 10 --max-retries 1 --results=100 -d $projectdir/eyewitness -f $projectdir/web_hosts.txt"

Run dirsearch (or replace with dirb/gobuster/etc)

for i in $(cat $projectdir/web_hosts.txt);do outname=$(echo $i |tr -d "/"|sed s/:/_/g);runuser $user -c "dirsearch -u $i --retries=2 --max-time=1200 -i 200,403 -o $projectdir/dirbusting/$project_id.$ip-dirsearch_default_20min.txt";done

Run nikto

# commented out; rarely adds more than wapati
# for i in $(cat $projectdir/web_hosts.txt);do ip=$(echo $i |cut -d '/' -f 3| cut -d ':' -f1);runuser $user -c "nikto -host $i -o nikto/nikto-$ip-x6.txt -maxtime 600 -Tuning x6";done

Run wapiti (leaving out unsafe methods)

*Need to modify ssrf and xxe checks as it calls out to wapiti external
for i in $(cat $projectdir/web_hosts.txt);do host=$(echo $i |cut -d '/' -f 3);ip=$(echo $host| cut -d ":" -f 1); if [[ $host =~ [':'] ]]; then port=$(echo $i |cut -d '/' -f 3| cut -d ':' -f2);else port='default';fi;echo "Scanning $ip:$port";runuser $user -c "wapiti -u $i -o $projectdir/wapiti/$project_id-$ip-$port-wapiti.txt -f txt -d 3 --max-attack-time 1200 --max-scan-time 600 -m backup,blindsql,cookieflags,crlf,csp,csrf,file,htaccess,http_headers,methods,redirect,shellshock,sql,xss" ;done

#### Optional - Run wapiti on the specific discovered urls (just for more depth if desired)
# for i in $(cat $projectdir/dirbusting/*  |grep -v "#" |grep -E "200|301" | awk '{print $3}'); do ip=$(echo $i |cut -d '/' -f 3| cut -d ':' -f1);runuser $user -c "wapiti -u $i -o $projectdir/wapiti/$project_id.$ip-wapiti_url.txt -f txt -d 3 --max-attack-time 1200 --max-scan-time 600 -m backup,blindsql,cookieflags,crlf,csp,csrf,file,htaccess,http_headers,methods,redirect,shellshock,sql,xss" ;done

Run nuclei

# need to incorporate more templates besides the defaults
# nuclei -ut
for i in $(cat $projectdir/web_hosts.txt);do ip=$(echo $i |cut -d '/' -f 3| cut -d ':' -f1);runuser $user -c "nuclei -u $i -o $projectdir/nuclei/$project_id.$ip-nuclei.txt" ;done

Phase 3: FULL TCP SCAN ON DISCOVERED HOSTS

Scan remaining TCP ports on the discovered live hosts from above

nmap -Pn -n -sS -p- -sV -O -T4 --max-retries 2 --min-hostgroup 64 -oA $projectdir/nmap/$project_id.livehosts-fullTCP --open -iL $projectdir/nmap/$project_id.livehosts.txt '$nmap_defaults'

## FINAL PORT SCANNING
### Scan remaining ports on the discovered live hosts from above

Conduct top 100 UDP scan of the discovered live hosts (or more if desired; change output name to match)

nmap -Pn -n -sU -sV -T4 --top-ports=100  --max-retries 1 --min-hostgroup 64 --version-intensity 0 -iL $projectdir/nmap/$project_id.livehosts.txt -oA $projectdir/nmap/$project_id.top100UDP '$nmap_defaults'

Create a new targetlist that removes the already scanned targets

cat target_ips.txt |grep -vf Parsed-Results/Host-Lists/Alive-Hosts-Open-Ports.txt > $projectdir/nmap/$custid-deadhosts.txt


# nmap -Pn -n -sS -p $rem_ports -sV -O -T4 --max-retries 2 --min-hostgroup 64 --host-timeout 5m -oA $projectdir/nmap/$custid.remaininghosts-fullTCP --open -iL $projectdir/nmap/$custid-deadhosts.txt

Parse final results using gnmap-parser

#*(alternatively, use msf: msfdb init; msfconsole -q ; workspace -c somename)*  
gnmap-parser.sh -p *.gnmap

Other Enumeration Scans

# MSSQL (for example)
nmap --script ms-sql-info,ms-sql-empty-password,ms-sql-xp-cmdshell,ms-sql-config,ms-sql-ntlm-info,ms-sql-tables,ms-sql-hasdbaccess,ms-sql-dac,ms-sql-dump-hashes --script-args mssql.instance-port=1433,mssql.username=sa,mssql.password=,mssql.instance-name=MSSQLSERVER -sV -p 1433 <IP>

Kick off any of the above scans in a screen session

sudo screen -dmS scans
sudo screen -S scans -p 0 -X stuff "nmap -sS ...\n"

TODO

# Add nmap vulnscan
# add skipfish
# consider parallel scan via xargs -P or parallel -j
# Convert nmap output to html
xsltproc nmapoutput.xml -o nmapoutput.html

# Convert nmap output to html
xsltproc nmapoutput.xml -o nmapoutput.html

# for each url endpoint, try skipfish:
# skipfish -o skiptest http://192.168.225.37/somesite/admin
# or skipfish teh whole site skipfish -o skiptest http://192.168.225.37/



# Create job list for each ip/port combo
# eg: 
# 192.168.1.2,tcp,443,apache,1.2.1,dirb 
# 192.168.1.2,tcp,443,apache,1.2.1,nikto
# 192.168.1.2,tcp,139,samba,1.9.2,nmap-smb-scripts
# 192.168.1.2,tcp,2049,nfs,NONE,nfs-scripts
# 192.168.1.7,tcp,21,proftpd,1.3.9,nmap-ftp-scripts+vuln

Need to run scraper to get dirb targets to determine what list to use

Incorporate parellel execution with bash

#run threaded/parallel command
`seq 1 254 | xargs -P 20 -I{} proxychains nmap -p 80,443,3389,445,22 -sT -Pn --open -n -T4 --min-parallelism 100 --min-rate 1 --oG proxychains_nmap --append-output 192.168.1.{}

#another way - Can do this faster using parallel or xargs:
cat <LIST_OF_RESOLVED_ASSOCIATED_DOMAINS> | parallel -j <NO_OF_CONCURRENT_JOBS> "amass enum -passive -d {} -o {}.out"
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment