Skip to content

Instantly share code, notes, and snippets.

@jhaddix
jhaddix / Github bash generated search links (from hunter.sh)
Created January 12, 2020 19:55
Github bash generated search links (from hunter.sh)
import requests
import re
import sys
from multiprocessing.dummy import Pool
def robots(host):
r = requests.get(
'https://web.archive.org/cdx/search/cdx\
?url=%s/robots.txt&output=json&fl=timestamp,original&filter=statuscode:200&collapse=digest' % host)