$ wget -e robots=off -r -np 'http://example.com/folder/'
- -e robots=off causes it to ignore robots.txt for that domain
- -r makes it recursive
- -np = no parents, so it doesn't follow links up to the parent folder
-
-
Save ChinnakornP/1850c057c15353b543a91dd52b580728 to your computer and use it in GitHub Desktop.
Recursive wget ignoring robots
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment