Take a list of domains and probe for working HTTP and HTTPS servers

Last update: Dec 28, 2022

Related tags

Overview

httprobe

Take a list of domains and probe for working http and https servers.

Install

▶ go get -u github.com/tomnomnom/httprobe

Basic Usage

httprobe accepts line-delimited domains on stdin:

▶ cat recon/example/domains.txt
example.com
example.edu
example.net
▶ cat recon/example/domains.txt | httprobe
http://example.com
http://example.net
http://example.edu
https://example.com
https://example.edu
https://example.net

Extra Probes

By default httprobe checks for HTTP on port 80 and HTTPS on port 443. You can add additional probes with the -p flag by specifying a protocol and port pair:

▶ cat domains.txt | httprobe -p http:81 -p https:8443

Concurrency

You can set the concurrency level with the -c flag:

▶ cat domains.txt | httprobe -c 50

Timeout

You can change the timeout by using the -t flag and specifying a timeout in milliseconds:

▶ cat domains.txt | httprobe -t 20000

Skipping Default Probes

If you don't want to probe for HTTP on port 80 or HTTPS on port 443, you can use the -s flag. You'll need to specify the probes you do want using the -p flag:

▶ cat domains.txt | httprobe -s -p https:8443

Prefer HTTPS

Sometimes you don't care about checking HTTP if HTTPS is working. You can do that with the --prefer-https flag:

▶ cat domains.txt | httprobe --prefer-https

Docker

Build the docker container:

▶ docker build -t httprobe .

Run the container, passing the contents of a file into stdin of the process inside the container. -i is required to correctly map stdin into the container and to the httprobe binary.

▶ cat domains.txt | docker run -i httprobe

Comments

Install command not work for macOS Mojave

I re-install go both from official and with brew installer. But there is no difference.

After command : go get -u github.com/tomnomnom/httprobe execute nothing is change. There is no output in terminal also.

opened by cyb3rsalih 8
Not download httprobe

Hi,

i have a problem with the Httprobe because not download..

sorry for my english XD

order for install..

Apt install golang (working done) now go get -u github.com/tomnomnom/httprobe (not error message, but dont download anything..)

attached screenshots.

https://ibb.co/c19P9Gb

https://ibb.co/QQYXKfF

upload photos from ImgBB.

opened by densdiego 3
Max concurrency?

Hello tomnomnom!

I have a short question, does httprobe have a max value for concurrency?

i am running it against +200k subdomains and sometimes even more, and it is a little bit slow with -c 50 provided. Is there a max value for the -c flag? any other method to speed this up without breaking everything?

Have a good day.

opened by marcelo321 2

Httprobe installation issue

go version go1.6.2 linux/amd64

PATH is set as follows:

export GOPATH=$HOME/go
export PATH=$PATH:$GOROOT/bin:$GOPATH/bin

~/go$ go get -u github.com/tomnomnom/httprobe
# github.com/tomnomnom/httprobe
src/github.com/tomnomnom/httprobe/main.go:57: unknown http.Transport field 'MaxIdleConns' in struct literal
src/github.com/tomnomnom/httprobe/main.go:58: unknown http.Transport field 'IdleConnTimeout' in struct literal
src/github.com/tomnomnom/httprobe/main.go:64: unknown http.Transport field 'DialContext' in struct literal
src/github.com/tomnomnom/httprobe/main.go:68: undefined: http.ErrUseLastResponse

opened by tacticthreat 2

Adding port templates
Currently, if you want to use httprobe and scan additional ports you have to use -p http:<PORT> which is fine for a few ports but for heavy scanning could become tedious.

I added two port templates to allow scanning for non-standard HTTP ports more easily. The idea is taken from aquatone.

The templates I implemented are:

xlarge for 80, 81, 300, 443, 591, 593, 832, 981, 1010, 1311, 2082, 2087, 2095, 2096, 2480, 3000, 3128, 3333, 4243, 4567, 4711, 4712, 4993, 5000, 5104, 5108, 5800, 6543, 7000, 7396, 7474, 8000, 8001, 8008, 8014, 8042, 8069, 8080, 8081, 8088, 8090, 8091, 8118, 8123, 8172, 8222, 8243, 8280, 8281, 8333, 8443, 8500, 8834, 8880, 8888, 8983, 9000, 9043, 9060, 9080, 9090, 9091, 9200, 9443, 9800, 9981, 12443, 16080, 18091, 18092, 20720, 28017

large for 80, 81, 443, 591, 2082, 2087, 2095, 2096, 3000, 8000, 8001, 8008, 8080, 8083, 8443, 8834, 8888

So if you want to use the xlarge template for example:

echo api.twitter.com | ./httprobe -p xlarge http://api.twitter.com https://api.twitter.com
opened by RiccardoAncarani 2
Unable to run httprobe from script

Hello,

I have been using httprobe for a while and everything is working fine so I decided to add it to my script, when I run the script I get an httprobe command not found. error. Every other tool in the script ( amass, nmap etc) is working fine and I can still use httprobe perfectly from the terminal in any directory (including the directory the script is in). Any thoughts? Thank you.

opened by MathematicsUnicorn 1
[Feature Request] Additional argument for printing hosts which weren't alive/responded

Hi @tomnomnom,

Thanks for the amazing tool, It really helps in HTTP probing.

As mentioned in the title, can we please have an additional argument if a user wants to print out the hosts which didn't respond or were down during the probing? Would help out in domains that point to a STALE record which is terminated etc.

opened by Anon-Exploiter 1
cannot run --prefer-https

cat abc.txt | httprobe --prefer-https -c 50

the above command is showing error

cat abc.txt | httprobe
#this command works fine though

I want the result to be either http or https and not both and if https is available I dont want http

opened by juniorhero 1
Read Full Response and Remove Defer Statement

The isListening function does not reads http bodies which may & do lead to error saying Too many open file descriptors as the connection does not gets closed. Adding an io.Copy to copy the response to ioutil.Discard solves the problem.

The second thing i did was remove the usage of defer from that code path as calling defer is more expensive than a simple statement.

opened by Ice3man543 1
Automatically figure out whether to prepend http protocol to url or not
Currently, the input url must be in the format:

google.com xyz.com

Pushing code to automatically prepend the HTTP protocol if it is not present, or if its present don't error out, continue with that url.

All the below URL's are valid input now

http://google.com google.com https://google.com
opened by adwaithks 0
Better -p Option Formatting
Hi Tom,

This pull request addresses the issue that you proposed about specifying a large number of ports being inconvenient given how values passed to the -p option are formatted (#3). The modifications that I made to httprobe allow the following to work:

$ cat domains.txt | httprobe -p 1234 $ cat domains.txt | httprobe -p 1234,4321 $ cat domains.txt | httprobe -p large $ cat domains.txt | httprobe -p large,1234 $ cat domains.txt | httprobe -p http:1234 -p https:1234,4321 $ cat domains.txt | httprobe -p http:large -p https:xlarge,1234

Determining which ports to probe on each of HTTP and HTTPS now happens only once (when the flags are parsed). Prior to this, httprobe would run a switch case for each host which is a waste of a few CPU cycles (I say this sarcastically :D).

Also note that I made the relevant changes to the README.md file (including what's mentioned in #23).

Best Regards, Adam
opened by NULLHE4D 0
add status code filter and follow redirects
To avoid getting massive 404 pages on large scopes I added two new flags:

-f: Filter the given status code

- --follow-redirect Use the default redirection policy, following up to a maximum of 10 times
opened by itasahobby 0
go get: installing executables with 'go get' in module mode is deprecated. Use 'go install [email protected]' instead.

Help on how to install because even after using the command: go install github.com/tomnomnom/[email protected]

The is no response from the httprobe in terminal returns: httprobe: command not found

opened by ZedTechInfo 3
Update README.md

Installing executables with "go get" in module mode is deprecated. "go install [email protected]" should be used instead. For more information, see https://golang.org/doc/go-get-install-deprecation

opened by adilsoybali 1