dcrawl - a multi-threaded web crawler


dcrawl is a simple, but smart, multi-threaded web crawler for randomly gathering huge lists of unique domain names.


How to get this tool

To use this tool, please use a method listed below.

In a Linux (Debian OS), run the following command(s).

git clone https://github.com/kgretzky/dcrawl.git 

cd dcrawl

go build dcrawl.go


Download directly from the following link:


How to execute

 ./dcrawl -url http://wired.com -out ~/domain_lists/domains1.txt -t 8





Next steps:


This article was contributed by Jason Jacobs from Guyana. Jason is a member of the Caribbean CSPA.

Was this article helpful?
0 out of 0 found this helpful



Article is closed for comments.