dcrawl - a multi-threaded web crawler

Overview

dcrawl is a simple, but smart, multi-threaded web crawler for randomly gathering huge lists of unique domain names.

 

How to get this tool

To use this tool, please use a method listed below.

In a Linux (Debian OS), run the following command(s).

git clone https://github.com/kgretzky/dcrawl.git 

cd dcrawl

go build dcrawl.go

 

Download directly from the following link:

 

How to execute

 ./dcrawl -url http://wired.com -out ~/domain_lists/domains1.txt -t 8

 

References:

 

-----

Next steps:

 

This article was contributed by Jason Jacobs from Guyana. Jason is a member of the Caribbean CSPA.

Was this article helpful?
0 out of 0 found this helpful

Comments

0 comments

Article is closed for comments.