DedSecInside / TorBot

                         ████████╗ ██████╗ ██████╗     ██████╗  ██████╗ ████████╗
                         ╚══██╔══╝██╔═══██╗██╔══██╗    ██╔══██╗██╔═████╗╚══██╔══╝
                            ██║   ██║   ██║██████╔╝    ██████╔╝██║██╔██║   ██║
                            ██║   ██║   ██║██╔══██╗    ██╔══██╗████╔╝██║   ██║
                            ██║   ╚██████╔╝██║  ██║    ██████╔╝╚██████╔╝   ██║
                            ╚═╝    ╚═════╝ ╚═╝  ╚═╝    ╚═════╝  ╚═════╝    ╚═╝

                            Open Source Intelligence Tool for the Dark Web

Status/Social links

Features

1. Onion Crawler (.onion)
2. Returns page title or host name if no page title is available and address with a short description of the site.
3. Save links to a database (Not done)
4. Output the HTML from a site or save it to an HTML file. (Not done)
5. Save the link tree as a JSON file.
6. Crawl custom domains
7. Check if the link is live
8. Built-in Updater
9. Build a visual tree of link relationships that can be quickly viewed or saved to a file

Recent improvements

• The CLI now allows --version and --update to run without requiring a URL.
• The crawler now resolves relative links against the current page and skips unsupported or malformed hrefs more gracefully.
• The optional TorBotApp desktop UI can be launched from the CLI with torbot app when installed as a sibling checkout.

...(will be updated)

Dependencies

• Tor (Optional)
• Python 3.9+
• pip

Quick start

Local setup

python3 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install -r requirements.txt
python -m pip install -e .

Run the CLI

python main.py --help
python main.py --version
python main.py -u https://example.com --depth 2 --visualize table

Common examples

# Gather basic site information
python main.py -u https://example.com --info

# Save the crawl tree as JSON
python main.py -u https://example.com --depth 2 --save json

# Disable SOCKS5 if you are not using Tor locally
python main.py -u https://example.com --disable-socks5 --visualize tree

# Launch the optional desktop app when TorBotApp is installed
torbot app

Options

usage: Gather and analyze data from Tor sites.

optional arguments:
  -u URL, --url URL     Specify a website link to crawl
  --depth DEPTH         Specify max depth of crawler (default 1)
  -h, --help            Show this help message and exit
  --host                Set IP address for SOCKS5 proxy (defaults to 127.0.0.1)
  --port                Set port for SOCKS5 proxy (defaults to 9050)
  -v                    Displays DEBUG level logging, default is INFO
  --version             Show the current version of TorBot.
  --update              Update TorBot to the latest stable version
  --app                 Run optional TorBot desktop app
  --app-dir APP_DIR     Path to a TorBotApp checkout
  -q, --quiet           Prevents display of header and IP address
  --save FORMAT         Save results in a file. (tree, JSON)
  --visualize FORMAT    Visualizes tree of data gathered. (tree, JSON, table)
  -i, --info            Info displays basic info of the scanned site
  --disable-socks5      Executes HTTP requests without using SOCKS5 proxy

-u/--url is required for crawl-related commands such as --info, --save, and --visualize.

If you are using Tor locally, keep the SOCKS5 proxy enabled. If you are not using Tor, add --disable-socks5 to avoid proxy connection errors.

Read more about torrc here: Torrc

Optional desktop app

TorBot remains a CLI-first Python tool. The desktop UI lives in the separate TorBotApp Electron project so Node and Electron are not required for CLI users.

To launch it from this CLI, install TorBotApp as a sibling checkout:

code/
+-- TorBot/
+-- TorBotApp/

Then run:

torbot app

If TorBotApp is somewhere else, use either:

TORBOT_APP_DIR=/path/to/TorBotApp torbot app
torbot --app --app-dir /path/to/TorBotApp

See Desktop App Architecture for how the CLI, optional UI, and backend services fit together.

Curated Features

• Visualization Module Revamp
• Implement BFS Search for webcrawler
• Improve stability (Handle errors gracefully, expand test coverage, etc.)
• Increase test coverage
• Save the most recent search results to a database
• Randomize Tor Connection (Random Header and Identity)
• Keyword/Phrase Search
• Social Media Integration
• Increase anonymity
• Screenshot capture

Contribution Guidelines

Found an issue?

If you face any issues in the project, please let us know by creating a new issue here.

Developer Guidelines

We welcome contributions to this project! Here are a few guidelines to follow:

1. Fork the repository and create a new branch for your contribution.
2. Make sure your code passes all tests by running pytest before submitting a pull request to dev branch.
3. Follow the PEP8 style guide for Python code.
4. Make sure to add appropriate documentation for any new features or changes.
5. When submitting a pull request, please provide a detailed description of the changes made.

References

1.  M. Glassman and M. J. Kang, “Intelligence in the internet age: The emergence and evolution of Open Source Intelligence (OSINT),” Comput. Human Behav., vol. 28, no. 2, pp. 673–682, 2012.
2.  D. Bradbury, “In plain view: open source intelligence,” Comput. Fraud Secur., vol. 2011, no. 4, pp. 5–9, 2011.
3.  B. Butler, B. Wardman, and N. Pratt, “REAPER: an automated, scalable solution for mass credential harvesting and OSINT,” 2016 APWG Symp. Electron. Crime Res., pp. 1–10, 2016.
4.  B. Zantout and R. A. Haraty, “I2P Data Communication System I2P Data Communication System,” no. April 2002, 2014.
5.  J. Qin, Y. Zhou, G. Lai, E. Reid, M. Sageman, and H. Chen, “The dark web portal project: collecting and analyzing the presence of terrorist groups on the web,” in Proceedings of the 2005 IEEE international conference on Intelligence and Security Informatics, 2005, pp. 623–624.
6.  D. Moore, T. Rid, D. Moore, and T. Rid, “Cryptopolitik and the Darknet Cryptopolitik and the Darknet,” vol. 6338, 2016.
7.  G. Weimann, “Going dark: Terrorism on the dark Web,” Stud. Confl. Terror., vol. 39, no. 3, pp. 195–206, 2016.
8.  A. T. Zulkarnine, R. Fran