Python Package Comparison for Webcrawlers and Website Mirroring, August 2026
Posted by Al Sweigart in misc
I wanted to get a local mirror copy of a website, but there's a lot of Python packages (as of August 2026) to do this. I decided to test 21 of them. This blog post is a human-written report of my experience using each one of these packages (as of August 2026). After reviewing them, I recommend website-downloader because it's a simple tool that does a simple task simply.
Making this blog post was a fun exercise in taking a broad look at various packages on PyPI. I found some serious, long-standing projects that I haven't heard of before and also various vibe coding experiments. It was also good exercise for skimming source code for signs of malware and directing AI to do the same.
Note: The WARC "web archive" file format is an archive file format (like zip files) with .warc file extension that is used by the Internet Archive for storing websites that have been crawled (and recrawled). The best WARC file viewer is ReplayWeb.page. (I hadn't heard of this file format before I started this blog post. It doesn't really come up unless you're creating archives of websites.)
website-downloader 2.6.1
pip install website-downloader (PyPI, MIT license, by Harsimran Sidhu)
The description reads, "Turn any website you're authorized to copy into a fast, browsable offline mirror — with one command." A modern, hackable alternative to wget --mirror and HTTrack — built in pure Python, without dragging in a heavy crawler framework.
There are only two releases for this package: 2.6.0 on July 6, 2026 and 2.6.1 on July 8, 2026. That and the abundant use of emoji in the README makes it almost certain that this was AI-generated. That's not a disqualifier in my book, but the hsidhu account behind website-downloader had no other posted projects and was also created on July 6, 2026. I did a cursory examination and also had Opus 4.8 examine the source for malware and didn't find anything.
The source code has next to no comments in it. Running website-downloader --help brings up some interesting choices: the default of max-pages is 50 and by default the crawler ignores robot.txt files. The package uses Playwrite to render JavaScript to find additional links, and by default it is not run in headless mode. It also does not show a progress bar by default.
Nevertheless, when I ran it, it worked great. It converted the downloaded links to work offline. I give it a passing grade.
pywebcopy 7.1
pip install pywebcopy (PyPI, by Raja Tomar)
The description reads, "PyWebCopy is a free tool for copying full or partial websites locally onto your hard-disk for offline viewing."
First released in 2018 and the most recent release is May 13, 2025 by Raja Tomar who has no other PyPI projects.
The readme also says, "What can PyWebCopy not do? PyWebCopy does not include a virtual DOM or any form of JavaScript parsing. If a website makes heavy use of JavaScript to operate, it is unlikely PyWebCopy will be able to make a true copy if it is unable to discover all of the website due to JavaScript being used to dynamically generate links."
It's not AI-generated, because the --quiet option was misspelled as --quite.
The downloaded files seem to work, though you have to supply the -s, --url, and --location arguments and these aren't marked as required.
archivebox 0.7.4
pip install archivebox (PyPI)
The archivebox package looks like a quite heavy application for serious web archiving efforts by librarians, lawyers, journalists, and research institutions. The project's website is https://archivebox.io/. It's a bit much for my needs, though I'm glad I at least know of the project.
webcloner
I skipped this. AI-slop. Emoji in the readme and only a single day of releases in July 2025 makes me think it's someone's vibe coding experiment.
web-snapshot-cli
I skipped this. AI-slop. Generic readme and a couple days of releases in July 2026 and the dev's social media makes me think this is just vibe coding experiments.
pagesnap
I don't recommend this. It's an interesting concept: it saves all of the images and media on a web page as embedded data URLs so you have a single .html file. Some warning signs were that it's really a single pagesnap.py file that was incorrectly packaged for PyPI and only has a couple releases back in May 2023 (so it predates AI). But when I ran it, it simply didn't work. I assume the project is abandoned. The dev's profile avatar is an anime image. The printed output is in Chinese, and there's something nice that people outside of the Anglosphere use Python and PyPI.
webmirror
I skipped this. AI-slop. A single release in May 2025 and a ton of emoji in the README makes me think this is AI-generated. The dev's profile pic is anime and the GitHub link and account are 404s. The entire source code is under 200 lines of code. I skimmed the source but haven't bothered to run it; I assume it's broken.
reap-web-harvester
Skipped. AI-slop. Single release in June 2026 and emoji in the README. While skimming the code, I found it requires an API token for Netlify, an AI-coding service. There's no reason a basic script like this needs to access an AI service. The image "optimizer" just uses Pillow to save images as a .webp file at 80 quality level.
brozzler 1.9.6
pip install brozzler (PyPI)
Advertised as “browser” | “crawler” = “brozzler”. The project goes back to 2016 and is affiliated with folks from the Internet Archive. It's a distributed crawler, and I assume it's what the Internet Archive uses. Looks like a heavy project (and above what my needs are), but I assume it works well for serious web archiving work. I skipped this.
pywb
pip install pywb (PyPI)
Web archiving tool that goes back to 2014. "This toolset forms the foundation of Webrecorder project, but also provides a generic web archiving toolkit that is used by other web archives, including the traditional “Wayback Machine” functionality." Again, looks like a heavy project above my needs, but I assume it works well. I skipped this. The developer, Ilya Kreymer, is the founder of Webrecorder and has an extensive history and maintains other projects on PyPI.
pywaybackup 4.2.3
pip install pywaybackup (PyPI)
Releases going back to 2024. This is a tool specifically for downloading archived web pages from the Wayback Machine (run by the Internet Archive). Interesting idea, but it's not the generic website mirroring tool that I need. I skipped this.
cdx-toolkit 0.9.39
pip install cdx-toolkit (PyPI)
From the Common Crawl Foundation, which in the last few years has been a resource for LLM training data. I'll just repost the project description: "cdx_toolkit is a set of tools for working with CDX indices of web crawls and archives, including those at the Common Crawl Foundation (CCF) and those at the Internet Archive's Wayback Machine. Common Crawl uses Ilya Kreymer's pywb to serve the CDX API, which is somewhat different from the Internet Archive's CDX API server. cdx_toolkit hides these differences as best it can. cdx_toolkit also knits together the monthly Common Crawl CDX indices into a single, virtual index."
Not what I'm looking for, but an interesting project.
Scrapy 2.17.0
pip install Scrapy (PyPI)
The venerable web crawler. One of the originals. I never did learn to use it, but I'm sure it has tons of features. I'm looking for something a bit more basic, but definitely look at this project if you're doing serious web crawling work.
scrapy-playwright 0.0.48
pip install scrapy-playwright (PyPI)
This project has releases going back to December 2020. It looks like some kind of plugin for Playwright for Scrapy. Looks well managed.
crawlee 1.9.0
pip install crawlee (PyPI, by Apify)
A project from Apify ("Apify is the largest marketplace of trusted tools for AI.") The first release was January 2024. The project description reads: "Crawlee covers your crawling and scraping end-to-end and helps you build reliable scrapers. Fast. Your crawlers will appear almost human-like and fly under the radar of modern bot protections even with the default configuration. Crawlee gives you the tools to crawl the web for links, scrape data and persistently store it in machine-readable formats, without having to worry about the technical details. And thanks to rich configuration options, you can tweak almost any aspect of Crawlee to suit your project's needs if the default settings don't cut it."
So I assume it's something you want to use to get around anti-bot scrapers. I skipped it, but it looks like a serious project.
wpull 2.0.1
pip install wpull (PyPI)
An older project with releases from 2014 to 2016, the description reads "Wpull is a Wget-compatible (or remake/clone/replacement/alternative) web downloader and crawler." Interesting if you need a pure Python alternative to wget, but I can just use wget.
