How to Install LibreCrawl: The Free Website Crawler That Lists Every SEO Problem

Checking a website one page at a time is like counting sand with tweezers. You'll be done sometime next year, and you'll still miss the page with no title.

A crawler does it for you. LibreCrawl is a free one. You give it a web address, it visits every page it can find, and it lists what's broken. Think of it as a free version of the big paid crawlers.

We installed it on Nov 2, 2026 and pointed it at a live Framer site. It crawled 40 pages and checked 142 outside links in about 45 seconds, and it found 89 issues. Here's the whole setup.

Quick words (one line each):

• Crawler: a program that clicks through every link on a site, like a very fast, very nosy visitor.

• H1: the main heading of a page. Every page gets exactly one.

• Meta description: the short blurb under your page title in Google results.

• Structured data (schema): hidden code that tells Google and AI what the page is about (business, product, article).

• Docker: a free app that runs a tool in its own sealed box so it doesn't mess with your computer.

• CSV: a plain spreadsheet file that opens in Excel or Google Sheets.

Step 1: Do a 60-second safety check

Never install a GitHub tool blind. Here's what LibreCrawl looked like when we checked it on Nov 2, 2026:

1. Licence: MIT. Free to use, change, and use for client work.

2. Last commit: Nov 1, 2026. Very active.

3. Stars: 1,091 (as of Nov 2, 2026).

4. Account or key: none. It runs on your own computer.

5. Code read: we read the code before running it. No hidden commands. The optional extras (a Zoho login and email) stay off unless you turn them on.

6. Its own tests: 60 out of 60 passed on our test machine.

Step 2: Install it (pick one way)

The easy way, with Docker:

1. Install Docker Desktop.

2. Download the code: git clone https://github.com/PhialsBasement/LibreCrawl.git

3. Go into the folder and run docker compose up

4. Open http://localhost:5000 in your browser.

The no-Docker way, with Python 3:

1. Download the code the same way.

2. Make a virtual environment (a private folder of Python parts for this one tool): python -m venv venv

3. Install the parts: pip install -r requirements.txt

4. Start it: python main.py --local

5. Open http://localhost:5000.

Step 3: Run your first crawl

1. Type the site's address into LibreCrawl.

2. Set a crawl delay. We used 1 second between pages. Be polite. Nobody likes the guy who hammers their server.

3. Leave JavaScript rendering off for the first run. It's optional (it uses Playwright, a robot browser) and the plain crawl is faster.

4. Hit start and grab a coffee. A small site is done before the coffee is.

Step 4: Read what it found

Here's what our Nov 2, 2026 crawl of a 40-page Framer site turned up:

1. No structured data on all 40 pages. Not one.

2. Meta description too long on 13 pages. Google cuts those off mid-sentence.

3. Title too short on 10 pages. Wasted space in search results.

4. No H1 on the homepage. The most important page had no main heading.

5. 7 pages still had the old template name in the title or web address. Template leftovers are sneaky.

6. One outside link answered 403 (the site was blocking crawlers, not broken).

That's a ready-made to-do list in under a minute.

Step 5: Export and fix page by page

1. Download the results as CSV, Excel, JSON or XML.

2. Open the CSV in a spreadsheet and sort by issue type.

3. Fix the homepage first: one H1 that says what the business does.

4. Then rewrite the long descriptions and short titles, one row at a time.

5. Remove or rename leftover template pages.

6. Add basic structured data for the business.

Step 6: Crawl again and show the before and after

1. Run the same crawl after your fixes.

2. Put the two issue counts side by side.

3. Hand that to the client. A before-and-after issue count beats any pie chart.

Step 7: Close it when you're done

LibreCrawl starts a small web server that listens on all network connections (it's fixed that way in the code), and it tries to open a browser when it starts. So:

1. Run it only on a private computer you chose on purpose.

2. Stop it (Ctrl+C, or docker compose down) when the crawl is finished.

Why this matters for AI search too

Exposure Ninja's Charlie Marchant and Jenny Nicholson (Oct 28, 2026) say the way a site is built is part of showing up in AI answers. Missing headings, dead links and confusing pages waste the work you put into getting mentioned. A crawler finds those fast.

What it costs

Nothing. MIT licence, runs on your computer, no paid key. The project's website has an optional donate link. Optional means optional.

Do this today

1. Install LibreCrawl with docker compose up.

2. Crawl your site with a 1-second delay.

3. Fix the homepage H1 first.

4. Export the CSV and fix titles and descriptions row by row.

5. Stop the app when you're done.

Related reading

• Install SEOmator: a free SEO audit with a score and a fix list

• Install siteprobe: check every page in your sitemap

• The 14-point on-page SEO checklist

Sources

• GitHub: PhialsBasement/LibreCrawl

• Exposure Ninja - Website Architecture: The Overlooked Side of AI Search (2026-10-28)

Don't worry if this seems like it's a pain in the ass for someone who doesn't build websites. That's because it is. The good news: just click the link below, pull the trigger, and we'll take care of the rest.