SEO for a self-hosted or static site is mostly not about tricks. It is about making pages easy to crawl, easy to understand, fast to load, and worth linking to.
This page maps the layers that matter, the open-source tools covered on FOSS Engineer that help with each one, and the checklist I actually applied to this Hugo blog.
The Short Version
| Layer | Question It Answers | Tools / Techniques |
|---|---|---|
| Crawlability | Can search engines find every page? | robots.txt, sitemap.xml, canonical URLs |
| Link health | Do any internal or external links break? | Lychee, LinkChecker |
| Public surface | What does my domain reveal: TLS, headers, DNS, social tags? | Web-Check |
| Structured data | Does Google understand what each page is? | JSON-LD: BlogPosting, BreadcrumbList, FAQPage, SoftwareApplication |
| On-page | Does each post clearly target a topic? | One H1, keyword early, internal and external links, FAQ |
| Performance | Is the page fast and stable? | Lighthouse, WebP covers, lazy loading, preloading |
| Indexing | Are pages actually indexed? | Google Search Console, Bing Webmaster Tools, IndexNow |
| Off-page | Does anyone link to it? | Awesome lists, communities, canonical cross-posts |
How They Fit Together
hugo --minify
-> Lychee --offline on ./public/ (broken internal links and assets, pre-deploy, seconds)
-> deploy (Cloudflare Pages, etc.)
-> Lychee on sitemap URLs (weekly) (external links rot over time)
-> LinkChecker on /sitemap.xml (periodic recursive crawl with an HTML report)
-> Web-Check on the domain (TLS, headers, DNS, robots, sitemap, social tags)
-> Lighthouse on top pages (Core Web Vitals)
-> Search Console / Bing / IndexNow (indexing and coverage)
-> refresh, interlink, repeat
The first two steps can run in CI. The rest are periodic checks, ideally after big template changes.
LinkChecker
LinkChecker is a GPL Python CLI that recursively checks links in local HTML, static-site output, sitemaps, and live websites. It respects robots.txt, checks anchors, and writes text, HTML, CSV, or other report formats.
Use LinkChecker when you want:
- broken-link checks on
./public/before deploying - a full crawl from
sitemap.xmlafter deploying - a deterministic CI step with exit codes and report files
- a Docker runner instead of a desktop crawler
| Target | Best For |
|---|---|
./public/ |
Pre-deploy checks of generated static output |
https://example.com/sitemap.xml |
Broad crawl of every published page |
https://example.com |
User-path crawl from the homepage and navigation |
hugo --minify
linkchecker --output=text/utf-8 ./public/
For this blog, the sitemap crawl is the most useful: it covers posts, docs, and app pages that are not all linked from the homepage. Start by collecting reports without failing the build, then make CI stricter once ignore rules are tuned.
Useful links:
Lychee
Lychee
is a fast, asynchronous link checker written in Rust (MIT/Apache-2.0), with an official GitHub Action
and Docker image. It checks the links found in the inputs you give it: HTML or Markdown files, folders, globs, URLs, or a list of URLs via --files-from.
Use Lychee when you want:
- a pre-deploy check that finishes in seconds, even on large sites
- a low-maintenance CI step with Markdown, JSON, or JUnit reports
- a request cache (
--cache) and a.lycheeignorefile for known false positives - GitHub token support to avoid rate limits on GitHub links
The most useful mode for a Hugo site is offline, against the generated output:
hugo --minify
docker run --rm -v "$PWD/public:/site:ro" lycheeverse/lychee:0.24.2 \
--offline --no-progress --root-dir /site --index-files index.html \
--remap 'https://fossengineer.com/(.*) file:///site/$1' \
'/site/**/*.html'
--root-dir resolves links like /docs/, and --remap turns absolute links to your own domain into local files, so they are checked too instead of skipped. On this blog the whole build (75,000+ links) is checked in about 4 seconds. The first run surfaced around 85 unique broken targets: posts scheduled for the future but already linked from live pages, plus old URLs that no longer exist.
For external links, run a weekly check over the sitemap URLs (--files-from urls.txt) with --cache and a couple of retries, since external sites rot over time and occasionally time out.
Lychee does not crawl recursively; that is why the sitemap list or the build folder is the input.
Lychee vs. LinkChecker
| Lychee | LinkChecker | |
|---|---|---|
| Speed | Very fast (Rust, async) | Slower (Python) |
| Crawling | No; checks the inputs you list | Yes; recursive, respects robots.txt |
| Best input | Build folder, sitemap URL list | Live site or sitemap |
| Reports | Compact, detailed, Markdown, JSON, JUnit | Text, HTML, CSV, and more |
| CI fit | Official GitHub Action, single binary | Docker image or pip |
| Best job | Fast pre-deploy gate and weekly external check | Periodic deep crawl with a readable report |
They combine well: Lychee as the fast gate on every build, LinkChecker as the occasional deep crawl.
Web-Check
Web-Check is an MIT-licensed, on-demand website intelligence tool. Point it at a domain and it inspects security, DNS, server, email, SEO, performance, and privacy signals in one dashboard.
Its SEO-relevant checks include robots.txt, sitemap, redirects, linked pages, social tags, and archive history, alongside TLS, HSTS, security headers, DNSSEC, and Lighthouse-style performance checks.
Use Web-Check when you want:
- a one-page public-surface review of your domain
- to confirm robots, sitemap, and social tags are served correctly
- to check TLS and security headers after moving behind a CDN
- a JSON report via its aggregate API
Running it against fossengineer.com locally confirmed valid TLS, Cloudflare in front of the site, and security headers such as content-security-policy, strict-transport-security, and x-frame-options.
Web-Check makes outbound requests to whatever target a user enters. Do not expose it publicly without authentication, rate limiting, and blocked hosts.
Useful links:
LinkChecker vs. Web-Check: LinkChecker crawls every page looking for broken links. Web-Check inspects one domain’s public configuration. They complement each other.
Technical SEO Checklist
These are template-level fixes. Do them once and every page benefits. All of these are in place on this blog:
robots.txtpointing to the sitemap- auto-generated
sitemap.xmlwith tuned priorities <link rel="canonical">on every pagemeta descriptionfrom frontmatter- Open Graph tags (
og:title,og:description,og:image,og:site_name,article:published_time,article:modified_time,article:tag) with a defaultog:imagefallback - full Twitter card tags (
summary_large_image) - JSON-LD
BlogPostingandBreadcrumbListon posts,WebSitewithSearchActionon the homepage - JSON-LD
FAQPagefrom FAQ sections,SoftwareApplicationon app pages, Author/Organization schema enableGitInfo: truesodateModifiedreflects real edits instead of build time- RSS feed
In Hugo, most of this lives in the head partial and a schema partial. Validate the output with Google’s Rich Results Test after changes.
On-Page Checklist
These are per-post content fixes:
- Exactly one H1. The title is the H1; the body starts at
##. Stray#headings in Markdown create duplicate H1s. - Primary keyword in the first 100 words, matching the title and URL slug.
- 3–5 internal links to related posts, so topic clusters are obvious.
- 2–3 authoritative external links: official docs, the GitHub repo, the license.
- 4–8 FAQ questions under a
## FAQsection to capture “people also ask” queries. - Keyword clusters: cover the related terms people search for, not just one phrase.
- Promote buried comparisons. An “X vs Y” hidden in one FAQ answer deserves its own section with a table.
When Search Console reported high-value posts as “Crawled – currently not indexed”, applying this checklist (fixing duplicate H1s, adding FAQs, and adding internal links) was the main recovery tool.
Performance and Core Web Vitals
- Self-host cover images. External image services fetched at render time are a common LCP bottleneck.
- Process images to WebP with responsive sizes via Hugo image processing.
- Lazy-load below-the-fold images and preload the hero image.
- Set image dimensions to avoid layout shift (CLS).
- Run Lighthouse on your top pages after each template change.
Hugo can only resize raster images. Guard image processing with reflect.IsImageResourceProcessable so an SVG cover does not break the build.
Indexing
- Google Search Console: submit the sitemap, watch the coverage report, and request reindexing of improved posts (the daily quota is small).
- Bing Webmaster Tools: worth the setup, since DuckDuckGo and other engines draw on Bing’s index.
- IndexNow: ping Bing and Yandex on every publish for near-instant discovery. A Cloudflare Pages deploy hook can do it.
- Publish at a steady cadence. Scheduled future-dated posts plus CI/CD are better than sudden bursts.
Off-Page
- Submit projects and guides to curated lists like awesome-selfhosted and selfh.st.
- Cross-post to dev.to or Hashnode with a canonical link back to the original.
- Answer questions in communities like r/selfhosted and r/homelab, linking only when it genuinely helps.
Choosing a Starting Point
| If You Want To… | Start With |
|---|---|
| Stop shipping broken links | Lychee --offline on ./public/ in CI |
| Catch external links that rotted | Weekly Lychee run over sitemap URLs |
| Audit every published page with a report | LinkChecker on sitemap.xml |
| Check what your domain exposes | Web-Check |
| Get rich results in Google | JSON-LD schema + Rich Results Test |
| Fix “crawled, not indexed” posts | On-page checklist, then reindex requests |
| Improve speed scores | Lighthouse + self-hosted WebP covers |
| Get indexed faster | Search Console + Bing + IndexNow |
Related Guides
- Create a Free Blog with Hugo to build the site in the first place
- Privacy Web Analytics to measure traffic without Google Analytics
- DNS and Domain Ops for the records Web-Check inspects
- Email and SMTP Deliverability for SPF, DKIM, and DMARC
- Homelab Security Hardening for running Web-Check and other tools safely
Comments