Python Script to Bulk Audit Broken Internal Links and 301 Redirect Chains from an XML Sitemap
You crawl a sitemap, follow every internal link, and write a CSV of anything that returns a 404, a 5xx, or a redirect chain longer than one hop. The script below does exactly that with requests and lxml, no crawler framework required. It reads your sitemap.xml, extracts every <loc>, fetches each URL with redirects disabled so you can inspect the chain hop by hop, then parses the HTML for internal <a href> targets and audits those too.
The one-liner that gets you started:
pip install requests lxml tqdm && python link_audit.py --sitemap https://example.com/sitemap.xml --out audit.csv
Known to work on Python 3.11 and 3.12 with requests==2.32.x and lxml==5.x. Nothing else is needed.
What the audit actually checks, and why redirect chains are the expensive part
A redirect chain is any URL that resolves through two or more 3xx responses before reaching a 200. Google’s own documentation on redirects states that each hop in a chain adds latency and that chains should be avoided in favor of a single redirect to the final destination. That is the rule the script enforces: one hop is a warning, two or more is a failure.
Broken internal links are simpler — any internal href that returns 404 or 410 is a hard failure. The script reports both, plus 5xx responses, because a 503 on a product page is worse than a 404 and gets missed by most tools that only look for “not found”.
| Finding | Typical cause | What the script reports | Fix |
|---|---|---|---|
| 404 / 410 on internal link | Page deleted, slug changed, typo in template | BROKEN with source page |
Update the link or add a 301 to the live target |
| Single 301 | Legitimate permanent move | REDIRECT_1 (info only) |
Update internal links to point at the destination |
| 301 → 301 → 200 | Old redirects chained by CMS migrations | CHAIN with hop count |
Collapse to one 301 pointing at the final URL |
| 301 → 302 → 200 | Mixed permanent/temporary rules | CHAIN with status list |
Pick one status; do not mix |
| Redirect loop | A → B → A in rewrite rules | LOOP after 10 hops |
Break the cycle in the server config |
| 5xx on internal link | Upstream timeout, bad deploy | SERVER_ERROR |
Check application logs, not the link |
The full script: sitemap parsing, redirect tracing, and internal link extraction
This script parses the sitemap (including nested sitemap indexes), fetches each URL with allow_redirects=False, walks the Location header manually up to a hard cap, and then extracts internal links from the final HTML. Save it as link_audit.py.
import argparse
import csv
import sys
from urllib.parse import urljoin, urlparse
import requests
from lxml import etree
from tqdm import tqdm
MAX_HOPS = 10
TIMEOUT = 15
HEADERS = {"User-Agent": "FreeFile-LinkAudit/1.0 (+https://freefile.store)"}
NS = {"sm": "http://www.sitemaps.org/schemas/sitemap/0.9"}
def fetch_sitemap_urls(sitemap_url, session, depth=0):
"""Return every <loc> from a sitemap, recursing into sitemap indexes."""
if depth > 3:
return []
r = session.get(sitemap_url, timeout=TIMEOUT, headers=HEADERS)
r.raise_for_status()
root = etree.fromstring(r.content)
if root.tag.endswith("sitemapindex"):
urls = []
for loc in root.findall(".//sm:sitemap/sm:loc", NS):
urls.extend(fetch_sitemap_urls(loc.text.strip(), session, depth + 1))
return urls
return [loc.text.strip() for loc in root.findall(".//sm:url/sm:loc", NS)
if loc.text]
def trace_redirects(url, session):
"""Follow redirects manually. Returns (status, final_url, chain, error)."""
chain = []
current = url
for _ in range(MAX_HOPS):
try:
resp = session.get(current, timeout=TIMEOUT, headers=HEADERS,
allow_redirects=False, stream=True)
except requests.RequestException as exc:
return None, current, chain, str(exc)
chain.append({"url": current, "status": resp.status_code})
if resp.status_code in (301, 302, 303, 307, 308):
location = resp.headers.get("Location")
if not location:
return resp.status_code, current, chain, "3xx without Location"
current = urljoin(current, location)
continue
return resp.status_code, current, chain, None
return None, current, chain, "LOOP: exceeded max hops"
def extract_internal_links(html_bytes, base_url, netloc):
"""Yield absolute internal hrefs found in the page body."""
try:
tree = etree.HTML(html_bytes)
except etree.ParserError:
return
if tree is None:
return
for a in tree.iter("a"):
href = a.get("href")
if not href or href.startswith(("mailto:", "tel:", "javascript:", "#")):
continue
absolute = urljoin(base_url, href)
if urlparse(absolute).netloc == netloc:
yield absolute.split("#")[0]
def classify(chain, final_status, error):
if error and error.startswith("LOOP"):
return "LOOP"
if error:
return "ERROR"
if final_status and final_status >= 500:
return "SERVER_ERROR"
if final_status in (404, 410):
return "BROKEN"
hops = [c for c in chain[:-1] if c["status"] in (301, 302, 303, 307, 308)]
if len(hops) >= 2:
return "CHAIN"
if len(hops) == 1:
return "REDIRECT_1"
return "OK"
def main():
ap = argparse.ArgumentParser()
ap.add_argument("--sitemap", required=True)
ap.add_argument("--out", default="audit.csv")
ap.add_argument("--max-pages", type=int, default=500)
args = ap.parse_args()
session = requests.Session()
netloc = urlparse(args.sitemap).netloc
sitemap_urls = fetch_sitemap_urls(args.sitemap, session)
print(f"Found {len(sitemap_urls)} URLs in sitemap", file=sys.stderr)
rows = []
seen = set()
for page_url in tqdm(sitemap_urls[:args.max_pages], desc="pages"):
status, final_url, chain, error = trace_redirects(page_url, session)
rows.append({
"source": page_url, "target": page_url,
"final": final_url, "status": status,
"hops": len(chain) - 1, "verdict": classify(chain, status, error),
"chain": " -> ".join(f'{c["status"]} {c["url"]}' for c in chain),
"error": error or "",
})
if status != 200:
continue
try:
body = session.get(final_url, timeout=TIMEOUT,
headers=HEADERS).content
except requests.RequestException:
continue
for link in extract_internal_links(body, final_url, netloc):
if link in seen:
continue
seen.add(link)
s, f, ch, err = trace_redirects(link, session)
rows.append({
"source": page_url, "target": link,
"final": f, "status": s, "hops": len(ch) - 1,
"verdict": classify(ch, s, err),
"chain": " -> ".join(f'{c["status"]} {c["url"]}' for c in ch),
"error": err or "",
})
with open(args.out, "w", newline="", encoding="utf-8") as fh:
writer = csv.DictWriter(fh, fieldnames=rows[0].keys())
writer.writeheader()
writer.writerows(rows)
bad = [r for r in rows if r["verdict"] not in ("OK", "REDIRECT_1")]
print(f"Wrote {len(rows)} rows, {len(bad)} problems to {args.out}",
file=sys.stderr)
if __name__ == "__main__":
main()
Why you must disable automatic redirect following to audit redirect chains
If you let requests follow redirects, you get the final status code and nothing else — the intermediate hops are invisible. The allow_redirects=False flag forces the library to return the first 3xx response so the script can read the Location header itself and record every hop. This is the failure mode that costs people hours: a page that returns 200 looks healthy in every report, while it silently costs three round trips on mobile.
Do not do this:
resp = session.get(url) # redirects followed, chain lost
print(resp.status_code) # 200 — tells you nothing about hops
Do this instead:
resp = session.get(url, allow_redirects=False)
print(resp.status_code, resp.headers.get("Location"))
How to handle sitemap indexes, gzip files, and the 50,000 URL limit
The sitemap protocol caps a single sitemap at 50,000 URLs and 50 MB uncompressed, per the sitemaps.org specification, which is why production sites almost always use a <sitemapindex> that points at child sitemaps. The script handles this by checking the root element tag — sitemapindex triggers recursion into each child, with a depth cap of 3 to prevent runaway loops.
If your sitemaps are gzipped (sitemap.xml.gz), requests decompresses Content-Encoding: gzip automatically, so no change is needed. If the file is served as application/gzip with a .gz extension and no content encoding, decompress manually:
import gzip
raw = session.get(sitemap_url, headers=HEADERS).content
xml_bytes = gzip.decompress(raw) if sitemap_url.endswith(".gz") else raw
root = etree.fromstring(xml_bytes)
How to filter the CSV down to the redirect chains worth fixing first
Sort by hop count descending and fix the longest chains first, because each removed hop is a measurable latency win on every page that links to it. A quick shell filter gets you the worst offenders:
python -c "
import csv
rows = list(csv.DictReader(open('audit.csv')))
bad = [r for r in rows if r['verdict'] in ('CHAIN','LOOP','BROKEN','SERVER_ERROR')]
bad.sort(key=lambda r: int(r['hops']), reverse=True)
for r in bad[:25]:
print(r['verdict'], r['hops'], r['target'], '->', r['final'])
"
For chains, the fix is almost always the same: collapse the intermediate redirect into a single 301 that points at the final URL, then update the internal links that reference the original. If you are on nginx, that means replacing stacked rewrite rules with a single return 301 to the canonical destination.
What concurrency settings keep a 5,000-page audit from getting rate-limited
Keep the default sequential loop for anything under about 1,000 URLs. Above that, thread it — but cap concurrency at 5 to 10 workers and add a small delay, because most origins will start returning 429 or 503 somewhere around 20 concurrent connections. The requests library does not rate-limit for you, and a 503 caused by your own crawler will show up in the CSV as a real SERVER_ERROR, poisoning the report.
If you do add threading, use concurrent.futures.ThreadPoolExecutor(max_workers=8) and keep one Session per thread — sharing a single session across threads is not guaranteed thread-safe in requests.
The failure mode to watch for: relative URLs and trailing-slash mismatches
The most common false positive in this kind of audit is a link that looks broken but is not, because the href was relative and got joined against the wrong base. Always join against the final URL after redirects (final_url), not the sitemap URL, or every link on a redirected page will resolve to the wrong path. The second false positive is the trailing-slash flip — /pricing and /pricing/ are different URLs to the server, and one of them will 301 to the other. That is a real finding, not a bug in the script: it means your internal links are inconsistent, and you should pick one canonical form and enforce it in the template.
The bottom line: run the audit weekly from cron, diff the CSV against last week’s, and treat any new CHAIN or BROKEN row as a deploy regression. Redirect chains accumulate silently after every CMS migration — a scheduled diff is the only thing that catches them before they cost you crawl budget.
Related reading: Best Free Monitoring Tools for Linux Servers.
If this helped, this next one will too: Automate Your Workflow with Python Scripts.