If You Love a Website, Back It Up: A Practical Guide to Archiving Web Content

The importance of backing up web content is more relevant than ever. With countless websites disappearing or changing their policies, knowing how to archive your favorite pages can save you from losing valuable information.

Why Archive Web Content?

Many of us have experienced the frustration of finding a link that once led to great content now returns a 404 error. This can happen due to expired domains, platform shutdowns, or changes in website policies. Simply bookmarking a page is not enough; links can break, and content can vanish. Archiving ensures that your favorite web content is preserved for future reference.

Tools for Archiving Web Content

There are several tools available for archiving web content. Here are three popular options:

1. monolith: This command-line tool saves web pages as single HTML files, including all necessary CSS, images, and JavaScript. This makes it easy to view the archived page offline without needing an internet connection.

2. wget: A powerful command-line utility that can download files from the web. It can mirror entire websites, making it suitable for backing up static sites.

3. ArchiveBox: An open-source, self-hosted tool that archives web content in various formats. It can save entire websites, individual pages, or collections of links and offers a user-friendly interface for searching through your archives.

Choosing the Right Tool

Before you start archiving, it's essential to determine what you want to save. Here’s a quick guide:

• Single Page: Use monolith or the SingleFile browser extension to save an entire webpage as a standalone HTML file.
• Entire Static Site: Use wget --mirror to download the whole site. Including the WARC (Web ARChive) format is recommended for long-term preservation.
• Long-Term Collection with Search Capability: Set up a self-hosted ArchiveBox instance to create a searchable archive of web content.

How to Use wget to Mirror a Website

To mirror a website using wget, follow these steps:

Step 1: Install wget

If you don’t have wget installed, you can easily add it. On macOS, you can use a package manager to install it.

Step 2: Create a Backup Script

You can create a simple script to automate the backup process. Here’s an example script:

#!/usr/bin/env bash
# mirror-site.sh - backup một site tĩnh, kèm file WARC
set -euo pipefail
URL="$1"
NAME=$(echo "$URL" | sed -E 's#https?://##; s#/.*##')
DATE=$(date +%Y%m%d)
OUT="$HOME/web-archive/$NAME/$DATE"
mkdir -p "$OUT"
cd "$OUT"
wget --mirror --convert-links --adjust-extension --page-requisites --no-parent --wait=1 --random-wait --limit-rate=500k --user-agent='Mozilla/5.0 (personal-archive)' --warc-file="$NAME-$DATE" --warc-cdx "$URL"
echo "Xong: $OUT"
du -sh "$OUT"

Step 3: Run the Script

Run the script by passing the URL of the website you want to archive:

bash mirror-site.sh https://example.com

Archiving Individual Pages with monolith

If you want to archive individual pages, especially from a blog with a sitemap, you can use monolith. Here’s how:

Step 1: Install monolith

Make sure you have monolith installed. You can find installation instructions on its official website.

Step 2: Create an Archiving Script

Here’s a Python script to archive pages listed in a sitemap:

#!/usr/bin/env python3
# archive_sitemap.py - Python 3.10+, cần: pip install requests
import hashlib
import subprocess
import sys
import time
import xml.etree.ElementTree as ET
from pathlib import Path
import requests

NS = {"sm": "http://www.sitemaps.org/schemas/sitemap/0.9"}
OUT_DIR = Path.home() / "web-archive" / "pages"

def get_urls(sitemap_url: str) -> list[str]:
    resp = requests.get(sitemap_url, timeout=30)
    resp.raise_for_status()
    root = ET.fromstring(resp.content)
    if root.tag.endswith("sitemapindex"):
        urls = []
        for loc in root.findall("sm:sitemap/sm:loc", NS):
            urls.extend(get_urls(loc.text.strip()))
        return urls
    return [loc.text.strip() for loc in root.findall("sm:url/sm:loc", NS)]

def archive(url: str) -> None:
    key = hashlib.sha1(url.encode()).hexdigest()[:12]
    target = OUT_DIR / f"{key}.html"
    if target.exists():
        print(f"[skip] {url}")
        return
    result = subprocess.run(["monolith", url, "-j", "-I", "-o", str(target)], capture_output=True, text=True, timeout=120)
    status = "ok" if result.returncode == 0 else "FAIL"
    print(f"[{status}] {url} - {target.name}")
    with open(OUT_DIR / "index.tsv", "a") as f:
        f.write(f"{key}\t{status}\t{url}\n")

if __name__ == "__main__":
    OUT_DIR.mkdir(parents=True, exist_ok=True)
    for u in get_urls(sys.argv[1]):
        archive(u)
        time.sleep(2)

Step 3: Run the Script

To run the script, use the command:

python3 archive_sitemap.py https://example.com/sitemap.xml

Automating with ArchiveBox

For larger collections of links, ArchiveBox can be very helpful. Here’s how to set it up:

Step 1: Install ArchiveBox

You can set up ArchiveBox using Docker. Create a directory for it and run:

mkdir -p ~/archivebox/data
cd ~/archivebox
curl -fsSL 'https://docker-compose.archivebox.io' -o docker-compose.yml
docker compose run archivebox init --setup
docker compose up -d

Step 2: Add Links

You can add individual links or a file of links:

docker compose run archivebox add 'https://example.com/article'
docker compose run archivebox add ~/links.txt

Step 3: Schedule Backups

You can automate the archiving process using a cron job. For example, to run a backup every Sunday at 3 AM:

0 3 * * 0 /home/user/bin/mirror-site.sh https://example.com/blog/ /var/log/mirror.log 2>&1

Conclusion

Archiving web content is essential to preserve valuable information from disappearing. By using tools like wget, monolith, and ArchiveBox, you can effectively back up your favorite websites and ensure they remain accessible in the future.

Merits

• Protects against data loss from broken links or expired domains.
• Allows offline access to archived content.
• Provides a searchable archive for easy retrieval of information.

Demerits

• Requires some technical knowledge to set up and use.
• May consume significant storage space depending on the amount of content archived.
• Regular maintenance and checks are necessary to ensure backups are usable.

Caution

This article is meant for educational purposes. Be sure to replace any placeholder values with your actual data when implementing these commands. Always verify claims against the original source before relying on them.

Frequently asked questions

• What is web archiving? — Web archiving is the process of collecting and storing web content to preserve it for future access.
• Why should I back up a website? — Backing up a website prevents loss of valuable information due to link rot, domain expirations, or platform changes.
• What tools can I use for archiving web content? — Popular tools include wget, monolith, and ArchiveBox.
• How do I use wget to mirror a website? — You can use wget --mirror along with various options to download an entire website.
• What is WARC format? — WARC (Web ARChive) is a file format used to store web pages and their associated metadata for long-term preservation.
• Can I archive JavaScript-rendered websites? — Yes, but you may need tools like SingleFile or ArchiveBox that can handle JavaScript.
• How can I automate the archiving process? — You can use cron jobs to schedule regular backups of your favorite websites.
• What is the 3-2-1 backup rule? — The 3-2-1 rule suggests keeping three copies of your data, on two different types of storage, with one copy stored offsite.

Tags

#webdev #archiving #selfhosted #devops #backup #wget #monolith #ArchiveBox #datarecovery #webpreservation