I run a Gitea instance at git.langille.org. Some of what’s on there is a pull mirror of a GitHub repo. Some of it exists nowhere else. That second group bothers me. If that server goes away, so do those repos.
I’m sure I can find another existing script and use that. Please point one out to me if you find it.
I want every repo on git.langille.org copied to another server, on FreeBSD, on ZFS, with snapshots so I can go back in time.
I didn’t write the script. Claude did. I described what I wanted, it wrote the script, and I reviewed it. More on that at the end. I have a 6-month trial of Claude. Let’s see how it can help.
What it does
- asks the Gitea API for every repo my token can see: mine, my orgs, and anything I collaborate on
- runs git clone --mirror for anything new, and git remote update --prune for everything else
- leaves snapshots to the jail host, which runs sanoid on the dataset
- prints nothing on a clean run, so cron only mails me about a new repo or a failure
- logs every run to syslog with logger, so a quiet run still leaves a trace
- pings watchgoose after a clean run, so I hear about it if the runs stop
If I delete a repo on the Gitea server, the copy stays. This is a backup, not a sync.
Setup
The script runs as me, in a jail called dbclone. The jail doesn’t manage any ZFS itself. The host, r730-03, creates the dataset and mounts it straight into the jail’s filesystem:
[17:39 r730-03 dvl ~] % sudo zfs create -o mountpoint=/jails/dbclone/usr/home/dvl/gitea-backups data01/gitea-backups
[17:39 r730-03 dvl ~] % I didn’t set compression. The dataset inherits lz4 from data01. I took this while the first run was still cloning, so the ratio covers only part of the data:
[17:39 r730-03 dvl ~] % zfs get all data01/gitea-backups | grep compress
data01/gitea-backups compressratio 1.03x -
data01/gitea-backups compression lz4 inherited from data01
data01/gitea-backups refcompressratio 1.03x -Even partway through, 1.03x is about what to expect. git already compresses everything it stores: loose objects are zlib-compressed, and packfiles are compressed and delta-encoded on top of that. There’s little left for ZFS to squeeze. I left lz4 on anyway. It gives up quickly on data it can’t compress, so it costs almost nothing, and it still helps with the uncompressed bits: refs, config, hooks, and the pack index (.idx) files.
Inside the jail, I make it mine and install what the script needs:
[17:40 dbclone dvl ~] % sudo chown dvl:dvl gitea-backups
[17:40 dbclone dvl ~] % sudo pkg install git curl jqBefore writing anything, the script checks that ~/gitea-backups really is a ZFS mount. If the dataset ever isn’t mounted, the mirrors won’t quietly fill up the jail’s own disk.
Next, a Gitea API token. The Gitea docs cover generating API tokens. In short: your avatar, Settings, Applications, Generate New Token. These are the minimum settings:
- Repository and Organization Access: All (public, private, and limited). Public only would leave out the private repos, which are the ones I most want backed up.
- repository: Read. This is for cloning, fetching, and listing repos.
- user: Read. Gitea requires this for anything under
/api/v1/user/, and the list of my repos is there. - Everything else: No Access. Nothing needs write.
Gitea shows the token once. It goes into a config file, ~/.gitea-mirror.conf, along with everything else the script needs. That file holds two secrets: the Gitea token, and the watchgoose URL (more on that below). I create it empty and lock it down first, so neither is ever readable by anyone else, not even for a moment:
[17:41 dbclone dvl ~] % touch ~/.gitea-mirror.conf
[17:41 dbclone dvl ~] % chmod 0600 ~/.gitea-mirror.conf
[17:41 dbclone dvl ~] % vi ~/.gitea-mirror.confIt’s read by /bin/sh, so it’s plain NAME=value lines, with no spaces around the =. Here it is with the secrets left out:
# gitea-mirror.sh settings
#
# Copy to ~/.gitea-mirror.conf and chmod 0600: both tokens below give
# access to something. This file is read by /bin/sh, so no spaces
# around the =.
# Where the Gitea API answers for this host.
GITEA_URL=https://git.langille.org
# Gitea API token: repository Read, user Read, all repos (not public only).
GITEA_TOKEN=
# The ZFS dataset r730-03 mounts into the jail.
DEST=$HOME/gitea-backups
# Pinged after each clean run, so a run which stops happening is noticed.
# The URL carries a token, which is why it is here and not in the script.
# Empty means do not ping.
WATCHGOOSE_GITEA_MIRROR=Then cron. This file goes in /usr/local/etc/cron.d in the jail. It runs the script as me every six hours, wrapped in lockf(1) so two runs never overlap:
[18:09 dbclone dvl /usr/local/etc/cron.d] % cat gitea-backups
# use /bin/sh to run commands, overriding the default set by cron
SHELL=/bin/sh
# mail any output to `dan', no matter whose crontab this is
MAILTO=dan@langille.org
PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/games:/usr/local/sbin:/usr/local/bin:/usr/X11R6/bin:$HOME/bin; export PATH
#minute hour mday month wday who command
0 */6 * * * dvl lockf -t 0 /tmp/gitea-mirror.lock $HOME/bin/gitea-mirror.shUnlike a personal crontab, a file in cron.d has a who column, so it can live with the rest of the system configuration and still run as me. Any output gets mailed. The script says nothing on a clean run, so a mail means there’s something to look at: a new repo, a failure, or lockf complaining because the previous run hasn’t finished.
The first time the job ran, this is what got mailed:
lockf: /usr/home/dvl/bin/gitea-mirror.sh: Permission deniedI’d copied the script into place and forgotten to make it executable. lockf runs the command it’s given directly, so it needs the execute bit:
[18:13 dbclone dvl ~] % ls -l /usr/home/dvl/bin/gitea-mirror.sh
-rw-r--r-- 1 dvl dvl 2997 2026.10.19 17:50 /usr/home/dvl/bin/gitea-mirror.sh
[18:13 dbclone dvl ~] % chmod u+x /usr/home/dvl/bin/gitea-mirror.sh
[18:13 dbclone dvl ~] % ls -l /usr/home/dvl/bin/gitea-mirror.sh
-rwxr--r-- 1 dvl dvl 2997 2026.10.19 17:50 /usr/home/dvl/bin/gitea-mirror.shAt least the mail arrived, so that part works.
Snapshots: sanoid on the host
Snapshots belong to r730-03, not to the jail, and sanoid takes care of them. This goes in /usr/local/etc/sanoid/sanoid.conf:
######################################
# /usr/local/etc/sanoid/sanoid.conf on r730-03
######################################
# Bare git mirrors written by gitea-mirror.sh in the dbclone jail.
[data01/gitea-backups]
use_template = gitea_backups
#############################
# templates below this line #
#############################
# The mirror runs every 6 hours, so most hourly snapshots are identical
# and cost next to nothing. They mean a bad run can be rolled back to
# the state just before it.
[template_gitea_backups]
frequently = 0
hourly = 24
daily = 30
weekly = 8
monthly = 12
yearly = 2
autosnap = yes
autoprune = yesSplitting it this way keeps the jail simple. It needs no zfs allow, no dataset delegation, and no snapshot code in the script. A snapshot can land partway through a run. That’s fine for git: objects are written before the refs that point to them, so each repo in a snapshot is in a usable state, and the next snapshot catches up.
Locking down the API
Requests for git.langille.org from outside go through a public proxy, which passes them to the nginx in front of Gitea. I don’t want the Gitea API open to the world. The script needs it to list repos, though. Only one host needs it: dbclone, the server taking the backups.
Every request through the public proxy reaches the backend nginx from the proxy’s own address. dbclone talks to the backend directly. So the backend can decide based on $remote_addr: dbclone gets the API, everyone else gets a 404. A 404 rather than a 403 doesn’t even admit the API is there.
In the http {} block on the backend:
# Gitea API: who may call /api/v1
geo $gitea_api_allowed {
default 0;
10.55.0.140 1; # dbclone
}Before this, the backend’s git.langille.org server {} block had a location /api that returned a 404 for everyone, and a location / holding proxy_pass and all the proxy_set_header lines. This is what it looks like now:
server_name git.langille.org;
error_log /var/log/nginx/git.langille.org-error.log;
access_log /var/log/nginx/git.langille.org-access.log combined;
# inherited by every location below
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Server $host;
proxy_set_header X-Forwarded-Port $server_port;
proxy_set_header X-Forwarded-Proto $scheme;
add_header X-Frame-Options "SAMEORIGIN";
# Gitea API: nothing under /api for the public
location /api {
return 404;
}
# Gitea API: only hosts listed in $gitea_api_allowed
location ^~ /api/v1 {
if ($gitea_api_allowed = 0) {
return 404;
}
proxy_pass http://127.0.0.1:3000;
}
location / {
proxy_pass http://127.0.0.1:3000;
}Three things are going on there:
- The
proxy_set_headerandadd_headerlines moved up fromlocation /to theserverlevel. nginx only inherits them into a location that sets none of its own. With them at the top, bothlocation /and the new/api/v1block get the same headers without copying them. location /apistays. Gitea has more under/apithan/api/v1:/api/swagger,/api/packages,/api/actions. I don’t use any of them from outside, so they stay closed to everyone, dbclone included.- nginx picks the longest matching prefix, so a request for
/api/v1/...goes to the^~ /api/v1block instead of the catch-all. There, only hosts in$gitea_api_allowedget through to Gitea.
I could have written if ($remote_addr != "10.55.0.140") and skipped the geo block. I went with geo because adding another host, or a whole subnet, is then one line. There’s no performance cost to speak of. nginx builds the table once, when it loads the config. It only looks it up for requests under /api/v1, and the lookup is about as cheap as the string compare. The if with just a return inside is one of the safe uses of if in nginx.
Testing. From dbclone, the API answers:
[17:49 dbclone dvl ~] % curl https://git.langille.org/api/v1/version
{"version":"1.27.3"}From pro05, it doesn’t:
[14:01 pro05 dvl ~] % curl https://git.langille.org/api/v1/version
<html>
<head><title>404 Not Found</title></head>
<body>
<center><h1>404 Not Found</h1></center>
<hr><center>nginx/1.30.6</center>
</body>
</html>That’s the same plain nginx 404 you’d get for any page that doesn’t exist. Nothing in it says there’s an API behind it.
And the rest of /api stays closed, even for dbclone:
[18:04 dbclone dvl ~] % curl https://git.langille.org/api/swagger
<html>
<head><title>404 Not Found</title></head>
<body>
<center><h1>404 Not Found</h1></center>
<hr><center>nginx/1.30.6</center>
</body>
</html>Cloning doesn’t use /api/v1, so git over HTTPS keeps working through the public proxy for everyone else.
Quiet, but not invisible
Silence on success is good for my inbox, but it doesn’t tell me anything. Did the job run and find nothing new, or did it not run at all? There are two answers to that.
The first is syslog. Each run logs has started and has completed: 67 mirrored (0 new), 0 failed under the tag gitea-mirror, plus a line for every new repo and every failure. Failures go out at user.err, everything else at user.notice. It’s the same logger pattern I use in the FreshPorts scripts.
The second is watchgoose. After a clean run, the script fetches a URL; if the pings stop arriving, watchgoose tells me. That catches what no amount of logging can: cron not running, the jail being down, the script hanging. The ping only goes out after a clean run. A ping after a failed run would report the check as healthy when it isn’t, which is worse than no monitoring at all. The URL carries a token, which is why it’s in the config file and not the script. Leave it empty and there’s no ping.
The ping uses fetch from the base system, with --retry so one dropped packet doesn’t turn into a false alarm.
I got to test the failure path sooner than planned. I’d named my config file wrong, and the next run sent me this:
2026-10-19 18:39:00 ERROR: cannot read /usr/home/dvl/.gitea-mirror.confand this went to /var/log/messages:
Oct 19 18:39:00 dbclone gitea-mirror[38173]: ERROR: cannot read /usr/home/dvl/.gitea-mirror.confOne mistake, two places telling me about it. With no clean run, there was no ping either, so watchgoose would have been the third if I hadn’t fixed it.
I’d left off the leading dot. The fix:
[18:39 dbclone dvl ~] % sudo mv -i ~/gitea-mirror.conf /usr/home/dvl/.gitea-mirror.conf
[18:39 dbclone dvl ~] % ls -l /usr/home/dvl/.gitea-mirror.conf
-rw------- 1 dvl dvl 731 2026.10.19 18:35 /usr/home/dvl/.gitea-mirror.confThe mv kept the 0600 mode and the owner, so the secrets stayed private the whole time.
The next run was clean. No email, just this in the log:
[18:40 dbclone dvl ~] % grep gitea-mirror /var/log/messages
Oct 19 18:39:00 dbclone gitea-mirror[38173]: ERROR: cannot read /usr/home/dvl/.gitea-mirror.conf
Oct 19 18:41:00 dbclone gitea-mirror[38513]: has started
Oct 19 18:41:07 dbclone gitea-mirror[39090]: has completed: 67 mirrored (0 new), 0 failedSeven seconds to check all 67 repos with nothing to fetch. That’s the quiet run I wanted: nothing in my inbox, but a record that it happened.
Details I liked
The token never appears on a command line. git gets it through GIT_CONFIG_COUNT / GIT_CONFIG_KEY_0 / GIT_CONFIG_VALUE_0 in the environment. curl reads the header from stdin with -H @-. Nothing shows up in ps, and nothing gets written into the mirrors’ config files. The usual trick of putting the token in the clone URL would have left it in every one of them.
The script
#!/bin/sh
#
# gitea-mirror.sh - keep bare mirrors of every repo on git.langille.org
#
# Runs as an ordinary user in the dbclone jail. The destination is a ZFS
# dataset the jail host (r730-03) creates and mounts; the host also takes
# and prunes snapshots of it with sanoid. Nothing here touches ZFS.
#
# Requires: git, curl, jq (pkg install git curl jq)
#
# Layout: $DEST/<owner>/<repo>.git (bare, created with git clone --mirror)
#
# Repos deleted on the server are left in place; this is a backup.
#
# Prints nothing unless a repo is new or something failed, so cron's
# mail means there's something to look at. Every run is logged to syslog
# under the tag gitea-mirror, and a clean run pings watchgoose.
#
# Settings live in $HOME/.gitea-mirror.conf (mode 0600; it holds two
# tokens). See gitea-mirror.conf.sample.
#
# Run from cron, wrapped in lockf(1) so two runs never overlap:
# 0 */6 * * * lockf -t 0 /tmp/gitea-mirror.lock $HOME/bin/gitea-mirror.sh
set -u
CONFIG=${GITEA_MIRROR_CONF:-$HOME/.gitea-mirror.conf}
# defaults; the config file overrides them
GITEA_URL=https://git.langille.org
GITEA_TOKEN=
DEST=$HOME/gitea-backups
WATCHGOOSE_GITEA_MIRROR=
LOGGER="logger -t gitea-mirror"
log() { printf '%s %s\n' "$(date '+%Y-%m-%d %H:%M:%S')" "$*"; }
die() {
log "ERROR: $*" >&2
$LOGGER -p user.err "ERROR: $*"
exit 1
}
[ -r "$CONFIG" ] || die "cannot read $CONFIG"
. "$CONFIG"
$LOGGER -p user.notice "has started"
for cmd in git curl jq; do
command -v "$cmd" >/dev/null 2>&1 || die "$cmd not found"
done
[ -n "$GITEA_TOKEN" ] || die "GITEA_TOKEN is not set in $CONFIG"
[ -d "$DEST" ] && [ -w "$DEST" ] || die "$DEST is missing or not writable"
# Write only into the dataset the host mounted, never the jail's own disk.
[ "$(df -T "$DEST" | awk 'NR == 2 { print $2 }')" = zfs ] ||
die "$DEST is not a mounted ZFS dataset"
# Hand the token to git through the environment, not the command line,
# so it never shows up in ps(1) or in the mirrors' config files.
export GIT_CONFIG_COUNT=1
export GIT_CONFIG_KEY_0=http.extraHeader
export GIT_CONFIG_VALUE_0="Authorization: token $GITEA_TOKEN"
export GIT_TERMINAL_PROMPT=0
api() {
# Header comes in on stdin for the same reason.
printf 'Authorization: token %s\n' "$GITEA_TOKEN" |
curl -fsS --max-time 60 -H @- "$GITEA_URL/api/v1$1"
}
# Every repo the token's user can see: their own, their orgs', collaborations.
list_repos() {
page=1
while :; do
json=$(api "/user/repos?limit=50&page=$page") || return 1
[ "$(printf '%s' "$json" | jq 'length')" -eq 0 ] && break
printf '%s' "$json" | jq -r '.[].full_name'
page=$((page + 1))
done
}
repos=$(list_repos) || die "could not list repos from $GITEA_URL"
[ -n "$repos" ] || die "no repos returned; check the token's scopes"
ok=0
new=0
failed=0
for repo in $repos; do
dir="$DEST/$repo.git"
if [ -d "$dir" ]; then
if git -C "$dir" remote update --prune >/dev/null 2>&1; then
ok=$((ok + 1))
else
log "FAILED update: $repo"
$LOGGER -p user.err "FAILED update: $repo"
failed=$((failed + 1))
fi
else
mkdir -p "${dir%/*}"
if git clone --quiet --mirror "$GITEA_URL/$repo.git" "$dir" 2>/dev/null; then
log "new: $repo"
$LOGGER -p user.notice "new: $repo"
ok=$((ok + 1))
new=$((new + 1))
else
log "FAILED clone: $repo"
$LOGGER -p user.err "FAILED clone: $repo"
rm -rf "$dir"
failed=$((failed + 1))
fi
fi
done
summary="$ok mirrored ($new new), $failed failed"
if [ "$failed" -ne 0 ]; then
log "$summary"
$LOGGER -p user.err "has completed: $summary"
exit 1
fi
$LOGGER -p user.notice "has completed: $summary"
#
# Tell watchgoose the run finished. On success only: a ping after a
# failed run would report the check as healthy when it is not, which is
# worse than no monitoring at all. Empty means do not ping.
#
if [ -n "$WATCHGOOSE_GITEA_MIRROR" ]; then
fetch -o /dev/null -q --retry --retry-delay=10 "$WATCHGOOSE_GITEA_MIRROR" ||
$LOGGER -p user.warning "could not ping watchgoose"
fi
exit 0Renaming or moving repos
A few hours later, the script did mail me:
2026-10-19 19:06:00 new: FreshPorts/DefaultVersions
2026-10-19 19:06:04 new: FreshPorts/freshports-versions
2026-10-19 19:06:04 new: FreshPorts/freshsource
2026-10-19 19:06:04 new: FreshPorts/freshsource-conversion
2026-10-19 19:06:05 new: FreshPorts/freshsource-phorumThey weren’t new to Gitea. I’d just transferred those five from my own account to the FreshPorts organization, and to the script a repo under a new owner is a new repo. The total stayed at 67: has completed: 67 mirrored (5 new), 0 failed.
The script never deletes anything, so the old copies under dvl/ stayed behind. They weren’t doing any harm; the API no longer lists them under dvl, so the script ignores them. But they’d never be updated again, and stale copies of a repo are how you end up restoring the wrong one. With sanoid snapshots holding them anyway, removing them was safe:
[19:08 dbclone dvl ~/gitea-backups/dvl] % cd ~/gitea-backups/dvl && rm -rf DefaultVersions.git freshports-versions.git freshsource.git freshsource-conversion.git freshsource-phorum.gitIf you transfer or rename a repo in Gitea, expect a mail like that one and a stale copy to clean up.
About Claude writing it
I’ve been using Claude Code for FreshPorts work lately. This one started with a sentence: write me a little script to mirror all my git.langille.org repos into a directory on another server, assume FreeBSD and ZFS. What came back was close to what I’d have written myself, with a few things I might not have bothered with, like keeping the token out of ps.
Read it before you run it. That goes for anything you find on a blog, whoever wrote it.
Gitea wikis (repo.wiki.git) aren’t included. I don’t use them. If you do, that’s a small addition.
