# Self-hosted GitHub Actions runner on `hzdev` (Dockerized, isolated)

This document records how this repo's workflows run on self-hosted runners on
the `hzdev` box, and how those runners are set up and operated.

> **Why this exists:** all jobs ran on `ubuntu-latest`. `hzdev` (8 cores,
> 62 GiB RAM) already hosts the runner families for `cca` (backend,
> `cca-backend-hzdev`), `cca-app` (frontend, `cca-hzdev`) and `api`
> (`api-hzdev`), and keeps warm package caches between runs — `test-downstream`
> in particular installs the full cca dependency set every run, which the warm
> uv cache cuts dramatically. This family follows the `cca` backend's design
> (see `cca/docs/ci/self-hosted-runner-hzdev.md`); everything here is namespaced
> `ecd-runner` / label `ecd-hzdev` to coexist with the other families. The repo
> only contains the `runs-on` changes, these deployable sources, and this doc;
> the live runner lives on `hzdev`.

## TL;DR

- Runners run **inside Docker containers** on `hzdev` (image `ecd-runner:latest`).
- Each job runs in a fresh **`--ephemeral`** container, then the container exits.
- **Four** host-side **systemd** template instances (`ecd-runner@1..4`) each
  supervise the loop: mint a short-lived registration token → launch one
  ephemeral container → repeat. Four slots ≈ a typical push's parallel job burst
  (5-version dateparser matrix + scripts + downstream) with modest queueing.
- The job label is **`ecd-hzdev`**; every workflow targets
  `runs-on: [self-hosted, linux, x64, ecd-hzdev]`.

## One image, every workflow

A single image (`ecd-runner:latest`) serves all jobs. Beyond the base toolchain
(build-essential, git, curl), it bakes in what GitHub-hosted runners preinstall
and this repo's workflows actually use:

- **Python** — a 3.12 baseline only; `actions/setup-python@v6` overlays each
  job's Python (3.10–3.14 across the matrices) as a writable, non-root install
  first on PATH, so `pip install` / `uv pip install --system` never touch the
  root-owned, PEP 668 externally-managed system Python.
- **uv** — `test-downstream.yml` installs the cca dependency set with it.
- **gh CLI** — `upgrade-deps.yml` shells out to `gh pr list` / `gh pr create`.
- **AWS CLI v2** — `release.yml` runs `aws s3 cp` / `aws sts` (credentials come
  per-job from the OIDC `configure-aws-credentials` action, never from the host).
- **No Node** — the runner agent bundles its own runtime for JS actions, and no
  job runs node/yarn directly.

## Isolation model

`hzdev` holds sensitive host credentials, so the runner is sandboxed the same
way as the cca backend family:

- **One mount only:** the non-secret `ecd-runner-cache` volume at
  `/home/runner/.cache` (uv **and** pip caches — this repo installs with both;
  each locks its cache, so concurrent slots are safe). The container never sees
  the rest of the host filesystem.
- **No Docker socket** is mounted — the container cannot control host Docker.
- **Non-root** (`runner`, **pinned uid/gid 1001**, same as the other families)
  **+ `--security-opt no-new-privileges`**, no `--privileged`.
- **The GitHub PAT never enters the container.** It lives on the host
  (`/etc/ecd-runner/runner.env`, root-only, `0600`) and is used host-side only
  to mint *registration tokens*. The container receives only the short-lived
  registration token.
- Job secrets (`DOWNSTREAM_APP_*`, `ANTHROPIC_API_KEY`, AWS OIDC) flow from
  GitHub per-job exactly as they did on `ubuntu-latest`; nothing secret is baked
  into the image or host env.

## Files on `hzdev`

| Path | Purpose |
|------|---------|
| `/opt/ecd-runner/Dockerfile` | Runner image: Ubuntu 24.04 + Python baseline + uv + gh CLI + AWS CLI v2 + build tools + `actions/runner`. |
| `/opt/ecd-runner/entrypoint.sh` | Container entrypoint: `config.sh --ephemeral` then `run.sh` (one job, then exit). |
| `/opt/ecd-runner/ecd-runner-supervisor.sh` | Host loop (one per instance): mint registration token from the PAT, `docker run` one ephemeral container, repeat. |
| `/opt/ecd-runner/build.sh` | Builds `ecd-runner:latest`, pinning the latest `actions/runner` release. |
| `/etc/ecd-runner/runner.env` | **Secret** (`0600`, root): `GITHUB_PAT`, `REPO`, `RUNNER_LABELS`. |
| `/etc/systemd/system/ecd-runner@.service` | systemd **template** unit; instantiate `@1 @2 @3 @4`. |
| `/opt/ecd-runner/ecd-runner-cleanup.sh` | Trims the cache volume once it exceeds `CACHE_MAX_GB`, using the api-runner drain protocol: pause flag → deregister idle runners (GitHub's DELETE fails atomically on a busy one, aborting harmlessly) → wipe only at zero registrations (run by the timer). |
| `/etc/systemd/system/ecd-runner-cleanup.{service,timer}` | Daily cleanup oneshot + its timer (05:00, offset from the other families). |
| Docker volume | `ecd-runner-cache` (warm uv + pip caches). |

The source of these files is kept in this repo under
[`docs/ci/ecd-runner/`](./ecd-runner/) so they can be re-deployed.

## The GitHub PAT

A **fine-grained PAT** is required to mint runner registration tokens:

- Resource owner: `EarningsCall`
- Repository access: **only** `earningscall-development`
- Repository permission: **Administration → Read and write**

Store it on the host (never commit it). Prompt for the token and send the file
over stdin so the PAT never appears on a command line (shell history, `ps`):

```bash
read -r -s -p "GitHub PAT: " GITHUB_PAT; printf '\n'
printf 'GITHUB_PAT=%s\nREPO=EarningsCall/earningscall-development\nRUNNER_LABELS=self-hosted,linux,x64,ecd-hzdev\n' "$GITHUB_PAT" |
  ssh hzdev 'mkdir -p /etc/ecd-runner && install -o root -g root -m 600 /dev/stdin /etc/ecd-runner/runner.env'
unset GITHUB_PAT
```

> This is a **separate** PAT from the `cca`, `cca-app`, and `api` runners' —
> a fine-grained PAT minting registration tokens must name the specific repo.

### Regenerating the PAT (it expires)

> **Current PAT expires 2027-07-19** (minted 2026-07-20). Update this line when
> you rotate.

When it expires, the supervisors can no longer mint registration tokens and the
runners stop coming online (`journalctl -u 'ecd-runner@*'` shows `failed to
mint registration token`); queued jobs sit in "Waiting for a runner" until a
new PAT is installed. To rotate it:

1. Create a new fine-grained PAT at
   <https://github.com/settings/personal-access-tokens/new>:
   - **Resource owner:** `EarningsCall`
   - **Repository access:** Only select repositories → **`earningscall-development`**
   - **Permissions → Repository → Administration:** **Read and write**
     (Metadata: Read-only is added automatically — leave it)
   - **Expiration:** your choice; note the new date here when you rotate.
   - **Generate token** and copy it (`github_pat_…`) — shown only once.
   - If the org requires approval, an org owner approves it under
     *Org Settings → Personal access tokens → Pending requests*.
2. Update the secret on `hzdev` with the command above and restart:
   ```bash
   ssh hzdev 'systemctl restart ecd-runner@1 ecd-runner@2 ecd-runner@3 ecd-runner@4'
   ```
3. Confirm they came back online:
   ```bash
   ssh hzdev "journalctl -u 'ecd-runner@*' -n 20 --no-pager"   # expect "Listening for Jobs"
   gh api repos/EarningsCall/earningscall-development/actions/runners \
     --jq '.runners[] | "\(.name) \(.status)"'
   ```
4. Delete the old token from GitHub once the new one works.

## Deploy / rebuild

```bash
# (on hzdev)
cd /opt/ecd-runner
./build.sh                      # builds ecd-runner:latest with latest actions/runner

# create the warm cache volume (empty, so it inherits runner ownership)
docker volume create ecd-runner-cache

# enable the four supervised ephemeral loops + the daily cleanup
sudo systemctl daemon-reload
sudo systemctl enable --now ecd-runner@1 ecd-runner@2 ecd-runner@3 ecd-runner@4
sudo systemctl enable --now ecd-runner-cleanup.timer
```

Confirm the runners registered (from anywhere with repo admin + gh):

```bash
gh api repos/EarningsCall/earningscall-development/actions/runners \
  --jq '.runners[] | "\(.name) \(.status) [\([.labels[].name]|join(","))]"'
```

### Important gotcha: cache volume ownership

The container runs as non-root `runner`. The named cache volume must be
**writable by `runner`**, or uv/pip can't write their caches. This is handled by
pre-creating `/home/runner/.cache` **owned by `runner`** in the Dockerfile
*before* that path becomes a volume mountpoint, so a fresh empty volume inherits
`runner` ownership. **If you recreate the volume, it must be empty when first
mounted** so it picks up that ownership:

```bash
docker volume rm ecd-runner-cache
docker volume create ecd-runner-cache
```

## Tailing logs on `hzdev`

```bash
# All four supervisors + which container is running which job:
ssh hzdev "journalctl -u 'ecd-runner@*' -f"

# The runner agent's stdout for a specific slot's current job:
ssh hzdev 'docker logs -f ecd-runner-1'   # or -2 / -3 / -4
```

## Operating

```bash
# Stop the runner slots. NOTE: this force-removes the containers (docker rm -f),
# which KILLS any in-flight job — check nothing is busy first if that matters:
#   gh api repos/EarningsCall/earningscall-development/actions/runners \
#     --jq '.runners[] | select(.busy) | .name'
# Killed idle listeners leave short-lived "offline" registrations; GitHub
# auto-removes offline ephemeral runners after about a day.
ssh hzdev 'sudo systemctl stop ecd-runner@1 ecd-runner@2 ecd-runner@3 ecd-runner@4'

# start again
ssh hzdev 'sudo systemctl start ecd-runner@1 ecd-runner@2 ecd-runner@3 ecd-runner@4'

# fully remove a registered runner from GitHub (e.g. a stale one)
gh api repos/EarningsCall/earningscall-development/actions/runners \
  --jq '.runners[] | "\(.id) \(.name)"'
gh api -X DELETE repos/EarningsCall/earningscall-development/actions/runners/<ID>
```

## Rollback

Revert the `runs-on:` changes in `.github/workflows/*.yml` (set the jobs back
to `ubuntu-latest`) and (optionally) on `hzdev`:

```bash
sudo systemctl disable --now ecd-runner@1 ecd-runner@2 ecd-runner@3 ecd-runner@4 \
  ecd-runner-cleanup.timer
docker rmi ecd-runner:latest
docker volume rm ecd-runner-cache
```
