Keep CI Running When GitHub Is Down

When GitHub Actions goes down, most teams stop shipping. Pull requests cannot be merged because the required checks never report. Releases wait. If you self-host your runners, you might assume you are insulated — we did, and we were wrong.

Self-hosted runners are not an independent CI system. They are idle capacity waiting for GitHub to hand them a job. When the orchestrator is unreachable, hardware you own and pay for sits doing nothing, and everyone waiting to merge sits with it.

This article describes the fallback we built at WP STAGING: a second control plane that can receive commits, run the full test suite and publish a release while github.com is unreachable. It doubles as our own operating manual — the commands below are the ones we run — and everything in it transfers to any team on GitHub Actions, whatever your stack.

What actually breaks in an outage

It helps to separate the pieces, because they fail independently.

The repository. You cannot push, and colleagues cannot fetch your branch. Local clones are intact, which matters more than it sounds.

The orchestrator. Nothing dispatches jobs. Runners connected to GitHub have nothing to do.

Hosted runner services. If you use a paid service for heavy suites — we use Blacksmith for end-to-end tests — it goes down with GitHub, because it is a GitHub Actions product. This one surprises people.

Everything your jobs quietly download from GitHub. This is the category that ruins fallback plans, and we return to it below.

The shape of the fallback

One small cloud VM runs Gitea, an open-source Git forge with a workflow engine that reads the same YAML syntax as GitHub Actions. It holds a copy of the repository, refreshed every ten minutes. Our existing runners register with it as a second fleet.

In normal operation nothing changes: GitHub is the repository of record, our own runners take unit and integration tests, and Blacksmith takes the end-to-end suite. During an outage, the Gitea instance dispatches the same jobs to the same machines. The end-to-end suite moves onto our own hardware and runs slower — which is a fine trade against not running at all, and the hardware is free at that moment precisely because the outage stopped everything else.

The cost is one small VM. The runners already exist.

The only thing that changes is which orchestrator hands out the work. The hosted runner service is unreachable during the outage too, so the end-to-end suite moves onto machines that are idle precisely because everything else has stopped.

One workflow definition, not two

The obvious trap is maintaining a second copy of every workflow. A copy drifts, and a fallback that drifts is worse than none, because you will not discover it until the day you depend on it.

Our workflows are generated from PHP job builders rather than hand-written YAML, so the fallback became a second render target instead of a fork:

./wpstg github:generate-workflows                 # → .github/workflows/
./wpstg github:generate-workflows --target=gitea  # → .gitea/workflows/

Both come from the same builders. The Gitea render swaps the runner labels and drops the steps that call the GitHub API, and the GitHub render stays byte-identical — we verify that on every change, because "no diff" is the only assurance that the fallback cannot break production CI.

If your workflows are hand-written YAML, the same idea applies with a different mechanism: keep the job bodies in composite actions or shell scripts that both forges call, so the duplicated part is the trigger block rather than the work.

One convenient detail: Gitea reads .gitea/workflows/ and, the moment that directory exists, ignores .github/workflows/ completely. GitHub never looks at .gitea/ at all. Each forge sees exactly one set of files, so neither can pick up the other’s jobs.

Two directories, one source. The fallback is a render target, not a fork, so it cannot drift from the pipeline it stands in for.

The dependencies that make "self-hosted" a lie

A job running on your own hardware still reaches out to GitHub several times before it does any real work. Each one is a single point of failure that only reveals itself during an outage.

Actions. uses: actions/checkout@v6 is a clone from github.com. Gitea can resolve bare action references against itself instead, so we mirror the handful we actually use — checkout, cache, upload-artifact and two others — into the fallback forge and point it there. An action nobody mirrored simply does not exist during an outage.

Packages. This is the one that caught us. Every dependency in our lock files — all 134 of them — resolves to an archive on api.github.com or codeload.github.com. Package metadata comes from Packagist and is fine; the downloads are not. We now run a local package mirror on the same VM, built with Satis from our own lock files so the mirrored set cannot drift from what CI installs. It is 34 MB, rebuilt nightly, and it is the difference between a fallback that builds and one that only looked like it would.

npm, PyPI and RubyGems users have the same exposure whenever a dependency resolves to a GitHub tarball, and comparable mirroring tools.

Caches. Warm caches hide all of this. With a populated dependency cache the install step never runs, the fallback goes green, and you have proved nothing. Test with the cache deliberately wiped.

Three outbound paths, three local interceptions. The mirrors refresh on a timer while GitHub is reachable, so during an outage they are frozen — a day stale at worst, and complete.

Running the tests

Both forges gate the expensive suites behind a label, because a full run is ten minutes of a machine somebody else is also waiting for. During an outage you add the same label you always do, or start a run by hand from the Actions tab.

Two portability notes if you copy this. Gitea reports label events differently from GitHub: the action arrives as label_updated rather than labeled, and the "which label was just added" field is empty, though the list of labels on the pull request is populated. A condition written for one forge is silently false on the other, which looks exactly like a broken gate. And the workflow-level types: filter is not applied as strictly, so gate at the job level rather than relying on the trigger.

The outage procedure

This is our runbook. Nothing in it contacts GitHub, and nothing in it can lose a commit — every push in either direction is fast-forward only, so a divergent branch fails loudly instead of being flattened.

Before you need it, add the remote. A remote you cannot add during an outage is a remote you do not have:

git remote add gitea ssh://git@gitea.example.com:2222/org/repo.git

During the outage, an admin freezes the mirror so the ten-minute copy from GitHub cannot run over work created on the fallback:

make dr_on

Then work as usual. Push your branch, open the pull request in the fallback’s web interface, add the label to start the tests, review, merge:

git push gitea my-branch

Releasing does not have to wait either, and this is worth thinking through for your own product. Our paid plugin is delivered by our own update server and the free plugin by the WordPress.org plugin directory over SVN. Neither knows GitHub exists. So the fallback builds both distributions exactly as the normal pipeline does and ships them to those two endpoints, defaulting to a dry run that reports what it would publish. What it deliberately leaves for later is the GitHub bookkeeping — the tag, the release, the pull request in the free plugin’s repository — because none of that is something customers wait for.

When GitHub returns, push the work back and unfreeze:

make recover        # dry run: reports what it would push
make recover_push   # fast-forward only, never deletes
make dr_off

A branch that cannot fast-forward is named and left alone. That case is a real merge conflict between work done on both sides, and it wants a human rather than a script.

Prove it by taking GitHub away

Everything above works while GitHub is up, which proves nothing at all. The only honest test is to block GitHub’s address ranges inside the runners and their container engine, then run the whole loop: push, pull request, unit suite, end-to-end suite, merge, release.

Ours found three real dependencies on its first run, each invisible in a green pipeline:

• a dependency directory that was never in the cache, so the build could not be assembled offline
• a package cache silently disabled because a shell variable was echoed instead of exported — every job had been re-downloading everything for months, on both forges
• a cache server advertising an address its clients could not reach, so every upload failed while every download looked healthy

None of these would have surfaced in normal operation. That is the entire argument for rehearsing.

One warning from our own drill: block the forge’s own address ranges, not everything it publishes. GitHub’s published list includes the cloud ranges behind hosted runners — thousands of entries, evaluated per packet — and blocking all of them timed out an unrelated service and produced a failure that had nothing to do with the exercise.

What to take from this

The tools matter less than the four decisions behind them.

Generate your workflows, so a second target is a render rather than a fork. Intercept every outbound path a job takes to the forge you are hedging against — actions, packages, the repository itself. Make the copy direction incapable of destroying work, which means fast-forward-only pushes and no force. And rehearse with the network actually cut, because a fallback you have never exercised is a belief, not a capability.

We built ours in a day on a small VM and hardware we already owned. The expensive part was not the infrastructure. It was finding out how many things quietly needed GitHub.