QRS Infrastructure Network, March to September 2026
We had a mess.We built a mesh.
QRS is RW Interactive’s Quality and Reliability System. This is the story of the QRS Infrastructure Network: how we went from machines spread across four clouds that nobody could fully explain to one environment where every host is declared, watched and recoverable.
2026-03-07 first commit → 2026-09-24 today
Act I: before the network
Nobody knewwhat was running.
Four providers, years of contractors, no handover.
Before the network
Years of reactive work, with no plan and no handover
Successive contractors had built the infrastructure one fix at a time. Nobody could say what ran where, what depended on what, or what was safe to switch off. The bill was substantial and unexplained, and the security state of the estate was unknown.
AWS
Jenkins CI, a Grafana/Loki stack and a NAT gateway, plus leftovers: AMIs, snapshots, Lambdas, S3 buckets and dormant IAM users.
Google Cloud
About 24 projects. All were deleted in the July clean-up.
Hetzner Cloud
18 hosts: GitLab, three Mailcow mail servers, wiki, Nextcloud, Odoo, Budibase, chat and product servers. One app had been dead for about six months and was still billed. One host had gone about two years without a reboot.
HostGator
Shared cPanel hosting with live mail on it. Around it: 44 domains at Namecheap and 3 zones on Cloudflare.
Service detail: the read-only estate audit of July 2026. Per-provider costs are left out of this version; the totals are on the cost slide.
The goal
The QRS Infrastructure Network: one sovereign, self-hosted, governed environment
Own the stack
Open-source and self-hosted over SaaS. Nothing that can be revoked, repriced or used to trap our data.
Governed, not improvised
Changes go through commands with risk classes, dry-runs, evidence and rollback. No raw shell improvisation on production.
Declared and recoverable
Every machine is declared as source: manifest → NetBox → DNS. Every workload has a backup and a restore drill that has been proven.
Zero-trust access
Nothing is reachable because someone guessed a hostname. Access is identity-aware, brokered and scoped to a role.
Zones × planes
Every component sits on two axes: where it operates (tools, develop, public) and what it is responsible for (access, vault, source, watchtower, recovery, office…).
Built for automation
The command surface that humans use is the same one automated operators drive. That makes reliability and evidence foundational.
Source: qrs-infrastructure, docs/vision/qrs-system-vision.md
Seven months at a glance
A quiet foundation, then a steep build
Each bar is the number of commits to the network’s infrastructure repository in that month: every decision, manifest, runbook and fix, as recorded.
Source: git log of qrs-infrastructure, counted per month. *September runs to the 24th, shown hatched. Commit count measures activity, not value. The access-client repository adds 801 more commits from June onwards.
Act II: March to August
Stop patching.Start declaring.
The foundation first, then the planes: access, vault, source, watchtower, recovery, office.
Mar–MayFoundation
- Day one. A builder machine is created to hold the infrastructure repository and to build everything else. Two redundant control-plane gateways follow, with WireGuard as the way in.
- A source of truth from the start. NetBox inventories the existing machines, and internal DNS (CoreDNS) is generated from it using the
{name}.{area}.rwiconvention. - Redundancy proven, not assumed. Gateway failover over a floating IP is exercised and validated before anything depends on it.
- Structural clean-up. The repository is reorganised into six canonical families (execution, infrastructure, operations, runtime…) with retention rules.
- Access-plane M2. Controlled user workflows, access packs and an identity-backed access prototype.
JuneAccess, rebuilt
- An identity-aware access gateway replaces the legacy stack. Access becomes identity-aware and zero-trust. WireGuard access packs are retired.
- QRS Access is born. We decide to build our own desktop client in Electron and Go (ADR-0001), so that people log in once and see only what their role allows.
- Canonical names everywhere. Manifests are normalised, superseded commands are removed, and every mutating command gets a contract.
- Hardening pass. OpenTofu is chosen over Terraform, and brokered authentication is decided. An internet-facing endpoint that answered without authentication is found and locked down the same day.
JulyThe planes go up and the audit begins
- Core planes decided. A self-hosted vault for secret custody, the Watchtower observability stack (Prometheus, Loki, Grafana, Mimir), internal TLS from our own PKI, redundant internal DNS, and a second recovery destination.
- The legacy audit. A read-only inventory of AWS, GCP, Hetzner and DNS. AWS and GCP are cleaned the same weekend, cutting the monthly bill. All 18 Hetzner hosts are inspected from the inside.
- First bare-metal node. An on-prem Proxmox hypervisor in the office becomes the develop zone.
- The network is self-contained. No reuse of outside services (ADR-0023). Legacy hosts can be read to plan a migration, never relied on.
- Operations plane complete, 8 of 8 items. Governed execution, command catalogue and maintenance windows.
- Public zones are a separate territory from the private island (ADR-0028). Customer-facing hosts get their own boundary.
- Forgejo goes load-bearing. Self-hosted Git and CI carry real project work.
AugustFilling the zones
- Develop zone populated. Every product gets its own dev and QA environments on the office hypervisor, seven products in all.
- How work flows. Mail consolidates onto one mail host inside the network. Identity and access get one authoritative file. Deploys become pull-based from an estate-owned agent. CI moves to a dedicated runner. Alertmanager becomes the single source of alerts.
- First outside server brought in. The machine that hosted a client product is migrated and rebuilt as a production host inside the network.
- Mattermost → Matrix decided (ADR-0056). Company chat is to move inside the network.
- One service catalogue. The control plane reconciles it, and session re-authentication is set to be a passkey, not a password form.
- Office plane chooses its tools. ERPNext for ERP, because an ERP must speak OIDC, and Frappe Helpdesk for support.
- Posture declared: pre-production. The doors are not open yet. One switch will turn every unmonitored host red on the day they open.
- AWS: zero machines.
Act III: the migration
Eighteen machines.One left standing.
Eight rebuilt inside the network. Nine destroyed once their work had moved. One still waiting on the chat cutover.
SeptemberCutover, then the automation layer
- Every legacy host measured against the provider, not against a spreadsheet. Three are confirmed gone.
- The big cutover day. Two client production apps and the company file store move into the network, each verified before its old host went: data parity, smoke tests, and a restore drill for the file store. Four legacy hosts are destroyed the same day, after archiving.
- GitLab and the old secrets box are retired. Each is snapshotted first and the snapshot kept for 90 days. GitLab's work already runs on Forgejo and Plane's on Leantime. The old secrets box's snapshot is its archive.
- A legacy host is rebuilt in place as the network’s automation host.
- The execution plane. The network gets its own daemons, each with a leased work queue and explicit per-command permissions. Their primary job does not depend on an AI layer.
- Every manual command gets a wizard in the QRS Access client, rendered from its own contract (ADR-0088). The Mattermost cutover is held on the day because a pre-check failed. HostGator is closed out: the account had already expired without renewal, and every archived copy is deleted.
- Routine maintenance on the new estate: Forgejo, ERPNext and version pins brought current.
The migration, host by host
Where each of the 18 legacy hosts went
8rebuilt inside the network9destroyed after their work moved1still legacy
“Rebuilt” means the machine was wiped and re-provisioned as a network host with its public address kept. “Destroyed” means its services were moved or archived first, then the machine was deleted.
| Legacy host | What it ran | Now |
|---|---|---|
| legacy-01 | Mail server | rwi-qrs-19-██████-████ |
| legacy-02 | Mail server | rwi-qrs-22-██████████-██████ |
| legacy-03 | Failover box | rwi-qrs-15-██████-███ |
| legacy-04 | Dead app, still billed | rwi-qrs-20-██████-████ |
| legacy-05 | App dev | rwi-qrs-21-████-████ |
| legacy-06 | Product dev | rwi-qrs-26-██████-████ |
| legacy-07 | Proof of concept | rwi-qrs-30-███████-████ |
| legacy-08 | Mail trap, reports | rwi-qrs-32-█████-██████████ |
| Legacy host | What it ran | Now |
|---|---|---|
| legacy-09 | Wiki | gone 2026-08-29 |
| legacy-10 | RWI mail | mail → rwi-qrs-19 |
| legacy-11 | Client back-office | gone, confirmed 09-03 |
| legacy-12 | File store, client product | → rwi-qrs-13 / -23 |
| legacy-13 | Client product | → rwi-qrs-23 |
| legacy-14 | Internal tools | role → rwi-qrs-31 |
| legacy-15 | Product server | archived 2026-09-11 |
| legacy-16 | GitLab, Plane | → Forgejo + Leantime |
| legacy-17 | Odoo, secrets stores | snapshot kept, 09-14 |
| legacy-18 | Mattermost chat | still legacy retires after Matrix cutover |
Source: qrs-infrastructure, infrastructure/manifests/legacy-estate.tsv. The estate check validates each “rebuilt” claim against that host’s own manifest, so a false claim fails the check.
Act IV: now
32 hosts.Every one declared.
Each has a manifest, a place in the inventory and a name that says where it lives and what it does. Names are redacted in this version.
Where we are now
32 hosts, one register, one set of rules
Tools zone
15Builder, inventory, vault, observability, object storage, source (Forgejo), CI runner, DNS, office apps (ERP, collaboration), operations and automation.
Public zone
12Customer production apps, web hosting, demos, mail, the public forge, the public vault and a public watchtower. Watched by their own public watchtower.
Access & control
3Two redundant access edges and the access control core. This is the only way in.
Develop zone
2The office bare-metal hypervisor and a cloud develop host. The hypervisor carries 14 product dev/QA environments and 3 shared dev services.
84 architecture decisions
Each is written down with its reasoning and who ruled on it.
Recoverable by proof
177 backup-timer series report in, and 22 restore-drill targets report back.
Honest status
Posture is still pre-production. The last live check (2026-09-25 08:49 UTC) showed 20/20 watched hosts reporting and 2 open alerts.
Sources: manifests/servers (32 hosts, 50 manifests including virtual guests), the ADR directory, and the estate probe of 2026-09-25 08:49 UTC. The cost comparison is on the next slide.
How it is organised
Two questions place every component: where does it run, and what is it for?
Zone: where it runs
Tools runs the network itself and the company’s office apps. Develop holds a dev and a QA environment for every product. Public holds what visitors are meant to reach. Access and control is the front door.
Plane: what it is for
Ten responsibilities, from the front door to the office suite. The same plane can run in more than one zone. A public site gets its own monitoring, not a pipe into ours.
So any machine can be explained in one line: its zone, its plane, and its name, which says both.
The ten planes
Each plane has one job
Access
The single front door. Log in once, reach only what your role allows.
Operations
Changes run as governed commands with risk classes, approvals, evidence and rollback. Never improvised shell.
Inventory
One source of truth for every machine and service. The estate can be rebuilt from it.
Vault
Secrets, keys and certificates, kept out of code and out of plain text.
Source
Code, CI and releases. Every deployable piece traces back to a release.
Data storage
Object storage and volumes under every stateful service, kept separate from compute.
Watchtower
Metrics, logs and alerts, so failures are found by the system, not by users.
Security
Protection and detection across every zone, not bolted onto one edge.
Recovery
Backups whose restores are proven, and plans to rebuild.
Office
The company’s own self-hosted suite: mail, chat, files and ERP.
The security model
Two territories, and no hallway between them
- The island holds everything we run for ourselves. No other network reaches it. The only way in is the access app: no login, no connection; no connection, no route.
- The public zone holds what visitors are meant to reach. It has its own machines, addresses, firewall and monitoring, and no link to the island’s private network.
- Neither side reaches into the other. The one allowed crossing is administration, from the island into the public zone. The sites are public; their control panels are not.
- Assume the worst. If a public site were taken over, the attacker should find nothing that leads to the island. The rule requires a recorded refusal as proof, not an assumption.
Letting visitors in, safely
Visitors get the shop floor, never the back office
| What | Who reaches it | How |
|---|---|---|
| Production sites and apps | Anyone | The public internet, by design |
| Development and QA | People with a role on that project | Only through the access app, after login |
| Demos | A prospect or client you invite | Today, a login we issue per demo. not built A temporary-token tool is decided and going into planning; it has no date yet |
| Control panels of public hosts | Operators | Only from inside the island |
| Everything we run for ourselves | People with a role | Only through the access app. One login is recognised by every service |
The rule is by class, decided 2026-09-10: production is public by nature and everything else is confined. Development and QA are not reachable from the internet. 16 of 24 open through the access app; the other 8 are not yet wired in (last checked 2026-09-25).
What it costs
More than half off the monthly bill
≈ €930a month before
≈ €350a month now
This is not like for like. “Now” pays for the 31 of the 32 declared hosts that run in the cloud (the 32nd is the office server), with backups, monitoring and restores that have been proven. “Before” paid for 18 hosts and four providers’ worth of services that nobody could fully explain.
Before: the company’s own account of the bills before the clean-up, dollar amounts converted at the ECB reference rate of 2026-09-24 (€1 = $1.1367). Now: Hetzner list prices read from its API on 2026-09-25, net of VAT, with customer-owned machines excluded. Not yet measured, so “now” is a floor: the backup Storage Box, any remaining AWS charges and the office hypervisor’s power. Domains are left out of both.
What’s left
Before the doors open
- Chat cutover. Mattermost → Matrix, held on 2026-09-23 until the promised Windows and macOS desktop packages are published. The last legacy host then goes read-only for 30 days, is snapshotted and is destroyed.
- Paperwork closures. The last AWS organisation, which only AWS Support can close, and two Google billing accounts, which now bill nothing.
- Finishing the newest host. The automation host is monitored and its rebuild drill has passed. It still needs its internal DNS record.
- Opening the doors. Flip the posture to production. From that moment every unmonitored host shows red.
Re-checked 2026-09-25 against the owning records in qrs-infrastructure: the legacy register, the 2026-09-15 rulings, the host manifests and the 2026-09-23 handoff. Google billing: zero, confirmed by the owner on 2026-09-25.