# Distribution How a TaxPilot AI release is built, signed, installed and undone. --- ## Why this is not the CMS's update system The CMS updates itself. It has to: the vendor cannot reach a customer's cPanel, so the installation pulls its own packages, verifies them, and applies them unattended. TaxPilot AI is the other way round. ADR-0001 gives one deployment per CMS install, and under the vendor-hosted model those deployments run on infrastructure the vendor controls. Nothing needs to pull. An operator pushes. That difference decides everything below: | | CMS | TaxPilot AI | | --- | --- | --- | | Who initiates | the installation, on a schedule | an operator | | Credential on the box | license key + install secret | none | | Transport | HTTPS from the License Server | a file, however it arrives | | Verification | offline RSA signature | the same offline RSA signature | | Rollback | restore code **and** a database dump | switch a pointer | The verification is deliberately identical — the same offline key, the same discipline of signing a canonical manifest rather than raw bytes, the same rule that nothing is extracted before the signature checks out. Only the transport differs, and it is separated well enough that a pull transport can be added later without touching anything underneath it. --- ## The signature One string is signed: ``` taxpilot-ai-release-v1|{version}|{sha256}|{size} ``` **The prefix is a product boundary, not decoration.** The same offline key signs CMS packages, where the prefix is `taxpilot-update-v1`. Without distinct prefixes a CMS package signature would verify perfectly well against an AI package of the same version and size — and the CMS installer would extract a Python tarball over a Laravel application and run `migrate --force` against it. Everything else a release declares — the migrations it carries, the Python it needs, the CMS version it expects — lives in `manifest.json` **inside** the archive. The digest covers the archive and the signature covers the digest, so the manifest is authenticated by inclusion. That keeps the signed string at four values, which matters because a signer and a verifier have to agree on it exactly, forever. The detached signature travels beside the archive as `.sig`: ```json { "manifest": "taxpilot-ai-release-v1|1.1.0|9f86d0…|4096", "signature": "base64 RSA-SHA256" } ``` The signed string travels **with** the signature rather than being rebuilt from the archive. That is what lets the installer verify before it opens anything — reading a manifest out of an unverified tarball to find out what to verify would put tar parsing inside the trust boundary. ### Verification is hand-written, and here is why `app/release/signing.py` implements RSA PKCS#1 v1.5 verification directly rather than depending on `cryptography`. "Don't roll your own crypto" is right, and the reasoning behind it does not reach this case. The dangerous parts of RSA are on the signing side, around a secret exponent; there is no secret here. The one classic verifier failure — Bleichenbacher's forgery — works against implementations that *parse* the padded block and ignore trailing data. This one never parses: it rebuilds the entire expected block from the digest and compares all of it. What it buys is that the most security-critical step in a deployment works on a bare Python, before anything is installed, on a machine that may have just been rebuilt. What makes it trustworthy is `tests/fixtures/release_signatures.json` — signatures produced by OpenSSL through PHP with the real 4096-bit release key — plus a keypair generated at test time by an independently written DER encoder. If this verifier ever disagrees with OpenSSL, the suite fails. --- ## Layout on disk ``` /opt/taxpilot-ai/ releases/ 1.0.0/ extracted, never modified after install 1.1.0/ shared/ .env configuration — no package ever contains one current -> releases/1.1.0 state.json what is installed, what came before, what happened ``` Switching versions is one atomic pointer move, and rolling back is the same move in reverse. Nothing is overwritten in place, so a failed install cannot leave a running service with a mixture of two releases. Configuration lives in `shared/` and is never packaged. That is precisely what makes a rollback a pointer change: the old release directory is still exactly as it was installed, still pointing at the same `.env` it always used. On a host without symlink privileges the pointer becomes `current.txt` holding the version name. Both forms are readable, and the symlink always wins on read because it is what a process actually follows. --- ## Installing ```bash python -m app.release install dist/taxpilot-ai-1.1.0.tar.gz --root /opt/taxpilot-ai ``` ``` verify signature ─ before the archive is opened at all extract ─ into a staging directory, never over a live release migrate ─ using the NEW release's own migrator health gate ─ `python -m app check` switch ─ one atomic pointer move restart ─ whatever the deployment uses verify again ─ readiness, over HTTP, against the running service roll back ─ on any failure from the health gate onwards ``` **Two verifications, not one.** The gate before the switch catches a release that cannot possibly work — wrong Python, unreachable database, a migration that fails — while the old version is still live and nothing has been disturbed. The check after the restart catches what only shows up once the new code is running. A single check on either side would miss one of those. `migrate` and `check` run as subprocesses **inside the new release**, not as imports. Importing them would run the currently-installed code against the new migrations, which is the subtle way this goes wrong. ### The readiness check asks which version answered A 200 from `/health/ready` only proves something is listening on that port. If the restart silently did nothing — a unit that failed to reload, a container that was never replaced — the old process is still there, still healthy, and still running the code the install was meant to replace. So the probe compares the version in the health payload against the release being installed. A release that reports no version at all is accepted on the 200: that is an older build being rolled back to, and refusing it would break the rollback path exactly when it is needed. **No restart configured means no probe.** Asking whether a service restarted when nothing restarted it either wastes the full timeout or, worse, passes against the old process. The result then says `not verified` rather than claiming health it did not establish. --- ## Rolling back ```bash python -m app.release rollback --root /opt/taxpilot-ai python -m app.release rollback --root /opt/taxpilot-ai --to 1.0.0 ``` Automatic on a failed health gate or a failed readiness check; manual at any time. ### What rollback honestly covers **Code, and the pointer. Not the database.** The CMS restores a SQL dump because it can afford to — it takes the site down, and a CMS being read-only for two minutes is survivable. A rollback here is meant to take seconds, and restoring a dump of the workflow database would discard every run that started since the install, including documents a client sent while it was happening. So the rule is expand/contract: > **A migration must leave the previous release able to run.** > > - Add columns; do not rename or drop them. > - Add tables; do not remove them. > - Make new columns nullable, or give them defaults. > - Remove something one release *after* nothing reads it. `state.json` records which migrations each release applied, so an operator can see exactly what a rollback leaves in place. This is a deliberate trade, not an omission — and it is the same trade every system that wants a fast rollback makes. --- ## Building and signing Building is one command. Signing is a separate deliberate act on the machine holding the offline key, which by design holds nothing else. ```bash python -m app.release build --version 1.1.0 --out dist ``` That prints the string to sign and writes it to `.manifest`. On the key machine: ```bash openssl dgst -sha256 -sign release-private.pem -out sig.bin taxpilot-ai-1.1.0.tar.gz.manifest base64 -w0 sig.bin ``` Then write `.sig` with the manifest and that signature, and check it before it goes anywhere: ```bash python -m app.release verify dist/taxpilot-ai-1.1.0.tar.gz ``` The build tool has no way to sign anything. A build step that could reach the private key would defeat the point of keeping it offline. ### Builds are reproducible Every timestamp, owner and permission bit is normalised, members are sorted, and gzip's own mtime field is zeroed. The same tree builds to the same bytes. That is not tidiness. The signature attests to a digest, so "is the thing you signed the thing in the repository?" is only answerable if the build is deterministic. Without it the honest answer is "trust me". --- ## Restarting Set one of these in `shared/.env`: ```bash TAXPILOT_RESTART_UNIT=taxpilot-ai # systemctl restart taxpilot-ai TAXPILOT_RESTART_COMMAND=docker compose restart ai ``` Neither set means the installer switches the release and tells you to restart it yourself. That is the default on purpose: an installer that guesses how to restart a service it does not manage will eventually guess wrong on someone's production box. A systemd unit that fits this layout: ```ini [Service] Type=simple WorkingDirectory=/opt/taxpilot-ai/current EnvironmentFile=/opt/taxpilot-ai/shared/.env ExecStart=/opt/taxpilot-ai/venv/bin/python -m app Restart=always ``` `WorkingDirectory` points at `current`, so a restart picks up whatever the pointer now names. --- ## The License Server, and the guard that had to exist first `update_versions` had no product column. `latestForChannel()` returned the newest published row in a channel, full stop, and every licensed CMS polling `/v1/update/check` was offered whatever that happened to be. Publishing a TaxPilot AI release into that table would have offered a Python tarball to every CMS installation, which would have verified its checksum and its signature — same offline key — and extracted it over its own application directory. So `product` now scopes every release lookup, `updates:sign --product=ai` signs with the AI prefix, and version numbers are unique per product rather than globally. Existing rows are CMS releases and clients that send no product get CMS releases, so nothing already deployed changes behaviour. **This migration has to be applied to the License Server before any AI package is published there.** ```bash php artisan migrate ``` Under the vendor-hosted model nothing consumes AI releases through the License Server yet — releases are pushed. The column exists because publishing one without it is the dangerous operation, and the guard belongs in place before anyone can perform it, not after. --- ## Reference | Command | | | --- | --- | | `build --version X --out dist` | package a release, print the string to sign | | `verify ` | check a package without installing it | | `install --root DIR` | verify, install, gate, switch, restart, verify | | `rollback --root DIR [--to X]` | return to the previous release, or a named one | | `status --root DIR [--json]` | what is live, what is installed, what happened | | Variable | | | --- | --- | | `TAXPILOT_RELEASE_ROOT` | the deployment directory | | `TAXPILOT_RELEASE_PUBLIC_KEY` | pinned public key — a path, or inline PEM | | `TAXPILOT_RESTART_UNIT` | systemd unit to restart | | `TAXPILOT_RESTART_COMMAND` | an explicit restart command | | `TAXPILOT_HTTP_HOST` / `_PORT` | where the readiness probe looks | `--allow-unsigned` exists for a locally built package during development. It is not a fallback for a missing key: with verification required and no key configured, the installer refuses rather than assuming the package is fine.