# Architect

> Architect hibernates idle Kubernetes pods and wakes them instantly (<50ms) when traffic arrives. No cold starts, no code changes, 30-50% lower compute costs.

Architect is a Kubernetes Availability Platform that provides pod hibernation and live migration capabilities that eliminates overprovisioning and wasted capacity. Architect adds automatic hibernation (scale-to-zero without deletion) and live migration to any workload using checkpoint/restore at the containerd shim layer, eBPF-based packet buffering, and restore in under 50ms. Hibernated pods stay registered with services, keep PVCs mounted, and preserve in-memory state across hibernate/wake cycles.

Architect is built by Loophole Labs (https://loopholelabs.io), a systems infrastructure company. This file is the complete Architect documentation; the curated index lives at https://architect.io/llms.txt.

Key capabilities:

- Automatic hibernation on idle with configurable timeouts per container
- Network-triggered wake via eBPF packet monitoring (under 50ms)
- Live migration across nodes, availability zones, and cloud providers
- Spot instance support with preemption-safe pod migration
- runc checkpoint/restore runtime support
- Health check proxy to prevent kubelet restarts during hibernation

---

# Architect Documentation

> Source: https://architect.io/docs

Documentation for Architect: Hibernate idle Kubernetes workloads. Wake them on traffic. Migrate running containers' memory state across nodes.

Architect scales Kubernetes pods to zero without [cold starts](/docs/glossary.md#cold-start). Instead of deleting idle pods, it checkpoints them: the running container's state is preserved on the node, resource requests drop to zero, and the pod stays scheduled with its Services and PVCs attached. When traffic returns, the pod wakes from its checkpoint in under 50ms and picks up exactly where it left off. No image pull, no init sequence, no JVM warmup; packets that arrive mid-wake are buffered, so nothing is dropped.

Enabling it requires no code changes: a runtime class and an annotation in your pod spec. The same checkpoint mechanism also migrates a running workload's in-memory state to its replacement when a pod moves across nodes.

## Start here

* [Quick start](/docs/quick-start.md): Hibernate and wake your first pod in minutes
* [How it works](/docs/how-it-works.md): The hibernation lifecycle and Architect's components

## Deploy

* [Installation](/docs/installation.md): Set up Architect in your cluster
* [Configuration](/docs/configuration.md): Node labels, Helm values, and pod annotations
* [Examples](/docs/examples.md): Deployment patterns for common workloads
* [Testing your application](/docs/testing.md): Validate compatibility before production

## Operate

* [Best practices](/docs/best-practices.md): Node config, capacity planning, and guidelines
* [Introspection](/docs/introspection.md): Monitoring and debugging
* [Troubleshooting](/docs/troubleshooting.md): Common issues
* [FAQ](/docs/faq.md): Compatibility, limits, and more
* [Uninstalling](/docs/installation/uninstalling.md): Remove Architect from your cluster


---

# Quick start

> Source: https://architect.io/docs/quick-start

Install Architect and watch a Valkey pod hibernate, wake instantly, and migrate across nodes with its in-memory state intact.

Architect scales Kubernetes pods to zero without unscheduling them, and wakes
or migrates them instantly with complete state. No code changes required.

**Prerequisites:**

* Kubernetes 1.33+ with at least 2 nodes
* Helm 3+
* On Amazon EKS: AL2023 AMI required (AL2 not supported)

## Install Architect

Sign into [console.architect.io](https://console.architect.io/), click
**+ Add Cluster**, and follow the instructions. The console provides a
pre-filled `helm install` command with your machine token, cluster name,
and Kubernetes distribution.

Verify the installation:

```bash
kubectl get pods -n architect
```

You should see `architect-admission-controller`, `architect-control-plane`, and
`architectd` on each labeled node. For GitOps setup, prerequisites, and
advanced Helm chart options, see the full [Installation](/docs/installation.md)
guide.

## Verify the install with the self-test

Before deploying anything of your own, confirm Architect's core functionality
works in *your* cluster: open the cluster in the
[Console](https://console.architect.io/), switch to the **Health** tab, and
click **Run self-test**. It runs hibernate, wake on exec, data persistence,
wake on network, and migration against short-lived workloads in a dedicated
namespace, and reports whether the cluster passed.

This is the fastest way to catch a cluster whose containerd config, CNI, or
kernel behaves differently from the tested matrix — worth doing after every
install or upgrade. See [Run the built-in
self-test](/docs/testing.md#run-the-built-in-self-test) for the checks, the
customization slots, and how to read a failing run's log.

## Deploy Valkey as an example

```bash
helm install \
  example-valkey oci://ghcr.io/loopholelabs/example-valkey-chart --wait
```

## Watch Valkey scale down and wake up

Watch the pod [scale down](/docs/glossary.md#hibernate-scale-down) after 10 seconds of inactivity:

```bash
watch "kubectl get pod -l app=example-valkey-valkey \
  -o custom-columns=\"\
NAME:.metadata.name,\
ARCHITECT:.metadata.labels['status\.architect\.loopholelabs\.io/valkey'],\
CPU:.spec.containers[0].resources.requests.cpu,\
MEM:.spec.containers[0].resources.requests.memory\""
```

Wake the pod up:

```bash
kubectl exec -it deployment/example-valkey-valkey \
  -c valkey -- valkey-cli ping
```

## Force Valkey migration across nodes

Find the initial node where the Valkey pod runs:

```bash
NODE=$(kubectl get pod -l app=example-valkey-valkey \
  -o jsonpath='{.items[0].spec.nodeName}')
```

Store a Valkey message in memory, on the initial node:

```bash
kubectl exec -it deployment/example-valkey-valkey -c valkey -- \
  valkey-cli set architect-quick-start \
  "This Valkey message was stored in memory on node: $NODE"
```

Confirm that the message can be read:

```bash
kubectl exec -it deployment/example-valkey-valkey -c valkey -- \
  sh -c "valkey-cli --raw get architect-quick-start; \
  echo \"Valkey pod is running on node: $NODE\""
```

Cordon the node and delete the Valkey pod:

```bash
kubectl cordon $NODE
kubectl delete pod -l app=example-valkey-valkey
```

Wait for the new pod to be running before continuing:

```bash
kubectl wait pod -l app=example-valkey-valkey \
  --for=condition=Ready --timeout=60s
```

Find the new node where the Valkey pod migrated to:

```bash
NEW_NODE=$(kubectl get pod -l app=example-valkey-valkey \
  -o jsonpath='{.items[0].spec.nodeName}')
```

Verify that Valkey message is still in memory, on the new node:

```bash
kubectl exec -it deployment/example-valkey-valkey -c valkey -- \
  sh -c "valkey-cli --raw get architect-quick-start; \
  echo \"Valkey pod is running on node: $NEW_NODE\""
```

Uncordon the initial node and delete the Valkey example:

```bash
kubectl uncordon $NODE
helm uninstall example-valkey
```

## Takeaways

You've now seen Architect's two core capabilities in action:

1. Valkey scaled to zero CPU and memory after 10 seconds of inactivity, then
   woke up instantly when you ran a command against it; all without being
   unscheduled.
2. You then forced a Valkey migration to a different node by cordoning and
   deleting the pod: the message you stored in memory was still there on the
   new node. No code changes and no special client logic needed. Valkey's
   in-memory state survived the move intact.

## More examples

Each chart below deploys a small workload with Architect annotations already
applied, so it scales down after a short idle period and wakes instantly.
Install any of them the same way, swapping in the chart name:

```bash
helm upgrade example-go \
  oci://ghcr.io/loopholelabs/example-go-chart --install --wait
```

Available charts: `example-go`, `example-java-tomcat`, `example-kafka`,
`example-php-wordpress`, `example-postgres`, `example-python`, `example-ruby`,
`example-rust-miniserve`, `example-spring-boot`.


---

# Installation

> Source: https://architect.io/docs/installation

How to install Architect: check prerequisites, install from the console, and confirm it's running.

Architect runs inside your cluster as an ordinary workload, with no host agent to
manage. You install it once per cluster, label the nodes it runs on, and your
workloads opt in with a runtime class.

Already installed via the [Quick start](/docs/quick-start.md)? Skip ahead to
[Introspection](/docs/introspection.md) or [Examples](/docs/examples.md).

## Steps

1. **[Prerequisites](/docs/installation/prerequisites.md)**: What your cluster, nodes, and tooling need.
2. **[Install](/docs/installation/install.md)**: Label your nodes and run the console's generated command.
3. **[Verify the install](/docs/installation/verification.md)**: Confirm the components registered, and diagnose a failed install.

Installing on a managed cloud? Start with [Prerequisites](/docs/installation/prerequisites.md#worker-nodes)
for the per-platform node-image requirements.


---

# Prerequisites

> Source: https://architect.io/docs/installation/prerequisites

What your cluster, nodes, and tooling need before installing Architect, with the commands to check each one.

Architect runs as an ordinary in-cluster workload, with no host agent to install.
Here is what your cluster and nodes need.

## Kubernetes

Kubernetes **1.33 or later**.

```bash
kubectl version
```

```
Client Version: v1.34.3
Kustomize Version: v5.7.1
Server Version: v1.34.3
```

## Tooling

`kubectl` and `helm` 3+ on your machine.

## Worker nodes

Architect supports these node images out of the box, with nothing to check:

| Platform   | Node image |
| ---------- | ---------- |
| Amazon EKS | AL2023     |

Bare metal or a custom image needs:

* **containerd 2.x** (not Docker, CRI-O, or containerd 1.7).
* **Linux 6.6+ recommended**, with checkpoint/restore support — 5.10+ is
  [partially supported](/docs/faq.md#can-i-use-architect-on-older-kernels).
* **amd64 or arm64**.

Check every node at once:

```bash
kubectl get nodes -o wide
```

`-o wide` also prints roles, age, version, and node IPs; the columns that matter
here are the last three (shown abbreviated):

```
NAME     STATUS   ...   OS-IMAGE            KERNEL-VERSION                 CONTAINER-RUNTIME
node-1   Ready    ...   Amazon Linux 2023   6.1.112-122.189.amzn2023      containerd://2.1.5
node-2   Ready    ...   Amazon Linux 2023   6.1.112-122.189.amzn2023      containerd://2.1.5
```

`CONTAINER-RUNTIME` should read `containerd://2.x`, and a `KERNEL-VERSION` of 6.6
or newer is recommended (5.10+ is
[partially supported](/docs/faq.md#can-i-use-architect-on-older-kernels)). On a
custom image, also confirm the kernel was built with checkpoint/restore:

```bash
# whichever your distro exposes
grep CHECKPOINT_RESTORE /boot/config-$(uname -r)
zcat /proc/config.gz | grep CHECKPOINT_RESTORE
```

```
CONFIG_CHECKPOINT_RESTORE=y
```

## Networking

* **Egress to `api.architect.io:443`** for authentication at install and heartbeats afterward.
* **Port 1337 between Architect's pods.** The daemon and control plane use it for checkpoint transfer. If you enforce pod-to-pod NetworkPolicies, allow it within the `architect` namespace.

Check egress from inside the cluster:

```bash
kubectl run egress-check --rm -it --restart=Never --image=curlimages/curl -- \
  curl -sS -o /dev/null -w '%{http_code}\n' https://api.architect.io/health
```

```
200
```

A `200` confirms DNS and egress to `api.architect.io` work; a hang or connection
error means they are blocked.

## S3 (optional)

Only needed for [persistent checkpoints](/docs/configuration/helm-values.md#persistent-checkpoint-storage-s3).
Any S3-compatible store works (AWS S3, Google Cloud Storage, MinIO, Cloudflare R2).
You provide the endpoint, region, bucket, and credentials as Helm values.


---

# Install

> Source: https://architect.io/docs/installation/install

Install Architect from the console: label your nodes, run the generated Helm command, and confirm the components are running.

Make sure your cluster meets the [prerequisites](/docs/installation/prerequisites.md) first.

You install Architect from the console. Sign into
[console.architect.io](https://console.architect.io/) and click **+ Add Cluster**;
it walks you through the three steps below with your machine token, cluster name,
and distribution already filled in.

## 1. Label the always-on nodes

Architect's control plane needs somewhere stable to run. Label at least one
always-on node (for example an on-demand instance) with `critical-node`:

```bash
kubectl label nodes <node> architect.loopholelabs.io/critical-node=true
```

## 2. Label the workload nodes

Label at least two of the nodes that run your own workloads with `node`.
Ephemeral nodes (for example spot instances) are a good fit, since Architect
migrates these workloads to a new node and scales them to zero when idle:

```bash
kubectl label nodes <node> architect.loopholelabs.io/node=true
```

See [Configuration: Node labels](/docs/configuration/node-labels.md) for what each
label means.

## 3. Install the chart

Run the command the console generated. The token, cluster name, and distribution
are filled in for you:

```bash
helm install architect oci://ghcr.io/loopholelabs/architect-chart \
  --namespace architect --create-namespace \
  --set kubernetesDistro=<distro> \
  --set apiUrl=https://api.architect.io \
  --set machineToken=<your-machine-token> \
  --set clusterName=<cluster-name>
```

Installing via GitOps? Reference the token from a Secret with `secretRef` instead
of passing it inline. See [Configuration: Helm values](/docs/configuration/helm-values.md#authentication).

## Confirm the install

A quick smoke check. For a full diagnostic walkthrough, see
[Verify the install](/docs/installation/verification.md).

```bash
kubectl get pods -n architect
```

You should see `architect-admission-controller`, `architect-control-plane`,
`architect-self-test` (the [built-in self-test](/docs/testing.md#run-the-built-in-self-test);
absent if you disabled it with `architectSelfTestEnabled: false`), and an
`architectd` pod on each labeled node. Then
[deploy an example application](/docs/quick-start.md#deploy-valkey-as-an-example).

For every chart value and tuning option, see
[Configuration: Helm values](/docs/configuration/helm-values.md).


---

# Verify the install

> Source: https://architect.io/docs/installation/verification

Confirm Architect installed correctly (pods running, RuntimeClass registered, nodes labeled, daemon scheduled) and diagnose a failed install.

After [installing](/docs/installation/install.md), work through these checks. Each
one isolates a different part of the install, so a failure points you straight
at the cause.

## 1. The components are running

```bash
kubectl get pods -n architect
```

You should see the four components (three if you disabled the self-test with
`architectSelfTestEnabled: false`), all `Running`:

```
NAME                                          READY   STATUS    RESTARTS   AGE
architect-admission-controller-<hash>-<id>    1/1     Running   0          1m
architect-control-plane-<hash>-<id>           1/1     Running   0          1m
architect-self-test-<hash>-<id>               1/1     Running   0          1m
architectd-<id>                               1/1     Running   0          1m
```

`architectd` is a DaemonSet, so expect one `architectd-<id>` pod per labeled node.

## 2. The RuntimeClass is registered

```bash
kubectl get runtimeclass runc-architect
```

```
NAME             HANDLER          AGE
runc-architect   runc-architect   1m
```

`architectd` creates this on startup. If it's missing, the daemon hasn't
finished initializing. Wait a moment and recheck, then look at its logs
(step 4).

## 3. Your nodes are labeled

```bash
kubectl get nodes -L architect.loopholelabs.io/node,architect.loopholelabs.io/critical-node
```

Every node that should run Architect workloads needs `node=true`, and at least
one stable node needs `critical-node=true`:

```
NAME     ...   NODE   CRITICAL-NODE
node-1         true   true
node-2         true
```

## 4. The daemon is on every labeled node

```bash
kubectl get pods -n architect -l app.kubernetes.io/name=architectd -o wide
```

There should be a `Running` `architectd` pod on each `node=true` node. To read
its logs (it also surfaces the shim's logs):

```bash
kubectl logs -n architect -l app.kubernetes.io/name=architectd --tail=50
```

## 5. The console shows the cluster as healthy

Open [console.architect.io](https://console.architect.io/) and select your
cluster. Status should be healthy, with a heartbeat in the last \~30 seconds. If
it stays pending, the daemon can't reach `api.architect.io` (see the table
below).

## 6. A workload actually hibernates

Deploy a test workload and confirm it hibernates. The [Quick start](/docs/quick-start.md)
walks through a Valkey example end to end; the key signal is the per-container
status label flipping after the idle timeout:

```bash
# For a container named "valkey" in your pod:
kubectl get pod <pod> -o jsonpath='{.metadata.labels.status\.architect\.loopholelabs\.io/valkey}'
# RUNNING  -> SCALED_DOWN once idle, and back to RUNNING when woken
```

## Diagnose a failed install

| Symptom                               | Likely cause                                                                         | What to check                                                                                          |
| ------------------------------------- | ------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------ |
| No `architectd` pod on a node         | node isn't labeled                                                                   | `kubectl get nodes -L architect.loopholelabs.io/node`; label it                                        |
| A component is `Pending`              | scheduling (taints, resources, `critical-node` unlabeled)                            | `kubectl describe pod -n architect <pod>`                                                              |
| A component is `CrashLoopBackOff`     | bad config or invalid machine token                                                  | `kubectl logs -n architect <pod>` and `kubectl describe pod`                                           |
| `ImagePullBackOff`                    | cluster can't pull from `ghcr.io/loopholelabs`                                       | registry egress / image pull secrets                                                                   |
| `runc-architect` RuntimeClass missing | `architectd` hasn't finished initializing                                            | wait, then `kubectl logs -n architect -l app.kubernetes.io/name=architectd`                            |
| Console stuck pending                 | daemon can't reach `api.architect.io:443`                                            | egress/NetworkPolicy, in-cluster DNS, machine token                                                    |
| Your pod never hibernates             | `runtimeClassName` not set, container not in `managed-containers`, or node unlabeled | confirm `spec.runtimeClassName: runc-architect` and the [annotations](/docs/configuration/annotations.md) |

Still stuck? See [Troubleshooting](/docs/troubleshooting.md).


---

# Uninstalling

> Source: https://architect.io/docs/installation/uninstalling

Safely uninstall Architect without breaking workloads: remove runtimeClassName, uninstall the Helm release, and clean up node labels.

To uninstall Architect without disrupting running workloads, follow these steps
in order. **Failure to update workloads before uninstalling will cause pod
errors.**

## 1. Prepare workloads

Before running the uninstall command, update your workload configurations to
ensure they no longer depend on the Architect runtime:

* **Remove `runtimeClassName`** from all active workloads. If you uninstall
  Architect while `runtimeClassName` is still set, all managed pods will
  immediately enter an `Error` state.
* **Remove annotations** (optional): delete any `architect.loopholelabs.io/*`
  annotations to keep your manifests clean.

## 2. Uninstall the Helm release

Once your workloads are updated, remove the Architect release:

```bash
helm uninstall -n architect architect
```

This also removes the self-test component's dedicated namespace
(`architect-self-test` by default) and any in-flight test workloads in it;
verify with `kubectl get ns architect-self-test`.

## 3. Clean up node labels

Remove any remaining Architect-specific labels from your cluster nodes:

```bash
kubectl label nodes <node-name> architect.loopholelabs.io/node-
kubectl label nodes <node-name> architect.loopholelabs.io/critical-node-
```


---

# Configuration

> Source: https://architect.io/docs/configuration

Reference for configuring Architect: node labels, Helm chart values, and pod annotations.

Architect is configured through three surfaces, each set by a different person at
a different stage:

* **[Node labels](/docs/configuration/node-labels.md)** mark which nodes Architect runs on. Set by the cluster admin when preparing the cluster.
* **[Helm values](/docs/configuration/helm-values.md)** configure the install. Set by the operator at install time.
* **[Annotations](/docs/configuration/annotations.md)** control per-workload behavior. Set by the application author in the pod spec.

A pod opts into Architect with `runtimeClassName: runc-architect` and lists the
containers to manage with the [`managed-containers`](/docs/configuration/annotations.md#managed-containers)
annotation. See [How it works](/docs/how-it-works.md) for the lifecycle this enables.


---

# Node labels

> Source: https://architect.io/docs/configuration/node-labels

The node labels that tell Architect which nodes to run on and which to keep stable.

Architect only schedules onto nodes you opt in with these labels, applied with
`kubectl label node <node> <label>=true`.

| Label                                     | Value  | Applies to      | Description                                                                                                                                              |
| ----------------------------------------- | ------ | --------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `architect.loopholelabs.io/node`          | `true` | Workload nodes  | Nodes where the `architectd` daemon and your managed workloads run. Required on every node that should run Architect workloads.                          |
| `architect.loopholelabs.io/critical-node` | `true` | Always-on nodes | Nodes for the control plane and admission controller. Required on at least one stable node you do not drain, so checkpoint coordination stays available. |

The console calls `critical-node` nodes "always-on" and `node` nodes
"ephemeral". The [install flow](/docs/installation/install.md) applies these labels
as its first step.


---

# Helm values

> Source: https://architect.io/docs/configuration/helm-values

Reference for every Architect Helm chart value: required settings, authentication, S3 storage, placement and sizing, images, and experimental flags.

Values for the `oci://ghcr.io/loopholelabs/architect-chart` chart. The
[console install flow](/docs/installation/install.md) sets the required ones for
you. Run `helm show values oci://ghcr.io/loopholelabs/architect-chart` to print
the live defaults.

## Required

| Value         | Type   | Default | Description                                              |
| ------------- | ------ | ------- | -------------------------------------------------------- |
| `clusterName` | string | `""`    | Identifier for this cluster in the console. Must be set. |

## Authentication

Provide the token inline with `machineToken`, or reference a Secret that holds it
with `secretRef`. Prefer `secretRef` for any non-interactive or GitOps install:
`--set machineToken` writes the token into your shell history and into the stored
Helm release values (`helm get values`), whereas a referenced Secret keeps it out
of both (and works with External Secrets, Sealed Secrets, and similar).

| Value              | Type   | Default              | Description                                                                                   |
| ------------------ | ------ | -------------------- | --------------------------------------------------------------------------------------------- |
| `machineToken`     | string | `""`                 | Install token, set inline. Mutually exclusive with `secretRef`.                               |
| `secretRef`        | string | `""`                 | Name of an existing Secret holding the token. The key inside it must be named `machineToken`. |
| `secretRefDefault` | string | `architectd-secrets` | Name of the Secret the chart creates when `secretRef` is unset.                               |

## Cluster

| Value              | Type   | Default                    | Description                                                       |
| ------------------ | ------ | -------------------------- | ----------------------------------------------------------------- |
| `kubernetesDistro` | string | `kind`                     | The cluster's distribution (the console sets this).               |
| `apiUrl`           | string | `https://api.architect.io` | Endpoint Architect authenticates against and sends heartbeats to. |
| `imagePullPolicy`  | string | `IfNotPresent`             | Pull policy for all Architect images.                             |

## Persistent checkpoint storage (S3)

Optional. Enables the daemon to store checkpoints for the
[start-from-persistent-checkpoint](/docs/configuration/annotations.md#start-from-persistent-checkpoint)
annotation in an S3-compatible bucket. Set `s3Bucket` and `s3Region` to turn it
on; everything else depends on where the bucket lives and how you authenticate.
When `secretRef` is unset, these values are written into the chart-created
Secret.

| Value               | Type   | Default | Description                                                                           |
| ------------------- | ------ | ------- | ------------------------------------------------------------------------------------- |
| `s3Bucket`          | string | `""`    | Bucket name. Required to enable S3.                                                   |
| `s3Region`          | string | `""`    | Bucket region. Required to enable S3.                                                 |
| `s3Endpoint`        | string | `""`    | Endpoint URL for an S3-compatible store (MinIO, Garage). Leave empty for real AWS S3. |
| `s3AccessKeyID`     | string | `""`    | Static access key ID. Leave empty to use workload identity (see below).               |
| `s3SecretAccessKey` | string | `""`    | Static secret access key. Leave empty to use workload identity (see below).           |

### Credentials

The daemon picks its credential source from what you set:

* **Static keys** — set `s3AccessKeyID` and `s3SecretAccessKey`. Required for
  S3-compatible stores (MinIO, Garage) and works against AWS S3.
* **Workload identity** — leave both keys empty and the daemon resolves
  credentials via the AWS SDK default chain (IRSA, EKS Pod Identity, instance
  profile, environment). Use this on EKS when a policy forbids long-lived keys.
  The daemon logs the resolved credential mode at startup.

Because leaving the keys empty now means "use ambient credentials" rather than
"S3 off", a bucket set with keys half-configured is treated as enabled and would
fail at the first upload rather than at startup. Set both keys or neither.

For **IRSA**, attach the role ARN to the daemon ServiceAccount via
[`architectdServiceAccountAnnotations`](#service-account):

```yaml
architectdServiceAccountAnnotations:
  eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/architect-s3
```

For **EKS Pod Identity**, no chart change is needed: create the Pod Identity
association on the AWS side against the `architectd-installer-daemon`
ServiceAccount in the release namespace.

The IAM role (IRSA or Pod Identity) needs these actions on the bucket and its
objects, since the transfer manager does concurrent multipart transfers:

```json
{
  "Effect": "Allow",
  "Action": [
    "s3:GetObject",
    "s3:PutObject",
    "s3:DeleteObject",
    "s3:AbortMultipartUpload",
    "s3:ListMultipartUploadParts",
    "s3:ListBucketMultipartUploads"
  ],
  "Resource": [
    "arn:aws:s3:::my-bucket",
    "arn:aws:s3:::my-bucket/*"
  ]
}
```

### Service account

| Value                                 | Type | Default | Description                                                                                                                 |
| ------------------------------------- | ---- | ------- | --------------------------------------------------------------------------------------------------------------------------- |
| `architectdServiceAccountAnnotations` | map  | `{}`    | Annotations added to the daemon (`architectd-installer-daemon`) ServiceAccount, e.g. the IRSA `eks.amazonaws.com/role-arn`. |

## Placement and sizing

Each component (`architectd`, `architectAdmissionController`, `architectControlPlane`)
takes a node selector, tolerations, and resource requests/limits.

| Value pattern             | Type | Default | Description                                     |
| ------------------------- | ---- | ------- | ----------------------------------------------- |
| `<component>NodeSelector` | map  | `{}`    | Node selector for the component's pods.         |
| `<component>Tolerations`  | list | `[]`    | Tolerations for the component's pods.           |
| `<component>Resources`    | map  | `{}`    | Resource requests and limits for the component. |
| `architectdHostAliases`   | list | `[]`    | Extra host aliases for `architectd` pods.       |

## Images

Each component image can be overridden; leave blank to use the pinned default.

| Value                                        | Type   | Default                                                                |
| -------------------------------------------- | ------ | ---------------------------------------------------------------------- |
| `architectdImage`                            | string | `ghcr.io/loopholelabs/architectd:latest`                               |
| `architectdInstallerImage`                   | string | `ghcr.io/loopholelabs/architectd-installer:latest`                     |
| `architectShimRuncImage`                     | string | `ghcr.io/loopholelabs/architect-shim-runc:latest`                      |
| `architectAdmissionControllerImage`          | string | `ghcr.io/loopholelabs/architect-admission-controller:latest`           |
| `architectAdmissionControllerInstallerImage` | string | `ghcr.io/loopholelabs/architect-admission-controller-installer:latest` |
| `architectControlPlaneImage`                 | string | `ghcr.io/loopholelabs/architect-control-plane:latest`                  |
| `architectHealthCheckProxyImage`             | string | `ghcr.io/loopholelabs/architect-health-check-proxy:latest`             |

## Self-test

The self-test component deploys short-lived workloads into a dedicated
testing namespace and runs core-functionality checks on demand from the
[Console](https://console.architect.io/). It is enabled by default; set
`architectSelfTestEnabled: false` to opt out. See
[Testing your application](/docs/testing.md#run-the-built-in-self-test) for how to
use it.

| Value                            | Type     | Default                                           | Description                                                               |
| -------------------------------- | -------- | ------------------------------------------------- | ------------------------------------------------------------------------- |
| `architectSelfTestEnabled`       | bool     | `true`                                            | Deploy the self-test component and advertise it to the Console.           |
| `architectSelfTestNamespace`     | string   | `architect-self-test`                             | Namespace the component creates test workloads in (created by the chart). |
| `architectSelfTestPollInterval`  | duration | `5s`                                              | How often the component polls the API for a pending run.                  |
| `architectSelfTestImage`         | string   | `ghcr.io/loopholelabs/architect-self-test:latest` | Self-test component image override.                                       |
| `architectSelfTestWorkloadImage` | string   | `ghcr.io/loopholelabs/example-go:latest`          | Test workload image override.                                             |
| `architectSelfTestNodeSelector`  | map      | `{}`                                              | Node selector for the component's pod.                                    |
| `architectSelfTestResources`     | map      | `50m`/`64Mi` req, `250m`/`128Mi` lim              | Component resource requests and limits.                                   |
| `architectSelfTestTolerations`   | list     | `[]`                                              | Tolerations for the component's pod.                                      |

## Experimental

> **Warning:** Only enable these when advised by Loophole Labs.

| Value                                          | Type     | Default                                             | Description                                        |
| ---------------------------------------------- | -------- | --------------------------------------------------- | -------------------------------------------------- |
| `features.liveMigrationBuffering`              | bool     | `false`                                             | Buffers in-flight traffic across a live migration. |
| `architectRouterAddr`                          | string   | `/ip4/0.0.0.0/tcp/8080`                             | Router listen multiaddr.                           |
| `architectRouterPort`                          | int      | `8080`                                              | Router port.                                       |
| `architectRouterGenericXDP`                    | bool     | `true`                                              | Use generic XDP (for drivers without native XDP).  |
| `architectRouterIngressIface`                  | string   | `eth0`                                              | Interface the router attaches to.                  |
| `architectRouterPassthroughPorts`              | string   | `""`                                                | Ports that bypass the router.                      |
| `architectRouterResources`                     | map      | `250m`/`256Mi` req, `1`/`512Mi` lim                 | Router resource requests and limits.               |
| `architectRouterShimPort`                      | int      | `8081`                                              | Router-shim port.                                  |
| `architectRouterShimResources`                 | map      | `100m`/`128Mi` req, `500m`/`256Mi` lim              | Router-shim resource requests and limits.          |
| `architectRouterShimTimeout`                   | duration | `10s`                                               | Router-shim request timeout.                       |
| `architectRouterImage`                         | string   | `ghcr.io/loopholelabs/architect-router:latest`      | Router image override.                             |
| `architectRouterShimImage`                     | string   | `ghcr.io/loopholelabs/architect-router-shim:latest` | Router-shim image override.                        |
| `architectShadowServiceEnabled`                | bool     | `false`                                             | Enable shadow Services for live network migration. |
| `architectShadowServicePortMin`                | int      | `30000`                                             | Low end of the shadow-service port range.          |
| `architectShadowServicePortMax`                | int      | `32767`                                             | High end of the shadow-service port range.         |
| `architectShadowServiceRouterPodLabelSelector` | string   | `app.kubernetes.io/name=architect-router`           | Selector for router pods.                          |


---

# Annotations

> Source: https://architect.io/docs/configuration/annotations

Reference for every Architect pod annotation: managed containers, scale-down timing, network wake, health-check proxy, shadow ports, migration, and persistent checkpoints.

Pod annotations control per-workload behavior. Set them on the pod template of a
pod that has `runtimeClassName: runc-architect`. Each entry lists its default and
what it requires.

## managed-containers

```yaml
architect.loopholelabs.io/managed-containers: '["container-1", "container-2"]'
```

Which containers Architect manages. Unlisted containers run normally.

**Default:** none. · **Requires:** `runtimeClassName: runc-architect`.

## scaledown-durations

```yaml
architect.loopholelabs.io/scaledown-durations: '{"container-1":"30s", "container-2":"60s"}'
```

Idle time before a container hibernates.

**Default:** `60s`. · **Requires:** `managed-containers`.

## initial-scaledown-delays

```yaml
architect.loopholelabs.io/initial-scaledown-delays: '{"container-1":"90s"}'
```

Grace period (a Go duration string) that suppresses hibernation for the
configured duration after the container's first scale-up. Useful for slow-starting workloads
(for example JVMs whose readiness probes take longer than `scaledown-durations`)
so they are not hibernated mid-startup. Normal activity-based scale-down resumes
after the window elapses. The window is not re-armed after a migration or
post-scale-down restart, since the workload is already past its slow startup by
then.

**Default:** `0` (disabled), values clamped to 24h. · **Requires:** `managed-containers`.

## network-monitor

```yaml
architect.loopholelabs.io/network-monitor: '{"container-1":"packets", "container-2":"connections"}'
```

Enables network-based wake: a scaled-down container wakes when it receives
network traffic. An eBPF program in the pod's network namespace watches the
container's declared ports and triggers a scale-up. Without this annotation, the
only way to wake a scaled-down container is `kubectl exec`.

Modes:

* `packets`: wake on any incoming TCP/UDP packet on a tracked port. Suits sporadic request/response workloads such as HTTP APIs and webhook receivers.
* `connections`: TCP only. Wake on connection establishment and stay awake while any TCP connection is open. Suits long-lived connection patterns such as databases, message brokers, and gRPC servers. A client that holds a pooled connection open indefinitely keeps the container awake.

Activity is tracked per port. Architect monitors only the ports the container
declares in its `ports` array. Shadow ports injected by
[`health-check-proxy`](#health-check-proxy) and [`shadow-ports`](#shadow-ports)
are added to that array so Kubernetes Services can target them, but Architect
ignores traffic on them when assessing activity. The traffic still reaches the
application; it just does not keep the container running.

Activity is also scoped per container. Sidecars sharing the pod's network
namespace (Istio sidecars, fluentd, and the like) do not keep the managed
container awake, and outbound traffic from an ephemeral source port does not
count. A workload that only does outbound traffic from ephemeral ports should
use [`disable-autoscaledown-containers`](#disable-autoscaledown-containers).

**Default:** off. · **Requires:** `managed-containers`.

## health-check-proxy

```yaml
architect.loopholelabs.io/health-check-proxy: '{"mappings":[{"containerName":"app","appPort":8080,"shadowPort":9080}]}'
```

Lets kubelet liveness, readiness, and startup probes pass while the container is
scaled down, without waking it. Probes are pointed at the `shadowPort`; Architect
injects an `architect-health-check-proxy` sidecar that forwards probes to the
application while it runs and answers them itself while it is scaled down, so
kubelet keeps seeing a healthy response. Without this, every probe hits the
application port and counts as activity, so a probed container never scales down.

Mapping fields:

* `containerName` (required): a container in `managed-containers`.
* `appPort` (required, 1 to 65535): the application's real probe port.
* `shadowPort` (required, 1 to 65535): the port to point probes at.

Duplicate `shadowPort` values across mappings are dropped with a warning. The
sidecar is not added (and a warning is logged) if `managed-containers` or
`network-monitor` is missing.

See [Examples](/docs/examples.md#web-api) for a worked example, and
[Troubleshooting](/docs/troubleshooting.md#health-probes-wake-the-container) if
probes still wake the container.

**Default:** none. · **Requires:** `managed-containers`, `network-monitor`.

## shadow-ports

```yaml
architect.loopholelabs.io/shadow-ports: '{"mappings":[{"containerName":"app","appPort":9090,"shadowPort":29090}]}'
```

Lets a scraper (Prometheus, an external health check, a debug tool) reach an
application port without counting as activity, so regular scrapes do not keep the
container awake. The scraper is pointed at the `shadowPort`; traffic still reaches
the application on the real port, and the application is unaware of the redirect.
Without this, a recurring scrape looks like continuous traffic and the container
never scales down.

Mapping fields:

* `containerName` (required): a container in `managed-containers`.
* `appPort` (required, 1 to 65535): the real port the application listens on.
* `shadowPort` (required, 1 to 65535): the port to point the scraper at.

Duplicate `shadowPort` values are dropped with a warning. The shadow ports are
not added (and a warning is logged) if `managed-containers` or `network-monitor`
is missing. When the scraper cannot be moved to a different port (for example it
is hard-coded in Prometheus discovery), use
[`ignore-activity-ports`](#ignore-activity-ports) instead.

See [Examples](/docs/examples.md#metrics-scraping) for a worked example, and
[Troubleshooting](/docs/troubleshooting.md#scrape-traffic-wakes-the-container) if
scrapes still wake the container.

**Default:** none. · **Requires:** `managed-containers`, `network-monitor`.

## ignore-activity-ports

```yaml
architect.loopholelabs.io/ignore-activity-ports: '{"container-1":[9091, 9100]}'
```

Marks specific ports on the container's existing port spec as conntrack-bypassed,
so traffic to them does not count as activity. Unlike [`shadow-ports`](#shadow-ports)
there is no DNAT and no new port is injected; the operator asserts that the listed
ports are already declared on the container and the application already listens on
them. Use this when a metrics scraper hits the real application port directly and
should not keep the workload awake.

**Default:** none. · **Requires:** `managed-containers`, `network-monitor`.

## postmigration-autoscaleup-containers

```yaml
architect.loopholelabs.io/postmigration-autoscaleup-containers: '["container-1"]'
```

Containers that automatically scale up after migration. By default they stay
hibernated to avoid a thundering herd.

**Default:** off (containers stay hibernated after migration). · **Requires:** `managed-containers`.

## disable-autoscaledown-containers

```yaml
architect.loopholelabs.io/disable-autoscaledown-containers: '["container-1"]'
```

Prevents automatic hibernation. Useful for background jobs that should migrate
but not hibernate on idle.

**Default:** off (containers hibernate on idle). · **Requires:** `managed-containers`.

## sleep-wake-groups

```yaml
architect.loopholelabs.io/sleep-wake-groups: '{"group-1":["container-1", "container-2"]}'
```

Groups managed containers in the same pod so they hibernate and wake together.
When any member of a group wakes (e.g. by `kubectl exec`, by
[`network-monitor`](#network-monitor) traffic, or by a post-migration
auto-scale-up), Architect wakes every other member of that group too. A member
is also held awake while any other member of its group is still active, and only
hibernates once the whole group has gone idle.

Use this when one container's work depends on another being awake, for example a
tools container that, on wake, must reach a companion container in the same pod.
Without it, each managed container sleeps and wakes on its own activity timer
independently.

The annotation is a JSON object keyed by an operator-chosen group name with the
member container names as the value. Members must be containers in the same pod
listed in `managed-containers`; a container may appear in more than one group, in
which case waking it wakes the union of those groups. A group with a single
member has no effect. A member that names a container which does not exist in the
pod or is not in `managed-containers` is ignored and logged as a warning, so a
typo'd name does not silently disable the group.

A group member that also has [`disable-autoscaledown`](#disable-autoscaledown-containers)
never hibernates, so the group as a whole never goes fully idle. Its group-mates
still hibernate on their own once that always-on member is quiet, and are held
awake again whenever it has genuine activity. Give such a member
[`network-monitor`](#network-monitor) so its traffic is tracked as activity;
without it, only its `kubectl exec` activity keeps the group awake.

**Default:** none (each container sleeps and wakes independently). · **Requires:** `managed-containers`.

## scaleup-timeout-containers

```yaml
architect.loopholelabs.io/scaleup-timeout-containers: '{"container-1": "60s"}'
```

How long to wait for a checkpoint during startup.

**Default:** `30s`.

## migrate-emptydir-containers

```yaml
architect.loopholelabs.io/migrate-emptydir-containers: '["container-1"]'
```

Preserves emptyDir volume data during migration. By default, emptyDir volumes are
not migrated.

**Default:** off (emptyDir not migrated). · **Requires:** `managed-containers`.

## sparse-files-containers

```yaml
architect.loopholelabs.io/sparse-files-containers: '{"container-1": ["/var/cache/app.db"]}'
```

Recreates the listed files as sparse files (same size and mode, contents zeroed)
at the destination instead of copying their bytes through the upper-layer
snapshot, and skips them on the source so the migration avoids the per-byte
snapshot cost. Use for workloads that re-scan or rewrite the file post-restore
(caches, generated artifacts, scratch space). Workloads that read the original
contents after migration see zeros.

**Default:** none.

## lazy-pages-migration-containers

> **Warning:** Experimental. Only enable when advised by Loophole Labs.

```yaml
architect.loopholelabs.io/lazy-pages-migration-containers: '["container-1"]'
```

Enables CRIU lazy-pages migration, fetching memory pages on demand from the source
pod during restore instead of copying everything upfront. Helps with memory-heavy
containers. Also applies to hibernated (scaled-down) containers, whose on-disk
checkpoint is served lazily during evacuation. Falls back to eager migration if
lazy-pages migration fails before the checkpoint is committed to the lazy layout;
after that, a restore that cannot fetch its pages from the source starts fresh.
The console reports the background page transfer as paired started/completed
events, including how many pages were transferred.

**Default:** off.

## lazy-pages-restore-timeout-containers

> **Warning:** Experimental. Only enable when advised by Loophole Labs.

```yaml
architect.loopholelabs.io/lazy-pages-restore-timeout-containers: '{"container-1":"30s"}'
```

Bounds how long a lazy-pages restore waits for memory pages from the source before
falling back to a fresh start. Useful when the source page-server is unreachable
but the underlying TCP connection appears healthy. Values are Go duration strings.

**Default:** `0` (disabled), clamped to 24h.

## rewrite-listener-addresses-containers

```yaml
architect.loopholelabs.io/rewrite-listener-addresses-containers: '["container-1"]'
```

Rewrites listener socket addresses in CRIU checkpoints during migration. When an
application binds to the pod IP (rather than `0.0.0.0`), the listener address
becomes invalid on the destination pod. This rewrites those addresses to
`INADDR_ANY` (`0.0.0.0`) or `in6addr_any` (`::`) so the restore succeeds.

**Default:** off.

## rewrite-established-addresses-containers

```yaml
architect.loopholelabs.io/rewrite-established-addresses-containers: '["container-1"]'
```

Rewrites the source IP of established TCP connections in CRIU checkpoints during
migration. The source pod's IP no longer exists on the destination pod, which
causes CRIU's socket restore to fail. This rewrites the source address to the new
pod's IP (read from `/etc/hosts`). Supports IPv4 and IPv6.

**Default:** off.

## start-from-persistent-checkpoint

```yaml
# Same namespace (name only):
architect.loopholelabs.io/start-from-persistent-checkpoint: "persistent-checkpoint-name"
# Cross-namespace (namespace/name):
architect.loopholelabs.io/start-from-persistent-checkpoint: "namespace/persistent-checkpoint-name"
```

Restore from a `PersistentCheckpoint` CRD on startup. With a bare name the
`PersistentCheckpoint` is looked up in the pod's namespace; use `namespace/name`
to reference one in a different namespace. When set, this takes priority over
pod-template-hash-based Checkpoint CRDs: on any failure (not found, empty,
download error, `registry` storage) the pod starts fresh rather than falling back
to the migration path.

**Default:** none.

## checkpoint-engine

> **Warning:** Experimental. Only enable when advised by Loophole Labs.

```yaml
architect.loopholelabs.io/checkpoint-engine: "cruise"
```

Selects the checkpoint/restore engine for the pod's managed containers. Set it to
`cruise` to route runc checkpoint/restore to the in-tree cruise engine instead of
CRIU. This is a pod-global setting; unmanaged containers in the pod are never
checkpointed.

**Default:** `criu`.


---

# Examples

> Source: https://architect.io/docs/examples

Annotated deployment patterns for Architect: web APIs, metrics scraping, sidecars, dev environments, and explicit persistent checkpoints (optionally to S3).

## Web API

Stateless service with network-based wake and health check proxy.

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: api-service
spec:
  replicas: 10
  selector:
    matchLabels:
      app: api-service
  template:
    metadata:
      labels:
        app: api-service
      annotations:
        architect.loopholelabs.io/managed-containers: '["api"]'
        architect.loopholelabs.io/scaledown-durations: '{"api":"30s"}'
        architect.loopholelabs.io/network-monitor: '{"api":"connections"}'
        architect.loopholelabs.io/health-check-proxy: '{"mappings":[{"containerName":"api","appPort":8080,"shadowPort":9080}]}'
    spec:
      runtimeClassName: runc-architect
      containers:
        - name: api
          image: mycompany/api:v2.1
          ports:
            - containerPort: 8080
          resources:
            requests:
              memory: "1Gi"
              cpu: "500m"
            limits:
              memory: "2Gi"
              cpu: "1000m"
```

## Metrics scraping

Expose `/metrics` on a shadow port so Prometheus can scrape without waking
the container. `health-check-proxy` keeps liveness and readiness probes
passing while the container is scaled down; `shadow-ports` routes scrape
traffic to the real metrics port without counting it as activity. See
[Configuration → shadow-ports](/docs/configuration/annotations.md#shadow-ports).

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: api-service
spec:
  replicas: 10
  selector:
    matchLabels:
      app: api-service
  template:
    metadata:
      labels:
        app: api-service
      annotations:
        architect.loopholelabs.io/managed-containers: '["api"]'
        architect.loopholelabs.io/scaledown-durations: '{"api":"30s"}'
        architect.loopholelabs.io/network-monitor: '{"api":"connections"}'
        architect.loopholelabs.io/health-check-proxy: '{"mappings":[{"containerName":"api","appPort":8080,"shadowPort":9080}]}'
        architect.loopholelabs.io/shadow-ports: '{"mappings":[{"containerName":"api","appPort":9090,"shadowPort":29090}]}'
    spec:
      runtimeClassName: runc-architect
      containers:
        - name: api
          image: mycompany/api:v2.1
          ports:
            - containerPort: 8080
            - containerPort: 9090 # metrics
          livenessProbe:
            httpGet:
              path: /healthz
              port: 9080 # shadow port, not 8080
          readinessProbe:
            httpGet:
              path: /readyz
              port: 9080
```

Point the scraper at the shadow port. With the Prometheus Operator:

```yaml
apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: api-service
spec:
  selector:
    matchLabels:
      app: api-service
  podMetricsEndpoints:
    - port: shadow-29090 # name auto-assigned by the admission controller
      path: /metrics
```

The shadow port appears on the container spec as `shadow-<port>` (TCP). Static
Prometheus `scrape_configs` work the same way: target port `29090` on the
pod IP.

## Microservices with sidecar

Only the main container hibernates. Sidecars are excluded from the managed list.

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: order-service
spec:
  replicas: 15
  selector:
    matchLabels:
      app: order-service
  template:
    metadata:
      labels:
        app: order-service
      annotations:
        architect.loopholelabs.io/managed-containers: '["order-service"]'
        architect.loopholelabs.io/scaledown-durations: '{"order-service":"60s"}'
    spec:
      runtimeClassName: runc-architect
      containers:
        - name: order-service
          image: mycompany/order-service:v1.5
          ports:
            - containerPort: 8080
        - name: logging-agent
          image: fluentd:latest
```

## Development environment

Short idle timeout for per-developer pods that are mostly idle.

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: dev-environment
  namespace: development
spec:
  replicas: 50
  selector:
    matchLabels:
      app: dev-environment
  template:
    metadata:
      labels:
        app: dev-environment
      annotations:
        architect.loopholelabs.io/managed-containers: '["dev-container"]'
        architect.loopholelabs.io/scaledown-durations: '{"dev-container":"5s"}'
    spec:
      runtimeClassName: runc-architect
      containers:
        - name: dev-container
          image: mycompany/dev-env:latest
          resources:
            requests:
              memory: "4Gi"
              cpu: "2000m"
```

## PersistentCheckpoint

`PersistentCheckpoint` CRDs create checkpoints explicitly while the pod keeps
running. Useful for golden images, backup snapshots, or pre-migration
checkpoints. `PersistentCheckpoint` produces one checkpoint per managed
container.

### Deployment

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: valkey-cache
spec:
  replicas: 1
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: valkey
  template:
    metadata:
      labels:
        app: valkey
      annotations:
        architect.loopholelabs.io/managed-containers: '["valkey"]'
        # Restore from PersistentCheckpoint (same namespace; use namespace/name for cross-namespace)
        architect.loopholelabs.io/start-from-persistent-checkpoint: "persistent-checkpoint-demo"
    spec:
      runtimeClassName: runc-architect
      containers:
        - name: valkey
          image: valkey/valkey:latest
          ports:
            - containerPort: 6379
          resources:
            requests:
              memory: "256Mi"
              cpu: "100m"
```

### Creating a checkpoint

```yaml
apiVersion: architect.loopholelabs.io/v1
kind: PersistentCheckpoint
metadata:
  name: persistent-checkpoint-demo
  namespace: default
spec:
  podName: valkey-cache-7d7f78c4f7-5f6ss
```

```bash
kubectl apply -f persistentcheckpoint.yaml

# Verify (populated once the checkpoint has been created and, if configured, uploaded)
kubectl get persistentcheckpoint persistent-checkpoint-demo \
  -o jsonpath='{.spec.checkpoints}'
```

### Storing checkpoints in S3

Set `spec.storage: s3` to upload the checkpoint archive (`tar.zst`) to a
configured S3-compatible store:

```yaml
apiVersion: architect.loopholelabs.io/v1
kind: PersistentCheckpoint
metadata:
  name: persistent-checkpoint-demo
  namespace: default
spec:
  podName: valkey-cache-7d7f78c4f7-5f6ss
  storage: s3
```

Deleting the CRD cleans up both the local archive and the S3 object. S3
requires the daemon to be configured with S3 credentials at install time; see
[Installation](/docs/installation.md).

### Restoring from a checkpoint

New deployments can restore from an existing checkpoint (same namespace; use
`namespace/name` for cross-namespace):

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: valkey-from-checkpoint
spec:
  replicas: 1
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: valkey-restored
  template:
    metadata:
      labels:
        app: valkey-restored
      annotations:
        architect.loopholelabs.io/managed-containers: '["valkey"]'
        # Restore from PersistentCheckpoint (same namespace; use namespace/name for cross-namespace)
        architect.loopholelabs.io/start-from-persistent-checkpoint: "persistent-checkpoint-demo"
    spec:
      runtimeClassName: runc-architect
      containers:
        - name: valkey
          image: valkey/valkey:latest
          ports:
            - containerPort: 6379
```

If the referenced `PersistentCheckpoint` doesn't exist or has no data, the pod
starts fresh.

### Deleting a checkpoint

```bash
kubectl delete persistentcheckpoint persistent-checkpoint-demo
```

Checkpoint files are cleaned automatically. `PersistentCheckpoint` CRDs persist
until explicitly deleted (unlike `Checkpoint` CRDs, which are consumed by new
pods).


---

# How it works

> Source: https://architect.io/docs/how-it-works

How Architect's four components (admission controller, control plane, daemon, and shim) drive the pod hibernate, wake, and migration lifecycle.

## Components

Architect deploys four components into your cluster:

* **Admission controller**: configures managed containers on pod creation
* **Control plane**: coordinates checkpoint transfers during migrations
* **Daemon**: per-node agent that orchestrates [hibernation](/docs/glossary.md#hibernate-scale-down) and [wake](/docs/glossary.md#wake-scale-up)
* **Shim**: performs checkpoint/restore for the `runc-architect` runtime class

```mermaid
graph TB
    subgraph "Each Node"
        Pod[Your Pods]
        D[Architect Daemon]
    end

    subgraph "Cluster"
        API[Kubernetes API]
        AC[Admission Controller]
        CP[Control Plane]
    end

    Pod <--> D
    D --> API
    D --> CP
    API --> AC
```

## Pod startup

When a pod starts, Architect checks for an available checkpoint:

```mermaid
graph TD
    A[Pod starts] --> PC{start-from-persistent-checkpoint<br/>annotation set?}
    PC -->|Yes| F[Restore from PersistentCheckpoint]
    F -->|Fails| G[Start fresh]
    PC -->|No| B{Checkpoint<br/>available?}
    B -->|Yes| C[Restore from checkpoint]
    B -->|No| G
    C -->|Fails| G
```

The `start-from-persistent-checkpoint` annotation, when set, is the only
source of checkpoint data: on any failure the pod starts fresh. Without the
annotation, pods find checkpoints from other pods with the same template hash
(this is how [migration](#migration) works).

## Scale-down

When a container has been idle for the configured duration:

```mermaid
sequenceDiagram
    participant Pod as Your Pod
    participant A as Architect

    Note over Pod: Idle threshold reached
    A->>Pod: Checkpoint and stop container
    A->>A: Buffer incoming packets
    A->>A: Reduce pod resources to zero
    Note over Pod: Status: SCALED_DOWN
```

## Scale-up

By default only `kubectl exec` wakes a container. To wake on network traffic,
enable [`network-monitor`](/docs/configuration/annotations.md#network-monitor).

When a wake trigger arrives (`kubectl exec` or network packet):

```mermaid
sequenceDiagram
    participant U as User / Traffic
    participant A as Architect
    participant Pod as Your Pod

    U->>A: Request arrives
    A->>Pod: Restore from checkpoint
    A->>A: Release buffered packets
    A->>A: Restore original resources
    Note over Pod: Status: RUNNING
    A->>U: Ready
```

## Migration

When a pod is deleted (e.g., node drain), Architect transfers state to the
replacement pod:

```mermaid
sequenceDiagram
    participant N1 as Node 1
    participant API as Kubernetes API
    participant N2 as Node 2

    Note over N1: Pod deleted
    N1->>API: Store checkpoint
    Note over N2: Replacement pod scheduled
    N2->>API: Find checkpoint
    API->>N2: Transfer checkpoint
    N2->>N2: Restore container
```


---

# Cruise

> Source: https://architect.io/docs/how-it-works/cruise

Cruise, Architect's own in-tree checkpoint/restore engine: what it is, why it exists, how it compares to CRIU, and how to opt a workload into it.

> **Warning:** Experimental. Cruise is under active development and supports a subset of the workloads CRIU handles. The default engine remains CRIU; only enable Cruise when advised by Loophole Labs.

Cruise is Architect's own [checkpoint engine](/docs/glossary.md#checkpoint-engine):
the component that takes a [checkpoint](/docs/glossary.md#checkpoint) of a running
container and [restores](/docs/glossary.md#restore) it later. Every
hibernation, wake, and migration goes through a checkpoint engine, and today
that engine is [CRIU](https://criu.org/). Cruise is the engine we are building
to take its place — written in Zig and built into Architect rather than pulled
in as an external dependency. For the background on why, see
[The Math Changed: Rewriting Stale Open Source In The AI Era](https://loopholelabs.io/blog/rewriting-oss-in-the-ai-era)
and [AI Took My Coding. What's Left For Me?](https://loopholelabs.io/blog/ai-took-my-coding-whats-left-for-me).

## Why Cruise

Checkpoint/restore is the foundation of what Architect does, so we want it to be
fast, predictable, and ours to fix when it breaks. Cruise is a focused engine
built specifically for the way Architect checkpoints and restores managed
containers:

* **Purpose-built.** Cruise implements the slice of checkpoint/restore that
  Architect's hibernate, wake, and migration flows rely on, and leaves out the
  parts we don't need.
* **Ours to fix.** Because the engine is in-tree rather than a large external
  dependency, Architect can optimize the checkpoint/restore path and ship fixes
  on its own timeline.
* **Room to grow beyond CRIU.** Because the engine is ours, we can extend
  checkpoint/restore to cases CRIU does not handle. Support for additional
  capabilities, such as processes that use `io_uring`, is being added.
* **Drop-in integration.** Cruise speaks CRIU's worker RPC protocol directly, so
  runc drives it exactly as it drives CRIU. Nothing else in the pipeline changes
  when you switch engines.
* **Cross-architecture.** Cruise runs on both `x86_64` and `arm64` nodes.

As Cruise's workload coverage grows, it is intended to fully replace CRIU as
Architect's default engine.

## Enabling Cruise

Cruise is opt-in per pod, through the
[`checkpoint-engine`](/docs/configuration/annotations.md#checkpoint-engine)
annotation:

```yaml
architect.loopholelabs.io/checkpoint-engine: "cruise"
```

This routes checkpoint/restore for the pod's
[managed containers](/docs/glossary.md#managed-container) to Cruise instead of CRIU.
It is a pod-global setting; unmanaged containers are never checkpointed. The
default is `criu`.

## Verified applications

Cruise is exercised against a growing set of real applications in Architect's
continuous test suite, each checkpointed and restored on every change.

"Verified" here means the application checkpoints and restores in place, as in a
hibernate/wake cycle on the same node. It does not imply live migration across
nodes with active connections preserved, which depends on the TCP support Cruise
does not yet have (see [Limitations](#limitations)). The current coverage:

{/*
  Update this table as cruise PRs add or change verified workloads. Source of
  truth: cruise/go/tests/runc_cr_test.go (the runcWorkloads list) and the
  per-app cruise/go/tests/runc_<name>_workload_test.go files. A workload belongs
  here only if it passes cruise-to-cruise checkpoint/restore (no Skip). Real
  applications only — the tick/toy programs in cruise/example-programs are
  intentionally excluded.
  */}

| Application               | Category                    | Tested version |
| ------------------------- | --------------------------- | -------------- |
| NGINX                     | Web server                  | 1.27           |
| Apache HTTP Server        | Web server                  | 2.4            |
| HAProxy                   | Load balancer               | 2.9            |
| Apache Tomcat             | Servlet container           | 10.1           |
| Node.js                   | JavaScript runtime          | 20             |
| Python (Flask + Gunicorn) | Web framework               | 3.11           |
| Ruby (Puma)               | Web server                  | 3.3            |
| PHP                       | Web runtime                 | 8.3            |
| Spring Boot               | JVM framework               | JRE 21 / 25    |
| .NET (ASP.NET Core)       | Web framework               | 8              |
| PostgreSQL                | Relational database         | 16             |
| MariaDB                   | Relational database         | 11.4           |
| MongoDB                   | Document database           | 7.0            |
| ClickHouse                | Analytical database         | 24.8           |
| Valkey                    | In-memory store             | 8.0            |
| SQLite                    | Embedded database           | 3              |
| etcd                      | Distributed key-value store | 3.5            |
| NATS                      | Message broker              | 2.10           |
| Prometheus                | Metrics database            | 3.1            |
| Grafana                   | Dashboards                  | 11.4           |

**Not yet supported.** A few real applications do not checkpoint/restore on
Cruise yet and should stay on the default CRIU engine: **Node.js 24** and
**MinIO**.

## Limitations

Cruise implements a subset of CRIU's capabilities. Some workloads that
checkpoint and restore cleanly under CRIU are not yet supported, and coverage
expands each release. If a checkpoint or restore is not yet supported under
Cruise, switch the workload back to CRIU by removing the `checkpoint-engine`
annotation.

**Networking.** Cruise's networking support is minimal. It does not currently
checkpoint or restore active TCP connections, so it should not be used for live
migration that relies on preserving established connections across the move
(the [`rewrite-established-addresses-containers`](/docs/configuration/annotations.md#rewrite-established-addresses-containers)
annotation). Workloads that depend on keeping connections alive through a
checkpoint/restore should stay on CRIU.


---

# Testing your application

> Source: https://architect.io/docs/testing

Validate your own application against Architect in staging before production: what to check through a hibernate/wake cycle, plus the tested-compatibility matrix.

Before you roll Architect out in production, validate your own application in a
staging environment. The [Quick start](/docs/quick-start.md) covers the mechanics of
hibernating and waking a pod; this page is about confirming *your* workload comes
through that cycle correctly.

## Deploy under Architect

Add the runtime class and annotations to your workload and deploy it to staging.
See [Configuration](/docs/configuration.md) for the full annotation reference.

```yaml
spec:
  template:
    metadata:
      annotations:
        architect.loopholelabs.io/managed-containers: '["my-container"]'
        architect.loopholelabs.io/scaledown-durations: '{"my-container":"60s"}'
    spec:
      runtimeClassName: runc-architect
```

## What to validate

Drive your app through a full hibernate and wake cycle (the
[Quick start](/docs/quick-start.md) has the exact commands) and confirm:

* **It hibernates.** After the idle timeout, the
  `status.architect.loopholelabs.io/<container>` label reads `SCALED_DOWN`.
* **It wakes.** A `kubectl exec`, or incoming traffic when
  [`network-monitor`](/docs/configuration/annotations.md#network-monitor) is set,
  returns the label to `RUNNING` and the app responds normally.
* **In-memory state survives.** Write a value, let the container hibernate, wake
  it, and read the value back; it should still be there.
* **Health checks pass after wake.** Liveness and readiness probes recover. If
  they keep the container awake or fail, enable the
  [health-check proxy](/docs/configuration/annotations.md#health-check-proxy).
* **Open work is not corrupted.** Connections, transactions, and background jobs
  resume cleanly rather than erroring.

## Run the built-in self-test

Beyond validating your own workload, Architect ships a **self-test** component
that verifies its core functionality *inside your cluster* — useful after an
install or upgrade, or when a customized cluster (containerd config, CNI,
kernel) might behave differently from the tested matrix. It is enabled by
default and needs no configuration; its test workloads run in a dedicated
namespace (`architect-self-test` by default). To disable the component, set
`architectSelfTestEnabled: false` in the Helm chart — see the
[Helm values](/docs/configuration/helm-values.md#self-test) for the full list.

Then open the cluster in the [Console](https://console.architect.io/) and switch
to the **Health** tab. By default **Run self-test** runs every check at once
and shows whether the cluster passed, along with past runs.

Each check in the results expands in place to show its individual steps, and
each step links to the pod it acted on — so a failing check tells you where it
failed and on which pod without changing any settings. A failing check is
expanded for you.

While a run is in progress you can **Cancel** it (its workloads are cleaned
up). **Export events** downloads the run's events *and* every event of each pod
the run touched, as NDJSON — the pods' own events are usually what explains a
failure. Results are stored as ordinary Architect events, so they also appear
in the cluster's regular event history.

Expand **Advanced** (under the run button) to change what the next run does:

* Pick which checks to run — **hibernate**, **wake on exec**, **data
  persistence**, **wake on network**, and **migration**, each a narrow version
  of the core flows this page describes — and how many run in parallel.
* Apply **customization slots** (labels, annotations, and a node selector) to
  the test workloads so they pass the same admission controllers and node
  selectors as
  your real workloads; the last-used slots are remembered.

A pass means Architect's core functionality works in your cluster; a failing
check's log narrows down where.

## Check logs on failure

```bash
kubectl logs -n architect -l app.kubernetes.io/name=architectd | grep <pod>
```

The [Architect Console](https://console.architect.io/) also shows per-pod events,
timings, and checkpoint details.

## Compatibility matrix

GPU workloads are not supported yet. These language and framework combinations
have been tested:

| Language | Application | Hibernation | Migration |
| -------- | ----------- | ----------- | --------- |
| C        | PostgreSQL  | Yes         | Yes       |
| C        | Valkey      | Yes         | Yes       |
| Go       | net/http    | Yes         | Yes       |
| Java     | Kafka       | Yes         | Yes       |
| Java     | Spring Boot | Yes         | Yes       |
| PHP      | WordPress   | Yes         | Yes       |
| Python   | http.server | Yes         | Yes       |
| Ruby     | TCPServer   | Yes         | Yes       |
| Rust     | miniserve   | Yes         | Yes       |
| C#       | ASP.NET     | No          | No        |
| JS       | Node.js     | No          | No        |

Node.js and ASP.NET aren't supported yet: their runtimes rely on kernel
features CRIU cannot checkpoint. For Node.js this is `io_uring`, which libuv
enables by default in current releases.


---

# Introspection

> Source: https://architect.io/docs/introspection

Monitor and debug Architect using the console dashboard, per-container status labels, resource tracking, and daemon logs.

## Console

The [Architect Console](https://console.architect.io/) provides a dashboard
for monitoring hibernation across your clusters:

* Per-pod event history (scale up, scale down, migration) over the last 30 days
* Live event stream with real-time updates
* Event timing breakdowns for diagnosing slow checkpoints or restores

![Console pod detail view showing the event timeline and per-container hibernation event history](/docs/console-pod-detail.png)

## Status labels

Architect labels each managed container (the ones you list in
[`managed-containers`](/docs/configuration/annotations.md#managed-containers)) with
its state:

```bash
kubectl get pods -l status.architect.loopholelabs.io/<container-name>=SCALED_DOWN
```

Values: `RUNNING` or `SCALED_DOWN`.

List all hibernated pods:

```bash
kubectl get pods -o json | jq '
  .items[]
  | select(
      .metadata.labels
      | to_entries[]
      | select(.key | startswith("status.architect.loopholelabs.io/"))
      | .value == "SCALED_DOWN"
    )
  | .metadata.name'
```

## Resource tracking

Architect stores original resource requests in annotations before reducing them
to zero:

```bash
kubectl get pod <pod-name> \
  -o jsonpath='{.metadata.annotations.architect\.loopholelabs\.io/cpu-requests}'
# {"container-name":"250m"}

kubectl get pod <pod-name> \
  -o jsonpath='{.metadata.annotations.architect\.loopholelabs\.io/memory-requests}'
# {"container-name":"6Gi"}
```

## Resource usage

Compare actual vs provisioned resources:

```bash
kubectl get pods -o custom-columns=\
NAME:.metadata.name,\
CPU:.spec.containers[0].resources.requests.cpu,\
MEMORY:.spec.containers[0].resources.requests.memory
```

On Kubernetes 1.33+ with metrics-server installed, you can also use
`kubectl top pods` to compare actual usage against requests.

## Logs

```bash
kubectl logs -n architect -l app.kubernetes.io/name=architectd --tail=100
kubectl logs -n architect -l app.kubernetes.io/name=architect-admission-controller --tail=100
kubectl logs -n architect -l app.kubernetes.io/name=architectd | grep <pod-name>
```


---

# Best practices

> Source: https://architect.io/docs/best-practices

Best practices for running Architect in production: node labeling, application suitability, configuration guidelines, and capacity planning.

## Node configuration

* **Label only intended nodes**: `architect.loopholelabs.io/node=true` controls
  where `architectd` runs. Only label nodes where you want Architect workloads.
* **Use critical-node labels for control components**: The
  `architect-admission-controller` and `architect-control-plane` require
  `architect.loopholelabs.io/critical-node=true`. Place these on stable nodes
  that are unlikely to be drained or preempted.
* **Use tolerations and node selectors**: The Helm chart exposes
  `architectdNodeSelector`, `architectControlPlaneNodeSelector`, and
  `architectAdmissionControllerNodeSelector` (plus matching toleration options)
  to control placement. See [Configuration → Helm values](/docs/configuration/helm-values.md#placement-and-sizing).

## Application suitability

**Well-suited applications:**

* Stateless web services and APIs
* Microservices with intermittent traffic
* Development and staging environments
* Services with predictable traffic patterns

See the [compatibility matrix](/docs/testing.md#compatibility-matrix) for tested
languages and frameworks.

**Not yet supported:**

* GPU workloads (CUDA state preservation is under development)

## Configuration guidelines

* **Start with the default timeout**: The default idle timeout is `60s`. Lower
  it gradually per-container via the `scaledown-durations` annotation once
  you've validated behavior.
* **Test in staging first**: Always validate hibernation behavior in
  non-production environments before rolling out. See
  [Testing your application](/docs/testing.md).
* **Enable network-monitor for web traffic**: Use the `network-monitor`
  annotation with `packets` or `connections` mode so containers wake on
  incoming requests instead of only on `kubectl exec`.
* **Use health-check-proxy for probed services**: If your workload has
  liveness, readiness, or startup probes, enable `health-check-proxy`.
  Without it the probes themselves prevent sleep (they hit the application
  port and count as activity), and removing them loses the safety net that
  catches a stuck process. See
  [Configuration → health-check-proxy](/docs/configuration/annotations.md#health-check-proxy).
* **Use shadow-ports for scraped metrics**: If Prometheus (or any other
  external scraper) hits the workload on a regular interval, enable
  `shadow-ports` to redirect scrape traffic to a port that doesn't count as
  activity. Same problem, same fix. Use `ignore-activity-ports` if you can't
  move the scraper to a different port. See
  [Configuration → shadow-ports](/docs/configuration/annotations.md#shadow-ports).

## Capacity planning

Hibernated pods consume zero CPU and memory while staying scheduled. This means
you can run more replicas for availability without proportional cost increase --
idle replicas hibernate automatically and wake in under 50ms when traffic
arrives.


---

# Security

> Source: https://architect.io/docs/security

The permissions Architect needs, the trust boundaries it enforces, and how checkpoint data is handled, for a production security review.

Architect is a cluster-scoped service, but its permissions are scoped to exactly
what it needs. **There is no cluster-admin binding in the default install**, and
nothing in Architect can grant itself more access at runtime (it has no verbs on
`roles`, `clusterroles`, or their bindings).

## Service accounts and permissions

The chart installs three ServiceAccounts in Architect's namespace (default
`architect`), each bound to narrow ClusterRoles. A fourth, `architect-router`, is
added only when live-migration buffering is enabled (see below).

### `architectd-installer-daemon` (the per-node daemon)

| Resource                                          | Verbs                             | Why                                                                                  |
| ------------------------------------------------- | --------------------------------- | ------------------------------------------------------------------------------------ |
| `node.k8s.io/runtimeclasses`                      | create, update, patch             | Installs the `runc-architect` RuntimeClass on startup.                               |
| `pods`, `pods/resize`                             | get, list, watch, patch           | Reads managed-pod status; zeroes and restores CPU/memory requests on hibernate/wake. |
| `events`                                          | create, patch                     | Emits lifecycle Events (`ScaleDownCompleted`, `CheckpointCreated`, and so on).       |
| `apps/deployments`                                | create, deletecollection, get     | Installs and uninstalls Architect's bootstrap Deployments.                           |
| `apps/replicasets`                                | get                               | Owner lookups for managed pods.                                                      |
| `architect.loopholelabs.io/checkpoints`           | create, patch, get, delete, watch | Manages the internal `Checkpoint` CRD records.                                       |
| `architect.loopholelabs.io/persistentcheckpoints` | get, list, watch, update, delete  | Consumes the user-facing `PersistentCheckpoint` CRD.                                 |
| `discovery.k8s.io/endpointslices`                 | list                              | Locates peer daemons for checkpoint transfer.                                        |

When `features.liveMigrationBuffering` is enabled, this role also gains
`architect.loopholelabs.io/networkmigrationstates` (full access), `services`
(get/list/watch), and `get` on `deployments`/`replicasets`/`statefulsets`/`daemonsets`
(to tell whether a migrating pod's owner is being torn down).

### `architect-admission-controller` (the admission controller)

| Resource                                                     | Verbs                                    | Why                                                                                                  |
| ------------------------------------------------------------ | ---------------------------------------- | ---------------------------------------------------------------------------------------------------- |
| `admissionregistration.k8s.io/mutatingwebhookconfigurations` | create, update, patch, get               | Registers the mutating webhook that injects the health-check-proxy sidecar.                          |
| `services`, `endpoints`                                      | get, list, watch, create, update, delete | Manages the webhook's service and the endpoints it injects (health-check-proxy and shadow services). |
| `pods`                                                       | get, list, watch, patch                  | Inspects and patches managed pods.                                                                   |

### `architect-control-plane` (the control plane)

| Resource           | Verbs        | Why                                                           |
| ------------------ | ------------ | ------------------------------------------------------------- |
| `pods`             | list         | Observes pods to coordinate migration and checkpoint locking. |
| `apps/deployments` | list, delete | Cleans up Architect's bootstrap Deployments on uninstall.     |

### `architect-router` (optional, live-migration buffering)

Installed only when `features.liveMigrationBuffering` is enabled. The router runs
as a DaemonSet and buffers in-flight traffic while a pod migrates.

| Resource                                           | Verbs                   | Why                                                  |
| -------------------------------------------------- | ----------------------- | ---------------------------------------------------- |
| `architect.loopholelabs.io/networkmigrationstates` | get, list, watch, patch | Reads and updates the migration-buffering state.     |
| `services`, `pods`                                 | get, list, watch        | Resolves the pods and services it routes traffic to. |

## What Architect cannot do

Across all three roles, Architect has **no** access to:

* `secrets` or `configmaps`: it cannot read your workloads' secrets or config.
* `roles`/`clusterroles`/bindings: it cannot escalate its own privileges.

It also cannot create or delete your StatefulSets or pods; the daemon only patches
managed pods' status and resources. It does hold cluster-scoped `create`/`delete`
on Deployments, which it uses to install and clean up its own bootstrap
Deployments (see the tables above), so that grant is not restricted to the
`architect` namespace.

Architect reads its own machine token from a mounted Secret (`architectd-secrets`
by default, or the one you pass via `secretRef`), not through the Kubernetes API.

## Checkpoint data is sensitive

Hibernation and migration use CRIU (Checkpoint/Restore In Userspace), which
captures the **full memory of the process**, including anything held in memory at checkpoint time (cached
credentials, decrypted data, in-flight requests). Treat a checkpoint with the
same sensitivity as a memory dump.

* **At rest (local):** checkpoints are written to node-local disk via a hostPath
  the daemon mounts, as zstd-compressed tar (`tar.zst`) archives. Architect does
  **not** encrypt them itself, so rely on node-disk encryption (EBS, GCE PD, or LUKS).
* **At rest (S3):** when [persistent checkpoints](/docs/configuration/helm-values.md#persistent-checkpoint-storage-s3) use S3,
  enable bucket-level encryption (SSE-S3 or SSE-KMS). Architect uploads over TLS
  but does not encrypt contents before upload.
* **In transit between nodes:** daemons stream `tar.zst` checkpoints to each
  other over **plain HTTP on port 1337**. Control-plane to daemon traffic is also
  plain HTTP on 1337. If your pod network is untrusted, add fabric-level
  encryption (a mesh, or WireGuard via your CNI).
* **Admission webhook:** served over **HTTPS** with a self-signed CA the
  installer generates and registers at install time.

## Admission webhook behavior

The mutating webhook registers cluster-wide on pod create. The handler
short-circuits any pod whose `runtimeClassName` is not `runc-architect`; those
pods pass through untouched. Its `failurePolicy` is **`Ignore`**, so if the admission
controller is down, pod creation proceeds without the Architect mutation and
Architect never blocks cluster operations.

## Audit trail

Every hibernate, wake, and migration emits a Kubernetes Event on the affected
pod:

```bash
kubectl get events --field-selector involvedObject.name=<pod>
```

Expect reasons like `ScaleDownCompleted`, `ScaleUpCompleted`, `CheckpointCreated`,
and `CheckpointDownloaded`. Events follow the standard one-hour retention, so ship
them to your log backend for longer history. Architect keeps no separate audit
log.

## Security review checklist

| Question                                 | Answer                                                                                                   |
| ---------------------------------------- | -------------------------------------------------------------------------------------------------------- |
| Does Architect need cluster-admin?       | No. See the ClusterRole tables above.                                                                    |
| Can it read my Secrets or ConfigMaps?    | No. It has no verbs on either.                                                                           |
| Can it escalate privileges?              | No. It has no verbs on roles, clusterroles, or bindings.                                                 |
| Does it capture process memory?          | Yes. CRIU checkpoints are full memory dumps; treat them as sensitive.                                    |
| Are checkpoints encrypted at rest?       | Not by Architect. Use node-disk encryption locally and SSE for S3.                                       |
| Is inter-component traffic encrypted?    | Only the admission webhook (HTTPS). Daemon-to-daemon and control-plane-to-daemon are plain HTTP on 1337. |
| Can a non-opted-in workload be affected? | No. The webhook ignores any pod without `runtimeClassName: runc-architect`.                              |

For anything not covered here, contact [security@loopholelabs.io](mailto:security@loopholelabs.io).


---

# Troubleshooting

> Source: https://architect.io/docs/troubleshooting

Diagnose common Architect issues: pods that won't hibernate or wake, unwanted wakes from probes or scrapes, sidecar injection, and slow wake times.

## Start with the self-test

Before digging into one workload, find out whether Architect works in the
cluster at all: open the cluster in the
[Console](https://console.architect.io/), switch to the **Health** tab, and
click **Run self-test**. It exercises hibernate, wake on exec, data
persistence, wake on network, and migration against its own short-lived
workloads.

That splits the problem in two:

* **The self-test fails too.** The problem is the cluster, not your workload.
  Expand the failing check to read its steps and open the pod it ran on, then
  work through the sections below with that pod.
* **The self-test passes.** Architect's core functionality is fine here, so the
  cause is specific to your workload — its annotations, probes, or traffic. The
  sections below cover the common ones.

Export a failing run (**Export events**) to attach the run's events and those
of every pod it touched to a support request. See [Run the built-in
self-test](/docs/testing.md#run-the-built-in-self-test) for the full description.

## Pod not hibernating

```bash
# Check idle timeout
kubectl get pod <pod-name> \
  -o jsonpath='{.metadata.annotations.architect\.loopholelabs\.io/scaledown-durations}'

# Verify container is managed
kubectl get pod <pod-name> \
  -o jsonpath='{.metadata.annotations.architect\.loopholelabs\.io/managed-containers}'

# Check status label
kubectl get pod <pod-name> \
  -o jsonpath='{.metadata.labels.status\.architect\.loopholelabs\.io/<container-name>}'

# Review daemon logs
kubectl logs -n architect -l app.kubernetes.io/name=architectd | grep <pod-name>
```

The [Architect Console](https://console.architect.io/) also shows per-pod
events, timings, and detailed debugging info.

## Pod not waking

```bash
# Test wake via exec
kubectl exec -it <pod-name> -- /bin/sh -c "echo test"

# Test wake via network
kubectl port-forward <pod-name> <port>:<port>
curl localhost:<port>

# Check events
kubectl describe pod <pod-name>

# Verify daemon is running on the pod's node
kubectl get pod <pod-name> -o wide
kubectl get pods -n architect -o wide | grep <node-name>
```

## Scale down and wake

### Health probes wake the container

If a managed container with `health-check-proxy` configured still wakes
whenever kubelet probes it:

```bash
# Confirm the sidecar was added
kubectl get pod <pod-name> \
  -o jsonpath='{.spec.containers[*].name}'

# Confirm probe ports target the shadow port, not the app port
kubectl get pod <pod-name> \
  -o jsonpath='{.spec.containers[?(@.name=="<container>")].livenessProbe}'

# Check the admission controller didn't skip the sidecar
kubectl logs -n architect -l app.kubernetes.io/name=architect-admission-controller \
  | grep -i 'health check proxy'
```

The first command lists every container in the pod; you should see
`architect-health-check-proxy` alongside your application container, e.g.:

```
my-app architect-health-check-proxy
```

Checklist:

* The probe's `port` field on each managed container must reference the
  `shadowPort`, not the `appPort`. Probes that still target the application
  port bypass the sidecar entirely.
* Both `managed-containers` and `network-monitor` annotations must be
  present. Without either, the admission controller logs a warning and skips
  sidecar injection.
* The sidecar (`architect-health-check-proxy`) must be present in
  `spec.containers`. If it isn't, check admission controller logs.

### Scrape traffic wakes the container

If a Prometheus scrape (or other external poller) wakes a managed container
that has `shadow-ports` configured:

```bash
# Confirm the shadow port is on the container spec
kubectl get pod <pod-name> \
  -o jsonpath='{.spec.containers[?(@.name=="<container>")].ports}'

# Check the admission controller didn't skip the shadow ports
kubectl logs -n architect -l app.kubernetes.io/name=architect-admission-controller \
  | grep -i 'shadow ports'
```

The first command lists the container's ports; the shadow port appears with
a `shadow-` name prefix, e.g.:

```
[{"containerPort":9090} {"containerPort":29090,"name":"shadow-29090","protocol":"TCP"}]
```

Checklist:

* The scraper must target the `shadowPort`, not the `appPort`. Verify your
  `ServiceMonitor`, `PodMonitor`, or `scrape_configs` references the shadow
  port (named `shadow-<port>` on the container spec).
* Both `managed-containers` and `network-monitor` annotations must be
  present. Without either, the admission controller logs a warning and skips
  injection.
* If you can't move the scraper to a new port, swap `shadow-ports` for
  `ignore-activity-ports` so the existing app port is exempted from activity
  tracking.

### Sidecar fails to inject

If `health-check-proxy` is set but no sidecar appears on the pod:

```bash
kubectl logs -n architect -l app.kubernetes.io/name=architect-admission-controller \
  | grep -i 'health check proxy\|shadow ports'
```

Checklist:

* The annotation JSON must parse; invalid JSON is logged and the feature
  is skipped.
* `managed-containers` must list the container referenced in each mapping.
* `network-monitor` must be set on the pod.
* All ports must be in the 1–65535 range; mappings outside the range are
  dropped with a warning.
* Duplicate `shadowPort` values across mappings are dropped with a warning.
  Only the first mapping per shadow port is used.

## High wake times

If wake times exceed 50ms:

* Check node CPU and memory availability; contention slows restore
* Large memory footprints produce larger checkpoints
* Verify no resource contention on the node
* Check daemon logs or the [Architect Console](https://console.architect.io/)
  for per-pod restore timings:

```bash
kubectl logs -n architect -l app.kubernetes.io/name=architectd --tail=500 \
  | grep -E "checkpoint|restore|error"
```

## Checkpoint failures

* GPU workloads are not supported yet
* Checkpoints use 50-200MB per pod; check node disk space:

```bash
kubectl get nodes \
  -o custom-columns=NAME:.metadata.name,DISK:.status.allocatable.ephemeral-storage
```

* Verify `runtimeClassName` is set and the node has the
  `architect.loopholelabs.io/node=true` label
* Check the [Architect Console](https://console.architect.io/) for checkpoint
  error details

## Runtime class errors after uninstall

Pods still referencing `runc-architect` will error.
Remove `runtimeClassName` from affected workloads. See
[Uninstalling](/docs/installation/uninstalling.md).


---

# FAQ

> Source: https://architect.io/docs/faq

Answers to common Architect questions: how it differs from scale-to-zero, app compatibility, HPA and StatefulSet support, and overhead.

## How is this different from scale-to-zero (KEDA, Knative)?

Scale-to-zero deletes pods entirely, causing 30-60+ second cold starts.
Architect hibernates pods in place. They wake in under 50ms, stay registered
with services, and keep PVCs mounted.

## What applications are compatible?

Most stateless and many stateful workloads work; GPU workloads are not supported
yet. See the [compatibility matrix](/docs/testing.md#compatibility-matrix) for the
languages and frameworks that have been tested.

## Does Architect work with HPA?

Yes. HPA manages replica count; Architect ensures idle replicas use zero
resources.

## Does Architect work with StatefulSets?

Yes. Same annotations and runtime class. PVCs stay mounted during hibernation.

## What happens to in-flight requests?

Packets arriving during or after hibernation are buffered and delivered on wake
(typically under 50ms). No packets are dropped.

## How much overhead does Architect add?

Less than 1% CPU and less than 50MB memory per node.

## What happens during Kubernetes upgrades?

Upgrade Architect first, then workloads. During a node drain, hibernated pods
migrate their in-memory state to another node and resume there.

## How much disk space do checkpoints use?

50-200MB per pod depending on memory footprint.

```bash
kubectl exec -n architect <architectd-pod> -- \
  du -sh /root/.local/state/architect/checkpoint/
```

## What happens if the daemon crashes?

Pods continue running but won't hibernate or wake until the DaemonSet
controller restarts `architectd`. Checkpoints are preserved.

## Where are checkpoints stored?

Locally on each node running `architectd`.

## How do I uninstall Architect?

See [Uninstalling](/docs/installation/uninstalling.md).

## Can I use Architect on older kernels?

We recommend Linux 6.6 or newer. On older kernels:

* **Hibernation and migration** (checkpoint/restore) work down to **5.10**.
* **Live-migration buffering** (experimental): requires **5.18+** to load. On
  **6.1** (e.g. AL2023) all traffic is supported except jumbo frames, which
  require **6.6+**.

Below 5.10, checkpoint/restore is unavailable, so Architect cannot hibernate pods.


---

# Glossary

> Source: https://architect.io/docs/glossary

Definitions of the core Architect terms: checkpoint, hibernate, migration, runtime class, managed container, and the components that run them.

Short definitions of the terms used throughout these docs.

### Admission controller

The component that configures a pod to work with Architect. When you create a pod
that requests Architect's runtime class, Kubernetes hands it to the admission
controller first, which reads your annotations and adjusts the pod so its
containers can be hibernated and restored. It runs as a Deployment named
`architect-admission-controller`, in the `architect` namespace, on a
[critical node](#critical-node).

### architectd (daemon)

The part of Architect that runs on every worker node and does the actual
hibernating, waking, and migrating of that node's containers. It runs as a
DaemonSet, one copy per node; if it stops on a node, the containers there keep
running but cannot hibernate or wake until it comes back.

### Checkpoint

A point-in-time snapshot of a running container's memory and process state. A
checkpoint can be written to disk as a checkpoint file, or exist only ephemerally
in memory. Restoring one resumes the container exactly where it left off, with no
cold start.

### Checkpoint engine

The component that creates a [checkpoint](#checkpoint) (and restores it again
later). This is an internal detail: in normal use you don't need to choose or
think about the engine. Architect uses [CRIU](https://criu.org/) today and is
moving to its own engine, Cruise, over time.

### Cold start

The delay when a container starts from scratch and has to re-initialize: load
code, warm caches, and rebuild in-memory state. Restoring from a
[checkpoint](#checkpoint) skips this, because the container resumes with that work
already done.

### Control plane

Architect's central coordinator. It keeps checkpoint handoffs between nodes
consistent during hibernation and migration, so container state can move around
the cluster safely. It runs as a Deployment named `architect-control-plane`, in
the `architect` namespace, on a [critical node](#critical-node).

### Critical node

A node labeled `architect.loopholelabs.io/critical-node=true`, where Architect's
[control plane](#control-plane) and [admission controller](#admission-controller)
run. Choose long-lived, on-demand nodes for these: not spot or preemptible
capacity, and not nodes you routinely drain or autoscale away, so these
components stay available.

### Cruise

Architect's own [checkpoint engine](#checkpoint-engine), the in-tree alternative
to [CRIU](https://criu.org/) that Loophole Labs is building to replace it.
Experimental and opt-in today; see [Cruise](/docs/how-it-works/cruise.md).

### Health-check proxy

A helper that answers a container's liveness and readiness probes while it is
hibernated or migrating, so Kubernetes does not mark the container unhealthy and
the probes themselves do not keep waking it. The
[admission controller](#admission-controller) injects it as a sidecar into the
managed pod, where it reads container state from the [shim](#shim-shim-runc).

### Hibernate (scale-down)

Checkpointing an idle [managed container](#managed-container) and dropping its
pod's CPU and memory requests to zero, while the pod stays scheduled and keeps its
IP, Services, and volumes. Architect hibernates a container after it has been idle
long enough. Its [status label](#status-label) reads `SCALED_DOWN` while it is
hibernated.

### Lazy-pages migration

A faster form of [migration](#migration) for containers with a large memory
footprint: instead of copying all the memory up front, the destination node fetches
each page (a small block of memory) from the source only as the container first
touches it.

### Managed container

A container that Architect hibernates and restores. You choose which containers
are managed; any you leave out (for example a logging sidecar) run normally and
are never hibernated.

### Migration

Moving a [managed container](#managed-container)'s in-memory state to its
replacement on another node when Kubernetes replaces the pod, for example during a
node drain, rolling update, or spot interruption. Architect checkpoints the
container on the old node and restores it in the new pod, so it keeps its running
state instead of starting cold. Only managed containers migrate this way.

### PersistentCheckpoint

A checkpoint you create deliberately and keep, captured while the container keeps
running. It saves a known-good or pre-warmed state, a golden image, that
other pods can start from to skip a slow cold start. It is stored on the node by
default, or in S3-compatible object storage when you configure it. Unlike the
checkpoints Architect takes automatically when hibernating, a PersistentCheckpoint
stays until you delete it.

### Restore

Re-creating a container from a [checkpoint](#checkpoint) so it resumes with its
previous memory and running state instead of starting fresh. Both waking and
migration end in a restore.

### Router (and router-shim)

The components that route network traffic to managed pods and buffer it during a
migration. The router runs as a DaemonSet named `architect-router`, in the
`architect` namespace, with the router-shim as a sidecar in its pods. They are
installed only when the experimental traffic-buffering feature is enabled in the
[Helm values](/docs/configuration/helm-values.md#experimental).

### Runtime class (`runc-architect`)

The Kubernetes RuntimeClass that opts a pod into Architect. Setting
`runtimeClassName: runc-architect` on a pod is what tells Architect to manage it.

### Shadow port

An extra port Architect exposes, named `shadow-<port>`, so health probes or
metrics scrapers can reach a container without that traffic counting as activity
that would wake it. Configured with the
[`shadow-ports`](/docs/configuration/annotations.md#shadow-ports) annotation.

### Shim (shim-runc)

Architect's shim is a containerd shim: a small program that sits between
containerd (the container runtime Kubernetes drives) and runc (the low-level tool
that actually starts and stops containers). Architect's version adds checkpoint
and restore at that layer, so a container can be hibernated and woken with no
changes to your workload. It runs on each node and coordinates with the
[architectd](#architectd-daemon) on that node.

### Status label

The label `status.architect.loopholelabs.io/<container>` that Architect sets to
`RUNNING` or `SCALED_DOWN` to show whether each
[managed container](#managed-container) is currently awake or hibernated.

### Wake (scale-up)

Restoring a hibernated container and returning its CPU and memory requests,
triggered by a `kubectl exec`, or by incoming network traffic if network-based
wake is enabled.
The [status label](#status-label) returns to `RUNNING`.