Six methods,
one cloud at a time.

There is a particular way that infrastructure software boxes itself in. It hard-codes not just which cloud it talks to, but which account. Mine did both. Twenty-plus @azure/* packages, and DefaultAzureCredential() constructed in about ten separate places, each of them reading the same process-wide environment variables. That is not one coupling problem. It is two.

The first is a vendor coupling: every call goes straight to an Azure SDK, so a second provider is a rewrite of every call site. The second is quieter and worse — a tenancy ceiling. Because credentials live in the environment, the app is not "an Azure management app." It is "a management app for exactly one Azure subscription, chosen at deploy time." A user cannot connect their account. There is no product in that. This is the story of the seam I inserted to fix both, and the far larger surface I deliberately refused to abstract.

01 · THE PROBLEMTwo couplings, stacked

I want to be honest about where this started, because the fix only makes sense against the mess it replaced. Quoting the Context section of the architecture decision that opens this work, ADR-004:

Problem A is vendor coupling. Every cloud call reaches straight into a vendor SDK. Adding a second provider means touching every one of them.

Problem B is the tenancy ceiling, and it is the one that actually hurts. Credentials sit in process.env; a serverless function reads them at cold start. So there is no multi-account story, no multi-tenant story, and therefore no product. Two more absences turn out to be the same fix: there was nowhere to put a credential that isn't an env var, and there was no action ledger. The existing Event model was telemetry, not an audit trail — and "we already log things" is the most common way an audit requirement gets quietly failed.

For honesty about the starting scale: 62 API routes across six route files, zero auth on any of them, managing live Azure infrastructure. That was logged as critical gap #1 and fixed before any of this — all 61 non-auth routes put behind requireAuth. The control-plane work here starts after that cleanup, on a codebase I had already audited and graded honestly: backend 2.5 out of 5, cross-lane coherence 2 out of 5. I mention the grade because a system is easier to trust when its architect writes down what is wrong with it first.

02 · THE WRONG TURNThe multi-cloud framework everyone reaches for

The reflex answer to "make it multi-cloud" is to reach for Pulumi, Terraform, Crossplane, or libcloud. It feels responsible. It is the wrong shape. From ADR-004's Alternatives Considered:

The category error is the whole point. An IaC engine's job is to converge desired state. My job was to press stop on a running machine. Those are not the same shape, and adopting a state-machine to do an imperative thing is exactly how a small seam quietly becomes a platform migration.

03 · THE WORSE WRONG TURNAbstract all sixty-two routes

The more seductive mistake was to put the whole existing Azure surface behind the new port — all eight route files, roughly twenty service domains, sixty-two routes. This is the trap, and the ADR's rationale for avoiding it is the single best line in the document:

A port is only worth its weight if every method is implementable by every future adapter.

"List compute, read cost, control a VM" maps cleanly onto Azure, AWS, and GCP. The twenty-service Azure-specific surface does not. Forcing it through a port would produce a leaky, Azure-shaped interface that no AWS adapter could honor. So the ADR did the thing most refactor plans never do: it declared a v1 Scope Reset and froze the legacy surface in place rather than porting it. "v1 ships the portable core; the Azure-specific richness stays in the frozen Express layer for anyone who needs it." And — this is the part I am proudest of — it wrote down the cost of that choice as a Negative, not a footnote.

That sentence is the thesis of the piece in miniature. Drift you have written down and bounded is a decision. Drift you haven't is rot.

There was a third refusal, and I made it by name. Sitting right next to the control plane was a live, useful subsystem — the Powerhouse Integration Hub, a content-integration landing zone with a service-health grid and a read-only Cosmos console. Every instinct says "abstract that too, it's cloud work." ADR-004 amended itself to say no, bluntly: the Integration Hub is explicitly de-scoped, a data-exploration concern categorically different from a cloud control plane, and forcing it through CloudProvider would corrupt the port with content-API and Cosmos-query methods no AWS or GCP adapter could ever honor. Any future "abstract the Integration Hub" request is a separate ADR, not an amendment to this one. Two systems, both cloud, both mine, kept apart on purpose. That discipline is what the rest of this is about.

04 · THE KEY INSIGHTSix methods, and the credential moves

The unlock has two halves, and neither is large. The first: the entire vendor abstraction is six methods. Verbatim from providers/types.ts:

export type ProviderKind = 'azure' | 'aws' | 'gcp';   // only 'azure' implemented in v1

export interface CloudProvider {
  readonly kind: ProviderKind;

  // connectivity / credential validation
  verifyConnection(): Promise<{ ok: boolean; identity?: string; error?: string }>;

  // read plane
  listComputeInstances(): Promise<ComputeInstance[]>;
  getCostSummary(period: { from: string; to: string }): Promise<CostSummary>;

  // control plane (VM lifecycle) — all audited + confirm-gated upstream
  startInstance(id: string): Promise<void>;
  stopInstance(id: string): Promise<void>;
  restartInstance(id: string): Promise<void>;
  resizeInstance(id: string, size: string): Promise<void>;
}

export interface CloudProviderFactory {
  forConnection(cred: CloudCredential): CloudProvider;   // one request, one credential
}

Three provider-neutral verbs — connect, list inventory, control. The names are the v1 realisation; the verbs are the contract.

The discipline that keeps this honest is that each verb was pinned to a plausible AWS realisation at design time, precisely so the port cannot quietly drift into an Azure shape. This mapping is straight out of ADR-004:

Any v2 adapter that can honor those three verbs slots in behind CloudProviderFactory.forConnection with zero handler, dashboard, or audit changes. That is the entire point of the port. To be exact about the state of play: aws and gcp are reserved members of the union type and nothing more — the factory throws on them today.

And here is the actual reveal — the credential moves. The port alone buys you nothing if the adapter still reads process.env. The whole multi-account capability comes from one function:

export async function resolveProvider(id: string, ownerEmail: string | null): Promise<CloudProvider> {
  await dbConnect();
  const doc = await Connection.findOne({ _id: id, createdBy: ownerEmail })   // owner-scoped ⇒ IDOR-safe
    .select('+secret')                                                       // sealed envelope is select:false
    .lean();
  if (!doc) throw AppError.notFound('Connection not found');
  const cred = decryptJson<CloudCredential>(doc.secret);                     // open in memory
  return providerFactory.forConnection(cred);                                // …and discard
}

Ten ambient-environment credential sites, replaced by one owner-scoped lookup that opens a sealed envelope and throws the plaintext away.

That is the change that turned a single-subscription demo into a multi-account product, and it is roughly a dozen lines. Everything else in this piece is consequence. The adapter constructor makes the shift visually obvious, and its own header comment says it out loud — built from a decrypted, per-connection credential, not process.env and not the old DefaultAzureCredential sites:

constructor(cred: AzureCredential) {
  this.credential = new ClientSecretCredential(cred.tenantId, cred.clientId, cred.clientSecret);
  this.subscriptionId = cred.subscriptionId;
}

The port makes "which cloud" a detail. The resolver makes "which account" a detail. Together they are the product.

06 · THE VAULTFour layers, and the tradeoff I kept

This is the section that earns the piece its credibility, because the security posture lives here rather than in a diagram. The envelope is AES-256-GCM under a single app master key — 32 raw bytes, base64, from the environment, never in the repo. GCM is authenticated encryption, so tampering fails on final(): integrity is verified on decrypt, not assumed. The code fails closed and fails loudly — a missing key does not degrade to plaintext, it throws with the remedy in the message (generate one with openssl rand -base64 32). And it leaves room for a rotation it hasn't done yet, by carrying an alg and a per-record envelope shape so a future key id slots in without a migration. Writing down the shape of a migration you haven't performed is the same instinct as writing down the coexistence cost in section 03.

The four defences are worth naming because they are layered deliberately. The secret is select:false, so a forgotten projection can't leak it into a response — leaking requires an explicit .select('+secret'), which appears in exactly one place. The public view is built from an allowlist projector, not a delete-list: the secret can never leak because it is never read there. The distinction between "I removed the dangerous field" and "I only copied the safe ones" is the whole game. meta holds display-safe identifiers only. And owner scoping lives in the query, not a middleware check.

Now the tradeoff I kept, stated plainly. One master key seals everything — cloud credentials and LLM keys alike. That is a deliberate simplicity call, and it means the blast radius of a leaked CONNECTION_ENC_KEY is the whole vault. The ADR logs it as a named risk — "the vault is only as safe as the app key" — with mitigation and no pretence that it's solved. The ultimate backstop is not the app at all: the stored service principal's own Azure RBAC bounds everything. The app can never provision beyond what the credential is permitted.

07 · THE DETAIL THAT SEPARATES REAL FROM DEMOI audited my own audit log, and it failed

The ledger, AuditLog, is append-only by construction, not by policy: timestamps: { createdAt: true, updatedAt: false }. There are no update or delete paths — entries are write-once, roughly the shape of CloudTrail. One subtlety is the correct call and the non-obvious one: writeAudit() never throws into the caller's flow, because auditing must not mask the underlying action result. A failing ledger must not turn a successful stop into a 500 — but it also means a silently broken ledger is possible, which is exactly what bit me.

The reference endpoint, POST /api/connections/[id]/vms/control, is short enough to show almost whole. It is the whole posture made concrete:

export const POST = route(async (req, { params }) => {
  await enforceRateLimit(req, 'inthub');
  const user = requireRole(getUserFromRequest(req), 'admin', 'operator');
  const { action, vmId, size } = await parseBody(req, vmControlBody);   // includes confirm: z.literal(true)

  const provider = await resolveProvider(params.id, user.email ?? null);
  const audit = { actor: user.email, action: `vm.${action}`, provider: provider.kind,
                  connectionId: params.id, resourceId: vmId, params: size ? { size } : {} };
  try {
    switch (action) { /* start | stop | restart | resize → the port, never the SDK */ }
  } catch (err) {
    await writeAudit({ ...audit, outcome: 'failure', error: (err as Error).message });   // failures too
    throw new AppError(`VM ${action} failed: …`, 502, 'UPSTREAM_ERROR');
  }
  await writeAudit({ ...audit, outcome: 'success' });
  return NextResponse.json({ success: true, action, vmId, status: 'accepted' });
});

Three details in there each earn a sentence. Failures are audited — most audit implementations record only what worked, but the interesting forensic signal, repeated denied attempts, lives in the failures. status: 'accepted', not 'done' — Azure VM power transitions take 30–60 seconds and more, so the adapter awaits acceptance of the long-running operation, not completion, because the transition is too long to block a serverless request; the response tells the truth about what happened. And "stop" means deallocate — beginDeallocate, not power-off, because someone who clicks stop wants the bill to stop. That's a product decision expressed in one method call.

The honest beat

The requirement was an append-only audit log on every mutation, with confirm-gating and operator/admin RBAC on all control endpoints. A scheduled security sweep enumerated every mutation route and graded each for requireRole, confirm, and writeAudit. The finding, verbatim:

Concretely: fifteen mutation exports across fourteen route files, and exactly one — vms/control — was complete. Five privileged credential mutations were RBAC-gated but unaudited. Two destructive deletes had no confirm gate at all. The failure mode is instructive and generalises: the audit log was built as part of the VM-control feature, so it was wired everywhere VM control went and nowhere else. The ledger was excellent. Its coverage was a feature-shaped accident. The ADR had even predicted the mechanism — "if a control action is added that bypasses the port, it escapes the audit log" — it just didn't anticipate that the vault's own mutations were already outside the port.

Reported honestly: the gap was written up as a line-precise coverage matrix, re-swept twice more on a schedule with dated NO DRIFT rows against a real HEAD, then fixed — writeAudit on success and failure for all five, plus a new auditCoverage.test.ts, verified in Docker (tsc clean, 13 suites and 195 tests green) and shipped as azure_app PR #32. And one thing was deliberately not fixed: the confirm-gate on the two destructive DELETEs. Adding a required confirm: true to an existing DELETE is a breaking API change needing a coordinated frontend release, so it was queued rather than smuggled in. "I found it, I fixed the part that was safe to fix now, and I wrote down why the rest waits" is a better ending than a clean sweep.

09 · THE NEIGHBOUR I DIDN'T ABSORBThe Integration Hub

The system I refused to abstract deserves its own section, because it is genuinely interesting engineering and because it is where real Powerhouse infrastructure grounds the piece. The Integration Hub is a landing zone surfacing the museum's content-integration plane — Azure subscription phm-dev-inthub, region australiaeast. Live components, verified at build time: a Linux Function App content API (funcPhmIntHubContentDxpDev), a Cosmos DB SQL-API account (cosmosphminthubdev, Session consistency), three ingest functions for Momentus, Tessitura and Sanity, three Event Grid CloudEvents topics, an APIM gateway, and Auth0 for identity.

The insight that shaped the build: the content API only HTTP-exposes the aggregated events; the richer per-source data — venues, rooms, productions, performances, zones, programs — is only reachable via direct Cosmos queries. So the Integration Hub backend has to talk to both the content API over HTTP and Cosmos directly over SQL. That is a coupling to two data planes that no cloud-control port could ever model cleanly, which is precisely why it stayed out of the port.

The read-only SQL console is defence-in-depth done right, and the honest framing belongs here verbatim: "Cosmos's query API is read-only by nature; the guard below is defense-in-depth + enforces the SELECT shape and a hard row cap." assertReadOnlySelect() enforces a single statement (a trailing semicolon is stripped, any remaining one rejects, which blocks statement stacking), a leading SELECT, and a forbidden-keyword list matched on whole-word boundaries so column names like created or updatedField aren't flagged by CREATE/UPDATE — plus a hard row cap and mandatory paging.

10 · WHERE IT WAS HEADINGADR → plan → IaC → deploy

The feature the ports exist to serve is the most ambitious thing in the repo: a user describes the architecture they want as an ADR, and the app turns that ADR into deployed cloud infrastructure, with a human review gate before anything is provisioned. I'll be precise about what is real — most of this is an accepted design, and only the front door has shipped.

The pivot idea is what makes this architecture rather than a prompt: an ADR is a cloud-agnostic statement of intent; the architecture plan is a provider-neutral resource graph — a message bus, object-store tiers, serverless transforms, a managed API gateway — that Azure maps to Event Grid, Blob, Functions and APIM, and that AWS would map to EventBridge, S3, Lambda and API Gateway. The AWS path is "swap the IaC-generation job and the deployer adapter," with no changes to intake, elicitation, plan generation, the review gate, or the data model.

The quiet best idea is the ADR template. It splits every ADR into narrative fields (the "why," reasoning context for the model) and machine-actionable fields emitted as a typed block. The rationale is the sharpest sentence in the plan: "the LLM is bad at being a deterministic config source and good at narrative reasoning. By keeping the deploy-driving facts in a typed block, the plan generator validates them with Zod and uses the narrative only as reasoning context. This bounds AI blast radius." The non-negotiable guardrails follow from that: generated Bicep is validated with bicep build before it is ever shown as ready; what-if is mandatory before deploy; the AI never triggers a deployment, a human does; guardrail cross-checks like budget and allowed regions are enforced by deterministic code, not the LLM; and the confirm-token is bound to plan version plus what-if hash, so approving plan v2 does not authorize deploying v3.

What actually shipped is BP-2, the ingestion front door, and it exercises the whole LLM port: POST /api/blueprints/adrs/upload → RBAC → multipart parse (15 MB cap) → text extraction by mime (Markdown as UTF-8, PDF via pdf-parse, DOCX via mammoth, all pure-Node and dynamically imported so the markdown path never loads the binary libs) → resolveLlmProvider → completeStructured against a JSON Schema → Zod-validate-with-one-repair → raw bytes streamed to GridFS → persist → 202. Two details are worth keeping: the raw bytes are retained so a future re-extraction with a better model needs no re-upload — the Bronze-layer, keep-the-raw-input instinct — and the GridFS key never leaves the server, redacted in the public projector and stripped in the model's toJSON. Belt and braces, because one of those layers will eventually be forgotten. Everything past BP-2 — context elicitation, plan generation, IaC generation, review gate, deploy — is specced and not built. So are the remaining CloudProvider tasks: real inventory and cost, the control dashboard, the install bundle.

11 · HOW IT WAS BUILTADR-first, agent-built, self-auditing

This is the part I'd share the piece for. The system was not just designed with ADRs; it was built by an autonomous multi-agent pipeline that the ADRs governed, and then re-audited on a schedule by that same pipeline. The architecture governed itself.

The mechanics are all verifiable in the repo. A parent orchestration repo, AZURE, holds the ADRs, specs, plans and state and has no build target of its own; the deployable code lives in azure_app, and every spec carries frontmatter saying exactly that. The specs are written to be executable by an agent with no prior context — each is headed "this document is the complete build brief," with a "current state (frozen, do not re-derive)" block, an interesting artefact of writing for a reader whose main failure mode is confidently re-inventing what already exists. Some specs even correct the task that spawned them: one opens by noting the fleet task called the methods listResources/getCost while the shipped port says listComputeInstances/getCostSummary — "use the shipped names, do not rename the port." The spec defends the contract against its own work order.

A scheduled headless runner worked a governance queue in ten-day improvement cycles — ADR freshness, doc-currency sweeps, parity audits, and the recurring audit/RBAC no-drift re-sweep — executed off-hours, each producing commits and dated verification rows. Crucially, the invariants are enforced by CI, not by hope: the control-dashboard spec makes provider-agnosticism a test — components bind to port-shaped DTOs, never to an @azure/* import and never to an if (kind === 'azure') branch — and that invariant is "an acceptance criterion and a CI grep gate." An architectural rule that isn't mechanically checked is just a preference. Verification was Docker-only throughout — no npm or node on the host — and when GitHub Actions died on billing mid-project, the fallback was a local-CI script that ran the real gates in a container and posted statuses only for checks that genuinely passed.

The architecture that could describe itself precisely enough for an agent to build it was the same architecture that could describe itself precisely enough to audit itself.

That is not a coincidence. The no-drift sweep re-enumerated every mutation route and re-graded it against three criteria with no human in the loop. A narrow, explicit contract isn't just cheaper to implement — it is the only kind a machine can check.

12 · THE PROOFWhat shipped, and what didn't

As with everything I write up, this is drawn against real HEADs and PR numbers, not a whiteboard. Production is Next.js Route Handlers on Vercel with MongoDB Atlas. Here is what is shipped and live:

• The CloudProvider port, CloudProviderFactory and AzureProvider — verifyConnection, listComputeInstances, and all four VM-lifecycle methods, resolved per-request from the vault (PR #28)
• The encrypted owner-scoped Connection vault — AES-256-GCM, select:false secret, allowlist projector, fail-closed on a missing master key
• Append-only AuditLog plus confirm-gated destructive RBAC on VM control (PR #29)
• The LlmProvider port, LlmKey vault and both adapters — native Anthropic and a baseUrl-switched OpenAI-compatible one — with a verify route (PR #31)
• Blueprint Pipeline BP-2: ADR upload → mime-aware text extraction → LLM structured extraction → Zod-validated document plus GridFS raw retention
• P1 audit-coverage remediation — writeAudit on success and failure across all five previously unaudited privileged mutations, plus auditCoverage.test.ts; verified in Docker, tsc clean, 13 suites / 195 tests green (PR #32)
• The Integration Hub end to end — live service-health grid, content-API proxy, Cosmos browse with continuation-token paging, guarded read-only SQL console, saved views
• The Express-API consolidation into Route Handlers — same-origin /api/*, CORS and NEXT_PUBLIC_API_URL deleted, one deploy on the free tier
• Six ADRs (001–006), each with a shipped-reality verification pass recorded against real HEAD SHAs; a self-documenting OpenAPI 3.0 surface and interactive docs

And here is what is designed, specced, and honestly not built — said plainly, because the line between the two is the whole point:

• CloudProvider real inventory and cost — getCostSummary still throws, and ComputeInstance.state is honestly 'unknown' in v1 because listAll() doesn't carry power state; the code refuses to invent a state it can't see
• The provider-agnostic control dashboard and the one-command install bundle
• The CloudDeployer port, AzureDeployer, Deployment model and what-if preview
• Blueprint stages 3–6: context elicitation, plan generation, IaC generation, review gate, deploy
• Confirm-gates on the two destructive vault DELETEs — a breaking change, deliberately queued
• AWS and GCP adapters, the Gemini adapter, and Azure AD / MSAL auth — ADR-001 is deferred, the MSAL packages are installed but unwired, and production auth is local JWT plus requireRole RBAC

The claim the shipped work supports, at exactly this size and no larger: a user can connect their own cloud account, have the credential sealed at rest, verify it authenticates and is authorized, list their compute inventory through a provider-neutral contract, control a VM behind role-and-confirm gating, and read an immutable ledger of every attempt — with the vendor name appearing in exactly one file.

One small detail that separates a real integration from a plausible one: verifyConnection is two calls, not one. Getting a token proves the service principal authenticates; it does not prove it is authorized on the target subscription. So the adapter does both — acquire the token, then one cheap subscription-scoped resourceGroups.list() — the analogue of AWS's sts:GetCallerIdentity plus a scoped probe. A green tick that only means "your password is right" is worse than no tick.