Billing Separator

Last Updated: 2026-03-06 Source: innovation-sandbox-on-aws-billing-seperator Captured SHA: 47ae71d

Executive Summary

The ISB Billing Separator is an explicitly temporary workaround that enforces a hard 91-day quarantine on sandbox accounts after cleanup, preventing billing attribution errors and quota exhaustion for subsequent users. It intercepts CloudTrail MoveAccount events via cross-account EventBridge forwarding from the Organization Management account (us-east-1) to the Hub account (us-west-2), redirecting accounts from the Available OU to a Quarantine OU. After 91 days, an EventBridge Scheduler triggers release back to Available. The entire repository is intended for archival once ISB implements native hard cooldown support (upstream issue #70).

Architecture Overview

The system deploys as two CDK stacks across two AWS accounts. The OrgMgmtStack in the Organization Management account (us-east-1) captures CloudTrail MoveAccount events and forwards them cross-account. The HubStack in the Hub account (us-west-2) processes events via SQS, moves accounts to quarantine, and schedules delayed release.

Component Architecture

Event Flow

Core Components

OrgMgmtStack (us-east-1)

Deployed to the Organization Management account. Contains a single EventBridge rule that captures MoveAccount CloudTrail events where the destination is the Available OU, forwarding them cross-account to the Hub's custom event bus via an IAM role.

Source: lib/org-mgmt-stack.ts

Additionally creates a self-managed IAM role (isb-billing-sep-org-mgt-{env}) that grants Organizations API access (MoveAccount, DescribeOrganizationalUnit, ListOrganizationalUnitsForParent, ListTagsForResource, UntagResource) to the Hub account's intermediate role.

HubStack (us-west-2)

The main compute stack containing all processing resources.

Source: lib/hub-stack.ts

Resource	Purpose
Custom EventBridge Bus	Receives forwarded events from OrgMgmt
EventBridge Rule	Filters MoveAccount events to Available OU
SQS Queue + DLQ	Event buffering with 5 retries, 14-day DLQ retention
Rule DLQ	EventBridge rule delivery failures
QuarantineLambda	Intercepts and quarantines accounts (30s timeout, ARM64)
UnquarantineLambda	Releases accounts after 91 days (30s timeout, ARM64)
Scheduler Group	isb-billing-separator group for one-shot schedules
Intermediate IAM Role	Hub-side of cross-account role chain
SNS Alert Topic	Operational alarm notifications
CloudWatch Alarms	DLQ depth, Lambda errors, rule DLQ
CloudWatch Metric Filters	QuarantineSuccessCount, UnquarantineSuccessCount, QuarantineBypassTagCount

QuarantineLambda

Processes SQS events containing CloudTrail MoveAccount data. For each event:

1. Validates the account exists in ISB tracking (DynamoDB)
2. Checks idempotency (skips if already in Quarantine)
3. Validates source is CleanUp OU (fresh lookup via ISB commons SandboxOuService)
4. Checks for do-not-separate bypass tag (one-shot skip, tag consumed on use)
5. Moves account from Available to Quarantine OU via ISB's transactional move
6. Creates an EventBridge Scheduler one-shot schedule for 91-day release

Uses SQS partial batch response pattern for granular failure handling.

Source: source/lambdas/quarantine/handler.ts

UnquarantineLambda

Triggered directly by EventBridge Scheduler after 91 days. For each invocation:

1. Validates scheduler payload via Zod schema
2. Checks account exists and is in Quarantine status
3. Moves account from Quarantine to Available OU
4. Deletes the triggering schedule (idempotent, handles ResourceNotFoundException)

Source: source/lambdas/unquarantine/handler.ts

Quarantine Bypass

New accounts with no billing history can skip quarantine using the do-not-separate tag:

aws organizations tag-resource \
  --resource-id 023138541607 \
  --tags Key=do-not-separate,Value=

The tag is consumed on use (one-shot). If tag check fails, quarantine proceeds normally (fail-safe).

Cross-Account Role Chain

Both Lambda functions use ISB's cross-account credential chain:

Lambda Execution Role -> Intermediate Role (Hub) -> Org Mgt Role (OrgMgmt Account)

The intermediate role is trusted by both Lambda execution roles and has permission to assume the OrgMgt role. The OrgMgt role grants Organizations API access and is trusted by the intermediate role.

Source: ISB commons fromTemporaryIsbOrgManagementCredentials

Why 91 Days

AWS billing operates on calendar-month boundaries. A 91-day quarantine (approximately 3 full billing months) ensures:

1. Billing attribution: Previous user's charges fully settle before account reuse
2. Quota recovery: AWS service quotas reset across billing periods (upstream issue #88)
3. Safety margin: Covers edge cases in AWS billing data propagation

Temporary Nature

This repository is explicitly temporary. The README states: "This entire repository should be archived and the infrastructure destroyed once ISB implements native cooldown support."

Known limitations:

• Race condition between MoveAccount event and quarantine interception
• ISB UI/API shows quarantined accounts as "Available"
• Two additional CDK stacks across two accounts add operational complexity
• Manual reconciliation required if solution is removed mid-quarantine

Technology Stack

Component	Technology
Runtime	Node.js 22, TypeScript, ARM64
Infrastructure	AWS CDK v2.240+
ISB Integration	Git submodule (deps/isb/) for ISB commons
Build	esbuild with NodejsFunction construct
Testing	Jest with CDK assertions
Validation	Zod v4 schemas
Tracing	AWS X-Ray (active on both Lambdas)
Logging	JSON structured logging
CI/CD	GitHub Actions with OIDC

Observability

• CloudWatch Alarms: DLQ depth >=3, QuarantineLambda errors >=3, UnquarantineLambda errors >=3, Rule DLQ >=1
• Custom Metrics (ISB/BillingSeparator namespace): QuarantineSuccessCount, UnquarantineSuccessCount, QuarantineBypassTagCount
• X-Ray Tracing: Active on both Lambda functions
• Structured Logging: JSON with action, accountId, timestamp, and contextual details
• SNS Alerts: Email subscription support for operational notifications

---

Generated from source analysis of innovation-sandbox-on-aws-billing-seperator at SHA f8f1bdc. See 00-repo-inventory.md for full inventory. Cross-references: 10-isb-core-architecture.md, 22-cost-tracking.md.