Safe Decommissioning of Legacy Cloud Infrastructure via Telemetry Analysis and Controlled Quarantine
Operational Challenge
Following the migration of a Learning Record Store (LRS) platform to a newly established AWS account, a decommissioning request was raised for the legacy infrastructure in the source account. The resources scheduled for deletion included an Elastic Beanstalk EC2 environment, a MySQL Amazon RDS instance, Kinesis streams, Lambda functions, and four DynamoDB tables holding over 2.15 TB of data. Production governance prohibited direct resource termination without dependency verification. Furthermore, CloudWatch metrics showed nearly 8,000 daily requests hitting the legacy Application Load Balancer, raising concerns about potential unmigrated clients. Additionally, there was a high risk of inadvertently decommissioning a shared VPC containing active network interfaces from independent mission-critical production systems.
Technical Diagnosis
The technical discovery began with an audit of Elastic Network Interfaces (ENIs) inside the VPC slated for decommission. This uncovered active dependencies including ElastiCache nodes, ECS task attachments, ancillary load balancers, and an operational jumpbox, leading to the immediate exclusion of the VPC from the decommissioning scope. To analyze the persistent ALB traffic, telemetry from the Traffic Shadowing Lambda was inspected. Detailed payload parsing demonstrated zero requests targeting the official application Host header or xAPI ingestion endpoints; all inbound traffic consisted of automated botnet scans hitting public ALB IP addresses directly due to a catch-all listener rule. Finally, an audit of DynamoDB disaster recovery posture revealed that the critical ‘statementV2’ table (304 GB) had Point-in-Time Recovery (PITR) disabled and no active backups, requiring manual remediation prior to isolation.
To audit CloudWatch Logs filtering strictly for legitimate incoming Host headers, the following query was executed:
START_TIME=$(date -d "48 hours ago" +%s000)
aws logs filter-log-events --log-group-name "/aws/lambda/lrs-shadow-processor-pro" --start-time "$START_TIME" --filter-pattern '{ $.request.headers.host = "lrs.legacy-domain.internal" }' --query "events[*]" --output json
The query returned an empty set over 48 hours, confirming that the legacy platform was fully decoupled from legitimate production traffic.
Implemented Solution
A phased decommissioning strategy was implemented, consisting of preventive data safeguarding, a 15-day controlled quarantine, and final resource termination. Prior to the maintenance window, an on-demand native backup of DynamoDB ‘statementV2’ was created to secure 304 GB of historical records. During the maintenance window, compute was isolated by scaling the Elastic Beanstalk Auto Scaling Group to zero instances, stopping the RDS MySQL instance, and setting reserved concurrency to zero on shadow Lambda functions to reject bot requests. Functional cutover verification was completed by issuing an end-to-end ingestion request against the new endpoint, which successfully returned an HTTP 200 code and transaction UUID. Following the incident-free 15-day quarantine, and accommodating an engineering request to retain ‘statementV2’ temporarily, the Beanstalk stack, Kinesis streams, Lambdas, and remaining DynamoDB tables were terminated, accompanied by a mandatory final snapshot of the RDS instance.
To isolate the legacy Beanstalk compute tier while preserving full rollback readiness, the following command was executed:
aws autoscaling update-auto-scaling-group --auto-scaling-group-name awseb-legacy-asg --min-size 0 --max-size 0 --desired-capacity 0
The rollback plan during the quarantine period required only restoring the Auto Scaling capacity to 3 instances and starting the RDS instance via CLI, ensuring recovery within 10 minutes if issues arose.
Key Metrics & Impact
/>Technology Stack
/>