INC-1088 Lab 02: The Hardened Subnet That Broke Everything
Reporting service cannot reach the database after a NACL hardening change
Cloud networking, at packet level
Reading about VPC routing does not build intuition. Debugging a broken one does. Every lab hands you a Terraform file that deploys a genuinely broken environment and an incident ticket — no hints, no answer key until you ask for it.
Trace a real failure — the outage behind Lab 01:
Run the trace to see where the packet dies.
INC-1043
A batch job in a private subnet cannot reach a third-party API. The route table looks correct, the security groups are wide open, and the NAT Gateway reports Available. Find out why every outbound connection still hangs.
Reported symptom
Outbound API calls from the private subnet time out after deploying the new VPC
Reported by Payments Platform
Reporting service cannot reach the database after a NACL hardening change
Newly onboarded spoke VPC cannot reach shared services; the existing spoke works
Application cannot reach the vendor API through its new PrivateLink endpoint
Address planning, subnet tiering, route table scoping, and the boundary rules that decide whether a packet ever leaves the VPC.
What a target actually sees as the source address, how health checks decide which zones receive traffic, and the defaults that silently change both.
NAT Gateway placement, per-AZ data processing charges, gateway versus interface endpoints, and where egress spend actually accrues.
Centralized inspection topologies, appliance mode, route table associations, and propagation across attachments.
Direct Connect, Site-to-Site VPN, and the BGP attributes that control which path your traffic actually takes on the way back.
Every instance can send 1,024 packets per second to the VPC resolver, per network interface, and the number is the same on a t3.nano and a u-24tb1.metal. Throttled queries never reach anything that logs, so the symptom is intermittent DNS failure with no evidence anywhere — and scaling up, the instinctive fix, changes nothing.
EC2 instance bandwidth is one number on the spec sheet and at least three in practice. Traffic through an internet gateway is capped at 5 Gbps on anything under 32 vCPUs, and a single flow is capped at 5 Gbps nearly everywhere. Neither is a fault, and both get filed as one.
A flow log with no REJECT lines is routinely read as proof that nothing was blocked. AWS documents ten categories of traffic that flow logs exclude by design, and several of them are precisely the traffic you would be looking for when something is wrong.
The zonal NAT Gateway has a placement rule, a per-AZ route table requirement, and a single-zone dependency, and each one is a documented outage. Regional mode does not fix those bugs. It removes the configuration they live in.