NAT Gateway Placement, Failure Modes, and What Egress Actually Costs

A NAT Gateway is one of the few AWS networking primitives where a single attribute — which subnet you put it in — simultaneously decides whether traffic flows, what happens during an AZ impairment, and how much you pay per gigabyte. Most teams get the first part right by copying a reference architecture and never revisit the other two.

What a NAT Gateway actually is#

It helps to be precise, because the mental model most people carry is wrong in a way that produces real outages.

A NAT Gateway is not a service endpoint that you point traffic at and it reaches the internet. It is a managed appliance that occupies an IP address in a specific subnet. When a packet arrives, it performs source network address translation and then forwards the result according to the route table associated with its own subnet.

That last clause is the whole thing. Two consequences follow:

1. A NAT Gateway must sit in a subnet whose route table has a default route to an internet gateway. In practice that means a public subnet.
2. A NAT Gateway placed in the private subnet it serves creates a routing loop. Traffic arrives, gets translated, and is handed back to the same route table that just sent it there.

The failure signature is a timeout, never a refusal. VPC routing drops packets without generating ICMP unreachable messages, so the client retransmits SYNs until it gives up. If you remember one diagnostic heuristic from this guide, make it that one:

| Symptom | What it eliminates | | --- | --- | | Connection refused | Routing is fine — you reached a host, nothing was listening | | Connection timed out | Packet died in transit — routing, NACL, or security group | | TLS or certificate error | Routing and TCP are fine — the problem is above layer 4 | | DNS resolution failure | Never reached the routing layer at all |

A timeout on outbound traffic from a private subnet means you are looking for something that forwards. Start with the middlebox, not the client.

Reading the two route tables together#

Diagnosing this class of problem requires looking at two route tables as a pair. Neither is wrong on its own.

Destination	Target	State	Origin
10.0.0.0/16	local	active	CreateRouteTable
0.0.0.0/0	nat-0f8e…	activeCorrect — the private tier should egress via NAT. Nothing to fix here.	CreateRoute

The private route table is exactly right, which is why teams stop looking after checking it.

Destination	Target	State	Origin
10.0.0.0/16	local	active	CreateRouteTable
0.0.0.0/0	igw-04c…	activeThis is the route the NAT Gateway needs to consult. It only does so if it is in THIS subnet.	CreateRoute

Both tables are individually valid. The defect is the relationship between them and the NAT Gateway's placement.

This is the most common shape of a cloud networking bug: every object is healthy, and the fault is in a relationship no single describe call reveals. When a checklist of individual resources all comes back green and traffic still fails, widen the aperture instead of re-checking the same resources harder.

Availability: one NAT Gateway per AZ#

A NAT Gateway is a zonal resource. It lives in one availability zone and it does not fail over.

A single shared NAT Gateway serving private subnets across three AZs produces two problems:

• Blast radius. If that AZ is impaired, egress breaks for all three AZs, not one. You have taken a zonal dependency for your entire outbound path.
• Cross-AZ data processing charges. Traffic from us-east-1b to a NAT Gateway in us-east-1a crosses an AZ boundary and is billed for it, on top of the NAT processing charge.

The correct pattern is one NAT Gateway per AZ, each in that AZ's public subnet, with each private subnet's route table pointing at the gateway in its own AZ.

What it costs#

NAT Gateway pricing has two components, and the second is the one that surprises people:

• An hourly charge per gateway, roughly $0.045/hour in us-east-1. About $32/month, billed whether or not a single packet flows. A misconfigured, non-functional NAT Gateway bills exactly the same as a working one.
• A data processing charge per gigabyte, roughly $0.045/GB, applied to all traffic through the gateway in either direction.

That per-GB charge is where budgets quietly break. It is levied in addition to any internet data transfer cost, and it applies to traffic that never leaves AWS.

The S3 case#

Consider an application in a private subnet pulling 10 TB/month from S3 in the same region.

Routed through a NAT Gateway, that 10 TB incurs the data processing charge — roughly $450/month — even though the traffic never touches the internet. A gateway VPC endpoint for S3 costs nothing per hour and nothing per gigabyte, and removes the NAT Gateway from that path entirely.

The fix is a route, not an application change:

Diff of endpoints.tf: 9 lines added, 1 lines removed.

1		-	# S3 traffic from private subnets egresses via NAT
2	1		resource "aws_route_table" "private" {
3	2		vpc_id = aws_vpc.main.id
4	3
⋯ 2 unchanged lines
7	6		nat_gateway_id = aws_nat_gateway.per_az["a"].id
8	7		}
9	8		}
	9	+
	10	+	# Gateway endpoints are free: no hourly charge, no per-GB charge. Associating
	11	+	# one injects a prefix-list route that bypasses the NAT Gateway for S3.
	12	+	resource "aws_vpc_endpoint" "s3" {
	13	+	vpc_id = aws_vpc.main.id
	14	+	service_name = "com.amazonaws.${var.region}.s3"
	15	+	vpc_endpoint_type = "Gateway"
	16	+	route_table_ids = [aws_route_table.private.id]
	17	+	}

Gateway endpoints exist for S3 and DynamoDB only. Everything else is an interface endpoint, which does carry an hourly charge.

Gateway versus interface endpoints#

The distinction matters for cost modelling:

• Gateway endpoints (S3, DynamoDB) are free. They work by injecting a route into the route tables you associate. There is no reason not to use them.
• Interface endpoints (everything else — SSM, ECR, Secrets Manager, and so on) are ENIs in your subnets, billed hourly per endpoint per AZ plus a per-GB charge. Cheaper than NAT for high-volume paths, more expensive for trivial ones. Model it before deploying dozens.

Guardrails#

Placement is not something to leave to review. Three checks worth automating:

1. A policy assertion that aws_nat_gateway.subnet_id resolves to a subnet tagged Tier = "public". Cheap in tflint, Sentinel, or Conftest, and it catches the outage class described above before apply.
2. A synthetic egress check from the private tier in CI. This failure mode is invisible until real traffic needs it — which, if you are unlucky, is during a batch run at month end.
3. A cost alarm on NatGateway-Bytes. A sudden step change usually means a new workload started pulling from S3 or ECR through NAT instead of through an endpoint.

Where to practice#

The failure mode at the top of this guide is Lab 01. It deploys a VPC where the NAT Gateway reports healthy and every outbound connection times out. Work the ticket before reading the debrief.