VPC in Depth — Subnet Design · Peering · Transit Gateway · PrivateLink

The first chapter of Part 5, operations · security · cost. Through Part 4 we put one service on ECS Fargate inside a single VPC and operated it. If Chapter 8 EC2 and VPC basics set the basics of VPC · subnet · route table · internet gateway · security group, this chapter covers how to design a production-scale network on top of that.

Once you get into operations, a single public subnet isn’t enough. The DB must not be reachable from the internet, multiple environments and multiple VPCs appear, and even private segments need to reach AWS services. The subnet tiering · inter-VPC connectivity · private-access model we set in this chapter carries straight over into the multi-account work of Chapter 29 security governance and the infrastructure skeleton of Part 6 Deploying a Fullstack App on AWS. The code in this chapter assumes the HCL from Chapter 25 Terraform intro.

From Chapter 8 to Part 5 #

Here’s what Chapter 8 covered.

• A VPC is a private network you create inside an account, and a CIDR block (e.g., 10.0.0.0/16) defines its IP range.
• A subnet is a partition that divides a VPC by AZ.
• A subnet whose routing connects to an internet gateway (IGW) is public; one that doesn’t is private.
• A security group is an instance-level firewall; a network ACL is a subnet-level firewall.

This chapter extends those basic elements into multiple tiers · multiple AZs · multiple VPCs.

Subnet Design — 3-tier and 4-tier #

The basic skeleton of a production VPC is 3-tier. You split subnets with different roles into tiers.

In heavily regulated environments you sometimes add a management-only tier (Bastion / ops tools) on top of this, splitting it into 4-tier. The key is that the isolated tier where the DB lives has no path out to the internet at all. The app goes out from private through NAT; the DB goes out nowhere. Access between tiers is opened one direction at a time with security groups — ALB → app (8000), app → DB (5432) — referencing the source security group to control by role rather than by IP.

resource "aws_security_group_rule" "app_from_alb" {
  type                     = "ingress"
  security_group_id        = aws_security_group.app.id
  source_security_group_id = aws_security_group.alb.id   # only what came from the ALB
  from_port                = 8000
  to_port                  = 8000
  protocol                 = "tcp"
}

resource "aws_security_group_rule" "db_from_app" {
  type                     = "ingress"
  security_group_id        = aws_security_group.db.id
  source_security_group_id = aws_security_group.app.id   # only what came from the app
  from_port                = 5432
  to_port                  = 5432
  protocol                 = "tcp"
}

CIDR planning #

Once you set a CIDR it’s a hassle to change. Carve it generously and regularly from the start. Here’s an example with two AZs × three tiers.

VPC            10.0.0.0/16     (65,536 IP)

AZ a
  public       10.0.0.0/20     (4,096)
  private-app  10.0.16.0/20    (4,096)
  isolated-db  10.0.32.0/20    (4,096)
AZ c
  public       10.0.128.0/20   (4,096)
  private-app  10.0.144.0/20   (4,096)
  isolated-db  10.0.160.0/20   (4,096)

• Take the VPC big, at /16. Leave room to add tiers or AZs later.
• Each subnet at around /20 (4,096 IPs) keeps IPs from drying up even as ECS Tasks grow. Since one Fargate Task uses one ENI = one IP, high container density turns IP exhaustion into a real outage. Also account for the 5 per subnet that AWS reserves (network/broadcast/gateway, etc.).
• Space out the high bits between AZs (10.0.0.x vs 10.0.128.x) to keep it readable later.
• If there’s any chance you’ll Peer with another VPC, make sure the CIDRs don’t overlap. Two VPCs with overlapping CIDRs can’t be peered. At the organization level, reserve numbers per VPC up front like 10.0.0.0/16, 10.1.0.0/16.

Route tables #

Keep a separate route table per tier. The difference in routing is precisely the definition of a tier.

# public: bidirectional to the internet
resource "aws_route" "public_internet" {
  route_table_id         = aws_route_table.public.id
  destination_cidr_block = "0.0.0.0/0"
  gateway_id             = aws_internet_gateway.main.id
}

# private-app: outbound only via per-AZ NAT
resource "aws_route" "private_nat" {
  for_each               = aws_nat_gateway.az          # one per AZ
  route_table_id         = aws_route_table.private[each.key].id
  destination_cidr_block = "0.0.0.0/0"
  nat_gateway_id         = each.value.id
}

# isolated-db: no default route — local only. cut off from the internet
# (don't put a 0.0.0.0/0 route on aws_route_table.isolated)

Not putting a 0.0.0.0/0 route on the isolated-db route table is exactly how you cut the DB off from the internet.

Internet Ingress/Egress — IGW · NAT · Egress-only IGW #

Apps in the private tier often need to go out (installing packages, calling external APIs, ECR pull). But inbound must be blocked. NAT resolves this asymmetry.

resource "aws_eip" "nat" {
  for_each = toset(["a", "c"])
  domain   = "vpc"
}

resource "aws_nat_gateway" "az" {
  for_each      = aws_eip.nat
  allocation_id = each.value.id
  subnet_id     = aws_subnet.public[each.key].id   # NAT lives in public
}

Watch NAT Gateway cost #

NAT Gateway is a surprisingly large line item on the operations bill. On top of the hourly rate (around $0.059/h in Seoul, ~$43/month per NAT) it’s billed per GB processed (about $0.059/GB). So when a private app goes in and out through NAT for traffic-heavy AWS services like S3 / ECR / DynamoDB, cost piles up fast.

For example, with NAT in 2 AZs processing 500GB a month, it’s roughly:

hourly      $0.059 × 24h × 30 days × 2     ≈ $85
throughput  $0.059 × 500GB                 ≈ $30
                                          ─────────
                                           about $115 / month

Ways to cut it:

• One NAT Gateway per AZ — avoid inter-AZ data transfer cost. Consolidating into a single NAT looks cheap, but on an AZ failure all AZs lose outbound, and apps in the other AZs cross over to the AZ that has the NAT, incurring inter-AZ transfer fees.
• Bypass NAT with VPC Endpoints — S3 and DynamoDB have free Gateway Endpoints. Send ECR · CloudWatch · Secrets Manager traffic through Interface Endpoints too, and it drops out of NAT processing cost. In container environments ECR pull is heavy traffic, so the effect is large.
• For learning / dev environments, putting the app in a public subnet instead of NAT, or turning NAT on only when needed, is also an option.

Look at this cost angle alongside Chapter 27 cost optimization and the multi-AZ design in Chapter 30 disaster recovery & backup.

VPC Endpoint and PrivateLink #

Even the private / isolated tiers need to reach AWS services (S3, Secrets Manager, ECR, etc.). Here, instead of going out to the internet and back, you make a private path with a VPC Endpoint.

# S3 — free Gateway Endpoint that attaches to the route table
resource "aws_vpc_endpoint" "s3" {
  vpc_id            = aws_vpc.main.id
  service_name      = "com.amazonaws.ap-northeast-2.s3"
  vpc_endpoint_type = "Gateway"
  route_table_ids   = [for rt in aws_route_table.private : rt.id]
}

# ECR — Interface Endpoint that creates an ENI in the private subnet
resource "aws_vpc_endpoint" "ecr_dkr" {
  vpc_id              = aws_vpc.main.id
  service_name        = "com.amazonaws.ap-northeast-2.ecr.dkr"
  vpc_endpoint_type   = "Interface"
  subnet_ids          = [for s in aws_subnet.private : s.id]
  security_group_ids  = [aws_security_group.endpoints.id]
  private_dns_enabled = true   # the existing ECR domain resolves to a private IP
}

With private_dns_enabled = true, existing domains like *.ecr.ap-northeast-2.amazonaws.com automatically resolve to the Endpoint’s private IP, so you don’t need to change app code.

PrivateLink is also the technology that exposes one VPC’s service to another VPC or account without the internet. Put an NLB in front of an internal API one team built and publish it with aws_vpc_endpoint_service, and the consumer VPC attaches to that service via an Interface Endpoint. PrivateLink’s strength over Peering is that it connects privately even when the two VPCs’ CIDRs overlap. An Interface Endpoint is also the clean path when an isolated app reads a secret from Chapter 20 Secrets Manager / Parameter Store.

Connecting VPCs — Peering and Transit Gateway #

Once you have two or more VPCs (environment separation, account separation, an acquired system, etc.), there are two ways to link them privately.

VPC Peering #

Connects two VPCs directly, 1:1. It only works after you create the connection and manually add the other side’s CIDR route to both route tables.

resource "aws_vpc_peering_connection" "a_to_b" {
  vpc_id      = aws_vpc.a.id
  peer_vpc_id = aws_vpc.b.id
  auto_accept = true            # when same account/region
}

resource "aws_route" "a_to_b" {
  route_table_id            = aws_route_table.a.id
  destination_cidr_block    = aws_vpc.b.cidr_block   # 10.1.0.0/16
  vpc_peering_connection_id = aws_vpc_peering_connection.a_to_b.id
}

resource "aws_route" "b_to_a" {
  route_table_id            = aws_route_table.b.id
  destination_cidr_block    = aws_vpc.a.cidr_block   # 10.0.0.0/16
  vpc_peering_connection_id = aws_vpc_peering_connection.a_to_b.id
}

• The setup is simple, and it works across same / different accounts · regions.
• It’s non-transitive. Peering A–B and B–C does not automatically connect A–C. With N VPCs the connections grow to N(N-1)/2, so even with just 5 you have to manage 10 peerings and twice as many routes.
• The two CIDRs must not overlap.

Transit Gateway #

A way of attaching many VPCs to a single hub.

resource "aws_ec2_transit_gateway" "hub" {
  description                     = "org hub"
  default_route_table_association = "enable"
  default_route_table_propagation = "enable"
}

resource "aws_ec2_transit_gateway_vpc_attachment" "a" {
  transit_gateway_id = aws_ec2_transit_gateway.hub.id
  vpc_id             = aws_vpc.a.id
  subnet_ids         = [for s in aws_subnet.a_private : s.id]
}
# VPCs b, c... attach the same way, just once each

• You attach each VPC to the Transit Gateway just once (a star topology). As VPCs grow, the number of connections increases only linearly.
• Use TGW route tables to control which VPCs can communicate. For example, you can keep dev and prod from seeing each other while letting both reach a shared-services VPC. On-premises (VPN / Direct Connect) attaches to the same hub too.
• There’s per-hour (per-attachment) + per-GB billing, so when you have only 2 ~ 3 VPCs, Peering is cheaper.

Decision rule: Peering for 2 ~ 3 VPCs, Transit Gateway once there are signs of growing beyond that. An organization headed toward multi-account governance (Chapter 29) is better off laying down a Transit Gateway from the start — it’s easier later. The Part 6 capstone uses neither since it’s a single VPC, but the moment a second environment account appears, this decision shows up.

IPv6 Adoption Decision Guide #

IPv6 isn’t needed for every project. Consider it when:

• Outbound traffic is heavy and NAT Gateway cost is a burden. IPv6 outbound is handled by the free Egress-only IGW, so you can cut NAT cost.
• Public IPv4 addresses are paid (billed per hour per public IPv4 in use since February 2024), so there’s room to save cost on workloads that need a large number of public IPs.
• You need to directly accept clients coming in over IPv6 from outside.

Most adopt it as dual-stack (IPv4 + IPv6 at the same time).

resource "aws_vpc" "main" {
  cidr_block                       = "10.0.0.0/16"
  assign_generated_ipv6_cidr_block = true     # AWS assigns a /56 IPv6
}

resource "aws_egress_only_internet_gateway" "main" {
  vpc_id = aws_vpc.main.id                     # IPv6 outbound (free)
}

resource "aws_route" "private_ipv6" {
  route_table_id              = aws_route_table.private["a"].id
  destination_ipv6_cidr_block = "::/0"
  egress_only_gateway_id      = aws_egress_only_internet_gateway.main.id
}

A single simple backend is fine on IPv4 alone, so bring in IPv6 when the cost or the requirement is clear.

Multi-VPC Mental Model #

Here’s the criteria for when to split a VPC.

• Single VPC + tiered subnets — for one service, one team, one environment, this skeleton is enough. It’s the model we used through Part 4.
• VPC per environment — splitting dev / staging / prod into VPCs confines the blast radius of an incident inside the environment. That said, true isolation is stronger with account separation (Chapter 29).
• Account + VPC combination — as scale grows, you split accounts by environment / team, put a VPC in each account, and link only what’s needed with Transit Gateway. This is the common shape of a production organization.

The primary motivation for splitting a VPC is usually the security boundary and blast radius. That deeper story continues into Organizations · SCP in the next Chapter 29 security governance.

Exercises #

1. Set your service’s VPC to 10.0.0.0/16 and assign the CIDRs for six subnets by hand across a 2 AZ × 3-tier (public / private-app / isolated-db) layout. For each subnet, write one sentence on whether its route table has IGW / NAT / local as the default route, and explain why the isolated tier has no 0.0.0.0/0 route. When you write the Terraform VPC module in Chapter 32 Deploying a Fullstack App on AWS, this table becomes the direct input.
2. Suppose a private app goes outbound to three places — ① S3, ② ECR (image pull), ③ an external payment API. Write which of NAT Gateway · Gateway Endpoint · Interface Endpoint each should be sent through to minimize cost and why, basing it on the estimate in §“Watch NAT Gateway cost.”
3. If you currently have 2 VPCs and plan to grow to 5 ~ 6 within a year, decide whether to start with Peering or Transit Gateway, and write one paragraph based on the connection count (N(N-1)/2 vs linear) and the routing-management burden.

In short: Design a production VPC as a three-tier layout of public / private-app / isolated-db, and cut the DB tier off from the internet by giving its route table no 0.0.0.0/0 route at all. Open inter-tier access in a role-based way with security group references. Send the private app’s outbound through per-AZ NAT Gateways, but bypass S3, DynamoDB, and similar services with Gateway Endpoints or Interface Endpoints to cut the per-hour and per-GB NAT cost. Stitch 2 ~ 3 VPCs together with Peering (non-transitive, manual routes on both sides), and more with Transit Gateway (star hub, controlled by TGW routing). PrivateLink exposes a service privately even when CIDRs overlap, and IPv6 is adopted as dual-stack plus Egress-only IGW when NAT or public IPv4 cost is a burden. The primary motivation for splitting a VPC is the security boundary and blast radius.

Next chapter #

In the next Chapter 29 security governance we cover the point at which you move from a single account to multi-account. We’ll lay out the model of grouping accounts and applying policies in bulk with AWS Organizations · SCP · Control Tower, and the governance of watching the whole account fleet with GuardDuty · Security Hub · Config. It’s the natural next step where this chapter’s multi-VPC expands into multi-account.