Hardware Basics #7: Virtualization and Containers — How One Physical Server Becomes Many

Infrastructure Hardware Virtualization
6 min read

Through #6 we’ve seen all four resources that make up one server. Yet the cloud sells one physical server as if it were dozens. An EC2 instance can’t be a whole physical server. What makes this split possible is virtualization, and containers make the split even lighter on top of it. This post covers the principle.

Where this post sits in the Hardware Basics series:

Why split one server #

A single physical server usually has resources to spare most of the time. It’s rare for one service to keep CPU and memory fully used at all times. Buy one server per service and the spare resources are money sitting idle.

Virtualization gathers up that spare and splits one server into several independent environments. Each looks like its own server and is isolated from the rest. Utilization rises, and you can create and destroy environments in minutes. This is exactly what lets the cloud “rent as much as you need.”

Virtualization — a hypervisor splits the hardware #

You put a layer called a hypervisor on top of the physical server, and several virtual machines (VMs) on top of that. The hypervisor carves up the physical resources and hands them to each VM, isolating the VMs so they can’t intrude on one another.

Each VM holds an entire operating system of its own. To the VM it looks like it’s running on real hardware, and you install Linux or Windows separately inside it. One EC2 instance corresponds to one such VM.

Virtualization — each VM holds its own OS 
┌──────────┐ ┌──────────┐ ┌──────────┐
│  App A   │ │  App B   │ │  App C   │
│  OS      │ │  OS      │ │  OS      │   ← one OS per VM
└──────────┘ └──────────┘ └──────────┘
        Hypervisor
        Physical server (CPU,memory,storage,network)

A VM’s isolation is strong. Whatever happens inside one VM has very little effect on the others or the physical server. In exchange, carrying a whole OS uses a lot of memory and disk, and it takes time to boot.

Containers — sharing the OS kernel #

A container takes a different path. Instead of a separate OS per VM, several containers share the host’s OS kernel. A container is merely isolated at the process level on top of it, without carrying a whole OS of its own.

Containers — sharing the kernel 
┌──────┐ ┌──────┐ ┌──────┐
│ App A│ │ App B│ │ App C│   ← just the app and its dependencies, no OS
└──────┘ └──────┘ └──────┘
   Container runtime
   Shared OS kernel
   Physical server or VM

Sharing one OS makes containers lightweight. They use less memory and start in an instant rather than tens of seconds. You can run far more of them on the same server than VMs. This lightness is why Docker and Kubernetes took hold. The trade-off is that sharing the kernel makes isolation weaker than a VM’s.

AspectVirtual machine (VM)Container
OSone per VMshares the host kernel
Weightheavy (GB)light (MB)
Start timetens of secondswithin seconds
Isolationstrongrelatively weak
How many per servera few to dozensdozens to hundreds

In practice the two are layered. The cloud splits a physical server into VMs (instances) to rent out, and inside that instance you run many containers again.

How resources are split and limited #

To keep the split environments from fighting over resources, each one’s allocation needs a limit. Linux uses a feature called cgroups to cap CPU, memory, and I/O per process group. Container resource limits sit on top of this.

  • CPU — the vCPU from #2 is split here. The containers inside an instance share its allocated vCPUs within their limits.
  • Memory — as #3 noted, without a memory limit one container can swallow the host’s memory and threaten the others. Cross the limit and that container’s process is terminated.

Overcommit and the noisy neighbor #

Virtualization has a downside: a cloud provider may allocate more total resources across its VMs than the physical hardware actually has, on the assumption that not all VMs use their allocation fully at once. This is overcommit.

Usually there’s no problem, but if another VM on the same physical server suddenly hogs resources, your VM’s performance wobbles. This is the noisy neighbor. It’s one cause of “an instance that was fine yesterday is slow today.”

Workloads where performance must not wobble pick an instance type that monopolizes resources, or use a dedicated option that rents a whole physical server. Such choices connect to the instance-type decision in #9.

Common pitfalls #

“A container is a lightweight VM” #

They look similar but differ. A VM holds a whole OS and is strongly isolated; a container shares the kernel, light but weakly isolated. Workloads where isolation level matters must weigh this difference.

“I didn’t set a limit on the container” #

Without a limit, one container monopolizes host resources and starves the others. In operations it’s safer to state CPU and memory limits.

“Instance performance is uneven and I can’t find why” #

It may be a noisy neighbor in an overcommit environment. If you need steady performance, consider a resource-monopolizing type or a dedicated instance.

“Containers on a VM are wasteful” #

It’s the standard setup. On top of a cloud instance’s strong VM isolation, you also get containers’ light deployment inside it.

Wrap-up #

What we covered:

  • Virtualization gathers spare resources and splits one physical server into several independent environments.
  • A hypervisor splits the hardware, and each VM holds a whole OS of its own. The EC2 instance is that VM.
  • A container shares the host kernel — light, but isolation is weaker than a VM’s. It’s the foundation of Docker and Kubernetes.
  • Resources are capped with cgroups. A container’s memory limit is safer stated explicitly.
  • Overcommit and the noisy neighbor are one reason cloud performance wobbles.

Next — cloud #

Virtualization is what lets the cloud sell resources in pieces. Now we sort out the picture above it, the cloud itself. #8 Cloud — From owning to renting, from on-prem to IaaS / PaaS / SaaS covers the shift of hardware from owning to renting, the difference between on-prem, colocation, and cloud, and how far IaaS, PaaS, and SaaS hide the hardware.

X