#Hardware
33 posts
Sizing Servers for Self-Hosted LLMs — From VRAM Math to Concurrent Users
A repeatable order for estimating the GPU scale a self-hosted LLM needs: computing VRAM from parameter count and precision, the KV cache and its relationship to concurrency, memory bandwidth and token speed, example configurations by scenario, and the break-even against API pricing.
AI Training Servers vs. Inference Servers — Where the Requirements Diverge
Why the same "AI server" question has two very different answers: batch versus request-serving workload shapes, GPU interconnect versus memory bandwidth as the bottleneck, precision and quantization, cost structures, and the trap of trying to share one fleet for both.
How to Read NVIDIA GPU Names — Architecture Generations and Product Lines
Reading H100, B200, GB300, L40S, and RTX 5090 as a system rather than a code: the scientist-named architecture axis and the datacenter/inference/consumer product-line axis, what the digits and prefixes mean, and the generation landscape and roadmap as of mid-2026.
GPU Servers vs. General Servers — Architecture, Power, Cooling, Price
Where a GPU server stops being a regular server with a graphics card in it: the inverted architecture and interconnects, power density an order of magnitude higher and the arrival of liquid cooling, the price structure, the driver stack, and why utilization management becomes the job.
Why Your Server Is Slow #4: Plenty of Bandwidth, Still Slow — Latency, RTT, Retransmissions
Tracking down why the network feels slow while the link sits mostly idle. Covers how RTT times round-trip count sets perceived speed, the handshake cost of TCP and TLS, BDP and window limits, checking packet loss and retransmissions, and connection reuse.
Why Your Server Is Slow #3: Slow Despite an SSD — Write Amplification, fsync, Queue Depth
Tracking down why storage is the bottleneck even on an SSD. Covers the fine print behind spec-sheet IOPS, the cost of durable writes through fsync, SLC cache exhaustion and write amplification, the physical limits of queue depth 1, and the IOPS ceilings of cloud volumes.
Why Your Server Is Slow #2: More Memory Didn't Help — Page Cache, Swap, Working Set
Tracking down why adding memory left performance unchanged. Covers what available really means, how spare memory already works as page cache, judging swap by movement rather than usage, sizing the working set, and the container limits and application heaps that upgrades never reach.
Why Your Server Is Slow #1: Low CPU Usage but Still Slow — Run Queue, I/O Wait, Lock Contention
Tracking down why a server is slow while CPU usage stays low. Covers run queue and scheduling delay, the iowait trap, lock contention and off-CPU waits, and steal time in the cloud — a diagnostic order for finding the waiting that utilization graphs miss.
Hardware Advanced #7: Firmware, BMC, and the Lifecycle — The Other Computer Inside Your Server
A look at the BMC, the management computer that stays on independently of the main CPU. It covers remote console and power control, IPMI and Redfish, the firmware stack and update operations, failure prediction with SMART and ECC counters, management-network security, and the lifecycle from warranty expiry to disk disposal — closing out the Hardware Advanced series.
Hardware Advanced #6: Data Center Cooling and Racks — Electricity Always Becomes Heat
Nearly all the power that enters a server comes back out as heat. Starting from the basic airflow contract of front intake and rear exhaust, this post maps out data center cooling end to end: hot/cold aisle containment, rack density and the limits of air cooling, liquid cooling with D2C and immersion, and how ASHRAE temperature guidelines tie into PUE.
Hardware Advanced #5: Datacenter Power — The Real Reason You Can't Rack More Servers
Even with empty slots in the rack, new servers get rejected — because of the power budget. This post walks the power environment a server lives in, from an operator's point of view: PSU redundancy and A/B feeds, per-rack kW contracts, PDUs and UPS, generators and ATS, PUE, and the power density that GPU servers have driven up.
Hardware Advanced #4: ZFS Deep Dive — When RAID and the Filesystem Become One
ZFS merged RAID, volume management, and the filesystem into a single layer, solving the structural problems of the traditional stack. This post walks through it all from an operations point of view: copy-on-write that eliminates the write hole, checksums that verify every read with self-healing, resilver that copies only live data, RAIDZ and the ARC, snapshots with send/recv, and lz4 compression.