Right-sizing without breaking production

Right-sizing sounds simple: find servers that are bigger than they need to be and make them smaller. Done carelessly, it’s also a reliable way to cause an outage at the worst possible moment, like the first day of a big campaign.

Here’s how I do it so the savings stick and production doesn’t notice.

Measure before you touch anything

Never size from a single day’s graph. Collect at least two to four weeks of metrics so you see a normal week, the busiest day and any monthly jobs.

For each server, look at:

• CPU: average and the busy peaks (the 95th percentile is more useful than the average)
• Memory: how much is actually used, not counting cache the OS will happily give back
• Disk: space used and how fast it grows, plus read/write activity
• Network: traffic at peak times

A server that averages 10% CPU but hits 90% every night during a batch job is not oversized. It’s sized for the batch job.

Know what each server is for

This is much easier when each server has one job. On a shared server running the app, database and cache together, you can’t tell which part needs which resource, so every change is a guess.

When I split a crashing fintech server into separate gateway, app, database and cache machines, each one could be sized for its own work. Only the database needed a larger instance. The rest ran on small ones.

Leave headroom on purpose

Don’t size to the peak you measured. Size so the busiest normal period still leaves room for the busy period you haven’t seen yet. As a rough starting point, I aim for peak CPU to stay comfortably below saturation and memory to stay well clear of the limit, then adjust based on how spiky the workload is.

Watch out for burstable instances

Burstable instance types (like AWS’s T family) look cheap and work well for spiky, mostly-idle workloads. Under steady load they can run out of CPU credits and slow down sharply, which looks like a mystery outage. If a workload is busy most of the time, a fixed-performance instance is often the safer choice even if it costs a little more.

Change one thing at a time

1. Start with non-production. Resize staging first and run your normal tests and load against it.
2. Change one production server at a time, and wait long enough to see a full busy cycle before the next one.
3. Make the change through code, so it’s reviewed, recorded and easy to reverse.
4. Plan the way back before you start. If response times or errors go up, how fast can you return to the previous size?

Be extra careful with databases

Databases are the riskiest place to save money. Memory affects how much data stays cached, and a smaller instance can turn fast queries into slow ones without any errors. Resize databases last, during a quiet window, with a tested backup and a clear rollback plan.

Check the result

After each change, compare response times, error rates and resource usage with the weeks before. A saving that makes the product slower isn’t a saving; it’s a cost moved to your users.

Keep it from drifting back

Right-sizing isn’t a one-time job. Load changes as the product grows. Review sizes every quarter, alert when a server runs hot for days at a time, and keep every size defined in code so you always know what you’re running and why.

Smaller servers are a nice result. Knowing exactly why each server is the size it is, that’s the real one.