Dedicated CPUs
We use AMD 9754 EPYC processors in our hypervisor platform. Each processor has 128 cores available for processing, and with 2 CPUs installed in the server, that makes a total of 256 cores total available! That’s how math works.
The high core count per hypervisor allows us to offer our VMs at really amazing prices for our customers. $10 for 100% of a cpu core is so much better than anything other cloud providers are doing. Other providers would price these VMs in the $40-$50 dollar / month range.
Why don’t we charge more? Well, I believe that the cloud should not be expensive. We charge enough to cover costs and provide a good paycheck to our selves.
Why dedicated CPUs? Why not do shared hosting and charge even less? Well, we are simply not interested in overloading our hypervisors and making more work for our selves. Overloaded CPUs cause issues with administrative tasks, with neighboring VMs, and with the hypervisor it self.
We simple do not believe that it is worth the trade off.
By choosing to not overload our hypervisors, we are making less work that needs to be done, and we can employ fewer people who would be dedicated to solving problems related to overloading. Those extra savings we are passing back to our customers. You’re welcome!
NVMe Drives
We use 7.68 TB NVMe drives in our hypervisor platform. Each hypervisor has slots for 8 drives (12 if we really wanted) and we only populate 6 of them. The drives are used in a ZFS pool in the hypervisor where each drive has a mirror for redundancy. It gives us a total capacity of 23.04 TB of storage.
If each hypervisor had 512 VMs running on it, each core would get 40 GB of storage. That comes to 20 TB of storage for all the VMs on a hypervisor. It leaves 3 TB for other things, and to ensure there is always enough room just incase it is needed.
ZFS is used because it allows for instantaneous snapshots, and allows for backups of those snapshots to be stored offsite in a ZFS cluster. Snapshots are instantaneous and basically free. No reason to charge extra for something that doesn’t cost us anything. You’re welcome!
Why don’t we use Ceph? Or why we don’t have a hyper converged infrastructure? The answer is simple: It’s extra complexity, it adds another layer of problems, and it’s more expensive.
By just having local resources, there is less work for us to do, and we can pass those savings onto our customers. You’re welcome!
2.25 TB of RAM
We provision our hypervisor with 2304 GB of ram. That’s a lot of ram. We have this much so that we can always reserve plenty for administrative tasks, and for extra wiggle room. If we put 512 VMs on a hypervisor, that would allocate 2048 GB of ram, and leave 256 GB available for other things.
Why don’t we allocate all available ram to VMs? Well hopefully you know the answer by now: we could, but it could cause issues if there weren’t resources available to do administrative things. If the hypervisor runs out of RAM and starts swapping, ALL the VMs will break. I worked somewhere before where that happened, lets just say it was bad.
By having spare RAM available at all times, there are less outages, we have to stay up less fixing things, we can work more on things that matter, features and availability, and we can pass those savings onto our customers. You’re welcome!
That’s a big server
When I started college, I spent extra money to buy 4GB of ram for my 2 core laptop. That was an amazing machine for the time: I was able to run a VM for my classes with mysql on it and not worry about contention. It’s amazing how computer performance has grown, not up, but out. I am looking forward to when we have 2048 cores in a single 1U Hypervisor, and able to sell a 4 core 16 GB VM for $10/month!