Lessons From The Inside Of The Cloud

The Impressive Infrastructure of the Modern Data Center

‍Last week our team took a group of non-engineering team members (like me!) to our data center in order to look under the hood of “the cloud.” Due to the ethereal nature of its nomenclature, “the cloud” conjures an image of data somehow floating out there in space ready to be accessed at any time. However, the fact of the matter is cloud computing requires an enormous amount of physical space along with complex engineering fail-safes and numerous redundancies. Data is vital to every business, and how data is stored, accessed, and secured is not only important but also fascinating. We got to see our own “Velo Cloud” server cluster that we offer to clients, but the trip also provided valuable lessons to consider when building on-premises IT infrastructure.

“Having never been to a data center, I did not know what to expect. Yet my expectations were exceeded! The architectural design and extensive attention to every detail of the environment to properly house the technology and equipment in the center is beyond impressive.”

- Julia Towry, Bookkeeper

‍Security

The first thing you realize when walking into the data center is that you are not yet “in” the data center. The lobby serves as the first layer of defense. Heat-sensitive fingerprint ID and badge scanners control access through a set of thick steel bars and each set of doors beyond requires the same. This prioritization of security is the first lesson. The data and technology systems housed within the concrete-encased 50,000 sq. ft. building are extremely valuable, and for many companies, it is critical that access is guaranteed 24/7/365. Credit card information, phone systems, software, legal documents… the list goes on. If your business has any important data or technologies, the physical security of your data should not be overlooked!

“I think the data center was impressive for a wide variety of reasons. The security, the precision, and execution of the team. The architecture and how everything was crafted with a specific reason in mind from the control of the airflow, the man traps, loading docks, and the raised floors where the server racks stood. The amount of redundancy that went into making sure that there was 0 downtime, from electrical backups to generator backups with the ability to run for almost 3 days without external electricity.”

– Ian DeRizzio, Hardware Engineer          

After checking in, we were taken to a bird’s-eye view of the facility (after two more security points.) It was here that the scope and magnitude of the data center came fully into view. Rows and rows of servers all meticulously climate-controlled and monitored. This brings us to the second lesson: there’s physical security and then there’s physical security. One is obviously pretty focused on access control, but the other less obvious precaution helps the data center avoid physical damage due to heat, sparks, fires, dust, power outages, fire sprinkler damage, etc. – you name it, they have a plan.

The datacenter has taken exhaustive steps to protect the servers held within from all manner of physical threats beyond intruders. Tornados, floods, ice storms (like the one that just hit Texas), and fire all pose a risk. Even power loss can wreck a system if not properly prepared for. The datacenter had an answer for each of these risks: double concrete walls, raised floors (which also serve cooling purposes), advanced fire extinguishing solutions that keeps fire sprinkler pipes dry until needed, and massive diesel generators (20-cylinder train engines) with 10,000 gallons of reserve fuel to keep the facility powered for several days if needed. It would of course be cost-prohibitive to design your own server with this level of security, but it presents important questions.

Food for thought as you consider the physical security and precautions you take in protecting your server room or network closet – Would power loss cripple your business? If you lose data can you recover it? Can you access your data whenever you need it? Could an intruder walk right into your IT closet unchecked if they wanted to?

Something as simple as a lock on your server room door can serve as an important security step.

Redundancy

“I was impressed with the amount of thought and detail put into disaster prevention, such as the 3 massive generators being stored indoors to allow them to power up without any warm-up time needed.”

– Sean Mair, Escalation Engineer        

From this bird’s eye view, we also learned the “N+1” philosophy that informs every decision they make. “N” is the variable representing how much they need of a certain technology or system to function (don’t worry this is simple algebra!) and the “+1” means they always have one more of than what they need. Certain systems (like power/electricity) are 2n+1 meaning they have double what they need plus backup options. This particular facility has generators that can handle over triple the electricity demand they currently have.

Therein lies the third lesson: redundancy is key. A phrase we live by here at Velo is “two is one and one is none” which echoes the same philosophy. One firewall? One hard drive? One power supply? In the event of equipment failure, this can quickly cripple or completely take a system offline. In designing an organization’s IT system, it is important to factor in redundancy to ensure all critical systems can remain functional even if one piece of equipment fails.

After this overview of the facility, we moved down into the heart of the datacenter and got to see the infrastructure that keeps the facility up and running. There is a large power room that can offer up to 40 megawatts – enough to power up to 10,000 homes. From there we briefly viewed the large diesel generators, as well as the meet-me room which pulls from several different internet service providers from north, south, east, and west to greatly decrease the likelihood of an outage.

24/7 Uptime Requires a Lot of Engineering

“One of the things that impressed me the most while touring the data center was the vast & in-depth knowledge of their team members. Each person we talked to could answer any question about the data center instantly and with exact statistics. This really demonstrated how passionate their team is about protecting their customers’ equipment.”

– Lindsey Ray, Sales Engineer          

After taking in some of the data center’s incredible infrastructure, I encountered a fourth realization and a critical item for consideration for those businesses using on-premises IT infrastructure. Behind the massive capital expenditure and feats of engineering that are necessary to run a successful data center, there is a simple mission: to keep technology and data and running 24/7/365. At any level, it is important to engineer an appropriate level of physical security, redundancy, and precaution when investing in IT infrastructure.

“The Cloud”

The final stop of this trip was our own server rack housed in the facility. Through a labyrinth of server cages and security measures we witnessed the power of modern computing and were all impressed at how much data and compute power was stored in such a relatively small space.

In a single server rack, Velo maintains hundreds of virtual machines (an effective way of partitioning processing power for different functions or software applications). In addition, there are hundreds of terabytes of data stored in the rack – an amount of data that would have taken up twice the space only 5 years ago. In its most simple form, this is what “the cloud” really looks like. Where you use AWS, Google, or a managed services provider who provides you a virtual private cloud, these cloud-driven data solutions show off a wealth of engineering genius in the pursuit of 24/7 up-time. If you ever get the opportunity to tour a data center yourself, I highly recommend it. Getting to take a look behind the curtain to look at the physicality of where that data lives made for a fascinating field trip “inside the cloud.”