Defining Infrastructure Through Kubernetes

Jeff Smith,Senior director, production operations, Centro

Jeff Smith,Senior director, production operations, Centro

As an industry, we continue to see the impact of transformative technologies. Virtualization, configuration management, cloud computing and now Kubernetes, have all transformed the way we run and manage our applications and products. But with each of these transformations comes a bit of a shift in how our teams operate. Some companies see these forces at work and pivot the technical organizations to better embrace the incoming technology change. Others keep their organizations exactly as-is and struggle to reap the full-benefits and promises of an emerging technology. Kubernetes is no different, as it has the ability to destabilize your environment, skillfully hidden inside the cloak of velocity and faster delivery.

Kubernetes continues the trend of "shifting-left", placing more work earlier in the process of software delivery. In the past you may have had a checkpoint where software has developed to the point where an engineer now feels it's ready to be placed on hardware that isn't their laptop. But now they must trigger some internal engagement workflow to get the infrastructure team to provision hardware, virtual machines or whatever your execution context of choice is. With Kubernetes, that same engineer is able to define her infrastructure during the development process using YAML files, testing that same infrastructure on their local laptop. The abstractions provided by Kubernetes helps to take what was once a complicated and nuanced configuration of servers and boil it down to a handful of lines of metadata. (Or at least that's the promise)

This shifting of responsibility does in fact help speed up the delivery pipeline. Software engineers are now more in-tune with the infrastructure that runs their applications. More importantly, engineers now have this sense of empowerment (and responsibility) as it relates to the total management and delivery of their application. The proverbial wall is gone. But before we cheer the tearing down of walls and silos, we should carefully examine why those walls exist and what tearing them down means to your organization.

Walls become gates

Whenever you have a wall between teams that are dependent on one another, you end up with a couple of unintended consequences. Beyond the communication and responsibility gaps, you also have the issue of unrecognized gating.

A gate is a step in a process in which one person or group has to approve the work of another group to continue their work. I will cleverly name the approving group "the approvers" and the requesting group "the requestors".

In an infrastructure context, the approvers group gains its power by access controls. The operations team (the approvers) has the access necessary to create and provision infrastructure that the software engineering team (the requestors) need in order to launch their shiny new application. This discrepancy in access gives the operations group a bit of control over what goes into production. It also assures that anything that is going to production will most likely need to be approved by the operations team, a sort of side-effect of the infrastructure request process. Despite most companies having a process for engaging operations teams earlier in the process, it's not an uncommon occurrence to have the operations engagement missed right up until servers need to be requisitioned.

Enter Kubernetes, where developers are defining infrastructure themselves. If you're trying to embrace the advantages of this technology, then engineers will have a level of access that might be new in your company. Creating new infrastructure and updating existing infrastructure become similar actions in a Kubernetes context, meaning that engineers can launch new services without the involvement of infrastructure teams. For companies where the infrastructure teams are the bottleneck, this can be a huge win. But now teams have to look at the unofficial gates that access controls were providing and ensure that there is a system in place for replacing those necessary gates.

The abstractions provided by Kubernetes helps to take what was once a complicated and nuanced configuration of servers and boil it down to a handful of lines of metadata.

As an example, say your organization has only a handful of approved message bus technologies. Historically the request to move to production triggers a review of the architecture. If the software engineering team is requesting something that isn't approved, there's a clear, clean-cut step on the road to production that affords the infrastructure team the ability to veto that decision. In a Kubernetes world, where a developer might have the ability to launch a service without the involvement of operations, an engineer can choose a different technology, not on the approved list, with a simple one line change to the image being deployed. Now your environment is suddenly running infrastructure that hasn't been approved.

Another example might be the high availability setup. Unbeknown to the software engineer, they may have created an infrastructure configuration that isolates all their infrastructure to a single failure domain. If you're running in a cloud provider like AWS, they offer options to ensure that your service executes in multiple availability zones, which are separate, distinct data centers that share very little infrastructure. The failure of one AZ doesn't impact another. This is important for workload distribution to ensure that your application doesn't run into availability issues in the event an AZ goes down. Do your software engineers have the expertise to make sure their Kubernetes environments aren't tied down to a single availability zone? Who is responsible for that deliverable? Where is the checkpoint for Infrastructure teams to verify that architecture?

Nothing I've outlined here is an intractable problem. But it's important to recognize the impact of new technologies and how they can create (and destroy) previous assumptions about the path to production. What key processes or responsibilities rely on your current path to production? Do software engineers understand their new set of responsibilities now that they can launch services on their own? And as a leader, are you onboard with these shifts? There are no right answers but make sure you're looking at these choices with eyes wide open.