AWS root account cannot access EKS Kubernetes master

I was consulting a team which just started to work on a new project. It was a startup that had switched its software development team completely. The previous team did not leave any docs and information about the project, and all their accounts and access rights were purged. As a result, AWS account owner created brand new admin accounts for the team. The project is running on AWS EKS cluster, and I was asked for help due to the team could not control the Kubernetes master. It sounds strange, right?

Yes, we like K8s — it provides some standard blocks to build on regardless of a cloud vendor. But we should remember that Kubernetes does not cover 100% of the infrastructure. For example, it abstracts from specific persistent storage, network load balancing, master authorization, and so on. Let’s call it the "other 10%" which depends on the infrastructure around our cluster, whether it’s a cloud provider or some on-premise solution. The three main horses, AWS/GCP/Azure, provide their glue to fill the gaps.

Here we talk about Kubernetes administration authorization, i.e. access control to the Kubernetes master. The K8s has its own authz mechanism (RBAC one) and AWS cloud has its well-known IAM. And AWS EKS adds the missing glue to map them, so we, as an AWS user from IAM perspective, may have access to a specific EKS cluster with a specified Kubernetes role.

Well, sounds great and the team wanted to add their AWS admin accounts to this specific aws-auth K8s ConfigMap. But how to do it without administration access to the cluster? Kind of a "chicken and egg" situation.

I quickly found out that the project has some Terraform files. There is no information on whether they are up-to-date or used at all. At a glance, K8s configuration was found within, with a couple of AWS user names mentioned to be masters of a Kubernetes cluster. They were user names of the previous team, and such users do not exist anymore.

As a simple and quick solution, we created a brand new AWS account with one of those names. This account was passed by the Kubernetes RBAC, we used it to re-configure aws-auth config map to allow new team members to operate the cluster. That’s it — team members were smiling at what just happened, and managers of the project were happy that the overall development and delivery process could go ahead.

New technologies strive to help us with additional simplification, standardization, development speed, and other benefits. But when they enter the existing market they have to integrate with existing players and solutions, that’s why we find some glue between Kubernetes and a cloud provider. And while the new tech may work perfectly well, the glue may still spawn some troubles on the go.

IT happens.

Copyright © Igor Ostapenko
(handmade content)