Book of Cloud Native Services

Nutanix Kubernetes Engine (formerly Nutanix Karbon)

» Download this section as PDF (opens in a new tab/window)

Nutanix Kubernetes Engine (NKE) is Nutanix’s certified enterprise Kubernetes management solution that enables turnkey provisioning, operations, and lifecycle management of Kubernetes.

Supported Configurations

The solution is applicable to the configurations below:

Core Use Case(s):

Management interfaces(s):

Supported Environment(s):

Supported node OS image(s):

Upgrades:

Compatible Features:

Nutanix Kubernetes Engine can be enabled via Prism Central. Any Nutanix AOS cluster registered with an NKE-enabled PC can be used as a target for provisioning Kubernetes clusters.

Architecture

An NKE-enabled Kubernetes cluster cannot span over multiple Nutanix HCI clusters

NKE Multi-cluster Architecture

NKE Architecture

NKE runs as a containerized service in Prism Central. When NKE is enabled on a PC, two containers are provisioned under the covers: the karbon-core container and the karbon-ui container.

Air-gapped environments

NKE can be enabled in air-gapped environments too (read more)

Kubernetes Cluster Configurations

OS Images

NKE provides the OS image for installing and scaling Kubernetes nodes. NKE uses the CentOS Linux-based operating system for NKE-enabled Kubernetes cluster creation. New OS image versions are periodically released including patches to fix vulnerabilities. For a list of supported OS image versions, check the NKE release notes (read more)

Operating System Images

Bringing your own OS image is not supported.

Compute

The recommended configurations include two options: development cluster and production cluster.

Networking

There are a total of three networks required by a Kubernetes cluster which can be grouped into virtual machines network and Kubernetes networks.

Pro tip

You can leave the service CIDR and pod CIDR ranges as default, but the ranges must not overlap with each other or with an existing network in your data center if a pod in the cluster will require access to that external network.

Networking

A production cluster with active-active control plane mode requires an external load balancer.

Storage

When deploying a Kubernetes cluster, the Nutanix container storage interface (CSI) driver is also deployed along with it.

A default StorageClass is created as well during the deployment, which uses Nutanix Volumes. This is required by the included add-ons such as Prometheus for monitoring, and EFK (Elasticsearch, Fluent Bit, and Kibana) logging stack, to store metrics and logs. After deployment, more storage classes can be added using the same CSI driver (see examples).

Apart from Nutanix Volumes, you can also create a StorageClass for file storage using Nutanix Files. Depending on what storage backend is configured in a StorageClass, different access modes are supported when creating a PersistentVolumeClaim.

Access modes supported by CSI driver and storage backend.
Storage backend ReadWriteOnce
RWO
ReadOnlyMany
ROX
ReadWriteMany
RWX
ReadWriteOncePod
RWOP
Volumes
-
-
Files
-
Security
Access and Authentication

There are two components to keep in mind when it comes to access and authentication: NKE in PC, and an NKE-enabled Kubernetes cluster.

Note that the kubeconfig generated by NKE is valid for 24-hours, after which the user will have to request a new kubeconfig file. This can be done using the NKE GUI, CLI, API, or the kubectl plug-in available here (recommended).

Nodes

The SSH access to the Kubernetes nodes is locked down using an ephemeral certificate - available in the NKE console, which expires after 24-hours. Installing software or changing settings in the node OS is unsupported, changes are not persistent during upgrades or when scaling out a node pool. The only reason for accessing the nodes via SSH is for troubleshooting at the discretion of Nutanix support.

CIS Benchmark for Kubernetes

Nutanix has evaluated NKE-enabled Kubernetes cluster against the CIS Kubernetes Benchmark-1.6. You can verify compliance through Kube Bench, an automated open-source tool available on GitHub. The report is available here.

Add-ons

NKE add-ons are open source software extensions that provide additional features to your deployment. The add-ons are automatically installed when you deploy a Kubernetes cluster.

Nutanix Kubernetes Engine includes the following add-ons:

These add-ons are for cluster internal use only. Their configuration is not designed for supporting the data generated by the applications running on the Kubernetes cluster. For collecting logs and metrics for the containerized applications, deploy dedicated instances of EFK and Prometheus, or re-use existing ones available in your environment.

Logging

The logging stack aggregates all the operating system and infrastructure logs from the Kubernetes nodes. The Kibana dashboard is accessible via the NKE console.

Since NKE 2.4, the logging stack can be disabled (more details) and just use Fluent Bit for log forwarding against an external existing logging stack (more details)

Monitoring

The Kubernetes clusters have the Prometheus operator installed and one instance of it deployed for collecting infrastructure metrics. Additional Prometheus instances can be deployed using the operator, for example, for application monitoring (blog).

Since NKE 2.4, SMTP-based alert forwarding to an e-mail address can be enabled (more details).

Lifecycle management

There are two different types of NKE upgrades:

NKE upgrade via LCM

To check the current version of Karbon or to upgrade to later versions, perform the inventory check in Prism Central using LCM. LCM upgrades the following NKE components:

Note
NKE Upgrades

Be aware when upgrading to a latest version of NKE, that all the Kubernetes clusters must be running or upgraded first to a supported version by the target NKE. Check the Nutanix portal for updated supported versions.

Kubernetes cluster upgrades

There are two aspects when it comes to upgrading a Kubernetes cluster:

Note
Kubernetes Cluster Upgrades

Be aware that node OS or Kubernetes version upgrades can be disruptive depending on your Kubernetes cluster type, development vs. production.

Node OS upgrade

When a node OS image upgrade is available, NKE displays an option to download the new image in the OS Images tab. NKE also displays an Upgrade Available icon next to the cluster in the Clusters view.

Kubernetes + add-ons version upgrade

Clusters that have a Kubernetes version eligible for an upgrade display the Upgrade Available icon in the table. As a part of the upgrade process, it will upgrade the Kubernetes version as well as any upgrade available for the installed add-ons.

NKE CLI and API

NKE CLI

The NKE CLI, karbonctl, gives users the ability to execute lifecycle management tasks for NKE and Kubernetes clusters. Certain advanced tasks can be done using karbonctl only.

To use karbonctl you have to SSH into a Prism Central instance. The path for the binary is /home/nutanix/karbon/karbonctl

Some common tasks you can run with karbonctl are:

karbonctl

NKE API

The NKE API lets users programmatically run management task for NKE and Kubernetes clusters. The API documentation is available at https://www.nutanix.dev/reference/karbon.