OpenShift 4.19


Chapter 1. Architecture overview
Chapter 2. OpenShift Container Platform architecture
Chapter 3. Installation and update
Chapter 4. Red Hat OpenShift Cluster Manager
Chapter 5. About the multicluster engine for Kubernetes Operator
Chapter 6. Control plane architecture
Chapter 7. Understanding OpenShift Container Platform development
Chapter 8. Red Hat Enterprise Linux CoreOS (RHCOS)
Chapter 9. Admission plugins


Chapter 1. Architecture overview

OpenShift Container Platform is a cloud-based Kubernetes container platform. The foundation of OpenShift Container Platform is based on Kubernetes and therefore shares the same technology. To learn more about OpenShift Container Platform and Kubernetes, see Chapter 2. OpenShift Container Platform architecture product architecture.


1.1. Glossary of common terms for OpenShift Container Platform architecture

access policies A set of roles that dictate how users, applications, and entities within a cluster interact with one another. An access policy increases cluster security.
admission plugins Admission plugins enforce security policies, resource limitations, or configuration requirements.
authentication To control access to an OpenShift Container Platform cluster, a cluster administrator can configure user authentication to ensure only approved users access the cluster. We authenticate by providing an OAuth access token or an X.509 client certificate in requests to the OpenShift Container Platform API.
bootstrap A temporary machine that runs minimal Kubernetes and deploys the OpenShift Container Platform control plane.
certificate signing requests (CSRs) A resource requests a denoted signer to sign a certificate. This request might get approved or denied.
Cluster Version Operator (CVO) An Operator that checks with the OpenShift Container Platform Update Service to see the valid updates and update paths based on current component versions and information in the graph.
compute nodes Nodes responsible for executing workloads for cluster users. Compute nodes are also known as worker nodes.
configuration drift A situation where the configuration on a node does not match what the machine config specifies.
containers Lightweight and executable images that consist of software and all of its dependencies. Because containers virtualize the operating system, we can run containers anywhere, such as data centers, public or private clouds, and local hosts.
container orchestration engine Software that automates the deployment, management, scaling, and networking of containers.
container workloads Applications packaged and deployed in containers.
control groups (cgroups) Partitions sets of processes into groups to manage and limit the resources processes consume.
control plane A container orchestration layer that exposes the API and interfaces to define, deploy, and manage the life cycle of containers. Control planes are also known as control plane machines.
CRI-O A Kubernetes native container runtime implementation that integrates with the operating system to deliver an efficient Kubernetes experience.
deployment A Kubernetes resource object that maintains the life cycle of an application.
Dockerfile A text file that contains the user commands to perform on a terminal to assemble the image.
hosted control planes A OpenShift Container Platform feature that enables hosting a control plane on the OpenShift Container Platform cluster from its data plane and workers. This model performs the following actions:

  • Optimize infrastructure costs required for the control planes.
  • Improve the cluster creation time.
  • Enable hosting the control plane using the Kubernetes native high level primitives. For example, deployments and stateful sets.
  • Allow a strong network segmentation between the control plane and workloads.
hybrid cloud deployments Deployments that deliver a consistent platform across bare metal, virtual, private, and public cloud environments. This offers speed, agility, and portability.
Ignition A utility that RHCOS uses to manipulate disks during initial configuration. It completes common disk tasks, including partitioning disks, formatting partitions, writing files, and configuring users.
installer-provisioned infrastructure The installation program deploys and configures the infrastructure that the cluster runs on. Must have direct internet access to pull images.
kubelet A primary node agent that runs on each node in the cluster to ensure that containers are running in a pod.
kubernetes manifest Specifications of a Kubernetes API object in a JSON or YAML format. A configuration file can include deployments, config maps, secrets, daemon sets.
Machine Config Daemon (MCD) A daemon that regularly checks the nodes for configuration drift.
Machine Config Operator (MCO) An Operator that applies the new configuration to the cluster machines.
machine config pools (MCP) A group of machines, such as control plane components or user workloads, based on the resources that they handle.
metadata Additional information about cluster deployment artifacts.
microservices An approach to writing software. Applications can be separated into the smallest components, independent from each other by using microservices.
mirror registry A registry that holds the mirror of OpenShift Container Platform images.
monolithic applications Applications that are self-contained, built, and packaged as a single piece.
namespaces A namespace isolates specific system resources that are visible to all processes. Inside a namespace, only processes that are members of that namespace can see those resources.
networking Network information of OpenShift Container Platform cluster.
node A worker machine in the OpenShift Container Platform cluster. A node is either a virtual machine (VM) or a physical machine.
OpenShift CLI (oc) A command-line tool to run OpenShift Container Platform commands on the terminal.
OpenShift Dedicated A managed RHEL OpenShift Container Platform offering on Amazon Web Services (AWS) and Google Cloud Platform (GCP). OpenShift Dedicated focuses on building and scaling applications.
OpenShift Update Service (OSUS) For clusters with internet access, Red Hat Enterprise Linux (RHEL) provides over-the-air updates by using an OpenShift update service as a hosted service located behind public APIs.
OpenShift image registry A registry provided by OpenShift Container Platform to manage images.
Operator The preferred method of packaging, deploying, and managing a Kubernetes application in an OpenShift Container Platform cluster. An Operator takes human operational knowledge and encodes it into software is packaged and shared with customers.
OperatorHub A platform that contains various OpenShift Container Platform Operators to install.
Operator Lifecycle Manager (OLM) OLM helps us to install, update, and manage the lifecycle of Kubernetes native applications. OLM is an open source toolkit designed to manage Operators in an effective, automated, and scalable way.
OSTree An upgrade system for Linux-based operating systems that performs atomic upgrades of complete file system trees. OSTree tracks meaningful changes to the file system tree using an addressable object store, and is designed to complement existing package management systems.
over-the-air (OTA) updates The OpenShift Container Platform Update Service (OSUS) provides over-the-air updates to OpenShift Container Platform, including RHCOS.
pod One or more containers with shared resources, such as volume and IP addresses, running in our OpenShift Container Platform cluster. A pod is the smallest compute unit defined, deployed, and managed.
private registry OpenShift Container Platform can use any server implementing the container image registry API as a source of the image which allows the developers to push and pull their private container images.
public registry OpenShift Container Platform can use any server implementing the container image registry API as a source of the image which allows the developers to push and pull their public container images.
RHEL OpenShift Container Platform Cluster Manager A managed service where we can install, modify, operate, and upgrade our OpenShift Container Platform clusters.
RHEL Quay Container Registry A Quay.io container registry that serves most of the container images and Operators to OpenShift Container Platform clusters.
replication controllers An asset that indicates how many pod replicas are required to run at a time.
role-based access control (RBAC) A key security control to ensure that cluster users and workloads have only access to resources required to execute their roles.
route Routes expose a service to allow for network access to pods from users and applications outside the OpenShift Container Platform instance.
scaling The increasing or decreasing of resource capacity.
service A service exposes a running application on a set of pods.
Source-to-Image (S2I) image An image created based on the programming language of the application source code in OpenShift Container Platform to deploy applications.
storage OpenShift Container Platform supports many types of storage, both for on-premise and cloud providers. We can manage container storage for persistent and non-persistent data in an OpenShift Container Platform cluster.
Telemetry A component to collect information such as size, health, and status of OpenShift Container Platform.
template A template describes a set of objects that can be parameterized and processed to produce a list of objects for creation by OpenShift Container Platform.
user-provisioned infrastructure We can install OpenShift Container Platform on the infrastructure that we provide. We can use the installation program to generate the assets required to provision the cluster infrastructure, create the cluster infrastructure, and then deploy the cluster to the infrastructure that we provided.
web console A user interface (UI) to manage OpenShift Container Platform.
worker node Nodes that are responsible for executing workloads for cluster users. Worker nodes are also known as compute nodes.

Additional resources


1.2. About installation and updates

We can install and deploy a cluster by using one of the following methods:


1.3. About the control plane

The control plane manages the worker nodes and the pods in the cluster. We can configure nodes with the use of machine config pools (MCPs). MCPs are groups of machines, such as control plane components or user workloads, that are based on the resources that they handle. OpenShift Container Platform assigns different roles to hosts. These roles define the function of a machine in a cluster. The cluster contains definitions for the standard control plane and worker role types.

We can use Operators to package, deploy, and manage services on the control plane. Operators are important components in OpenShift Container Platform because they provide the following services:

Additional resources


1.4. About containerized applications for developers

As a developer, we can use different tools, methods, and formats to develop containerized application:

We can also create a Kubernetes manifest and store it in a Git repository. Kubernetes works on basic units called pods. A pod is a single instance of a running process in the cluster. Pods can contain one or more containers. We can create a service by grouping a set of pods and their access policies. Services provide permanent internal IP addresses and host names for other applications to use as pods are created and destroyed. Kubernetes defines workloads based on the type of application.


1.5. About Red Hat Enterprise Linux CoreOS (RHCOS) and Ignition

As a cluster administrator, we can perform the following Red Hat Enterprise Linux CoreOS (RHCOS) tasks:

The OpenShift Container Platform installation program creates the Ignition configuration files that we need to deploy the cluster. Red Hat Enterprise Linux CoreOS (RHCOS) uses Ignition during the initial configuration to perform common disk tasks, such as partitioning, formatting, writing files, and configuring users. During the first boot, Ignition reads its configuration from the installation media or the location that we specify and applies the configuration to the machines.

We can learn how Chapter 8. Red Hat Enterprise Linux CoreOS (RHCOS) Ignition works, the process for a Red Hat Enterprise Linux CoreOS (RHCOS) machine in an OpenShift Container Platform cluster, view Ignition configuration files, and change Ignition configuration after an installation.


1.6. About admission plugins

We can use admission plugins to regulate how OpenShift Container Platform functions. After a resource request is authenticated and authorized, admission plugins intercept the resource request to the master API to validate resource requests and to ensure that scaling policies are adhered to. Admission plugins are used to enforce security policies, resource limitations, configuration requirements, and other settings.


1.7. About Linux cgroup version 2

OpenShift Container Platform uses Linux control group version 2 (cgroup v2) in the cluster.

cgroup v2 offers several improvements over cgroup v1, including a unified hierarchy, safer sub-tree delegation, features such as Pressure Stall Information, and enhanced resource management and isolation. However, cgroup v2 has different CPU, memory, and I/O management characteristics than cgroup v1. Therefore, some workloads might experience slight differences in memory or CPU usage on clusters that run cgroup v2.


Chapter 2. OpenShift Container Platform architecture


2.1. Introduction to OpenShift Container Platform

OpenShift Container Platform is a platform for developing and running containerized applications. It is designed to allow applications and the data centers that support them to expand from just a few machines and applications to thousands of machines that serve millions of clients.

With its foundation in Kubernetes, OpenShift Container Platform incorporates the same technology that serves as the engine for massive telecommunications, streaming video, gaming, banking, and other applications. We can extend containerized applications beyond a single cloud to on-premise and multi-cloud environments.

Red Hat OpenShift Kubernetes Engine View larger image


2.1.1. About Kubernetes

Although container images and the containers that run from them are the primary building blocks for modern application development, to run them at scale requires a reliable and flexible distribution system. Kubernetes is the defacto standard for orchestrating containers.

Kubernetes is an open source container orchestration engine for automating deployment, scaling, and management of containerized applications.

Kubernetes has seen massive cloud and on-premise adoption. The open source development model allows many people to extend Kubernetes by implementing different technologies for components such as networking, storage, and authentication.


2.1.2. The benefits of containerized applications

Using containerized applications offers many advantages over using traditional deployment methods. Where applications were once expected to be installed on operating systems that included all their dependencies, containers let an application carry their dependencies with them. Creating containerized applications offers many benefits.


2.1.2.1. Operating system benefits

Containers use small, dedicated Linux operating systems without a kernel. Their file system, networking, cgroups, process tables, and namespaces are separate from the host Linux system, but the containers can integrate with the hosts seamlessly when necessary. Being based on Linux allows containers to use all the advantages that come with the open source development model of rapid innovation.

Because each container uses a dedicated operating system, we can deploy applications that require conflicting software dependencies on the same host. Each container carries its own dependent software and manages its own interfaces, such as networking and file systems, so applications never need to compete for those assets.


2.1.2.2. Deployment and scaling benefits

If we employ rolling upgrades between major releases of applications, we can continuously improve applications without downtime and still maintain compatibility with the current release.

We can also deploy and test a new version of an application alongside the existing version. If the container passes our tests, simply deploy more new containers and remove the old ones.

Since all the software dependencies for an application are resolved within the container itself, we can use a standardized operating system on each host in our data center. We do not need to configure a specific operating system for each application host. When our data center needs more capacity, we can deploy another generic host system.

OpenShift Container Platform offers a standard way of scaling any containerized service. For example, if we build applications as a set of microservices rather than large, monolithic applications, we can scale the individual microservices individually to meet demand. This capability allows we to scale only the required services instead of the entire application, which can allow we to meet application demands while using minimal resources.


2.1.3. OpenShift Container Platform overview

OpenShift Container Platform provides enterprise-ready enhancements to Kubernetes, including the following enhancements:

Although Kubernetes excels at managing applicationss, it does not specify or manage platform-level requirements or deployment processes. Powerful and flexible platform management tools and processes are important benefits that OpenShift Container Platform 4.19 offers.


2.1.3.1. Custom operating system

OpenShift Container Platform uses Red Hat Enterprise Linux CoreOS (RHCOS), a container-oriented operating system specifically designed for running containerized applications from OpenShift Container Platform and works with new tools to provide fast installation, Operator-based management, and simplified upgrades.

RHCOS includes:

In OpenShift Container Platform 4.19, we must use RHCOS for all control plane machines, but we can use Red Hat Enterprise Linux (RHEL) as the operating system for compute machines, which are also known as worker machines. If we choose to use RHEL workers, we must perform more system maintenance than if we use RHCOS for all of the cluster machines.


2.1.3.2. Installation and update process

With OpenShift Container Platform 4.19, if we have an account with the right permissions, we can deploy a production cluster in supported clouds by running a single command and providing a few values. We can also customize our cloud installation or install the cluster in our data center if we use a supported platform.

For clusters that use RHCOS for all machines, updating, or upgrading, OpenShift Container Platform is a highly-automated process. Because OpenShift Container Platform completely controls the systems and services that run on each machine, including the operating system itself, from a central control plane, upgrades are designed to become automatic events. If the cluster contains RHEL worker machines, the control plane benefits from the streamlined update process, but we must perform more tasks to upgrade the RHEL machines.


2.1.3.3. Other key features

Operators are both the fundamental unit of the OpenShift Container Platform 4.19 code base and a convenient way to deploy applications and software components for applications to use. In OpenShift Container Platform, Operators serve as the platform foundation and remove the need for manual upgrades of operating systems and control plane applications. OpenShift Container Platform Operators such as the Cluster Version Operator and Machine Config Operator allow simplified, cluster-wide management of those critical components.

Operator Lifecycle Manager (OLM) and the OperatorHub provide facilities for storing and distributing Operators to people developing and deploying applications.

The Red Hat Quay Container Registry is a Quay.io container registry that serves most of the container images and Operators to OpenShift Container Platform clusters. Quay.io is a public registry version of Red Hat Quay that stores millions of images and tags.

Other enhancements to Kubernetes in OpenShift Container Platform include improvements in software defined networking (SDN), authentication, log aggregation, monitoring, and routing. OpenShift Container Platform also offers a comprehensive web console and the custom OpenShift CLI (oc) interface.


2.1.3.4. OpenShift Container Platform lifecycle

Figure 2.1. High level OpenShift Container Platform overview

High-level OpenShift Container Platform flow View larger image


2.1.4. Internet access for OpenShift Container Platform

In OpenShift Container Platform 4.19, we require access to the internet to install the cluster.

We must have internet access to perform the following actions:

Important

If the cluster cannot have direct internet access, we can perform a restricted network installation on some types of infrastructure that we provision. During that process, we download the required content and use it to populate a mirror registry with the installation packages. With some installation types, the environment that we install the cluster in will not require internet access. Before we update the cluster, we update the content of the mirror registry.


Chapter 3. Installation and update


3.1. About OpenShift Container Platform installation

The OpenShift Container Platform installation program offers four methods for deploying a cluster...

Each method deploys a cluster with the following characteristics:


3.1.1. About the installation program

We can use the installation program to deploy each type of cluster. The installation program generates the main assets, such as Ignition config files for the bootstrap, control plane, and compute machines. We can start an OpenShift Container Platform cluster with these three machine configurations, provided we correctly configured the infrastructure.

The OpenShift Container Platform installation program uses a set of targets and dependencies to manage cluster installations. The installation program has a set of targets that it must achieve, and each target has a set of dependencies. Because each target is only concerned with its own dependencies, the installation program can act to achieve multiple targets in parallel with the ultimate target being a running cluster. The installation program recognizes and uses existing components instead of running commands to create them again because the program meets the dependencies.

Figure 3.1. OpenShift Container Platform installation targets and dependencies

OpenShift Container Platform installation targets and dependencies View larger image


3.1.2. About Red Hat Enterprise Linux CoreOS (RHCOS)

Post-installation, each cluster machine uses Red Hat Enterprise Linux CoreOS (RHCOS) as the operating system. RHCOS is the immutable container host version of Red Hat Enterprise Linux (RHEL) and features a RHEL kernel with SELinux enabled by default. RHCOS includes the kubelet, which is the Kubernetes node agent, and the CRI-O container runtime, which is optimized for Kubernetes.

Every control plane machine in an OpenShift Container Platform 4.19 cluster must use RHCOS, which includes a critical first-boot provisioning tool called Ignition. This tool enables the cluster to configure the machines. Operating system updates are delivered as a bootable container image, using OSTree as a backend, deployed across the cluster by the Machine Config Operator. Actual operating system changes are made in-place on each machine as an atomic operation by using rpm-ostree. Together, these technologies enable OpenShift Container Platform to manage the operating system like it manages any other application on the cluster, by in-place upgrades that keep the entire platform up to date. These in-place updates can reduce the burden on operations teams.

If we use RHCOS as the operating system for all cluster machines, the cluster manages all aspects of its components and machines, including the operating system. Because of this, only the installation program and the Machine Config Operator can change machines. The installation program uses Ignition config files to set the exact state of each machine, and the Machine Config Operator completes more changes to the machines, such as the application of new certificates or keys, after installation.


3.1.3. Supported platforms for OpenShift Container Platform clusters

Platform Platform>Installer-provisioned infrastructure [1] Platform>User-provisioned infrastructure [2] Platform>Agent-based Installer Platform>Assisted Installer
Amazon Web Services (AWS) X X
Bare metal X X X X
External X X
Google Cloud Platform (GCP) X X
IBM Cloud Classic X
IBM Cloud Virtual Private Cloud (VPC) X
IBM Power X X X
IBM Z or IBM LinuxONE X X X
Microsoft Azure X X
Microsoft Azure Stack Hub X X
None X X
Nutanix X X
Oracle Cloud Infrastructure (OCI) X X
Red Hat OpenStack Platform (RHOSP) [3] X X
VMware vSphere X X X X

  1. For installer-provisioned infrastructure: All machines, including the computer that we run the installation process on, must have direct internet access to pull images for platform containers and provide telemetry data to Red Hat.

    After installation, the following changes are not supported:

    • Mixing cloud provider platforms.
    • Mixing cloud provider components. For example, using a persistent storage framework from a another platform on the platform where we installed the cluster.

  2. For user-provisioned infrastructure: Depending on the supported cases for the platform, we can perform installations on user-provisioned infrastructure so that we can run machines with full internet access, place the cluster behind a proxy, or perform a disconnected installation.

    In a disconnected installation, we can download the images that are required to install a cluster, place them in a mirror registry, and use that data to install the cluster. While we require internet access to pull images for platform containers, with a disconnected installation on vSphere or bare-metal infrastructure, the cluster machines do not require direct internet access.

  3. For Red Hat OpenStack Platform (RHOSP): The latest OpenShift Container Platform release supports both the latest RHOSP long-life release and intermediate release. For complete RHOSP release compatibility, see the OpenShift Container Platform on RHOSP support matrix.

The OpenShift Container Platform 4.x Tested Integrations page contains details about integration testing for different platforms.


3.1.4. Installation process

Except for the Assisted Installer, when we install an OpenShift Container Platform cluster, we must download the installation program from the appropriate Cluster Type page on the OpenShift Cluster Manager Hybrid Cloud Console. This console manages:

In OpenShift Container Platform 4.19, the installation program is a Go binary file that performs a series of file transformations on a set of assets. The way we interact with the installation program differs depending on the installation type. Consider the following installation use cases:

For the installation program, the program uses three sets of files during installation: an installation configuration file named install-config.yaml, Kubernetes manifests, and Ignition config files for the machine types.

We can modify Kubernetes and the Ignition config files that control the underlying RHCOS operating system during installation. However, no validation is available to confirm the suitability of any modifications that we make to these objects. If we modify these objects, we might render the cluster non-functional. Because of this risk, modifying Kubernetes and Ignition config files is not supported unless we are following documented procedures or are instructed to do so by Red Hat support.

The installation configuration file is transformed into Kubernetes manifests, and then the manifests are wrapped into Ignition config files. The installation program uses these Ignition config files to create the cluster.

The installation configuration files are all pruned when we run the installation program, so be sure to back up all the configuration files that we want to use again.

We cannot modify the parameters that we set during installation, but we can modify many cluster attributes after installation.


The installation process with the Assisted Installer

Installation with the Assisted Installer involves creating a cluster configuration interactively by using the web-based user interface or the RESTful API. The Assisted Installer user interface prompts us for required values and provides reasonable default values for the remaining parameters, unless we change them in the user interface or with the API. The Assisted Installer generates a discovery image, which we download and use to boot the cluster machines. The image installs RHCOS and an agent, and the agent handles the provisioning for you. We can install OpenShift Container Platform with the Assisted Installer and full integration on Nutanix, vSphere, and bare metal. Additionally, we can install OpenShift Container Platform with the Assisted Installer on other platforms without integration.

OpenShift Container Platform manages all aspects of the cluster, including the operating system itself. Each machine boots with a configuration that references resources hosted in the cluster that it joins. This configuration allows the cluster to manage itself as updates are applied.

If possible, use the Assisted Installer feature to avoid having to download and configure the Agent-based Installer.


The installation process with Agent-based infrastructure

Agent-based installation is similar to using the Assisted Installer, except that we must initially download and install the Agent-based Installer. An Agent-based installation is useful when we want the convenience of the Assisted Installer, but we need to install a cluster in a disconnected environment.

If possible, use the Agent-based installation feature to avoid having to create a provisioner machine with a bootstrap VM, and then provision and maintain the cluster infrastructure.


The installation process with installer-provisioned infrastructure

The default installation type uses installer-provisioned infrastructure. By default, the installation program acts as an installation wizard, prompting us for values that it cannot determine on its own and providing reasonable default values for the remaining parameters. We can also customize the installation process to support advanced infrastructure scenarios. The installation program provisions the underlying infrastructure for the cluster.

We can install either a standard cluster or a customized cluster. With a standard cluster, we provide minimum details that are required to install the cluster. With a customized cluster, we can specify more details about the platform, such as the number of machines that the control plane uses, the type of virtual machine that the cluster deploys, or the CIDR range for the Kubernetes service network.

If possible, use this feature to avoid having to provision and maintain the cluster infrastructure. In all other environments, we use the installation program to generate the assets that we require to provision the cluster infrastructure.

With installer-provisioned infrastructure clusters, OpenShift Container Platform manages all aspects of the cluster, including the operating system itself. Each machine boots with a configuration that references resources hosted in the cluster that it joins. This configuration allows the cluster to manage itself as updates are applied.


The installation process with user-provisioned infrastructure

Install OpenShift Container Platform on infrastructure that we provide. We use the installation program to generate the assets that we require to provision the cluster infrastructure, create the cluster infrastructure, and then deploy the cluster to the infrastructure that we provided.

If we do not use infrastructure that the installation program provisioned, we must manage and maintain the cluster resources yourself. The following list details some of these self-managed resources:

If the cluster uses user-provisioned infrastructure, we have the option of adding RHEL compute machines to the cluster.


Installation process details

When a cluster is provisioned, each machine in the cluster requires information about the cluster. OpenShift Container Platform uses a temporary bootstrap machine during initial configuration to provide the required information to the permanent control plane. The temporary bootstrap machine boots by using an Ignition config file that describes how to create the cluster. The bootstrap machine creates the control plane machines that make up the control plane. The control plane machines then create the compute machines, which are also known as worker machines. The following figure illustrates this process:

Figure 3.2. Creating the bootstrap, control plane, and compute machines

Creating bootstrap View larger image

Important

While planning to deploy the cluster, ensure that we are familiar with the recommended practices for performance and scalability, particularly the requirements for input/output (I/O) latency for etcd storage and the requirements for the recommended control plane node sizing. For more information, see “Recommended etcd practices” and “Control plane node sizing”.

After the cluster machines initialize, the bootstrap machine is destroyed. All clusters use the bootstrap process to initialize the cluster, but if we provision the infrastructure for the cluster, we must complete many of the steps manually.

Important

Bootstrapping a cluster involves the following steps:

  1. The bootstrap machine boots and starts hosting the remote resources required for the control plane machines to boot. If we provision the infrastructure, this step requires manual intervention.
  2. The bootstrap machine starts a single-node etcd cluster and a temporary Kubernetes control plane.
  3. The control plane machines fetch the remote resources from the bootstrap machine and finish booting. If we provision the infrastructure, this step requires manual intervention.
  4. The temporary control plane schedules the production control plane to the production control plane machines.
  5. The Cluster Version Operator (CVO) comes online and installs the etcd Operator. The etcd Operator scales up etcd on all control plane nodes.
  6. The temporary control plane shuts down and passes control to the production control plane.
  7. The bootstrap machine injects OpenShift Container Platform components into the production control plane.
  8. The installation program shuts down the bootstrap machine. If we provision the infrastructure, this step requires manual intervention.
  9. The control plane sets up the compute nodes.
  10. The control plane installs additional services in the form of a set of Operators.

The result of this bootstrapping process is a running OpenShift Container Platform cluster. The cluster then downloads and configures remaining components needed for the day-to-day operations, including the creation of compute machines in supported environments.

Additional resources


Installation scope

The scope of the OpenShift Container Platform installation program is intentionally narrow. It is designed for simplicity and ensured success. We can complete many more configuration tasks after installation completes.

Additional resources


3.2. About the OpenShift Update Service

The OpenShift Update Service (OSUS) provides update recommendations to OpenShift Container Platform, including Red Hat Enterprise Linux CoreOS (RHCOS). It provides a graph, or diagram, that contains the vertices of component Operators and the edges that connect them. The edges in the graph show which versions we can safely update to. The vertices are update payloads that specify the intended state of the managed cluster components.

The Cluster Version Operator (CVO) in the cluster checks with the OpenShift Update Service to see the valid updates and update paths based on current component versions and information in the graph. When we request an update, the CVO uses the corresponding release image to update the cluster. The release artifacts are hosted in Quay as container images.

To allow the OpenShift Update Service to provide only compatible updates, a release verification pipeline drives automation. Each release artifact is verified for compatibility with supported cloud platforms and system architectures, as well as other component packages. After the pipeline confirms the suitability of a release, the OpenShift Update Service notifies we that it is available.

The OpenShift Update Service (OSUS) supports a single-stream release model, where only one release version is active and supported at any given time. When a new release is deployed, it fully replaces the previous release.

The updated release provides support for upgrades from all OpenShift Container Platform versions starting after 4.8 up to the new release version.

Important

The OpenShift Update Service displays all recommended updates for our current cluster. If an update path is not recommended by the OpenShift Update Service, it might be because of a known issue related to the update path, such as incompatibility or availability.

Two controllers run during continuous update mode. The first controller continuously updates the payload manifests, applies the manifests to the cluster, and outputs the controlled rollout status of the Operators to indicate whether they are available, upgrading, or failed. The second controller polls the OpenShift Update Service to determine if updates are available.

Important

Only updating to a newer version is supported. Reverting or rolling back the cluster to a previous version is not supported. If the update fails, contact Red Hat support.

During the update process, the Machine Config Operator (MCO) applies the new configuration to the cluster machines. The MCO cordons the number of nodes specified by the maxUnavailable field on the machine configuration pool and marks them unavailable. By default, this value is set to 1. The MCO updates the affected nodes alphabetically by zone, based on the topology.kubernetes.io/zone label. If a zone has more than one node, the oldest nodes are updated first. For nodes that do not use zones, such as in bare metal deployments, the nodes are updated by age, with the oldest nodes updated first. The MCO updates the number of nodes as specified by the maxUnavailable field on the machine configuration pool at a time. The MCO then applies the new configuration and reboots the machine.

Warning

The default setting for maxUnavailable is 1 for all the machine config pools in OpenShift Container Platform. It is recommended to not change this value and update one control plane node at a time. Do not change this value to 3 for the control plane pool.

If we use Red Hat Enterprise Linux (RHEL) machines as workers, the MCO does not update the kubelet because we must update the OpenShift API on the machines first.

With the specification for the new version applied to the old kubelet, the RHEL machine cannot return to the Ready state. We cannot complete the update until the machines are available. However, the maximum number of unavailable nodes is set to ensure that normal cluster operations can continue with that number of machines out of service.

The OpenShift Update Service is composed of an Operator and one or more application instances.


3.3. Support policy for unmanaged Operators

The management state of an Operator determines whether an Operator is actively managing the resources for its related component in the cluster as designed. If an Operator is set to an unmanaged state, it does not respond to changes in configuration nor does it receive updates.

While this can be helpful in non-production clusters or during debugging, Operators in an unmanaged state are unsupported and the cluster administrator assumes full control of the individual component configurations and upgrades.

An Operator can be set to an unmanaged state using the following methods:


3.4. Next steps


Chapter 4. Red Hat OpenShift Cluster Manager

Red Hat OpenShift Cluster Manager is a managed service where we can install, modify, operate, and upgrade our Red Hat OpenShift clusters. This service allows we to work with all of our organization’s clusters from a single dashboard.

OpenShift Cluster Manager guides we to install OpenShift Container Platform, Red Hat OpenShift Service on AWS (ROSA), and {product-short-name} clusters. It is also responsible for managing both OpenShift Container Platform clusters after self-installation as well as our ROSA and {product-short-name} clusters.

We can use OpenShift Cluster Manager to do the following actions:


4.1. Accessing Red Hat OpenShift Cluster Manager

We can access OpenShift Cluster Manager with our configured OpenShift account.

Prerequisites

Procedure


4.2. General actions

On the top right of the cluster page, there are some actions that a user can perform on the entire cluster:


4.3. Cluster tabs

Selecting an active, installed cluster shows tabs associated with that cluster. The following tabs display after the cluster’s installation completes:


4.3.1. Overview tab

The Overview tab provides information about how the cluster was configured:


4.3.2. Access control tab

The Access control tab allows the cluster owner to set up an identity provider, grant elevated permissions, and grant roles to other users.


4.3.2.1. Identity providers

We can create the cluster’s identity provider in this section. See the Additional resources for more information.


4.3.2.2. Cluster roles and acess

We can create a dedicated-admins role for {product-short-name} clusters or cluster-admins role for OpenShift Container Platform clusters.

Procedure

  1. Click the Add user button.
  2. Enter the ID of the user we want to grant cluster admin access.
  3. Select the appropriate group for our user. Either dedicated-admins for {product-short-name} clusters, or cluster-admins for clusters.

4.3.2.3. OCM roles and access

Prerequisites

Procedure

  1. Click the Grant role button.
  2. Enter the Red Hat account login for the user that we wish to grant a role on the cluster.

  3. Select the role from following options:

    • Cluster editor allows users or groups to manage or configure the cluster.
    • Cluster viewer allows users or groups to view cluster details only.
    • Identity provider editor allows users or groups to manage and configure the identity providers.
    • Machine pool editor allows users or groups to manage and configure the machine pools.
  4. Click the Grant role button on the dialog box.


4.3.3. Add-ons tab


4.3.4. Insights Advisor tab

The Insights Advisor tab uses the Remote Health functionality of the OpenShift Container Platform to identify and mitigate risks to security, performance, availability, and stability. See Using Insights to identify issues with the cluster in the OpenShift Container Platform documentation.


4.3.5. Machine pools tab

The Machine pools tab allows the cluster owner to create new machine pools if there is enough available quota, or edit an existing machine pool.

Selecting the kebab > Edit option opens the "Edit machine pool" dialog. In this dialog, we can change the node count per availability zone, edit node labels and taints, and view any associated AWS security groups.


4.3.6. Support tab

In the Support tab, we can add notification contacts for individuals that should receive cluster notifications. The username or email address that we provide must relate to a user account in the Red Hat organization where the cluster is deployed.

Also from this tab, we can open a support case to request technical support for the cluster.


4.3.7. Settings tab

The Settings tab provides a few options for the cluster owner:


4.4. Additional resources


Chapter 5. About the multicluster engine for Kubernetes Operator

One of the challenges of scaling Kubernetes environments is managing the lifecycle of a growing fleet. To meet that challenge, we can use the multicluster engine Operator. The operator delivers full lifecycle capabilities for managed OpenShift Container Platform clusters and partial lifecycle management for other Kubernetes distributions. It is available in two ways:


5.1. Cluster management with multicluster engine on OpenShift Container Platform

When we enable multicluster engine on OpenShift Container Platform, we gain the following capabilities:

The multicluster engine is included with our OpenShift Container Platform support subscription and is delivered separately from the core payload. To start to use multicluster engine, we deploy the OpenShift Container Platform cluster and then install the operator. For more information, see Installing and upgrading multicluster engine operator.


5.2. Cluster management with Red Hat Advanced Cluster Management

If we need cluster management capabilities beyond what OpenShift Container Platform with multicluster engine can provide, consider Red Hat Advanced Cluster Management. The multicluster engine is an integral part of Red Hat Advanced Cluster Management and is enabled by default.


5.3. Additional resources

For the complete documentation for multicluster engine, see Cluster lifecycle with multicluster engine documentation, which is part of the product documentation for Red Hat Advanced Cluster Management.


Chapter 6. Control plane architecture

The control plane, which is composed of control plane machines, manages the OpenShift Container Platform cluster. The control plane machines manage workloads on the compute machines, which are also known as worker machines. The cluster itself manages all upgrades to the machines by the actions of the Cluster Version Operator (CVO), the Machine Config Operator, and a set of individual Operators.


6.1. Node configuration management with machine config pools

Machines that run control plane components or user workloads are divided into groups based on the types of resources they handle. These groups of machines are called machine config pools (MCP). Each MCP manages a set of nodes and its corresponding machine configs. The role of the node determines which MCP it belongs to; the MCP governs nodes based on its assigned node role label. Nodes in an MCP have the same configuration; this means nodes can be scaled up and torn down in response to increased or decreased workloads.

By default, there are two MCPs created by the cluster when it is installed: master and worker. Each default MCP has a defined configuration applied by the Machine Config Operator (MCO), which is responsible for managing MCPs and facilitating MCP updates.

For worker nodes, we can create additional MCPs, or custom pools, to manage nodes with custom use cases that extend outside of the default node types. Custom MCPs for the control plane nodes are not supported.

Custom pools are pools that inherit their configurations from the worker pool. They use any machine config targeted for the worker pool, but add the ability to deploy changes only targeted at the custom pool. Since a custom pool inherits its configuration from the worker pool, any change to the worker pool is applied to the custom pool as well. Custom pools that do not inherit their configurations from the worker pool are not supported by the MCO.

Note

A node can only be included in one MCP. If a node has multiple labels that correspond to several MCPs, like worker,infra, it is managed by the infra custom pool, not the worker pool. Custom pools take priority on selecting nodes to manage based on node labels; nodes that do not belong to a custom pool are managed by the worker pool.

It is recommended to have a custom pool for every node role we want to manage in the cluster. For example, if we create infra nodes to handle infra workloads, it is recommended to create a custom infra MCP to group those nodes together. If we apply an infra role label to a worker node so it has the worker,infra dual label, but do not have a custom infra MCP, the MCO considers it a worker node. If we remove the worker label from a node and apply the infra label without grouping it in a custom pool, the node is not recognized by the MCO and is unmanaged by the cluster.

Important

Any node labeled with the infra role that is only running infra workloads is not counted toward the total number of subscriptions. The MCP managing an infra node is mutually exclusive from how the cluster determines subscription charges; tagging a node with the appropriate infra role and using taints to prevent user workloads from being scheduled on that node are the only requirements for avoiding subscription charges for infra workloads.

The MCO applies updates for pools independently; for example, if there is an update that affects all pools, nodes from each pool update in parallel with each other. If we add a custom pool, nodes from that pool also attempt to update concurrently with the master and worker nodes.

There might be situations where the configuration on a node does not fully match what the currently-applied machine config specifies. This state is called configuration drift. The Machine Config Daemon (MCD) regularly checks the nodes for configuration drift. If the MCD detects configuration drift, the MCO marks the node degraded until an administrator corrects the node configuration. A degraded node is online and operational, but, it cannot be updated.

Additional resources


6.2. Machine roles in OpenShift Container Platform

OpenShift Container Platform assigns hosts different roles. These roles define the function of the machine within the cluster. The cluster contains definitions for the standard master and worker role types.

Note

The cluster also contains the definition for the bootstrap role. Because the bootstrap machine is used only during cluster installation, its function is explained in the cluster installation documentation.


6.2.1. Control plane and node host compatibility

The OpenShift Container Platform version must match between control plane host and node host. For example, in a 4.19 cluster, all control plane hosts must be 4.19 and all nodes must be 4.19.

Temporary mismatches during cluster upgrades are acceptable. For example, when upgrading from the previous OpenShift Container Platform version to 4.19, some nodes will upgrade to 4.19 before others. Prolonged skewing of control plane hosts and node hosts might expose older compute machines to bugs and missing features. Users should resolve skewed control plane hosts and node hosts as soon as possible.

The kubelet service must not be newer than kube-apiserver, and can be up to two minor versions older depending on whether our OpenShift Container Platform version is odd or even. The table below shows the appropriate version compatibility:

Platform>OpenShift Container Platform version Platform>Supported kubelet skew
Odd OpenShift Container Platform minor versions [1] Up to one version older
Even OpenShift Container Platform minor versions [2] Up to two versions older

  1. For example, OpenShift Container Platform 4.11, 4.13.
  2. For example, OpenShift Container Platform 4.10, 4.12.


6.2.2. Cluster workers

In a Kubernetes cluster, worker nodes run and manage the actual workloads requested by Kubernetes users. The worker nodes advertise their capacity and the scheduler, which is a control plane service, determines on which nodes to start pods and containers. The following important services run on each worker node:

Note

For information about how to enable runC instead of the default crun, see the documentation for creating a ContainerRuntimeConfig CR.

In OpenShift Container Platform, compute machine sets control the compute machines, which are assigned the worker machine role. Machines with the worker role drive compute workloads that are governed by a specific machine pool that autoscales them. Because OpenShift Container Platform has the capacity to support multiple machine types, the machines with the worker role are classed as compute machines. In this release, the terms worker machine and compute machine are used interchangeably because the only default type of compute machine is the worker machine. In future versions of OpenShift Container Platform, different types of compute machines, such as infrastructure machines, might be used by default.

Note

Compute machine sets are groupings of compute machine resources under the machine-api namespace. Compute machine sets are configurations that are designed to start new compute machines on a specific cloud provider. Conversely, machine config pools (MCPs) are part of the Machine Config Operator (MCO) namespace. An MCP is used to group machines together so the MCO can manage their configurations and facilitate their upgrades.


6.2.3. Cluster control planes

In a Kubernetes cluster, the master nodes run services that are required to control the Kubernetes cluster. In OpenShift Container Platform, the control plane is comprised of control plane machines that have a master machine role. They contain more than just the Kubernetes services for managing the OpenShift Container Platform cluster.

For most OpenShift Container Platform clusters, control plane machines are defined by a series of standalone machine API resources. For supported cloud provider and OpenShift Container Platform version combinations, control planes can be managed with control plane machine sets. Extra controls apply to control plane machines to prevent we from deleting all of the control plane machines and breaking the cluster.

Note

Exactly three control plane nodes must be used for all production deployments. However, on bare metal platforms, clusters can be scaled up to five control plane nodes.

Services that fall under the Kubernetes category on the control plane include the Kubernetes API server, etcd, the Kubernetes controller manager, and the Kubernetes scheduler.

Platform>Component Platform>Description
Kubernetes API server The Kubernetes API server validates and configures the data for pods, services, and replication controllers. It also provides a focal point for the shared state of the cluster.
etcd etcd stores the persistent control plane state while other components watch etcd for changes to bring themselves into the specified state.
Kubernetes controller manager The Kubernetes controller manager watches etcd for changes to objects such as replication, namespace, and service account controller objects, and then uses the API to enforce the specified state. Several such processes create a cluster with one active leader at a time.
Kubernetes scheduler The Kubernetes scheduler watches for newly created pods without an assigned node and selects the best node to host the pod.

There are also OpenShift services that run on the control plane, which include the OpenShift API server, OpenShift controller manager, OpenShift OAuth API server, and OpenShift OAuth server.

Platform>Component Platform>Description
OpenShift API server The OpenShift API server validates and configures the data for OpenShift resources, such as projects, routes, and templates.

The OpenShift API server is managed by the OpenShift API Server Operator.

OpenShift controller manager The OpenShift controller manager watches etcd for changes to OpenShift objects, such as project, route, and template controller objects, and then uses the API to enforce the specified state.

The OpenShift controller manager is managed by the OpenShift Controller Manager Operator.

OpenShift OAuth API server The OpenShift OAuth API server validates and configures the data to authenticate to OpenShift Container Platform, such as users, groups, and OAuth tokens.

The OpenShift OAuth API server is managed by the Cluster Authentication Operator.

OpenShift OAuth server Users request tokens from the OpenShift OAuth server to authenticate themselves to the API.

The OpenShift OAuth server is managed by the Cluster Authentication Operator.

Some of these services on the control plane machines run as systemd services, while others run as static pods.

Systemd services are appropriate for services that we need to always come up on that particular system shortly after it starts. For control plane machines, those include sshd, which allows remote login. It also includes services such as:

CRI-O and Kubelet must run directly on the host as systemd services because they need to be running before we can run other containers.

The installer-* and revision-pruner-* control plane pods must run with root permissions because they write to the /etc/kubernetes directory, which is owned by the root user. These pods are in the following namespaces:

Additional resources


6.3. Operators in OpenShift Container Platform

Operators are among the most important components of OpenShift Container Platform. They are the preferred method of packaging, deploying, and managing services on the control plane. They can also provide advantages to applications that users run.

Operators integrate with Kubernetes APIs and CLI tools such as kubectl and the OpenShift CLI (oc). They provide the means of monitoring applications, performing health checks, managing over-the-air (OTA) updates, and ensuring that applications remain in our specified state.

Operators also offer a more granular configuration experience. We configure each component by modifying the API that the Operator exposes instead of modifying a global configuration file.

Because CRI-O and the Kubelet run on every node, almost every other cluster function can be managed on the control plane by using Operators. Components that are added to the control plane by using Operators include critical networking and credential services.

While both follow similar Operator concepts and goals, Operators in OpenShift Container Platform are managed by two different systems, depending on their purpose:

Cluster Operators Managed by the Cluster Version Operator (CVO) and installed by default to perform cluster functions.
Optional add-on Operators Managed by Operator Lifecycle Manager (OLM) and can be made accessible for users to run in their applications. Also known as OLM-based Operators.


6.3.1. Cluster Operators

In OpenShift Container Platform, all cluster functions are divided into a series of default cluster Operators. Cluster Operators manage a particular area of cluster functionality, such as cluster-wide application logging, management of the Kubernetes control plane, or the machine provisioning system.

Cluster Operators are represented by a ClusterOperator object, which cluster administrators can view in the OpenShift Container Platform web console from the Administration → Cluster Settings page. Each cluster Operator provides an API for determining cluster functionality. The Operator hides the details of managing the lifecycle of that component. Operators can manage a single component or tens of components, but the end goal is always to reduce operational burden by automating common actions.

Additional resources


6.3.2. Add-on Operators

Operator Lifecycle Manager (OLM) and OperatorHub are default components in OpenShift Container Platform that help manage Kubernetes-native applications as Operators. Together they provide the system for discovering, installing, and managing the optional add-on Operators available on the cluster.

Using OperatorHub in the OpenShift Container Platform web console, cluster administrators and authorized users can select Operators to install from catalogs of Operators. After installing an Operator from OperatorHub, it can be made available globally or in specific namespaces to run in user applications.

Default catalog sources are available that include Red Hat Operators, certified Operators, and community Operators. Cluster administrators can also add their own custom catalog sources, which can contain a custom set of Operators.

Note

OLM does not manage the cluster Operators that comprise the OpenShift Container Platform architecture.

Additional resources


6.4. Overview of etcd

etcd is a consistent, distributed key-value store that holds small amounts of data that can fit entirely in memory. Although etcd is a core component of many projects, it is the primary data store for Kubernetes, which is the standard system for container orchestration.


6.4.1. Benefits of using etcd

By using etcd, we can benefit in several ways:


6.4.2. How etcd works

To ensure a reliable approach to cluster configuration and management, etcd uses the etcd Operator. The Operator simplifies the use of etcd on a Kubernetes container platform like OpenShift Container Platform. With the etcd Operator, we can create or delete etcd members, resize clusters, perform backups, and upgrade etcd.

The etcd Operator observes, analyzes, and acts:

  1. It observes the cluster state by using the Kubernetes API.
  2. It analyzes differences between the current state and the state that we want.
  3. It fixes the differences through the etcd cluster management APIs, the Kubernetes API, or both.

etcd holds the cluster state, which is constantly updated. This state is continuously persisted, which leads to a high number of small changes at high frequency. As a result, it is critical to back the etcd cluster member with fast, low-latency I/O. For more information about best practices for etcd, see "Recommended etcd practices".

Additional resources


Chapter 7. Understanding OpenShift Container Platform development

To fully leverage the capability of containers when developing and running enterprise-quality applications, ensure our environment is supported by tools that allow containers to be:

Containers’ widespread acceptance, and the resulting requirements for tools and methods to make them enterprise-ready, resulted in many options for them.

The rest of this section explains options for assets we can create when we build and deploy containerized Kubernetes applications in OpenShift Container Platform. It also describes which approaches we might use for different kinds of applications and development requirements.


7.1. About developing containerized applications

We can approach application development with containers in many ways, and different approaches might be more appropriate for different situations. To illustrate some of this variety, the series of approaches presented starts with developing a single container and ultimately deploys that container as a mission-critical application for a large enterprise. These approaches show different tools, formats, and methods that we can employ with containerized application development. This topic describes:


7.2. Building a simple container

We have an idea for an application and we want to containerize it.

First we require a tool for building a container, like buildah or docker, and a file that describes what goes in our container, which is typically a Dockerfile.

Next, we require a location to push the resulting container image so we can pull it to run anywhere we want it to run. This location is a container registry.

Some examples of each of these components are installed by default on most Linux operating systems, except for the Dockerfile, which we provide yourself.

The following diagram displays the process of building and pushing an image:

Figure 7.1. Create a simple containerized application and push it to a registry

Creating and pushing a containerized application View larger image

If we use a computer that runs Red Hat Enterprise Linux (RHEL) as the operating system, the process of creating a containerized application requires the following steps:

  1. Install container build tools: RHEL contains a set of tools that includes podman, buildah, and skopeo that we use to build and manage containers.
  2. Create a Dockerfile to combine base image and software: Information about building our container goes into a file named Dockerfile. In that file, we identify the base image we build from, the software packages we install, and the software we copy into the container. We also identify parameter values like network ports that we expose outside the container and volumes that we mount inside the container. Put our Dockerfile and the software we want to containerize in a directory on our RHEL system.
  3. Run buildah or docker build: Run the buildah build-using-dockerfile or the docker build command to pull our chosen base image to the local system and create a container image stored locally. We can also build container images without a Dockerfile by using buildah.
  4. Tag and push to a registry: Add a tag to our new container image that identifies the location of the registry in which we want to store and share our container. Then push that image to the registry by running the podman push or docker push command.
  5. Pull and run the image: From any system that has a container client tool, such as podman or docker, run a command that identifies our new image. For example, run the podman run <image_name> or docker run <image_name> command. Here <image_name> is the name of our new container image, which resembles quay.io/myrepo/myapp:latest. The registry might require credentials to push and pull images.

For more details on the process of building container images, pushing them to registries, and running them, see Custom image builds with Buildah.


7.2.1. Container build tool options

Building and managing containers with buildah, podman, and skopeo results in industry standard container images that include features specifically tuned for deploying containers in OpenShift Container Platform or other Kubernetes environments. These tools are daemonless and can run without root privileges, requiring less overhead to run them.

Important

Support for Docker Container Engine as a container runtime is deprecated in Kubernetes 1.20 and will be removed in a future release. However, Docker-produced images will continue to work in the cluster with all runtimes, including CRI-O. For more information, see the Kubernetes blog announcement.

When we ultimately run our containers in OpenShift Container Platform, we use the CRI-O container engine. CRI-O runs on every worker and control plane machine in an OpenShift Container Platform cluster, but CRI-O is not yet supported as a standalone runtime outside of OpenShift Container Platform.


7.2.2. Base image options

The base image we choose to build our application on contains a set of software that resembles a Linux system to our application. When we build our own image, our software is placed into that file system and sees that file system as though it were looking at its operating system. Choosing this base image has major impact on how secure, efficient and upgradeable our container is in the future.

Red Hat provides a new set of base images referred to as Red Hat Universal Base Images (UBI). These images are based on Red Hat Enterprise Linux and are similar to base images that Red Hat has offered in the past, with one major difference: they are freely redistributable without a Red Hat subscription. As a result, we can build our application on UBI images without having to worry about how they are shared or the need to create different images for different environments.

These UBI images have standard, init, and minimal versions. We can also use the Red Hat Software Collections images as a foundation for applications that rely on specific runtime environments such as Node.js, Perl, or Python. Special versions of some of these runtime base images are referred to as Source-to-Image (S2I) images. With S2I images, we can insert our code into a base image environment ready to run that code.

S2I images are available for we to use directly from the OpenShift Container Platform web UI. In the Developer perspective, navigate to the +Add view and in the Developer Catalog tile, view all of the available services in the Developer Catalog.

Figure 7.2. Choose S2I base images for apps that need specific runtimes

OpenShift Container Platform Developer Catalog View larger image


7.2.3. Registry options

Container registries are where we store container images so we can share them with others and make them available to the platform where they ultimately run. We can select large, public container registries that offer free accounts or a premium version that offer more storage and special features. We can also install our own registry that can be exclusive to our organization or selectively shared with others.

To get Red Hat images and certified partner images, we can draw from the Red Hat Registry. The Red Hat Registry is represented by two locations: registry.access.redhat.com, which is unauthenticated and deprecated, and registry.redhat.io, which requires authentication. We can learn about the Red Hat and partner images in the Red Hat Registry from the Container images section of the Red Hat Ecosystem Catalog. Besides listing Red Hat container images, it also shows extensive information about the contents and quality of those images, including health scores that are based on applied security updates.

Large, public registries include Docker Hub and Quay.io. The Quay.io registry is owned and managed by Red Hat. Many of the components used in OpenShift Container Platform are stored in Quay.io, including container images and the Operators that are used to deploy OpenShift Container Platform itself. Quay.io also offers the means of storing other types of content, including Helm charts.

If we want our own, private container registry, OpenShift Container Platform itself includes a private container registry installed with OpenShift Container Platform and runs on its cluster. Red Hat also offers a private version of the Quay.io registry called Red Hat Quay. Red Hat Quay includes geo replication, Git build triggers, Clair image scanning, and many other features.

All of the registries mentioned here can require credentials to download images from those registries. Some of those credentials are presented on a cluster-wide basis from OpenShift Container Platform, while other credentials can be assigned to individuals.


7.3. Creating a Kubernetes manifest for OpenShift Container Platform

While the container image is the basic building block for a containerized application, more information is required to manage and deploy that application in a Kubernetes environment such as OpenShift Container Platform. The typical next steps after we create an image are to:


7.3.1. About Kubernetes pods and services

While the container image is the basic unit with docker, the basic units that Kubernetes works with are called pods. Pods represent the next step in building out an application. A pod can contain one or more than one container. The key is that the pod is the single unit that we deploy, scale, and manage.

Scalability and namespaces are probably the main items to consider when determining what goes in a pod. For ease of deployment, we might want to deploy a container in a pod and include its own logging and monitoring container in the pod. Later, when we run the pod and need to scale up an additional instance, those other containers are scaled up with it. For namespaces, containers in a pod share the same network interfaces, shared storage volumes, and resource limitations, such as memory and CPU, which makes it easier to manage the contents of the pod as a single unit. Containers in a pod can also communicate with each other by using standard inter-process communications, such as System V semaphores or POSIX shared memory.

While individual pods represent a scalable unit in Kubernetes, a service provides a means of grouping together a set of pods to create a complete, stable application that can complete tasks such as load balancing. A service is also more permanent than a pod because the service remains available from the same IP address until we delete it. When the service is in use, it is requested by name and the OpenShift Container Platform cluster resolves that name into the IP addresses and ports where we can reach the pods that compose the service.

By their nature, containerized applications are separated from the operating systems where they run and, by extension, their users. Part of our Kubernetes manifest describes how to expose the application to internal and external networks by defining network policies that allow fine-grained control over communication with containerized applications. To connect incoming requests for HTTP, HTTPS, and other services from outside the cluster to services inside the cluster, we can use an Ingress resource.

If the container requires on-disk storage instead of database storage, which might be provided through a service, we can add volumes to our manifests to make that storage available to our pods. We can configure the manifests to create persistent volumes (PVs) or dynamically create volumes that are added to our Pod definitions.

After we define a group of pods that compose our application, we can define those pods in Deployment and DeploymentConfig objects.


7.3.2. Application types

Next, consider how our application type influences how to run it.

Kubernetes defines different types of workloads that are appropriate for different kinds of applications. To determine the appropriate workload for our application, consider if the application is:


7.3.3. Available supporting components

The application we write might need supporting components, like a database or a logging component. To fulfill that need, we might be able to obtain the required component from the following Catalogs that are available in the OpenShift Container Platform web console:

We can configure the supporting Operators and templates to the specific needs of our development team and then make them available in the namespaces in which our developers work. Many people add shared templates to the openshift namespace because it is accessible from all other namespaces.


7.3.4. Applying the manifest

Kubernetes manifests let we create a more complete picture of the components that make up our Kubernetes applications. We write these manifests as YAML files and deploy them by applying them to the cluster, for example, by running the oc apply command.


7.3.5. Next steps

At this point, consider ways to automate our container development process. Ideally, we have some sort of CI pipeline that builds the images and pushes them to a registry. In particular, a GitOps pipeline integrates our container development with the Git repositories that we use to store the software required to build applications.

The workflow to this point might look like:


7.4. Develop for Operators

Packaging and deploying our application as an Operator might be preferred if we make our application available for others to run. As noted earlier, Operators add a lifecycle component to our application that acknowledges that the job of running an application is not complete as soon as it is installed.

When we create an application as an Operator, we can build in our own knowledge of how to run and maintain the application. We can build in features for upgrading the application, backing it up, scaling it, or keeping track of its state. If we configure the application correctly, maintenance tasks, like updating the Operator, can happen automatically and invisibly to the Operator’s users.

An example of a useful Operator is one set up to automatically back up data at particular times. Having an Operator manage an application’s backup at set times can save a system administrator from remembering to do it.

Any application maintenance that has traditionally been completed manually, like backing up data or rotating certificates, can be completed automatically with an Operator.


Chapter 8. Red Hat Enterprise Linux CoreOS (RHCOS)


8.1. About RHCOS

Red Hat Enterprise Linux CoreOS (RHCOS) represents the next generation of single-purpose container operating system technology by providing the quality standards of Red Hat Enterprise Linux (RHEL) with automated, remote upgrade features.

RHCOS is supported only as a component of OpenShift Container Platform 4.19 for all OpenShift Container Platform machines. RHCOS is the only supported operating system for all node types in OpenShift Container Platform. RHCOS is deployed in OpenShift Container Platform 4.19 in two general ways:


8.1.1. Key RHCOS features

The following list describes key features of the RHCOS operating system:

For RHCOS systems, the layout of the rpm-ostree file system has the following characteristics:


8.1.2. Choosing how to configure RHCOS

RHCOS is designed to deploy on an OpenShift Container Platform cluster with a minimal amount of user configuration. In its most basic form, this consists of:

Because RHCOS systems in OpenShift Container Platform are designed to be fully managed from the OpenShift Container Platform cluster after that, directly changing an RHCOS machine is discouraged. Although limited direct access to RHCOS machines cluster can be accomplished for debugging purposes, we should not directly configure RHCOS systems. Instead, if we need to add or change features on our OpenShift Container Platform nodes, consider making changes in the following ways:

Here are examples of customizations we could do on day 1:

To accomplish these tasks, we can augment the openshift-install process to include additional objects such as MachineConfig objects. Those procedures that result in creating machine configs can be passed to the Machine Config Operator after the cluster is up.

Note


8.1.3. Choosing how to deploy RHCOS

Differences between RHCOS installations for OpenShift Container Platform are based on whether we are deploying on an infrastructure provisioned by the installer or by the user:

The Ignition facility runs only when the RHCOS system is first set up. After that, Ignition configs can be supplied later using the machine config.


8.1.4. About Ignition

Ignition is the utility used by RHCOS to manipulate disks during initial configuration. It completes common disk tasks, including partitioning disks, formatting partitions, writing files, and configuring users. On first boot, Ignition reads its configuration from the installation media or the location that we specify and applies the configuration to the machines.

Whether we are installing the cluster or adding machines to it, Ignition always performs the initial configuration of the OpenShift Container Platform cluster machines. Most of the actual system setup happens on each machine itself. For each machine, Ignition takes the RHCOS image and boots the RHCOS kernel. Options on the kernel command line identify the type of deployment and the location of the Ignition-enabled initial RAM disk (initramfs).


8.1.4.1. How Ignition works

To create machines by using Ignition, we need Ignition config files. The OpenShift Container Platform installation program creates the Ignition config files that we need to deploy the cluster. These files are based on the information that we provide to the installation program directly or through an install-config.yaml file.

The way that Ignition configures machines is similar to how tools like cloud-init or Linux Anaconda kickstart configure systems, but with some important differences:

8.1.4.2. The Ignition sequence

The Ignition process for an RHCOS machine in an OpenShift Container Platform cluster involves the following steps:

At the end of this process, the machine is ready to join the cluster and does not require a reboot.


8.2. Viewing Ignition configuration files

To see the Ignition config file used to deploy the bootstrap machine, run the following command: 

openshift-install create ignition-configs --dir $HOME/testconfig
$ openshift-install create ignition-configs --dir $HOME/testconfig

After we answer a few questions, the bootstrap.ign, master.ign, and worker.ign files appear in the directory we entered.

To see the contents of the bootstrap.ign file, pipe it through the jq filter. Here’s a snippet from that file: 

cat $HOME/testconfig/bootstrap.ign | jq
{
  "ignition": {
    "version": "3.2.0"
  },
  "passwd": {
    "users": [
      {
        "name": "core",
        "sshAuthorizedKeys": [
          "ssh-rsa AAAAB3NzaC1yc...."
        ]
      }
    ]
  },
  "storage": {
    "files": [
      {
        "overwrite": false,
        "path": "/etc/motd",
        "user": {
          "name": "root"
        },
        "append": [
          {
            "source": "data:text/plain;charset=utf-8;base64,VGhpcyBpcyB0aGUgYm9vdHN0cmFwIG5vZGU7IGl0IHdpbGwgYmUgZGVzdHJveWVkIHdoZW4gdGhlIG1hc3RlciBpcyBmdWxseSB1cC4KClRoZSBwcmltYXJ5IHNlcnZpY2VzIGFyZSByZWxlYXNlLWltYWdlLnNlcnZpY2UgZm9sbG93ZWQgYnkgYm9vdGt1YmUuc2VydmljZS4gVG8gd2F0Y2ggdGhlaXIgc3RhdHVzLCBydW4gZS5nLgoKICBqb3VybmFsY3RsIC1iIC1mIC11IHJlbGVhc2UtaW1hZ2Uuc2VydmljZSAtdSBib290a3ViZS5zZXJ2aWNlCg=="
          }
        ],
        "mode": 420
      },
...
$ cat $HOME/testconfig/bootstrap.ign | jq
{
  "ignition": {
    "version": "3.2.0"
  },
  "passwd": {
    "users": [
      {
        "name": "core",
        "sshAuthorizedKeys": [
          "ssh-rsa AAAAB3NzaC1yc...."
        ]
      }
    ]
  },
  "storage": {
    "files": [
      {
        "overwrite": false,
        "path": "/etc/motd",
        "user": {
          "name": "root"
        },
        "append": [
          {
            "source": "data:text/plain;charset=utf-8;base64,VGhpcyBpcyB0aGUgYm9vdHN0cmFwIG5vZGU7IGl0IHdpbGwgYmUgZGVzdHJveWVkIHdoZW4gdGhlIG1hc3RlciBpcyBmdWxseSB1cC4KClRoZSBwcmltYXJ5IHNlcnZpY2VzIGFyZSByZWxlYXNlLWltYWdlLnNlcnZpY2UgZm9sbG93ZWQgYnkgYm9vdGt1YmUuc2VydmljZS4gVG8gd2F0Y2ggdGhlaXIgc3RhdHVzLCBydW4gZS5nLgoKICBqb3VybmFsY3RsIC1iIC1mIC11IHJlbGVhc2UtaW1hZ2Uuc2VydmljZSAtdSBib290a3ViZS5zZXJ2aWNlCg=="
          }
        ],
        "mode": 420
      },
...

To decode the contents of a file listed in the bootstrap.ign file, pipe the base64-encoded data string representing the contents of that file to the base64 -d command. Here’s an example using the contents of the /etc/motd file added to the bootstrap machine from the output shown above: 

echo VGhpcyBpcyB0aGUgYm9vdHN0cmFwIG5vZGU7IGl0IHdpbGwgYmUgZGVzdHJveWVkIHdoZW4gdGhlIG1hc3RlciBpcyBmdWxseSB1cC4KClRoZSBwcmltYXJ5IHNlcnZpY2VzIGFyZSByZWxlYXNlLWltYWdlLnNlcnZpY2UgZm9sbG93ZWQgYnkgYm9vdGt1YmUuc2VydmljZS4gVG8gd2F0Y2ggdGhlaXIgc3RhdHVzLCBydW4gZS5nLgoKICBqb3VybmFsY3RsIC1iIC1mIC11IHJlbGVhc2UtaW1hZ2Uuc2VydmljZSAtdSBib290a3ViZS5zZXJ2aWNlCg== | base64 --decode
$ echo VGhpcyBpcyB0aGUgYm9vdHN0cmFwIG5vZGU7IGl0IHdpbGwgYmUgZGVzdHJveWVkIHdoZW4gdGhlIG1hc3RlciBpcyBmdWxseSB1cC4KClRoZSBwcmltYXJ5IHNlcnZpY2VzIGFyZSByZWxlYXNlLWltYWdlLnNlcnZpY2UgZm9sbG93ZWQgYnkgYm9vdGt1YmUuc2VydmljZS4gVG8gd2F0Y2ggdGhlaXIgc3RhdHVzLCBydW4gZS5nLgoKICBqb3VybmFsY3RsIC1iIC1mIC11IHJlbGVhc2UtaW1hZ2Uuc2VydmljZSAtdSBib290a3ViZS5zZXJ2aWNlCg== | base64 --decode

Example output 

This is the bootstrap node; it will be destroyed when the master is fully up.

The primary services are release-image.service followed by bootkube.service. To watch their status, run e.g.

  journalctl -b -f -u release-image.service -u bootkube.service
This is the bootstrap node; it will be destroyed when the master is fully up.

The primary services are release-image.service followed by bootkube.service. To watch their status, run e.g.

  journalctl -b -f -u release-image.service -u bootkube.service

Repeat those commands on the master.ign and worker.ign files to see the source of Ignition config files for each of those machine types. We should see a line like the following for the worker.ign, identifying how it gets its Ignition config from the bootstrap machine: 

"source": "https://api.myign.develcluster.example.com:22623/config/worker",
"source": "https://api.myign.develcluster.example.com:22623/config/worker",

Here are a few things we can learn from the bootstrap.ign file:


8.3. Changing Ignition configs after installation

Machine config pools manage a cluster of nodes and their corresponding machine configs. Machine configs contain configuration information for a cluster. To list all machine config pools that are known: 

oc get machineconfigpools
$ oc get machineconfigpools

Example output 

NAME   CONFIG                                  UPDATED UPDATING DEGRADED
master master-1638c1aea398413bb918e76632f20799 False   False    False
worker worker-2feef4f8288936489a5a832ca8efe953 False   False    False
NAME   CONFIG                                  UPDATED UPDATING DEGRADED
master master-1638c1aea398413bb918e76632f20799 False   False    False
worker worker-2feef4f8288936489a5a832ca8efe953 False   False    False

To list all machine configs: 

oc get machineconfig
$ oc get machineconfig

Example output 

NAME                                      GENERATEDBYCONTROLLER   IGNITIONVERSION   CREATED   OSIMAGEURL

00-master                                 4.0.0-0.150.0.0-dirty   3.5.0             16m
00-master-ssh                             4.0.0-0.150.0.0-dirty                     16m
00-worker                                 4.0.0-0.150.0.0-dirty   3.5.0             16m
00-worker-ssh                             4.0.0-0.150.0.0-dirty                     16m
01-master-kubelet                         4.0.0-0.150.0.0-dirty   3.5.0             16m
01-worker-kubelet                         4.0.0-0.150.0.0-dirty   3.5.0             16m
master-1638c1aea398413bb918e76632f20799   4.0.0-0.150.0.0-dirty   3.5.0             16m
worker-2feef4f8288936489a5a832ca8efe953   4.0.0-0.150.0.0-dirty   3.5.0             16m
NAME                                      GENERATEDBYCONTROLLER   IGNITIONVERSION   CREATED   OSIMAGEURL

00-master                                 4.0.0-0.150.0.0-dirty   3.5.0             16m
00-master-ssh                             4.0.0-0.150.0.0-dirty                     16m
00-worker                                 4.0.0-0.150.0.0-dirty   3.5.0             16m
00-worker-ssh                             4.0.0-0.150.0.0-dirty                     16m
01-master-kubelet                         4.0.0-0.150.0.0-dirty   3.5.0             16m
01-worker-kubelet                         4.0.0-0.150.0.0-dirty   3.5.0             16m
master-1638c1aea398413bb918e76632f20799   4.0.0-0.150.0.0-dirty   3.5.0             16m
worker-2feef4f8288936489a5a832ca8efe953   4.0.0-0.150.0.0-dirty   3.5.0             16m

The Machine Config Operator acts somewhat differently than Ignition when it comes to applying these machine configs. The machine configs are read in order (from 00* to 99*). Labels inside the machine configs identify the type of node each is for (master or worker). If the same file appears in multiple machine config files, the last one wins. So, for example, any file that appears in a 99* file would replace the same file that appeared in a 00* file. The input MachineConfig objects are unioned into a "rendered" MachineConfig object, which will be used as a target by the operator and is the value we can see in the machine config pool.

To see what files are being managed from a machine config, look for "Path:" inside a particular MachineConfig object. For example: 

oc describe machineconfigs 01-worker-container-runtime | grep Path:
$ oc describe machineconfigs 01-worker-container-runtime | grep Path:

Example output 

            Path:            /etc/containers/registries.conf
            Path:            /etc/containers/storage.conf
            Path:            /etc/crio/crio.conf
            Path:            /etc/containers/registries.conf
            Path:            /etc/containers/storage.conf
            Path:            /etc/crio/crio.conf

Be sure to give the machine config file a later name (such as 10-worker-container-runtime). Keep in mind that the content of each file is in URL-style data. Then apply the new machine config to the cluster.


Chapter 9. Admission plugins

Admission plugins are used to help regulate how OpenShift Container Platform functions.


9.1. About admission plugins

Admission plugins intercept requests to the master API to validate resource requests. After a request is authenticated and authorized, the admission plugins ensure that any associated policies are followed. For example, they are commonly used to enforce security policy, resource limitations or configuration requirements.

Admission plugins run in sequence as an admission chain. If any admission plugin in the sequence rejects a request, the whole chain is aborted and an error is returned.

OpenShift Container Platform has a default set of admission plugins enabled for each resource type. These are required for proper functioning of the cluster. Admission plugins ignore resources that they are not responsible for.

In addition to the defaults, the admission chain can be extended dynamically through webhook admission plugins that call out to custom webhook servers. There are two types of webhook admission plugins: a mutating admission plugin and a validating admission plugin. The mutating admission plugin runs first and can both modify resources and validate requests. The validating admission plugin validates requests and runs after the mutating admission plugin so that modifications triggered by the mutating admission plugin can also be validated.

Calling webhook servers through a mutating admission plugin can produce side effects on resources related to the target object. In such situations, we must take steps to validate that the end result is as expected.

Warning

Dynamic admission should be used cautiously because it impacts cluster control plane operations. When calling webhook servers through webhook admission plugins in OpenShift Container Platform 4.19, ensure that we have read the documentation fully and tested for side effects of mutations. Include steps to restore resources back to their original state prior to mutation, in the event that a request does not pass through the entire admission chain.


9.2. Default admission plugins

Default validating and admission plugins are enabled in OpenShift Container Platform 4.19. These default plugins contribute to fundamental control plane functionality, such as ingress policy, cluster resource limit override and quota policy.

Important

Do not run workloads in or share access to default projects. Default projects are reserved for running core cluster components.

The following default projects are considered highly privileged: default, kube-public, kube-system, openshift, openshift-infra, openshift-node, and other system-created projects that have the openshift.io/run-level label set to 0 or 1. Functionality that relies on admission plugins, such as pod security admission, security context constraints, cluster resource quotas, and image reference resolution, does not work in highly privileged projects.

The following lists contain the default admission plugins:

Example 9.1. Validating admission plugins

Example 9.2. Mutating admission plugins


9.3. Webhook admission plugins

In addition to OpenShift Container Platform default admission plugins, dynamic admission can be implemented through webhook admission plugins that call webhook servers, to extend the functionality of the admission chain. Webhook servers are called over HTTP at defined endpoints.

There are two types of webhook admission plugins in OpenShift Container Platform:

When an API request comes in, mutating or validating admission plugins use the list of external webhooks in the configuration and call them in parallel:

Communication between the webhook admission plugin and the webhook server must use TLS. Generate a CA certificate and use the certificate to sign the server certificate used by our webhook admission server. The PEM-encoded CA certificate is supplied to the webhook admission plugin using a mechanism, such as service serving certificate secrets.

The following diagram illustrates the sequential admission chain process within which multiple webhook servers are called.

Figure 9.1. API admission chain with mutating and validating admission plugins

API admission stage View larger image

API admission stage

An example webhook admission plugin use case is where all pods must have a common set of labels. In this example, the mutating admission plugin can inject labels and the validating admission plugin can check that labels are as expected. OpenShift Container Platform would subsequently schedule pods that include required labels and reject those that do not.

Some common webhook admission plugin use cases include:

Note

The maximum default webhook timeout value in OpenShift Container Platform is 13 seconds, and it cannot be changed.


9.4. Types of webhook admission plugins

Cluster administrators can call out to webhook servers through the mutating admission plugin or the validating admission plugin in the API server admission chain.


9.4.1. Mutating admission plugin

The mutating admission plugin is invoked during the mutation phase of the admission process, which allows modification of resource content before it is persisted. One example webhook that can be called through the mutating admission plugin is the Pod Node Selector feature, which uses an annotation on a namespace to find a label selector and add it to the pod specification.

Sample mutating admission plugin configuration

apiVersion: admissionregistration.k8s.io/v1beta1
kind: MutatingWebhookConfiguration 
metadata:
  name: <webhook_name> 
webhooks:
- name: <webhook_name> 
  clientConfig: 
    service:
      namespace: default 
      name: kubernetes 
      path: <webhook_url> 
    caBundle: <ca_signing_certificate> 
  rules: 
  - operations: 
    - <operation>
    apiGroups:
    - ""
    apiVersions:
    - "*"
    resources:
    - <resource>
  failurePolicy: <policy> 
  sideEffects: None
apiVersion: admissionregistration.k8s.io/v1beta1
kind: MutatingWebhookConfiguration
1 
metadata:
  name: <webhook_name>
2 
webhooks:
- name: <webhook_name>
3 
  clientConfig:
4 
    service:
      namespace: default
5 
      name: kubernetes
6 
      path: <webhook_url>
7 
    caBundle: <ca_signing_certificate>
8 
  rules:
9 
  - operations:
10 
    - <operation>
    apiGroups:
    - ""
    apiVersions:
    - "*"
    resources:
    - <resource>
  failurePolicy: <policy>
11 
  sideEffects: None

1 Specifies a mutating admission plugin configuration.
2 The name for the MutatingWebhookConfiguration object. Replace <webhook_name> with the appropriate value.
3 The name of the webhook to call. Replace <webhook_name> with the appropriate value.
4 Information about how to connect to, trust, and send data to the webhook server.
5 The namespace where the front-end service is created.
6 The name of the front-end service.
7 The webhook URL used for admission requests. Replace <webhook_url> with the appropriate value.
8 A PEM-encoded CA certificate that signs the server certificate used by the webhook server. Replace <ca_signing_certificate> with the appropriate certificate in base64 format.
9 Rules that define when the API server should use this webhook admission plugin.
10 One or more operations that trigger the API server to call this webhook admission plugin. Possible values are create, update, delete or connect. Replace <operation> and <resource> with the appropriate values.
11 Specifies how the policy should proceed if the webhook server is unavailable. Replace <policy> with either Ignore (to unconditionally accept the request in the event of a failure) or Fail (to deny the failed request). Using Ignore can result in unpredictable behavior for all clients.

Important

In OpenShift Container Platform 4.19, objects created by users or control loops through a mutating admission plugin might return unexpected results, especially if values set in an initial request are overwritten, which is not recommended.


9.4.2. Validating admission plugin

A validating admission plugin is invoked during the validation phase of the admission process. This phase allows the enforcement of invariants on particular API resources to ensure that the resource does not change again. The Pod Node Selector is also an example of a webhook which is called by the validating admission plugin, to ensure that all nodeSelector fields are constrained by the node selector restrictions on the namespace.

Sample validating admission plugin configuration

apiVersion: admissionregistration.k8s.io/v1beta1
kind: ValidatingWebhookConfiguration 
metadata:
  name: <webhook_name> 
webhooks:
- name: <webhook_name> 
  clientConfig: 
    service:
      namespace: default  
      name: kubernetes 
      path: <webhook_url> 
    caBundle: <ca_signing_certificate> 
  rules: 
  - operations: 
    - <operation>
    apiGroups:
    - ""
    apiVersions:
    - "*"
    resources:
    - <resource>
  failurePolicy: <policy> 
  sideEffects: Unknown
apiVersion: admissionregistration.k8s.io/v1beta1
kind: ValidatingWebhookConfiguration
1 
metadata:
  name: <webhook_name>
2 
webhooks:
- name: <webhook_name>
3 
  clientConfig:
4 
    service:
      namespace: default
5 
      name: kubernetes
6 
      path: <webhook_url>
7 
    caBundle: <ca_signing_certificate>
8 
  rules:
9 
  - operations:
10 
    - <operation>
    apiGroups:
    - ""
    apiVersions:
    - "*"
    resources:
    - <resource>
  failurePolicy: <policy>
11 
  sideEffects: Unknown

1 Specifies a validating admission plugin configuration.
2 The name for the ValidatingWebhookConfiguration object. Replace <webhook_name> with the appropriate value.
3 The name of the webhook to call. Replace <webhook_name> with the appropriate value.
4 Information about how to connect to, trust, and send data to the webhook server.
5 The namespace where the front-end service is created.
6 The name of the front-end service.
7 The webhook URL used for admission requests. Replace <webhook_url> with the appropriate value.
8 A PEM-encoded CA certificate that signs the server certificate used by the webhook server. Replace <ca_signing_certificate> with the appropriate certificate in base64 format.
9 Rules that define when the API server should use this webhook admission plugin.
10 One or more operations that trigger the API server to call this webhook admission plugin. Possible values are create, update, delete or connect. Replace <operation> and <resource> with the appropriate values.
11 Specifies how the policy should proceed if the webhook server is unavailable. Replace <policy> with either Ignore (to unconditionally accept the request in the event of a failure) or Fail (to deny the failed request). Using Ignore can result in unpredictable behavior for all clients.



9.5. Configuring dynamic admission

This procedure outlines high-level steps to configure dynamic admission. The functionality of the admission chain is extended by configuring a webhook admission plugin to call out to a webhook server.

The webhook server is also configured as an aggregated API server. This allows other OpenShift Container Platform components to communicate with the webhook using internal credentials and facilitates testing using the oc command. Additionally, this enables role based access control (RBAC) into the webhook and prevents token information from other API servers from being disclosed to the webhook.

Prerequisites

  • An OpenShift Container Platform account with cluster administrator access.
  • The OpenShift Container Platform CLI (oc) installed.
  • A published webhook server container image.

Procedure

  1. Build a webhook server container image and make it available to the cluster using an image registry.
  2. Create a local CA key and certificate and use them to sign the webhook server’s certificate signing request (CSR).

  3. Create a new project for webhook resources:

      oc new-project my-webhook-namespace
      $ oc new-project my-webhook-namespace
    1

    1 Note that the webhook server might expect a specific name.

  4. Define RBAC rules for the aggregated API service in a file called rbac.yaml: 
    apiVersion: v1
kind: List
items:

- apiVersion: rbac.authorization.k8s.io/v1  
  kind: ClusterRoleBinding
  metadata:
    name: auth-delegator-my-webhook-namespace
  roleRef:
    kind: ClusterRole
    apiGroup: rbac.authorization.k8s.io
    name: system:auth-delegator
  subjects:
  - kind: ServiceAccount
    namespace: my-webhook-namespace
    name: server

- apiVersion: rbac.authorization.k8s.io/v1  
  kind: ClusterRole
  metadata:
    annotations:
    name: system:openshift:online:my-webhook-server
  rules:
  - apiGroups:
    - online.openshift.io
    resources:
    - namespacereservations  
    verbs:
    - get
    - list
    - watch

- apiVersion: rbac.authorization.k8s.io/v1  
  kind: ClusterRole
  metadata:
    name: system:openshift:online:my-webhook-requester
  rules:
  - apiGroups:
    - admission.online.openshift.io
    resources:
    - namespacereservations 
    verbs:
    - create

- apiVersion: rbac.authorization.k8s.io/v1  
  kind: ClusterRoleBinding
  metadata:
    name: my-webhook-server-my-webhook-namespace
  roleRef:
    kind: ClusterRole
    apiGroup: rbac.authorization.k8s.io
    name: system:openshift:online:my-webhook-server
  subjects:
  - kind: ServiceAccount
    namespace: my-webhook-namespace
    name: server

- apiVersion: rbac.authorization.k8s.io/v1  
  kind: RoleBinding
  metadata:
    namespace: kube-system
    name: extension-server-authentication-reader-my-webhook-namespace
  roleRef:
    kind: Role
    apiGroup: rbac.authorization.k8s.io
    name: extension-apiserver-authentication-reader
  subjects:
  - kind: ServiceAccount
    namespace: my-webhook-namespace
    name: server

- apiVersion: rbac.authorization.k8s.io/v1  
  kind: ClusterRole
  metadata:
    name: my-cluster-role
  rules:
  - apiGroups:
    - admissionregistration.k8s.io
    resources:
    - validatingwebhookconfigurations
    - mutatingwebhookconfigurations
    verbs:
    - get
    - list
    - watch
  - apiGroups:
    - ""
    resources:
    - namespaces
    verbs:
    - get
    - list
    - watch

- apiVersion: rbac.authorization.k8s.io/v1
  kind: ClusterRoleBinding
  metadata:
    name: my-cluster-role
  roleRef:
    kind: ClusterRole
    apiGroup: rbac.authorization.k8s.io
    name: my-cluster-role
  subjects:
  - kind: ServiceAccount
    namespace: my-webhook-namespace
    name: server
    apiVersion: v1
kind: List
items:

- apiVersion: rbac.authorization.k8s.io/v1
    1 
      kind: ClusterRoleBinding
  metadata:
    name: auth-delegator-my-webhook-namespace
  roleRef:
    kind: ClusterRole
    apiGroup: rbac.authorization.k8s.io
    name: system:auth-delegator
  subjects:
  - kind: ServiceAccount
    namespace: my-webhook-namespace
    name: server

- apiVersion: rbac.authorization.k8s.io/v1
    2 
      kind: ClusterRole
  metadata:
    annotations:
    name: system:openshift:online:my-webhook-server
  rules:
  - apiGroups:
    - online.openshift.io
    resources:
    - namespacereservations
    3 
        verbs:
    - get
    - list
    - watch

- apiVersion: rbac.authorization.k8s.io/v1
    4 
      kind: ClusterRole
  metadata:
    name: system:openshift:online:my-webhook-requester
  rules:
  - apiGroups:
    - admission.online.openshift.io
    resources:
    - namespacereservations
    5 
        verbs:
    - create

- apiVersion: rbac.authorization.k8s.io/v1
    6 
      kind: ClusterRoleBinding
  metadata:
    name: my-webhook-server-my-webhook-namespace
  roleRef:
    kind: ClusterRole
    apiGroup: rbac.authorization.k8s.io
    name: system:openshift:online:my-webhook-server
  subjects:
  - kind: ServiceAccount
    namespace: my-webhook-namespace
    name: server

- apiVersion: rbac.authorization.k8s.io/v1
    7 
      kind: RoleBinding
  metadata:
    namespace: kube-system
    name: extension-server-authentication-reader-my-webhook-namespace
  roleRef:
    kind: Role
    apiGroup: rbac.authorization.k8s.io
    name: extension-apiserver-authentication-reader
  subjects:
  - kind: ServiceAccount
    namespace: my-webhook-namespace
    name: server

- apiVersion: rbac.authorization.k8s.io/v1
    8 
      kind: ClusterRole
  metadata:
    name: my-cluster-role
  rules:
  - apiGroups:
    - admissionregistration.k8s.io
    resources:
    - validatingwebhookconfigurations
    - mutatingwebhookconfigurations
    verbs:
    - get
    - list
    - watch
  - apiGroups:
    - ""
    resources:
    - namespaces
    verbs:
    - get
    - list
    - watch

- apiVersion: rbac.authorization.k8s.io/v1
  kind: ClusterRoleBinding
  metadata:
    name: my-cluster-role
  roleRef:
    kind: ClusterRole
    apiGroup: rbac.authorization.k8s.io
    name: my-cluster-role
  subjects:
  - kind: ServiceAccount
    namespace: my-webhook-namespace
    name: server

    1 Delegates authentication and authorization to the webhook server API.
    2 Allows the webhook server to access cluster resources.
    3 Points to resources. This example points to the namespacereservations resource.
    4 Enables the aggregated API server to create admission reviews.
    5 Points to resources. This example points to the namespacereservations resource.
    6 Enables the webhook server to access cluster resources.
    7 Role binding to read the configuration for terminating authentication.
    8 Default cluster role and cluster role bindings for an aggregated API server.

  5. Apply those RBAC rules to the cluster: 
    oc auth reconcile -f rbac.yaml
    $ oc auth reconcile -f rbac.yaml

  6. Create a YAML file called webhook-daemonset.yaml used to deploy a webhook as a daemon set server in a namespace: 
    apiVersion: apps/v1
kind: DaemonSet
metadata:
  namespace: my-webhook-namespace
  name: server
  labels:
    server: "true"
spec:
  selector:
    matchLabels:
      server: "true"
  template:
    metadata:
      name: server
      labels:
        server: "true"
    spec:
      serviceAccountName: server
      containers:
      - name: my-webhook-container  
        image: <image_registry_username>/<image_path>:<tag>  
        imagePullPolicy: IfNotPresent
        command:
        - <container_commands>  
        ports:
        - containerPort: 8443 
        volumeMounts:
        - mountPath: /var/serving-cert
          name: serving-cert
        readinessProbe:
          httpGet:
            path: /healthz
            port: 8443 
            scheme: HTTPS
      volumes:
      - name: serving-cert
        secret:
          defaultMode: 420
          secretName: server-serving-cert
    apiVersion: apps/v1
kind: DaemonSet
metadata:
  namespace: my-webhook-namespace
  name: server
  labels:
    server: "true"
spec:
  selector:
    matchLabels:
      server: "true"
  template:
    metadata:
      name: server
      labels:
        server: "true"
    spec:
      serviceAccountName: server
      containers:
      - name: my-webhook-container
    1 
            image: <image_registry_username>/<image_path>:<tag>
    2 
            imagePullPolicy: IfNotPresent
        command:
        - <container_commands>
    3 
            ports:
        - containerPort: 8443
    4 
            volumeMounts:
        - mountPath: /var/serving-cert
          name: serving-cert
        readinessProbe:
          httpGet:
            path: /healthz
            port: 8443
    5 
                scheme: HTTPS
      volumes:
      - name: serving-cert
        secret:
          defaultMode: 420
          secretName: server-serving-cert

    1 Note that the webhook server might expect a specific container name.
    2 Points to a webhook server container image. Replace <image_registry_username>/<image_path>:<tag> with the appropriate value.
    3 Specifies webhook container run commands. Replace <container_commands> with the appropriate value.
    4 Defines the target port within pods. This example uses port 8443.
    5 Specifies the port used by the readiness probe. This example uses port 8443.

  7. Deploy the daemon set: 
    oc apply -f webhook-daemonset.yaml
    $ oc apply -f webhook-daemonset.yaml

  8. Define a secret for the service serving certificate signer, within a YAML file called webhook-secret.yaml: 
    apiVersion: v1
kind: Secret
metadata:
  namespace: my-webhook-namespace
  name: server-serving-cert
type: kubernetes.io/tls
data:
  tls.crt: <server_certificate>  
  tls.key: <server_key>  
    apiVersion: v1
kind: Secret
metadata:
  namespace: my-webhook-namespace
  name: server-serving-cert
type: kubernetes.io/tls
data:
  tls.crt: <server_certificate>
    1 
      tls.key: <server_key>
    2 
    




    

    
1 
     References the signed webhook server certificate. Replace <server_certificate> with the appropriate certificate in base64 format.
 
    
2 
     References the signed webhook server key. Replace <server_key> with the appropriate key in base64 format.
       
    


  9.  Create the secret:
     


    oc apply -f webhook-secret.yaml
    


    $ oc apply -f webhook-secret.yaml
    





  10.  Define a service account and service, within a YAML file called webhook-service.yaml:
     


    apiVersion: v1
kind: List
items:

- apiVersion: v1
  kind: ServiceAccount
  metadata:
    namespace: my-webhook-namespace
    name: server

- apiVersion: v1
  kind: Service
  metadata:
    namespace: my-webhook-namespace
    name: server
    annotations:
      service.beta.openshift.io/serving-cert-secret-name: server-serving-cert
  spec:
    selector:
      server: "true"
    ports:
    - port: 443  
      targetPort: 8443  
    


    apiVersion: v1
kind: List
items:

- apiVersion: v1
  kind: ServiceAccount
  metadata:
    namespace: my-webhook-namespace
    name: server

- apiVersion: v1
  kind: Service
  metadata:
    namespace: my-webhook-namespace
    name: server
    annotations:
      service.beta.openshift.io/serving-cert-secret-name: server-serving-cert
  spec:
    selector:
      server: "true"
    ports:
    - port: 443  
    1


          targetPort: 8443  
    2


    




    

    
1 
     Defines the port that the service listens on. This example uses port 443.
 
    
2 
     Defines the target port within pods that the service forwards connections to. This example uses port 8443.
       
    


  11.  Expose the webhook server within the cluster:
     


    oc apply -f webhook-service.yaml
    


    $ oc apply -f webhook-service.yaml
    





  12.  Define a custom resource definition for the webhook server, in a file called webhook-crd.yaml:
     


    apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: namespacereservations.online.openshift.io  
spec:
  group: online.openshift.io  
  version: v1alpha1  
  scope: Cluster  
  names:
    plural: namespacereservations  
    singular: namespacereservation  
    kind: NamespaceReservation  
    


    apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: namespacereservations.online.openshift.io  
    1


    spec:
  group: online.openshift.io  
    2


      version: v1alpha1  
    3


      scope: Cluster  
    4


      names:
    plural: namespacereservations  
    5


        singular: namespacereservation  
    6


        kind: NamespaceReservation  
    7


    




    

    
1 
     Reflects CustomResourceDefinition spec values and is in the format <plural>.<group>. This example uses the namespacereservations resource.
 
    
2 
     REST API group name.
 
    
3 
     REST API version name.
 
    
4 
     Accepted values are Namespaced or Cluster.
 
    
5 
     Plural name to be included in URL.
 
    
6 
     Alias seen in oc output.
 
    
7 
     The reference for resource manifests.
       
    


  13.  Apply the custom resource definition:
     


    oc apply -f webhook-crd.yaml
    


    $ oc apply -f webhook-crd.yaml
    





  14.  Configure the webhook server also as an aggregated API server, within a file called webhook-api-service.yaml:
     


    apiVersion: apiregistration.k8s.io/v1beta1
kind: APIService
metadata:
  name: v1beta1.admission.online.openshift.io
spec:
  caBundle: <ca_signing_certificate>  
  group: admission.online.openshift.io
  groupPriorityMinimum: 1000
  versionPriority: 15
  service:
    name: server
    namespace: my-webhook-namespace
  version: v1beta1
    


    apiVersion: apiregistration.k8s.io/v1beta1
kind: APIService
metadata:
  name: v1beta1.admission.online.openshift.io
spec:
  caBundle: <ca_signing_certificate>  
    1


      group: admission.online.openshift.io
  groupPriorityMinimum: 1000
  versionPriority: 15
  service:
    name: server
    namespace: my-webhook-namespace
  version: v1beta1
    




    

    
1 
     A PEM-encoded CA certificate that signs the server certificate used by the webhook server. Replace <ca_signing_certificate> with the appropriate certificate in base64 format.
       
    


  15.  Deploy the aggregated API service:
     


    oc apply -f webhook-api-service.yaml
    


    $ oc apply -f webhook-api-service.yaml
    





  16.  Define the webhook admission plugin configuration within a file called webhook-config.yaml. This example uses the validating admission plugin:
     


    apiVersion: admissionregistration.k8s.io/v1beta1
kind: ValidatingWebhookConfiguration
metadata:
  name: namespacereservations.admission.online.openshift.io  
webhooks:
- name: namespacereservations.admission.online.openshift.io  
  clientConfig:
    service:  
      namespace: default
      name: kubernetes
      path: /apis/admission.online.openshift.io/v1beta1/namespacereservations  
    caBundle: <ca_signing_certificate>  
  rules:
  - operations:
    - CREATE
    apiGroups:
    - project.openshift.io
    apiVersions:
    - "*"
    resources:
    - projectrequests
  - operations:
    - CREATE
    apiGroups:
    - ""
    apiVersions:
    - "*"
    resources:
    - namespaces
  failurePolicy: Fail
    


    apiVersion: admissionregistration.k8s.io/v1beta1
kind: ValidatingWebhookConfiguration
metadata:
  name: namespacereservations.admission.online.openshift.io  
    1


    webhooks:
- name: namespacereservations.admission.online.openshift.io  
    2


      clientConfig:
    service:  
    3


          namespace: default
      name: kubernetes
      path: /apis/admission.online.openshift.io/v1beta1/namespacereservations  
    4


        caBundle: <ca_signing_certificate>  
    5


      rules:
  - operations:
    - CREATE
    apiGroups:
    - project.openshift.io
    apiVersions:
    - "*"
    resources:
    - projectrequests
  - operations:
    - CREATE
    apiGroups:
    - ""
    apiVersions:
    - "*"
    resources:
    - namespaces
  failurePolicy: Fail
    




    

    
1 
     Name for the ValidatingWebhookConfiguration object. This example uses the namespacereservations resource.
 
    
2 
     Name of the webhook to call. This example uses the namespacereservations resource.
 
    
3 
     Enables access to the webhook server through the aggregated API.
 
    
4 
     The webhook URL used for admission requests. This example uses the namespacereservation resource.
 
    
5 
     A PEM-encoded CA certificate that signs the server certificate used by the webhook server. Replace <ca_signing_certificate> with the appropriate certificate in base64 format.
       
    


  17.  Deploy the webhook:
     


    oc apply -f webhook-config.yaml
    
 oc apply -f webhook-config.yaml





     
  18.  Verify that the webhook is functioning as expected. For example, if we have configured dynamic admission to reserve specific namespaces, confirm that requests to create those namespaces are rejected and that requests to create non-reserved namespaces succeed.