Cloud / Kubernetes / OpenShift

OpenShift 4: Pushing my cluster to the limit

When I involved in the design of Red Hat OpenShift 4 platform for customer deployment, the common question coming from the customer was how much my cluster can take the load, how many object my cluster can take and ultimately how I can push my cluster to the limit.

There is no magic number to this, each cluster in each specific environment will give different behaviour and result of the performance due to multiple factors (such as network performance, cpu capacity and capability, etc).

There are many ways of load testing (or soak testing) the cluster. Here we are going to talk about the cluster loader.

Cluster loader works by creating object defined by the user and the same time measuring performance metrics (from the OpenShift monitoring Grafana dashboard).

The cluster loader can be executed by running a origin-tests container image:

#> podman pull quay.io/openshift/origin-tests:4.9

There are built-in test suite, as an example:

#> podman run -v ${LOCAL_KUBECONFIG}:/root/.kube/config:z -i quay.io/openshift/origin-tests:4.9 /bin/bash -c 'export KUBECONFIG=/root/.kube/config && openshift-tests run-test "[sig-scalability][Feature:Performance] Load cluster should populate the cluster [Slow][Serial] [Suite:openshift]"'

We also can set cluster loader configuration YAML file to suite our need, as an example to set the tuning set (stepping, delay etc):

tuningsets: 
    - name: default
      pods:
        stepping: 
          stepsize: 5
          pause: 0 s
        rate_limit: 
          delay: 0 ms

Once we have the test result, we now can have a better view of how much our cluster can take. This also give us some hint if we need to give more resource to the cluster.

It is recommended to extend this with application load testing as well. It is a peace of mind to know what is the maximum load the cluster can take with the given resource with the maximum object deployed while application is running at maximum capacity.

Therefore, we can do a well informed capacity planning and expansion when required.

Disclaimer:

The views expressed and the content shared in all published articles on this website are solely those of the respective authors, and they do not necessarily reflect the views of the author’s employer or the techbeatly platform. We strive to ensure the accuracy and validity of the content published on our website. However, we cannot guarantee the absolute correctness or completeness of the information provided. It is the responsibility of the readers and users of this website to verify the accuracy and appropriateness of any information or opinions expressed within the articles. If you come across any content that you believe to be incorrect or invalid, please contact us immediately so that we can address the issue promptly.

Tags: kubernetes · OpenShift

Comments