RHACM GitOps – Kustomize for Dev & Prod Environments

This blog demonstrates how to use RHACM GitOps to manage the MySQL application in the development and Production environment.

MySQL does not need persistent storage in development. And Persistent storage is required in production. environment. The development environment has one cluster and the production environment has the other cluster. The routes are different since applications have different domain names on 2 clusters. The rest should be the same for both environments.

We create a ‘base’ directory to store the common resource yaml files, the ‘overlays/dev’ directory has the overlay resources files for the development environment, and the ‘overlays/prod’ directory has the overlay resources files for the production environment

Create a Branch on GitHub

The first thing is to create a branch ‘MySQL’ for the application.

$ git checkout -b MySQL
Switched to a new branch 'MySQL'

Bases Resource files

Below are the resource yaml files under the directory base. All resource files are the common setting for both development and production environments.

$ tree base/
base/
├── deployment-frontend.yaml
├── deployment.yaml
├── kustomization.yml
├── service-frontend.yaml
└── service.yaml

If there is kustomization.yaml file in the Git folder, kustomize is applied. This file defined all the resource files.

$ cat base/kustomization.yml 
kind: Kustomization

resources:
- deployment-frontend.yaml
- deployment.yaml
- service.yaml
- service-frontend.yaml

Below is the deployment resource file. Note that the volume resource has only the name `db-volume`. It will have a different setting in development and production environments.

cat base/deployment.yaml 
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mysql
  labels:
    app: todonodejs
    name: mysql
spec:
  replicas: 1
  selector:
    matchLabels:
      app: todonodejs
      name: mysql
  template:
    metadata:
      labels:
        app: todonodejs
        name: mysql
    spec:
      containers:
      - image: registry.redhat.io/rhel8/mysql-80:1-152
        name: mysql
        env:
        - name: MYSQL_ROOT_PASSWORD
          value: r00tpa55
        - name: MYSQL_USER
          value: user1
        - name: MYSQL_PASSWORD
          value: mypa55
        - name: MYSQL_DATABASE
          value: items
        ports:
        - containerPort: 3306
          name: mysql
        volumeMounts:
        - mountPath: "/var/lib/mysql"
          name: db-volume
      volumes:
      - name: db-volume
      - name: db-init
        emptyDir: {}

Below is the overlay directory structure.

Overlay Resource files

$ tree overlays/
overlays/
├── dev
│   ├── dbclaim-pvc.yaml
│   ├── kustomization.yaml
│   ├── route.yaml
│   └── volume.yaml
└── prod
    ├── dbclaim-pvc.yaml
    ├── kustomization.yaml
    ├── route.yaml
    └── volume.yaml

The directory ‘dev’ has all the overlay resource files for the development environment.

The directory ‘prod’ has all the overlay resource files for the production environment.

Development Resource Files

Below is the kustomization.yaml file for the development environment.

We can use ‘spec.packageOverrides‘ to override kustomization in bases at the subscription deployment time.

The ‘resources' adds the new resource file(s) into the kustomization in bases.

The ‘namePrefix’ add the prefix to the resource name.

$ cat overlays/dev/kustomization.yaml 
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
bases:
- ../../base

patchesStrategicMerge:
- volume.yaml
resources:
- route.yaml
namePrefix: dev-

This is the ‘volume.yaml file. The lines above ’emptyDir: {}’ are used to detect the target resource to overwrite in bases. The target is the volume ‘db_volume’ inside the ‘mysql’ deployment. Please take note we cannot specify the ‘namespace’ in both bases and overlays resource files. The namespace setting will cause failure. We leave the namespace to be specified in RHACM subscription.

$ cat overlays/dev/volume.yaml 
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mysql
  labels:
    app: todonodejs
    name: mysql
spec:
  template:
    spec:
      volumes:
      - name: db-volume
        emptyDir: {}

Since the route domain ‘apps.bn7z2-m-dev.sandbox1558.opentlc.com’ is specific to the dev cluster, it should be in the overlay instead of the base.

$ cat overlays/prod/route.yaml 
apiVersion: route.openshift.io/v1
kind: Route
metadata:
  labels:
    app: todonodejs
    name: frontend
  name: frontend
spec:
  host: todo.apps.bn7z2-m-dev.sandbox1558.opentlc.com
  path: "/todo"
  to:
    kind: Service
    name: dev-frontend
    weight: 100
  wildcardPolicy: None

Use the below command to check the final settings. You can see the volume in deployment ‘mysql’ is set to ’emptyDir’, and route resource is added.

$ kubectl kustomize .
...
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: todonodejs
    name: mysql
  name: dev-mysql
spec:
  ...
  template:
    ...
    spec:
      containers:
      ...
      volumes:
      - emptyDir: {}
        name: db-volume
      ...
---
apiVersion: route.openshift.io/v1
kind: Route
metadata:
  labels:
    app: todonodejs
    name: frontend
  name: dev-frontend
spec:
  host: todo.apps.bn7z2-m-dev.sandbox1558.opentlc.com
  path: /todo
  to:
    kind: Service
    name: dev-frontend
    weight: 100
  wildcardPolicy: None

$ git add . ; git commit -a -m "update templates" ; git push --set-upstream origin MySQL

Once the resources’ settings are right, you can push new content to the branch in GitHub.

Production Resource Files

The settings in the prod overlay are similar to the dev environment, except it has the extra file ‘dbclaim-pvc.yaml’ for persistent storage.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: mysql-pv-claim-02
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 50Mi
  storageClassName: gp2

In the ‘kustomization.yaml’ file, ‘dbclaim-pvc.yaml’ is added under the ‘resources’.

$ cat overlays/prod/kustomization.yaml 
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
bases:
- ../../base

patchesStrategicMerge:
- volume.yaml
resources:
- dbclaim-pvc.yaml
- route.yaml
namePrefix: prod-

The pvc name ‘mysql-pv-claim-02’ is added to the file ‘volume.yaml’ which is used to update volume ‘db-volume’ for ‘mysql’ deployment.

$ cat overlays/prod/volume.yaml 
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mysql
  labels:
    app: todonodejs
    name: mysql
spec:
  template:
    spec:
      volumes:
      - name: db-volume
        persistentVolumeClaim:
          claimName: mysql-pv-claim-02

Since the route domain ‘aaps.bn7z2-m-prod.sandbox1558.opentlc.com’ is specific to the dev cluster, it should be in the overlay instead of the base.

$ cat overlays/prod/route.yaml 
apiVersion: route.openshift.io/v1
kind: Route
metadata:
  labels:
    app: todonodejs
    name: frontend
  name: frontend
spec:
  host: todo.apps.bn7z2-m-prod.sandbox1558.opentlc.com
  path: "/todo"
  to:
    kind: Service
    name: dev-frontend
    weight: 100
  wildcardPolicy: None

RHACM

From the RHACM console, deploy the development customized MySQL application.

• On the left navigation bar, navigate to Applications, click Create application, and then click Subscription.
• On the Applications page, set YAML: On to view the YAML in the console as you create the application.
• In both the Name and Namespace fields, type mysql.
• Click Repository location for resources and choose Git for the repository type for your deployable resources.
• For the channel source, enter the URL https://github.com/alpha-wolf-jin/acm-kustomize repository in the URL field. Enter the username and token since we use the private Git repository.
• Specify the Git Path ‘overlays/dev’ for the development environment.
• Scroll down, click Select clusters to deploy to, then select Deploy application resources only on clusters matching specified labels configure a placement rule for the application. Type purpose in the Label field and dev in the Value field.
• Click Create button

Note: specify the namespace here instead of resource yaml files.

Update the subscription name to ensure it is unique and meaningful for your environment.

---
apiVersion: apps.open-cluster-management.io/v1
kind: Subscription
metadata:
  name: dev-mysql-subscription-1
  ...
spec:
  ...
  placement:
    placementRef:
      name: dev-mysql-placement-1
      kind: PlacementRule
...
---
apiVersion: apps.open-cluster-management.io/v1
kind: PlacementRule
metadata:
  name: dev-mysql-placement-1
  ...
spec:
  clusterSelector:
    matchLabels:
      purpose: development

Repeat the same steps for the production environment.

Field	Value
Name	mysql
Namespace	mysql
Repository types	Git
URL	https://github.com/alpha-wolf-jin/acm-kustomize/
username	****
Access token	***************
Branch	MySQL
Path	overlays/prod
Label	purpose
Value	prod
Deployment window	Always Active

Update the subscription name to ensure it is unique and meaningful for your environment.

---
apiVersion: apps.open-cluster-management.io/v1
kind: Subscription
metadata:
  name: prod-mysql-subscription-1
  ...
spec:
  ...
  placement:
    placementRef:
      name: prod-mysql-placement-1
      kind: PlacementRule
...
---
apiVersion: apps.open-cluster-management.io/v1
kind: PlacementRule
metadata:
  name: prod-mysql-placement-1
  ...
spec:
  clusterSelector:
    matchLabels:
      purpose: prod

On the Topology page, navigate to Subscription →All Subscription. You can see both dev and prod subscriptions.

The above is one way to use RHACM GitOps to manage multi-clusters. Here, we use the Git branch for the application (you can have more branches for various applications), and the Git path for various environments. We use the cluster label to match the Git contents to the clusters. Of course, it is not the only way.

Of course, it is not the only way. I hope this sharing is able to give a clue on how to use RHACM GitOps to manage multi-clusters.