Get up to 50% off on CKA, CKAD, CKS, KCNA, KCSA exams and courses!

OpenShift 3: DevSecOps with OpenShift – Image Signing

INTRODUCTION

Just like an RPM signing, container image singing can be used to verify authenticity of a container image created by users/developers throughout image life-cycle.

The concept of verification are similar to RPM gpgcheck verification.

Refer here for containerized image signing and scanning operator framework produced by Red Hat container-cop, automatically by requesting the signing/scanning service. As for understanding purposes, steps in this blog are manual step.

CONFIGURATION

Blog Prerequisites

While OpenShift can have varieties of container run-time option. This blog will focus for below environment. The selection was due to nature that docker by default not like other run-time will not honor policy.json out-of-the-box.

  • OCP running on full RHEL (7.5)
  • Docker container run-time (not CRI-O or others)
  • OCP 3.11 (Tested on)

Configuration Steps

We going to use master01 for signing node, while for production purpose, it is a best practise to have particular node for signing and kept very secure.

Overview of steps will be taken

  1. Create GPG Key Pair.
  2. Enable V2 endpoint for OCP registry.
  3. Configure atomic trust.
  4. Enforce Docker signature verification.
  5. Sign, Push and Verify image from OCP.
  6. Docker GPG check pull test.

1. Create GPG Key Pair

Existing GPG keypair can be used for this activity, in case we dont have any keypair yet.

[root@master01 ~]# gpg2 --gen-key

List generated keypair:

[root@master01 ~]# gpg --list-secret-keys
 /root/.gnupg/secring.gpg
 sec   2048R/687C588F 2019-02-24
 uid                  OCP Image Signer (Key for OCP Image) 
[[email protected]](mailto:[email protected])
 ssb   2048R/A31CA531 2019-02-24
 [root@master01 ~]# gpg --list-key
 /root/.gnupg/pubring.gpg
 pub   2048R/687C588F 2019-02-24
 uid                  OCP Image Signer (Key for OCP Image) 
[[email protected]](mailto:[email protected])
 sub   2048R/A31CA531 2019-02-24
 [root@master01 ~]# gpg --fingerprint
 /root/.gnupg/pubring.gpg
 pub   2048R/687C588F 2019-02-24
       Key fingerprint = 2AA9 DA41 1D89 A57D 9353  6F5B 4ABF BCBA 687C 588F
 uid                  OCP Image Signer (Key for OCP Image) 
[[email protected]](mailto:[email protected])
 sub   2048R/A31CA531 2019-02-24
 [root@master01 ~]#

On all nodes, ensure below public key content exported and exists. Copy /etc/pki/container/key.pub to all OPC nodes. This will act as verification key against signer key, just like RPM signing.

[root@master01 ~]# mkdir -p /etc/pki/containers/
 [root@master01 ~]# gpg2 --armor --export --output /etc/pki/containers/key.pub [email protected]

2. Enable V2 endpoint for OCP registry

Let`s freeze the automatic rollout for docker-registry.

[root@master01 ~]# oc rollout pause deploymentconfig docker-registry -n default

Set registry environment to enable V2 schema.

[root@master01 ~]#  oc set env dc/docker-registry REGISTRY_MIDDLEWARE_REPOSITORY_OPENSHIFT_ACCEPTSCHEMA2=true

Resume docker-registry rollout

[root@master01 ~]# oc rollout  resume dc/docker-registry -n default
 deploymentconfig.apps.openshift.io/docker-registry resumed
 [root@master01 ~]# oc rollout  latest dc/docker-registry -n default
 deploymentconfig.apps.openshift.io/docker-registry rolled out

Now lets see the header (expecting 401 return since no token provided) and look for X-Registry-Supports-Signatures: 1:

[root@master01 ~]# curl -kv https://docker-registry.default.svc:5000/v2/
 About to connect() to docker-registry.default.svc port 5000 (#0)
 Trying 10.51.1.99…
 Connected to docker-registry.default.svc (10.51.1.99) port 5000 (#0)
 Initializing NSS with certpath: sql:/etc/pki/nssdb
 skipping SSL peer certificate verification
 SSL connection using TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256
 Server certificate:
 subject: CN=10.51.1.99
 start date: Feb 12 14:17:40 2019 GMT
 expire date: Feb 11 14:17:41 2021 GMT
 common name: 10.51.1.99
 issuer: CN=openshift-signer@1549980226 
   GET /v2/ HTTP/1.1
   User-Agent: curl/7.29.0
   Host: docker-registry.default.svc:5000
   Accept: 
/
   < HTTP/1.1 401 Unauthorized
   < Content-Type: application/json; charset=utf-8
   < Docker-Distribution-Api-Version: registry/2.0
   < Www-Authenticate: Bearer realm="https://docker-registry.default.svc:5000/openshift/token"
   < X-Registry-Supports-Signatures: 1
   < Date: Sun, 24 Feb 2019 13:33:45 GMT
   < Content-Length: 87
   < 
   {"errors":[{"code":"UNAUTHORIZED","message":"authentication required","detail":null}]}
      Connection #0 to host docker-registry.default.svc left intact
   [root@master01 ~]#

Next we going to enable and configure atomic trust, hence only allowed registry and/or signed image allowed to be pulled into the node.

3. Configure atomic trust

This step can be automated via Ansible or other automation mean. In this example we are using master01 as configuration node and copy configs to all nodes.

As an example of Ansible play:

- hosts: OSEv3
   tasks:
 name: Create /etc/pki/containers directory
 file: path=/etc/pki/containers state=directory
 name: Create /etc/containers/registries.d directory
 file: path=/etc/containers/registries.d state=directory
 name: Copy trusted public keys
       copy: src=/etc/pki/containers/ dest=/etc/pki/containers
 name: Copy container trust policy
 copy: src=/etc/containers/policy.json
       dest=/etc/containers/policy.json
 name: Copy signature server configuration files
 copy: src=/etc/containers/registries.d/
       dest=/etc/containers/registries.d/

As default policy, reject all:

[root@master01 ~]# atomic trust default reject

Now we add our internal OCP registry(or any registry required to pull/push image):

[root@master01 ~]# atomic trust add docker-registry-default.apps.bytewise.com.my  --pubkeys /etc/pki/containers/key.pub
 [root@master01 ~]# atomic trust add docker-registry.default.svc:5000 --pubkeys /etc/pki/containers/key.pub
 [root@master01 ~]# atomic trust add quay.bytewise.com.my -t insecureAcceptAnything

Let list our atomic trust:

[root@master01 ~]# atomic trust show
 (default)                         reject                               
 docker-registry-default.apps.bytewise.com.my signed [email protected]              
 docker-registry.default.svc:5000    signed [email protected]              
 quay.bytewise.com.my                accept

/etc/containers/policy.json looks like this:

{
     "default": [
         {
             "type": "reject"
         }
     ], 
     "transports": {
         "docker": {
             "docker-registry-default.apps.bytewise.com.my": [
                 {
                     "keyType": "GPGKeys", 
                     "type": "signedBy", 
                     "keyData": "#########"
                 }
             ], 
             "quay.bytewise.com.my": [
                 {
                     "type": "insecureAcceptAnything"
                 }
             ], 
             "docker-registry.default.svc:5000": [
                 {
                     "keyType": "GPGKeys", 
                     "type": "signedBy", 
                     "keyData": "#########"
                 }
             ]
         }, 
         "docker-daemon": {
             "": [
                 {
                     "type": "insecureAcceptAnything"
                 }
             ]
         }
     }
 }

4. Enforce Docker signature verification

To enforce docker to verify image signature ( on all OCP nodes ):

NOTE : Ensure no duplicate line for ‘”signature-verification”: true’ in /etc/sysconfig/docker, or you may use configuration here as well instead of daemon.json.

[root@master01 ~]# cat /etc/docker/daemon.json
 {
     "log-driver": "journald",
     "signature-verification": true
 }
 [root@master01 ~]# systemctl restart docker

5. Sign, Push and Verify image from OCP

Now we going to push one test image (note debug is on for testing purposes):

[root@master01 ~]# atomic --debug push --type atomic --sign-by [email protected] docker-registry.default.svc:5000/django/httpd:latest

Once the image pushed, inspect the imageStream, where you can see the Status: Unverified:

[root@master01 sigstore]#  oc describe istag httpd:latest
 Image Name:          sha256:ce3878bf368075638dcd5cce8770443dcbc29f8d9bbe30029e65d34cbf31dc33
 Docker Image:          docker-registry.default.svc:5000/django/httpd@sha256:ce3878bf368075638dcd5cce8770443dcbc29f8d9bbe30029e65d34cbf31dc33
 Name:               sha256:ce3878bf368075638dcd5cce8770443dcbc29f8d9bbe30029e65d34cbf31dc33
 Created:          29 minutes ago
 Annotations:          image.openshift.io/dockerLayersOrder=ascending
                image.openshift.io/manifestBlobStored=true
                openshift.io/image.managed=true
 Image Size:          49.39MB in 5 layers
 Layers:               22.5MB     sha256:6ae821421a7debccb4151f7a50dc8ec0317674429bec0f275402d697047a8e96
                153B     sha256:0ceda4df88c8d23a0cd3c077725e3ac77c63786eca7bca89051e3cc2c7400aae
                10.33MB     sha256:24f08eb4db686b2136baac67538688d99064dc5a4a6a3b9ec821b7e4af71d6c3
                16.55MB     sha256:ddf4fc3180816e62e990fdb516292a3d4c20faf53e044d722848e5f0bdcc7d19
                302B     sha256:fc5812428ac05db13815379a1e08657f76d8906d9cd56fce6d1d161a10fdca80
 Image Signatures:      
                Name:     sha256:ce3878bf368075638dcd5cce8770443dcbc29f8d9bbe30029e65d34cbf31dc33@c902aa576b9214ded37ed4e89afebbcf
                Type:     atomic
                Status:     Unverified
 Image Created:          11 days ago
 Author:               
 Arch:               amd64
 Command:          httpd-foreground
 Working Dir:          /usr/local/apache2
 User:               
 Exposes Ports:          80/tcp
 Docker Labels:          
 Environment:          PATH=/usr/local/apache2/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
                HTTPD_PREFIX=/usr/local/apache2
                HTTPD_VERSION=2.4.38
                HTTPD_SHA256=7dc65857a994c98370dc4334b260101a7a04be60e6e74a5c57a6dee1bc8f394a
                HTTPD_PATCHES=
                APACHE_DIST_URLS=https://www.apache.org/dyn/closer.cgi?action=download&filename=      https://www-us.apache.org/dist/      https://www.apache.org/dist/      https://archive.apache.org/dist/

Now lets verified our image and save it in OCP:

[root@master01 ~]# oc adm verify-image-signature sha256:ce3878bf368075638dcd5cce8770443dcbc29f8d9bbe30029e65d34cbf31dc33 --expected-identity docker-registry.default.svc:5000/django/httpd:latest --public-key /etc/pki/containers/key.pub --save
 image "sha256:ce3878bf368075638dcd5cce8770443dcbc29f8d9bbe30029e65d34cbf31dc33" identity is now confirmed (signed by GPG key "4ABFBCBA687C588F

Now let re-inspect out the imageStream, and it should shown ‘ Status: Verified `:

[root@master01 pem]#  oc describe istag httpd:latest -n django
 Image Name:          sha256:ce3878bf368075638dcd5cce8770443dcbc29f8d9bbe30029e65d34cbf31dc33
 Docker Image:          docker-registry.default.svc:5000/django/httpd@sha256:ce3878bf368075638dcd5cce8770443dcbc29f8d9bbe30029e65d34cbf31dc33
 Name:               sha256:ce3878bf368075638dcd5cce8770443dcbc29f8d9bbe30029e65d34cbf31dc33
 Created:          34 minutes ago
 Annotations:          image.openshift.io/dockerLayersOrder=ascending
                image.openshift.io/manifestBlobStored=true
                openshift.io/image.managed=true
 Image Size:          49.39MB in 5 layers
 Layers:               22.5MB     sha256:6ae821421a7debccb4151f7a50dc8ec0317674429bec0f275402d697047a8e96
                153B     sha256:0ceda4df88c8d23a0cd3c077725e3ac77c63786eca7bca89051e3cc2c7400aae
                10.33MB     sha256:24f08eb4db686b2136baac67538688d99064dc5a4a6a3b9ec821b7e4af71d6c3
                16.55MB     sha256:ddf4fc3180816e62e990fdb516292a3d4c20faf53e044d722848e5f0bdcc7d19
                302B     sha256:fc5812428ac05db13815379a1e08657f76d8906d9cd56fce6d1d161a10fdca80
 Image Signatures:      
                Name:          sha256:ce3878bf368075638dcd5cce8770443dcbc29f8d9bbe30029e65d34cbf31dc33@c902aa576b9214ded37ed4e89afebbcf
                Type:          atomic
                Status:          Verified
                Issued By:     4ABFBCBA687C588F
                :          Signature is Trusted (verified by user "ocpadmin" on 2019-02-24 19:07:21 +0800 +08)
                :          Signature is ForImage ( on 2019-02-24 19:07:21 +0800 +08)
 Image Created:          11 days ago
 Author:               
 Arch:               amd64
 Command:          httpd-foreground
 Working Dir:          /usr/local/apache2
 User:               
 Exposes Ports:          80/tcp
 Docker Labels:          
 Environment:          PATH=/usr/local/apache2/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
                HTTPD_PREFIX=/usr/local/apache2
                HTTPD_VERSION=2.4.38
                HTTPD_SHA256=7dc65857a994c98370dc4334b260101a7a04be60e6e74a5c57a6dee1bc8f394a
                HTTPD_PATCHES=
                APACHE_DIST_URLS=https://www.apache.org/dyn/closer.cgi?action=download&filename=      https://www-us.apache.org/dist/      https://www.apache.org/dist/      https://archive.apache.org/dist/

6. Docker GPG check pull test

Signed image:

[root@master01 ~]# atomic trust show
 (default)                         accept                               
 docker-registry-default.apps.bytewise.com.my signed [email protected]              
 docker-registry.default.svc:5000    signed [email protected]              
 quay.bytewise.com.my                accept  
 [root@master01 ~]# docker pull docker-registry.default.svc:5000/openshift/django-psql-example:latest
 Trying to pull repository docker-registry.default.svc:5000/openshift/django-psql-example … sha256:1331148170ce4c0c7befc07d038560a7c01ed3d19d1b26f2e12c731cfe268157: 
 Pulling from docker-registry.default.svc:5000/openshift/django-psql-example23113ae36f8e: Already exists d134b18b98b0: Already exists e9c030a1a5e3: 
 Pull complete 784f9bf04822: Pull complete 8d1fafd69d24: 
 Pull complete 29d3ba629150: Pull complete Digest: sha256:1331148170ce4c0c7befc07d038560a7c01ed3d19d1b26f2e12c731cfe268157Status: 
 Downloaded newer image for docker-registry.default.svc:5000/openshift/django-psql-example:latest

Unsigned image:

[root@master01 ~]# atomic trust show
 (default)                         accept                               
 docker-registry-default.apps.bytewise.com.my signed [email protected]              
 docker-registry.default.svc:5000    signed [email protected]              
 quay.bytewise.com.my                accept  
 [root@master01 ~]# docker pull docker-registry.default.svc:5000/openshift/hello-world:latestTrying to pull repository docker-registry.default.svc:5000/openshift/hello-world … docker-registry.default.svc:5000/openshift/hello-world:latest isn't allowed: A signature was required, but no signature exists

Test with wrong gpg public key:

[root@worker02 ~]# atomic trust show
 (default)                         reject                               
 docker-registry.default.svc:5000    signed [email protected] 
 [root@worker02 ~]# docker pull docker-registry.default.svc:5000/httpd/httpd:latest
 Trying to pull repository docker-registry.default.svc:5000/httpd/httpd … 
 docker-registry.default.svc:5000/httpd/httpd:latest isn't allowed: Invalid GPG signature: gpgme.Signature{Summary:128, Fingerprint:"4ABFBCBA687C588F", Status:gpgme.Error{err:0x7000009}, Timestamp:time.Time{wall:0x0, ext:63686601828, loc:(
time.Location)(0x2560640)}, ExpTimestamp:time.Time{wall:0x0, ext:62135596800, loc:(
time.Location)(0x2560640)}, WrongKeyUsage:false, PKATrust:0x0, ChainModel:false, Validity:0, ValidityReason:error(nil), PubkeyAlgo:1, HashAlgo:2

CONCLUSION

  1. Above configuration are true for docker run-time environment, while atomic or cri-o as an example will honour policy.json by default.

While above is manual steps, docker should able to block unverified image automatically via policy.json with signature verification sets to true.

  1. If we do not want to configure docker signature check and only using OCP objects, the only way that possible is to check image signature via  oc adm verify-image-signature status and annnotate the image with  images.openshift.io/deny-execution: true via pipeline.

  2. There is one issue/RFE created in upstream to automatically check image signature during deployment:

OpenShift should automatically verify signatures on images during pod deployments. · Issue #21689 · openshift/origin · G…

Muhammad Aizuddin Zali

Muhammad Aizuddin Zali

RHCA | AppDev & Platform Consultant | DevSecOps


Note

Disclaimer: The views expressed and the content shared in all published articles on this website are solely those of the respective authors, and they do not necessarily reflect the views of the author’s employer or the techbeatly platform. We strive to ensure the accuracy and validity of the content published on our website. However, we cannot guarantee the absolute correctness or completeness of the information provided. It is the responsibility of the readers and users of this website to verify the accuracy and appropriateness of any information or opinions expressed within the articles. If you come across any content that you believe to be incorrect or invalid, please contact us immediately so that we can address the issue promptly.

Tags :
Share :

Related Posts

OpenShift Cluster – How to Drain or Evacuate a Node for Maintenance

OpenShift Cluster – How to Drain or Evacuate a Node for Maintenance

Image : www.oemoffhighway.com As we know OpenShift clusters are bundled with multiple compute nodes, master nodes, infra nodes etc, it’s not a big …

How to find the pod details from container in OpenShift

When we have issue with container and we are not sure which pod the container belongs to; we can find the pods details as below.

How to Create, Increase or Decrease Project Quota in OpenShift

How to Create, Increase or Decrease Project Quota in OpenShift

Image Credit : https://www.joc.com Usually application owner or project owner will specify the quota settings (Memory and CPU) during project …