INTRODUCTION
Just like an RPM signing, container image singing can be used to verify authenticity of a container image created by users/developers throughout image life-cycle.
The concept of verification are similar to RPM gpgcheck verification.
Refer here for containerized image signing and scanning operator framework produced by Red Hat container-cop, automatically by requesting the signing/scanning service.
As for understanding purposes, steps in this blog are manual step.
Contents
hide
CONFIGURATION
Blog Prerequisites
While OpenShift can have varieties of container run-time option. This blog will focus for below environment. The selection was due to nature that docker by default not like other run-time will not honor policy.json out-of-the-box.
- OCP running on full RHEL (7.5)
- Docker container run-time (not CRI-O or others)
- OCP 3.11 (Tested on)
Configuration Steps
We going to use master01 for signing node, while for production purpose, it is a best practise to have particular node for signing and kept very secure.
Overview of steps will be taken
- Create GPG Key Pair.
- Enable V2 endpoint for OCP registry.
- Configure atomic trust.
- Enforce Docker signature verification.
- Sign, Push and Verify image from OCP.
- Docker GPG check pull test.
1. Create GPG Key Pair
Existing GPG keypair can be used for this activity, in case we dont have any keypair yet.
[root@master01 ~]# gpg2 --gen-key
List generated keypair:
[root@master01 ~]# gpg --list-secret-keys
/root/.gnupg/secring.gpg
sec 2048R/687C588F 2019-02-24
uid OCP Image Signer (Key for OCP Image) [email protected]
ssb 2048R/A31CA531 2019-02-24
[root@master01 ~]# gpg --list-key
/root/.gnupg/pubring.gpg
pub 2048R/687C588F 2019-02-24
uid OCP Image Signer (Key for OCP Image) [email protected]
sub 2048R/A31CA531 2019-02-24
[root@master01 ~]# gpg --fingerprint
/root/.gnupg/pubring.gpg
pub 2048R/687C588F 2019-02-24
Key fingerprint = 2AA9 DA41 1D89 A57D 9353 6F5B 4ABF BCBA 687C 588F
uid OCP Image Signer (Key for OCP Image) [email protected]
sub 2048R/A31CA531 2019-02-24
[root@master01 ~]#
On all nodes, ensure below public key content exported and exists. Copy /etc/pki/container/key.pub to all OPC nodes. This will act as verification key against signer key, just like RPM signing.
[root@master01 ~]# mkdir -p /etc/pki/containers/
[root@master01 ~]# gpg2 --armor --export --output /etc/pki/containers/key.pub [email protected]
2. Enable V2 endpoint for OCP registry
Let`s freeze the automatic rollout for docker-registry.
[root@master01 ~]# oc rollout pause deploymentconfig docker-registry -n default
Set registry environment to enable V2 schema.
[root@master01 ~]# oc set env dc/docker-registry REGISTRY_MIDDLEWARE_REPOSITORY_OPENSHIFT_ACCEPTSCHEMA2=true
Resume docker-registry rollout
[root@master01 ~]# oc rollout resume dc/docker-registry -n default
deploymentconfig.apps.openshift.io/docker-registry resumed
[root@master01 ~]# oc rollout latest dc/docker-registry -n default
deploymentconfig.apps.openshift.io/docker-registry rolled out
Now lets see the header (expecting 401 return since no token provided) and look for `X-Registry-Supports-Signatures: 1`:
[root@master01 ~]# curl -kv https://docker-registry.default.svc:5000/v2/
About to connect() to docker-registry.default.svc port 5000 (#0)
Trying 10.51.1.99…
Connected to docker-registry.default.svc (10.51.1.99) port 5000 (#0)
Initializing NSS with certpath: sql:/etc/pki/nssdb
skipping SSL peer certificate verification
SSL connection using TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256
Server certificate:
subject: CN=10.51.1.99
start date: Feb 12 14:17:40 2019 GMT
expire date: Feb 11 14:17:41 2021 GMT
common name: 10.51.1.99
issuer: CN=openshift-signer@1549980226
GET /v2/ HTTP/1.1
User-Agent: curl/7.29.0
Host: docker-registry.default.svc:5000
Accept: /
< HTTP/1.1 401 Unauthorized
< Content-Type: application/json; charset=utf-8
< Docker-Distribution-Api-Version: registry/2.0
< Www-Authenticate: Bearer realm="https://docker-registry.default.svc:5000/openshift/token"
< X-Registry-Supports-Signatures: 1
< Date: Sun, 24 Feb 2019 13:33:45 GMT
< Content-Length: 87
<
{"errors":[{"code":"UNAUTHORIZED","message":"authentication required","detail":null}]}
Connection #0 to host docker-registry.default.svc left intact
[root@master01 ~]#
Next we going to enable and configure atomic trust, hence only allowed registry and/or signed image allowed to be pulled into the node.
3. Configure atomic trust
This step can be automated via Ansible or other automation mean. In this example we are using master01 as configuration node and copy configs to all nodes.
As an example of Ansible play:
- hosts: OSEv3
tasks:
name: Create /etc/pki/containers directory
file: path=/etc/pki/containers state=directory
name: Create /etc/containers/registries.d directory
file: path=/etc/containers/registries.d state=directory
name: Copy trusted public keys
copy: src=/etc/pki/containers/ dest=/etc/pki/containers
name: Copy container trust policy
copy: src=/etc/containers/policy.json
dest=/etc/containers/policy.json
name: Copy signature server configuration files
copy: src=/etc/containers/registries.d/
dest=/etc/containers/registries.d/
As default policy, reject all:
[root@master01 ~]# atomic trust default reject
Now we add our internal OCP registry(or any registry required to pull/push image):
[root@master01 ~]# atomic trust add docker-registry-default.apps.bytewise.com.my --pubkeys /etc/pki/containers/key.pub
[root@master01 ~]# atomic trust add docker-registry.default.svc:5000 --pubkeys /etc/pki/containers/key.pub
[root@master01 ~]# atomic trust add quay.bytewise.com.my -t insecureAcceptAnything
Let list our atomic trust:
[root@master01 ~]# atomic trust show
(default) reject
docker-registry-default.apps.bytewise.com.my signed [email protected]
docker-registry.default.svc:5000 signed [email protected]
quay.bytewise.com.my accept
/etc/containers/policy.json looks like this:
{
"default": [
{
"type": "reject"
}
],
"transports": {
"docker": {
"docker-registry-default.apps.bytewise.com.my": [
{
"keyType": "GPGKeys",
"type": "signedBy",
"keyData": "#########"
}
],
"quay.bytewise.com.my": [
{
"type": "insecureAcceptAnything"
}
],
"docker-registry.default.svc:5000": [
{
"keyType": "GPGKeys",
"type": "signedBy",
"keyData": "#########"
}
]
},
"docker-daemon": {
"": [
{
"type": "insecureAcceptAnything"
}
]
}
}
}
4. Enforce Docker signature verification
To enforce docker to verify image signature (on all OCP nodes):
NOTE: Ensure no duplicate line for ‘”signature-verification”: true’ in /etc/sysconfig/docker, or you may use configuration here as well instead of daemon.json.
[root@master01 ~]# cat /etc/docker/daemon.json
{
"log-driver": "journald",
"signature-verification": true
}
[root@master01 ~]# systemctl restart docker
5. Sign, Push and Verify image from OCP
Now we going to push one test image (note debug is on for testing purposes):
[root@master01 ~]# atomic --debug push --type atomic --sign-by [email protected] docker-registry.default.svc:5000/django/httpd:latest
Once the image pushed, inspect the imageStream, where you can see the `Status: Unverified`:
[root@master01 sigstore]# oc describe istag httpd:latest
Image Name: sha256:ce3878bf368075638dcd5cce8770443dcbc29f8d9bbe30029e65d34cbf31dc33
Docker Image: docker-registry.default.svc:5000/django/httpd@sha256:ce3878bf368075638dcd5cce8770443dcbc29f8d9bbe30029e65d34cbf31dc33
Name: sha256:ce3878bf368075638dcd5cce8770443dcbc29f8d9bbe30029e65d34cbf31dc33
Created: 29 minutes ago
Annotations: image.openshift.io/dockerLayersOrder=ascending
image.openshift.io/manifestBlobStored=true
openshift.io/image.managed=true
Image Size: 49.39MB in 5 layers
Layers: 22.5MB sha256:6ae821421a7debccb4151f7a50dc8ec0317674429bec0f275402d697047a8e96
153B sha256:0ceda4df88c8d23a0cd3c077725e3ac77c63786eca7bca89051e3cc2c7400aae
10.33MB sha256:24f08eb4db686b2136baac67538688d99064dc5a4a6a3b9ec821b7e4af71d6c3
16.55MB sha256:ddf4fc3180816e62e990fdb516292a3d4c20faf53e044d722848e5f0bdcc7d19
302B sha256:fc5812428ac05db13815379a1e08657f76d8906d9cd56fce6d1d161a10fdca80
Image Signatures:
Name: sha256:ce3878bf368075638dcd5cce8770443dcbc29f8d9bbe30029e65d34cbf31dc33@c902aa576b9214ded37ed4e89afebbcf
Type: atomic
Status: Unverified
Image Created: 11 days ago
Author:
Arch: amd64
Command: httpd-foreground
Working Dir: /usr/local/apache2
User:
Exposes Ports: 80/tcp
Docker Labels:
Environment: PATH=/usr/local/apache2/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
HTTPD_PREFIX=/usr/local/apache2
HTTPD_VERSION=2.4.38
HTTPD_SHA256=7dc65857a994c98370dc4334b260101a7a04be60e6e74a5c57a6dee1bc8f394a
HTTPD_PATCHES=
APACHE_DIST_URLS=https://www.apache.org/dyn/closer.cgi?action=download&filename= https://www-us.apache.org/dist/ https://www.apache.org/dist/ https://archive.apache.org/dist/
Now lets verified our image and save it in OCP:
[root@master01 ~]# oc adm verify-image-signature sha256:ce3878bf368075638dcd5cce8770443dcbc29f8d9bbe30029e65d34cbf31dc33 --expected-identity docker-registry.default.svc:5000/django/httpd:latest --public-key /etc/pki/containers/key.pub --save
image "sha256:ce3878bf368075638dcd5cce8770443dcbc29f8d9bbe30029e65d34cbf31dc33" identity is now confirmed (signed by GPG key "4ABFBCBA687C588F
Now let re-inspect out the imageStream, and it should shown ‘Status: Verified`:
[root@master01 pem]# oc describe istag httpd:latest -n django
Image Name: sha256:ce3878bf368075638dcd5cce8770443dcbc29f8d9bbe30029e65d34cbf31dc33
Docker Image: docker-registry.default.svc:5000/django/httpd@sha256:ce3878bf368075638dcd5cce8770443dcbc29f8d9bbe30029e65d34cbf31dc33
Name: sha256:ce3878bf368075638dcd5cce8770443dcbc29f8d9bbe30029e65d34cbf31dc33
Created: 34 minutes ago
Annotations: image.openshift.io/dockerLayersOrder=ascending
image.openshift.io/manifestBlobStored=true
openshift.io/image.managed=true
Image Size: 49.39MB in 5 layers
Layers: 22.5MB sha256:6ae821421a7debccb4151f7a50dc8ec0317674429bec0f275402d697047a8e96
153B sha256:0ceda4df88c8d23a0cd3c077725e3ac77c63786eca7bca89051e3cc2c7400aae
10.33MB sha256:24f08eb4db686b2136baac67538688d99064dc5a4a6a3b9ec821b7e4af71d6c3
16.55MB sha256:ddf4fc3180816e62e990fdb516292a3d4c20faf53e044d722848e5f0bdcc7d19
302B sha256:fc5812428ac05db13815379a1e08657f76d8906d9cd56fce6d1d161a10fdca80
Image Signatures:
Name: sha256:ce3878bf368075638dcd5cce8770443dcbc29f8d9bbe30029e65d34cbf31dc33@c902aa576b9214ded37ed4e89afebbcf
Type: atomic
Status: Verified
Issued By: 4ABFBCBA687C588F
: Signature is Trusted (verified by user "ocpadmin" on 2019-02-24 19:07:21 +0800 +08)
: Signature is ForImage ( on 2019-02-24 19:07:21 +0800 +08)
Image Created: 11 days ago
Author:
Arch: amd64
Command: httpd-foreground
Working Dir: /usr/local/apache2
User:
Exposes Ports: 80/tcp
Docker Labels:
Environment: PATH=/usr/local/apache2/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
HTTPD_PREFIX=/usr/local/apache2
HTTPD_VERSION=2.4.38
HTTPD_SHA256=7dc65857a994c98370dc4334b260101a7a04be60e6e74a5c57a6dee1bc8f394a
HTTPD_PATCHES=
APACHE_DIST_URLS=https://www.apache.org/dyn/closer.cgi?action=download&filename= https://www-us.apache.org/dist/ https://www.apache.org/dist/ https://archive.apache.org/dist/
6. Docker GPG check pull test
Signed image:
[root@master01 ~]# atomic trust show
(default) accept
docker-registry-default.apps.bytewise.com.my signed [email protected]
docker-registry.default.svc:5000 signed [email protected]
quay.bytewise.com.my accept
[root@master01 ~]# docker pull docker-registry.default.svc:5000/openshift/django-psql-example:latest
Trying to pull repository docker-registry.default.svc:5000/openshift/django-psql-example … sha256:1331148170ce4c0c7befc07d038560a7c01ed3d19d1b26f2e12c731cfe268157:
Pulling from docker-registry.default.svc:5000/openshift/django-psql-example23113ae36f8e: Already exists d134b18b98b0: Already exists e9c030a1a5e3:
Pull complete 784f9bf04822: Pull complete 8d1fafd69d24:
Pull complete 29d3ba629150: Pull complete Digest: sha256:1331148170ce4c0c7befc07d038560a7c01ed3d19d1b26f2e12c731cfe268157Status:
Downloaded newer image for docker-registry.default.svc:5000/openshift/django-psql-example:latest
Unsigned image:
[root@master01 ~]# atomic trust show
(default) accept
docker-registry-default.apps.bytewise.com.my signed [email protected]
docker-registry.default.svc:5000 signed [email protected]
quay.bytewise.com.my accept
[root@master01 ~]# docker pull docker-registry.default.svc:5000/openshift/hello-world:latestTrying to pull repository docker-registry.default.svc:5000/openshift/hello-world … docker-registry.default.svc:5000/openshift/hello-world:latest isn't allowed: A signature was required, but no signature exists
Test with wrong gpg public key:
[root@worker02 ~]# atomic trust show
(default) reject
docker-registry.default.svc:5000 signed [email protected]
[root@worker02 ~]# docker pull docker-registry.default.svc:5000/httpd/httpd:latest
Trying to pull repository docker-registry.default.svc:5000/httpd/httpd …
docker-registry.default.svc:5000/httpd/httpd:latest isn't allowed: Invalid GPG signature: gpgme.Signature{Summary:128, Fingerprint:"4ABFBCBA687C588F", Status:gpgme.Error{err:0x7000009}, Timestamp:time.Time{wall:0x0, ext:63686601828, loc:(time.Location)(0x2560640)}, ExpTimestamp:time.Time{wall:0x0, ext:62135596800, loc:(time.Location)(0x2560640)}, WrongKeyUsage:false, PKATrust:0x0, ChainModel:false, Validity:0, ValidityReason:error(nil), PubkeyAlgo:1, HashAlgo:2
CONCLUSION
1. Above configuration are true for docker run-time environment, while atomic or cri-o as an example will honour policy.json by default.
2. While above is manual steps, docker should able to block unverified image automatically via policy.json with signature verification sets to true.
3. If we do not want to configure docker signature check and only using OCP objects, the only way that possible is to check image signature via `oc adm verify-image-signature` status and annnotate the image with `images.openshift.io/deny-execution: true` via pipeline.
4. There is one issue/RFE created in upstream to automatically check image signature during deployment:
Disclaimer:
The views expressed and the content shared in all published articles on this website are solely those of the respective authors, and they do not necessarily reflect the views of the author’s employer or the techbeatly platform. We strive to ensure the accuracy and validity of the content published on our website. However, we cannot guarantee the absolute correctness or completeness of the information provided. It is the responsibility of the readers and users of this website to verify the accuracy and appropriateness of any information or opinions expressed within the articles. If you come across any content that you believe to be incorrect or invalid, please contact us immediately so that we can address the issue promptly.
Tags:
Latest posts
Comments
Leave a Reply