NOTE: This article focused on technical feasibility and features. For production usage this require detail implementation discussion.
Objective.
We want to ship containers and apiserver audit log, objects and K8s metrics (not OpenShift monitoring metrics) to external Splunk via HEC.
Prereq.
- Helm3 (Latest, and avoid tiller security issues)
- Splunk (we use free Splunk with internal Splunk self signed certificate)
Steps.
NOTE: Ensure you are logged in to cluster as cluster-admin via oc client prior run below steps.
1. Go and get free splunk and we are going to install it.
2. We have the RPM ready and execute the installation:
#> ls -rlt splunk*
-rw-r--r--. 1 root root 510124954 Feb 13 00:48 splunk-8.0.2-a7f645ddaf91-linux-2.6-x86_64.rpm
#> rpm -ivh splunk-8.0.2-a7f645ddaf91-linux-2.6-x86_64.rpm
warning: splunk-8.0.2-a7f645ddaf91-linux-2.6-x86_64.rpm: Header V4 RSA/SHA256 Signature, key ID b3cd4420: NOKEY
Verifying... ################################# [100%]
Preparing... ################################# [100%]
Updating / installing...
1:splunk-8.0.2-a7f645ddaf91 ################################# [100%]
complete
3. Start splunk server:
#> cd /opt/splunk/bin
#> ./splunk start --accept-license
This appears to be your first time running this version of Splunk.
Splunk software must create an administrator account during startup. Otherwise, you cannot log in.
Create credentials for the administrator account.
Characters do not appear on the screen when you type in credentials.
Please enter an administrator username: admin
Password must contain at least:
* 8 total printable ASCII character(s).
Please enter a new password:
Please confirm new password:
Copying '/opt/splunk/etc/openldap/ldap.conf.default' to '/opt/splunk/etc/openldap/ldap.conf'.
Generating RSA private key, 2048 bit long modulus
........................................................................+++++
.....................................................................+++++
e is 65537 (0x10001)
writing RSA key
Generating RSA private key, 2048 bit long modulus
.+++++
..+++++
e is 65537 (0x10001)
writing RSA key
Moving '/opt/splunk/share/splunk/search_mrsparkle/modules.new' to '/opt/splunk/share/splunk/search_mrsparkle/modules'.
Splunk> Another one.
Checking prerequisites...
Checking http port [8000]: open
Checking mgmt port [8089]: open
Checking appserver port [127.0.0.1:8065]: open
Checking kvstore port [8191]: open
Checking configuration... Done.
Creating: /opt/splunk/var/lib/splunk
Creating: /opt/splunk/var/run/splunk
Creating: /opt/splunk/var/run/splunk/appserver/i18n
Creating: /opt/splunk/var/run/splunk/appserver/modules/static/css
Creating: /opt/splunk/var/run/splunk/upload
Creating: /opt/splunk/var/run/splunk/search_telemetry
Creating: /opt/splunk/var/spool/splunk
Creating: /opt/splunk/var/spool/dirmoncache
Creating: /opt/splunk/var/lib/splunk/authDb
Creating: /opt/splunk/var/lib/splunk/hashDb
New certs have been generated in '/opt/splunk/etc/auth'.
Checking critical directories... Done
Checking indexes...
Validated: _audit _internal _introspection _metrics _telemetry _thefishbucket history main summary
Done
Checking filesystem compatibility... Done
Checking conf files for problems...
Done
Checking default conf files for edits...
Validating installed files against hashes from '/opt/splunk/splunk-8.0.2-a7f645ddaf91-linux-2.6-x86_64-manifest'
All installed files intact.
Done
All preliminary checks passed.
Starting splunk server daemon (splunkd)...
Generating a RSA private key
....+++++
..+++++
writing new private key to 'privKeySecure.pem'
-----
Signature ok
subject=/CN=bastion.local.bytewise.my/O=SplunkUser
Getting CA Private Key
writing RSA key
Done
[ OK ]
Waiting for web server at # to be available.... Done
If you get stuck, we're here to help.
Look for answers here: http://docs.splunk.com
The Splunk web interface is at #
Do not forget to open up a firewall rules for 8000/tcp 8089/tcp 8191/tcp 8088/tcp. Or any other custom port you configured.
4. Install helm as Splunk HEC will depending on helm to deploy the HEC stack on OpenShift.
5. Next create a project for splunk HEC called “splunk-connect”.
#> oc new-project splunk-connect
#> oc project splunk-connect6. Create new index as per below list. Settings > Indexes > New Index
- openshift (events)
- openshiftlab1_logging (events)
- openshiftlab1_metrics (metrics)
- openshiftlab1_object (events)
7. Enable HEC (HTTP Event Collector), Settings > Data Inputs > HTTP Event Collector > Global Settings > Default Index as “Default” > Save.
8. Create new HEC token, Settings > Data inputs > HTTP Event Collector > New Token > Name as “openshiftlab1” > Next (Input Settings, add allowed indexes like below” > Review > Submit. Note the Token Value, we going to use this for next step.
9. Now prepare the splunk HEC values.yaml to be use as config file, replace token with the value above:
#> cat splunk-hec.yaml
global:
logLevel: info
journalLogPath: /run/log/journal
splunk:
hec:
host: bastion.local.bytewise.my
port: 8088
token: xxxxxxxxxxxxxxxxxx
protocol: https
indexName: openshift
insecureSSL: true
#clientCert:
#clientKey:
#caFile:
kubernetes:
clusterName: "openshiftlab"
openshift: true
splunk-kubernetes-logging:
enabled: true
logLevel: debug
splunk:
hec:
host: bastion.local.bytewise.my
port: 8088
token: xxxxxxxxxxxxxxxxxx
protocol: https
indexName: openshiftlab1_logging
insecureSSL: true
#clientCert:
#clientKey:
#caFile:
containers:
logFormatType: cri
logs:
kube-audit:
from:
file:
path: /var/log/kube-apiserver/audit.log
splunk-kubernetes-metrics:
enabled: true
splunk:
hec:
host: bastion.local.bytewise.my
port: 8088
token: xxxxxxxxxxxxxxxxxx
protocol: https
indexName: openshiftlab1_metrics
insecureSSL: true
#clientCert:
#clientKey:
#caFile:
kubernetes:
openshift: true
splunk-kubernetes-objects:
enabled: true
kubernetes:
openshift: true
splunk:
hec:
host: bastion.local.bytewise.my
port: 8088
token: xxxxxxxxxxxxxxxxxx
protocol: https
insecureSSL: true
indexName: openshiftlab1_objects
#clientCert:
#clientKey:
#caFile: 10. At this stage we are now ready to deploy Splunk HEC via Helm3 cli:
#> helm install splunk-kubernetes-logging -f splunk-hec.yaml https://github.com/splunk/splunk-connect-for-kubernetes/releases/download/1.4.0/splunk-connect-for-kubernetes-1.4.0.tgz
#> oc get pods
NAME READY STATUS RESTARTS AGE
splunk-kubernetes-logging-76l9k 1/1 Running 0 68s
splunk-kubernetes-logging-gmjrm 1/1 Running 0 68s
splunk-kubernetes-logging-sbtx7 1/1 Running 0 68s
splunk-kubernetes-logging-splunk-kubernetes-metrics-2vtbm 1/1 Running 0 68s
splunk-kubernetes-logging-splunk-kubernetes-metrics-6jjmh 1/1 Running 0 69s
splunk-kubernetes-logging-splunk-kubernetes-metrics-agg-662qp7q 1/1 Running 0 69s
splunk-kubernetes-logging-splunk-kubernetes-metrics-m6qwj 1/1 Running 0 69s
splunk-kubernetes-logging-splunk-kubernetes-metrics-mdr65 1/1 Running 0 68s
splunk-kubernetes-logging-splunk-kubernetes-metrics-pg2zr 1/1 Running 0 69s
splunk-kubernetes-logging-splunk-kubernetes-objects-7b9864jw9wh 1/1 Running 0 69s
splunk-kubernetes-logging-xgmh8 1/1 Running 0 68s
splunk-kubernetes-logging-zvrzn 1/1 Running 0 69s
IMPORTANT: Ensure all Splunk deployment service account has Privileged SCC. e.g
# > oc adm policy add-scc-to-user privileged -z splunk-hec-splunk-kubernetes-logging -z splunk-hec-splunk-kubernetes-objects
securitycontextconstraints.security.openshift.io/privileged added to: ["system:serviceaccount:splunk-k8s:splunk-hec-splunk-kubernetes-logging" "system:serviceaccount:splunk-k8s:splunk-hec-splunk-kubernetes-objects"]
11. Ensure all the collector pod running normally.
12. Do index and metric search from UI and we can see data ingested and indexed properly on Splunk:
Summary
Having centralized and aggregated SIEM is essential as part of organization infrastructure defends. Splunk is one of the well-known SIEM solution. We just go trough how OpenShift 4 via Splunk HEC can pipe logs, metrics and some k8s object off cluster to Splunk.