KUBERNETES CKA CKS¶

REPO PER ESERCITAZIONI¶

CKA

CKAD

https://kodekloud.com/courses/kubernetes-certification-course/lectures/6743640

https://kodekloud.com//courses/certified-kubernetes-administrator-with-practice-tests/lectures/9808147

Be fast with Kubectl 1.19 (CKS|CKA|CKAD) | FAUN

Creare risorse velocemente:
Cat > nome.yml
Copia dalla documentazione “pod overview”
To copy and paste limited amounts of text (1−2 lines) please use
For Windows: Ctrl+Insert to copy and Shift+Insert to paste.
For Linux: select text for copy and middle button for paste (or both left and right

Vi nome.yml customizza

INSTALLAZIONE¶

KUBEADM
https://github.com/tuxerrante/kubernetes-utils/tree/main/local-install/multipass

Per usare il driver KVM
https://minikube.sigs.k8s.io/docs/reference/drivers/kvm2/
# yum install qemu-kvm libvirt libvirt-python libguestfs-tools virt-install
virt-host-validate

grep -E --color 'vmx|svm' /proc/cpuinfo

Altrimenti Docker
https://docs.docker.com/engine/install/centos/
sudo yum install docker-ce docker-ce-cli containerd.io --nobest
minikube config set driver docker

cat \<\<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
EOF

dnf install -y kubectl

# install minikube
echo downloading minikube, check version
curl -Lo minikube https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64

chmod +x minikube
# mv minikube /usr/local/bin

echo at this point, reboot your Fedora Workstation.
curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt

AVVIO¶

https://kubernetes.io/docs/setup/learning-environment/minikube/#managing-your-cluster

$ minikube start --memory 4096 --dry-run
kubectl get all --all-namespaces

ip address show docker0
kubectl cluster info

autocomplete
echo 'source \<(kubectl completion bash)' >>~/.bashrc

export MINIKUBE_ACTIVE_DOCKERD="minikube"
Oppure
minikube docker-env

minikube stop

Basic Commands:
create Create a resource from a file or from stdin.
expose Take a replication controller, service, deployment or pod and expose it as a new Kubernetes Service
run Run a particular image on the cluster
set Set specific features on objects

explain Documentation of resources
get Display one or many resources
edit Edit a resource on the server
delete Delete resources by filenames, stdin, resources and names, or by resources and label selector

Deploy Commands:
rollout Manage the rollout of a resource
scale Set a new size for a Deployment, ReplicaSet or Replication Controller
autoscale Auto-scale a Deployment, ReplicaSet, or ReplicationController

Cluster Management Commands:
certificate Modify certificate resources.
cluster-info Display cluster info
top Display Resource (CPU/Memory/Storage) usage.
cordon Mark node as unschedulable
uncordon Mark node as schedulable
drain Drain node in preparation for maintenance
taint Update the taints on one or more nodes

Troubleshooting and Debugging Commands:
describe Show details of a specific resource or group of resources
logs Print the logs for a container in a pod
attach Attach to a running container
exec Execute a command in a container
port-forward Forward one or more local ports to a pod
proxy Run a proxy to the Kubernetes API server
cp Copy files and directories to and from containers.
auth Inspect authorization

Advanced Commands:
diff Diff live version against would-be applied version
apply Apply a configuration to a resource by filename or stdin
patch Update field(s) of a resource using strategic merge patch
replace Replace a resource by filename or stdin
wait Experimental: Wait for a specific condition on one or many resources.
convert Convert config files between different API versions
kustomize Build a kustomization target from a directory or a remote url.

Settings Commands:
label Update the labels on a resource
annotate Update the annotations on a resource
completion Output shell completion code for the specified shell (bash or zsh)

CLUSTER INFO¶

kubectl cluster-info |head -n1 |egrep -o "([0-9]*\.){3}[0-9]+"
172.17.0.14

SCHEDULER¶

The scheduler decides where to put the pod, Kubelet generated that pod on the node

sudo less /etc/kubernetes/manifests/kube-scheduler.yaml

KUBELET¶

PODS¶

The containers are incapsulated in pods.

kubectl run nginx --image=nginx --port=80 --expose=true --dry-run=client --output yaml kubectl run echo --image=gcr.io/google-containers/echoserver:1.10 kubectl get pods -o wideTo create a YAML (declarative config)explain api-resourcesapi-version K delete pod X --grace-period=0 --forcek describe pods	grep -C 10 "Image:" k get po --sort-by=.metadata.nameK describe pods Li descrive tutti

DEPLOYMENT¶

kubectl create deployment nginx --image=nginx:alpine --dry-run --output=yaml

kubectl create deployment nginx --image=nginx:alpine

kubectl expose deployment nginx --port=80 --type=NodePort

kubectl exec nginx-86c57db685-jq796 -- hostname -i

CONTEXT¶

k config use-context ferrari_dev

NODES¶

k describe nodes | grep -i "cpu "

kubectl describe nodes |grep -A 2 -e "^\\s*cpu.*%"
cpu 250m (12%) 33500m (1675%)
memory 2232Mi (15%) 2974Mi (20%)
ephemeral-storage 0 (0%) 0 (0%)
--
cpu 270m (13%) 33500m (1675%)
memory 2754Mi (18%) 3486Mi (23%)
ephemeral-storage 0 (0%) 0 (0%)

alias k_util='kubectl get nodes | grep node | awk '\''{print $1}'\'' | xargs -I {} sh -c '\''echo {} ; kubectl describe node {} | grep Allocated -A 5 | grep -ve Event -ve Allocated -ve percent -ve -- ; echo '\'''

SERVICES¶

Cluster-ip
Node-port
Load balancer
External name

https://kubernetes.io/docs/concepts/services-networking/service/#publishing-services-service-types

You can also use Ingress to expose your Service. Ingress is not a Service type, but it acts as the entry point for your cluster. It lets you consolidate your routing rules into a single resource as it can expose multiple services under the same IP address

CLUSTER-IP¶

A Service receives a Virtual IP address, known as its ClusterIP. This Virtual IP address is used for communicating with the Service and is accessible only from within the cluster.

NODE PORT ¶

in addition to a ClusterIP, a high-port, dynamically picked from the default range 30000-32767, is mapped to the respective Service, from all the worker nodes.

LOAD BALANCER¶

NodePort and ClusterIP are automatically created, and the external load balancer will route to them
The Service is exposed at a static port on each worker node
The Service is exposed externally using the underlying cloud provider's load balancer feature.

Oracle Load Balancing

EXTERNAL IP¶

A Service can be mapped to an ExternalIP address if it can route to one or more of the worker nodes. Traffic that is ingressed into the cluster with the ExternalIP (as destination IP) on the Service port, gets routed to one of the Service endpoints.

apiVersion: v1
kind: Service
metadata:
name: my-service
spec:
selector:
app: MyApp
ports:
- name: http
protocol: TCP
port: 80
targetPort: 9376
externalIPs:
- 80.11.12.10

EXTERNAL NAME¶

Services of type ExternalName map a Service to a DNS name, not to a typical selector such as my-service or cassandra. You specify these Services with the spec.externalName parameter.
Please note that ExternalIPs are not managed by Kubernetes. The cluster administrator has to configure the routing which maps the ExternalIP address to one of the nodes.

apiVersion: v1
kind: Service
metadata:
name: my-service
namespace: prod
spec:
type: ExternalName
externalName: my.database.example.com

INGRESS¶

To manage access to multiple application Services from the external world, administrators can configure a reverse proxy - an ingress, and define rules that target specific Services within the cluster.

An API object that manages external access to the services in a cluster, typically HTTP.
Ingress may provide load balancing, SSL termination and name-based virtual hosting.

decouple the routing rules from the application and centralize the rules management, we can then update our application without worrying about its external access.

TLS (Transport Layer Security)
Name-based virtual hosting
Fanout routing
Load Balancing
Custom rules.

it merely accepts the definitions of traffic routing rules. The ingress is fulfilled by an Ingress Controller, which is a reverse proxy responsible for traffic routing based on rules defined in the Ingress resource.

KONG INGRESS CONTROLLER
Getting started with the Kubernetes Ingress Controller - v1.1.x
Kong/kubernetes-ingress-controller: Kong for Kubernetes: the official Ingress Controller for Kubernetes.
Can you expose your services with an API gateway in Kubernetes?

| alias k=kubectl #### Crea tutte le risorse del controller Kong (namespace, service, ingress ecc..) kubectl create -f https://bit.ly/k4k8s #### Salva IP alla quale è accessibile il servizio kong (load balancer) dall’esterno: export PROXY_IP=\((kubectl get service \-n kong kong-proxy \--no-headers \--output=custom-columns=:.spec.clusterIP) \#\#\#\# CREA SERVIZIO E DEPLOYMENT PER IL CONTAINER ECHO kubectl apply \-f https://bit.ly/echo-service \#\#\#\#\# CREATE AN INGRESS RULE FOR THE API 1.0 OF THE ECHO APP echo " apiVersion: extensions/v1beta1 kind: Ingress metadata: name: demo annotations: kubernetes.io/ingress.class: kong spec: rules: \- http: paths: \- path: /api/1.0 backend: serviceName: echo servicePort: 80 " | kubectl apply \-f \- \#\#\#\#\#\# APPLY A FAKE V2 VERSION OF THE SAME APP "ECHO" echo " apiVersion: v1 kind: Service metadata: labels: app: echo-v2 name: echo-v2 spec: ports: \- port: 8080 name: high protocol: TCP targetPort: 8080 \- port: 80 name: low protocol: TCP targetPort: 8080 selector: app: echo-v2 \--- apiVersion: apps/v1 kind: Deployment metadata: labels: app: echo-v2 name: echo-v2 spec: replicas: 1 selector: matchLabels: app: echo-v2 template: metadata: labels: app: echo-v2 spec: containers: \- image: gcr.io/kubernetes-e2e-test-images/echoserver:2.2 name: echo-v2 ports: \- containerPort: 8080 env: \- name: NODE\_NAME valueFrom: fieldRef: fieldPath: spec.nodeName \- name: POD\_NAME valueFrom: fieldRef: fieldPath: metadata.name \- name: POD\_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace \- name: POD\_IP valueFrom: fieldRef: fieldPath: status.podIP resources: {} " | kubectl apply \-f \- \#\#\# UPDATE AN INGRESS RULE FOR THE API 2.0 OF THE ECHO APP echo " apiVersion: extensions/v1beta1 kind: Ingress metadata: name: demo annotations: kubernetes.io/ingress.class: kong spec: rules: \- http: paths: \- path: /api/1.0 backend: serviceName: echo servicePort: 80 \- path: /api/2.0 backend: serviceName: echo-v2 servicePort: 80 " | kubectl apply \-f \- curl \--insecure \-i \--stderr \- http://\){PROXY_IP}/api/1.0 |grep pod curl --insecure -i --stderr - http://${PROXY\_IP}/api/2.0 |grep pod \# pod name: echo-v2-9b78d55f7-b4rcz \# pod namespace: default \# pod IP: 172.18.0.6 \#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\# \#\#\#\#\# NINJA LEVEL \==\> remove the api version before sending it to the container \#\#\#\#\# https://docs.konghq.com/kubernetes-ingress-controller/1.1.x/guides/using-rewrites/\#rewriting-the-path kubectl patch ingress demo \-p '{"metadata":{"annotations":{"konghq.com/strip-path":"true"}}}' curl \--insecure \--stderr \- http://${PROXY_IP}/api/2.0/login |grep "Request Information" -A3 #Request Information: # client_address=172.18.0.4 # method=GET # real path=/ | | :---- |

KUBE PROXY ¶

watches the API server on the master node for the addition, updates, and removal of Services and endpoints.
on each node, kube-proxy configures iptables rules to capture the traffic for its ClusterIP and forwards it to one of the Service's endpoints.

$ kubectl get daemonsets -n kube-system NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGEkube-proxy 1 1 1 1 1 kubernetes.io/os=linux 6d3h

SERVICE DISCOVERY ¶

Environment Variables¶

As soon as the Pod starts on any worker node, the kubelet daemon running on that node adds a set of environment variables in the Pod for all active Services.
the Pods will not have the environment variables set for Services which are created after the Pods are created

DNS¶

Kubernetes has an add-on for DNS (CoreDNS), which creates a DNS record for each Service and its format is my-svc.my-namespace.svc.cluster.local. Services within the same Namespace find other Services just by their names.

VOLUMES¶

A Volume is essentially a mount point on the container's file system backed by a storage medium. The storage medium, content and access mode are determined by the Volume Type.

A Volume is linked to a Pod and can be shared among the containers of that Pod. Although the Volume has the same life span as the Pod, meaning that it is deleted together with the Pod, the Volume outlives the containers of the Pod - this allows data to be preserved across container restarts.

emptyDir
An empty Volume is created for the Pod as soon as it is scheduled on the worker node. The Volume's life is tightly coupled with the Pod. If the Pod is terminated, the content of emptyDir is deleted forever.
hostPath
With the hostPath Volume Type, we can share a directory between the host and the Pod. If the Pod is terminated, the content of the Volume is still available on the host.
secret
With the secret Volume Type, we can pass sensitive information, such as passwords, to Pods.
configMap
With configMap objects, we can provide configuration data, or shell commands and arguments into a Pod.
persistentVolumeClaim
We can attach a PersistentVolume to a Pod using a persistentVolumeClaim.

PERSISTENT VOLUME¶

A Persistent Volume is a storage abstraction backed by several storage technologies, which could be local to the host where the Pod is deployed with its application container(s), network attached storage, cloud storage, or a distributed storage solution. A Persistent Volume is statically provisioned by the cluster administrator.

PersistentVolumes can be dynamically provisioned based on the StorageClass resource. A StorageClass contains pre-defined provisioners and parameters to create a PersistentVolume. Using PersistentVolumeClaims, a user sends the request for dynamic PV creation, which gets wired to the StorageClass resource.

A PersistentVolumeClaim (PVC)¶

is a request for storage by a user. Users request for PersistentVolume resources based on type, access mode, and size. There are three access modes:

ReadWriteOnce (read-write by a single node),
ReadOnlyMany (read-only by many nodes), and
ReadWriteMany (read-write by many nodes).

Once a suitable PersistentVolume is found, it is bound to a PersistentVolumeClaim.

Once a user finishes its work, the attached PersistentVolumes can be released.
The underlying PersistentVolumes can then be

reclaimed (for an admin to verify and/or aggregate data),
deleted (both data and volume are deleted), or
recycled for future usage (only data is deleted),

based on the configured persistentVolumeReclaimPolicy property.

Dynamic Volume Provisioning¶

Dynamic volume provisioning allows storage volumes to be created on-demand. Without dynamic provisioning, cluster administrators have to manually make calls to their cloud or storage provider to create new storage volumes, and then create PersistentVolume objects to represent them in Kubernetes.

CONFIG MAPS¶

ConfigMaps allow us to decouple the configuration details from the container image.
Using ConfigMaps, we pass configuration data as key-value pairs, which are consumed by Pods or any other system components and controllers, in the form of

environment variables,
sets of commands and arguments, or
volumes.

We can create ConfigMaps from

literal values,
from configuration files,
from one or more files or directories.

...
containers:
- name: myapp-full-container
image: myapp
envFrom:
- configMapRef:
name: full-config-map
…

Load configMap as Volume
...
containers:
- name: myapp-vol-container
image: myapp
volumeMounts:
- name: config-volume
mountPath: /etc/config
volumes:
- name: config-volume
configMap:
name: vol-config-map
…

SECRETS¶

the Secret data is stored as plain text inside etcd, therefore administrators must limit access to the API server and etcd.
However, Secret data can be encrypted at rest while it is stored in etcd, but this feature needs to be enabled at the API server level.

QUOTAS¶

Compute Resource Quota
We can limit the total sum of compute resources (CPU, memory, etc.) that can be requested in a given Namespace.
Storage Resource Quota
We can limit the total sum of storage resources (PersistentVolumeClaims, requests.storage, etc.) that can be requested.
Object Count Quota
We can restrict the number of objects of a given type (pods, ConfigMaps, PersistentVolumeClaims, ReplicationControllers, Services, Secrets, etc.).

An additional resource that helps limit resources allocation to pods and containers in a namespace, is the LimitRange, used in conjunction with the ResourceQuota API resource. A LimitRange can:

Set compute resources usage limits per Pod or Container in a namespace.
Set storage request limits per PersistentVolumeClaim in a namespace.
Set a request to limit ratio for a resource in a namespace.
Set default requests and limits and automatically inject them into Containers' environments at runtime.

AUTOSCALING¶

Horizontal Pod Autoscaler (HPA)
HPA is an algorithm-based controller API resource which automatically adjusts the number of replicas in a ReplicaSet, Deployment or Replication Controller based on CPU utilization.
Vertical Pod Autoscaler (VPA)
VPA automatically sets Container resource requirements (CPU and memory) in a Pod and dynamically adjusts them in runtime, based on historical utilization data, current resource availability and real-time events.
Cluster Autoscaler
Cluster Autoscaler automatically re-sizes the Kubernetes cluster when there are insufficient resources available for new Pods expecting to be scheduled or when there are underutilized nodes in the cluster.

JOB ¶

creates one or more Pods to perform a given task.
The Job object takes the responsibility of Pod failures.
It makes sure that the given task is completed successfully. Once the task is complete, all the Pods have terminated automatically. Job configuration options include:

parallelism - to set the number of pods allowed to run in parallel;
completions - to set the number of expected completions;
activeDeadlineSeconds - to set the duration of the Job;
backoffLimit - to set the number of retries before Job is marked as failed;
ttlSecondsAfterFinished - to delay the clean up of the finished Jobs.

we can also perform Jobs at scheduled times/dates with CronJobs,

DAEMONSET¶

In cases when we need to collect monitoring data from all nodes, or to run a storage daemon on all nodes, then we need a specific type of Pod running on all nodes at all times. A DaemonSet is the object that allows us to do just that.
The kube-proxy agent running as a Pod on every single node in the cluster is managed by a DaemonSet.

STATEFULSET¶

The StatefulSet controller is used for stateful applications which require a unique identity, such as name, network identifications, or strict ordering. For example, MySQL cluster, etcd cluster.
The StatefulSet controller provides identity and guaranteed ordering of deployment and scaling to Pods. Similar to Deployments, StatefulSets use ReplicaSets as intermediary Pod controllers and support rolling updates and rollbacks.

Unlike a Deployment, a StatefulSet maintains a sticky identity for each of their Pods. These pods are created from the same spec, but are not interchangeable: each has a persistent identifier that it maintains across any rescheduling.

CUSTOM RESOURCES¶

There are two ways to add custom resources:

Custom Resource Definitions (CRDs)
This is the easiest way to add custom resources and it does not require any programming knowledge. However, building the custom controller would require some programming.
API Aggregation
For more fine-grained control, we can write API Aggregators. They are subordinate API servers which sit behind the primary API server. The primary API server acts as a proxy for all incoming API requests - it handles the ones based on its capabilities and proxies over the other requests meant for the subordinate API servers.

Should I use a configMap or a custom resource?¶

Kubernetes Cluster Federation¶

We can manage multiple Kubernetes clusters from a single control plane. We can sync resources across the federated clusters and have cross-cluster discovery.
This allows us to perform Deployments across regions, access them using a global DNS record, and achieve High Availability.

SECURITY¶

Security Contexts allow us to set Discretionary Access Control for object access permissions, privileged running, capabilities, security labels, etc. However, their effect is limited to the individual Pods
In order to apply security settings to multiple Pods and Containers cluster-wide, we can define Pod Security Policies.

SNYK Webinar: Live Hack: Hack My Mis-Configured Kubernetes, sponsored by Snyk

https://kubernetes.io/docs/tasks/configure-pod-container/security-context/

dal POD con root
nmap -sn IP_ETh0/netmask
sugli host trovati
nmap -sS

https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.20/#securitycontext-v1-core
https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.20/#podsecuritycontext-v1-core
privileged: false
runAsNonRoot: true
allowPrivilegeEscalation: false

Network Policies¶

are sets of rules which define how Pods are allowed to talk to other Pods and resources inside and outside the cluster. Pods not covered by any Network Policy will continue to receive unrestricted traffic from any endpoint.

The Network Policy API resource specifies podSelectors, Ingress and/or Egress policyTypes, and rules based on source and destination ipBlocks and ports.

METRICS¶

Prometheus

LOGGING¶

A popular method to collect logs is using Elasticsearch together with fluentd with custom configuration as an agent on the nodes. fluentd is an open source data collector, which is also part of CNCF.

Helm¶

client queries the Chart repositories for Charts based in search parameters, downloads a desired Chart, and then it requests the API server to deploy in the cluster the resources defined in the Chart.
Charts submitted for Kubernetes are available here.

Service Mesh¶

is a third party solution to the Kubernetes native application connectivity and exposure achieved with Services paired with Ingress Controllers.

Consul by HashiCorp
Envoy built by Lyft, currently a CNCF project
Istio is one of the most popular service mesh solutions, backed by Google, IBM and Lyft
Kuma by Kong
Linkerd a CNCF project

https://servicemesh.es/
http://leanpub.com/s/8duzgr_S1vlLyGjptlKzZA.pdf

https://docs.google.com/spreadsheets/d/1Bxf8VW9n-YyHeBiKdXt6zytOgw2cQlsDnK1gLUvsZ4A/edit#gid=907731238

CKAD¶

Crea un pod con data immagine, replicas, namespace
Pod con due container con volume persistente condiviso
Sidecar container
Volume hostPath
Storage class
Pod con due container, uno con dei comandi “while …”

https://kubernetes.io/docs/tasks/inject-data-application/define-command-argument-container/

Pod con secret come env var
Pod con config map come volume
Crea deployment
Inserisci readiness e liveness al pod -> errore se provo con k edit
https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/#define-startup-probes
Associa un service account ad un pod
Salva i log di un pod su file

Problemi

Non ho trovato un modo per creare velocemente un deployment
Il deployment falliva per errore sul selector/label
Associando readiness e liveness ad un pod, questi campi risultavano read only
Differenze tra command e arg

AMBASSADOR PATTERN

ADAPTER PATTERN

CKS¶

sudo tail /proc/self/stack

-> sudo grep 'Name\|Cap\|Sec' /proc/self/status

Name: grep
CapInh: 0000000000000000
CapPrm: 000001ffffffffff
CapEff: 000001ffffffffff
CapBnd: 000001ffffffffff
CapAmb: 0000000000000000
Seccomp: 0
Seccomp_filters: 0

➜ capsh --decode=000001ffffffffff
0x000001ffffffffff=cap_chown,cap_dac_override,cap_dac_read_search,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_linux_immutable,cap_net_bind_service,cap_net_broadcast,cap_net_admin,cap_net_raw,cap_ipc_lock,cap_ipc_owner,cap_sys_module,cap_sys_rawio,cap_sys_chroot,cap_sys_ptrace,cap_sys_pacct,cap_sys_admin,cap_sys_boot,cap_sys_nice,cap_sys_resource,cap_sys_time,cap_sys_tty_config,cap_mknod,cap_lease,cap_audit_write,cap_audit_control,cap_setfcap,cap_mac_override,cap_mac_admin,cap_syslog,cap_wake_alarm,cap_block_suspend,cap_audit_read,cap_perfmon,cap_bpf,cap_checkpoint_restore

➜ grep 'Name\|Cap\|Sec' /proc/self/status
Name: grep
CapInh: 0000000000000000
CapPrm: 0000000000000000
CapEff: 0000000000000000
CapBnd: 000001ffffffffff
CapAmb: 0000000000000000
Seccomp: 0
Seccomp_filters: 0

➜ man syscalls

===============================================================

CIS = centre for internet security benchmark¶

It provides tools for people and governments to respect best security practices.
- verbose logging
/var/log/containers
find /var/log/ -type f -exec grep configmap {} +
Install rsyslog
- auditing access
- FS
/dev/shm
Ensure /tmp is a separate partition (tmpfs)
mounting new devices is disabled
Ensure strong password hashing algorithm is configured
- network
Ensure SSH access is limited
Ensure SSH PAM is enabled
- debugging
- Ggm
- userspace
- no duplicate id, gid, usernames…
- Jobs
permissions on /etc/crontab are configured

CIS-CAT tool
> ./Assessor-CLI.sh -html --reports-dir /var/www/html/ --no-timestamp -report-prefix index --benchmark benchmarks/CIS_Ubuntu_Linux_20.04_LTS_Benchmark_v2.0.0-xccdf.xml --profile xccdf_org.cisecurity.benchmarks_profile_Level_1_-_Server

Releases · inspec/inspec (github.com)
dev-sec/cis-docker-benchmark: CIS Docker Benchmark - InSpec Profile (github.com)

wget -O /root/cis-cat.zip https://learn.cisecurity.org/e/799323/l-799323-2019-11-15-3v7x/2mnnf/145238200?h=XB2O2vOkVGHhHbfAxiAyYhdsSGW-osVs3NvDASSOSps

> curl https://omnitruck.chef.io/install.sh | sudo bash -s -- -P inspec
git clone https://github.com/dev-sec/cis-docker-benchmark
inspec exec cis-docker-benchmark

===============================================================

AQUASECURITY KUBE-BENCH¶

curl -sLO $(curl -s https://api.github.com/repos/aquasecurity/kube-bench/releases/latest |grep "browser_download_url.*linux_amd64.tar.gz" |tr -s ' ' |cut -d'"' -f4 )

GitHub - aquasecurity/kube-bench: Checks whether Kubernetes is deployed according to security best practices as defined in the CIS Kubernetes Benchmark

kube-controller-manager | Kubernetes

===============================================================

General Principles¶

Confidentiality: no unauthorised read
Integrity: no unauthorised modification
Availability: data is accessible even during an attack
Non Repudiation/Accountability: non deniability of own actions

Others

Auditing: logging
Identification & Authentication: requires an Identity Authorization Manager and a login process (user/psw, AD, SSH cert, SSH PAM…)
Authorization: identities permission management

Containers¶

Containers images¶

Containers runtime¶

Pause container: Every Kubernetes Pod includes an empty pause container, which bootstraps the Pod to establish all of the cgroups, reservations, and namespaces before its individual containers are created.
Containers in a pod share the network
Mounted host paths like
/dev host devices
/sys process info
/proc launch new containers
Linux capabilities
CAP_SYS_ADMIN
CAP_NET_RAW
CAP_SET_UID
CAP_SYS_PTRACE
Namespace can be inferred by .metadata headers and by /etc/resolv.conf

Security levels in the cloud¶

Code
SBOM analysis
Vuln scanner
Static code analysis
Container
SHA256 digest tags
Signed images
Build from scratch
Security context
1. Privileged
2. readOnlyFilesystem
3. runAsUser
Cluster
Network policies
Admission controllers
Secret management (key vault)
RBAC through AD
Seccomp / Sellinux / Apparmor -> Security Profiles Operator
Rebuild of old containers
Operators can see deployment secrets and have Cluster wide role bindings
CPU and MEM resource limits
Kubesec
Kubescan
Falco IDS
Tracee IDS
CloudProvider
Vulnerability in the hardware
Bugged services

Sandboxing¶

VM -> QEMU .> KVM -> Linux Kernel

Linux Kernel Virtual Machine is a module that runs a nested kernel as hypervisor for the container
Kata
Executes any syscalls but it is slower since it has a QEMU kernel replication
Gvisor
Syscalls are limited but very fast
Firecracker

Persistence attacks

PKI¶

Cosign¶

Sign docker images
Sign build artefacts

SPIFFE/SPIRE¶

Assign identity to k8s workloads

OverlayFS
Tmpfs

K8S Certificates¶

Use curl instead of kubectl (github.com)

>> kubectl unable to connect to server: x509: certificate signed by unknown authority
cat .kube/config
openssl s_client -showcerts -connect 127.0.0.1:6443
kubectl --insecure-skip-tls-verify cluster-info dump
crictl ps -a |grep api-server
crictl logs 569cfa02f7716 --follow

grep 2379 /etc/kubernetes/manifests/kube-apiserver.yaml
netstat -tlnp |grep etcd
openssl s_client -showcerts -connect 127.0.0.1:2379
openssl verify -CApath /etc/kubernetes/pki/ /etc/kubernetes/pki/ca.key

openssl rsa -check -noout -in akshay.key
openssl req -verify -noout -in akshay.csr
openssl req -in akshay.csr -text -noout

# Take values from kubeconfig and decode base64
curl \
--cacert cluster_admin_dev_pau.ca \
--cert cluster_admin_dev_pau.crt \
--key cluster_admin_dev_pau.key \
https://dev-pau-aks-55062799.hcp.westeurope.azmk8s.io:443

curl --cacert cluster_admin_dev_pau.ca --cert cluster_admin_dev_pau.crt --key cluster_admin_dev_pau.key https://dev-pau-aks-55062799.hcp.westeurope.azmk8s.io:443/apis/apps/v1/namespaces/dev-pau-testing/deployments/pythontestappdeployment

# https://docs.github.com/en/rest/releases/releases?apiVersion=2022-11-28#get-the-latest-release
latest_tag=$(curl -sL -H "X-GitHub-Api-Version: 2022-11-28" https://api.github.com/repos/mikefarah/yq/releases/latest |jq -r .tag_name)

curl -fsSL https://github.com/mikefarah/yq/releases/download/${latest_tag}/yq_linux_amd64 -o yq

yq_linux_amd64_checksum="$(curl \-fsSL https://github.com/mikefarah/yq/releases/download/${latest_tag}/checksums |grep "yq_linux_amd64 " |tr --squeeze-repeats [:space:] |cut -d' ' -f2- )"

grep $(sha256sum yq_linux_amd64 |cut -d' ' -f1) yq_linux_amd64_checksum

mv yq_linux_amd64 yq
chmod +x /usr/local/bin/yq
sudo mv yq /usr/local/bin/yq

# Choose a user with the certificate data inside kube config
export CLUSTER_USER="clusterAdmin_dev-pau-infra-group_dev-pau-aks"
export CLUSTER_NAME="dev-pau-aks"

SERVER="$(yq eval '.clusters[] | select(.name == env(CLUSTER_NAME)) | .cluster.server' $KUBECONFIG)"

SYSTEM HARDENING¶

Least Privilege¶

Account management
Identity profiling
Session locks
Session terminations
Information flow enforcement

SSH Hardening¶

/etc/ssh/sshd_config
visudo

permitRootLogin no
passwordAuthentication no
usermod --shell /usr/sbin/nologin –lock himanshi

Privilege Escalation

Remove Obsolete packages

systemctl list-units --type service --state active
apt list --installed

Kernel Modules

Secrets
kubectl create secret
tls
webhook-server-tls
--namespace webhook-demo
--cert /root/keys/webhook-server-tls.crt
--key /root/keys/webhook-server-tls.key

Sandboxing¶

https://stackoverflow.com/a/50150710/3673430

KUBERNETES CKA CKS¶

REPO PER ESERCITAZIONI¶

INSTALLAZIONE¶

AVVIO¶

CLUSTER INFO¶

SCHEDULER¶

KUBELET¶

PODS¶

DEPLOYMENT¶

CONTEXT¶

NODES¶

SERVICES¶

CLUSTER-IP¶

NODE PORT¶

LOAD BALANCER¶

EXTERNAL IP¶

EXTERNAL NAME¶

INGRESS¶

KUBE PROXY¶

SERVICE DISCOVERY¶

Environment Variables¶

DNS¶

VOLUMES¶

PERSISTENT VOLUME¶

A PersistentVolumeClaim (PVC)¶

Dynamic Volume Provisioning¶

CONFIG MAPS¶

SECRETS¶

QUOTAS¶

AUTOSCALING¶

JOB¶

DAEMONSET¶

STATEFULSET¶

CUSTOM RESOURCES¶

Should I use a configMap or a custom resource?¶

Kubernetes Cluster Federation¶

SECURITY¶

Network Policies¶

METRICS¶

LOGGING¶

Helm¶

Service Mesh¶

CKAD¶

CKS¶

CIS = centre for internet security benchmark¶

AQUASECURITY KUBE-BENCH¶

General Principles¶

Containers¶

Containers images¶

Containers runtime¶

Security levels in the cloud¶

Sandboxing¶

PKI¶

Cosign¶

SPIFFE/SPIRE¶

K8S Certificates¶

SYSTEM HARDENING¶

Least Privilege¶

SSH Hardening¶

Sandboxing¶

NODE PORT ¶

KUBE PROXY ¶

SERVICE DISCOVERY ¶

JOB ¶