In today’s post, we are going to explore mirrord. Mirrord is a process mirroring tool which mirrors your local running process and related dependencies to a Kuebrnetes cluster. We will see how to quickly get started with mirrord, and how it works.
31st October 2024
This will take a minute.
I am currently operating a M2 Mac so running the command below to install the mirrord CLI.
brew install metalbear-co/mirrord/mirrord
We need a service running in the Kubernetes cluster against which we can test our process mirroring. We will be sending a HTTP GET request to the echo-server through process mirroring.
Start minikube.
minikube startLet’s install the echo-server with the following command.
helm repo add ealenn https://ealenn.github.io/charts
helm repo update
helm install echo-server ealenn/echo-server --namespace echo-server --force --create-namespace Run the command below to send HTTP GET request to the echo-server ClusterIP Service.
mirrord exec curl echo-serverThe echo-server Service is only available from within the context of the Kubernetes cluster (and namespace). The command above runs curl echo-server in the context the Kubernetes cluster and we get the output below.
* Running binary "/var/folders/v7/yqyq_d6x2996hfnnwvgs5f080000gn/T/mirrord-bin-ghu3278mz/usr/bin/curl" with arguments: ["echo-server"].
* mirrord will run without a target, no configuration file was loaded
* operator: the operator will be used if possible
* env: all environment variables will be fetched
* fs: file operations will default to read only from the remote
* incoming: incoming traffic will be mirrored
* outgoing: forwarding is enabled on TCP and UDP
* dns: DNS will be resolved remotely
⠁ mirrord exec
✓ running on latest (3.122.1)!
✓ ready to launch process
✓ layer extracted
✓ operator not found
✓ agent pod created
✓ pod is ready
✓ arm64 layer library extracted
✓ config summary
{"host":{"hostname":"echo-server","ip":"::ffff:10.244.0.10","ips":[]},"http":{"method":"GET","baseUrl":"","originalUrl":"/","protocol":"http"},"request":{"params":{"0":"/"},"query":{},"cookies":{},"body":{},"headers":{"host":"echo-server","user-agent":"curl/8.4.0","accept":"*/*"}},"environment":{"PATH":"/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin","HOSTNAME":"echo-server-cbc7ddb7b-s9c6q","ENABLE__COOKIES":"true","ENABLE__HTTP":"true","ENABLE__HEADER":"true","ENABLE__HOST":"true","ENABLE__REQUEST":"true","LOGS__IGNORE__PING":"false","PORT":"80","ENABLE__ENVIRONMENT":"true","ENABLE__FILE":"true","KUBERNETES_PORT_443_TCP_ADDR":"10.96.0.1","ECHO_SERVER_SERVICE_HOST":"10.103.113.87","ECHO_SERVER_PORT_80_TCP":"tcp://10.103.113.87:80","ECHO_SERVER_PORT_80_TCP_ADDR":"10.103.113.87","KUBERNETES_SERVICE_PORT":"443","KUBERNETES_SERVICE_PORT_HTTPS":"443","KUBERNETES_PORT_443_TCP_PROTO":"tcp","KUBERNETES_PORT_443_TCP_PORT":"443","KUBERNETES_SERVICE_HOST":"10.96.0.1","KUBERNETES_PORT_443_TCP":"tcp://10.96.0.1:443","ECHO_SERVER_SERVICE_PORT":"80","ECHO_SERVER_SERVICE_PORT_HTTP":"80","ECHO_SERVER_PORT_80_TCP_PROTO":"tcp","ECHO_SERVER_PORT_80_TCP_PORT":"80","KUBERNETES_PORT":"tcp://10.96.0.1:443","ECHO_SERVER_PORT":"tcp://10.103.113.87:80","NODE_VERSION":"16.16.0","YARN_VERSION":"1.22.19","HOME":"/root"}}%The last part of the output after “config summary” was returned from the echo-server Service.
The mirrord-agent acts as a proxy to the local process due to which it can run in the context of the Kubernetes cluster. More on the agent here. When we run `mirrord exec curl echo-server` the CLI basically creates a Job called the mirrord-agent which looks like the one below.
k get job
NAME COMPLETIONS DURATION AGE
mirrord-agent-95u3z3q18t 0/1 6s 6sAnd below are the logs for the agent. Had to quickly run it as the exec command runs.
kubectl logs -f job/$(kubectl get jobs --output=jsonpath='{.items[*].metadata.name}')
agent ready - version 3.122.1
2024-10-30T22:21:14.035109Z WARN mirrord_agent::outgoing::udp: interceptor_task -> no messages left
at mirrord/agent/src/outgoing/udp.rs:225 on ThreadId(5)
2024-10-30T22:21:19.172187Z WARN mirrord_agent::outgoing::udp: interceptor_task -> no messages left
at mirrord/agent/src/outgoing/udp.rs:225 on ThreadId(7)
2024-10-30T22:21:19.172908Z INFO mirrord_agent::entrypoint: main -> mirrord-agent `start` exiting successfully.
at mirrord/agent/src/entrypoint.rs:824 on ThreadId(1)We don’t have any critical information which we can work with here for sure except an observation that there is no UDP data available which can be forwarded to the user on the local host, hence the warning messages above. Apart from that the agent process is exiting successfully.
Let’s take apart the Job spec now. Let’s run the command below and stare at the spec for a bit.
kubectl edit job/$(kubectl get jobs --output=jsonpath='{.items[*].metadata.name}') We get the following output. I have annotated important/relevant lines,
apiVersion: batch/v1
kind: Job
metadata:
annotations:
batch.kubernetes.io/job-tracking: ""
linkerd.io/inject: disabled
sidecar.istio.io/inject: "false"
creationTimestamp: "2024-10-30T22:32:02Z"
generation: 1
labels:
app: mirrord
kuma.io/sidecar-injection: disabled ###
name: mirrord-agent-q4xcvitguv
namespace: echo-server
resourceVersion: "7817"
uid: 24167a9e-4c32-4ef7-b233-e9c98c2e73b3
spec:
backoffLimit: 0
completionMode: NonIndexed
completions: 1
parallelism: 1
selector:
matchLabels:
batch.kubernetes.io/controller-uid: 24167a9e-4c32-4ef7-b233-e9c98c2e73b3
suspend: false
template:
metadata:
annotations:
linkerd.io/inject: disabled ###
sidecar.istio.io/inject: "false" ###
creationTimestamp: null
labels:
app: mirrord
batch.kubernetes.io/controller-uid: 24167a9e-4c32-4ef7-b233-e9c98c2e73b3
batch.kubernetes.io/job-name: mirrord-agent-q4xcvitguv
controller-uid: 24167a9e-4c32-4ef7-b233-e9c98c2e73b3
job-name: mirrord-agent-q4xcvitguv
kuma.io/sidecar-injection: disabled
spec:
containers:
- command:
- ./mirrord-agent
- -l
- "33863"
- targetless ###
env:
- name: RUST_LOG
value: info
- name: MIRRORD_AGENT_STEALER_FLUSH_CONNECTIONS ##
value: "true"
- name: MIRRORD_AGENT_NFTABLES
value: "false"
- name: MIRRORD_AGENT_JSON_LOG
value: "false"
image: ghcr.io/metalbear-co/mirrord:3.122.1
imagePullPolicy: IfNotPresent
name: mirrord-agent
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 1m
memory: 1Mi
securityContext:
privileged: false ###
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
dnsPolicy: ClusterFirst
restartPolicy: Never
schedulerName: default-scheduler
securityContext: {}
terminationGracePeriodSeconds: 30
tolerations:
- operator: Exists
ttlSecondsAfterFinished: 1
status:
active: 1
ready: 1
startTime: "2024-10-30T22:32:02Z"
uncountedTerminatedPods: {}Sidecar injection disabled
Based on the labels and annotations given above we can tell that these have the Kuma, Istio and LinkerD sidecar injection disabled.
Targetless
Targetless is the mode because of which we were able to run `curl echo-server` in the content of the Kubernetes cluster without specifying a particular target pod or container. The opposite of this,l targets a particular pod and intercepts/mirrors the traffic and environment of the process in the pod (Targeted mode).
Flush existing connections
MIRRORD_AGENT_STEALER_FLUSH_CONNECTIONS is an environment variable is set to true which means that existing connections are flushed for more accurate interception of traffic.
Unprivileged Container
The agent container runs in an unprivileged mode but needs the below Linux capabilities to act as a proxy for processes. These capabilities in Linux allow granular control to be given to processes based on what they can or cannot do. We can see these being used in the targeted mode and not in the targetless mode we were using right now.
CAP_NET_ADMINandCAP_NET_RAW- used for modifying routing tables.CAP_NET_ADMINallows network interface, firewall, routing tables, socket permissions configurations etc.CAP_NET_RAWallows for creation of RAW and PACKET socket types.
CAP_SYS_PTRACE- used for reading target pod environment.Gives the ability to use the `ptrace` system call which is usually used for debugging and process control and in this case we can use it to get information about the process running in the Pod.
CAP_SYS_ADMIN- used for joining target pod network namespace.Gives the ability to mount filesystems, change system time, manage devices etc.
