Linux transparent proxy internals

I've always thought the transparent proxying functionality in the Linux kernel is one of the coolest but hardest to understand.

This may have to do with the sheer lack of documentation on the subject or maybe the fact that configuring it requires quite a few moving pieces like IP rules, socket options, and routing.

Let's explore how transparent proxying works and build a toy transparent proxy server to drive the concepts home.

Most people will have colloquial familiarity with the term "proxy" as a networking infrastructure.

A proxy sits between a client, the source, and a server, the destination, and manages the connections and data between them.

If this is a TCP proxy, it will terminate the client's connection, holding a socket to it, and will connect to the server, holding a socket for this "leg" as well.

With both sockets in-hand the proxy will pipe the data between them, performing whatever actions on the data is required.

Proxies can do all sorts of things, but we just need to understand the fundamentals right now.

The above is what, I think, most people imagine when they think 'proxy', though technically, this is a reverse proxy where the proxy sits in front of the server.

To understand what a transparent proxy is they must think about what IP addresses the client will use to connect to the server, and the client IPs seen by the server.

Let's use the non-transparent proxy example above, both the client connection and the server's connections are well aware a proxy server sits between them.

A transparent proxy is fundamentally different.

Notice, in the transparent case the client perceives itself connected directly to the server and the server perceives itself connected directly to the client.

Both client and server maintain their IP addresses, across both "legs" of the proxy.

The example above is sometimes called a "fully transparent proxy" since the source IP of the client is maintained from the server's perspective. It's not uncommon for the server to see the proxy's original IP when, for example, routing the client's IP back to the proxy is difficult, among other reasons.

While the explanation is simple, like a lot of things with networking, the devil's in the details.

Let's start building a proxy server which we will modify into a transparent proxy as we explain how we can accomplish the above.

We need a small testing environment which creates separate layer 3 networks for the client, proxy, and server.

We can do this with just Linux network namespaces, the ip tool, and veths.

The above diagram illustrates the topology, three network namespaces connected by veths.

Makefile

We can now test end-to-end connectivity with the command:

This will ping from the client's IP network stack to the server's using the proxy network namespace as a router; it should succeed.

Let's first build a non-transparent proxy server as a stepping stone.

This transparent proxy will be very dumb, simply to demonstrate the transparent proxying requirements.

It will only accept one connection at a time, wait for the client to write, return the response from the server back to the client, and then close both the client and server connections.

proxy.c

Let's also update the Makefile to build the proxy server.

Run make to build the proxy binary.

At this point we can test the proxy server in a few terminals.

term1: deploy topology, start an echo server using nc

term2: start the proxy

term3: issue a request to the proxy using nc

Once the action in term3 is completed we should see the following new lines in term1 and term2

term1:

term2:

We just built a very dumb, yet functional, proxy server which terminates the client connection, opens a new connection to the server, and streams the bits between both client and server sockets.

Notice, from the server's perspective, the proxy is connecting to it.

Additionally, the client connects to the proxy, not to the server.

Recall, in the transparent proxying case, we want the client to connect to the proxy using the echo server's address and port, and we want the proxy to connect to the echo server using the original client's address and port.

For me, it was always easiest to understand how transparent proxying works by first outlining the obstacles it presents in the normal Linux network stack.

Let's start at the client who we want to send a request directly to the server 10.0.6.1:8080.

We can do this right now, since we have end-to-end connectivity, and it will work but we bypass the proxy altogether, not what we want.

Consider the current network path for this 10.0.6.1:8080 packet sent from a client which bypasses the proxy:

The packet is received on proxy-a veth inside the proxy network namespace.

The primary layer 2 function handles the packet, determines it's an IPv4 protocol packet, and calls into ip_rcv.

ip_rcv is the ingress IP packet handling function which does a bit of prep work and quickly calls into ip_rcv_finish_core.

In the core function a route lookup is done on the packet, this "host" doesn't have the 10.0.6.1 IP assigned to any of its interfaces, but it does have a route to it found by calling ip_route_input_noref.

Because the kernel has determined it can be routed the ip_forward function is called to eventually transmit the packet out the interface toward its next hop.

Herein lies the obstacles transparent proxying must overcome.

1. The routing layer must accept the packet for local delivery, not forward it.

2. Once accepted for local delivery, the proxy needs to deliver it to the proxy's TCP socket bound to a port that does not match the packet's destination.

3. Once the client side is connected the proxy must connect to the server with the original client's address and port.

4. The return traffic must pass through the proxy to ensure TCP connections are properly maintained.

Let's tackle these individually.

It's not really a trick, more of a common practice used in a not-so-common way.

Firstly, we can force a packet, based on its destination, to be locally delivered with a specific routing table entry.

This is what the local keyword does when adding a route with ip route.

Let's take this route for example:

This route would take all traffic and accept it as local delivery.

Of course, it's not that simple, since this will also route traffic leaving the host, back into the network stack, which is not what we want.

We will need to use policy routing to select the traffic we want to force local delivery for.

This is done with a combination of packet marking, IP rules, and a dedicated routing table.

We can configure iptables to mark all packets destined for the echo server 10.0.6.1 with the value 0x1.

Next, we can create a policy routing rule with the ip rule tool, telling the kernel to use routing table 100 when performing route lookups for packets with the 0x1 mark.

Finally, we add the route we discussed earlier into table 100, instructing the kernel to deliver these packets locally, even if the IP address is not present on the host.

The above diff in the Makefile does exactly this.

If we now rebuild the topology, start the echo and proxy servers, and issue the client request, you will see it just hang.

This is because, despite being accepted for local delivery by the routing subsystem, layer 4 cannot find a socket for the ingress packet.

To deliver the packet to the proxy, we must hijack layer 4 delivery.

The current socket state in the proxy network namespace results in a dropped packet.

We can accept the packet now for local delivery and get past layer 3.

However, layer 4 will attempt to lookup a listening socket bound to either 10.0.6.10:8080 or the wild card *:8080 which does not exist in this namespace and drop the packet.

We need a form of socket redirection which does not rely on modifying the packet's destination port, like a NAT function would.

This exists and is primarily implemented by the TPROXY iptables target, though, it requires a change in the application code as well.

For TPROXY redirection to work we must do two things:

1. Introduce a new iptables rule which redirects the marked traffic to the proxy port for delivery

2. Ensure the proxy's listening socket is set to IP_TRANSPARENT

The newly introduced TPROXY rule runs in the mangle table, prior to any routing decision being made on the packet.

It will run for any packets with the 0x1 mark and will transparently redirect the packet to the proxy's listening port.

Next, we update the proxy's code to enable IP_TRANSPARENT on the listening socket.

This allows the proxy to accept connections for destinations it is not bound to, exactly what we need to accept a SYN packet destined to the echo server's 10.0.6.1 address.

If we re-deploy the topology and send a request, guess what?

Everything will work!

term2: proxy

term3: echo server

term1, the client, is omitted since we simply run the same command as previously shown.

The client's packets were locally delivered and redirected into the proxy's listening socket.

We can see this with tcpdump in the client's namespace.

In the above dump we see the client sending a SYN to the echo server.

Once the proxy accepts the incoming packet, because the connection is over a transparent socket, the kernel is smart and will send the SYN,ACK and all subsequent traffic sourced as the destination of the SYN.

What we've built so far is an example of a "non-fully transparent proxy", for lack of a better term.

We are transparent for the client, but the server still sees the traffic originating from the proxy, not the original client.

Let's make the proxy fully transparent next.

In the previous sections we have the proxy performing client side transparency, but the connection to the server is still made from the proxy's source address.

Typically, Linux will not allow an application to bind to an address that's not assigned to an interface on the host.

This can be, yet again, overcome with the IP_TRANSPARENT socket option coupled with the bind function call.

However, this is not enough.

Consider, the reply traffic entering the proxy's network namespace now.

The reply goes through layer 2 into layer 3 where it is simply routed out.

This is because there is no mechanism to deliver the return traffic locally and perform the subsequent socket redirection required.

The return traffic is unique however, since a socket which matches the return packets will exist.

We can exploit this fact and use a socket iptables match function.

The socket match function will inspect the incoming tuple and determine if a socket lookup for it would successfully return a socket, which in our reply case above, it would.

We can then pair this match with a MARK target to mark the packet for local delivery, using the same exact policy routing mechanisms we already configured.

Once delivered locally, layer 4 will attempt the same socket lookup the socket match function did, find the transparent socket, and deliver it to the transparent proxy.

Rebuild the proxy and redeploy the topology.

Issuing a new client request will now show a complete transparent proxy flow!

The interesting bit is the output of the echo server

term3: connection received from proxy, using client's address.

Congratulations you have a fully transparent proxy server!

We now understand the full end-to-end configuration of transparent proxying in the Linux kernel.

Let's turn our attention to how the crux of the operation works, the TPROXY target.

The TPROXY target is implemented within the kernel's source code located in the file xt_TPROXY.c.

The function which is invoked when the target is expressed follows.

xt_TPROXY.c

The tproxy_tg4_v0 function exists solely to extract the arguments of the TPROXY target and supply them to tproxy_tg4 proper.

Consider the target used in this scenario.

We will be supplying the tgi->lport argument to tproxy_tg4 given the --on-port target argument translation.

We won't be supplying either of the tgi->mark_mask or tgi->laddr arguments, though it is nice to know they exist:

Let's turn our attention to the tproxy_tg4 function now.

1.

The above gets a pointer to the packet's header and stores it into a struct udphdr, which is used regardless of whether layer 4 is UDP or TCP, since both protocols' wire formats start with the source and destination ports.

Next, a socket lookup is attempted for the current packet header details.

This socket lookup is done to find connected sockets that were already accepted by our transparent proxy and used to forward the request to the server.

2.

Next, the laddr and lport arguments are resolved, preferring the arguments to the TPROXY target or using the packet's wire data in lieu of the former.

3.

Let's brush over the TIME_WAIT complexities for now.

If no socket was found, this is a new packet never seen by our transparent proxy before.

At this point, the kernel will perform a socket lookup using the arguments provided to the target.

This lookup will find the transparent proxy's listening socket, noting the NF_TPROXY_LOOKUP_LISTENER argument to nf_tproxy_get_sock_v4.

Because the proxy is listening on *:6666 and the lookup is being done with lport == 6666, the proxy's listening socket will be found.

4.

Finally, if a socket was found previously, and the socket is marked transparent, an optional mark value is applied and, crucially, the skb is assigned the socket.

Once the skb is assigned the socket, it will now continue to layer 3, where our policy routing rules exist to force it into the local delivery path.

The skb then continues to layer 4, but now a socket is already assigned to the skb.

This socket will be used for subsequent delivery to an application, finalizing the usage of the TPROXY target.

For the sake of a clear explanation I took the longer route to achieve the full transparent proxy demo.

For one, its so common to mark packets for TPROXY that the TPROXY target can do this on its own.

We can remove the MARK rule and tell TPROXY to do the marking for us, collapsing the two rules for the client side leg to one.

This is optional of course, but is preferred, especially on nodes with a lot of iptables rules.

I always enjoy going over the transparent proxy flow.

It's a clinic on how to think about Linux networking, or networking in general, in layers.

Once we start to understand that each layer of the network stack has a set of rules, we can start to understand how to bend them to get to a solution like transparent proxying.