Securing Kubernetes Resources Without a VPN

Securing kubernetes resources that you want to expose to only some users externally is often done through IP allowlisting and a VPN. While this is a tried and true method, there are some drawbacks.

VPNs may require manual installation and configuration on your coworkers’ devices. This is especially true for smaller companies who may not have dedicated IT teams. Depending on the configuration and type of VPN, it could also leave your users with slow connections and a less than ideal experience.

As part of a smaller engineering team, it’s important to focus on changes that minimize manual work. New VPN implementations can be daunting, and it might seem like overkill if you only need to share a handful of internal web pages to the broader company. Instead, consider an alternative option that could save you a ton of implementation time and avoid “helpdesk” style slack follow ups from your coworkers: OAuth2 Proxy.

What is OAuth2 Proxy?

In short, OAuth2 Proxy determines whether a user should be allowed to access a resource or not. We can let it do the hard work and verify that users are logged into the company Google or Microsoft organization before letting them access the resource, no VPN needed! You will need to pair oauth2-proxy with a suitable webserver. In my example, I’ll be leveraging ingress-nginx, but there are plenty of other ingress controllers and webserver that can get the job done. Let’s dive in!

OAuth2 Proxy + Ingress-Nginx = A perfect match

OAuth2 Proxy can be used as either a reverse proxy or middleware for another tool, like nginx, to protect services with Oauth2 or OIDC. Nginx can receive a request, redirect you to oauth2-proxy if you aren’t authenticated, and then validate that you are on consecutive requests in order to grant access to a resource. This lets you expose ingress-nginx resources directly to the internet, and still prevent anyone outside of your organization from accessing those resources. I’ve included a diagram from the oauth2-proxy README that highlights the flow we’ll be using.

Stepping Back — IP Allowlisting with Ingress-Nginx

Let’s take a quick step back and talk about IP allowlisting with ingress-nginx. I’m most familiar with AWS and Kubernetes clusters running on EKS, and this article will be focused on that use case for implementation specifics.

Assuming you have ingress-nginx installed on a cluster and it’s linked to your cloud service provider’s load balancer, the first thing you’ll want to look at is the `nginx.ingress.kubernetes.io/whitelist-source-range` annotation. This annotation allows you to specify one or more CIDR ranges that should be able to access the resource, and everyone else should get a 403. Let’s try it!

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/whitelist-source-range: "26.221.232.245/32"
  name: redsun-allowedlisted-ingress
  namespace: default
spec:
  ingressClassName: nginx
  rules:
  - host: ip.0xredsun.gg
    http:
      paths:
      - backend:
          service:
            name: internal-service
            port:
              number: 80
        path: /
        pathType: Prefix

Note that I’m omitting certificate and SSL related configurations (I generally use cert-manager for this) and focusing on the ingress-nginx configuration.

In the example above, you can see we’ve added a whitelist for our VPN IP. So we should be able to access our service now, let’s give it a shot!

Weird! Even though we’ve confirmed our IP is correct, and that our ingress is up-to-date, it’s not working. After a bit of debugging we might see the nginx is actually blocking requests because the source IP is the IP address of our load balancer itself, and not of the original requester.

In order to figure out how to get the original source IP, let’s take a look at the ingress-nginx docs. There are some different options that apply depending on the the specifics of your load balancer and the traffic that you are routing to the cluster. In my case, I elected to use the proxy protocol between the load balancer and ingress-nginx to resolve the issue. This required me to re-deploy the loadbalancer for nginx and to use a network load balancer(NLB) as opposed to a standard elastic load balancer(ELB) in AWS.

Once I took those steps — I was be able to access the new service ONLY from the allowlisted IP. Now that we have a basic implementation for granting access via VPN — let’s talk about OAuth2-Proxy.

OAuth2 Proxy

Oauth2-proxy can use either OAuth 2.0 or OpenID Connect as authentication methods. There’s a lot of great documentation about OAuth and OIDC — I’d specifically recommend taking a look at Okta’s documentation if you want to dive deeper. For now, suffice it to say that oauth2-proxy will integrate with many different Identity Providers or IDPs such as Google, Github, Microsoft, and many more. It also supports “Generic” OIDC which allows you to integrate it with any identity provider which supports the standard. In my case, I plan to integrate it with my Google workspace so that only members of my workspace will be able to access my protected resources.

I’m not going to dive into all of the configuration or setup details in this article, but the ingress-nginx repo has some sample YAML files that cover the basics of the implementation you can reference. Additionally, the oauth2-proxy github repo has sample provider configuration guides to help you get everything set up with your identity provider of choice. You can refer to my oauth2-proxy deployment, service, and ingress below and compare to the sample files if you want for some extra context.

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    k8s-app: oauth2-proxy
  name: oauth2-proxy
  namespace: default
spec:
  replicas: 2
  selector:
    matchLabels:
      k8s-app: oauth2-proxy
  template:
    metadata:
      labels:
        k8s-app: oauth2-proxy
    spec:
      volumes:
      - name: google-service-account
        secret:
          secretName: google-service-account-oauth
      containers:
      - args:
        - --provider=google
        - --email-domain=0xredsun.gg
        - --cookie-domain=.0xredsun.gg
        - --http-address=0.0.0.0:4180
        - --whitelist-domain=".0xredsun.gg"
        - --whitelist-domain=".myotherdomains.com"
        - --google-group=admins@0xredsun.gg
        - --google-group=everyone@0xredsun.gg
        - --google-admin-email=bpsizemore@0xredsun.gg
        - --google-service-account-json=/etc/google-service-account/serviceaccount
        env:
        - name: OAUTH2_PROXY_CLIENT_ID
          value: 'SOME_ID'
        - name: OAUTH2_PROXY_CLIENT_SECRET
          value: 'SOME_SECRET'
        - name: OAUTH2_PROXY_COOKIE_SECRET
          value: 'COOKIE_SECRET'
        image: quay.io/oauth2-proxy/oauth2-proxy:latest
        imagePullPolicy: Always
        name: oauth2-proxy
        ports:
        - containerPort: 4180
          protocol: TCP
        volumeMounts:
        - name: google-service-account
          mountPath: /etc/google-service-account
          readOnly: true
---
apiVersion: v1
kind: Service
metadata:
  labels:
    k8s-app: oauth2-proxy
  name: oauth2-proxy
  namespace: default
spec:
  ports:
  - name: http
    port: 4180
    protocol: TCP
    targetPort: 4180
  selector:
    k8s-app: oauth2-proxy
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: oauth2-proxy
  namespace: default
spec:
  ingressClassName: nginx
  rules:
  - host: authenticate.0xredsun.gg
    http:
      paths:
      - path: /oauth2
        pathType: Prefix
        backend:
          service:
            name: oauth2-proxy
            port:
              number: 4180
---

I want to specifically highlight some of the values in the args section of my deployment.

- args:
        - --provider=google
        - --email-domain=0xredsun.gg
        - --cookie-domain=.0xredsun.gg
        - --whitelist-domain=".0xredsun.gg"
        - --whitelist-domain=".myotherdomains.com"
        - --google-group=admins@0xredsun.gg
        - --google-group=everyone@0xredsun.gg
        - --google-admin-email=bpsizemore@0xredsun.gg

The email domain flag determines what domain authenticated users will come from, this should match your organization.

The cookie domain flag will force the cookie to be scoped to that particular domain.

The whitelist-domain flag allows you to whitelist what domains are acceptable to redirect to. As an example — if you wanted to protect secret.myotherdomains.com using oauth2 proxy hosted at authenticate.0xredsun.gg you would need to add it as a whitelisted domain like I did above. This is true for every TLD you want to authenticate for. The leading . is a wildcard allowing all subdomains to work.

The google-group flag allows you to restrict login to only the included groups. You can reuse this flag as much as you like to add more groups that you might want to use to restrict resources. You can specify in individual ingresses which groups are authorized to access a resource, but you must specify in the configuration here that it should be passing along information about these groups to the ingress or it won’t work.

Finally, the google-admin-email flag is related to the google oauth setup. An admin account is used to query information about the individual users during the authentication process.

You can find specifics and more details on configuration flags here.

Leveraging OAuth2 Proxy to authenticate an Ingress

Now that our oauth2-proxy is running, all we need to do is leverage some annotations on any of our ingress objects to restrict it to our domain and groups.

Let’s take a look at an example ingress for ingress-nginx.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/auth-url: "https://internal.0xredsun.gg/oauth2/auth?allowed_groups=everyone@0xredsun.gg"
    nginx.ingress.kubernetes.io/auth-signin: "https://internal.0xredsun.gg/oauth2/start?rd=$scheme%3A%2F%2F$host$escaped_request_uri"
  name: redsun-oauth-ingress
  namespace: default
spec:
  ingressClassName: nginx
  rules:
  - host: oauth.0xredsun.gg
    http:
      paths:
      - backend:
          service:
            name: internal-service
            port:
              number: 80
        path: /
        pathType: Prefix

Let’s examine the two relevant lines:

nginx.ingress.kubernetes.io/auth-url: 
  "https://internal.0xredsun.gg/oauth2/auth?allowed_groups=everyone@0xredsun.gg"

The auth-url annotation is used to tell Nginx where it reach out to determine if the user is authorized to view the resource. The specifics of the URL is determined by the service and what specific functionality it supports (like group membership) that are required for a user to be authorized. Oauth2-proxy supports a few different options. In this case, we point it at the oauth2-proxy we just set up and exposed to the internet. On every request to the ingress — nginx will forward along the cookies and request data to the auth-url which will reply with a status that indicates authorized or not authorized. If you are unauthorized nginx will redirect you to the auth-signin url.

nginx.ingress.kubernetes.io/auth-signin: 
  "https://internal.0xredsun.gg/oauth2/start?rd=$scheme%3A%2F%2F$host$escaped_request_uri"

The rd flag is a redirect, and the query here simply redirects us back to the internal resource we had originally requested before nginx required us to login.

Assuming that oauth2-proxy is correctly configured and our ingress-nginx is working correctly we should be required to login to our domain in order to access the resource.

When attempting to access the page from a private browser — I’m prompted to login. The name 0xredsun Internal Services is configured in my Google Oauth Consent screen as part of the oauth2-proxy set up.

After logging in — I’m greeted with my internal page.

When trying to access the page from an email with a different domain, I’m presented with the following page.

Wrapping Up

At my organization not everyone uses a VPN and this setup makes it straightforward for our engineering team to control access to “internal” applications without a VPN. There’s no need to spend time trying to onboard users to the VPN and no new login process that anyone needs to learn since it leverages the company’s primary login, Google in my case.

You could also leverage this with groups to make a straightforward access control mechanism for your apps. Add the — google-group=someapp.access@yourdomain.com flag to oauth2-proxy and specify in the ingress that the allowed group is someapp.access@yourdomain.com At this point you can assign someone as a group manager or group owner, and they can manage access to the application as needed.