Preserve Source IP when using an Ingress Controller

By default, when using Civo and NGINX Ingress Controller or Traefik, the incoming request will not pass the source IP of packets through to the Kubernetes service and ultimately to the containerized application (running in a pod). Instead, the application will see an intermediary IP (e.g. in-cluster NGINX pod’s IP as the source). With the following configuration change, the application will be able to see the source IP of packets and proceed with its business logic (i.e. If the IP is from the Asia Pacific region, redirect to the APAC corporate site, otherwise, stay on the current/global site).

This figure shows the preservation of the client’s public IP when both the “externalTrafficPolicy” on the LoadBalancer service is set to “Local” as well as the “enable Proxy protocol” annotation is added and set to “send-proxy”. When the client sends the request to the Kubernetes service (1), the packet header is changed, and the request will now preserve the client’s public IP (i.e. 148.73.223.18) all the way to the pod.

This tutorial provides instructions on how to preserve the source IP for applications running on Kubernetes, with specific sections tailored for users of NGINX Ingress Controller and Traefik.

Prerequisites

Before you start with the setup to preserve the source IP for applications running on Kubernetes using NGINX Ingress Controller or Traefik on Civo, ensure you have the following prerequisites in place:

• Civo Kubernetes Cluster: You must have an active Civo account and a Kubernetes cluster set up on Civo.
• Kubectl: You must have kubectl installed on your host, in order to interact with the Kubernetes cluster via the command-line.
• Helm: Helm, the package manager for Kubernetes, is required for installing the NGINX Ingress Controller or Traefik. If you haven't installed Helm yet, you can do so through our Civo Marketplace when setting up your cluster.
• Access to Your Cluster: Ensure you have downloaded the Kubeconfig from the Civo dashboard, and the appropriate role is assigned to your user in order to create deployments, services, and ingress resources within your cluster.
• DNS Configuration (Optional): If you plan to test the ingress with a real domain, have access to your DNS provider to configure DNS records. This is optional for testing purposes, as you can also test using the LoadBalancer IP/DNS directly.
• Cloudflare DNS (Recommended for Testing): For faster DNS propagation during testing, it's recommended to use Cloudflare's DNS servers (1.1.1.1 and 1.0.0.1). This step is optional but can help reduce wait times during DNS propagation.
• jq (Optional): Some commands may utilize jq, a lightweight and flexible command-line JSON processor, for parsing command output. While not mandatory, installing jq could enhance your command-line experience. Visit the jq documentation for installation instructions.
• Familiarity with Kubernetes Concepts: A basic understanding of Kubernetes concepts, such as deployments, services, and ingress, is necessary to follow the instructions effectively.
• Network Policies (If applicable): If your cluster has network policies in place, ensure they are configured to allow traffic as required for the ingress controllers to function properly.

Installing NGINX Ingress Controller

The following will guide you through the process of creating a deployment using the kennethreitz/httpbin image, creating a service, installing the NGINX Ingress Controller via Helm chart, changing the NGINX service configuration, and creating an ingress to ensure the preservation of the source IP.

Step 1. Create a HTTPbin Deployment using kennethreitz/httpbin image:

kubectl create deployment my-httpbin --image=kennethreitz/httpbin

Step 2. Create a Service for Deployment:

kubectl expose deployment my-httpbin --port=80 --name=my-httpbin-svc

Step 3. Install Nginx ingress controller:

helm upgrade --install ingress-nginx ingress-nginx \
    --repo https://kubernetes.github.io/ingress-nginx \
    --namespace ingress-nginx --create-namespace

Step 4. Edit the ingress-nginx-controller Service in ingress-nginx namespace:

kubectl -n ingress-nginx edit svc ingress-nginx-controller

Add this annotation:

kubernetes.civo.com/loadbalancer-enable-proxy-protocol: send-proxy

Step 5. Edit the ingress-nginx-controller ConfigMap in ingress-nginx namespace

kubectl -n ingress-nginx edit cm ingress-nginx-controller

Add these lines under data:

real-ip-header: proxy_protocol
use-proxy-protocol: "true"

Step 6. Create an Ingress:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: my-httpbin-ingress
  namespace: default
spec:
  ingressClassName: nginx
  rules:
  - http:
      paths:
      - pathType: Prefix
        path: "/"
        backend:
          service:
            name: my-httpbin-svc
            port:
              number: 80

Step 7. Hit the LoadBalancer's IP/DNS to test. One can obtain the LoadBalancer’s IP and DNS via the Civo Dashboard, below the custer information.

Once you have obtained the Public IP or the DNS Name form the Civo dashboard, we can test the communication to that IP/DNS via curl (e.g. curl 131.153.226.68 or curl 288d2dc3-0dc6-4bb9-b030-0a74fa32f999.lb.civo.com).

Installing Traefik Ingress Controller through an Helm chart

Civo K3s clusters are created by default with a Traefik Ingress Controller. If you removed this default when creating the cluster,you can add it through a Helm chart.

Step 1. Create a HTTPbin Deployment using kennethreitz/httpbin image:

kubectl -n default create deployment my-httpbin --image=kennethreitz/httpbin

Step 2. Create a Service for Deployment above:

kubectl -n default expose deployment my-httpbin --port=80 --name=my-httpbin-svc

Step 3. Create traefik-values.yaml file (NOTE: This file can be saved anywhere on your local filesystem):

service:
  annotations:
    kubernetes.civo.com/loadbalancer-enable-proxy-protocol: send-proxy
  spec:
    externalTrafficPolicy: Local

additionalArguments:
  - "--entrypoints.web.proxyProtocol.trustedIPs=10.0.0.0/8,172.16.0.0/12,192.168.0.0/16"
  - "--entrypoints.websecure.proxyProtocol.trustedIPs=10.0.0.0/8,172.16.0.0/12,192.168.0.0/16"

Step 4. Install Nginx ingress controller:

helm upgrade --install traefik traefik/traefik --values traefik-values.yaml --namespace traefik --create-namespace

Step 5. Create an Ingress Resource:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: my-httpbin-ingress
  namespace: default
spec:
  rules:
  - http:
      paths:
      - pathType: Prefix
        path: "/"
        backend:
          service:
            name: my-httpbin-svc
            port:
              number: 80

Step 6. Use curl to send traffic to the LoadBalancer's IP/DNS to test (NOTE: You can obtain the IP/DNS in the same fashion as above in step 7 of the Installing NGINX Ingress Controller section of this article):

curl <LB_IP>/ip

Or…

curl <LB_DNS>/ip

Security Implications

When enabling the proxy protocol and exposing real client IPs to applications, several security implications must be carefully considered:

Troubleshooting

Encountering issues during or after setup is not uncommon. Here are some troubleshooting tips for common issues related to LoadBalancer configuration, DNS propagation delays, or errors in applying the YAML files:

LoadBalancer Issues:

• Ensure the LoadBalancer supports the proxy protocol.
• Verify the annotations used for the LoadBalancer service is correctly applied and supported by your cloud provider or infrastructure.
• Check the LoadBalancer's logs for any errors or warnings that might indicate misconfiguration or operational issues.

DNS Propagation Delays:

• Use tools like dig or nslookup to verify that your DNS records are correctly set up and propagating. Remember, DNS propagation can take time, varying from a few minutes to 48 hours.
• Modify your /etc/hosts file (or its equivalent on non-Unix systems) to map the domain name directly to the LoadBalancer's IP for immediate testing.

Errors Applying YAML Files:

• Verify the syntax and integrity of your YAML files. Tools like yamllint can help check for syntax errors.
• Ensure that you have the necessary permissions to apply the configurations. Running kubectl auth can-i <action> <resource> can help identify permission issues.
• Check for deprecated or unsupported Kubernetes API versions in your YAML files, especially if you're using an older version of Kubernetes.

General Troubleshooting:

• Review the logs of your Ingress controller and application pods. These logs can provide valuable insights into what might be going wrong.
• Ensure your Kubernetes cluster is healthy and that all nodes are ready and not experiencing issues.
• Use kubectl describe commands to inspect the state and events related to your deployments, services, and ingress resources for any anomalies or errors.

Summary

Configuring NGINX or Traefik Ingress Controllers on a Civo Kubernetes cluster to preserve source IP is vital for applications that rely on client IP for functionality or security. This tutorial outlines the steps to achieve source IP preservation, including setting up the necessary tools, modifying service annotations, and adjusting ingress resources.

Key security considerations such as IP spoofing risks, firewall adjustments, and privacy implications are highlighted, ensuring that applications handle client IP information securely and in compliance with regulations.

Ultimately, this tutorial provides a clear pathway for configuring your Kubernetes environment on Civo to preserve client IP information, enhancing application functionality and security.

Related Resource

Here are concise resources to enhance your Kubernetes ingress setup on Civo:

• Intro to NGINX Ingress Controller: Start with the basics of setting up and using NGINX as your ingress controller.
• Rate-Limiting with Nginx Ingress: Learn to apply rate limiting to your applications for better performance and security.
• Managing IP Addresses: Essential insights on reserving and managing IP addresses on Civo for a stable ingress configuration.
• Gateway API Tutorial: Delve into the Gateway API for advanced ingress management and source IP handling in complex setups.