Kubernetes Networking: CNI, Cilium with eBPF and Network Policy
Kubernetes abstracts the network in a masterly way: each Pod gets a routable IP address, containers within the same Pod share the network namespace, and Pods can communicate with each other on any node without NAT. Yet, the apparent simplicity hides a real complexity: how does this model actually work? Who deals with assign IPs, route traffic between nodes, implement network policies?
The answer lies in Container Network Interface (CNI), a standard that defines how networking plugins should integrate with Kubernetes. In this article we'll explore the Kubernetes networking model from the inside: how kube-proxy works, why Cilium with eBPF is it replacing traditional solutions, and how implement Network Policy to isolate workloads in production clusters.
What You Will Learn
- The Kubernetes network model: IP-per-Pod, flat network, no NAT
- How the Container Network Interface (CNI) works and its main plugins
- The difference between kube-proxy (iptables/IPVS) and Cilium with eBPF
- Because eBPF brings -30% latency and +60% throughput compared to iptables
- How to install and configure Cilium as a CNI
- Network Policy: syntax, practical examples, default-deny and namespace isolation
- CiliumNetworkPolicy for Layer 7 rules (HTTP, gRPC, Kafka)
The Kubernetes Network Model
Kubernetes imposes a networking model with four fundamental requirements that any CNI implementation must comply with:
- All Pods can communicate with all other Pods without NAT
- All nodes can communicate with all Pods without NAT
- The IP that a Pod sees as its own and the same IP that others use to reach it
- Containers within a Pod share network namespaces and IPs
This "flat network model" greatly simplifies application reasoning: a microservice does not need to know whether the service it calls is on the same node or on a node remote. The complexity is moved to the cluster network layer.
How Communication Between Pods Works
When a Pod A wants to communicate with a Pod B on different nodes, the typical flow is the following with a solution based on overlay network (e.g. VXLAN):
- The package exits Pod A's container through the interface
eth0 - Enter the node namespace through a
veth pair - The CNI plugin intercepts it and encapsulates it in VXLAN (or GRE, Geneve...)
- The packet traverses the physical network to the Pod B node
- The CNI on the destination node de-encapsulates the packet
- The packet arrives at Pod B through its veth pair
With solutions based on BGP or native routing (like Cilium in native routing mode), encapsulation is not necessary and performance improves significantly.
Container Network Interface (CNI)
The CNI is a CNCF specification that defines how container runtimes should invoke network plugins. Kubernetes uses CNI to delegate network management to plugins: when the kubelet creates a Pod, it calls the configured CNI plugin to allocate the address IP and configure the network interface.
Main CNI Plugins
| Plugins | Technology | Network Policy | L7 Policy | Use Case |
|---|---|---|---|---|
| Flannel | VXLAN overlay | No (native) | No | Development, simple clusters |
| Calico | BGP/overlay | Si | No | On-premise, performance |
| Cilium | eBPF | Si | Yes (HTTP, gRPC) | Production, service mesh |
| AWS VPC CNI | ENI native | Yes (SG) | No | EKS |
| Azure CNI | Native VNet | Yes (NSG) | No | AKS |
kube-proxy: The Old Approach
kube-proxy is the Kubernetes component responsible for implementing the Services: when a client calls a Service, kube-proxy ensures that the traffic arrives at one of the Pod backend. Traditionally uses iptables for this purpose.
The problem with iptables is that complexity scales linearly with the number of rules. In a cluster with 10,000 Services and 100,000 endpoints, iptables manages millions of rules. Every packet must pass through this chain of rules, with significant impact on the latency and CPU of the node.
The in-tree alternative e IPVS (IP Virtual Server), which uses a hash table for O(1) lookup instead of iptables linear scan. But IPVS also has limitations: It does not support advanced policies and still requires the management of additional iptables rules.
iptables Scaling Problem
With 10,000 Services, kube-proxy with iptables creates around 40,000 rules for Services alone. The update time of these rules grows from milliseconds to minutes. Clustered large, this leads to high latencies during scaling events and deployment updates. This is one of the main reasons why Cilium is replacing kube-proxy.
Cilium and eBPF: The Future of Kubernetes Networking
eBPF (extended Berkeley Packet Filter) is a Linux kernel technology which allows you to run sandboxed programs directly in the kernel, without modifying it e without kernel modules. Cilium uses eBPF to intercept and process network traffic at the kernel level, completely bypassing iptables and kube-proxy.
Benefits of Cilium with eBPF
- Performance: Latency reduced by up to 30%, throughput increased by 60% compared to iptables
- Scalability: O(1) lookup with BPF maps instead of iptables linear scan
- Observability: Hubble provides real-time visibility of L3/L4/L7 traffic
- Layer 7 Policy: Policies based on HTTP path, method, header, gRPC method, Kafka topic
- Service Mesh without sidecar: mTLS and load balancing implemented in the kernel, zero sidecar overhead
- Kube proxy replacement: Cilium can completely replace kube-proxy
Cilium installation
We install Cilium on a Kubernetes cluster using Helm, configuring it to replace kube-proxy and enable Hubble for observability:
# Aggiungi il repo Helm di Cilium
helm repo add cilium https://helm.cilium.io/
helm repo update
# Installa Cilium con kube-proxy replacement e Hubble abilitati
helm install cilium cilium/cilium \
--version 1.16.0 \
--namespace kube-system \
--set kubeProxyReplacement=true \
--set k8sServiceHost=API_SERVER_HOST \
--set k8sServicePort=API_SERVER_PORT \
--set hubble.relay.enabled=true \
--set hubble.ui.enabled=true \
--set ipam.mode=kubernetes
# Verifica installazione
cilium status --wait
# Verifica connettivita
cilium connectivity testCilium in Production: Advanced Configuration
For a production cluster, here is a complete Helm values configuration with native routing (without VXLAN encapsulation) to maximize performance:
# cilium-values-production.yaml
kubeProxyReplacement: true
k8sServiceHost: "10.0.0.1" # indirizzo API server
k8sServicePort: "6443"
# Native routing mode (senza overlay VXLAN)
# Richiede che la rete sottostante supporti il routing dei pod CIDR
tunnel: disabled
autoDirectNodeRoutes: true
ipv4NativeRoutingCIDR: "10.244.0.0/16"
# IPAM
ipam:
mode: kubernetes
# BGP per annunciare i pod CIDR ai router
bgp:
enabled: true
announce:
podCIDR: true
lbIP: true
# Hubble - osservabilita Layer 7
hubble:
enabled: true
metrics:
enabled:
- dns:query;ignoreAAAA
- drop
- tcp
- flow
- icmp
- http
relay:
enabled: true
replicas: 2
ui:
enabled: true
# Monitoring con Prometheus
prometheus:
enabled: true
serviceMonitor:
enabled: true
# Encryption (WireGuard)
encryption:
enabled: true
type: wireguard
# Load Balancing con DSR (Direct Server Return)
loadBalancer:
mode: dsr # elimina un hop di rete nel path di ritorno
# Limita connessioni per Pod
bpf:
mapDynamicSizeRatio: 0.0025Network Policy in Kubernetes
By default, all Pods in a Kubernetes cluster can freely communicate with each other. This is convenient for development but is a security issue in production. The NetworkPolicy allow you to define entry and egress rules for Pods.
Attention: CNI Plugin Required
NetworkPolicies are a Kubernetes resource, but their implementation depends on the CNI plugins. Flannel does not support NetworkPolicy natively. To use them, you need Cilium, Calico, or another CNI that implements them. The resource is accepted by the server API but ignored if the CNI does not support it.
Default Deny: The Fundamental Best Practice
The first step to isolate workloads and apply a policy default-deny on each namespace. This denies all traffic not explicitly authorized:
# default-deny-all.yaml
# Nega tutto il traffico ingress e egress nel namespace
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-deny-all
namespace: production
spec:
podSelector: {} # seleziona TUTTI i pod nel namespace
policyTypes:
- Ingress
- Egress
---
# Permetti il traffico DNS (necessario per la risoluzione dei nomi)
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-dns
namespace: production
spec:
podSelector: {}
policyTypes:
- Egress
egress:
- ports:
- protocol: UDP
port: 53
- protocol: TCP
port: 53NetworkPolicy for a Typical Application
Let's see how to isolate a three-tier application: frontend, backend, database. Only the frontend accepts traffic from outside, only the backend can reach the DB:
# network-policies-app.yaml
# Frontend: accetta traffico dall'ingress controller
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-ingress-to-frontend
namespace: production
spec:
podSelector:
matchLabels:
app: frontend
policyTypes:
- Ingress
- Egress
ingress:
- from:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: ingress-nginx
podSelector:
matchLabels:
app.kubernetes.io/name: ingress-nginx
ports:
- protocol: TCP
port: 8080
egress:
- to:
- podSelector:
matchLabels:
app: backend
ports:
- protocol: TCP
port: 8000
- ports:
- protocol: UDP
port: 53
---
# Backend: accetta solo dal frontend, puo raggiungere DB
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-frontend-to-backend
namespace: production
spec:
podSelector:
matchLabels:
app: backend
policyTypes:
- Ingress
- Egress
ingress:
- from:
- podSelector:
matchLabels:
app: frontend
ports:
- protocol: TCP
port: 8000
egress:
- to:
- podSelector:
matchLabels:
app: database
ports:
- protocol: TCP
port: 5432
- ports:
- protocol: UDP
port: 53
---
# Database: accetta solo dal backend
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-backend-to-database
namespace: production
spec:
podSelector:
matchLabels:
app: database
policyTypes:
- Ingress
ingress:
- from:
- podSelector:
matchLabels:
app: backend
ports:
- protocol: TCP
port: 5432CiliumNetworkPolicy: Policy Layer 7
Standard Kubernetes NetworkPolicies operate at Layer 3/4 (IP and port). Cilium extends this with CiliumNetworkPolicy, which allows content-based policies of communication: HTTP path, gRPC method, Kafka topic, DNS query.
HTTP Policy with Cilium
We allow the frontend to call only specific backend endpoints:
# cilium-http-policy.yaml
apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
name: frontend-to-backend-l7
namespace: production
spec:
endpointSelector:
matchLabels:
app: backend
ingress:
- fromEndpoints:
- matchLabels:
app: frontend
toPorts:
- ports:
- port: "8000"
protocol: TCP
rules:
http:
- method: "GET"
path: "/api/v1/products.*"
- method: "POST"
path: "/api/v1/orders"
- method: "GET"
path: "/health"DNS Policy with Cilium
Limit the DNS domains that a Pod can resolve (useful for preventing data exfiltration):
# cilium-dns-policy.yaml
apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
name: restrict-dns-egress
namespace: production
spec:
endpointSelector:
matchLabels:
app: backend
egress:
# Permetti solo DNS verso il cluster DNS
- toEndpoints:
- matchLabels:
k8s:io.kubernetes.pod.namespace: kube-system
k8s:k8s-app: kube-dns
toPorts:
- ports:
- port: "53"
protocol: ANY
rules:
dns:
- matchPattern: "*.internal.company.com"
- matchPattern: "*.svc.cluster.local"
- matchPattern: "api.stripe.com"Hubble: Network Observability
Hubble and the Cilium observability layer. Provides real-time visibility on all cluster network traffic, with communications identities based on Kubernetes labels instead of IP addresses.
# Installa il CLI di Hubble
export HUBBLE_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/hubble/master/stable.txt)
curl -L --fail --remote-name-all \
https://github.com/cilium/hubble/releases/download/$HUBBLE_VERSION/hubble-linux-amd64.tar.gz
tar xzvf hubble-linux-amd64.tar.gz
sudo mv hubble /usr/local/bin
# Port-forward al relay Hubble
cilium hubble port-forward &
# Osserva il traffico in tempo reale
hubble observe --namespace production --follow
# Filtra per Pod specifici
hubble observe \
--namespace production \
--from-pod frontend-7d9d6b8f-abc12 \
--to-pod backend-5c4f8d9-xyz99 \
--follow
# Mostra solo i drop (traffico bloccato dalle policy)
hubble observe \
--namespace production \
--verdict DROPPED \
--follow
# Statistiche per service
hubble observe \
--namespace production \
--output json | jq '.flow.destination.namespace'Network Policy Debugging and Troubleshooting
Debugging NetworkPolicies is one of the most common challenges in Kubernetes. Here's one approach systematic:
# 1. Verifica quali NetworkPolicy si applicano a un Pod
kubectl get networkpolicies -n production -o wide
# 2. Test di connettivita con un Pod temporaneo
kubectl run test-pod \
--image=nicolaka/netshoot \
--rm \
-it \
--restart=Never \
-n production \
-- bash
# All'interno del Pod:
# Test TCP
nc -zv backend-service 8000
# Test DNS
nslookup backend-service.production.svc.cluster.local
# Test HTTP
curl -v http://backend-service:8000/health
# 3. Con Cilium, usa il tool di policy verification
cilium policy get # mostra tutte le policy caricate
# 4. Testa la connettivita specifica
kubectl exec -n production frontend-pod -- \
curl -v http://backend-service:8000/api/v1/products
# 5. Con Hubble, vedi perche un pacchetto viene droppato
hubble observe \
--namespace production \
--verdict DROPPED \
--from-pod frontend-7d9d6b8f \
--followMulti-Namespace Isolation
In multi-tenant clusters, it is critical to isolate namespaces. By default, NetworkPolicies they do not block cross-namespace traffic. Here's how to implement complete isolation:
# Isola completamente un namespace da tutti gli altri
# (ma permette il traffico interno al namespace)
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: namespace-isolation
namespace: tenant-a
spec:
podSelector: {}
policyTypes:
- Ingress
- Egress
ingress:
# Permetti solo traffico dallo stesso namespace
- from:
- podSelector: {}
# Permetti da monitoring namespace
- from:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: monitoring
egress:
# Permetti solo traffico verso lo stesso namespace
- to:
- podSelector: {}
# Permetti DNS
- ports:
- protocol: UDP
port: 53
# Permetti verso monitoring namespace
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: monitoringPerformance Benchmark: Cilium vs kube-proxy
Benchmarks show significant differences between Cilium with eBPF and kube-proxy with iptables in clusters with a high number of Services:
| Metric | kube-proxy iptables | Cilium eBPF | Improvement |
|---|---|---|---|
| P50 latency (10K svc) | 450 µs | 130 µs | -71% |
| P99 latency (10K svc) | 2.1 ms | 320 µs | -85% |
| Throughput (Gbps) | 22Gbps | 36Gbps | +64% |
| CPU update rules (10K svc) | 180 sec | 2 sec | -99% |
| Connections/sec | 220K | 380K | +73% |
Best Practices for Kubernetes Networking
Production Networking Checklist
- Choose the right CNI now: Migare CNI in production and complex. Consider Cilium if you plan on advanced network policies or service mesh
- Default-deny in every namespace: Always start with a deny everything policy, then add exceptions
- Consistent labels on Pods: NetworkPolicies depend on labels; use clear conventions (app, tier, version)
- Test policies before deploying: USA
cilium connectivity testor test pods to check - Enable Hubble: In production, traffic visibility is critical for debugging and compliance
- Track drops: Configure alerts on Prometheus for unexpected dropped traffic
- Don't block DNS: Always remember to allow UDP/TCP 53 to kube-dns in your egress policies
- Document policies: Use Kubernetes annotations to describe the purpose of each NetworkPolicy
Anti-Patterns to Avoid
Common Errors with NetworkPolicies
- Too lax policies: Use
namespaceSelector: {}without matchLabels allows traffic from ALL namespaces, including compromised ones - Forgetting DNS: If you block all egress and forget port 53, your Pods will no longer resolve any hostnames
- Labels not updated: If you add new Pods without the correct labels, NetworkPolicies do not protect them
- Tests in development only: NetworkPolicies can block unexpected traffic in production. Always test in staging with realistic traffic
- Use IP instead of selectors: Pod IPs change; always use podSelector and namespaceSelector, never ipBlock for internal traffic
Conclusions and Next Steps
The Kubernetes networking model, with its flat, IP-per-Pod approach, is elegant in design its simplicity yet sophisticated in implementation. Choosing the CNI plugin is one of the most important architectural decisions for a production cluster: influence the performance, security and observability features available.
Cilium with eBPF is the most advanced CNI available today: replace kube-proxy with superior performance, offers Layer 7 policies, integrates observability with Hubble and can replace a traditional service mesh for mTLS. For new clusters in production, and the choice recommended by the CNCF community and large players such as Google, Amazon and Microsoft who use it in their managed Kubernetes services.
Network policies, implemented correctly with a default-deny approach, reduce the attack surface drastically if a Pod is compromised. I'm not optional in production: they are a fundamental safety requirement.
Upcoming Articles in the Kubernetes at Scale Series
Related Series
- MLOps and Machine Learning in Production — GPU workloads on Kubernetes
- Platform Engineering — Internal Developer Platforms on K8s
- Observability and OpenTelemetry — cluster monitoring