Daniel Watrous on Software Engineering

A Collection of Software Problems and Solutions

Software Engineering

Istio Ingress vs. Kubernetes Ingress

For years I have appreciated the clean and simple way Kubernetes approached Ingress into container workloads. The idea of an IngressController that dynamically reconfigures itself based on the current state of Ingress resources seemed very clean and easy to understand. Istio, on the other hand, felt more confusing, so I set out to correlate what I refer to as “traditional kubernetes ingress” with Istio ingress. The following diagram will help visualize my comments below.

Dynamic Ingress Control

Load Balancer at the Edge

Both approaches are very similar in how they treat traffic at the edge. The demonstrations usually attempt to bypass DNS management and use something like a NodePort for convenience. In general practice, the edge traffic will arrive at a Load Balancer (think ELB or Google Cloud Load Balancing) that will distribute traffic across multiple instances of the IngressController (traditional) or Istio Ingress Gateway. A kubernetes Service defines the Load Balancer and associates it with the IngressController/Istio Ingress Gateway. With this scheme, the Load Balancer will always dynamically reconfigure based on changes to the IngressController/Istio Ingress Gateway Pods.

DNS and decision making

In my experience, a combination of wildcard DNS and API automation of DNS entries is used to manage resolution for workloads coming in to the cluster. The LoadBalancer Service configures the load balancer to pass all traffic on ports 80 and 443 through to the IngressController/Istio Ingress Gateway. In this sense it becomes a dumb LoadBalancer which leaves all decisions about what to do with a request to the IngressController/Istio Ingress Gateway.

Proxy technologies

There are many implementations of the IngressController spec for the traditional kubernetes routing, such as nginx, haproxy and traefik, to name a few. These come with various features (e.g. Let’s Encrypt integration), but all of them satisfy the specification that requires them to be aware of all Ingress resources and route traffic accordingly.

The Istio Ingress Gateway exclusively uses Envoy. Whether this lack of choice is a problem for you will depend on your specific use cases, but Envoy is a solid, very fast proxy that is battle tested by some of the biggest sites in the world.

Automatic (re)configuration

Both approaches implement a type of server-side Service Discovery pattern. The proxy monitors kubernetes resources and configures and reconfigures itself. The diagram below illustrates how this works.

Although the Istio ingress mechanism is more complicated with three possible kubernetes resources contributing to the Envoy configuration, the overall approach is almost identical. A mechanism in the ingress proxy observes changes to either the Ingress or Gateway, VirtualService and DestinationRule resources. When changes are observed relative to the current configuration, the configuration is updated. Each of the numbered steps above are continuous.

TLS

Both solutions accommodate TLS certificates at two levels. The first level is at the IngressController (at least this is true with nginx) and Istio Ingress Gateway. The second level is with the IngressController or Gateway. Both solutions make use of a kubernetes Secret to store the TLS certificate and key. However, they work differently in practice, so I’ll provide a description for each solution below.

IngressController (nginx)

The nginx IngressController defines a base/default TLS certificate for all Ingress resources that don’t define their own. If a TLS certificate is not provided, a fake is used. Some IngressController implementations also integrate with Let’s Encrypt. In my case I created a custom Let’s Encrypt automation for kubernetes which could work with any Ingress Controller, or even Istio Ingress Controller.

The Ingress resource can override the default TLS certificate by referencing an a different kubernetes Secret. When this happens, the Ingress specific Secret is mounted into the IngressController and added to the configuration for that route.

Istio Ingress Gateway

The Istio Ingress Gateway also consumes a secret, but it is based on a naming convention and must exist in the same namespace that hosts the Istio Ingress Gateway. Unlike the IngressGateway, there is not default use of TLS. Instead it must be defined in every Gateway resource, and that definition is a path which is determined by convention. More details available here, and here. Adding a TLS route to the Gateway and VirtualService can be seen in the links to those resource definitions above.

In the documentation, it seems the Gateway can reference a kubernetes Secret to override the TLS secret, but I wasn’t able to make this work. I’ll update this post if I eventually succeed.

Service used for endpoints

When using an IngressController, traffic is routed to a Service, which load balances traffic across available Pods. In the case of the Istio Ingress Gateway, the kubernetes Service is only used to get a list of endpoints (Pods). Requests are then send directly to the Envoy proxy in the Pod, bypassing the Service.

Leave A Comment

This site uses Akismet to reduce spam. Learn how your comment data is processed.