Overview and Comparison of Ingress Controllers for Kubernetes
When launching a Kubernetes cluster for a specific application, you should understand what requirements the application itself, business, and developers pose to this resource. If you have this information, you can begin to make an architectural decision and, in particular, to select a specific Ingress controller, of which there are already a large number today. To get a basic idea of the options available without having to study a lot of articles / documentation, etc., we prepared this review by including the main (production ready) Ingress controllers in it.
We hope that it will help colleagues in choosing an architectural solution - at least, will be the starting point for more detailed information and practical experiments. Previously, we studied other similar materials on the network and, oddly enough, did not find a single more or less complete, and most importantly - structured - review. So, fill this gap!
In order to, in principle, make comparisons and get any useful result, you need to understand not just the subject area, but also have a specific list of criteria that will determine the research vector. Without pretending to analyze all possible cases of using Ingress / Kubernetes, we tried to highlight the most general requirements for controllers - be prepared that you will have to study all your specifics and particulars in any case separately.
But I'll start with the characteristics that have become so familiar that they are implemented in all solutions and are not considered:
- dynamic discovery of services (service discovery);
- SSL termination;
- work with websockets.
Now about the comparison points:
One of the fundamental criteria for selection. Your software may not work on standard HTTP, or it may require work on many protocols at once. If your case is non-standard, be sure to take this factor into account so that you do not have to reconfigure the cluster later. For all controllers, the list of supported protocols varies.
There are several application options on which the controller is based. Popular ones are nginx, traefik, haproxy, envoy. In the general case, it may not affect the way traffic is received and transmitted, but it is always useful to know the potential nuances and features of what is “under the hood”.
On the basis of what can we decide on the direction of traffic to a particular service? This is usually host and path, but there are additional features.
Namespace (namespace) - the ability to logically split resources in Kubernetes (for example, on stage, production, etc.). There are Ingress controllers that must be set separately in each namespace (and then it can only direct traffic to the pods of this space). And there are some (and their overwhelming majority) that work globally on the entire cluster - in them traffic is directed to any pod of the cluster, regardless of the namespace.
Samples for upstream
How is traffic directed to healthy instances of the application, services? There are options with active and passive checks, retries, circuit breakers (for more details see them, for example, in the article about Istio ) , custom health check implementations, etc. A very important parameter if you have high requirements for accessibility and timely withdrawal of failed services from the balance.
There are many options: from traditional round-robin to exotic ones like rdp-cookies , as well as some features like sticky sessions .
What authorization schemes does the controller support? Basic, digest, oauth, external-auth - I think that these options should be familiar. This is an important criterion if you use a lot of circuits for developers (and / or just closed ones) that are accessed through Ingress.
Does the controller support such commonly used mechanisms for traffic distribution as canary rollouts, A / B testing, mirroring traffic (mirroring / shadowing)? This is a really sore subject for applications that require accurate and accurate traffic control for productive testing, debugging product errors not in combat (or with minimal losses), traffic analysis, etc.
Is there a paid option for the controller, with advanced functionality and / or technical support?
Graphical User Interface (Web UI)
Is there any graphical interface to control the configuration of the controller? Basically, for “handiness” and / or for those who need to make some changes to the configuration of Ingress, but working with “raw” templates is inconvenient. It may come in handy if developers want to carry out any experiments with traffic on the fly.
The presence of built-in JSON verification of web tokens for authorization and validation of the user to the final application.
Features for config customization
Extensibility of templates in the sense of having mechanisms to add their own directives, flags, etc. to standard configuration templates
Basic DDOS protection mechanisms
Simple rate limit algorithms or more complicated options for filtering traffic based on addresses, white lists, countries, etc.
Opportunities for monitoring, tracking and debugging requests from Ingresss to specific services / pods, and ideally between services / pods too.
Support for application firewall .
The list of controllers was based on the official Kubernetes documentation and this table . We excluded some of them from the review because of their specificity or low prevalence (early stage of development). The remaining ones are discussed below. We start with a general description of the solutions and continue with the pivot table.
Ingress by Kubernetes
License: Apache 2.0
This is the official controller for Kubernetes, which is developed by the community. Obviously from the name, it is based on nginx and is supplemented by a different set of Lua plugins used to implement additional features. Due to the popularity of nginx itself and minimal modifications to it when used as a controller, this option may be the simplest and most understandable configuration average engineer (with experience in the web).
Ingress by NGINX Inc
License: Apache 2.0
Official product of nginx developers. It has a paid version based on NGINX Plus . The main idea is a high level of stability, constant backward compatibility, the absence of any extraneous modules and the declared increased speed (compared with the official controller) achieved due to the rejection of Lua.
The free version is significantly reduced, including even when compared with the official controller (due to the lack of the same Lua-modules). Paid at the same time has a fairly wide additional functionality: real-time metrics, JWT validation, active health checks, and more. An important advantage over NGINX Ingress is full support for TCP / UDP traffic (and in the community version too!). The downside is the lack of features for traffic distribution, which, however, “has the highest priority for developers,” but it takes time to implement.
License: Apache 2.0
Product developed by Kong Inc. in two versions: commercial and free. It is based on nginx, the capabilities of which are expanded by a large number of modules on Lua.
Initially, it was focused on processing and routing API requests, i.e. like the API Gateway, but at the moment it has become a full-fledged Ingress controller. Main advantages: many additional modules (including from third-party developers) that are easy to install and configure and with which a wide range of additional features is implemented. However, the built-in features already offer many features. Work configuration is done using CRD resources.
An important feature of the product - working within the same circuit (instead of cross-namespaced) is a controversial topic: for some it seems a drawback (you have to produce entities for each circuit), and for someone it’s a feature ( a higher level of isolation, because if one controller is broken, then the problem is limited to one circuit only).
Proxy, which was originally created to work with the routing of requests for microservices and their dynamic environment. Hence, many useful features: updating the configuration completely without reboots, supporting a large number of balancing methods, a web interface, forwarding metrics, supporting various protocols, REST APIs, canary releases, and much more. A nice feature is also support for Let's Encrypt certificates out of the box. The disadvantage is that for the organization of high availability (HA), the controller will need to install and connect its own KV-storage.
License: Apache 2.0
HAProxy has long been known as a proxy and traffic balancer. Within the Kubernetes cluster, it offers a “soft” configuration update (without loss of traffic), DNS-based service discovery, dynamic configuration using the API. Full customization of the config template by replacing the CM'a, as well as the possibility of using the functions of the Sprig library in it, can become attractive. In general, the main emphasis of the solution is on high speed, its optimization and efficiency in consumed resources. The advantage of the controller is the support of a record number of different balancing methods.
License: Apache 2.0
Based on HAproxy controller, which is positioned as a universal solution that supports wide capabilities on a large number of providers. The opportunity is proposed for balancing traffic on L7 and L4, and balancing TCP L4 traffic as a whole can be called one of the key features of the solution.
License: Apache 2.0
The basis of this solution was not only laid down by Envoy: it was developed jointly with the authors of this popular proxy. An important feature is the ability to split Ingress resource management using IngressRoute CRD resources. For organizations with many development teams using a single cluster, this helps to maximize the safety of traffic in neighboring circuits and protect them from errors when changing Ingress resources.
It also offers an expanded set of balancing methods (there is mirroring of requests, auto-retry, restriction on the rate of requests and much more), detailed monitoring of traffic flow and failures. Perhaps for some there will be a significant drawback of the lack of support for sticky sessions (although work is already underway ).
License: Apache 2.0 A
comprehensive service mesh solution that is not only an Ingress controller that controls incoming traffic from the outside, but also controls all traffic within the cluster. Under the Hood, Envoy is used as a sidecar proxy for each service. In essence, this is a large combine that “can do anything”, and its main idea is maximum manageability, extensibility, security and transparency. With it, you can fine-tune the routing of traffic, authorization of access between services, balancing, monitoring, canary releases, and much more. Read more about Istio in the Back to Microservices with Istio series of articles .
License: Apache 2.0
Another solution based on Envoy. Has a free and commercial version. It is positioned as “completely native to Kubernetes”, which brings corresponding advantages (tight integration with methods and entities of the K8s cluster).
So, the culmination of the article is this huge table: It is clickable for more detailed viewing and is also available in the Google Sheets format .
The purpose of the article is to provide a more complete understanding (though not completely exhaustive!) Of what choice to make in your particular case. As usual, each controller has its own advantages and disadvantages ... The
classic Kubernetes Ingress is good for its accessibility and provenness, quite rich in capabilities - in general, it should be "eye-catching." However, if there are increased requirements for stability, level of features and development, it is worth paying attention to Ingress with NGINX Plus and a paid subscription. Kong has a rich set of plugins (and, accordingly, the capabilities they provide), and in the paid version there are even more of them. It has ample opportunities for working as an API Gateway, dynamic configuration based on CRD resources, as well as basic Kubernetes services.
With increased requirements for balancing and authorization methods, take a look at Traefik and HAProxy. These are Open Source projects, proven over the years, very stable and actively developing. Contour has been around for a couple of years, but it still looks too young and has only basic features added on top of Envoy. If there are requirements for the presence / embedding of WAF before the application, you should pay attention to the same Ingress from Kubernetes or HAProxy.
And the richest in features are products built on the basis of Envoy, especially Istio. It seems to be a complex solution that “can do anything”, which, however, also means a significantly higher threshold for entering configuration / launch / administration than other solutions.
We have selected and still use Kubernetes Ingress, which covers 80-90% of the needs, as the standard controller. It is quite reliable, easy to configure, expand. In the general case, in the absence of specific requirements, it should suit most clusters / applications. Of the same versatile and relatively simple products, Traefik and HAProxy can be recommended.
Read also in our blog: