SQR-111

Gateway migration for Gafaelfawr#

Abstract

Gafaelfawr, the identity management and access control system for Phalanx environments including the Rubin Science Platform, is currently tightly integrated with ingress-nginx. This ingress provider will no longer be maintained after March 2026, and Kubernetes is encouraging a migration to the new Gateway API. This tech note outlines the planned transition process and necessary changes for Gafaelfawr.

Note

This is part of a tech note series on identity management for the Rubin Science Platform. The other two primary documents are DMTN-224, which describes the implementation; and SQR-069, which provides a history and analysis of the decisions underlying the design and implementation. See the references section of DMTN-224 for a complete list of related documents.

Problem statement#

Gafaelfawr is the identity management and access control system for all Phalanx environments including the Rubin Science Platform. It currently runs primarily as an NGINX auth_request handler and is tightly integrated with the ingress-nginx ingress controller used by Phalanx. Incoming web requests to any Phalanx web service generate subrequests to Gafaelfawr, which then does authentication, authorization, and adjusts the HTTP headers of the request. This integration is done via GafaelfawrIngress custom Kubernetes resources, which the Gafaelfawr Kubernetes operator converts into Ingress resources with the appropriate ingress-nginx-specific configuration.

Development of ingress-nginx is coming to an end. Feature development has already ceased. After March of 2026, it will no longer receive security fixes. Phalanx therefore must migrate to a different software stack for handling incoming requests, and Gafaelfawr must be updated to integrate with that new stack.

Additionally, the Kubernetes Ingress API has been frozen in favor of the new Gateway API, and the Kubernetes project recommends all users switch to the new API. Since replacement of ingress-nginx will require a significant migration project, ideally the migration to the new Gateway API would be done at the same time and avoid the need for another subsequent large migration.

Integration points#

Gafaelfawr relies on the following integration points with ingress-nginx. Each will have to be converted to the Gateway API and a new gateway controller or, if that is not possible, redesigned.

  • Incoming web requests must first be sent to Gafaelfawr URL (the parameters of which will vary by target service) and the request must be rejected and not sent to the backend if Gafaelfawr returns failure.

  • All headers in the incoming web request must be sent to Gafaelfawr.

  • Gafaelfawr must be able to return its choice of status codes. All of 400, 401, 403, and 429 are currently returned in some situations, and more may be needed in the future. Gafaelfawr must be able to include a body and arbitrary headers in these responses. The current integration with ingress-nginx allows this but is very awkward.

  • Gafaelfawr must be able to inject additional request headers that are added to the request sent to the backend after approval.

  • The ingress or gateway controller handling the request must be able to cache the Gafaelfawr response for up to five minutes, but not use a cached response if the method or the Authorization or Cookie request headers have changed. This is used to reduce authentication traffic for services with large numbers of requests from the same user in short intervals, such as Nublado, the Portal, or the Butler server.

  • Gafaelfawr must be able to inject additional response headers that are added to the response from the backend before it is sent to the user. This is used for rate limiting status headers and will eventually be used to comply with the IVOA authentication standard.

Other Phalanx applications rely on the following integration points with ingress-nginx, unrelated to Gafaelfawr.

  • The Portal relies on cookie-based session affinity to consistently route a user to the same instance of the Portal.

  • The Portal and the CADC TAP server rely on proxy URL rewriting to adjust the URLs returned by the backend application before returning them to the user.

  • Multiple Phalanx applications rely on URL rewriting to present a different URL to the backend service than the URL the client sent. I believe all instances of this only remove, add, or replace a static path prefix.

  • Some third-party applications may only support Ingress resources and not Gateway resources. Any ingress-nginx replacement should therefore be able to serve Ingress resources, but does not need to support Gafaelfawr integration with an Ingress.

Migration plan#

The high-level migration steps are:

  1. Test promising candidates for a Gateway API implementation by installing them in a development cluster parallel to ingress-nginx using a separate external IP and testing them with hand-crafted Gateway resources. Muster exists for this purpose; its ingresses cover a wide range of expected Gafaelfawr functionality, and mobu can run verification tests against Muster.

  2. Choose a Gateway API implemenation.

  3. Add support to Gafaelfawr’s Kubernetes operator for generating Gateway resources and any associated resources, such as the custom middleware configuration required to send external authentication requests to the correct Gafaelfawr endpoint. Initially, this should accept GafaelfawrIngress resources, if possible, and transform them into equivalent Gateway resources. This will probably require adding a new field to GafaelfawrIngress to select the type of resource to generate so that we can enable this one-by-one. If that proves too complex, we can go directly to a new GafaelfawrGateway resource and corresponding Gafaelfawr implementation.

  4. Add any necessary additional support to Gafaelfawr required by the chosen Gateway API implementation. We may also have to further optimize a hot path for Gafaelfawr authorization requests if the chosen Gateway API doesn’t support caching.

  5. Add Phalanx support for choosing between ingress-nginx and the new Gateway API, and write migration steps for how to convert an existing cluster.

  6. Migrate Phalanx environments to the new implementation, refining the migration steps with each conversion.

  7. Add Gafaelfawr support for a new GafaelfawrGateway resource, if that was not required earlier.

  8. Convert all GafaelfawrIngress resources in Phalanx to GafaelfawrGateway. Also convert as many third-party chart ingresses to gateways as possible, based on whether upstream supports gateways.

USDF will require special attention because they currently deploy a separate ingress-nginx service outside of the vCluster clusters that are visible to Phalanx environment administrators. As part of this migration, this practice should stop for Phalanx-managed clusters and the gateway API implementation should be managed by Phalanx as it is in any other Phalanx environment.

Gateway API options#

The Kubernetes project maintains a list of implementations. Below are evaluation notes for some of the options.

Unfortunately, I have not been able to find a gateway API implementation that supports caching of external authentication results the way that ingress-nginx did. This means none of the implementations discussed below meet our basic requirements, and we may have to find other workarounds. See Gaps for more details.

Based on an initial review, Envoy appears to be the best option, although this will require a significant redesign of Gafaelfawr if we want to use the GRPC protocol. Traefik could be made to work, but we would lose the ability to inject headers into the response. The other options I evaluated looked less interesting.

Traefik#

Traefik supports external auth via gateway middleware (ForwardAuth) and a Traefik-specific resource. On failure, the full response from the external authentication provider is returned to the client, which is exactly the behavior we want. URL rewriting is supported with middleware (ReplacePathRegex and StripPathRegex, among others).

Traefik interestingly has some built-in support for recognizing ingress-nginx annotations on Ingress resources and honoring them as much as Traefik is able. Unfortunately, this support is quite limited and does not include many of the annotations we use.

Traefik appears to be missing the following necessary features:

  • Traefik does not support caching of external authentication results.

  • There does not appear to be a way to inject headers from the external authentication provider into the response, only into the request. This would mean loss of the rate limit headers on successful requests, unless we added code to every backend to mirror those headers in replies. It will also complicate complying with the IVOA authentication requirement to include a response header with the authenticated username.

  • There does not appear to be any support for rewriting URLs returned by the backend before sending them to the client.

Envoy#

The Envoy gateway is a controller that implements the Gateway protocol by dynamically configuring the Envoy proxy. The bare proxy supports everything we need (except auth caching), but the gateway operator does not yet expose all of the necessary config to the Kubernetes API.

External auth is configured by using the ext_authz filter. This filter enables calling out to either an HTTP or GRPC-based auth server and manipulating the request to the backend and the response to the client based on the response from the auth server.

Both the gRPC protocol for external auth and the HTTP configuration support all of the auth features we need, but the gateway API does not expose all of the necessary config. This is not a problem with the gRPC interface because all of the necessary info is in the gRPC messages themselves, and nothing needs to be configured in the proxy. Gafaelfawr could gain support for the gRPC protocol, and in some ways it would be an improvement over the way external auth is currently handled. For example, only one endpoint would be required, since all of the necessary information is in the request body. Gafaelfawr could maintain its own internal database of authorization rules, based on gathered Kubernetes resources, avoiding the need to create a separate external auth rule for every ingress or gateway. This would require a substantial redesign of Gafaelfawr’s request flow, however.

Table 1 HTTP auth proxy support#

Feature

Proxy Support

Gateway Support

Denied: response code

yes

yes

Denied: body

yes

yes

Denied: headers to client

yes

yes [1]

Accepted: headers to backend

yes

yes

Accepted: headers to client

yes

no [2]

Redirect to login

no

no [3]

Cache auth responses

no

no

One other way The HTTP option is incompatible with Gafaelfawr is that it always appends the path of the request to the auth server path. For example, if the client request is to /somepath, and the auth proxy is configured to request /authpath?foo=bar, then the actual auth server path requested is /authpath?foo=bar/somepath. We would likely be able to disable this path appending soon, based on this GitHub issue and associated PR.

Envoy supports URL rewriting via the standardized HTTPURLRewriteFilter approach, which is cleaner than Traefik’s use of custom middleware. This approach is more limited in what it can do, but I believe it will satisfy our requirements.

Envoy appears to be missing the following necessary features:

Unlike some of the other new gateways, Envoy is written in C++, rather than in a memory-safe language such as Go. This is not as bad as being written in C, but it somewhat increases the risk of security vulnerabilities.

Kgateway#

Kgateway is, so far as I can tell, a wrapper around Envoy with a lot of tacked-on AI marketing buzzwords. It appears to have all of the same tradeoffs as Envoy for Phalanx. None of the advantages over Envoy appear to be relevant for our use case.

The documentation is quite nice and easy to follow, though, so it may be worth a closer look.

NGINX Gateway Fabric#

NGINX Gateway Fabric is the NGINX-driven Gateway API implementation maintained by the NGINX maintainers themselves. It therefore in theory would support everything that we are currently doing with ingress-nginx, since the full capabilities of NGINX are available.

Unfortunately, very few of the features that we need appear to be supported out of the box. Unlike ingress-nginx, which provided simple annotation-driven configuration to enable NGINX features, it looks like using NGINX Gateway Fabric would require we write and maintain most of the low-level NGINX configuration ourselves and inject it with their SnippetsFilter API. Given the complexity of NGINX configuration, this is rather unappealing.

So far as I can tell, this includes all support for external auth. There does not appear to be any native support for setting it up, so the full backend configuration and auth_request block would have to be written directly in the NGINX configuration language and injected via SnippetsFilter.

NGINX is written in C and does not have a great security track record. Ideally, this replacement project would let us migrate away from an ingress or gateway written in a memory-unsafe language prone to security issues.

HAProxy#

HAProxy has an initial implementation of the Kubernetes Gateway API, but it is appears to be incomplete, largely undocumented, and possibly unmaintained. I was unable to determine whether it supported any of the features we need.

Gaps#

The two main gaps in the available Gateway API implementations appear to be response URL rewriting and caching of external authentication replies.

Response URL rewriting#

Currently, the Portal and the CADC TAP server both rely on ingress-nginx’s ability to rewrite Location headers in responses from the backend. This is configured with the proxy-redirect-from and proxy-redirect-to annotations. So far as I can tell, none of the Gateway API implementations support this.

Given that we have contact with and some influence over the implementation of both software packages, we should hopefully be able to eliminate the need for this type of rewriting. The application needs to take a configuration option specifying the base path for its URLs and use that configuration when constructing URLs in Location headers. See, for example, how Argo CD supports this via the server.basehref and server.rootpath configuration parameters.

External authentication caching#

Nublado, the Portal, and the Butler server rely on caching of external authentication responses from Gafaelfawr for up to five minutes. Without this support, every resource request to those services requires a request to Gafaelfawr, which can mean rather high load on Gafaelfawr and on the gateway given the number of resource requests. For Nublado and Portal, this means an extra request for every JavaScript AJAX request. For Butler, uncached authorization checks caused noticable performance degredation for common use cases before caching was enabled.

There does not appear to be a straightforward path to resolve this gap. Neither the Traefik nor the Envoy projects appear interested in adding support, although the Traefik developers did indicate they would look at community PRs.

We could attempt to optimize the Gafaelfawr request rather than avoid it. Currently, every authorization check requires a call to Redis. That call, at least, could be avoided by adding an additional in-memory cache similar to that used in ingress-nginx currently, using the method and the Authorization and Cookie headers as keys. It’s not clear whether that would be sufficient to make an approach without gateway-side caching viable, since each request would still have to pay the cost of parsing the Gafaelfawr request in Python.

Using Envoy does open up some additional possibilities since with Envoy the external auth request could be a gRPC request. We would have to experiment to see if those requests were more efficient; they might be, since the parsing of the request is done via protobuf and should be faster than the Python-based parsing of conventional HTTP query parameters.

In the most extreme case, it may be possible to implement a caching layer for the external authorization check in another, faster programming language (Rust or Go) with gRPC support, and have it call out to the Gafaelfawr Python backend only when needed. This is a lot of additional complexity, however, and should only be considered if the lack of caching causes unacceptable performance degredation and we can’t find a better alternative.

It is possible to set up a cache between the gateway and Gafaelfawr, via a separate Vinyl cache instance or something similar, but this is very unappealing from a complexity standpoint. Performance of a Vinyl cache as a sidecar container in the Gafaelfawr pod seems very close to a native NGINX cache, despite the extra network hop involved. There is a slightly odd pattern in the data where requests-per-second goes higher and lower every ~60s, but this pattern is present in the no-cache tests too. You can see this in graphs of metrics over time by opening the files in the auth-cache-test-results/2026-03-16-120408-40-users-600s-duration directory in a browser.

These are data from 40 simulated users hitting Muster endpoints as fast as they can for 10 minutes: (“vus” is virtual users):

Listing 1 No auth cache#
 EXECUTION
 iteration_duration.............: avg=149.04ms min=29.83ms med=137.41ms max=876.41ms p(90)=196.48ms p(95)=204.26ms
 iterations.....................: 161010 268.292831/s
 vus............................: 40     min=40          max=40
 vus_max........................: 40     min=40          max=40
Listing 2 Vinyl cache#
 EXECUTION
 iteration_duration.............: avg=66.17ms  min=21.32ms med=43.13ms max=1.46s    p(90)=100.17ms p(95)=104.54ms
 iterations.....................: 362524 604.174278/s
 vus............................: 40     min=40          max=40
 vus_max........................: 40     min=40          max=40
Listing 3 NGINX cache#
 EXECUTION
 iteration_duration.............: avg=65.42ms  min=19.91ms med=42.25ms max=1.21s    p(90)=100.01ms p(95)=104.17ms
 iterations.....................: 366684 611.103082/s
 vus............................: 40     min=40          max=40
 vus_max........................: 40     min=40          max=40

Migrate to another Ingress controller#

Another option is to migrate to another Ingress-based controller that will remain supported, instead of going all the way to a Gateway-based controller. Even though the Ingress API has been frozen, there are no plans to deprecate or remove it. If it is significantly easier, we may want to migrate to another Ingress-based controller first so we can at least be using something that is still getting security patches.

NGINX Ingress#

The NGINX Ingress controller is the official NGINX ingress controller maintained by F5, the owners of NGINX. It theoretically supports all of the functionality of ingress-nginx because it too uses NGINX as the proxy.

Unfortunately, it has the same problem as the NGINX Gateway Fabric: very few of the features that we need appear to be supported out of the box. We would be required to write and maintain explicit NGINX configuration ourselves.