Langfuse and JWT for LLM: proxy solution for corporate systems

Integrating Langfuse with LLM via JWT: Solving the Dynamic Authorization Problem

Langfuse is a powerful tool for tracing LLM requests, managing prompts, and evaluating models. However, in enterprise environments, a critical issue arises: the system requires dynamic JWT tokens for access to the inference gateway, while Langfuse only supports static API keys. This mismatch makes direct integration impossible. The solution is to create a thin proxy service that converts static authorization to dynamic.

Let's examine an architectural solution compatible with OpenAI-like APIs. It maintains transparency for Langfuse while meeting enterprise infrastructure security requirements.

Proxy Architecture: Bridging the Gap Between Static and Dynamic

The core problem is a clash of paradigms:

• Langfuse uses fixed connection configs with API keys
• Enterprise inference gateways require short-lived JWT tokens with brief TTLs

The proxy acts as an adapter, performing three critical operations:

• Receiving OpenAI-compatible requests from Langfuse
• Dynamically obtaining JWTs via the client_credentials flow
• Transparently forwarding requests to the upstream API with Authorization header replacement

This preserves the standard Langfuse workflow without code changes, while complying with internal infrastructure security policies.

Technical Implementation: Step-by-Step Build

Step 1: Asynchronous Core with FastAPI

The proxy is an I/O-bound service where the main load comes from network calls. An asynchronous architecture using FastAPI and httpx.AsyncClient delivers:

• Non-blocking handling of parallel requests
• Efficient connection management
• Maximum throughput with minimal resources

Project structure:

app/
├── __init__.py
├── app.py          # Entry point and lifecycle management
├── routers.py      # Request routing
├── proxy_service.py # Proxy logic
├── jwt_provider.py # Token retrieval and caching
└── settings.py     # Pydantic-based configuration

Example lifecycle management implementation:

@asynccontextmanager
async def lifespan(app: FastAPI):
    configure_logging()
    logger = get_logger()
    logger.info("Microservice is starting...")

    proxy_service = LLMProxyService()
    app.state.proxy_service = proxy_service

    try:
        yield
    finally:
        await proxy_service.aclose()
        logger.info("Microservice is shutting down...")

Step 2: Routing OpenAI-Compatible Endpoints

Minimally required routes:

• /v1/chat/completions — main endpoint for generation
• /v1/models — check available models
• /healthz — health check for orchestrators

It's critical to preserve original paths to avoid breaking compatibility with client libraries. Router implementation:

@root_router.post("/v1/chat/completions")
async def proxy_chat_completions(request: Request):
    proxy_service = _get_proxy_service(request)
    return await proxy_service.proxy_chat_completions(request)

Step 3: Dynamic Configuration via Pydantic Settings

Using pydantic-settings enables flexible parameter management across environments. Key parameters:

• LLM_BASE_URL — inference gateway address
• SYSTEM_ID_HEADER_NAME — header for client identifier
• JWT_ISSUER and JWT_SCOPE — OIDC provider parameters
• Timeouts for all external calls

Example settings class:

class Settings(BaseSettings):
    llm_base_url: str = Field(default="", alias="LLM_BASE_URL")
    system_id_header_name: str = Field(
        default="systemId", alias="SYSTEM_ID_HEADER_NAME")
    jwt_issuer: str = Field(default="", alias="JWT_ISSUER")

Step 4: Smart JWT Provider with Caching

A critical optimization is caching tokens until expiration. Workflow algorithm:

• Extract client_id and client_secret from incoming request
• Auto-discover token_endpoint via OIDC discovery
• Request token using client_credentials flow
• Cache access_token accounting for expires_in

Without caching, the auth service becomes a bottleneck under high load. Implementation with LRU cache:

@lru_cache(maxsize=128)
async def get_token(
    self,
    client_id: str,
    client_secret: str
) -> str:
    # Token retrieval and caching logic

Step 5: Transparent Request Proxying

Key implementation aspects:

• Preserve original query parameters and request body
• Filter transport headers (Host, Content-Length)
• Replace Authorization with fresh JWT
• Pass through all HTTP status codes unmodified

Never modify the payload — this ensures compatibility with Langfuse client libraries. Example handling:

async def _forward(self, request: Request, path: str) -> Response:
    client_id = request.headers.get(self._settings.system_id_header_name)
    authorization = request.headers.get("authorization")
    _, _, client_secret = authorization.partition(" ")

    token = await self._token_provider.get_token(
        client_id=client_id,
        client_secret=client_secret,
    )

    return await self._send(
        request=request,
        url=f"{self._settings.llm_base_url}{path}",
        token=token,
    )

Key Takeaways: What Matters

• Proxy as Essential Adapter: With short-lived tokens, direct Langfuse integration is impossible — an intermediary layer is required
• Token Caching is Critical: Without it, the auth service bottlenecks under load
• Maintain Transparency: Don't modify payloads or HTTP status codes — this ensures compatibility
• Data Security: Never log Authorization headers; restrict proxy access
• Flexible Configuration: .env parameterization makes it easy to adapt across environments

Langfuse Setup: Final Steps

Configuration in the Langfuse interface:

• Set Base URL to the proxy address (e.g., http://proxy-service/v1)
• Enter client_secret in API Key (passed as Bearer credential)
• Add header with name from SYSTEM_ID_HEADER_NAME in Custom Headers

This creates a closed loop: Langfuse sends requests with fixed parameters → proxy dynamically fetches JWT → request reaches inference gateway in the required format.

Important to test scenarios:

• Token expiration
• Auth service errors
• High load (validate caching efficiency)

This architecture retains all Langfuse benefits for LLM request monitoring while meeting strict enterprise security requirements. Implementation takes 1-2 days with basic FastAPI and OIDC protocol skills.

— Editorial Team