Home - News ( United States | United Kingdom | Italy | Germany ) - Football scores

Showing content from https://learn.microsoft.com/en-us/azure/api-management/azure-openai-token-limit-policy below:

Azure API Management policy reference - azure-openai-token-limit

APPLIES TO: Developer | Basic | Basic v2 | Standard | Standard v2 | Premium | Premium v2

The azure-openai-token-limit policy prevents Azure OpenAI in Foundry Models API usage spikes on a per key basis by limiting consumption of language model tokens to a specified rate (number per minute), a quota over a specified period, or both. When a specified token rate limit is exceeded, the caller receives a 429 Too Many Requests response status code. When a specified quota is exceeded, the caller receives a 403 Forbidden response status code.

By relying on token usage metrics returned from the OpenAI endpoint, the policy can accurately monitor and enforce limits in real time. The policy also enables precalculation of prompt tokens by API Management, minimizing unnecessary requests to the OpenAI backend if the limit is already exceeded.

The policy is used with APIs added to API Management from the Azure OpenAI in Foundry Models of the following types:

gpt-4

gpt-4o

gpt-4o-mini

o1

o3

text-embedding-3-small

text-embedding-ada-002

gpt-4o-mini (Version: 2024-07-18)

gpt-4.1 (Version: 2025-04-14)

gpt-4.1-nano (Version: 2025-04-14)

gpt-4.1-mini (Version: 2025-04-14)

gpt-image-1 (Version: 2025-04-15)

o3 (Version: 2025-04-16)

o4-mini (Version: `2025-04-16)

Note

Traditional completion APIs are only available with legacy model versions and support is limited.

For current information about the models and their capabilities, see Azure OpenAI in Foundry Models.

<azure-openai-token-limit counter-key="key value"
        tokens-per-minute="number"
        token-quota="number"
        token-quota-period="Hourly | Daily | Weekly | Monthly | Yearly"
        estimate-prompt-tokens="true | false"    
        retry-after-header-name="custom header name, replaces default 'Retry-After'" 
        retry-after-variable-name="policy expression variable name"
        remaining-quota-tokens-header-name="header name"  
        remaining-quota-tokens-variable-name="policy expression variable name"
        remaining-tokens-header-name="header name"  
        remaining-tokens-variable-name="policy expression variable name"
        tokens-consumed-header-name="header name"
        tokens-consumed-variable-name="policy expression variable name" />

When set to false, the remaining tokens per counter-key are calculated using the actual token usage from the response of the model. This could result in prompts being sent to the model that exceed the token limit. In such case, this will be detected in the response, and all succeeding requests will be blocked by the policy until the token limit frees up again.

• Policy sections: inbound
• Policy scopes: global, workspace, product, API, operation
• Gateways: classic, v2, self-hosted, workspace

• This policy can be used multiple times per policy definition.
• This policy can optionally be configured when adding an API from the Azure OpenAI using the portal.
• Where available when estimate-prompt-tokens is set to false, values in the usage section of the response from the Azure OpenAI API are used to determine token usage.
• Certain Azure OpenAI endpoints support streaming of responses. When stream is set to true in the API request to enable streaming, prompt tokens are always estimated, regardless of the value of the estimate-prompt-tokens attribute. Completion tokens are also estimated when responses are streamed.
• For models that accept image input, image tokens are generally counted by the backend language model and included in limit and quota calculations. However, when streaming is used or estimate-prompt-tokens is set to true, the policy currently over-counts each image as a maximum count of 1200 tokens.
• API Management uses a single counter for each counter-key value that you specify in the policy. The counter is updated at all scopes at which the policy is configured with that key value. If you want to configure separate counters at different scopes (for example, a specific API or product), specify different key values at the different scopes. For example, append a string that identifies the scope to the value of an expression.
• This policy tracks token usage independently at each gateway where it is applied, including workspace gateways and regional gateways in a multi-region deployment. It doesn't aggregate token counts across the entire instance.

In the following example, the token rate limit of 5000 per minute is keyed by the caller IP address. The policy doesn't estimate the number of tokens required for a prompt. After each policy execution, the remaining tokens allowed for that caller IP address in the time period are stored in the variable remainingTokens.

<policies>
    <inbound>
        <base />
        <azure-openai-token-limit
            counter-key="@(context.Request.IpAddress)"
            tokens-per-minute="5000" estimate-prompt-tokens="false" remaining-tokens-variable-name="remainingTokens" />
    </inbound>
    <outbound>
        <base />
    </outbound>
</policies>

In the following example, the token quota of 10000 is keyed by the subscription ID and resets monthly. After each policy execution, the number of remaining tokens allowed for that subscription ID in the time period is stored in the variable remainingQuotaTokens.

<policies>
    <inbound>
        <base />
        <azure-openai-token-limit
            counter-key="@(context.Subscription.Id)"
            token-quota="100000" token-quota-period="Monthly" remaining-quota-tokens-variable-name="remainingQuotaTokens" />
    </inbound>
    <outbound>
        <base />
    </outbound>
</policies>

• Rate limiting and quotas
• llm-token-limit policy
• azure-openai-emit-token-metric policy

For more information about working with policies, see:

• Tutorial: Transform and protect your API
• Policy reference for a full list of policy statements and their settings
• Policy expressions
• Set or edit policies
• Reuse policy configurations
• Policy snippets repo
• Policy playground repo
• Azure API Management policy toolkit
• Get Copilot assistance to create, explain, and troubleshoot policies

RetroSearch is an open source project built by @garambo | Open a GitHub Issue

Search and Browse the WWW like it's 1997 | Search results from DuckDuckGo

HTML: 3.2 | Encoding: UTF-8 | Version: 0.7.4