A RetroSearch Logo

Home - News ( United States | United Kingdom | Italy | Germany ) - Football scores

Search Query:

Showing content from https://python.github.io/peps/pep-0677/ below:

PEP 677 – Callable Type Syntax

PEP 677 – Callable Type Syntax
Author:
Steven Troxler <steven.troxler at gmail.com>, Pradeep Kumar Srinivasan <gohanpra at gmail.com>
Sponsor:
Guido van Rossum <guido at python.org>
Discussions-To:
Python-Dev list
Status:
Rejected
Type:
Standards Track
Topic:
Typing
Created:
13-Dec-2021
Python-Version:
3.11
Post-History:
16-Dec-2021
Resolution:
Python-Dev message
Table of Contents Abstract

This PEP introduces a concise and friendly syntax for callable types, supporting the same functionality as typing.Callable but with an arrow syntax inspired by the syntax for typed function signatures. This allows types like Callable[[int, str], bool] to be written as (int, str) -> bool.

The proposed syntax supports all the functionality provided by typing.Callable and typing.Concatenate, and is intended to work as a drop-in replacement.

Motivation

One way to make code safer and easier to analyze is by making sure that functions and classes are well-typed. In Python we have type annotations, the framework for which is defined in PEP 484, to provide type hints that can find bugs as well as helping with editor tooling like tab completion, static analysis tooling, and code review.

Consider the following untyped code:

def flat_map(func, l):
    out = []
    for element in l:
        out.extend(func(element))
    return out


def wrap(x: int) -> list[int]:
    return [x]

def add(x: int, y: int) -> int:
    return x + y

flat_map(wrap, [1, 2, 3])  # no runtime error, output is [1, 2, 3]
flat_map(add, [1, 2, 3])   # runtime error: `add` expects 2 arguments, got 1

We can add types to this example to detect the runtime error:

from typing import Callable

def flat_map(
    func: Callable[[int], list[int]],
    l: list[int]
) -> list[int]:
    ....

...


flat_map(wrap, [1, 2, 3])  # type checks okay, output is [1, 2, 3]
flat_map(add, [1, 2, 3])   # type check error

There are a few usability challenges with Callable we can see here:

Possibly as a result, programmers often fail to write complete Callable types. Such untyped or partially-typed callable types do not check the parameter types or return types of the given callable and thus negate the benefits of static typing. For example, they might write this:

from typing import Callable

def flat_map(
    func: Callable[..., Any],
    l: list[int]
) -> list[int]:
    ....

...


flat_map(add, [1, 2, 3])  # oops, no type check error!

There’s some partial type information here - we at least know that func needs to be callable. But we’ve dropped too much type information for type checkers to find the bug.

With our proposal, the example looks like this:

def flat_map(
    func: (int) -> list[int],
    l: list[int]
) -> list[int]:
    out = []
    for element in l:
        out.extend(f(element))
    return out

...

The type (int) -> list[int] is more concise, uses an arrow similar to the one indicating a return type in a function header, avoids nested brackets, and does not require an import.

Rationale

The Callable type is widely used. For example, as of October 2021 it was the fifth most common complex type in typeshed, after Optional, Tuple, Union, and List.

The others have had their syntax improved and the need for imports eliminated by either PEP 604 or PEP 585:

The typing.Callable type is used almost as often as these other types, is more complicated to read and write, and still requires an import and bracket-based syntax.

In this proposal, we chose to support all the existing semantics of typing.Callable, without adding support for new features. We made this decision after examining how frequently each feature might be used in existing typed and untyped open-source code. We determined that the vast majority of use cases are covered.

We considered adding support for named, optional, and variadic arguments. However, we decided against including these features, as our analysis showed they are infrequently used. When they are really needed, it is possible to type these using callback protocols.

An Arrow Syntax for Callable Types

We are proposing a succinct, easy-to-use syntax for typing.Callable that looks similar to function headers in Python. Our proposal closely follows syntax used by several popular languages such as Typescript, Kotlin, and Scala.

Our goals are that:

Consider this simplified real-world example from a web server, written using the existing typing.Callable:

from typing import Awaitable, Callable
from app_logic import Response, UserSetting


def customize_response(
    response: Response,
    customizer: Callable[[Response, list[UserSetting]], Awaitable[Response]]
) -> Response:
   ...

With our proposal, this code can be abbreviated to:

from app_logic import Response, UserSetting

def customize_response(
    response: Response,
    customizer: async (Response, list[UserSetting]) -> Response,
) -> Response:
    ...

This is shorter and requires fewer imports. It also has far less nesting of square brackets - only one level, as opposed to three in the original code.

Compact Syntax for ParamSpec

A particularly common case where library authors leave off type information for callables is when defining decorators. Consider the following:

from typing import Any, Callable

def with_retries(
    f: Callable[..., Any]
) -> Callable[..., Any]:
    def wrapper(retry_once, *args, **kwargs):
        if retry_once:
            try: return f(*args, **kwargs)
            except Exception: pass
        return f(*args, **kwargs)
    return wrapper

@with_retries
def f(x: int) -> int:
    return x


f(y=10)  # oops - no type error!

In the code above, it is clear that the decorator should produce a function whose signature is like that of the argument f other than an additional retry_once argument. But the use of ... prevents a type checker from seeing this and alerting a user that f(y=10) is invalid.

With PEP 612 it is possible to type decorators like this correctly as follows:

from typing import Any, Callable, Concatenate, ParamSpec, TypeVar

R = TypeVar("R")
P = ParamSpec("P")

def with_retries(
    f: Callable[P, R]
) -> Callable[Concatenate[bool, P] R]:
    def wrapper(retry_once: bool, *args: P.args, **kwargs: P.kwargs) -> R:
        ...
    return wrapper

...

With our proposed syntax, the properly-typed decorator example becomes concise and the type representations are visually descriptive:

from typing import Any, ParamSpec, TypeVar

R = TypeVar("R")
P = ParamSpec("P")

def with_retries(
    f: (**P) -> R
) -> (bool, **P) -> R:
    ...
Comparing to Other Languages

Many popular programming languages use an arrow syntax similar to the one we are proposing here.

TypeScript

In TypeScript, function types are expressed in a syntax almost the same as the one we are proposing, but the arrow token is => and arguments have names:

The names of the arguments are not actually relevant to the type. So, for example, this is the same callable type:

Kotlin

Function types in Kotlin permit an identical syntax to the one we are proposing, for example:

It also optionally allows adding names to the arguments, for example:

(x: Int, y: String) -> Bool

As in TypeScript, the argument names (if provided) are just there for documentation and are not part of the type itself.

Scala

Scala uses the => arrow for function types. Other than that, their syntax is the same as the one we are proposing, for example:

Scala, like Python, has the ability to provide function arguments by name. Function types can optionally include names, for example:

(x: Int, y: String) => Bool

Unlike in TypeScript and Kotlin, these names are part of the type if provided - any function implementing the type must use the same names. This is similar to the extended syntax proposal we describe in our Rejected Alternatives section.

Function Definitions vs Callable Type Annotations

In all of the languages listed above, type annotations for function definitions use a : rather than a ->. For example, in TypeScript a simple add function looks like this:

function higher_order(fn: (a: string) => string): string {
  return fn("Hello, World");
}

Scala and Kotlin use essentially the same : syntax for return annotations. The : makes sense in these languages because they all use : for type annotations of parameters and variables, and the use for function return types is similar.

In Python we use : to denote the start of a function body and -> for return annotations. As a result, even though our proposal is superficially the same as these other languages the context is different. There is potential for more confusion in Python when reading function definitions that include callable types.

This is a key concern for which we are seeking feedback with our draft PEP; one idea we have floated is to use => instead to make it easier to differentiate.

The ML Language Family

Languages in the ML family, including F#, OCaml, and Haskell, all use -> to represent function types. All of them use a parentheses-free syntax with multiple arrows, for example in Haskell:

Integer -> String -> Bool

The use of multiple arrows, which differs from our proposal, makes sense for languages in this family because they use automatic currying of function arguments, which means that a multi-argument function behaves like a single-argument function returning a function.

Specification Typing Behavior

Type checkers should treat the new syntax with exactly the same semantics as typing.Callable.

As such, a type checker should treat the following pairs exactly the same:

from typing import Awaitable, Callable, Concatenate, ParamSpec, TypeVarTuple

P = ParamSpec("P")
Ts = TypeVarTuple('Ts')

f0: () -> bool
f0: Callable[[], bool]

f1: (int, str) -> bool
f1: Callable[[int, str], bool]

f2: (...) -> bool
f2: Callable[..., bool]

f3: async (str) -> str
f3: Callable[[str], Awaitable[str]]

f4: (**P) -> bool
f4: Callable[P, bool]

f5: (int, **P) -> bool
f5: Callable[Concatenate[int, P], bool]

f6: (*Ts) -> bool
f6: Callable[[*Ts], bool]

f7: (int, *Ts, str) -> bool
f7: Callable[[int, *Ts, str], bool]
Grammar and AST

The proposed new syntax can be described by these AST changes to Parser/Python.asdl:

expr = <prexisting_expr_kinds>
     | AsyncCallableType(callable_type_arguments args, expr returns)
     | CallableType(callable_type_arguments args, expr returns)

callable_type_arguments = AnyArguments
                        | ArgumentsList(expr* posonlyargs)
                        | Concatenation(expr* posonlyargs, expr param_spec)

Here are our proposed changes to the Python Grammar <https://docs.python.org/3/reference/grammar.htm>:

expression:
    | disjunction disjunction 'else' expression
    | callable_type_expression
    | disjunction
    | lambdef

callable_type_expression:
    | callable_type_arguments '->' expression
    | ASYNC callable_type_arguments '->' expression

callable_type_arguments:
    | '(' '...' [','] ')'
    | '(' callable_type_positional_argument*  ')'
    | '(' callable_type_positional_argument* callable_type_param_spec ')'

callable_type_positional_argument:
    | !'...' expression ','
    | !'...' expression &')'

callable_type_param_spec:
    | '**' expression ','
    | '**' expression &')'

If PEP 646 is accepted, we intend to include support for unpacked types in two ways. To support the “star-for-unpack” syntax proposed in PEP 646, we will modify the grammar for callable_type_positional_argument as follows:

callable_type_positional_argument:
    | !'...' expression ','
    | !'...' expression &')'
    | '*' expression ','
    | '*' expression &')'

With this change, a type of the form (int, *Ts) -> bool should evaluate the AST form:

CallableType(
    ArgumentsList(Name("int"), Starred(Name("Ts")),
    Name("bool")
)

and be treated by type checkers as equivalent to or Callable[[int, *Ts], bool] or Callable[[int, Unpack[Ts]], bool].

Implications of the Grammar Precedence of ->

-> binds less tightly than other operators, both inside types and in function signatures, so the following two callable types are equivalent:

(int) -> str | bool
(int) -> (str | bool)

-> associates to the right, both inside types and in function signatures. So the following pairs are equivalent:

(int) -> (str) -> bool
(int) -> ((str) -> bool)

def f() -> (int, str) -> bool: pass
def f() -> ((int, str) -> bool): pass

def f() -> (int) -> (str) -> bool: pass
def f() -> ((int) -> ((str) -> bool)): pass

Because operators bind more tightly than ->, parentheses are required whenever an arrow type is intended to be inside an argument to an operator like |:

(int) -> () -> int | () -> bool      # syntax error!
(int) -> (() -> int) | (() -> bool)  # okay

We discussed each of these behaviors and believe they are desirable:

async Keyword

All of the binding rules still work for async callable types:

(int) -> async (float) -> str | bool
(int) -> (async (float) -> (str | bool))

def f() -> async (int, str) -> bool: pass
def f() -> (async (int, str) -> bool): pass

def f() -> async (int) -> async (str) -> bool: pass
def f() -> (async (int) -> (async (str) -> bool)): pass
Trailing Commas

Allowing trailing commas also gives autoformatters more flexibility when splitting callable types across lines, which is always legal following standard python whitespace rules.

Disallowing ... as an Argument Type

Under normal circumstances, any valid expression is permitted where we want a type annotation and ... is a valid expression. This is never semantically valid and all type checkers would reject it, but the grammar would allow it if we did not explicitly prevent this.

Since ... is meaningless as a type and there are usability concerns, our grammar rules it out and the following is a syntax error:

We decided that there were compelling reasons to do this:

Incompatibility with other possible uses of * and **

The use of **P for supporting PEP 612 ParamSpec rules out any future proposal using a bare **<some_type> to type kwargs. This seems acceptable because:

Compatibility with Arrow-Based Lambda Syntax

To the best of our knowledge there is no active discussion of arrow-style lambda syntax that we are aware of, but it is nonetheless worth considering what possibilities would be ruled out by adopting this proposal.

It would be incompatible with this proposal to adopt the same a parenthesized ->-based arrow syntax for lambdas, e.g. (x, y) -> x + y for lambda x, y: x + y.

Our view is that if we want arrow syntax for lambdas in the future, it would be a better choice to use =>, e.g. (x, y) => x + y. Many languages use the same arrow token for both lambdas and callable types, but Python is unique in that types are expressions and have to evaluate to runtime values. Our view is that this merits using separate tokens, and given the existing use of -> for return types in function signatures it would be more coherent to use -> for callable types and => for lambdas.

Runtime Behavior

The new AST nodes need to evaluate to runtime types, and we have two goals for the behavior of these runtime types:

Evaluation and Structured API

We intend to create new builtin types to which the new AST nodes will evaluate, exposing them in the types module.

Our plan is to expose a structured API as if they were defined as follows:

class CallableType:
    is_async: bool
    arguments: Ellipsis | tuple[CallableTypeArgument]
    return_type: object

class CallableTypeArgument:
    kind: CallableTypeArgumentKind
    annotation: object

@enum.global_enum
class CallableTypeArgumentKind(enum.IntEnum):
    POSITIONAL_ONLY: int = ...
    PARAM_SPEC: int = ...

The evaluation rules are expressed in terms of the following pseudocode:

def evaluate_callable_type(
    callable_type: ast.CallableType | ast.AsyncCallableType:
) -> CallableType:
    return CallableType(
       is_async=isinstance(callable_type, ast.AsyncCallableType),
       arguments=_evaluate_arguments(callable_type.arguments),
       return_type=evaluate_expression(callable_type.returns),
    )

def _evaluate_arguments(arguments):
    match arguments:
        case ast.AnyArguments():
            return Ellipsis
        case ast.ArgumentsList(posonlyargs):
            return tuple(
                _evaluate_arg(arg) for arg in args
            )
        case ast.ArgumentsListConcatenation(posonlyargs, param_spec):
            return tuple(
                *(evaluate_arg(arg) for arg in args),
                _evaluate_arg(arg=param_spec, kind=PARAM_SPEC)
            )
        if isinstance(arguments, Any
    return Ellipsis

def _evaluate_arg(arg, kind=POSITIONAL_ONLY):
    return CallableTypeArgument(
        kind=POSITIONAL_ONLY,
        annotation=evaluate_expression(value)
    )
Backward-Compatible API

To get backward compatibility with the existing types.Callable API, which relies on fields __args__ and __parameters__, we can define them as if they were written in terms of the following:

import itertools
import typing

def get_args(t: CallableType) -> tuple[object]:
    return_type_arg = (
        typing.Awaitable[t.return_type]
        if t.is_async
        else t.return_type
    )
    arguments = t.arguments
    if isinstance(arguments, Ellipsis):
        argument_args = (Ellipsis,)
    else:
        argument_args = (arg.annotation for arg in arguments)
    return (
        *arguments_args,
        return_type_arg
    )

def get_parameters(t: CallableType) -> tuple[object]:
    out = []
    for arg in get_args(t):
        if isinstance(arg, typing.ParamSpec):
            out.append(t)
        else:
            out.extend(arg.__parameters__)
    return tuple(out)
Additional Behaviors of types.CallableType

As with the A | B syntax for unions introduced in PEP 604:

Rejected Alternatives

Many of the alternatives we considered would have been more expressive than typing.Callable, for example adding support for describing signatures that include named, optional, and variadic arguments.

To determine which features we most needed to support with a callable type syntax, we did an extensive analysis of existing projects:

We decided on a simple proposal with improved syntax for the existing Callable type because the vast majority of callbacks can be correctly described by the existing typing.Callable semantics:

Features that other, more complicated proposals would support account for fewer than 2% of the use cases we found. These are already expressible using callback protocols, and since they are uncommon we decided that it made more sense to move forward with a simpler syntax.

Extended Syntax Supporting Named and Optional Arguments

Another alternative was for a compatible but more complex syntax that could express everything in this PEP but also named, optional, and variadic arguments. In this “extended” syntax proposal the following types would have been equivalent:

class Function(typing.Protocol):
    def f(self, x: int, /, y: float, *, z: bool = ..., **kwargs: str) -> bool:
        ...

Function = (int, y: float, *, z: bool = ..., **kwargs: str) -> bool

Advantages of this syntax include: - Most of the advantages of the proposal in this PEP (conciseness, PEP 612 support, etc) - Furthermore, the ability to handle named, optional, and variadic arguments

We decided against proposing it for the following reasons:

We confirmed that the current proposal is forward-compatible with extended syntax by implementing a grammar and AST for this extended syntax on top of our reference implementation of this PEP’s grammar.

Syntax Closer to Function Signatures

One alternative we had floated was a syntax much more similar to function signatures.

In this proposal, the following types would have been equivalent:

class Function(typing.Protocol):
    def f(self, x: int, /, y: float, *, z: bool = ..., **kwargs: str) -> bool:
        ...

Function = (x: int, /, y: float, *, z: bool = ..., **kwargs: str) -> bool

The benefits of this proposal would have included:

Key downsides that led us to reject the idea include the following:

Other Proposals Considered Functions-as-Types

An idea we looked at very early on was to allow using functions as types. The idea is allowing a function to stand in for its own call signature, with roughly the same semantics as the __call__ method of callback protocols:

def CallableType(
    positional_only: int,
    /,
    named: str,
    *args: float,
    keyword_only: int = ...,
    **kwargs: str
) -> bool: ...

f: CallableType = ...
f(5, 6.6, 6.7, named=6, x="hello", y="world")  # typechecks as bool

This may be a good idea, but we do not consider it a viable replacement for callable types:

Hybrid keyword-arrow Syntax

In the Rust language, a keyword fn is used to indicate functions in much the same way as Python’s def, and callable types are indicated using a hybrid arrow syntax Fn(i64, String) -> bool.

We could use the def keyword in callable types for Python, for example our two-parameter boolean function could be written as def(int, str) -> bool. But we think this might confuse readers into thinking def(A, B) -> C is a lambda, particularly because Javascript’s function keyword is used in both named and anonymous functions.

Parenthesis-Free Syntax

We considered a parentheses-free syntax that would have been even more concise:

We decided against it because this is not visually as similar to existing function header syntax. Moreover, it is visually similar to lambdas, which bind names with no parentheses: lambda x, y: x == y.

Requiring Outer Parentheses

A concern with the current proposal is readability, particularly when callable types are used in return type position which leads to multiple top-level -> tokens, for example:

def make_adder() -> (int) -> int:
    return lambda x: x + 1

We considered a few ideas to prevent this by changing rules about parentheses. One was to move the parentheses to the outside, so that a two-argument boolean function is written (int, str -> bool). With this change, the example above becomes:

def make_adder() -> (int -> int):
    return lambda x: x + 1

This makes the nesting of many examples that are difficult to follow clear, but we rejected it because

We also considered requiring parentheses on both the parameter list and the outside, e.g. ((int, str) -> bool). With this change, the example above becomes:

def make_adder() -> ((int) -> int):
    return lambda x: x + 1

We rejected this change because:

Making -> bind tighter than |

In order to allow both -> and | tokens in type expressions we had to choose precedence. In the current proposal, this is a function returning an optional boolean:

(int, str) -> bool | None  # equivalent to (int, str) -> (bool | None)

We considered having -> bind tighter so that instead the expression would parse as ((int, str) -> bool) | None. There are two advantages to this:

We decided against this for a few reasons:

Introducing type-strings

Another idea was adding a new “special string” syntax and putting the type inside of it, for example t”(int, str) -> bool”. We rejected this because it is not as readable, and seems out of step with guidance from the Steering Council on ensuring that type expressions do not diverge from the rest of Python’s syntax.

Improving Usability of the Indexed Callable Type

If we do not want to add new syntax for callable types, we could look at how to make the existing type easier to read. One proposal would be to make the builtin callable function indexable so that it could be used as a type:

callable[[int, str], bool]

This change would be analogous to PEP 585 that made built in collections like list and dict usable as types, and would make imports more convenient, but it wouldn’t help readability of the types themselves much.

In order to reduce the number of brackets needed in complex callable types, it would be possible to allow tuples for the argument list:

callable[(int, str), bool]

This actually is a significant readability improvement for multi-argument functions, but the problem is that it makes callables with one arguments, which are the most common arity, hard to write: because (x) evaluates to x, they would have to be written like callable[(int,), bool]. We find this awkward.

Moreover, none of these ideas help as much with reducing verbosity as the current proposal, nor do they introduce as strong a visual cue as the -> between the parameter types and the return type.

Alternative Runtime Behaviors

The hard requirements on our runtime API are that:

Alternative APIs

We considered having the runtime data types.CallableType use a more structured API where there would be separate fields for posonlyargs and param_spec. The current proposal was was inspired by the inspect.Signature type.

We use “argument” in our field and type names, unlike “parameter” as in inspect.Signature, in order to avoid confusion with the callable_type.__parameters__ field from the legacy API that refers to type parameters rather than callable parameters.

Using the plain return type in __args__ for async types

It is debatable whether we are required to preserve backward compatibility of __args__ for async callable types like async (int) -> str. The reason is that one could argue they are not expressible directly using typing.Callable, and therefore it would be fine to set __args__ as (int, int) rather than (int, typing.Awaitable[int]).

But we believe this would be problematic. By preserving the appearance of a backward-compatible API while actually breaking its semantics on async types, we would cause runtime type libraries that attempt to interpret Callable using __args__ to fail silently.

It is for this reason that we automatically wrap the return type in Awaitable.

Backward Compatibility

This PEP proposes a major syntax improvement over typing.Callable, but the static semantics are the same.

As such, the only thing we need for backward compatibility is to ensure that types specified via the new syntax behave the same as equivalent typing.Callable and typing.Concatenate values they intend to replace.

There is no particular interaction between this proposal and from __future__ import annotations - just like any other type annotation it will be unparsed to a string at module import, and typing.get_type_hints should correctly evaluate the resulting strings in cases where that is possible.

This is discussed in more detail in the Runtime Behavior section.

Reference Implementation

We have a working implementation of the AST and Grammar with tests verifying that the grammar proposed here has the desired behaviors.

The runtime behavior is not yet implemented. As discussed in the Runtime Behavior portion of the spec we have a detailed plan for both a backward-compatible API and a more structured API in a separate doc where we are also open to discussion and alternative ideas.

Open Issues Details of the Runtime API

We have attempted to provide a complete behavior specification in the Runtime Behavior section of this PEP.

But there are probably more details that we will not realize we need to define until we build a full reference implementation.

Optimizing SyntaxError messages

The current reference implementation has a fully-functional parser and all edge cases presented here have been tested.

But there are some known cases where the errors are not as informative as we would like. For example, because (int, ...) -> bool is illegal but (int, ...) is a valid tuple, we currently produce a syntax error flagging the -> as the problem even though the real cause of the error is using ... as an argument type.

This is not part of the specification per se but is an important detail to address in our implementation. The solution will likely involve adding invalid_.* rules to python.gram and customizing error messages.

Resources Background and History

PEP 484 specifies a very similar syntax for function type hint comments for use in code that needs to work on Python 2.7. For example:

def f(x, y):
    # type: (int, str) -> bool
    ...

At that time we used indexing operations to specify generic types like typing.Callable because we decided not to add syntax for types. However, we have since begun to do so, e.g. with PEP 604.

Maggie proposed better callable type syntax as part of a larger presentation on typing simplifications at the PyCon Typing Summit 2021.

Steven brought up this proposal on typing-sig. We had several meetings to discuss alternatives, and this presentation led us to the current proposal.

Pradeep brought this proposal to python-dev for feedback.

Acknowledgments

Thanks to the following people for their feedback on the PEP and help planning the reference implementation:

Alex Waygood, Eric Traut, Guido van Rossum, James Hilton-Balfe, Jelle Zijlstra, Maggie Moss, Tuomas Suutari, Shannon Zhu.

Copyright

This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive.


RetroSearch is an open source project built by @garambo | Open a GitHub Issue

Search and Browse the WWW like it's 1997 | Search results from DuckDuckGo

HTML: 3.2 | Encoding: UTF-8 | Version: 0.7.4