📜 ⬆️ ⬇️

Multiple pitfalls of static typing in Python


I think we are slowly getting used to the fact that Python has type annotations: they were delivered two releases back (3.5) to the annotation of functions and methods ( PEP 484 ), and in the last release (3.6) to variables ( PEP 526 ).


Since both of these PEPs were inspired by MyPy , I’ll tell you what everyday joys and cognitive dissonances awaited me when using this static analyzer, as well as the typing system as a whole.


Disclamer: I do not raise the question of the need or harmfulness of static typing in Python. Just talking about the pitfalls that came across in the process of working in a statically-typed context.

Generics (typing.Generic)


It's nice to use in annotations something like List[int] , Callable[[int, str], None] .
It is very nice when the analyzer highlights the following code:


 T = ty.TypeVar('T') class A(ty.Generic[T]): value: T A[int]().value = 'str' # error: Incompatible types in assignment # (expression has type "str", variable has type "int") 

However, what to do if we write a library, and a programmer using it will not use a static analyzer?
To force the user to initialize the class with a value, and then store its type?


 T = ty.TypeVar('T') class Gen(Generic[T]): value: T ref: Type[T] def __init__(self, value: T) -> None: self.value = value self.ref = type(value) 

Somehow not user-friendly.
And what if you want to do so?


 b = Gen[A](B()) 

In search of an answer to this question, I ran a little modulo typing , and plunged into the world of factories.

The fact is that after initialization of the instance of the Generic class, it has the attribute __origin_class__ , which has the attribute __args__ , which is a type tuple. However, there is no access to it from __init__ , as well as from __new__ . It is also not in the __call__ metaclass. And the trick is that at the time of initialization of the Generic subclass, it turns into another metaclass _GenericAlias , which sets the final type, either after initializing the object, including all the methods of its metaclass, or at the time of calling __getithem__ on it. Thus, there is no way to get generic types when constructing an object.


We throw out this garbage, promised a more universal solution.

Therefore, I wrote myself a small descriptor that solves this problem:


 def _init_obj_ref(obj: 'Gen[T]') -> None: """Set object ref attribute if not one to initialized arg.""" if not hasattr(obj, 'ref'): obj.ref = obj.__orig_class__.__args__[0] # type: ignore class ValueHandler(Generic[T]): """Handle object _value attribute, asserting it's type.""" def __get__(self, obj: 'Gen[T]', cls: Type['Gen[T]'] ) -> Union[T, 'ValueHandler[T]']: if not obj: return self _init_obj_ref(obj) if not obj._value: obj._value = obj.ref() return obj._value def __set__(self, obj: 'Gen[T]', val: T) -> None: _init_obj_ref(obj) if not isinstance(val, obj.ref): raise TypeError(f'has to be of type {obj.ref}, pasted {val}') obj._value = val class Gen(Generic[T]): _value: T ref: Type[T] value = ValueHandler[T]() def __init__(self, value: T) -> None: self._value = value class A: pass class B(A): pass b = Gen[A](B()) b.value = A() b.value = int() # TypeError: has to be of type <class '__main__.A'>, pasted 0 

Of course, in consequence, it will be necessary to rewrite for more universal use, but the essence is clear.


[UPD]: In the morning I decided to try to do the same as in the typing module itself, but simpler:


 import typing as ty T = ty.TypeVar('T') class A(ty.Generic[T]): # __args are unique every instantiation __args: ty.Optional[ty.Tuple[ty.Type[T]]] = None value: T def __init__(self, value: ty.Optional[T]=None) -> None: """Get actual type of generic and initizalize it's value.""" cls = ty.cast(A, self.__class__) if cls.__args: self.ref = cls.__args[0] else: self.ref = type(value) if value: self.value = value else: self.value = self.ref() cls.__args = None def __class_getitem__(cls, *args: ty.Union[ty.Type[int], ty.Type[str]] ) -> ty.Type['A']: """Recive type args, if passed any before initialization.""" cls.__args = ty.cast(ty.Tuple[ty.Type[T]], args) return super().__class_getitem__(*args, **kwargs) # type: ignore a = A[int]() b = A(int()) c = A[str]() print([a.value, b.value, c.value]) # [0, 0, ''] 

[UPD]: typing developer Ivan Levinsky said that both options could break unpredictably.


Anyway, you can use whatever way. Maybe __class_getitem__ is even slightly better __class_getitem__ .

Functions and aliases


Yes, with generics is not easy at all:
For example, if we take a function somewhere as an argument, its annotation automatically turns from covariant to contravariant:


 class A: pass class B(A): pass def foo(arg: 'A') -> None: # принимает инстанции A и B ... def bar(f: Callable[['A'], None]): # принимает функции с аннотацией не ниже A ... 

And in principle, I have no claims to logic, only it has to be solved through generic aliases:


 TA = TypeVar('TA', bound='A') def foo(arg: 'B') -> None: # принимает инстанции B и сабклассов ... def bar(f: Callable[['TA'], None]): # принимает функции с аннотациями A и B ... 

In general, the section on type variation should be read carefully, and not on time.


backward compatibility


This is not so hot: from version 3.7 Generic - subclass ABCMeta , which is very convenient and good. It's bad that it breaks the code if it is running on 3.6.


Structural Inheritance (Stuctural Suptyping)


At first I was very happy: the interfaces were delivered! The role of interfaces is performed by the Protocol class from the typing_extensions module, which, in combination with the @runtime decorator, allows you to check whether the class implements an interface without direct inheritance. MyPy is also highlighted at a deeper level.


However, I didn’t notice any particular practical use in runtime compared to multiple inheritance.
It seems that the decorator checks only the presence of a method with the required name, without even checking the number of arguments, not to mention the typing:


 import typing as ty import typing_extensions as te @te.runtime class IntStackP(te.Protocol): _list: ty.List[int] def push(self, val: int) -> None: ... class IntStack: def __init__(self) -> None: self._list: ty.List[int] = list() def push(self, val: int) -> None: if not isinstance(val, int): raise TypeError('wrong pushued val type') self._list.append(val) class StrStack: def __init__(self) -> None: self._list: ty.List[str] = list() def push(self, val: str, weather: ty.Any=None) -> None: if not isinstance(val, str): raise TypeError('wrong pushued val type') self._list.append(val) def push_func(stack: IntStackP, value: int): if not isinstance(stack, IntStackP): raise TypeError('is not IntStackP') stack.push(value) a = IntStack() b = StrStack() c: ty.List[int] = list() push_func(a, 1) push_func(b, 1) # TypeError: wrong pushued val type push_func(c, 1) # TypeError: is not IntStackP 

On the other hand, MyPy, in turn, behaves more intelligently, and highlights the incompatibility of types:


 push_func(a, 1) push_func(b, 1) # Argument 1 to "push_func" has incompatible type "StrStack"; # expected "IntStackP" # Following member(s) of "StrStack" have conflicts: # _list: expected "List[int]", got "List[str]" # Expected: # def push(self, val: int) -> None # Got: # def push(self, val: str, weather: Optional[Any] = ...) -> None 

Operator Overloading


Very fresh topic, because When operators are overloaded with full type safety, all the fun disappears. This question has already emerged more than once in the MyPy bug tracker, but it still curses in some places, and it can be safely turned off.
I explain the situation:


 class A: def __add__(self, other) -> int: return 3 def __iadd__(self, other) -> 'A': if isinstance(other, int): return NotImplemented return A() var = A() var += 3 # Inferred type is 'A', but runtime type is 'int'? 

If the composite assignment method returns NotImplemented , Python first looks for __radd__ , then uses __add__ , and voila.


The same applies to overloading of any subclass methods of the form:


 class A: def __add__(self, x : 'A') -> 'A': ... class B(A): @overload def __add__(self, x : 'A') -> 'A': ... @overload def __add__(self, x : 'B') -> 'B' : ... 

In some places, the warnings have already moved to the documentation, in some places they are still on sale. But the general conclusion of the contributors: to leave such overloads acceptable.



Source: https://habr.com/ru/post/437018/