RFC: Enable type annotations and Mypy checks on Python projects in Fedora Infrastructure
Hi everyone,
RFC: Enable type annotations and Mypy checks on Python projects in Fedora Infrastructure
We have many Python applications under Fedora Infrastructure, and we also on a path to move to the Python3 world. I am proposing the following changes for our Python applications:
* Add type annotations in any new code we write. * Add type annotations in code we change * Enable Mypy checks for every pull requests and commits in CI. * During Flock we try to enable type annotations in our favorite Python applications.
## What is Type Annotation
This is a feature of Python3, also backported to the Python2 world using the python2-typing module. Type annotations or type hints are purely for external type checking tools, they do not affect the Python interpreter anyway. PEP-484 discusses more on the type hints [1].
## Example (which works in both Python2 and Python3)
def add(a, b): # type: (int, int) -> int "Adds two numbers" return a + b
## What is Mypy?
Mypy [2] is an open source type checking tool, most of the developers are in Dropbox. Mypy can be used on a codebase gradually, means we don't have to add all the type information in one go. We can slowly keep improving the type annotations.
## What are benefits of enable type annotations and Mypy?
### Readability of the code
Because of the checks, we will always have right data types mentioned, which increases the readability of the code a lot for any future maintainer.
### Finding actual logical bugs
There are times when we have functions which generally returns a boolean, but it some corner cases, maybe it returns an empty string or None value. These may cause runtime errors. Using Mypy on your codebase improves the chances of finding any such issue in the development cycle only.
### Refactoring becomes easier
As the tool itself checks for all of the API use cases, refactoring larger codebases will be easier with Mypy.
## Which are the few big projects using Mypy?
Dropbox has around 700K lines of Python code, and Zulip project has around 100K of Python with type information. They also have an excellent blog post at [3].
[1] https://www.python.org/dev/peps/pep-0484/ [2] http://mypy.readthedocs.io/en/latest/ [3] http://blog.zulip.org/2016/10/13/static-types-in-python-oh-mypy/
Kushal
On Tue, Jun 20, 2017 at 8:06 AM, Kushal Das mail@kushaldas.in wrote:
Hi everyone,
RFC: Enable type annotations and Mypy checks on Python projects in Fedora Infrastructure
We have many Python applications under Fedora Infrastructure, and we also on a path to move to the Python3 world. I am proposing the following changes for our Python applications:
- Add type annotations in any new code we write.
- Add type annotations in code we change
- Enable Mypy checks for every pull requests and commits in CI.
- During Flock we try to enable type annotations in our favorite Python applications.
## What is Type Annotation
This is a feature of Python3, also backported to the Python2 world using the python2-typing module. Type annotations or type hints are purely for external type checking tools, they do not affect the Python interpreter anyway. PEP-484 discusses more on the type hints [1].
## Example (which works in both Python2 and Python3)
def add(a, b): # type: (int, int) -> int "Adds two numbers" return a + b## What is Mypy?
Mypy [2] is an open source type checking tool, most of the developers are in Dropbox. Mypy can be used on a codebase gradually, means we don't have to add all the type information in one go. We can slowly keep improving the type annotations.
## What are benefits of enable type annotations and Mypy?
### Readability of the code
Because of the checks, we will always have right data types mentioned, which increases the readability of the code a lot for any future maintainer.
### Finding actual logical bugs
There are times when we have functions which generally returns a boolean, but it some corner cases, maybe it returns an empty string or None value. These may cause runtime errors. Using Mypy on your codebase improves the chances of finding any such issue in the development cycle only.
### Refactoring becomes easier
As the tool itself checks for all of the API use cases, refactoring larger codebases will be easier with Mypy.
## Which are the few big projects using Mypy?
Dropbox has around 700K lines of Python code, and Zulip project has around 100K of Python with type information. They also have an excellent blog post at [3].
[1] https://www.python.org/dev/peps/pep-0484/ [2] http://mypy.readthedocs.io/en/latest/ [3] http://blog.zulip.org/2016/10/13/static-types-in-python-oh-mypy/
FWIW +1 from me
I'm a big fan of mypy and type annotations. I'm currently using type annotations in my custom ansible modules in the releng-automation project space.
-AdamM
Kushal
Fedora Cloud Engineer CPython Core Developer https://kushaldas.in https://dgplug.org _______________________________________________ infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-leave@lists.fedoraproject.org
I'm also +1 on this. I haven't used mypy or type annotations before, but I like well-documented code and having something to lint types would be extremely helpful.
I'm also a +1. I've been working on documenting Bodhi's code with a PEP-257 checker, and part of that work has been to determine types on some arguments.
Can mypy handle finding the types from docblocks, or is it strict about the format being like:
# type: (int, bool)
?
On Tue, 2017-06-20 at 16:27 -0400, Randy Barlow wrote:
Can mypy handle finding the types from docblocks, or is it strict about the format being like:
# type: (int, bool)
?
To elaborate on my question, I've been doing docblocks in this style:
def add(x, y): """ Return the sum of x and y.
Args: x (int): The first number. y (int): The second number. Returns: int: The sum of x and y. """ return x + y
It'd be suuuuuper nice if mypy could do that… ☺ If not, I guess I could include both annotatations, or perhaps only the mypy annotation so that I'm not adding duplicate info.
On Tue, Jun 20, 2017 at 5:02 PM, Randy Barlow bowlofeggs@fedoraproject.org wrote:
On Tue, 2017-06-20 at 16:27 -0400, Randy Barlow wrote:
Can mypy handle finding the types from docblocks, or is it strict about the format being like:
# type: (int, bool)
?
To elaborate on my question, I've been doing docblocks in this style:
def add(x, y): """ Return the sum of x and y.
Args: x (int): The first number. y (int): The second number. Returns: int: The sum of x and y. """ return x + yIt'd be suuuuuper nice if mypy could do that… ☺ If not, I guess I could include both annotatations, or perhaps only the mypy annotation so that I'm not adding duplicate info.
To the best of my knowledge it does not support doing that and you would need to add PEP 484 conformant type annotations.
-AdamM
infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-leave@lists.fedoraproject.org
On Tue, Jun 20, 2017 at 6:06 AM, Kushal Das mail@kushaldas.in wrote:
Type annotations or type hints are purely for external type checking tools, they do not affect the Python interpreter anyway.
Note that this statement is incorrect. Code like this: https://docs.python.org/3/library/typing.html#user-defined-generic-types where inheritance of a class from the typing module is used affects running code. Mark Shannon gave a talk at the pycon language summit and plead for the typing module to divorce type checking from inheritance and pointed out that the implementation slows down runtime. He further pointed out that using inheritance for type information like this was not a proper fit; that sometimes the class inherited from should not have anything to do with the type information.
-Toshio
As someone who has a very big research interest in type theory, huge +1 here. I'd love to see us moving in the direction where types are leveraged.
On Tue, Jun 20, 2017 at 7:37 PM, Toshio Kuratomi a.badger@gmail.com wrote:
On Tue, Jun 20, 2017 at 6:06 AM, Kushal Das mail@kushaldas.in wrote:
Type annotations or type hints are purely for external type checking tools, they do not affect the Python interpreter anyway.
Note that this statement is incorrect. Code like this: https://docs.python.org/3/library/typing.html#user-defined-generic-types where inheritance of a class from the typing module is used affects running code. Mark Shannon gave a talk at the pycon language summit and plead for the typing module to divorce type checking from inheritance and pointed out that the implementation slows down runtime. He further pointed out that using inheritance for type information like this was not a proper fit; that sometimes the class inherited from should not have anything to do with the type information.
-Toshio _______________________________________________ infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-leave@lists.fedoraproject.org
On 20/06/17, Toshio Kuratomi wrote:
On Tue, Jun 20, 2017 at 6:06 AM, Kushal Das mail@kushaldas.in wrote:
Type annotations or type hints are purely for external type checking tools, they do not affect the Python interpreter anyway.
Note that this statement is incorrect. Code like this: https://docs.python.org/3/library/typing.html#user-defined-generic-types where inheritance of a class from the typing module is used affects running code. Mark Shannon gave a talk at the pycon language summit and plead for the typing module to divorce type checking from inheritance and pointed out that the implementation slows down runtime. He further pointed out that using inheritance for type information like this was not a proper fit; that sometimes the class inherited from should not have anything to do with the type information.
Thanks for pointing this one out, I somehow missed both of Mark's talks in the language summit. Now, I went back and read the report on his talk[1]. I hope the core-denels will come up with solutions to the problem he pointed up in proper time.
Thanks once again for pointing this one.
[1] https://lwn.net/Articles/724639/
Kushal
infrastructure@lists.fedoraproject.org
-
Adam Miller -
Jeremy Cline -
Kushal Das -
Randy Barlow -
Ricky Elrod -
Toshio Kuratomi