Understanding the basic syntax for assignment isn’t a high bar to clear. While it’s maybe not ideal to use colons for both the “assignment-like” separator between keys and values inside dictionaries and the annotation of types, introducing any new syntax features into a language that’s been around for decades is never without compromise. Colons are also used at the end of block statements and in index ranges and nobody complains.
In this case you hit the most benign sort of change: Code that was previously syntactically invalid is now syntactically valid but has silly semantics. There’s never going to be a way to avoid this with full generality, and your code never worked — type annotations just delayed your runtime error by one line.
When annotations were introduced in 3.4, there was no requirement they be used to annotate types instead of something else, so any valid Python expression is allowed syntactically. In 3.6 the use of annotations for anything other than types was provisionally deprecated, and in 3.7 is fully deprecated. In 3.8, non-type usage will be disallowed as a TypeError, and the only allowable non-type annotations will be strings, (which will be interpreted as bare identifiers for backward compatibility with pre-3.7 code that uses them as forward references). At this point your code will revert to raising an exception on exactly the same line it used to.
The real problem, as others point out, is not using a linter to catch errors before runtime. Mypy flags this problem through static analysis.
You’re right I’m being uncharitable by saying “understanding”. I’m annoyed he bothered writing an accusatory blog post about this aspect of the language without mentioning any of the alternative syntax proposals helpfully discussed in the PEP, or exploring why annotations work this way. The designers have been very transparent and conservative in considering different ways of doing it, and the OP doesn’t offer any ideas for improving the interpreter, or note that the behavior he’s complaining about will be fixed in 3.8.
> Mypy flags this problem through static analysis.
The problem is that there's not really such a thing as static analysis in python. For example, if I wrap it in a function like this, mypy doesn't complain at all.
Mypy intentionally does not check annotations inside untyped `def` statements. This is a feature meant to make it easier to gradually introduce types in code that wasn't intended to be typechecked. If you call mypy with the flag `--check-untyped-defs` or just annotate the function with anything at all, you will see mypy checks the inside of the function:
from typing import Any
def blah() -> Any:
...
I'm not sure what you mean by "there's not really such a thing as static analysis" in Python -- that's exactly what Mypy does.
There is a way to avoid this kind of problem; have sufficiently complicated syntax that most random permutations caused by "fat-fingering" are incorrect. Python's problem is that the syntax is so minimalist, and the semantics so rich, that any random typo is liable to do something.
For what it's worth, I think Python's approach here is the right one (at least, for Python.) As others have noted, the common answer to this among Python users is to just use a linter to catch these kinds of issues, although I find that unsatisfying- linters can't catch everything, and they require you to have your stuff together well enough to use them (e.g., you can't be a student who is already overwhelmed with the bare necessary stack.) There are things that seem like they could be done to alleviate these kinds of problems within the language itself; most obviously, for the particular problem in the OP, a variable annotation expression could require the right-hand to be a type expression. It's not clear to me why these kinds of moves aren't taken.
> a variable annotation expression could require the right-hand to be a type expression. It's not clear to me why these kinds of moves aren't taken
Because when annotations were introduced in PEP 3107 they were not meant exclusively for type hinting. We're in a period now where non-type usage is deprecated but allowed for compatibility with old code. In 3.8 a non-type evaluation of the RHS will be disallowed.
This is a weird article to me. There seems to be an implicit condemnation of Python for this new feature resulting in non-error, unexpected behavior under certain typoes, despite the fact that that is true for literally any typo that results in valid code in any language (such as, e.g., = instead of ==). I don't understand how it would be possible to add a new syntactic option without being "bit".
Regarding type hints' ability to be arbitrary code, I suspect this is to allow you to use nested type hints from the `typing` library, like `Tuple[int, int]`, as well as more complex or user-defined types. Plus it just gives more flexibility in general.
I'd be interested to hear the author's suggestions on alternative syntax or ways to catch when the programmer intended something other than a type hint, or even arguments that type hints are extraneous and should not have been added - personally, I've found them quite useful recently, in combination with pylint.
I read it as a more general cautionary tale rather than an attack on this particular feature or language. Be aware of new features, because even ones you don't know about and don't use can cause you problems. Or alternatively for language developers, be aware that new features can make the language harder to use even for people who don't use them.
There were some people who thought that trying to bolt a type system on to Python was a bad idea.
A common response to them was "you don't have to use it; you can ignore it and you're no worse off than you were before". I think it's reasonable for him to point out that this isn't quite true.
I mean technically it's a feature you're using, just unintentionally. Kinda like setting a mutable object as a default argument.
But I'll admit I got bit by exactly this recently, wrote up an "assignment" using a colon instead of an equal sign and it didn't do anything, but didn't fail either.
Somewhat frustratingly, pycharm didn't warn about it either. I should probably check their bug tracker to see if somebody already reported it.
I don't think so, because 42 is not a type in Python (it could be one day, though - in TypeScript, 42 can be used as a type, basically an enum with only one value). I think (naively) that it should throw an error.
Annotations were specifically designed as general-purpose, not specifically type annotations. So the ability to put arbitrary expressions in annotations was very much intended.
In fact, the type hints of the typing module are not actually types.
What I wanted when I found the PEP a few days ago were docstrings for instance variables, which isn't one of the things that this provides, so I stopped reading. I ended up using properties but it's a bit verbose.
It's great that typings are in a module so it can be extended or replaced!
I'm not sure how feasible throwing an error is - Type hints in Python can be arbitrary classes or instances thereof, in order to type hint for stuff like nested data structures (`Dict[str: int]`). You could disallow instances of specific built-in classes, but that seems... clunky, especially as some can be used in valid code (strings are used as forward references).
In the void of enforcing a speficic one, you allow people flexibility to do custom things.
For instance linters not understanding type specificatins beyond a certain complexity but still using them in your code and some custom decorator to check them at run/dev time
In any language that undergoes syntactic extension, there is new syntax that is a syntax error in an old version of the language. Erroneous programs written for the old version which land on that previously bad syntax will undergo a change in behavior.
This is true even in languages that don't undergo syntactic extension. If you call a nonexistent function today, your program bombs. Tomorrow, that function could be added, so now something unexpected happens, perhaps.
Lisps are susceptible. Today (foo bar) is diagnosed; tomorrow it's new syntax (not simply a new function), because foo exists as a macro operator.
However, functions and macros, being named, can be namespaced to curtail such issues. Random, ad hoc read syntax extensions, not so much.
One approach is that files, or sections of files, can support annotations which indicate what version of the syntax is required.
Under GCC, if you want only C89, so that any syntactic extensions from later dialects are invalid syntax, you use '-ansi' on the command line to select that dialect. For oinstance, support for // comments disappears, and so x //blah/ y is x / y, rather than x followed by a comment-to-end-of-line.
This is true. I don't think Python is to blame for it, though. Any language has potential for making a mistake that's syntactically valid but doesn't have the intended meaning. It's like when a former manager of mine typed "call me at your continence" when he meant to say "call me at your convenience".
Languages can help manage change, though, providing features such that programmers working against older dialects of the language, who make mistakes, get diagnostics according to the old version, rather than accidentally stepping on new syntax.
It seems odder to me that 42 is a valid type hint than that the LHS can be an expression. But I share with most of the others here the opinion that this really isn’t a problem with Python. Every language makes trade offs, and python gives you incredible flexibility and expressiveness in exchange for a lack of certain guardrails.
Once a coworker came to me with a program that was unexpectedly failing, and it turned out that somewhere in the code there was a function that takes a callable, but instead of
something(iterable, func)
he had written
something(iterable, func())
There were no IDE errors/warning (in fact PyCharm tends to induce this error by auto inserting the parens) and the behavior that resulted made it tricky to catch.
Nonetheless, personally I’d take that over most other options for run of the mill progamming tasks.
This isn't a good article to post. It's just one person complaining about a change he didn't know about.
It's really just a syntax change. I probably wouldn't have chosen that particular syntax using colons, or I would have forced another keyword like "set" in front of it, but it is what it is. He created a bug and he likely won't do it again, so problem solved.
Ok I don't really see the issue. You could just as easily write x is 42 as x: 42 and you have the same issue. it's not really a language problem. If you see keyerror, you should know that that value didn't properly get added to the dictionary. When investigating why, you should see your error of using the wrong operator. Language is functioning properly
There are common libraries which do things incompatible with the way pypy works. I've never gotten it to do anything but abort, when run over a non-trivial program.
mypy is a very cool experimental project (as their own webpage calls it), but I wouldn't fault anyone for not running it on their program.
> Despite considerable discussion about a standard type parameterisation syntax, it was decided that this should also be left to third-party libraries. ([7], [8], [9]).
> Despite yet more discussion, it was decided not to standardize a mechanism for annotation interoperability. Standardizing interoperability conventions at this point would be premature. We would rather let these conventions develop organically, based on real-world usage and necessity, than try to force all users into some contrived scheme. ([14], [15], [16]).
In this case you hit the most benign sort of change: Code that was previously syntactically invalid is now syntactically valid but has silly semantics. There’s never going to be a way to avoid this with full generality, and your code never worked — type annotations just delayed your runtime error by one line.
When annotations were introduced in 3.4, there was no requirement they be used to annotate types instead of something else, so any valid Python expression is allowed syntactically. In 3.6 the use of annotations for anything other than types was provisionally deprecated, and in 3.7 is fully deprecated. In 3.8, non-type usage will be disallowed as a TypeError, and the only allowable non-type annotations will be strings, (which will be interpreted as bare identifiers for backward compatibility with pre-3.7 code that uses them as forward references). At this point your code will revert to raising an exception on exactly the same line it used to.