Most typesystems are limited in what they can express, and as they get more complicated in what they can express they tend to get more complicated to understand.
A typesystem is a limitation you place on your code which you accept in the hope that the things which it proves true (problems it avoids) are worth the limitations that you've accepted. You can set everything to Object but the culture that develops around statically typed code tends to mean that people are really afraid to ever do this or be ostracised.
From a data moving point of view there's a couple of examples I hit in production where a typesystem isn't helping me =>
* DataFrames tend to gain/lose columns on every operation, so to represent them in types I'd need a zillion types with code duplication or an interface for each field name. Some operations aren't even possible to type like pivot_table. So tons of my code is really not helped by typing
* Everything turns into a DAG of data flows, you can do this with typing but the overhead would mean that noone woudl come up with something like mdf (I don't use mdf in prod but it's impressive for specific applications)
If you want a better defense of dynamism google Rich Hickey, a point he made:
There's many choices to be made for typesystems, you can have generics/not, higher kinded types, dependent typing. Maybe it's better not to enforce these choices right at the root language level and allow them to happen at a higher level (be plugged in).
> DataFrames tend to gain/lose columns on every operation, so to represent them in types I'd need a zillion types with code duplication or an interface for each field name. Some operations aren't even possible to type like pivot_table.
For pivoting? I don't know about dependent types, sometimes it's possible in theory to know the type of the pivot columns because you know the possible combinations of column fields that you're pivoting from, but is that dependent-typable?
I think there's a kind of Motte and Bailey that goes on with people who think typesystems are the one true god. They choose the simplest typesystem as their example of the cognitive/learning overhead, (say C#?)
But if you question the limitations they go to "but it's possible in X obscure typesystem".
But that's exactly to the OP's point: You need to think about this. You need to figure it out beforehand. In Python, you can express this without that mental overhead--particularly while prototyping.
A typesystem is a limitation you place on your code which you accept in the hope that the things which it proves true (problems it avoids) are worth the limitations that you've accepted. You can set everything to Object but the culture that develops around statically typed code tends to mean that people are really afraid to ever do this or be ostracised.
From a data moving point of view there's a couple of examples I hit in production where a typesystem isn't helping me => * DataFrames tend to gain/lose columns on every operation, so to represent them in types I'd need a zillion types with code duplication or an interface for each field name. Some operations aren't even possible to type like pivot_table. So tons of my code is really not helped by typing * Everything turns into a DAG of data flows, you can do this with typing but the overhead would mean that noone woudl come up with something like mdf (I don't use mdf in prod but it's impressive for specific applications)
If you want a better defense of dynamism google Rich Hickey, a point he made:
There's many choices to be made for typesystems, you can have generics/not, higher kinded types, dependent typing. Maybe it's better not to enforce these choices right at the root language level and allow them to happen at a higher level (be plugged in).