I have used numpy, but don't understand what it has to do with dataframe apis
Take two examples of dataframe apis, dplyr and ibis. Both can run on a range of SQL backends because dataframe apis are very similar to SQL DML apis.
Moreover, the SQL translation for tools for pivot_longer in R are a good illustration of complex dynamics dataframe apis can support, that you'd use something like dbt to implement in your SQL models. duckdb allows dynamic column selection in unpivot. But in some SQL dialects this is impossible. dataframe apis -> SQL tools (or dbt) enable them in these dialects.
I think piping and method chaining are a little bit different.
Piping generally chains functions, by passing the result of one call into the next (eg result is first argument to the next).
Method chaining, like in Python, can't do this via syntax. Methods live on an object. Pipes work on any function, not just an object's methods (which can only chain to other object methods, not any function whose eg first argument can take that object).
For example, if you access Polars.DataFrame.style it returns a great_tables.GT object. But in a piping world, we wouldn't have had to add a style property that just calls GT() on the data. With a pipe, people would just be able to pipe their DataFrame to GT().
I think it's often a syntax convenience. For example, Polars and Pandas both have DataFrame.pipe(...) methods, that create the same effect. But it's a bit cumbersome to write.
It's neat to see tablets discussed in the context of modern tools. I recently helped edit an article for Great Tables[1] that discusses the history of tables like this, and recently Hannes mentioned a protocuniform tablet in his duckdb keynote at posit::conf()[2].
There's something really inspiring from realizing how far back tables go.
"Table" and "tablet" literally have the same root. It's flat surface, a two-dimensional blank space that is perfect for laying out data, dinner, or anything else you'd like to display.
This is an interesting case, since the pigeon study is about what happens when the underlying process is random.
But if the shape drawing process isn't random, I think the author's experience of feeling unable to articulate the rules AND gravitating to a set of behaviors is a good example of procedural memory (implicit vs explicit).
Explicit rules would probably help speed things up, though!
As someone who did statistics and psychology, I'm very surprised by this take, for a few reasons:
1. Many of the early pioneers in statistics were psychologists.
2. The econ x psych connection is strong (eg econometrics and psychometrics share a lot in common and know of each other)
3. Many of the people I see with math chops trying to do psychology are bad at the philosophy side (eg what is a construct; how do constructs like intelligence get established)
As in many fields, the strength of statistical practices continually improve. And the parent comment has it right about the difficulty. In physics its much easier to ensure your sample is representative (heterogeneity is huge), and you have no way of ensuring that last sample of 100 participants have the same characteristics as your next sample of 100.
I'm sorry, maybe a didn't communicate clearly. SubiculumCode commented the main part of what I wanted to convey, so I won't repeat.
1. Yes! But that's doesn't exactly change things, in fact, it's part of my point. A big part of why this happened (and still does!) is due to the inherent difficulties and the lack of existing tools. If you ever get a chance, go look at a university physics lab. Even Columbia's nuclear reactors (fusion or fission!) and I think many will be surprised how "janky" it looks. It's because they build the tools along the way, not because lack of monetary resources (well... That too...) but because the tools don't exist!
My critique about the psych field is that this is not more embraced. You have to embrace the fuzziness! The uncertainty. But the field is dominated by people publishing studies that use very simple statistical models, low sample sizes, and put a lot of faith in unreliable metrics with arbitrary cutoffs (most well known being the p-value). Many people will graduate grad school without a strong background in statistics and calculus (it's also easier to think this is stronger than it is. And of course, there are also plenty who would be indistinguishable from mathematicians. But on average?). There are rockstars in every field, even when not recognized as rockstars. But it matters who the field follows.
And I must be absolutely clear, this is not to say that work and those results are useless. Utility and confidence are orthogonal. You might need 5 sigma confidence verified by multiple teams and replicated on different machines to declare discovery of a particle but before that there's many works published with only a few sigma and plenty of purely theoretical works. (Note: in physics replication is highly valued. Most work is not novel and it is easy to climb the academic ladder without leading novel research. This is a whole other conversation though) This is why I discussed Copernicus and Brahe but would not call them astronomers. That's not devaluing them, but rather nothing a categorical difference due to the paradigm shift Kepler caused. Mind you, chemistry even later!
2. I specifically mention economists (my partner is one). I could highlight them more but I feel this would only add to confusion. I believe those close to the details will have no doubt to their role. I don't want to detract from their contribution but I also don't want to convolute my message which is already difficult to accurately convey.
3. I think this is orthogonal. I'm happy to bash on other fields if that makes my comment feel less like an attack rather than a critique (or wakeup call). I'm highly critical of my own community (ML) and believe it is important that we all are *most* critical of our own tribes, because even if we don't dictate where the ship goes we're not far removed from those that do. I'll rant all day if you want (or check my comment (recent) history if you want to know if I'm honest about this). I'll happily critique physicists who can solve high order PDEs in their sleep but struggle with for loops. Or the "retired engineer" trope that every physicist knows and most have experienced.
But it is hard to be critical while not offending (there's a few comments about this too). Maybe you disagree with my critique but I hope you can reread the end of comment as hopeful and encouraging. I want psychology to be taken more seriously. But if the field is unable to recognize why the other fields are dismissive of them, then this won't happen. Sure, there's silly reasons that aren't reasonable, but that doesn't mean there's no reason.
It is a matter of fact that the (statistical) confidence of studies in psychology is much lower than those of the "hard" sciences (physics, chemistry, and yes, even biology (the last part is a joke. Read how you will)). In part this is due to the studies and researchers themselves, but the truth is that the biggest contributing factor is the nature of the topic. That is not an easy thing to deal with and I have a lot of sympathy for it. But how to handle it is up to the field.
From a more applied angle, a book like "10 steps to complex learning" might be helpful.
I come from a similar cog psych background as the Bjork Lab, so am a big fan of their research, but books like 10 steps come from instructional design, which is a bit more focused on the big picture (designing a whole course vs individual mechanisms).
Hey one of the co-maintainers of Great Tables, along with Rich Iannone, here!
I just wanted to say that Rich is the only software developer I know, who when asked to lay out the philosophy of his package, would give you 5,000 years of history on the display of tables. :)
I got into embroidery as a break from software engineering. Then, I got into machine embroidery and found myself running inkstitch in a docker container and submitting GitHub issues. Whoops!
Can't speak highly enough of the inkstitch maintainers, though--really welcoming community!
Take two examples of dataframe apis, dplyr and ibis. Both can run on a range of SQL backends because dataframe apis are very similar to SQL DML apis.
Moreover, the SQL translation for tools for pivot_longer in R are a good illustration of complex dynamics dataframe apis can support, that you'd use something like dbt to implement in your SQL models. duckdb allows dynamic column selection in unpivot. But in some SQL dialects this is impossible. dataframe apis -> SQL tools (or dbt) enable them in these dialects.