I think out-of-order or unknown-order execution is the biggest source of my own mistakes / bugs. If the order in which your cells are executed affects your result, you've got a bug waiting to bite you or someone else. I've been there. And we tend to run cells more than once, while refining our work.

This potentially affects reproducibility, which is a tenet of scientific computing. So it's important in our world.

I personally think there might be a couple of ways to deal with this issue. One is to put handcuffs on Jupyter and inadvertently make it more difficult or cumbersome to use. But this might be valuable if the size of your project and importance of your result warrants it.

Another way is for us to recognize that at some point we have to grow up and learn some techniques from the programmers. In my revisionist history, software engineering began when programs got too big to see the whole thing on one screen and intuitively understand.

One of my little habits is that I do a "restart kernel and run all cells" when I'm ready to take a break or close out at the end of the day. If if fails, the causes of the problem tend to be superficial, and nothing to lose sleep over.

Another habit is to take cells that work and are nice, and enclose their innards in a function so that their impact on the global variable space is minimized. This can be done during a sleepy time in the afternoon when you're not really in flow anyway.

To avoid the state problem, I start each cell (or, in org-mode, each src block) with a command to set up the environment for that cell, usually by importing a script named something like setup_abc123.py. Cells are not allowed to reference code or results from other cells except through the filesystem, which is under the watchful eye of git. In this workflow, cells exist only to write plots or tables into the notebook for human consumption.

This workflow, for me, has completely eliminated errors due to hidden state dependencies. Before I adopted it I made a serious state-related error every couple of days, which was intolerable.

As a bonus, using a single script to set up the analysis environment makes it trivial to start a new analysis, even with an entirely different tool chain.

Laughing, politely, at that summary of "history." Software engineering started when there were no screens. Code was written on typewriters (called "terminals") and then printed to cards or long rolls of paper (and occasionally using switches to indicate the 1's and 0's you wanted set as bits in your code). Before the Mac a "screen" of code was pretty universally 24 lines of green or white text on a black background, 80 monospace characters per line. A few specialized terminals had 25x80 so they could show a status line below the 24x80 standard area, but they were rare beasts. Yes, there were a few systems like Plato that predated the Mac but if you need a tl;dr it's the observation that, like most revisionist histories, this one is unfortunately total nonsense when looked at in the context of actual history (even if lots of people who didn't experience the actual history believe it).

Before screens, there were pages, before pages there were cards... going back to plugboards, etc.

Each of us got in at a certain time point. When I learned programming, we had a workstation that simulated a card punch, let you see the contents of one card at a time on a little vacuum-fluorescent display, and stored the data on an 8" floppy. You handed the floppy to the computer operator, and received a printout on green-bar paper.

The other way I got into "programming" was staring at a nest of wires, reading my output on an oscilloscope, and making corrections with a soldering iron. ;-)

The common theme was that a program gets too large for somebody to look at and understand it, much less for a team to work on it all at once, unless disciplines are adopted that tend to come from software engineering. When I'm mentoring colleagues who are just beginning to get into scientific programming, some of these simple techniques, such as avoiding globals and creating functions, are in fact an improvement.

When I'm doing some bit of data wrangling or just exploratory work with data I quite like it - at least at first. As you pointed out, it's really easy to extremely quickly iterate on things and start to get an idea of what's in the data, what techniques work, and which don't. It's great.

Until it isn't. I'm probably "using it wrong" but all the suggestions I've seen for doing it "right" start cutting into the iteration time. If I'm in Jupyter it's because I want to flail around real fast and see what happens. And that causes all sorts of problems. Which cells do I need to rerun together? Which cells should I not run again. What the heck is actually in all these variables right now anyway, because I don't remember which order I've run (and rerun) all the cells in. And what was that one approach or parameter that really worked well that one time? I don't remember.

These sorts of things aren't an issue with a non-Jupyter approach. And because I'm taking my time anyway I'm probably being diligent with source control, and all that. Even pulling code out of a notebook into a more stable workflow is a pain, because you have no assurances you can just copy/paste a cell and have it work. In my experience, it never will.

It's one of those things that has a pretty heavy costs and benefits. And unfortunately they aren't really possible to separate.

Not coincidentally, I feel the same about dynamic languages and even less usefully typed static languages. Jupyter takes an already fairly extreme position on the stable(safe)/easy continuum and really ramps it up to a point where it's hard to wrangle time and work done easy into some sort of stability. It's great, and it's awful.

I gave a talk at JupyterCon this week about Cocalc which partly solves this one problem by providing a Time travel slider with complete history.

Right. Get that all the time. My solution is to try to condense the useful code built up across different cells into a reusable block, most often a function. Do that while the 'state' is still fresh in your mind and you remember the order of execution.

It also helps to break up a long notebook into sections using headings and markdown cells with comments. I often start by writing down a question/problem I am trying to solve in prose, that makes it easier to recover the context of the code cells that follow.

Right, that's the sort of thing I was talking about in terms of "using it right." It makes sense and is really a good idea, it just starts eating into the advantages you get from using Jupyter in the first place.

I'm still trying to figure out how best to integrate it in my workflow. So far I've mostly settled on using it to get some rough ideas about the data and libraries I'm using, and then just reworking from scratch in a more stable setting. As needed I'll go back and try things out there, possibly in a new notebook entirely and just do my best to copy/paste the setup over. In getting it working the setup stuff at least gets consolidated. It's... well awful but eh, I haven't really found a better way.

1) Extracting functions is harder 2) Your git commits become completely borked 3) Opening some data-heavy notebooks is neigh impossible once they have been shut down 4) Import of other modules you have in local is pretty non-trivial. 5) Refactoring is pretty hard 6) Sphinx for autodoc extraction is pretty much out of the picture 7) Non-deterministic re-runs - depending on the cell execution order you can get very different results. That's an issue when you are coming back to your code a couple of months later and try to figure what you did to get there.

There are likely work-arounds for most of these problems, but the issue is that with my standard workflow they are non-issues to start with.

In my experience, Jupyter is pretty good if you rely only on existing libraries that you are piecing together, but once you need to do more involved development work, you are screwed.

It isn't that you aren't getting good results and doing good work. It is that there are many practices that were hammered out in software development that were completely tossed out the window. 4GL comes to mind as a comparison. Makes great demos. Is not likely to lead to good engineering.

1. Interactive computation and plotting, run data through a pipeline.

2. Quick prototyping and testing. Once a function/class is ready, it goes into a module, together with some unit tests.

Nothing beats Jupyter for that second use case, 90% of the bugs are squashed through interaction and inspection in the notebook cells, not through unit tests.

As a tool Excel has provided astronomical real world value.

There are only a small handful of other tools that even come close.

I think the counter is there is probably a lot of damage excel has done, as well. I'm not convinced other tools would have resulted in no bugs/problems.

If I need a file browser or I want to quickly view two notebooks side-by-side, Jupyter Lab is still more convenient than Spyder. Although it doesn't support all the features of the notebook, and you end up installing additional extensions managed through yarn and webpack (jeez).

I never used Jupyter before getting involved in this kind of coding, data wangling and stats. And it took me a while to get used to it. The "lack of state" is confusing, as well as the apparent non-linearity of a jupyter notebook (it can appear that certain commands are done in order from top to bottom, when if you look at the number next to them, they may have been executed in a completely different order). I actually chalked this up to my background in software development, figuring I was more wired to think of state as something that would be cleared every time a script was run. Analysts (statisticians, engineers, data scientist types) like to interact back and forth very iteratively with their data, and I think they're less likely to be tripped up by some back of mind assumption that you can determine the state of variables and so forth by looking at a bunch of commands as if they were run top to bottom in a file.

It got me thinking about when I'd use jupyter notebooks for my own work. In short, if I were writing a program, even just a one page script, I wouldn't use an interactive notebook (I have opened .py files in jupyter and just used it as a text editor).

But if I wanted to "do my math homework"? A one off where I need to get an answer to some complicated questions that will require a lot of data wrangling and stats? Yeah, I'd probably reach for jupyter notebook or something like it.

Otherwise? I can't remember the O'Reilly book where this was written (sorry I wish I had the cite) but the author wrote that his favorite dev environment is python and a text editor. That's still the case for me as well.

I hate to conclude with the disappointing "the problems are when you use it for the wrong task", mainly because I've objected when people have made this argument in the past. If a tool tends to get used for the wrong task, that's partly a problem with who is using it... but we shouldn't let the tool itself off the hook too easily. It's still worth thinking through. I'm worried I'm falling into the trap of making this argument when I like the tool, and objecting to it when I don't. I'll have to think about that a bit.

As it stands, though, I haven't had a huge problem with Jupyter Notebook, and I like it a lot, probably because I don't use it when Python + vi (or a "better" but still basic text editor) would do the trick.