Hey, Stencila developer here. Thanks to the original poster for sharing the link and for all the interest. Unfortunately, the site is not handling all that interest too well (R session hosting instances filling up, timeouts,...). I'm working on it but please bear with me while I try to stabilize things.
Thanks again for the interest. I've launched a few more workers on EC2 for all of you attempting to play with the demo sheet (each of you gets a Docker container running an R session). Things seem to have stabilized for now. A good robustness test - more work needed! Sigh.
Hey great work. I like that people are re-thinking spreadsheets as users get more technically inclined to use more sophisticated tools.
What do you think about fixing R notation to not describe cells, but the named array? i.e. instead of Average(A2:B2) it's Average(this_array) where this_array is the name for the row/column data and when the Average cell is clicked, highlight cells A2:B2?
There are many cases around detecting the proper array name or the user not inputing column or row names, but due to the ability to quickly read and understand the code as it relates to the sheet may make certain trade-offs more acceptable (i.e. forcing he user to enter array names).
Your suggestion about named arrays is important and relates the discussion here around the approach taken in the OSX Numbers app and the now dead Lotus Improv. Its connected to a planned feature in Stecnila sheets that I call cell "mapping" - effectively projecting an R/Python object onto grid cells. There is some more about that here: https://github.com/stencila/stencila/issues/118.
I've been shifting a bunch of data analysis out of spreadsheets (and some adhoc SQL) to Jupyter/Pandas, and we've found some unexpected tradeoffs on both sides.
The lack of testability and version control (and really, long-term maintainability) is what drove us out of spreadsheets into Jupyter. We've found though that the workflow in Jupyter, even with Pandas, is not great for exploring and getting a feel for data --- we end up missing the very quick "poking around" you can do in excel or in a good sql client.
I'd have to use Stencila more to know if it strikes the right balance for the kind of analysis work I do, but I'm glad to see such a thoughtful attempt to try a new balance.
Try R and RStudio. It excels at 'poking around' data.
In addition, you can show any data frame as a spreadsheet of data, import/export CSV files, etc... Knitr, Shiny, Plotly and other technologies also make producing documents, graphs and whatnot super easy.
This Stencila also looks cool though. Spreadsheet + R....
Matt, check out my EasyMorph (http://easymorph.com) -- it's like Excel for tabular data with a comprehensive set of pre-built transformations to replace Python scripting. It's intended for both "quick poking around" and doing complex analysis.
Lotus 1-2-3 got people to use computers. One could argue that spreadsheets are the single biggest business innovation of the late 20th century. I don't think they're going away that easily.
I know people who type whole letters in Excel. I know an accountant in particular who I showed how to use MS Word for writing letters, but he prefers Excel. He writes full documents in Excel, prints them out and mails them.
Laugh if you want but spreadsheets are not going away in my lifetime.
My experience has been that it always ends up in Google Docs. Collaboration in Docs is just so stupidly easy compared to emailing spreadsheets around. And yeah I've been through several cycles of this:
start tracking in google docs, we'll get a system eventually.
-> OK we have bought a new PM system, let's start using it.
-> Everyone please start using the new system. Move your google docs there today.
-> I ... why is this still in Google Docs?
-> Ok we've paid a bunch more money to get all of the requested features in our new system. Please please please start using it
-> Sigh. Just share me on the Google Doc.
Purpose-built software systems allow you to ask a lot of important questions, but they don't allow you to ask all of the important questions. It's useful to have a tool that you can fall back on that gives you a basic representation of the data that you can manipulate with simple tools. For most people that will be a spreadsheet. You can also do it with *nix tools but they're really unapproachable for the uninitiated.
This is by no means observed in my experiences. It's certainly understandable how it could get there and what excel could do, but if that was a trend, its one thats faded.
Yes, it goes from Excel to Microsoft Project to the plotter to the wall. And then beings the multi-year process of watching the baseline and targets drift away from each other like ships at sea...
I knew a guy who did everything in MS Works spreadsheet. In 1998 those files were pretty much locked to that software.
It's useful experience though, watching someone use a screw-driver to hammer in a nail. For him it worked - he didn't have to faff around with tab stops or margins. He just clicked a different cell.
in win95 days, Winword.exe wasn't particularly stable, nor ergonomically consistent. I quickly resorted to type documents in Excel, abusing table layout ala html4 days. A bit heavy but clear and solid. Excel was a workhorse.
I think the more usual form of that phrase is "Spreadsheets are dead, long live spreadsheets!". And that captures the truth of the situation better.
Despite all the known weaknesses of spreadsheets, it is sheer hubris to believe that they will be supplanted by reactive programming (or databases, or anything else for that matter). It seems that users will give up their spreadsheets only when they are pried from their cold, dead hands.
European Spreadsheet Risks Interest Group has some great links to articles about the pros/cons/whattodo of spreadsheets:
http://www.eusprig.org/
Jocelyn Ireson-Paine has done lots of work with spreadsheets and their problems. Lots of links to spreadsheet sites from there:
http://www.j-paine.org/
Sorry, I can't resist indulging in a moment of pedantry here.
The phrase "The king is dead, long live the king!", which this title and many others play off of, is referring to two separate individuals, the now-dead king in the first clause and his successor in the second. The phrase is representative of a theory of monarchy in which, upon the death of the sovereign, their authority passes instantaneously to their heir, without the need for a coronation or a formal investiture, in order to foreclose on the possibility of pretenders to the throne arising during an interregnum.
So the correct (and I know you said 'usual and not 'correct', but bear with me) form of the phrase would be 'spreadsheets are dead, long live [whatever is succeeding spreadsheets]'. Of course, historically speaking, it was not at all uncommon for dead royals to suddenly reappear and reassert their claim to the throne....
> Sorry, I can't resist indulging in a moment of pedantry here.
I've always got the impression that this is, rather than an abuse, instead an intentional re-purposing of the slogan to convey exactly the meaning that it appears to have (instead of the one that it historically should). I'm a die-hard prescriptivist on most matters; but, since the sentiment that the grandparent expressed is common and the one the the literal historical slogan expresses doesn't seem to be, it's hard to see why we should cling to the latter and avoid the former.
Thanks for the comment. "Spreadsheets are dead, long live spreadsheets!" does actually capture the intent of the article more accurately. But I used "..., long live reactive programming environments!" because I wanted to emphasise the point that spreadsheets are actually reactive programming environments. The article isn't try to suggest that the spreadsheet interface is going to be replaced. Rather is says that by combining it with a "code-based" approach we might be able to create a whole lot of benefits for testability, audit-ability and reproducibility.
"Quite simply, we do not think the way we think we think. Human cognition is built on complex mechanisms that inherently sacrifice some accuracy for speed. Speed in thinking was the overriding evolutionary force for our hunter ancestors, and although occasional errors caused the loss of hunters, new hunters were inexpensive and fun to make."
Spreadsheets can be a pain, but be careful what you ask for to replace them...
Having contracted at two big american banks (cough jp, cough boa) that decided to build their own proprietary technology sinking ship solutions (cough athena, quartz ), it is a horrible experience.
The business get sold on the idea of the "benefits" of this new "solution" so they pay lots for it. Lots and lots. Then all tech projects are guided towards it, otherwise it becomes a "political" problem if you don't use it, even if you can present a technical case.
Basically, it works, just about, if you are building the equivalent of a simple option spreadsheet pricer. Even then, the effort required should ring alarm bells. Still, job for life for those that maintain it...
I think notebook environments like iPython provide greater flexibility in terms of programming. I don't think they can replace spreadsheet nor spreadsheet would develop functionality to replace notebook environments.
The missing link is a seamless experience between spreadsheet and reproducible data analysis (programming).
Hey, Stencila developer here. Thanks, nxsheet looks really nice - great work too!
I agree, somehow getting spreadsheets into the workflow of reproducible data analysis is important. They are a tool that I for one had ignored (focussing instead of reproducibility of text + code) - but there are too many people that use them to ignore them!
Iteration on spreadsheets is great, and there's opportunity for tons of innovation in this space. But this suffers from the same problems that most spreadsheets have. Errors like not averaging over the right set of data (which you pointed out) don't come up because the code can't be diffed in git. They come up because there's not enough clarity in whether a set of cells are consistent or not. Making this text-based doesn't add clarity because ultimately there will be too much text to read.
Here's a test for the most common kind of error. If I add data in A5 and B5, will that data be represented in the chart? How about in the averages? Will I be able to even see that?
Here's the pivot: Break your data into regions, where regions have repeated elements. Something like this:
The dataset associated with that sub-block can be clearly annotated to an editing user as such, with simple tools for adding a row to that dataset, or sorting it without effecting the rest of the sheet. Within the dataset, there's no corruption of the formulas (the third row's C cant be different than the second's), you've still got your diffing (probably better because it's clear that data changed but formulas different) and it's extremely hard to make the calculated average come out wrong.
Yeah, that exact representation isn't great. Maybe a "view" metaphor, so that headers and footers can be attached to the dataset instead of floating outside it. But once you've gone this direction, there's all sorts of amazing things possible by sharing datasets, linking them, transforming them, etc.
The Numbers app on OSX does something quite like this. It's highly underrated, but having a number of smaller, floating spread sheets in one big sandbox is waaaaay better for many things than a single sheet. I think the concept should be pushed even further.
And of course, Excel can flag cells that, according to its heuristics, seem to use inconsistent formulas, such as using empty cells in a summation, or having one cell in a column use a different formula.
Hey, Stencila developer here. Yes, I agree. I recently came across the Numbers app when I was writing this. I haven't used it but from what I have read it seems to have some really good ideas for improving the spreadsheet interface.
Kayia (kayia.org) is a spreadsheet/reactive programming environment but where the code can be 'diff'ed. All values in the spreadsheet are versioned, so you can either look at the current value of, say, Cell B2 or its immutable value at a specific point in time, or see how its value has changed over time. And since the code is stored in the same way as the data, version control of the source is a first-class part of the language.
Yes, you're right this doesn't solve all the issues with spreadsheets as a programming environment. The "averaging over the right data" is a big one, and yes, being able to diff a text file won't help that.
Thanks a lot for the suggested syntax - it's really good to get some feedback. I need more time to think it through and work out exactly how this would work.
In the meantime, one solution to the problem, that I would like to implement soon, is cell mapping - in effect projecting native data structures onto grid cells. That way you can still display and modify your data in the spreadsheet but use a sensible formula like `mean(data$height)` instead of `mean(A2:A5)`. Would love comments on that idea - there's an issue on it https://github.com/stencila/stencila/issues/118
This article reminded me of an interesting video by Chris Granger, co-author of the Light Table code editor, titled "In Search of Tomorrow: What does programming look like in 10 years?" He's designing a visual/code block hybrid editor.
Spreadsheets somehow hit a vein with the public, so that tells you something. I've been kind of obsessed with the Tup build system, and in trying to distill what it does into one sentence, I find it helpful to say, It's like a spreadsheet for files, where the formulas are shell commands. Even though I can grok FRP on its own, I still find that analogy helpful.
Calca is a beast — one of the most unique and useful OS X apps on my machine. FYI, the iOS version is free right now. (EDIT: I see that the author switched to a pay-if-you-want model for the iOS version.)
Exciting stuff. I'm a financial analyst and I spend most of my time in Excel because I'm not about to send an R script to our executive team. Everybody understands spreadsheets.
But spreadsheets are terrible at everything except their original use case (small bore financial modelling).
Right now I'm transitioning a lot of my workflow to MSFT's Power BI tools, but I'd love to have a justification for using more R at work. Stencila could be that.
Has anyone here ever used Lotus Improv? Does this program solve some of the flaws that this article highlights? I keep on hearing how Improv was one of the great programs that never took off, and I'm curious to now try it.
For those of you that don't know (or weren't even born yet), Lotus Improv was a spreadsheet alternative that was released for the NeXT computer in the early '90s that was well-reviewed but never sold well. It was eventually abandoned when IBM bought lotus in the mid-90's.
Hi, article author here. I have never used Improv but while researching for this work I did read about it and yes, it did seem to have introduced several innovations to address flaws in the spreadsheet approach. The OSX Numbers app seems to have picked up some of these ideas. And, I'm hoping to apply some of them to Stencila Sheets too.
This looks very sexy, and I don't say that often about technology. This is something I've wanted to see for a long time - the ability to seamlessly move from a spreadsheet to an application
Edit: Quick feedback
Cell names appear to be case sensitive. While this makes sense from a programming standpoint, if I can't have a cell named "a1" the application should convert that to "A1" for me.
Not being able to tab between cells is annoying. Was there a decision made not to capture the tab event?
Regarding cell names. Currently every cell has an "id" e.g. A1 which is represented in the background R session as the variable A1. If you enter a formula like "pi = 3.14" into cell A1 then that cell has an id of A1 as well as a "name" (like an alias) of "pi". You can name a cell "a1" e.g. enter "a1 = 42" into cell A1 - but that could be confusing.
Yes, not being able to tab between cells is annoying - no decision was made on that - we just haven't got round to it. Just created an issue: https://github.com/stencila/stencila/issues/162. Thanks for the prod!
It might be good user experience to either make variable names case-insensitive (could be done any time before R), or default a1/A1 as the same pointer.
You'll really want to appeal to less technical folk, and they have trouble with case-sensitive things.
It seems, perhaps, Resolver One's time has finally come. Alas, Resolver One (and Resolver Systems) died a slow death and ceased to be in 2012. Perhaps this will fare better.
I always thought Resolver was a brilliant idea, but they never gained any traction against the heavy weights (Excel).
I think another change is needed for spreadsheets to be a more reliable tool.
Right now a sheet is a full document thing. Meaning that it starts with A1 in top left, and spreads out from there.
but what if you could break a "sheet" down into units?
So that you can have say a constants unit, with its own A1 to whatever as needed, and then another unit, perhaps called results, that hold the formula in their own A1 to whatever, each unit movable and extendable on the screen at the same time.
Now if formula A1 wants to refer to something in constants A1, the entry would look something like constants(A1).
This way one would have everything on screen, while avoiding the worry that as the sheet grows and one move things around to make it more readable, the formulas are referencing the wrong cells.
One project I've recently been wanting play around with is Beaker, which touts itself as a polyglot data science notebook tool. The pitch is that you can use the right tool for the job, no matter what questions you want to ask next to each other. It looks like an iPython notebook on steroids.
This kind of makes me want to brush off POI and attach it to Scala to develop a git-able way to create spreadsheets.
Create your spreadsheet, import it to an IDE, and create a way to intelligently create reports that can be connected to business intelligence suites... Someone must have done this.
Hey, that's really cool, thanks for the link. Right now you can't do that...but give us a few weeks. We are planning to implement conditional cell formatting using CSS i.e. cell CSS styles as a function of R expressions https://github.com/stencila/stencila/issues/97. Once that is done, it would be fun to see if we could create these types of pretty picture.
very well done! a plaintext spreadsheet. I think it has a lot of appeal, and for all the reasons outlined. Looking at it raw, is fairly understandable. Awesome! Coolest thing i've seen all month on HN.
hi - quick question. Is your Dockerfile open source ? We have been building an R powered backend for one of our analytics services and it's been a struggle getting a scalable R-based infrastructure going.
BTW - another killer usecase would be an easy way to build database backed spreadsheets. instead of the TSV, it would be cool to figure out how to interface to postgresql for example.
Hey, yeah it's all open. The Dockerfile is here: https://github.com/stencila/stencila/blob/master/docker/ubun... I'm no expert, but it seems to be doing the job. Would appreciate any suggestions you might have! I'd like to make it smaller i.e. use a smaller base image.
just one comment - your entire js is basically building a reactive workflow through vanilla js. it is a very commendable piece of work!
however, I'm wondering if using Reactjs+Redux would not reduce the amount of js by many orders of magnitude. You would get all your reactive processing for free.
Yes, I think that they certainly help at making spreadsheets more readable. At present Stencila sheets only has named cells but one of the things we want to add soon is mapping of cells onto underlying `data.frames` or matrices. In other words to "project" data objects onto the spreadsheet grid.
OT and just my opinion: it seems to exists an unfortunate trend in the promotion of new ideas where, instead of simply exposing the virtues of the idea on its own, it first needs to knock down the (proven and world adopted) predecessor as something unpolished, unplanned, untested and, in general, a bad idea from the start.
Compare this trend, for instance, with the first Linux announcement by Linus Torvalds [1] below.
It is an unfortunate trend because it ends up polarizing and creating unnecessary division among the early adopters of the new idea and the current userbase of the old one.
I’m doing a (free) operating system (just a hobby, won’t be big and professional like gnu) for 386(486) AT clones. This has been brewing since april, and is starting to get ready. I’d like any feedback on things people like/dislike in minix, as my OS resembles it somewhat (same physical layout of the file-system (due to practical reasons) among other things).
I’ve currently ported bash(1.08) and gcc(1.40), and things seem to work. This implies that I’ll get something practical within a few months, and I’d like to know what features most people would want. Any suggestions are welcome, but I won’t promise I’ll implement them :-)
Linus (torvalds@kruuna.helsinki.fi)
PS. Yes – it’s free of any minix code, and it has a multi-threaded fs. It is NOT protable (uses 386 task switching etc), and it probably never will support anything other than AT-harddisks, as that’s all I have :-(.
The article itself is much more positive on spreadsheets than the headline -- it suggests improvements to the underlying data / computational model while preserving the interface.
And it's about time somebody said something nice about spreadsheets. Go to your average (non-software) engineer and ask to see an engineering model. Chances are, it's in Excel. It's less buggy than you expect, because your average engineer is going to notice and fix nonsensical results. Unlike the scientists the author is talking about, your average engineer is looking at very similar data from day to day and is attuned to anomalous behavior.
I think there's a lot of room for a tool like the one described by the article that includes built-in validation and is git-friendly. However, given the overwhelming dominance of Excel, it's going to have to integrate with, rather than replace Excel.
Thanks for the comments. Actually, I was just working on importing Excel spreadsheets into Stencila sheets last night. Some interesting aspects involved in translating Excel formulas into R expressions e.g. `AVERAGE(A1:A5)` to `mean(A1:A5)`. To do it properly looks like we'll need to build an Excel formula parser. But do-able - and could be fun!
How many times have we heard over decades now "yada yada yda is dead" as a headline? Cobol, CD's, PC's, even vinyl records, nothing really dies and especially Excel which will probably still be used for decades to come.
Thanks for the comment. Yes, the title of the article does sound negative (I was just trying to for something that caught the eye - something that is perhaps harder to do these days than back in 1991 when Linux was released). But the body of article is supportive of the spreadsheet approach and looks to ways that we can improve it and reach out to the existing user base - not to knock it down.
You would be surprised how many people have done this. The average office worker doesn't have a development environment on their computer, but they have Excel, and they have a coworker who was a whiz who said, "Did you know you could do this?"
They just kept automating task after task, silently, without the IT department ever knowing. 90% of them never thought of it as coding, and 10% realized they could make more money as an expert who knows how to coerce excel combined with their domain knowledge than they could as a programmer.
I've been at top 20 banks in the world that run 14 hour jobs in Excel that import data from web services, run analysis, and FTP result sets elsewhere. They were used to analyze risk on tens of billions of dollars worth of securities.
They were riddled with bugs, but it let the front-line traders iterate quickly and prototype something that the backend team (my team) could turn into a "real" reporting process.
Absolutely. I think it is the rapid prototyping aspect of spreadsheets that is perhaps the most powerful. For analysing large amounts of data they are obviously not great. But they excel (pardon the pun) at providing an environment in which to quickly formulate and visualise numerical models - because they are reactive so you can see what happens when you change inputs - without having to recompile, or manually rerun code.
I've done this a lot. I've always programmed but I'd say the most used things I've ever done are Excel workbooks. It's great, you don't really have to worry too much about compatibility. You're never like hey install this program so you can use this other program. I don't have to make a 'ui'.
Sure they're prone to breakage, they're kind of fiddly and inelegant. But they work and get the work done which is what I need to do at the end of the day.
And all these little macros could be one of the largest sources of increased productivity attributable to computers in history. Maybe even _the_ largest. I'm completely serious.
I've seen office workflows sped up by 20X or more with a little 1-2-3 or Excel macro hacking. Everyday minor miracles to the people who previously slaved at manually munging data. A student intern spends a few hours one day on macros, and the office staff each save three hours every day for years to come.
The spreadsheet has always and will always be the killer app of the PC.
"If you have been doing programming in excel for the past 20 years then you are simply a fool - sorry."
You've got to be kidding! Excel programmers rarely worry about the next paycheck. Programming in Excel may not be the most invigorating intellectual pursuit but it's a helluva steady job - there's always something that needs fixing!
I remember tweaking Excel input 'system' and decorate a spreadsheet with additional hints and checks so I can go furiously fast on a very large input-operator level task.
This stencila thing seems to be a marriage of the two: you enter your data in a spreadsheet type environment, and then you can write a reactive document backed by that data.
