It's easy to call Excel the cause, but it's not, any more than the existence of memory management is the cause of memory leaks.
Putting a very powerful (if archaic) tool in the hands of people without the experience or understanding to use it effectively and then passing around the results for editing by other people with similarly (or differently) deficient ability with the program is the problem.
The tool is simply too flexible to be used for many of the things it's doing by many of the people who are using it. This is one of the good reasons that we (application developers) do our best to gather requirements: Excel is usually good at what it does, but it does so many things that understanding what it does that you don't need it to do, or how and why it does things you've accidentally done or need to watch out for, has become as -- if not more -- important as/than understanding what it was you wanted to do in the first place.
So yes... Excel plays a role in some very bad things. But no, it's not Excel's fault.
Are people taking this article seriously? I'm fairly certain that it's a tongue-in-cheek jab at various other articles recently blaming Excel for the Reinhart-Rogoff thing.
I don't know how anyone could see the last bit about Native Americans and think that the article was actually serious.
It's serious, though playful. Some of the cases are very interesting, particularly the one about Barclays getting burned by the lack of access control on data in spreadsheets.
the "hide vs delete" problem is incredibly prevalent in corporate financial statements.
Accountants during preparation of these documents often "hide" for a reason - they want to try out how certain numbers work, and want to keep their work-in-progress easily accessible without deleting them. And often corporate secretaries just send out these documents "as ready" without thinking that there's tons of confidential and sometimes very wrong (or very incriminating) numbers "hidden" as work-in-progress.
This happens so often on Quarterly financial statements that it's an embarrassment, frankly. Disclaimer: I worked as a lowly proofreader of financial documents, quarterly and annual reports, etc.
The problem is not Access Control. The problem is people failing to recognize when a document is "ready for public consumption" vs an internal working draft.
The real story of "lacking access control" is when these excel worksheets actually link into MS Access databases hosted inside the firewalls of these companies - and holes were poked in the firewall to let the data out because executives wanted to work on these documents in their lodge in aspen, etc.
I see this all the time - it's really freaky when the excel worksheet I was proofreading start reaching out over the net and into supposedly private networks deep within financial houses. If I run something like little snitch, it would show access attempts over nonstandard ports, etc.
Well, of course "it's always a people problem" as Jerry Weinberg says. But the Barclays example is interesting* precisely for its technical detail. The spreadsheet author used Excel's "hide" feature to hide some data. Had Excel given them the ability to restrict read access to some portion of the sheet, it seems obvious that they would have done that instead.
* Interesting to me at least, because I work on things like this. Not contradicting your comment at all.
I guess the poing might be where there is no peer review or quality assurance - these cultures are prone to produce this kind of errors. And Excel culture is most of the time a one man show for every spreadsheet. They are hard to audit.
I agree. It seem to me that the article is taking the Mickey out of these high profile people / companies who are quick to place blame on Excel. I would see most of these a simply human errors which could have been avoided had the people involved been more thorough. If they are able to model complexes financial and economic problems in a spreadsheet surely they are also capable of building safeguards and double checks into their sheets or have someone proof read the results. Evidently, they didn't ... Though luck.
Excel is part of the cause. It simply mangles and hides programming. So it creates just the atmosphere were people think they can "avoid" programming and use Excel. Not realising that they are programming but in a very bad way.
So don't try to hide and dumb down the programming part. This of course doesn't mean that mistakes wouldn't happen with better tools.
> It simply mangles and hides programming. So it creates just the atmosphere were people think they can "avoid" programming and use Excel.
This attitude bothers me somewhat. I'll certainly agree that the combination of Excel's power and ease of use allowing people to model really quite complex calculations and scenarios quickly and easily, and the ease of which they can do it leads to a inappropriate level of "respect" for the gravity of what they are calculating, insufficient testing, double checks, etc, leading to some massive blunders.
But many people seem to go beyond this, asserting that one cannot do such things properly in Excel. I've actually heard one person with an absolutely straight face say that Excel should literally be taken away from engineers and finance folks, and any calculations they need to do should be custom implemented by "professional" developers who will do it "properly".
People who make that suggestion don`t know that there are tools like ResolverONE. It is a spreadsheet that your finance people can use as usual. But when it comes time to validate assumptions and check the calculations, a professional developer can take the generated Python code, write unit tests and other kind of tests to make sure that it does what is intended. Creating spreadsheets could become a collaboration between professionals in finance and professionals in software development. The tools exist to allow this, but where is the will?
When modeling something where mistakes are expensive in Excel, a reasonable person checks their work. I've made Excel spreadsheets that are responsible for millions of dollars in inventory purchases. You better believe I check the shit out of those. I'm paranoid about making mistakes because a small number multiplied by a million is usually a dollar amount of consequence.
In my opinion, people who don't check their work, just don't check their work. Whether they're using Excel or a "real programming language" doesn't matter. You can't blame Excel because people are reckless.
One company I worked at was making billion dollar decisions based on tools made in spreadsheets. It was the only option really, the instances where we did actually have the bespoke tools we wanted were even worse! There is no way whoever made this felt they were avoiding programming though - this buggy, unmaintainable monstrosity was three-quarters VBA. I bet someone on work experience or an internship thought "I know how I can make this tedious manual process easier for everyone!" and (with the very best of intentions) created something that no-one down the line would ever be in a position to fix or update.
This reminds me of Edward Tufte's criticisms of PowerPoint [1,2]. Both PowerPoint and Excel are widely abused, in part due to their flexibility and initial ease of use.
However, in addition to problems caused by its users, there have also been problems resulting from design deficiencies in Excel: automatically and irreversibly converting strings representing gene names into dates [3], displaying some numbers incorrectly [4], and incorrectly evaluating some statistical procedures [5].
"Too flexible" sounds like someone implemented all the requirements. A programming language is too flexible too. That's not really the problem, which is more about the opaqueness and lack of DRY.
In the end it's always "pilot error", right? But some blame rests on Excel for creating a tool that engendered a level of confidence that was unwarranted.
Excel is designed to be used for mission critical applications under certain circumstances. And it does a piss-poor job of delineating what those circumstances are, it deserves a lot of criticism for encouraging people to misuse it.
I guess you would say the same for C, right? So are you saying that Excel works correctly but because its easy to get started with it creates unrealistic confidence? I disagree.
Well, what a about detailed study? Quoting from abstract:
"Microsoft’s continuing inability to correctly fix errors is discussed. No statistical procedure in Excel should be used until Microsoft documents that the procedure is correct; it is not safe to assume that Microsoft Excel’s statistical procedures give the correct answer. Persons who wish to conduct statistical analyses should use some other package."
The problem with Reinhardt-Rogoff has nothing to do with Excel. It was a programing error that could have happened anywhere. The problem is that it took them years to reluctantly share their spreadsheet. Open data research and computation is the solution to that problem.
Partly because a bad workman chooses poor tools to begin with. Tools do matter. It's easier to find a mistake in say, an SQL query or some R code than it is to find a mistake in an Excel spreadsheet, where you are trying to catch the difference between SUM(A3:A12) and SUM(A3:A10) in a thousand different cells.
> where you are trying to catch the difference between SUM(A3:A12) and SUM(A3:A10) in a thousand different cells.
Excel does a pretty good job in highlighting which cells are selected when you edit a formula, and it does a pretty good job of maintaining the meaning of the formula under sheet transformations. For example, if you inserted a row between rows 8 and 9 then the two formulae would be =SUM(A3:A13) and =SUM(A3:A11) respectively.
This may have been a genuine error, but the two researchers definitely started the process with a goal in mind, and when the results agreed with their goals they didn't bother to check.
This may have been a genuine error, but the two researchers definitely started the process with a goal in mind...
What would that goal be? According to Megan Mcardle, Rogoff was a mild proponent of stimulus. For example, he said this in 2012:
"Back in 2008-9, there was a reasonable chance, maybe 20% that we’d end up in another Great Depression. Spending a trillion dollars is nothing to knock that off the table."
Yes, it does highlight the cells. But if a mistake does somehow make it in, it is still exceptionally tedious to find the error. It is easy enough to expand a formula for additional cells but accidentally miss a cell, leaving it with the old formula. If you were using SQL, R, or Python, this class of error would never happen.
I guess it is just that to do a proper audit you need to have your logic on one page. Otherwise it is extreemly hard to follow the code. In excel every formula is sitting essentially on it's own page. It is easy to code this way but hard to audit.
Recent versions of excel do warn you if formulas omit adjacent cells. I agree with your sentiment to some extent but with excel 2011/2013 you have to actively suppress these warnings.
I'm not sure I understand why it is easier with SQL than Excel. Both seem to offer similar roadblocks and both seem to offer similar solutions.
I think we're thinking there is a technological solution to a human problem, and personally I don't think there is or could ever be.
This is a process problem. And by process I mean the process of building complex datasets, validating them for "correctness," and judging the quality/correctness of different data sets.
Unit tests are a massive asset to programmers. I wonder if unit tests (i.e. "sanity checks") would also help in this situation? I mean you would have to force people to write them and monitor them, but once they've been created they pay for themselves by picking up unexpected errors.
One of the big problems with science is the proliferation of Excel "databases". It is very easy to look at a lot of numbers and make quick calculations with them. However, when you start getting into extremely large datasets, your propensity to make mistakes increases. A2:A10 here, B3:B11 there, etc... This is one reason why recent versions of Excel warn you when your formulas aren't in sync.
However, all of this is fixed when you are using SQL to properly query a database. Why? Because you are forced to write a SQL statement that details exactly what you want done. With Excel it can all be hidden away behind the cells. With SQL, it's out in front, so it's easier to check.
People like to use Excel because they can get an answer quickly without all that "programming". The problems start to arise when you need better tools, but only know Excel. So, in this case, it's not a matter of a craftsman blaming their tools, it's closer to an amateur trying to pretend to be a professional.
Excel is a wonderful spreadsheet. It is a horrible database.
Unless you write SQL queries that use stored procedures or pre-calculated results tables, and then you just wind up in the same situation as Excel.
I mean Excel is at its heart a query language. So your logic equally applies to Excel, why not write a massive query in Excel that does all the calculations in one go so you can see the inner workings?
All you're really doing is playing musical chairs with the data. SQL query language for Excel query language, and data moved from tables to worksheets.
As I said earlier, unless there are procedural changes upstream nothing will change. A tool is just a tool. You can use it in a way to minimise mistakes, or not. Humans are the weak point.
Now there are tools that automatically identify common mistakes but neither Excel or SQL/relational database engines are in that class.
> I mean Excel is at its heart a query language. So your logic equally applies to Excel
The least readable query language ever. Instead of names, everything must be addressed as column,row. Imagine a C program that instead of declaring variables with sane names as needed, simply created a large array of each type and used constant numeric indecies. That's what Excel requires.
Excel cells have been nameable since forever... It also has supported types since forever. I'm literally talking Excel 2000 or older functionality[1][2].
With respect do you even know how to use Excel? Because everything you just said is incorrect.
Only one of the tens of heavy users of Excel I have worked with know of this feature on others like it. I am not a heavy use of Excel but my theory is that this is due to a failure of explanation by the software. Excel does not by its design encourage good coding, and most of the users of Excel do not have any programming background so they do not realize that these features should even be there so they wont know to look it up in the manual.
And it's easy to write horrifically bad, but working, code in C or C++. How exactly do you propose Excel butt into your workflow to expose advanced features?
If you plan to use Excel to aid you in making serious money (as countless business do), can't you fork out a thousand extra for an employee who actually knows how to use Excel properly?
Naming a single cell doesn't solve the problem of accidentally using a different range of values. In a normal programming language you would define a variable such as GDP containing a list of values for the countries of interest, in Excel every formula has to restate the range, which is what caused the mistake.
I hate it when people generate these contrived examples where they intentionally stack the scenario to favour whichever way proves their point the most.
Which is easier to understand:
=SUM(profits01_2012:profits12_2012)
Or:
SELECT * FROM dassaddasads WHERE date > 1325394000 AND date < 1356930000
I have seen literally hundreds of Excel spreadsheets from banks, consultancies, and VC firms and I have never, ever seen anyone name individual cells the way you are proposing here. If I had to name individual cells in a monthly model spanning the next 10 years over 250 line items, I'd quit that job.
I'm not saying that Excel can't do it, but a) it sounds like it would take a huge amount of time, so b) it is not practically done in "the real world."
They aren't stereotypical of the kind of Excel that gets written by people who use it as a major part of their job. They're stereotypical of people writing poor Excel queries against unnamed cells via location reference.
It was one of many problems. The biggest problem, in my opinion, was excluding some years from their analysis without any explanation or reasoning behind why they did so. Again, peer review would have crushed this paper before it got out the door.
The really amazing thing is that the authors have not retracted the study, and, last I heard, they claimed that their analysis was correct in spite of the errors and exclusion of data which did not support their thesis.
Once again: this business with Excel mistakes seems like a red herring. The bigger problem is that the Reinhart-Rogoff result never made much sense to begin with. Correlation isn't causation; in fact, there's a strong intuitive case that it's the reverse with debt and growth.
Yup, but the Excel error is the soundbite. Anyone can understand it and it makes the authors look ridiculous, which is so much more potent than just being wrong, especially if what you're wrong about is something abstruse that in the end will only get reported as "economists disagree".
I've often wondered about this aspect of political discourse. It's not rational, it's symbolic. Yet most of the time when a symbolic soundbite takes off, even though it's wrong or misleading per se, there's still some poetic justice to it. That is, it's rare that such a soundbite is completely unfair; usually it turns out to be a metonym for an arguably legitimate criticism. This is a case of that.
Both situations are intuitive. High debt causes high servicing of debt causes inefficient use of private market resources paid in taxes.
Additionally, slow growth causes lower govt revenue causes less incurring of debt to make up the shortfall in tax base.
Debt and growth are both inputs and outputs that are codependent. The degree to which they impact the overall financial picture depends HEAVILY upon the opportunity cost of the servicing of the incurred debt.
For bonus, consider what will happen to heavily indebted countries when interest rates begin to rise. Servicing of debt will become more difficult leading to a decrease in our ability to grow the economy, leading to more difficulty in servicing the debt, and on and on.
The original paper doesn't make any claims about causation at all, does it? I haven't read every single word, but the Abstract certainly doesn't say that. The closest the paper says is that high debt and low growth are "associated."
Obviously, Excel isn't exactly at fault -- it's the people using it, as echoed by many comments here.
HOWEVER, spreadsheets are rather unique in the fact that they hide the very formulas used to generate everything, unless you have a specific cell selected.
So, opposite of computer programs, they're almost impossible to "read through" and verify that everything makes sense. If 99 cells have identical formulas, and 1 in the middle has a slight typo, how on earth are you ever going to see that?
I wonder if there are other similar spreadsheet-like tools that one might use, that make a point of showing everything going on in the spreadsheet, rather than trying to hide it?
If we want to enter 99 identical (or relatively identical) formulas into 99 cells, then we should be able to select all 99 cells, and enter the formula once without any dragging. If we want to change that formula, we should be able to change it once and not need to remember to redrag. We should be able to enter a formula into the column heading and know it'll apply whenever relevant to every cell in a column. That we can't means Excel (and Lotus and OpenOffice) is defective.
What QA is to blame for isn't letting unfinished spreadsheets out the door, but rather for letting users in a company use defective tools like Excel and Lotus and OpenOffice in the first place.
Many of us developers learnt the hard way that visual programming tools from the 1990's don't work, which is why scripting languages became popular. Users in corporations (and the IT staff who authorize what tools that can use) need learn the same lesson.
> we should be able to select all 99 cells, and enter the formula once without any dragging.
Agreed. You can do this by creating your own macros, but it's a pain. Considering how often I do this for personal spreadsheets, you'd think that this would have been simplified long ago.
And, this deficiency has been ported to both Numbers and Google spreadsheets.
After reading your comment, I did some googling to find something that could convert an existing spreadsheet into code. I found a few commercial solutions that turn Excel into C++ or Java. I think the idea is that Excel becomes the prototyping tool for non-coders. That seems like a better idea than treating Excel as the final product. I feel like there's a new paradigm for both spreadsheets and programming in there somewhere but I need a bigger brain to see it.
I suspect the reason those commercial solutions aren't very prominent is that they don't work very well. Spreadsheet compilation is a harder problem than it appears to be.
Related to this is that there are no descriptive variable names either. Just a massive three dimensional grid of, non typesafe, memory indexed by letter and number: sheet/row/column.
Imagine programming like this:
A5 = (C6*C6 * Sheet5:D1 + Sheet6:D1 / B3)
I'm not saying you have to use excel like this, there are options to do for example Sum(myTable[myRow]) and pivottable reduces alot of manual equations also. However both of these alternatives require that you format everything nicely into one table per sheet like a proper database and use the "Format as table" option, you almost need a DBA to use excel properly! The problem is that normal people don't use excel like this, they paste random data around everywhere, they want to have the sum, the average and lots of other calculations both below and on the right side of the data so you can see both simultaneously but once you do this you've screwed up your table and we're back on square one.
If you've got 99 cells with identical formulas, and one without, it's not far fetched for Excel to speculate that there may be a mistake (particularly if they're within X% of being identical). At which point it can provide a simple, non-intrusive hint.
And going further up the wishful thinking tree, into more difficult territory, Excel should be able to grasp what you're attempting to do by understanding, say, the top 10,000 (arbitrary number) tasks commonly performed by people using spreadsheets. That 'understanding' could then tip Excel toward deciding whether there's a problem, and whether it should leave you a nice little illuminated icon at the top right of the program with an exclamation mark.
Microsoft should have insane amounts of data on everything people do with spreadsheets, and they should be able to make it radically more intelligent (without being obnoxious or intrusive - it should be optional and highly passive).
If you've got 99 cells with identical formulas, and one without, it's not far fetched for Excel to speculate that there may be a mistake (particularly if they're within X% of being identical). At which point it can provide a simple, non-intrusive hint.
It already does this, actually. Enter some data in two columns, like:
4 1
2 3
5 6
3 2
4 2
2 1
Then enter a sum formula in the next column like =SUM(A1:B1) and drag it all the way down. Change one in the middle to something else, like =SUM(B3:B3) instead of =SUM(A3:B3) and a little green arrow appears in the corner. Hover above it an it explains there is an "inconsistent formula".
You can actual show formulas in excel, instead of the values.
But that's not to say that it'd be easy to spot the one that is different. This is why excel has a little tooltip that pops up and says 'inconsistent formula' when it finds a cell that doesn't seem to belong.
HOWEVER, spreadsheets are rather unique in the fact that they hide the very formulas used to generate everything, unless you have a specific cell selected.
Unless you click the button to "show formulas" in Excel, then you can view them all at once.
I got screwed again by my damn notepad. I needed a quart of buttermilk for a recipe, but wrote a pint instead on the memo pad.
If only I had used a recipe management package that would have automatically told me that I wouldn't have enough buttermilk to make waffles for 5 kids.
It gives you an error if the dimensions don't match, but unfortunately the error messages are hideous.
Anybody can define new units just by giving a conversion factor and a string to display. These new units immediately work with other units. You can probably define new "dimensions" as well, but that's bound to be more involved.
All the units are represented as empty types, so it should have no overhead at runtime.
So it is very nice syntactically, with the only real problem being the error messages and the fact that it uses a whole bunch of extensions and only supports very new versions of GHC. I think it's pretty cool.
F# has units of measure[1], which lets you use units on numeric types. It also does unit conversion, so if you have a function (x * x), the type sig would be "x:int<m> -> int<m ^ 2>". Pretty neat.
Without the neat inherent aspects, any language with lightweight support for phantom types can achieve some level of this functionality.
For instance, you could define a sum type with only one case taking the type to wrap. Thus you can easily have TransactionId, AccountId, etc. as distinct types, without spending more than a line to declare them.
This should be easy in many languages - inherit from a numeric type or wrap one and overload the operators. This is actually not uncommon - in areas with high risk potential this is standard practice up to using different data types for x, y and z coordinates to avoid that your flight control accidentally adds wrong things together.
Excel is an incredibly powerful environment to code. I learned it many years ago; I started thinking it was some arcane tool that accountants used and ended being able to deliver amazingly powerful tools to users orders of magnitude faster than systems developers. Yes they had bugs, but... so did the industrial systems!
Excel's real limitations came around scalability of developers (beyond one developer you are in a bad place) and performance that can drop of a cliff beyond a certain size. But it is an astonishingly powerful tool.
Let us all stop pretending that people were pursuing austerity because of some paper by Reinhart and Rogoff. Cutting government services to citizens for the sake of repaying foreign debt has been going on as long as there has been foreign debt. And that is longer than three years.
Reinhart and Rogoff were important because their idea supported an existing financial theory. If their paper had reached an opposite conclusion, there still would have been austerity.
Now that their paper has been shown to be in error, there will still be austerity. And that will be because the people who benefit from it have the power to impose it. Where they don't have the power, it will not be imposed.
In bioinformatics work, Excel auto-formatting can screw up genomic data http://nsaunders.wordpress.com/2012/10/22/gene-name-errors-a... . The problem with it is that formatting will modify entered text by default, and the user has to be actively aware of this behavior to undo it. Regardless, spreadsheets are invaluable for visualizing and processing tabular data -- but default behavior and user-end awareness are the lessons here.
When I started my thesis there was a graph passed around essentially outlining the difference between Latex and traditional word processing, and it essentially looked like http://bit.ly/ZAtwNi, with word processing packages starting off easier but exponentially getting more difficult to use and Latex being harder to start with, but easier in the long run.
I feel like the same thing applies to Excel. Essentially what gets built with Excel are small programs where all the variables are unnamed, everything passed around is a numerical value and there's almost no capacity to see the entire 'program'.
These always get built because Excel is easier to start with (and you can always pass an Excel spreadsheet around and let other people 'run' it), but it means in the long run with the bigger programs you hit exponential complexity pretty quickly and by then you're invested in Excel, even though R or Pandas would have been more appropriate for the size of the program. Just my two cents (coming from businesses that LOVE excel for completely insane purposes).
From a recent Nassim Taleb facebook posting: "Rejecting a macroeconomic idea (Rogoff and Reinhard) over an excel error is exactly like falsifying astrology over a computer glitch."
Yeah, whatever. The only thing worse than Excel is the typical corporate-developed big Java application produced by a bloated team of mediocre programmers grown to further some middle manager's empire-building desire rather than to produce quality software. I'm not surprised that people doing organizationally key work give up on interfacing with little wannabe-CIOs counting subordinates and just get the work done with a spreadsheet.
Not to mention Excel often ends up having better performance than most home-grown solutions, especially now its core functions are properly multithreaded.
The problem is people being satisfied with a free application for doing mission critical calculations. Yes, you buy an MS Office licence and Excel comes free. That is fine for run of the mill everyday calculations, but it is not OK for mission critical work and it is NOT ok for the CFO or anyone who advises the CFO on his decisions. Those important financial people should use a spreadsheet like this http://www.resolversystems.com/products/resolver-one/ backed up by a team of software developers who write unit tests for the spreadsheets and validate the calculations.
TLDR is that the ResolverONE spreadsheet writes Python code as you manipulate the spreadsheet. A software developer can then organize that Python code, write unit tests for the code and validate that the calculations do what the CFO intended. Once this extra work is done, then the business can have confidence in the results of the calculations. But the user interface, i.e. the spreadsheet that end users see, is the same as it was originally.
This allows mission critical software development discipline to be applied to the development of spreadsheets. I suspect that the existence of a tool like ResolverONE is one of the factors that led the SEC in the USA to require Python source code for new derivatives that are created, so that there is a standard way to express the calculations of the derivative.
I'd say Excel is superior to those systems for any one-person project. Especially monsters like Bugzilla, which shine for huge projects but impose a lot of overhead on tiny projects.
The trick with Excel is knowing when you've crossed that inflection point where you need something better. Unfortunately most people can't see that inflection since they only know Excel. And for others, it might be hard to transition away because they have all of their data in Excel.
I just choked on my soda. Bugzilla packages install in a command line and you can be up and running with a product, a couple of components for categorization, and user accounts 15 minutes later.
The only time I don't take that 15 minutes is if I'm working on a project where I'll NEVER be working directly with anyone else. Even then, it's often nice "just in case" someone else comes along later and wants to collaborate, you need to keep up with issues fixed in various releases, etc.
In particular, any piece of software that needs an MTA in order to run is not going to be pleasant when you are running a one-person project. What, are you going to want to receive email updates from yourself whenever you update a bug that you created on the computer you are using right now?
I wasn't even talking about installation, however. Go to the Landfill if you don't believe me (http://landfill.bugzilla.org/). You want to create a new bug? You first have to create a product, then you have to create components. You have to fill out the entire form when filing a bug, even if you only have one product, one component, and only target one operating system.
Then think about the cost of running Bugzilla. You are now running a web server. What if you don't have a server? What if you just have a laptop for the road and a desktop at home? How are you going to do database backups?
Tell me: does this beat an Excel spreadsheet for a one-person project?
Edit: I'm not saying Bugzilla is bad software. I'm saying it's like a stick shift. If you put an automatic transmission in a car, you alienate users who prefer manual. If you put a manual transmission in a car, you alienate users who don't know how to use it.
any piece of software that needs an MTA in order to run
What distro doesn't already have an MTA installed?
Create product: Click the "new product" link then type in "My New Software Project" enter.
Create components: Under your new product, click the "new component" linke then type in "Miscellaneous", press enter. There, you have your product and a catch-all component for all your new issues. Divide into more meaningful components as time goes on.
What if you don't have a server?
Now I see the misunderstanding here. I was referring to professional software projects where I've had to work with other people, some of whom here and there wished to track bugs in spreadsheets. Any such project that I work on has at least one server available. Hell, for small consulting projects I just use one of the servers running in my house to bootstrap things.
You know, you can get a RaspberryPi and plenty of flash storage space for running as a server (with Bugzilla, an MTA, and everything) for projects for less than $50.
If you're referring to small personal projects then sure. Use a text file or a spreadsheet or strings tied to your fingers to remind you of things.
> I'd say Excel is superior to those systems for any one-person project.
> I was referring to professional software projects where I've had to work with other people
We're having different discussions. It happens. I was pretty careful about choice of words: I said "tiny" and "one-person" rather than "small". For a "small" project I'd use Bugzilla or Trac or whatever makes the grass greener with the least fertilizer.
Not every developer wants to be a sysadmin. Some developers are hobbyists with a single Windows machine—for those developers, Bugzilla administration might be an arduous task which requires skills they don't want to waste graymatter on.
> Bugzilla is dependent on the availability of an e-mail system for its user authentication and for other tasks.
> This is not entirely true. It is possible to completely disable email sending, or to have Bugzilla store email messages in a file instead of sending them. However, this is mainly intended for testing, as disabling or diverting email on a production machine would mean that users could miss important events (such as bug changes or the creation of new accounts).
Yeah, I've worked in groups like that. Simplicity works great until testers need to verify that the bug was fixed and see the way that the issue was found and how to reproduce it or god forbid the developer rationale in what he did. Then rather than looking in an easy to search system or record they're stopping developers from writing code so they can answer the questions. Then as is typical with informal systems, they have those conversations over the phone which means that the knowledge never goes beyond those two people.
All these failures just get magnified when the software is then distributed out in the field or new developers join the team down the road.
You can misuse any system. JIRA or BugZilla doesn't magically give the testers that information; the burden is on the developers to document what they did.
Personally, as a developer with other things to do, I'd rather just jot my notes in my bugs.txt file instead of remembering to put them in BugZilla's "Steps to reproduce:" text box.
I don't understand your point about searchability. It's easier to grep through a text file than remember the proper incantation to go in <random bug tracker package's search box>
I look at excel like a car. What's happening is that most people who use don't have a license. What's worse they've mainly learned to drive on the farm.
Now rather than hire a professional or take performance driving courses they try and do the same in high end usage and things go wrong.
It's not the sports car that's at fault by someone driving it at the edge of its performance without the proper skill. If the just drove it around town they'd be fine, but taking a high performance car to the track with no experience, well bad things happen. We've all seen youtube clips of high performance crashes from inexperience ;)
I wondered if there might be a market for an Excel correctness checker. If billions are on the line, it might be worth a few bucks. But apparently Excel can advise you of common problems like these already.
Probably that means such an add-on would not be sellable? However it wouldn't be the first time someone paid lots of money for something that software they owned already did just fine.
There is probably not much you could check. You could probably do some type checking but beyond that I see - ad hoc - not much one could do. Excel does already catch syntax errors and a few other errors mentioned in the link. Maybe you could make these errors more prominent. But should we really encourage more development using office software? Shouldn't we solve the actual problem?
The hard part would be designing a machine-readable and illiterate-writable language for formal specification of Excel spreadsheet requirements to validate against. And getting said illiterates to use it.
There is no other way to detect errors involving somebody using SUM instead of AVG or aggregating data from wrong fields (examples from the article).
Use a hammer. Hit your thumb instead of the nail. Blame the hammer.
Use a table saw. Cut off three fingers instead of wood. Blame the table saw.
Go scuba diving. Return to the surface too fast. Get the bends. Blame the equipment.
Drive a car. Have an accident while texting. Blame the phone.
Go fishing. Be careless. Put a hook through your hand. Blame the fishing rod, the reel and the hook.
Use Excel. Don't make an effort to learn what you are doing. Mindlessly copy and paste. Do not seek out peer review. Cause massive financial losses. Blame Excel.
I'm reminded of a similar situation involving treasury secretary Timothy Geithner and TurboTax. He initially blamed the software for underpayment of tax (or at least he said he used turbotax) and later admitted that he fudged the numbers.
Huh, a second ago there was a comment here asking how this article was on the front page... then it was disappeared. Call Alex Jones.
I wanted to voice my thought anyway... this article prominently features the information that the mainly cited study showing that national debt is bad is no longer valid. Those wanting to fight for higher debt levels are absolutely ecstatic and will up-vote any article that even tangentially mentions the Reinhart/Rogoff failure.
People could use R or python and do similar mistakes. I am creating a small algorithm to analyze stock prices and I am just realizing how easy is to mess up in these things.
Unfortunately, these things are treated like earthquakes in some countries: it's rare, we don't care. However, bad quality of reports is not rare. But the problem is: companies do not have other ways. They are stuck with the big players (SAP, Oracle, etc.) and they can't understand why their IT stack is crap.
I wish from all this Excel bashing we could have a serious talk about these tools. BEFORE writing any line of code, we must first understand deeply why professionals use Excel in this way and what we can do about it.
Finally, realize that the "cloud" is not considered always "safe". People put in Excel all kind of "secret" data and they are scared of having them on a third party server. Please, consider this too when thinking about a possible solution.
There is an error in their reporting. The paper calling Reinhart-Rogoff into question came out of UMass Amherst, not Amherst (which implies Amherst College). Very different.
One is a large state school, the other an elite liberal arts school. UMass Amherst has a notoriously leftist econ program, Amherst College your typical neo-liberal.
A bad mechanic will always blame his tools. Having extensively used Excel once upon a time there is nothing like it out there, period. If you want a powerful spreadsheet, Excel is your goto guy. Just because people are misusing Excel does not make it bad.
Excel do not seem to be the cause - poor knowledge of using the tool is. You can argue that excel needs to be simpler than it is to learn - but then until something better for ubiquitous use (not forgetting SPSS and other tools which are way more complex) comes along - excel literacy is what is needed. Excel has its own quirks, but it also is capable of doing powerful modelling if you are ready to get out of the novice levels - but you also need advanced levels mathematics and statistics knowledge to be able to use them. Programming is not a panacea - financial and statistical modelling is beyond average programmer's level of proficiency.
People have raised some great issues with Excel. For me, the biggest problem that I dealt with all the time when I worked on Wall Street was just the lack of one, central place for all Excel "code" - to me, it makes it exceedingly difficult to understand a spreadsheet or debug it. The "smartest" use I've seen is essentially just using Excel to store the data and then converting the "code" to VB script. Of course, one can argue at that point you might as well have the data in SQL or other database as well.
There is a whole organization with an annual conference for the past 10 years, that is dedicated to analyzing spreadsheet errors. http://www.eusprig.org/index.htm
This is a big deal. In fact, the ResolverONE spreadsheet was created as a way to reduce the risk of the kind of errors that are discussed at the EUSPRIG conferences.
"A tribe of Native Americans entered the wrong row into the cell of the Excel ... Just kidding. The error was made in Lotus." - I wonder if Afrika is not making a big mistake with Ubuntu now :-)
Excel isn't the problem. Mediocrity is. Or, I should say, mediocrity when you need something more (like reliability, because you're making major decisions based on the data furnished).
Excel is great at providing a mediocre database, presentation template, calculation engine, and model builder, all-in-one. It's even Turing complete (although it makes it hard to write decent code, since that's not the point.) It does a large number of things, and a mediocre job of all. Often, this is enough. It gets the job done for small toy problems, and often that's a start. However, basing billion-dollar decisions based on Excel code? Are you kidding me?
This is exactly why I rail so hard (and without giving, ever) against Corporate injections into software. Corporate America is founded on mediocrity being good enough. Sometimes it is. Sometimes, it's fucking not. When it's not, stop using Excel as a database (it's not one) and fire all your CommodityJavaDevelopers writing SingletonVisitorFactories and start doing things right. Since you don't know how, you have to hire someone like me and give that person lots of autonomy, but it can be done.
Excuse me, but given your history of writing overly verbose, self-important Grand Proclamations: I imagine the only reason someone would have to hire you is if they wanted tomes written about how they could Do Things Right while probably not getting much actual work product done in return.
When someone spouts big talk like "start doing things right. Since you don't know how, you have to hire someone like me and give that person lots of autonomy, but it can be done" and all they're known for (please correct me if I'm wrong here) is writing a lot of text on HN, I don't think a little snark is out of line.
I get that railing against unsympathetic targets like big corporations makes for entertaining reading, but I don't think that actually qualifies anyone for the job he was describing.
while probably not getting much actual work product done in return.
That was fucking low, man.
I'm actually extremely productive when things work right, but when people persist in fuck-uppery it's like a low-grade chronic earthquake-- not exactly dangerous, but jarring and impossible to ignore and eventually exhausting.
By the way, the danger of startups isn't failure. It's that you end up like me: someone who can't tolerate the mediocrity, idiocy, and resistance to creativity that most people bring to their working lives. In most corporate environments, those are affordable background noise. In startups, they're existential risks. Unfortunately, it's hard to unlearn an allergy to fuck-uppery.
I am humble in the sense that I have a keen awareness of my limitations. I don't know what happens after death. I don't know if there is a God. I don't know more than 1.5 natural languages. I'm a mediocre athlete. On most topics, I'm less smart than a person with a passing knowledge of the field, and my opinion is consequently less useful.
I'm arrogant insofar as I've seen through the corporate nonsense and, after having watched people way inferior to me making huge decisions that affect peoples' lives, I feel an urge to step up. I know for a fact that these intellectual children are not the best people to be making such calls, because I am better.
Now, that said, I'm far from the smartest person out there and if someone smarter than me steps up so I don't have to, then that's great! I don't care about being the leader and I'd rather not. I want a competent leader. If that person is someone other than me, then great! But usually the people who are smarter than me shy away from power; they're smart enough to realize that that competition is utterly soul-raping.
The world needs people like me with actual competence and the courage to, at least, try to turn it into something.
Notice I am not objecting to you making the statements you have made; I'm just saying that with such statements, will come attacks. Justified or not, expect them.
I like animated disagreement. My wife and I argue constantly. It's great, because although I have a lot of good ideas, I have plenty of bad ones. Ideas are best tempered with fire. Let's beat the damn thing until we have something good. It's a difficult, chaotic process and it takes a lot of passionate minds. Corporate processes tend toward creative mediocrity by default because no one cares. Well, if no one cares, then why are we having this fucking meeting? Just go off and do it, then.
Ad hominem attacks piss me off because they're just so fucking useless. They add nothing, but they change the discussion to a toxic one quickly.
Maybe he wasn't making an ad hominem attack. Maybe I'm just insecure because so much of my career has been wasted on manager-blessed bullshit and I'm finally waking up. I lost someone last winter, so the value of time has really been impressed on me of late.
There's a rather complex relationship between business success and being "better" in any sense other than the mere tautological fact of business success. It is not a very good way to "prove" anything beyond that, without either additional evidence, or axiomatic assumptions.
Mmmh, you write so much that it would be easy to write you
off as crazy lunatic (as some here do for sure). But you amassed crazy Karma in just 2 years and I very much agree with your last two blog posts, so I'll give you the benefit of my doubt. Also this comment about Excel is spot-on.
I think mediocrity is a function of the risk your company is willing to take. The more risk-averse, the more mediocre. I don't have data to back this up, but I'd bet dollars to donuts there's a correlation between risk-aversion and company size. For example, here's a speech by a guy working for IBM during days they were developing the PC explaining his gripe with the organization: https://www.youtube.com/watch?v=IbRmaIzGTOM#t=5m55s
My thoughts: there's good risk and bad risk and, in the technological economy, you need to know the difference.
Good risk: trusting smart people with their own time. Don't take this risk and you will fucking fail because everyone with an IQ over 125 will sabotage you (or, at least, abandon you for better things.)
Bad risk: not backing up your database, skimping on code correctness in safety-critical code, playing with guns.
People who don't know the difference should not be making decisions that affect other people.
Mediocrity != average. The average MIT CS grad student isn't mediocre, but even above-average corporate cultures (e.g. Google, which is legitimately top-20%) are mediocre.
I feel like mediocrity (as a pejorative) is an artifact of a convex performance function where excellence is 10x as potent as a half-assed performance, as opposed to concave work where it's 1.2x. We don't gripe about cab drivers doing "mediocre" work. We ask them to take us somewhere and if they do it, we're usually happy.
[ETA: I realize I didn't answer your question. The reason corporate mediocrity is evil is that it's an artifact of the wrong people making decisions. If you wish to excel, you'll make a real effort to listen to people and get the right ideas into implementation. But a lot of companies tolerate executive bikeshedding because they don't care about excellence.]
Most people pay lip service to excellence, but what they really want is consistency. In a corporation, a career developer that works his 40 hours and consistently delivers uninspired but clean and correct code is more valuable than one rockstar that creates a brilliant system in one weekend and spends the rest of the month burned out.
it would be easy to write you off as crazy lunatic (as some here do for sure).
Only the 10 or so Google trolls.
But you amassed crazy Karma in just 2 years and I very much agree with your last two blog posts, so I'll give you the benefit of my doubt. Also this comment about Excel is spot-on.
I'm considering starting an anti-consulting service focused on corporate culture. It's $40k for one month with a catch: after I do X amount of fieldwork and figure out all the things you're screwing up HR and culture-wise, I'd be contractually precluded from working with you in the next 5 years. Why such a restrictive and seemingly self-destructive structure? That structure means I will do something most people (who are looking to retain a long-term employment relationship) won't, and that's tell the truth. Burned Bridges Consulting, LLC.
I am considering launching as a freelancer and I'm trying to figure out how to finance it in the context of my bidirectionally extreme (high magnitude; both directions) reputation (which is somewhat an exaggeration of my actual personality; when I write, I'm trying to communicate and sometimes I drive hard.)
I'd prefer to spend most of my time on actual work. I'm getting back into Clojure and rediscovering the joy of coding (which disappeared, with fleeting breaks in it, when I went to Google and had to answer to a middle manager, which just sucks all the life out of what we do). That said, if someone is willing to give me $40,000 to tell him what he's doing wrong, I'll do so.
(In person, I'm much less abrasive, though. If I actually respect someone, and I respect most people, I'm nice.)
Burned Bridges Consulting would work because it's structured to produce the only employment relationship in which telling the truth is possible: one where there is no possibility of continuance.
I don't know if you've ever been in management, but if not I think that would be the wall you'd run into.
Imagine if a manager had tons of advice for programmers, ie; 'listen to what management says, build what we want not what you think we want, pay attention when we interrupt your coding!'
If you've already done the whole management thing also and it worked out well then good luck to you.
I'm actually a believer in removing management, at least of the traditional do-as-I-say variety.
The only credibility that has any place in anything is the credibility that comes from making others' lives easier-- solving their problems, teaching them, building things they want.
Are you able to actually solve problems (with leadership) instead of just pointing them out? You have no track record of that.... in fact, just the opposite. But if you managed to change things around you could become an easy hire and own a new niche. Maybe even start a nice big company around it. Work pro bono on two major things, show measured improvements on dimensions you promised to improve, and then ask for your contract rate.
To be honest, I'd rather solve problems with real work and lead by doing. I enjoy work a lot more than I enjoy yelling at people.
I think I'm poisoned by corporate environments (a startup being the worst) where you have to yell at people for days to get the permissions necessary to do a few hours of real work. I fucking hate that shit. Eventually, people stop yelling at each other about work (because they forget how to work) and are just yelling about the yelling. That's what Corporate America really is: recursive wankery with almost no one really working.
I get your point, and agree that Excel is not the problem. What I reason is the issue is that Excel was marketed as the program to use to do any kind of business data calculations and manipulations. As a result, people think that Excel is a great solution to a myriad of problems. You and me know it is not. But try explaining that to a multi million business that grew to that size whilst using Excel. Sure, they need something better, yet will almost never understand why. People are also afraid of change. They are convinced Excel is the mesiah of business software. How could you convince anyone to give up their mesiah?
> How could you convince anyone to give up their mesiah?
Yeah, you don't. But you let them fail, and then convince the next generation that their elder's messiah is a fraud. Mainly by being very vocal, and using the failures to showcase your point.
In a while, either everybody realise what they're worshiping is just some normal piece of data, or you started another holly war.
That's not why people don't want to give up Excel. It's because in Excel they can do what they want themselves rather than having to rely on programmers. That's a really big deal and it's not something we programmers can easily appreciate.
To use the old saying, when all you have is a hammer, everthing begins to look like a nail. To most office workers, Excel is their hammer. You can stretch it to extreme limits by using things like pivot tables, and complicated lookup functions. It's easier for them this way, that is to say they have fluency in excel, but none in programming, which means they fail to realize that a simple well written script could save them from every having to repeat a task.
Excel is extremely powerful for data analysis, and is getting more powerful still as new versions are released.
If a dataset is under 200k or so rows, I'll generally do analysis in Excel. Larger than that, and I'm turning to Python.
Even then, the end result has be be passed to someone who isn't an analyst/developer, so where does it go? You guessed it: into an Excel spreadsheet, formatted nice and neat, and sent up the chain.
You can't spell "autonomy" without "money". Well, yes you can. But the point stands -- replace autonomy with money in that sentence. There are organizational pathologies that lead to big companies not hiring the most talented and independent thinkers, but in so many cases the problem is that they don't want to spend sufficient money for a staff with the skills, experience, and creativity to provide solutions better than "works well enough".
The program is only as good as the input it gets. Garbage in, Garbage out. Lazy and incompetent people would misuse any tool no matter how good it was.
Putting a very powerful (if archaic) tool in the hands of people without the experience or understanding to use it effectively and then passing around the results for editing by other people with similarly (or differently) deficient ability with the program is the problem.
The tool is simply too flexible to be used for many of the things it's doing by many of the people who are using it. This is one of the good reasons that we (application developers) do our best to gather requirements: Excel is usually good at what it does, but it does so many things that understanding what it does that you don't need it to do, or how and why it does things you've accidentally done or need to watch out for, has become as -- if not more -- important as/than understanding what it was you wanted to do in the first place.
So yes... Excel plays a role in some very bad things. But no, it's not Excel's fault.