Academic papers today are not meant to be discussion forums. To be fair, undergrads and early-stage graduate students often use academic papers as a way to figure out what kind of debates are taking place in their field. But once you get into advanced levels, academic papers are little more than archives of discussions that you had many months and years ago. All the real discussion happens in labs, offices, colloquiums, conferences, by e-mail, and through countless other channels that everyone knows are much more efficient than academic papers. Only those who are relatively isolated from the rest of the academic community, like scientists working far away from major research centers, rely on academic papers to catch up with what everyone else is up to.
Also, sometimes it doesn't matter that the paper gets published a few months or even a couple of years late. Not every academic field progresses at the rate of cutting-edge science. In my philosophy PhD dissertation I criticized works that were 10-16 years old, and even older if you count how long they took to be published. But those works are still considered a hot topic in their field. Hardly surprising in a discipline that has been moving slowly and steadily for thousands of years. The same might be said of many disciplines in the humanities and social sciences, though perhaps to a slightly lesser extent. So whether or not papers can be a suitable discussion forum depends on the field and how people expect their discussion forum to work. Blanket statements that cover everything from philosophy to mathematics to the latest nanotechnology are not helpful.
Some of the other problems you mention, though, are real.
This article seems terribly confused. Of course academic papers are a terrible discussion forum. They're not supposed to be any kind of discussion forum.
Academic papers are supposed to be contributions to science. They take by definition a long time to publish because they need to go through a peer review process, and they are only interest to a narrow group of people because they typically advance the understanding of a very narrowly defined problem deep within their field.
Academics don't just sit in a dusty office communicating with the world through journal articles. We meet in conferences and work in joint projects, have informal discussion forums and correspond through a multitude of different channels. Many academics also have non-academic publications about their research.
Academic papers serve a very specific purpose and they do it well. Just because they don't serve some other purpose that you'd like them to serve doesn't make the process broken.
After the first year or two, you'll have some idea of what subfield you are going to be working in. At this point-or even earlier-it's important to get plugged into the Secret Paper Passing Network. This informal organization is where all the action in AI really is. Trend-setting work eventually turns into published papers-but not until at least a year after the cool people know all about it. Which means that the cool people have a year's head start on working with new ideas.
How do the cool people find out about a new idea? Maybe they hear about it at a conference; but much more likely, they got it through the Secret Paper Passing Network. Here's how it works. Jo Cool gets a good idea. She throws together a half-assed implementation and it sort of works, so she writes a draft paper about it. She wants to know whether the idea is any good, so she sends copies to ten friends and asks them for comments on it. They think it's cool, so as well as telling Jo what's wrong with it, they lend copies to their friends to Xerox. Their friends lend copies to their friends, and so on. Jo revises it a bunch a few months later and sends it to AAAI. Six months later, it first appears in print in a cut-down five-page version (all that the AAAI proceedings allow). Jo eventually gets around to cleaning up the program and writes a longer revised version (based on the feedback on the AAAI version) and sends it to the AI Journal. AIJ has almost two years turn-around time, what with reviews and revisions and publication delay, so Jo's idea finally appears in a journal form three years after she had it-and almost that long after the cool people first found out about it. So cool people hardly ever learn about their subfield from published journal articles; those come out too late.
You, too, can be a cool people. Here are some heuristics for getting connected: ...
TL;DR Academic papers are for presenting final results and assigning credit, not discussion.
"A reference graph is a web of citations: paper A cites papers B and C, B cites C and D, C cites D, and so on."
A bit off topic, but where is the computer generated version of this graph. This was the inspiration to Googles Page Rank algorithm, right? We do have Google Search and even Google Scholar now. But where can I access this gigantic graph of published research?
So, what you are saying is, that it doesn't exist? Why is no one building this? I understand, that there is less demand than for web search. But there still is a lot of demand from smart and skilled people, no?
Thx for your reply. Makes sense. Other than abstracts, the reference are usually behind a pay wall.
But how does, e.g. Google Scholar estimate the number of citations? Do they only look at papers which are not behind a pay wall or do they buy a subscription to all journals?
It still buggles my mind, that the science loving google founders Larry and Brian haven't tackled this yet.
With respect to item #1 (huge time lag), it's worth mentioning that in some fields—I'm familiar with psychology, neuroscience, and biomedical engineering, but this likely applies to others as well—many if not most labs are in some form of contact during the course of the year, whether it takes the form of planned or serendipitous meetings at conferences, email conversations, second-hand talks through mutual collaborators and similarly-minded faculty, or other channels. In these fields, it's pretty unusual to be working in a total vacuum, where your only method of communication is through academic publications.
I think the author is implicitly talking about academic philosophy papers.
Regarding point 8, the "bottom line" rule (http://lesswrong.com/lw/js/the_bottom_line/) is about process, not actual placement of the bottom line. It is about asking how the conclusion and evidence came about: is the conclusion derived from the evidence, or was the conclusion stated, and the evidence filled in afterwards to support it?
Sure, I will state my conclusions in the abstract of my (computer science, mostly systems) papers. But I arrived at those conclusions after designing and experimenting with the artifact I'm presenting in the paper. That my conclusion comes first in the paper is not what matters.
I assume the author does not mean to apply point 8 to engineering and scientific papers. But I want to point this out since he didn't say so.
But many of the author's points don't apply to philosophy papers, either - when philosophers are deciding what to write they don't start with their conclusion any more than scientists and engineers do. Likewise, the author's claim not to have been convinced by a philosophical paper seems odd to me, and suggests he doesn't have much familiarity with philosophy. If anything, I think there are more influential papers than books in philosophy, particularly of the analytic sort; think of Quine's "Two Dogmas of Empiricism," Gettier's "Is Justified True Belief Knowledge," or Putnam's "The Meaning of Meaning," for instance, all of which are more influential than any of the books written by these authors.
Academic papers fail as a Reddit-style discussion forum because they were never meant to be a Reddit-style discussion forum.
If you need to discuss things with people, join a mailing list or go to a conference or get to know them. The paper is not supposed to be the transitional step where you are bouncing ideas off people.
I agree that academic journals should be open access. I don't agree that it is unfair for subject matter experts not to publish every crank who comes down the pike in every journal.
If you want to talk about enforcing conformity as a bad thing, a discussion forum (particularly one with upvotes and downvotes) is the last thing you should use as a model.
I guess you are more directly engaged with Less Wrong and Singularity Institute and other online fora which lack any academic credibility, but sort of pretend to have some.
Who claimed that the primary purpose of peer-reviewed academic papers was to serve as an all-inclusive, real-time, 'democratic' discussion forum?
Next up: "4chan flame comments are a terrible way to incrementally advance the state-of-the-art in a narrow academic field, presenting theorems, experimental data, and citations to closely-related work."
This is the author. Where did I say anything about "all-inclusive", "real-time" or "democratic"? The closest thing I actually said is that discussions shouldn't take years, but I then pointed out an example of a good discussion midway between "real-time" and "years" (in that case, a few weeks).
>Next up: "4chan flame comments are a terrible way to incrementally advance the state-of-the-art in a narrow academic field, presenting theorems, experimental data, and citations to closely-related work."
Now I'm tempted to test that, by submitting my next paper as a 4chan comment and seeing what kind of response it gets.
Say what you will about academic papers, but I needed to know how to solve a tridiagonal matrix efficiently on a distributed computer, and damned if I didn't find some that told me how.
As a survey of the field, maybe papers in journals aren't so great, but as a way to find heavyweight answers to heavyweight questions? You bet your ass they're good.
(Of course my University pays through the nose so that I can do that, but that has no bearing on the format).
There is a reason why conferences are so hugely important. Often, what will be published as an article is presented beforehand on conferences, among knowledgable and interested peers. And while some conferences are invite-only or have to be paid for, most universities have a lot of conferences either completely open to the public or at least open to students. Often, just asking someone of one is allowed to just audit the talks without going to all he social events aimed at invited people will be enough in order to be "let it".
Overall, what bothers me most about this blog post is this:
>> Papers are, intentionally, written for an audience of specialists rather than a general interest group, which reduces both the tendency and ability of non-specialists to read them when asked
This repeats the same bias that everyone can say something about everything. Nobody here would complain that mailing list discussions about bugs, features or programming questions are hard to understand for non-programmers. However, a scientific discussion in a paper is supposed to cater to everyone. In reality, especially within the confined spaces of an article, one has to assume some baseline of knowledge in the reader, because one can't afford to explain all the basics one implicitly builds upon. To give an example: I do literary studies, and often people assume that they know just as much without having properly dealt with it, because hey, everyone can read a book, right? right? While that part might be true, every scientific discipline over time builds up a plethora of instruments for analysis of their respective topics, an understanding of the matter's and the subject's history, its pitfalls, shortcomings and strengths.
Also, I do not understand his paragraph regarding the "bottom line", since I know noone who writes the abstract first. The abstract is the very last bit to be written, after the paper's structure, arguments, theses, supporting evidence and rebuttals are decided, fixed and usually written. What both the cited Paul Graham essay and his text fail to see is the process of writing a paper. It is true that the actual writing process usually doesn't start with a thesis statement and a chain of arguments. But structuring a paper is just as much part of writing as finding words for what one wants to express.
The author hints at being an undergrad in maths, having trouble with the understanding of state of the art research. A professor I know once said this about degrees: undergrad & grad work are a necessity, they teach how to start a car, how to accelerate, break and also drive on basic roads (research methods/skills). A PhD then allows (and teaches) you to drive on the autobahn (research), where you can start to close the gap somewhat to all the people already on it going faster than you do. The actual entry into novel research doesn't happen until after the PhD. That's where journals and conferences, workshops, groups, funding etc. become interesting and important. First then are you within the actual target group for academic papers, since you are seen as an equal, equal also in terms of discussing on eye level.
Nobody expects complex technology etc. to be understandable by everyone, but one could say: "Everyone thinks they are a writer, much like many people stand in front of contemporary art saying they could produce the same thing." What they fail to understand is the complexity involved in the writing process. What sometimes looks easy or even "trivial" in a finished paper often has a highly complex, dense and far-reaching background.
[edit] Mhm...I apologise for the length of this comment, it grew larger than I originally thought it would... [/edit]
I disagree about the value of conferences, especially large ones, for actual discussion.
If you go to a large conference (e.g. the International Studies Association annual conference), you are almost guaranteed to be fielding questions from an audience that has not read your paper.
You would get more valuable feedback on your actual work (rather than your academic powerpoint/presentation notes) from a small colloquium than from a large academic conference. The only person is guaranteed to have read your work is the discussant, and if you're lucky the panel chair and other presenters.
Conferences are a good way to network and get a feel for the overall state of the field, but for having actual discussion of a problem or issue area they're pretty ineffectual.
I suppose I should have defined "conferences" more closely, which for me includes workshops, workgroups etc. Right now, my subject isn't the biggest, so people usually know each other, each other's interests and what most other people are currently working on.
So yes, different experiences with conferences here. However, you are obviously right that the bigger the conference, the less constructive the overall discussion. This in a way corresponds to the aforementioned amount of readers targeted. If you aim to reach a broader audience, discussion will be less in-depth, if you specialise and focus on a smaller amount, discussion is more likely to be more relevant.
I totally agree that conferences are important, and lack some of the "bugs" I identified. However, that wasn't the point I was trying to argue.
Nowhere did I argue that "everyone can say something about everything". That's a strawman. In fact, in the last paragraph I said the opposite: "All forums need some sort of moderation to avoid degenerating."
I do think that being readable by a larger audience is an advantage, all else being equal, to being readable by a smaller audience. Especially since, as explained in http://lesswrong.com/lw/kh/explainers_shoot_high_aim_low/, we tend to vastly overestimate the size of the audience who can understand us. However, there's a huge difference between "can read" and "can contribute equally to the author".
In an experimental paper, sure, the abstract is written last. However, as one of the other comments noted, my post is (by virtue of being originally written for Less Wrong) targeted more at philosophy papers, like in point #4. In experimental and mathematical papers, you're right that this is less of a concern.
For the record, I have already graduated college (with a major in math and economics).
>> For the record, I have already graduated college (with a major in math and economics).
Oh, the undergrad bit wasn't in any way meant as an ad hominem or the like. Apologies if it came across as such.
>> I do think that being readable by a larger audience is an advantage, all else being equal, to being readable by a smaller audience.
I don't necessarily disagree with this statement, all I am pointing out is that there are limits to how wide an audience a given paper can address. Specifically in more abstract papers (e.g. philosophical papers) is a profound knowledge important, especially in the history of thought and the foundations of our modern philosophy, together with the objections raised along the way. To give just one example, if you want to actually talk about something as abstract as postmodernism (an internet favourite, as shown also by xkcd etc.), you will need to know about its genesis, about its criticism of structuralism, modernism, idealism, society, art as an institution, the avant-garde movements etc. All I am saying is while addressing a wider audience is an admirable goal, it becomes harder and harder to achieve the higher the paper's complexity.
>> In an experimental paper, sure, the abstract is written last.
Coming from a non-experimental background, I still don't know anyone writing their abstract first. First comes an interest, an idea, something one doesn't understand, believes to understand, something one finds irritating, interesting etc. Then one keeps digging deeper, looking at other people's work and interpretation while simultaneously structuring one's ideas, arguments and problems. The abstract of a paper -- in my opinion -- can only be given after this process of sorting and structuring, of testing, evaluating and discarding is over. So I would like to disagree that a philosophical paper is the result of a vastly different thought process in terms of structure and argumentation.
The real problem is that 99% of research publications are, to put it bluntly, worthless. This is the root problem of the entire system which leads to many of the problems the author here is listing.
These 99% are often churned out because they help sustain someone's carrier or lab. To be published, crap research thus needs to convince people that it is smart and important. This is usually done by being written poorly.
The remaining 1% will progress human knowledge and help other scientists to continue down the right path. If they are important enough for wider dissemination, well, that's what science journalists are for.
I cringed at this point. Especially in mathematical fields where the abstract is in some sense an API -- it tells you what theorems you might use from this paper and how to apply them.
This post might be more useful if it offered more specific alternatives, rather than mostly criticizing the academic process. Imagine the same ideas reworded as suggestions for improving the process. Then the posts here could discuss the new ideas (useful!), rather than debating the criticism (tedious!).
Perhaps peer review should be done in public, as a page-rank style rating according to the authority of each reviewer in that domain. Would it be completely open, single blind? double blind? Good questions. But it might reduce delays and decrease the effect of politics on the process.
I think the central urge in there is to get scientific discussion more out in the open. The single biggest issue by far, is not the pace of the publication process or the level of writing. It is access.
I would argue that access would actually solve a number of his other complaints (poor moderation, percieved time lag, people don't wanna read 'em, academia selecting for conformity). If we raised the quality of content on the open internet, the quality of discourse would go up.
Could this be a niche for a software solution enabling scientists to comment/share/contribute on each other's papers faster and more efficiently? Perhaps in the end speeding up research times and in the grand scheme of things furthering the scientific process?
This article has some good points and some bad ones too.
e.g. Academic papers violate the "bottom line" rule.
If you were making stuff up as you write it down then it would be true that starting with a conclusion might bias your arguments towards justifying that conclusion. This might describe how a (bad) journalist writes, but not how scientific papers are usually written.
The majority of the work for a scientific publication usually comes well before writing the first word. Theorists will discard many theories before they find one promising enough to do extensive work (primarily aimed at proving that theory wrong) before they even consider publication. Experimenters must work with theories that have been through the above, conduct a complex (and usually expensive) experiment, and then analyze the results extensively before they start writing. The writing process can trigger some consistency checks and digging through references may reveal new spins for or weaknesses in the theory or experiment. However, if the entire point of the paper was wildly misguided to begin with, it's unlikely writing the paper in a different order (so that the authors aren't beguiled by their conclusion) will be what reveals it.
Now, the big benefit to writing scientific papers with the conclusion presented right up front is that you don't have to read entire papers to figure out what they're talking about. Scientists generally don't read papers to be led down a grand story with a surprise payoff at the end. The first thing they want to know is what this paper has to say so they can decide if it's worth reading. Science is big. Really big. Go to arxiv.org (a preprint repository for both articles that will be published and some that won't). Pick one specific sub-field and look at how many papers were submitted yesterday. The reason few people read material from outside their bailiwicks is because they barely have enough hours in the day to keep up with their own field and do their own work. If scientific papers followed the "bottom line" rule things would get much, much worse.
Personally, my biggest complaint about journals is how expensive they are. The only reason university researchers have "free" access to journals is because the universities pay (frequently exorbitant) subscription fees to the journals. It's better than having every researcher pay $30 every time they want to look at a paper, but it's still far from cheap. The really weird thing is that it costs several thousand dollars to publish a paper in a decent scientific journal. Yes, you do ground-breaking work, write a beautiful paper about it, and then you have to pay the journal to publish it if they accept it, and color figures are almost always extra! Where does all the money go? Journals have remarkably few expenses. In terms of producing content the only people they really have to pay is their editors. The authors pay and the referees work for free. Publishing the content does require some infrastructure, like servers and code. The print version requires type-setting, production, distribution, etc... However, I find it very difficult to believe that journals need to charge what they do to either creators of the content or the consumers of it. I'd love to see journals open up their books and reveal where the money is going.
Well, there's at least one way in which academic papers are a great discussion forum: People think a lot before posting and do thorough research on their arguments - and those of their opponents.
Also, sometimes it doesn't matter that the paper gets published a few months or even a couple of years late. Not every academic field progresses at the rate of cutting-edge science. In my philosophy PhD dissertation I criticized works that were 10-16 years old, and even older if you count how long they took to be published. But those works are still considered a hot topic in their field. Hardly surprising in a discipline that has been moving slowly and steadily for thousands of years. The same might be said of many disciplines in the humanities and social sciences, though perhaps to a slightly lesser extent. So whether or not papers can be a suitable discussion forum depends on the field and how people expect their discussion forum to work. Blanket statements that cover everything from philosophy to mathematics to the latest nanotechnology are not helpful.
Some of the other problems you mention, though, are real.