DARPA is effective, but this document paints a rosy picture that is far from the truth. I've been a part of DARPA projects both commerical, university and on the government side.
The real process is:
1. PM gets picked due to knowing someone or being a former employee.
2. The biggest test of a program isn't if it's doable or a good idea, but if its able to be transitioned to another government agency with deeper pockets.
3. Most contracts are lost before you begin writing, as people have insider information about what the PM wants. This is done through just talking with each other (remember that most of the PMs come from the same companies), and not through any other formal process.
4. DARPA has some really cool stuff, but fails to transition it well enough (leading to 2.)
DARPA is not without it's problems, but has a better track record then NSF (NIH has them beat). What is funny is that you quickly realize how much bunk there is in scientific research and how many papers are not replicated.
this aligns with my experience. NIH has a better track record than DARPA, which has a better track record than NSF and the worst is the DOE. The DOE knows this and is trying to cargo-cult DARPA through stuff like ARPA-E. Predictable results have ensued.
Ummm.. The DOE run National labs have pioneered fundamental science. A lot of modern materials science technique development originally happened at some DOE National Lab.
Judging basic science by productivity is the absolute worst metric to choose. DOE Labs are going down that path, and are becoming glorified university research groups all in the name of productivity.
Projects achieving their initial goals. Government keeps metrics on all grants, projects and contracts. The reason NSF is low is because they rely on universities who have poor quality control standards as they don't pay their researchers enough.
The real process is:
1. PM gets picked due to knowing someone or being a former employee.
2. The biggest test of a program isn't if it's doable or a good idea, but if its able to be transitioned to another government agency with deeper pockets.
3. Most contracts are lost before you begin writing, as people have insider information about what the PM wants. This is done through just talking with each other (remember that most of the PMs come from the same companies), and not through any other formal process.
4. DARPA has some really cool stuff, but fails to transition it well enough (leading to 2.)
DARPA is not without it's problems, but has a better track record then NSF (NIH has them beat). What is funny is that you quickly realize how much bunk there is in scientific research and how many papers are not replicated.