> If the toolchain wasn't corrupted, and this was in the original source code (C?) - it's not going to be too difficult to find in code review, if you are looking for it

I was part of a research project, in which, as part of the evaluation, we were given small Android applications (1-10k lines of code, not obfuscated). We were told that each of the apps contained a single piece of embedded malicious functionality hidden in its otherwise known/innocuous behavior. We were given a few weeks to find the malicious functionality. The main goal was to use our prototype software analysis tools. But in those cases in which we had to perform manual source analysis, the implanted functionality often eluded us for 10-25% of the cases.

Now, you might be able to find people who are more experienced at software auditing than us (a bunch of Ph.D. students in security and compiler-related areas). But you might also find more experienced implant writers as well. Also, in a case like this, you don't know how many implants there are, how much of the internal company network is also compromised, how many (if any) of your remaining employees is an attacker, etc.

Fair point - but I'm thinking that a company like Juniper, which has critical branding and revenue on the lines, can afford to hire the 100 top-of-the-line pen testers for 90 days to scrub through their code (with various members have different areas of speciality - which would be important in a situation like ScreenOS which has so much breadth). Figure a top-end pen tester goes for around $300-$400K/year + 50% consulting Markup - $600K/year works out to $150K/90 days * 100 pen testers = $15mm.

The level of performance out of a $400k/year pen tester is pretty good. 100 of them can really scrub a code base pretty well.

$15mm might sounds like a lot of money to you and I - but you would be amazed to see how quickly the money flows when there is a security issue at stake. Also - since the Target CEO (http://www.cbc.ca/news/business/tony-fisher-fired-as-target-...) got axed as a result of a security event, budgeting for security responses has strong CxO level support.

In addition to the consulting workforce you can pull in for this type of project, you also presumably have a highly motivated internal workforce that will be pulling long hours, and working hard to support the pen testers as well.

You won't find that many pen testers making 600K a year, I don't think you find a single one that's paid that much for actual technical delivery. Pen-testers are also not the people that should be doing code reviews necessarily as they most likely lack experience in that regard.

Money doesn't solve everything, you can bring out 200 people and they'll still won't find everything and being able to figure out how everything works.

The hardcoded password backdoor was well obfuscated but at least something still "easily" detectable the Dual-EC vuln where some one replaced the values with their own pre-computed pair I don't even want to know how they found that out other than by chance.

And when you'll start going deeper It's just gets more complicated It won't be easy to find a 100 people that will be able to do code review on say the kernel of ScreenOS world wide, heck considering the specific skill set that they'll need to have: being able to understand the source code, assembly, being able to figure out how if it can be leveraged in any way which will affect the security of product on a software or hardware level (we are dealing with mother of all registers here after all) and being available for hire that's not something that trivial.

And lastly this might not be solvable with money at all identifying backdoors in unknown code is very difficult the technology that can do so is very limited and the manual approach is prone to human errors and oversight. Even if you get a team of a 100 leading code review, malware and security experts in the world I would not bet money that they'll be able to scrub anything within a year. And I would not even attempt to hire a 100 of them, the time it will take to bring them up to speed would be pretty exhaustive, a safer bet would be to have your own developers which worked on the products from year to go over small parts of the code and flag something unusual.

If Juniper is smart they'll crowd source that internally and have various code snippets pop up on the screens of randomly selected developers and have them being evaluated, when a snippet gets enough flags forward it to an expert and have them review it. But that again only works if a) the backdoor is localized and not staged across various steps in the logic flaw it self, and b) the backdoor is in the actual source code and not only present in the binaries because the tool chain itself was compromised.

Life isn't simple.

Sr. Pen testers make $300-$400K/year. The companies they work for charge a 50% markup. For short projects on a tight timeline, the costs are even higher.

I don't know of any other profession that does do code review with explicit tasks of finding vulnerabilities in the code.

And pen testers love to critique the many, many, many ways in which CSPRNG can fail - even when appropriate algorithms are chosen, and correctly implemented - there are still ways in which compiler options can expose you to side-channel attacks. The Juniper Dual-EC vulnerability would have been identified in the first 5 minutes (Dual_EC_DRBG - WTF?!) someone skilled in that arena was looking at that section of code.

A hundred people gives you enough breadth to find people who are not only familiar with the higher level security concepts (IPSec, Firewalling, packet filtering, SSL, SSH, etc, etc...) in the ScreenOs code, but also people who can evaluate tool chains, compiler output, etc.. And get the job done in a reasonable period of time (90 days).

There is absolutely no way you could scrub your own code with your own developers in a short period of time - not only do they have endless internal responsibilities, they aren't experts in vulnerability assessment (typically) -, there just aren't enough of them to do this in a reasonable period of time. I can't believe that Juniper has large numbers of $400k/year pen testers just sitting on staff. Companies usually only hire a few of those people for several weeks a year - relying on code review the rest of the year.

Also - remember, it's unclear whether someone internal in Juniper installed this code in the first place - you want a third party to check everything over. Leadership has to do something very meaningful here, beyond just, "we had our engineers review the code" - bringing in a massive third-party high-level audit is the sort of thing that will demonstrate transparency.

I work in the field and I know very senior people even in the US NYC/Bay and none of them makes 300K. Sr. Pentesters top out around 200K in NYC and that's really top of the line (world known) guys.

Now don't get me wrong some one who's wearing additional hats, like a CEO/CTO of a small consulting group might make that money while still doing some technical delivery, but they make that money despite of that fact not because of it.

But sorry there isn't a company on the planet that pays 300-400K to their testing staff if you know of one please let me know, I'll relocate there in a heartbeat :) Heck the average pay in the US for pen testers I would say is even lower than London. http://www.payscale.com/research/US/Job=Penetration_Tester/S...