This took place around 2009. Back then I was working for Rapid7, on their network vulnerability scanner Nexpose. The codebase was mostly Java and was relatively large at 1M+ lines of code. We had many unit tests. Running the entire test suite took up to 40 minutes on Windows, and 20 minutes on Linux (Windows was always about twice slower on everything: building, product's startup time, etc.) The company had grown quickly to at least 30-50 software engineers. The problem was that every time one of them ran a build on his or her local machine (which happened multiple times a day) it would have to run the test suite and waste up to 40 minutes of this person's time. 40 minutes × dozens of engineers = lots of inefficiencies in the company.
I loved solving performance issues so one day I remember arriving at the office and making it my mission to investigate if there was an easy way to speed up the test suite. Our build system was based on Ant, and our tests used the JUnit framework. After a little time profiling Ant with basic tools (top, strace, ltrace), and taking a few Java stack traces, I realized that most of the wasted time was not actually running the individual tests, but many instances of the JVM kept being started and destroyed between each test. Our Ant build file was running the JUnit test with fork=yes, which was required for a reason I don't recall at the moment. This forks the JVM for running the tests. Then a little googling lead me to this:
While reading this documentation, I stumbled upon an unknown parameter to me: forkmode. What does it do?
"forkmode: Controls how many JVMs get created if you want to fork some tests. Possible values are perTest (the default), perBatch and once. once creates only a single JVM for all tests while perTest creates a new JVM for each TestCase class. perBatch creates a JVM for each nested <batchtest> and one collecting all nested <test>s."
Our Ant build file did not set forkmode, so it meant we were forking a new JVM for every test!
I immediately tried forkmode=perBatch and... the test suite ran 10× faster! 40 minutes down to 4 minutes on Windows. And Linux ran it in 2 minutes instead of 20 minutes. I told my boss right away but he was unbelieving. He asked that I check with our most-experienced Java developer. I showed him my 1-line patch speeding the test suite 10× and he said "I guess you are right, we can commit that." By lunch time the fix was committed and everyone loved me :)
Several years ago, I did a short contract for Apple Retail Software Engineering. They build all the software and systems that are used to communicate between Retail HQ and the stores, handle all training, etc.... Apple employees know that they’re not allowed to even acknowledge the existence of this software, unless you’re in a secure location, like the store back-of-house. More than a few times, I would go into a store for personal reasons, and as part of the regular chit-chat they would ask where I worked, and I would tell them. They would smile at me and maybe a little wink, but that was always as far as it went.
My first task was to automate the build process for all their iOS and macOS front-end software, as well as the Linux-based back-end support systems. They didn’t want their Engineers to have their laptops taken over for thirty minutes to an hour or more, every time they did a build.
So, I delivered a Jenkins CI/CD system for them. And although I’m not a developer, I was able to find and fix the problems in their code base that kept it from building in a server environment instead of on their laptops. I also added a lot of static code analysis tools, and they developed a lot more regression suite tests, once the builds were automated on the server.
And then I found out I had automated myself out of a job, and my contract ended. I was a DevOps Engineer, with a heavy emphasis on the Ops side of the house, and because I wasn’t a Developer, they didn’t have any more work for me.
After finding out that my contract would be ending, the first thing I did was to call my wife and tell her the good news — I would be moving back to Austin, and I would be there in time for Christmas.
As a contractor, I had a high hourly pay rate, which meant I was just barely able to afford to get a 1BR apartment in the Cupertino/San Jose area. But if they had offered me a full-time position, they would have had to offer to pay me at least $250k-$350k per year, for me to be able to afford to keep that same apartment. And that doesn’t include what they would have had to pay me to be able to afford to have anything remotely like our house here in Austin.
I really enjoyed most aspects of working there, and it was a really good experience, but I do feel like I really dodged a bullet there. Or maybe it was a grenade.
> But if they had offered me a full-time position, they would have had to offer to pay me at least $250k-$350k per year, for me to be able to afford to keep that same apartment.
Huh? Rent is expensive in Cupertino, but not THAT expensive. You can live quite comfortably with half that.
Ha! I did something like that for a company, in this case. Their core app was slow and not scaling and they were failing SLAs, I was doing sysadmin work then and not on the dev team then but I dove into the code base, found the bug fixed it, increased speed by 3000%. They had tons of DB clients inserting one row at a time to the DB. Batched them. Company went nuts everyone was happy since this has been hunting the business for months, VP came into the office and gave me a high five. CEO sent me a thanks email. Then economy shrank and pay raise was frozen. I got $0 after performance review which was excellent a few months later. My boss at that time dipped into his pocket and gave me a $100 gift card to Best Buy. I bought a VCR, VCR then was $120. :-D
I’ve always wondered, does this happen in US companies? Or is it only in very small shops? I’ve worked in a variety of big European firms, and salaries and promotions are discussed once a year, at review time. Even when changing jobs within the same firm, the salary will only be aligned at the next cycle
Except if you have some leverage like an offer from another company, then sometimes you can make your current employer to almost match the offer. Once I improved some workflow greatly because I took extra time for it (it was about some Lua debugging, I don't really remember the circumstances), I was told that this is part of my job, so no raise. Now I don't expect raises for things like that but I only invest extra time if an issue annoys me and even then I open a ticket first. Long build times are annoying.
I’m playing devils advocate, but the point is he used initiative. He wasn’t asked to improve the build, he was curious and found a solution.
Is that worth a raise? Probably not on it’s own but I was saying it’s a good time to ask for one if you are secretly thinking you should get one as you have a good good forward.
On a meta level a company that doesn’t taskmaster or track everyone’s time usage will get this kind of result from time to time. Curious professional just making things better. Continuous improvement.
Didn't ask for a raise right away. In general the company liked me a lot; so a few months earlier I had asked for and already obtained a +19% raise, and during perf review a few months later I got another +7% raise.
This took place around 2009. Back then I was working for Rapid7, on their network vulnerability scanner Nexpose. The codebase was mostly Java and was relatively large at 1M+ lines of code. We had many unit tests. Running the entire test suite took up to 40 minutes on Windows, and 20 minutes on Linux (Windows was always about twice slower on everything: building, product's startup time, etc.) The company had grown quickly to at least 30-50 software engineers. The problem was that every time one of them ran a build on his or her local machine (which happened multiple times a day) it would have to run the test suite and waste up to 40 minutes of this person's time. 40 minutes × dozens of engineers = lots of inefficiencies in the company.
I loved solving performance issues so one day I remember arriving at the office and making it my mission to investigate if there was an easy way to speed up the test suite. Our build system was based on Ant, and our tests used the JUnit framework. After a little time profiling Ant with basic tools (top, strace, ltrace), and taking a few Java stack traces, I realized that most of the wasted time was not actually running the individual tests, but many instances of the JVM kept being started and destroyed between each test. Our Ant build file was running the JUnit test with fork=yes, which was required for a reason I don't recall at the moment. This forks the JVM for running the tests. Then a little googling lead me to this:
https://ant.apache.org/manual/Tasks/junit.html
While reading this documentation, I stumbled upon an unknown parameter to me: forkmode. What does it do?
"forkmode: Controls how many JVMs get created if you want to fork some tests. Possible values are perTest (the default), perBatch and once. once creates only a single JVM for all tests while perTest creates a new JVM for each TestCase class. perBatch creates a JVM for each nested <batchtest> and one collecting all nested <test>s."
Our Ant build file did not set forkmode, so it meant we were forking a new JVM for every test!
I immediately tried forkmode=perBatch and... the test suite ran 10× faster! 40 minutes down to 4 minutes on Windows. And Linux ran it in 2 minutes instead of 20 minutes. I told my boss right away but he was unbelieving. He asked that I check with our most-experienced Java developer. I showed him my 1-line patch speeding the test suite 10× and he said "I guess you are right, we can commit that." By lunch time the fix was committed and everyone loved me :)