Yes, other bottlenecks might be preventing us from seeing overall productivity improvements. We might require large organisational changes across the industry in order to take advantage of the improvements.
I guess we will see if smaller startups without many of our bottlenecks are suddenly able to be much more competitive.
> How are you measuring developer productivity?
We use a host of quantitative and qualitative measures. None of them show any positive improvements. These include the basics like roadmap reviews, demo sessions, feature cycle time, etc as well as fairly comprehensive business metrics.
In some teams every developer is using copilot and yet we can't see any correlation with it and improved business metrics.
At the same time we can measure the impact from changing the label on a button on our UI on these business metrics.
> Were those that adopted copilot and chatgpt now enabled to finally keep up with their faster peers
No.
> Is developer satisfaction improved, and therefore retention?
> We use a host of quantitative and qualitative measures. None of them show any positive improvements. These include the basics like roadmap reviews, demo sessions, feature cycle time, etc as well as fairly comprehensive business metrics.
Those are very high level. If there's no movement on those, I'd guess there are other things bottlenecking the teams. They can code as fast as possible and things still move at the same pace overall. Nice thing to know.
If you want to really test the hypothesis that Copilot and ChatGPT have no impact on coding speed, look at more granular metrics to do with just coding. The average time from the moment a developer picks up a work item to the time it gets merged (assuming code reviews happen in a timely fashion). Hopefully you have historical pre-AI data on that metric to compare to.
Edit: and average number of defects discovered from that work after merge
> look at more granular metrics to do with just coding. The average time from the moment a developer picks up a work item to the time it gets merged (assuming code reviews happen in a timely fashion)
We do collect this data.
I personally don't put a lot of stock in these kinds of metrics because they depend far too much on the way specific teams operate.
For example perhaps Copilot helps developers understand the codebase better so they don't need to break up the tasks into such small units. Time to PR merge goes up but total coding time could easily go down.
Or perhaps Copilot works well with very small problem sizes (IMO it does) so developers start breaking the work into tiny chunks Copilot works well with. Time to PR merge goes way down but total code time for a feature stays the same.
For what it is worth I do not believe there have been any significant changes with these code level metrics either at the org level.
I guess we will see if smaller startups without many of our bottlenecks are suddenly able to be much more competitive.
> How are you measuring developer productivity?
We use a host of quantitative and qualitative measures. None of them show any positive improvements. These include the basics like roadmap reviews, demo sessions, feature cycle time, etc as well as fairly comprehensive business metrics.
In some teams every developer is using copilot and yet we can't see any correlation with it and improved business metrics.
At the same time we can measure the impact from changing the label on a button on our UI on these business metrics.
> Were those that adopted copilot and chatgpt now enabled to finally keep up with their faster peers
No.
> Is developer satisfaction improved, and therefore retention?
No.