I've worked on HW/SW codesign, for mixed signal chips, initially for convnets and image recognition, and later as a more general DL architecture. My advisor is the guy who "found the missing memristor" at HP in 2008.
Looking back now I regret going into HW field. I should have applied to CS rather than ECE, and focus on DL algorithms (especially RL), or maybe even something like what Numenta is doing, because my primary interest is AI, not hardware to run AI.
The hardest part has been being the only DL/ML specialist in my group. Everyone else here is more HW-oriented. I only understood this when I did an internship where I worked with people I could learn from/discuss ideas with.
I looked at your CV, and my guess is that for you, the hardest part will be to focus on one thing for the next 4 years.