I used to give a lightning talk at Google called "Life of a Data Engineer" (~6+ years ago), where I tried to explain to Google Software Engineers and Product Managers the concept of a "Data Engineer" in other parts of industry, as well as Google Cloud Customers.
In that talk, I outlined three (3) primary archetypes which still exist today:
- Data Engineering for Business Intelligence (often a rebranding of yesterday's "DBAs")
- Data Engineering for Data Science
- Data Engineering for Machine Learning (precursor for today's "ML Engineers")
Having worn all of these hats in the past, and many others, in different contexts, one of the key points I wanted to convey was that, in practice:
- The Role of a Data Engineer on a Team is Complementary and Often Defined By The Tasks That Others Don’t (Want To) Do Well
There are many guides, tutorials, and courses that like to talk about tools and technologies, and yet very few that get real about organizational structures and the dynamics that cause these archetypes to emerge, peoples' desires to rebrand themselves as "Data Engineers," "ML Engineers," or "(Data) Software Engineers" instead.
There was an interesting survey of ~600 "Data Engineers" by DataKitchen from 2021 ago where they had some very relatable, yet eye-opening findings:
- 97% report experiencing burnout in their day-to-day jobs.
- 70% say they are likely to leave their current company for another data engineering job in the next 12 months.
- 79% have considered leaving the industry entirely.
- 91% report frequently receiving requests for analytics with unrealistic or unreasonable expectations.
- 87% say they are blamed when things go wrong.
- 69% say their company’s data governance policies make their day-to-day job more difficult.
- 89% report frequent disruptions to work-life balance due to unplanned work.
In that talk, I outlined three (3) primary archetypes which still exist today:
- Data Engineering for Business Intelligence (often a rebranding of yesterday's "DBAs")
- Data Engineering for Data Science
- Data Engineering for Machine Learning (precursor for today's "ML Engineers")
Having worn all of these hats in the past, and many others, in different contexts, one of the key points I wanted to convey was that, in practice:
- The Role of a Data Engineer on a Team is Complementary and Often Defined By The Tasks That Others Don’t (Want To) Do Well
There are many guides, tutorials, and courses that like to talk about tools and technologies, and yet very few that get real about organizational structures and the dynamics that cause these archetypes to emerge, peoples' desires to rebrand themselves as "Data Engineers," "ML Engineers," or "(Data) Software Engineers" instead.
There was an interesting survey of ~600 "Data Engineers" by DataKitchen from 2021 ago where they had some very relatable, yet eye-opening findings:
- 97% report experiencing burnout in their day-to-day jobs.
- 70% say they are likely to leave their current company for another data engineering job in the next 12 months.
- 79% have considered leaving the industry entirely.
- 91% report frequently receiving requests for analytics with unrealistic or unreasonable expectations.
- 87% say they are blamed when things go wrong.
- 69% say their company’s data governance policies make their day-to-day job more difficult.
- 89% report frequent disruptions to work-life balance due to unplanned work.
(source: https://datakitchen.io/white_papers/2021-data-engineering-su...)