Why Data Engineers Are Ignored (And How to Fix It)

0:01 hello hey everyone how's everybody doing today good okay now so great all right well thank you everyone for being here I'm so excited you guys taking the time being here for me and for shakar um first of all i' like to introduce myself so my name is she the X in my name pronounced like s so I'm the founder of the data engineer things community and running

0:32 this community is actually my side hustle because I have this passion for creating resources and place where dat Engineers connect and learn from each other and um of course I need money to feed my family so I also work as a data engineer at Netflix and right next to me is Shar would you like to introduce yourself to everybody yeah thank you and I also appreciate you all being here and I just wanted to say it's a it's a real

0:58 honor for me to do it with you I mean Shenron and I have been sort of working together for the past one year and I absolutely adore everything that you do um in our first conversation shinran when we were talking about the data engineering things and I asked her uh why do you do this she said I wanted to have this resource that I never had when I started my career and that was for me one of my most relatable things I've

1:23 been in data for uh about 20 years now uh I started as a DBA and grown up the runs and I've been in Daya and Leadership but I never had that sort of community that you've built which is amazing so thank you and um in my last role I was director of data engineering at meta I was leading the data engineering function at meta or Facebook in London growing it from four data Engineers to more than 200 and I've been

1:51 the director of data engineering for all of Facebook Instagram and Messengers trust and safety problems um I left Facebook about a year ago to become an independent advisor and today I just work with uh companies helping them with a data Journey working with data startups and I'm also working with a few individuals on their career growth and their career Journey yeah yeah thank you for the intro Che so before we kick off this discussion um I want to start it

2:21 with the story it was several years ago I was working as a state engineer at a company and we were hiring for new data engineer and this person would really like he accepted the offer and right before he started we decided to grab coffee to connect and he asked me what is this most frust frustrating thing you had with your team or your company and was thinking all right you already

2:52 accept your offer so I'm going to be transparent with you and I told him that my most frustrating thing with the team is that as dat Engineers I feel invisible I really don't know how many people at the company actually know that this engineering team exists and when I told him and I thought okay this is not be going to be great but I want to be honest with him and he told me actually isn't isn't it like that everywhere so

3:23 this is actually one I'm saying that shakar and I want to talk about because this is one of the most common challenges that did engineering teams face this days it's not it's feeling invisible and not getting a seat at a table so shakar why do you think that's the case and what would be your suggestion for those teams who's facing this problem yeah um I think it's a great question and it's a question that came up a lot in my career all of my

3:53 data engineering teams and my data teams throughout my career always had that uh challenge it's not new and it's not unque by any means um I think that there is something that data people need to understand about the Dynamics with other teams that sometimes I think they other didn't spend time thinking about or they struggle to accept it but the reality we operate in is this right if you want to build a house you only need Builders

4:22 right you need people who can lay bricks one on top of the other you need electricians you need plumbers like these are the people who build you don't need anything else and the reason I say that is because sometimes there are other people who are involved like Architects interior designers there are people who think about the the the flow in the house the energies and all of that and they feel like you know they don't get their fair

4:48 share but the reality is you don't need these people to build a house yeah it may not be an amazing house it may not be beautiful it may not be super functional but there will be a house and in our reality I think that uh software Engineers are like the builders right you can build any software without data almost any software right you can build without data metrics help you make it better makes metrics can help you do

5:13 experimentation they can help you understand if you have product Market fit they can help you understand if you're heading in the right direction identify issues like data is super critical but it's it's not essential for building the software right um and that's something we have to realize because um if I when when we moved to our new house I I started working with an interior designer and she helped us think about the house and it's been

5:39 amazing experience like she was thinking about lines and she was thinking about colors and how different things in the house connect together and that was a very different way of thinking that I'd never seen before and I'm never going to you know move into another house again without getting the the help and support of interior designer but I had to see it once first to understand the value to understand oh okay so that's what you do

6:04 and that's why I need to work with you um and I think that data people need to understand that once people see the value once they show the value and they show why is it that you need to work with data Engineers why why is it that important that you have good data foundations that you have metrics I think the rest will come naturally and I've been in teams when when I joined people didn't even know that team

6:29 existed and and you know after less than a year people are asking hey you know we want to give you more headcount because we need that team to grow it's becoming a bottleneck right so that's the difference between these two states is people have seen the value they now know why why you exist and why is it important and they want to work with you and unfortunately it's on us sometimes to show that value and to to help them

6:56 understand yeah that is absolutely true and um to me part of whether you can get a seat at a table as a DAT engineer really have to do with whether you can make that connection between your technical work and also the business impact to be honest I feel so lucky in a way that I had a manager who very intentionally Tau me that when I got my first job as a DA engineer here's this

7:27 three thing that my first manager Tau me um when I first get started um as a engineer which was my first full-time job after graduation and she always invit me to the meetings that she have with different business partners at that point um the data engineering team I was part of was supporting marketing and membership and she always take me to those meetings that she have as business

7:57 partner and at first first honestly I didn't quite understand why I had to be there I just need to know okay where I need to move the data to I thought that was a most important thing at the time but from those meetings I get to learn so much about the business problems we were trying to solve at the time and I also made those personal connection with those Partners we have there you know there were data scientists there were um

8:24 business partner who were making decision about how we should be advertising for our products and how we want to distribute in different social medias I learned so much from those meetings and also I made those connections that was really helpful for me later in the career and another thing that she made me do at that point was giving presentations to everybody um I was literally giving presentations like every week it was sometimes giving presentation about my work to my

8:53 teammates and sometime was giving presentations to the data scientists and or to the business partners at first I don't really know why am I spending so much time creating slides I could be writing more data pipelines but later on realized that was when I was making that connection between my technical work and the business so that they understand what it took to enable those business initiative that they were trying to achieve um my joke was at that point

9:23 that my most fluent language was PowerPoints was now P um but any was a joke but also so that it was such an intentional thing that she made me do and in realize it was such a valuable skill that you need to build as an engineer if you want to make that connection between your work and business impact and one last thing I just remember when I first joined the company and the team she gave me this

9:47 list of 20 people that I need to have one onone with and I was an introvert I was so intimidated because I was like those are strangers I don't know them and I had to go grab coffee with some I already go to work meetings with some why am I having oneone with some but then realize that those are the important small talk you want to have with people outside the work meeting because sometimes during those talks

10:15 people seem to be more vulnerable they want to tell you about what's the challenge they have with their data so that you really get to know their pain point and also you get those personal connections that it just make it easier to work with some later to summarize those are the three things that I learned about um making connection between your work and business impact one is that you want to make your work visible to other people

10:42 for example business partners or your other data Partners through showing off your work what you have done how that enable those business and also at the same time go to those business meetings understand what's the thing that you want to enable them to be able to do through you know making data available and third thing is you know never um never ignore the values of having oneon-one with the people you work with yeah and I love it that she

11:11 made you present uh things to people I think that's one of the most effective ways to actually tell people what you do and I think it's not just about presenting it's the way you talk about your work makes a big difference and I don't know why um but a lot of data practitioners especially data Engineers the way they talk about the work is is is is not right like it's there's something wrong about it and I'll give

11:34 you an example like um I I'll actually tell you a story um a couple of years ago um we me and my wife wanted to build a cabinet for my daughter in a room because she was three years old at the time and she had a lot of stuff in the room and she didn't have a lot of space to play and I we bought this big Ikea cabinet that I thought okay I'm going to build it in two hours and spend two days

11:56 building it um and as I was building it I was reflecting on an interview I just I had just had a couple of days before with with a data engineer and I asked him okay tell me about your work tell me what you do and they said I use Python I use spark I use kalala which is the equivalent of you asking me what did you do this weekend and I'll tell you I used Hammer I used Nails I used a screwdriver

12:20 which is true I did I built a cabinet but that's that's not the most important thing like that's not even interesting and that doesn't tell the story but for some reason and that's the way a lot of people a lot of data Engineers talk about the work they describe the tools that um they use I don't know why but that's what they do and then some some people some data Engineers the more advanced ones that I've met they tell

12:43 you I build a pipeline I build the dashboard which is the equivalent of you asking me Shaha what did you do this weekend and I tell you I built a cabinet which is which is true that's a better description of what had happened that weekend but the best data practition as worked with and not just data but any any sort of professionals have worked with when you ask them what you do they say oh so we have a problem with a

13:07 marketing funnel we're acquiring the wrong customers and we're basically paying too much money on customers that their lifetime value is too low so I'm building this uh pipeline to basically bring all the data in one place and I'm going to build a funnel that we can slice and dice by different dimensions and I'm I'm basically going to be able to identify the points in the funnel where we are spending money acquiring the wrong customers so that we can

13:30 optimize our marketing funnel that's basically telling what is the problem that you're solving right uh a lot of when you talk to product managers that's how they talk about the problem they explain okay we have these situation this is why we do this software engineers in a lot of cases they also talk like that not always but in many cases they explain okay this is the problem I have this is what I'm trying to solve so I think if you get into the

13:53 habit of and of of of talking about your working the right way and talking about it in the right order I think it will be more interesting and inspiring to people and they would be able to connect with that because if you just say I'm building a pipeline people frankly just don't understand why is that important yeah that's definitely true that also reminds me um I did this resume critique session at one point with um a recruiter I work with and

14:22 that's actually also true when you're creating your resume create your narrative is that you not only want to talk about how big is this data pipeline how much data you're moving but also mentioning the biggest impact you're making with the technical work you're doing there too 100% And I wanted to ask you shinran actually because a lot of people ask me the question which company should I go to as a data engineer should I go to big tech companies should I go

14:48 to startups should I go to smaller companies should I be a consultant what what do you think about that yeah that's a really good question and to be honest I had to make those decisions at several different time in my career as well and to be honest like making those uh making those decision making those decisions are actually really important because those companies are indeed different um I did have experience working at like kind of middle siiz company tech company

15:16 and also startup my experience as startup which is actually true in most startup or smaller company that if you get hired as a data engineer you are not just a data engineer you have to work four or five different heads you're data engineer you also have to work on different different uh data infrastructure you probably also data analyst you Pro probably also um data scientists as well so and you're going to be alone yeah and also you have to

15:47 basically build that breast of knowledge and um to be honest um it's also it depends on what you want to do as well like for example when you work at a smaller company where you get to exercise and learn so many different skills in a way it's helpful for you to really see what is kind of the Upstream of data what does it takes to build a reliable or scalable data infrastructure also at the same time when you wear the

16:16 hat as a data analyst or data scientist you kind of build those embasy for your data consumers as well you'll start realize hey this data set that was building it didn't work and why and that was also very important experience to have whereas at a bigger company your role as a engineer are typically a lot more in depth and focused because at a bigger company they have a lot more people working in data you're going to

16:43 be part of data engineer team and you're going to work with maybe a data platform team data infrastructure team a dedicated data scientist team so as part of that you get to be a lot more focused as data engineer and really build those skills in depth and to me I don't think there I mean I don't think there's a absolutely right answer for everybody because there's some pros and cons and different grows opportunities for those type of grows and what's the most

17:13 important thing here is to be intentional when you choose your next career move because they don't offer the same girls opportunity for you like for example if you are a junior dat engineer my recommendation for you is to really find a team that you can be surrounded by people who are more experienced for you so you can get those mentorship and really learn those best practice whereas if you join a small company and a lot of

17:43 times you might be the first dat engineer there and you really don't get as much mentorship and another thing is you might be managed by somebody who's who was never a DAT engineer so that was be one example like where you might be in a a situation that you might want to go to bigger company where there will be more senior people who can you know who you can learn from you know also another scenario would be that say if you want

18:09 to transition into from a IC individual contributor to a leadership role and when you are evaluating all this opportunity in front of you you want to think about will this Ro provide me the opportunity to become a leader for example do I get to Mentor other people do I get to lead in projects or this team going to grow fast enough that I could be transitioning into a manager so those are also the factors that not all

18:36 the teams will be able to provide you those growth opportunities yeah I love it and I'll I'll add to that two more quick things one is there's some skills that are always important but in bigger companies they become even more important like communication skills influence um you know the ability to work with different people collaboration conflict resolution things like that that maybe still important but less in in in much smaller companies so think about that and choose a role that plays

19:04 to your strength and the other thing I will say is that there is one big advantage to to bigger company and that's the the scope that you can have and the impact you can have and just to give one example I remember an interview I've listened to with Jeff Bezos in 2021 where he said that Amazon basically cut down on um 7% of the plastic that they use in the non-recyclable plastic that they use in their packaging and that was

19:34 the equivalent of 100,000 metric tons a year um so big companies do have the advantage of scale and then even though you might be working on something that is quite narrow the impact you can have on the world is is massive right and if that's something that's appealing to you and that's something that you look for and I'm not some people are being cynical about that but you can really have big positive impact on the world

19:57 just because you work on large scale problems yeah talking about like um making career transition or a career choice I'm curious one question that people usually have that should I stay as I see or should I become manager what's your thought on that it's a good question I get that I get that a lot and um sorry I get that a lot and I think the

20:28 first thing I'll say is play to your strengths right think about what are the things that you like doing what are the things that you don't like doing if you really like just sitting down and coding for six hours not talking to anyone then you probably want to be an icy right or or you know techly um but but if you like coaching mentoring people if you like you know presenting things if you like solving conflict and you like

20:55 organizational Dynamics and you genuinely enjoy that then management is is a good option for you um I will also say that there are some things that are part of the um I mean everybody starts as an IC right so the question is do you want to transition to a manager role or not and I will say there are few things that are actually part of the part of the role of every manager that you need to ask

21:18 yourself are you comfortable with that and would you be be willing to do that so for example Performance Management right genuinely think about that situation I've had people who were interested in management Transitions and when I was talking to them and I said would you be willing to manage performance of of a low performance and potentially even let them go if they're not able to perform at the level required they were like no I'm absolutely not not able to do that

21:44 um hiring people is is can be difficult for some people I mean I know people who genuinely don't like the feeling of interviewing somebody and having to make this decision about their career and you know to be able to say yes or no and someone's career they they just don't like it so these are things that are inherently part of the role of every manager and you just need to ask yourself would I be enjoying doing these

22:07 things would I be able to do these things yeah totally and um and also another saying that you don't have to be a manager to learn leadership skill as well because when you are senior I seeo there's so many opportunities that you could also pick up leadership skill for example how do you influence people on your team to work towards the same goal how do you influence your partners to get buying so they understand that this

22:36 is a project that will be really good for the long term even though it doesn't enable any new business initiative and those are the scills that you could also build as part um being IC as well without actually managing a team 100% And and I'll say that when you work for big tech companies right it's very very common to work with people who are at senior IC level who's new level is the equivalent of a director senior director

23:02 even VP I've worked with people like that they are some of the best leaders I've worked with and except for the fact that they didn't have a team they didn't have an organization they were absolutely leading they were taking some of the most difficult challenges and problems that nobody else could solve and they would be the ones leading it so I know we only have five minutes left I wanted to ask you shinran what do you think makes a great data engineer

23:28 um it's a great question and to be honest I had very different answers kind of throughout my career why um when I was a junior engineer honestly I I thought being a good data engineer means that to say yes to everything that comes my way and get things done as fast as possible and at that point I saw that was what a good engineer looks like but now I think about what it means to be a

23:58 greatest engineer is actually those who are good at saying no to those requests that come into your way and to be honest what's the hardest thing about saying no it's to build that trust with the people you're working with so that even if you're saying no they know that you are doing it for the good for the company for the bigger team even though you're not saying yes to things and there's different aspect to it for example you want your partners to

24:28 come Sol with you before they say yes to the things that come to their way that means that you want to get involved in the process as much as early as possible so that they get your opinions and input before they commit to anything because some of the worst thing you can run into that your partner come to you like hey I need to get this dashboard on Matrix executive this week can you get it down

24:52 for me and you learn that this is not possible because we don't even have the data available and that creates friction however when you realize it it's because that you know they did not involve you early in the project to get your pennings on the data availability and the B uh your bandas priority and also um the level efforts on those type of things and also another thing is saying no also means push back a lot of times

25:19 people might come to you with request like hey can you build the data engineering system for me to solve this problem but a lot of times you'll realize that after you get to know that know the problem you're trying to solve you realize the initial proposal was actually not the best thing to do so sometime you had to push back on that to saying that hey maybe that's not the best thing to do can we solve it

25:41 upstream or Solve IT Downstream and being able to push back on those requests it's also a really important skill to have I love it I'll just add to that that that will be the last story before we leave but um just one story from the best data engineer I've worked with in my entire career the person I consider to be the best data engineer um when uh when we worked at meta there was there's a piece of infrastructure that's

26:06 called Product catalog we were working on ads and it's a piece of infrastructure but that basically enables you to upload an entire catalog of products to meta so that you can create uh ads based on these products and if you think of big advertisers like Amazon for example or Shopify they have billions of of products right they don't want to have to create ad for each and every specific product they just want to upload a catalog and say these are all

26:33 of my products now here A bunch of signals and then we want you to show this ad to this person and dynamically create them so this data engineer started working on on on data catalog and a few weeks after he started working with them he came back to me in one of our onet to ones and he said jar I'm really concerned about that team and I said oh why why are you concerned about that team tell me more and he said

26:56 because they are prioritizing the wrong things they said the wrong goals and I said what do you mean they said the wrong goals they're building catalog and he said yeah yeah but I I'll tell you why I I'm saying that I I joined the team I started trying to create a catalog in the UI I tried to upload a bunch of of csvs and all of that to to to create the catalog I get error messages it's not working it's failing

27:17 it's slow so I created a lot of bugs and the team is just not prioritizing that so I asked the software Engineers why are you not solving the bugs why not fixing it it's production bugs and they said oh the UI is not a problem and how the UI cannot be a problem there are two ways to create the catalog UI and API how can the UI not be like how how can that not be a priority of course

27:37 it's a priority that's how advertisers upload their cataloges right and he said I didn't understand that until I looked at the the team's Topline dashboard and I saw that the the the key metric the one that they're going on is the number of items in catalog and if that's your key metric that's what you're trying to do then you don't care about small uh uh businesses who use the the UI to upload you only care about the big ones you use

28:01 the API and he said I said okay that's that's I I agree with your assessment what are you going to do about it and he said oh that's fine I've already spoken to the PM and the engineering manager and data scientist and I told them guys you're you're completely wrong and I've proposed a different metric and um and and this is what they're going to use for their going for next half and it's a data engineer who did that so the

28:24 ability to step back to look at the big picture and and to think think broadly like your job is not just to build the pipeline your job is to make the business better if you can do that then you're an amazing data engineer yeah and I think we need to wrap here uh it's about time but thank you everyone so much for being with us today thank you shinran for coming all the way here and for uh doing this and please let's let's

28:47 connect on LinkedIn and let's continue the conversation if you're not part of data engineering things yeah thank you Shar thank you everyone for being here and just a quick shout out we have a meet up at 600 pm. today um it's for the data engineer science community so if you have time please feel free to join us thank you