[00:01:03] Nima Gardideh: Welcome to The Hypergrowth Experience. This is your host Nima Gardideh. Today we're gonna have Alexey Komissarouk on the pod. He's the ex-Head of Growth Engineering at Masterclass and he did similar work at Opendoor for quite a few years. , I'm quite grateful for him for waking up super early, in Japan. He's, recently had, a move there and has been living there for the past year or so, with his partner and kid.

[00:01:28] Nima Gardideh: I've actually had a pretty long history with him, and we've really been talking about. Growth engineering work, paid growth, and the connection between engineering and these channels for the past few years. And he's done a very good job at these organizations that he's been at. And I would consider them sort of the state of the art when it comes to the process that they implement and the way they run their teams on how to use engineering as a massive leverage point for.

[00:02:01] Helping marketers, deploy their budgets better and think about the product in the top of the funnel all the way to, to retention in a systematic way and try to sort of run, run experiments. and if you're a technical founder or recently stopped being a technical founder, his path of, Going from the startup world into starting these sort of growth engineering teams at companies.

[00:02:28] Is quite attractive. He actually is, really a big fan of hiring, founders that have recently either, sort of shut down their companies or, or moved on from their companies to be in these roles. technical founders that have a very strong sort of business mindset. writing a book on this matter.

[00:02:44] In general, growth engineering it's a topic that is not really spoken about enough, in my opinion. There's not enough content out there on it, so I'm very happy that he's doing this. He's also one of the experts in the field, so I think it's going to be very helpful. , I tried to get into it with him and, and got in there, in the different aspects of what it's like to create a team like that in an organization.

[00:03:05] What should be in charge of, and how, the organization should be built out over time. So if you're thinking about building one of these themes or thinking about how engineering can, feed into your marketing organization. , this should be a really good, an interesting conversation for you. We start with him sharing his path from being a startup founder to leading these growth engineering teams at Open Door and Masterclass and what really attracted him to these fast scale, fast growing companies.

[00:03:34] , so I hope you enjoy the episode. Here is Alexey. 

[MUSIC FADES OUT]

[00:03:40] Alexey Komissarouk: so for me, growth engineering is sort of a retirement path for, for technical co-founders that have burnt out. , I did a lot of the traditional startup stuff. Started a venture back company, raised a bunch of money, had a big co-founder breakup. I did a bootstrap company, which did moderately well.

[00:03:58] , and then at some point I was just like, wow, this is really stressful. What's it like just having a job? All, all my college friends went to FANG and they seem pretty happy. I don't know that I want to go to quite a company that big, but like, what, what if I just had a job? , and that was about 2016 and ended up joining Open Door for four years in the masterclass for three and fell in love with growth engineering because it's sort of a, it's a lot of the early founder stuff of like, you have to go solve the business problem.

[00:04:27] You have to go figure out to make, how to make things work. , but also ultimately it's the form of engineering. And being good at engineering makes you significantly better at growth. And so for the last three years I was in charge of the growth engineering department at, masterclass until very recently.

[00:04:43] Nima Gardideh: I like that positioning in general. I think there's a lot of paths, founders of early stage companies can take. , this is probably one of my favorite ones, especially for the technical, founders. A lot of folks end up in products because they just want to make some decisions. But, I'm curious how you think about that, but before we get into the role itself or the sort of craft itself really, 

[00:05:06] could you tell us just, to set context, like what was the stage open door was at when you joined, what were the challenges there?

[00:05:14] And then similarly we'll get to masterclass after that, but, I wanna maybe pause on open store for a little bit. , give us sort of like a picture of when did you join, why did you join that company to begin with, and what were the challenges?

[00:05:29] Alexey Komissarouk: I joined Open Door in 2016. , it was maybe a half billion dollar company. I think it was engineer nber 15 or so. , I mean, so o open door, for folks that don't know is a company that, will, like the services will buy your house. So you don't have to deal with the pain of having to sell your house.

[00:05:48] And so you can have it for people, especially who are selling their house because they're moving, having the convenience of, the price, you get the offer right away, exactly what's gonna move. It's not gonna fall through, is huge. , on the flip side, as, as a business that's a terrifyingly thin margin business.

[00:06:03] , and so it's very important to acquire the customer as effectively and efficiently as possible. , so when I joined, we were in the process of launching the third market opener has to go launch on a city by city basis because the process of purchasing homes is very different on a state by state and city, by city basis.

[00:06:22] , we were launching Las Vegas and I mean the problem that I spent the vast majority of my time working on was, hey, how do we more efficiently get people from where they've heard of us to where to contract with us? And that's everywhere from. Performance marketing to the landing page, to the conversion funnel, to telling us about your home to, hopping on a call with you to making sure that you're getting all the touches you need from the sales and support people.

[00:06:49] , just optimizing every possible step in that entire funnel, to ultimately optimize A, how many people sell their homes to us, and B, what our margin is gonna be like on those homes and how the cost of acquiring the customer, will impact that margin.

[00:07:04] Nima Gardideh: And so is that something like that the growth team was in charge of, was the product in caring? This, was it just a C-suite caring about these things? Like how were the teams structured? There was only 15 engineers at the time. 

[00:07:18] , were you on this project at first? Like, how, walk us through that part.

[00:07:22] Alexey Komissarouk: absolutely. So, open Door was, and I think largely still is, four business units. It's, again, you can kind of think of it as a marketplace, because you have to acquire the home and then eventually you have to sell it. So the four business units are fundamentally, the ones interacting with the seller, the ones interacting with the buyer, the ones interacting in the middle that have to actually go and clean up the home and make it ready for inspection.

[00:07:48] , and then the data science or pricing team, which is responsible for figuring out, how much can we actually offer on this house and how are we gonna get all the financing to, to work for that. , and so seller buyer homes and pricing is roughly the four units. And then, effectively, I was on the seller team for most of the time there.

[00:08:11] And then the seller team itself acted effectively. , I mean, again, effective, effectively acted as a growth team. And the way to think about this is, open Door is one of these products where. The customer acquisition funnel is the product. , it's not like you buy a home for us and then we need you to use it every month.

[00:08:31] Right. Like, like if you've sold us your home, we're done. Thank you very much. It, it's been a pleasure. So it's, the whole product is, is fundamentally like a growth, like, a funnel. And at the end of the funnel, at the end of the checkout flow, you're done. We have your home, you, you have a bunch of money in escrow.

[00:08:47] Nima Gardideh: interesting. That business whole, that's like the whole business is the funnel. If the funnel works, like the business is working. So, that makes a lot of, that. The service in it in itself is kind of like an acquisition, service. ,

[00:08:59] Alexey Komissarouk: so due to your other question, like who cared about it? I mean, yeah. The C-Suite cared about it. At the end of the day, our reporting was how many homes did we buy this month and, and at what margin? , and the kinds of projects that I did, so the, when I first joined the person running the business unit was jd, who was one of the co-founders.

[00:09:19] He was effectively acting as the pm but he was, I mean, doing a lot of things. , and so one of the first projects that I ended up working on was working with the seller salespeople. And the, at the time, the interface that they had was really painful. , it was just something su like, like in any startup, somebody built something super early on.

[00:09:39] It doesn't quite work, but it's sort of, it's what we have and let's keep going. , and there would be things like, Hey, this person hasn't signed their contract in four days. We should probably follow up with them. , but there was no easy way to see that. , and I think at the time one of the best sales people had organized a like spreadsheet where they were manually like duplicating the tracking of every single one of the, homes assigned to them to make sure they knew when to follow up, which effectively means your salesperson spending a lot of time, on bureaucracy as opposed to just, showing up to work and being like, all right, who are the 10 people I need to follow up with today?

[00:10:15] , And so, one thing I would expect many people to say at this point is like, oh yeah, so you just integrated Salesforce. , no, Salesforce was, was, would actually be a terrible idea. Salesforce really doesn't work well for a business like Open Door, and I know that because we've spent years, seeing, seeing if we can get it to integrate well and ultimately like, no.

[00:10:36] , and the answer there was to do something a bit more custom, which a lot of the same principles, but fundamentally gives somebody the dashboard that says, here's what's important to you today that you can, follow up on. And so I built that and it was a pretty significant difference for, the efficiency of our salespeople.

[00:10:57] I think one fun story from there actually is before I built it, I became, one of our, seller cx or salespeople, I think, for a couple of weeks. , 5% of the new homes that were, coming into us in Phoenix, I was a salesperson for, so I had to call them. I have to go chase them, like. It was great.

[00:11:16] I bought this one house from this, from this very friendly couple. , it was fun.

[00:11:21] Nima Gardideh: And so you did that in order to learn the process and understand it very well.

[00:11:26] Alexey Komissarouk: Yeah, exactly, because, so I was partnering with the head of that, sales team. , but at the end of the day, like, it's sort of still secondhand knowledge. I think I still got a lot more understanding by effectively being, just, just doing the job

[00:11:43] just to gain a better intuition.

[00:11:45] Nima Gardideh: so let's zoom out a little bit just to touch on, masterclass.

[00:11:50] So you were at Open Door for four years. I asse that was in their sort of like hyper-growth stage. Did they go public before you left or

[00:12:00] were you there right after? , so what was the state in Masterclass was in, because I think they were around already for quite a while when you joined them. Right.

[00:12:11] Alexey Komissarouk: Yeah, I think, they were in a similar state valuation wise, it was about a half billion dollar company. , and they were at maybe 30 or 40 engineers, so a little bit larger. , but similarly entering this hypergrowth stage of, especially this was at the start of Covid. And so as you can imagine, COVID, especially during the early days, people were stuck at home and you can only watch so much Netflix until you feel bad about yourself.

[00:12:35] , and then ultimately you wanna say, cool, I wanna make sure I'm doing something productive with my time at home. And then masterclass was such a strong answer to that.

[00:12:43] Nima Gardideh: Yeah, that was, I remember we were talking when you already had open door. , and then the masterclass change was, was pretty clear to me, but just, maybe like zoom out on the career. And what is your, what was your logic in joining these companies? That the, it sounds like the exact same sort of like moment.

[00:13:07] What, why, why, why, do you, do you just like hypergrowth? What's going on? Do you

[00:13:11] Alexey Komissarouk: I mean, I do like hypergrowth, masterclass is a particularly interesting thing because when I met the VPN engine, they started talking about the problems they had. , incidentally, they ended up in the exact same tech stack. , 

[00:13:25] and, and it was like, well, how do we solve this and this and this?

[00:13:28] And I and I basically had exactly the answers. Cause I had had exactly those problems with the growth info tech stack at, at open Door. So it was. It was just magical opportunity. Cause , after you've worked somewhere for a while, you've accrued a bunch of tech that you, that you understand you will never be able to fix, right?

[00:13:43] You're like, yeah, no, we should have done done it this other way, but there's no way we're gonna do a six month project and fix this up. Like the value's not there. Like, I understand what the el elegant solution is now, but it's just not worth it. , versus you come to a company that's at the same stage and you're like, oh, you guys haven't solved that yet.

[00:13:58] Okay, well here's the four pitfalls that we're gonna avoid and here's the thing that you really have to do that's gonna make it significantly less painful for your lifecycle people to iterate or something like that, or make it significantly less painful to, get your experiment analysis rate. , and so I think joining masterclass was, for me, the difference between being, and ic him and an EM on that team versus running the engineering organization, was just an opportunity to do things right and, to see what it's like to, to do it from scratch.

[00:14:26] Nima Gardideh: Yeah. And, and I think that's like a worthy cause for sure, cuz it, correct me if I'm wrong, but it's quite painful to go through this stage with these companies where you're sort of like scaling up and , going through these, these sort of mass hiring and then dealing with all sorts of problems both on the people side as well as just maintaining how do you like, manage all this growth and

[00:14:49] Alexey Komissarouk: I love it, man. , yeah, I'm sure there's painful parts to it, but, but I love it. I think it's so much fun. I think it's, it, it, it lets me get my like, sort of co-founder, like interesting problem solving head on. But at the end of the day, like, it's not my company, right? So as much as I would like for it to succeed, and as much as I'm like stock incentivized for it to succeed, like, I can go ho if I'm not on call, I'm turning my phone off, right?

[00:15:13] It's, like I get to live closer to a normal life.

[00:15:17] Nima Gardideh: Yeah. Yeah, I can, I can feel that. Do you feel like there was just less an anxiety attached to it as well, because you're not, on the hook for it?

[00:15:24] Alexey Komissarouk: I think that that was a big part of my motivation. And then also, like, if it's your company and , you argue with your co-founders and you lose the argent, you're like, ah, like, ah, we're gonna fucking lose. We're gonna, this whole company's gonna die cause we made this stupid decision. And, and you're so like, it's your baby.

[00:15:38] , versus if you work them, like, you, you'd like to do the right thing, but if you get overruled, like, hey, like that's cool. Like I can look at that.

[00:15:48] Nima Gardideh: Yeah, that's an interesting one. I want to like, maybe talk about that for, to some extent of what does it feel like to be over Overruled. 

[00:15:57] Like, do you think that these companies had similar cultures when it, when it came to that, was there sort of like a consensus culture? Was there sort of like super down, top down 

[00:16:06] culture? Like how do they work?

[00:16:08] Alexey Komissarouk: They were actually super different. , open door. Open door feels very east, Coasty. Open door is very like, disagree and commit and let's go. Right? Like, very comfortable with disagreement, and then sort of moving forward and, and executing. , like open door will, will tolerate some amount of dissent.

[00:16:27] And then at some point, hey, all right, let's all, let's all just keep going. , versus masterclass is much more consensus building. , I think much more of like an la like San Francisco company of like, Hey, like, but have you checked in with this like ninth person who's like nominally related to this project that we should also check with?

[00:16:45] , which I mean, I appreciated both. It was nice to, to sort of experience the difference of being a manager in both. But, yeah, very different cultures.

[00:16:56] Nima Gardideh: Do you feel like there's one that's inherently better than the other?

[00:17:00] Alexey Komissarouk: no, I think you just have to pick your, , what's the word? Yeah. , you just have to pick what you're gonna be unhappy about, right? Like, no, no. Culture is perfect. I think it's, pros and cons in, in each case

[00:17:15] Nima Gardideh: Yeah. It's interesting cuz there's like probably literature on both of these and I have friends that run both types of companies and I'm investors in companies that are. Basically on, maybe both sides of these. And I would say there's probably like a third one, that's a little bit more like a cratic trying to be basically not managing at all.

[00:17:37] But, I don't love that one as much as I used to. I I when I was younger, I 

[00:17:42] Alexey Komissarouk: it, it's like communes. It sounds like a great idea.

[00:17:44] Nima Gardideh: Yeah. , it just, they don't get anything done is the main problem. But the inherent thing you mentioned seems like pretty true to me, which is just pick it and then stick to it. , and then you'll self-select the people that like that version, of management and culture, and you'll end up be able to skill up a business that way. Let's go to sort of, how you entered Masterclass. So you saw an opportunity to sort of build things, in a way where all the mistakes you had learned from at Opendoor would not be. Repeat it. 

[00:18:25] So what do you mind, explaining maybe in a little bit more detail what those mistakes were in your opinion and, not mistakes or just like things you learned, I guess, through the process, and what you set up hopefully better at Masterclass as, as you did at, unless you think you, you, you found even more things

[00:18:46] Alexey Komissarouk: No, I definitely found even more mistakes. Yeah. If you're not making mistakes, you're just not being introspective enough. , I think, I don't know if this is the most interesting thing cuz it does get a little bit in the technical weeds, but, , mask or open door got into this place where we spent so much time maintaining and cleaning up the, lifecycle integration, right?

[00:19:07] Where the means by which we got data into our like, marketing architect marketing. , automation platform, which at the time was Iterable was, just became kind of like the vast majority of the background jobs running in production. , and at the time I was so adamant that like, listen, like you need to have realtime data and interval so that any email can be realtime, and in practice, like maintaining the sort of graph dependency between anything that could change in production and getting it into interval right away, and, oh, well, but now we rely on a new table that isn't getting updated, or, oh, this got migrated out, , ended up being, becoming this huge project.

[00:19:46] , versus, the more reasonable way to do that, for the vast majority of use cases is just to have a nightly sync and not to try to maintain this like, realtime data export, because it just doesn't, it doesn't solve a real problem, where ultimately the lifecycle team realistically for the vast majority of their email is nightly as good enough.

[00:20:09] Nima Gardideh: Do you recommend some, were you, I guess these tools started bubbling up in your masterclass journey. What were reverse ETL tools a thing when you were, working at Open Door or Masterclass? Were you building that type of stuff yourself?

[00:20:26] Alexey Komissarouk: Yeah, we built it ourselves. , well, we had five Tran at least. So five Tran, was really nice for, and again, for, for folks, for whom this is relevant. , the problem that reverse et TL tools solve is, you have a bunch of vendors and you need to get your data out to them. And the most naive thing you'll, you can do is you can write the code for that yourself.

[00:20:47] , you can have the engineers write that yourself. You have to understand exactly what the Facebook or Google API is and put that all together. And then you have to maintain it as it changes. , and base, it's not terrible, but basically now you have some data engineer, some full stack engineer, desperately trying to learn data engineering, like full-time or at least part-time.

[00:21:07] Maintaining that, versus a reverse ETL will let you do it with just sql, right? We'll just say, Hey, I sit on top of your data warehouse, tell me the data that's interesting. And then I know how to take the sequel query with the right colns into the relevant, like places for Facebook and Google, to know so that they understand which customers are converting or so I can send it to your, marketing automation platform so they can learn all the right things about your customers, et cetera.

[00:21:30] , so we did 

[00:21:31] Nima Gardideh: Yeah. And just to 

[00:21:31] like go one 

[00:21:32] level up 

[00:21:33] for folks, SQL is the language in which you will ask a database for information and it comes back, if you're like a non-technical marketer, cuz we have some

[00:21:42] Alexey Komissarouk: That, that, that's really one level up my bad. This thing that you had to do with code, can now be done by a data analyst or a technical pm , or still an engineer, but that engineer is now, spending a lot less time on it because the integration maintenance is, being handled for them.

[00:21:59] And we did end up bringing, census into masterclass and I was quite happy with it.

[00:22:04] Nima Gardideh: Shout out to census. They're friends of the pot. , yeah. What, what, into masterclass. So you, you did decide not to build that again, which is great. , and so are you still, even with these reversal TL two s, do you think, you should be going for like a nightly sync for these things? And, and just to like address, when you mentioned vendors, I asse you're talking about things like Iterable, Facebook ads, Google Ads, basically any marketing tool in, in your stack or product analytics tool, or C R M and so on and so forth, right?

[00:22:39] Alexey Komissarouk: Exactly. Yeah. I think nightly is about right. You could do more frequent, I mean, I think the important thing is just not real time. Like you should not attempt to, keep your lifecycle, your, your marketing automation platform tool, real time. , because again, you, you're just gonna spend a bunch of time chasing it, chasing, that dream.

[00:22:59] And it's actually. Not that valuable except for a small nber of key emails that you wanna send. And for those, just make sure that the data you have for those emails is available in real time.

[00:23:08] Nima Gardideh: All right, so unless it's transactional, you should not care about real time data being available.

[00:23:15] Alexey Komissarouk: Yeah, we're getting really into the fine grains here. , you could think of, like transactional marketing emails are this weird spectrum. So for example, the welcome email, right? So somebody buys the product, you send 'em their receipt and then you say, Hey, welcome to masterclass. Here's five cool things to check out.

[00:23:33] Right? You could argue that that's a transactional email because, well, they purchased and like, this is now what the product is. You could argue it's a marketing email. Cause we're actually like trying to tell you more about the product and explore other features. , and so if it, if it's kind of marketing, then the marketing team wants to be able to iterate on it.

[00:23:49] So then you put it in, interval and then. Now, now you're in this world, but that, that email needs to go out right away when somebody's bought. Right? So now you can't, but you wanna say, Hey Nema, you don't wanna say, Hey, first name, or, Hey, new customer. , but NEMA is the first name on the user profile, which we just said should be updated, nightly, right?

[00:24:12] , and you, you don't want to be one of those people sending a hey first name, email. , and so now you need to say, okay, well, okay, this is the one exception where this email does need to, does need to know what your name is, even though maybe that isn't updated yet. And in cases like that, my recommendation, and again, we're getting super in the leads here, but, in that case is my recommendation is just to stick, to stick any of that extra context into the, like user has just signed up.

[00:24:38] Here is a welcome email event, that you sent to, rather than trying to update it properly through the user profile.

[00:24:43] Nima Gardideh: Mm, gotcha. Yes. There are some like edge cases here, and you just have to figure out some creative solution out of, 

[00:24:52] Alexey Komissarouk: Yeah. 

[00:24:53] Nima Gardideh: approach.

[00:24:53] Alexey Komissarouk: And, and there's, right, and the challenge is like day one on your job, on the job, you've like, there's no way I would've understood that. , I think this, this just comes from years of having to maintain that system at Open Door and, and then at Masterclass and realizing, okay, like I tried really hard to support this realtime in variant and like the juice was not worth the weeds.

[00:25:14] , so there's all these heuristics, I mean this one's specifically around lifecycle, but there's a lot about like experimentation as well, that you don't realize are important until you are in a place that does it a certain way and, and to change the way it, it will do, it will be sufficiently painful.

[00:25:30] But you now realize that wave is not optimal.

[00:25:33] Nima Gardideh: Yeah, the juicy things you learn with experience. ,

[00:25:40] Alexey Komissarouk: It's a

[00:25:40] Nima Gardideh: okay. Let's, let's zoom out a little 

[00:25:41] bit. I wanted, I wanna talk to you about growth engineering as a craft, and I think, first of all, you, you're doing some writing and I really appreciate you sharing this writing with me before it goes out about this, this specific type of role.

[00:25:57] , and I'm quite excited about it because I think it's seldom talked about and it's very unclear to people what that role actually means. , it, it feels like one of those roles that became popular. Vernacular, but no one understood what, what it actually meant. And so, I'm thinking like growth hacker or growth marketer and, , these types of names that the job description was unclear for.

[00:26:28] So let's start with what is the job description of a growth engineer? In your eyes,

[00:26:34] Alexey Komissarouk: Yeah. , I think a growth, a growth engineer is when, like a growth hacker learns about C I C D. , is is maybe one way to put that, 

[00:26:43] , 

[00:26:43] Nima Gardideh: I C D?

[00:26:45] Alexey Komissarouk: Continuous integration, continuous deployment, like, like a proper, 

[00:26:48] you 

[00:26:48] know, the proper engineering way of, of getting things done. I think, early on, a lot of companies, will hire, I mean, what used to be called a growth hacker, which is, we don't have time for a proper growth team.

[00:27:01] We're just gonna have one person and see what kind of crazy magic they can do. It's like, Hey, we got this 20 year old who just flamed out of yc. Let's, let, let, let's see what they can figure out. A, for us, is the typical stereotype. , and honestly, that works surprisingly well, like more often than you would expect.

[00:27:19] , but then what happens is the company grows and you can't just have a one man marching band, doing growth anymore because you have stakeholders and you have priorities. And like you, you're, you're a real company now. And so then you build a proper growth org. You have, you have a head of growth, you have a product manager, you have performance marketing, and ultimately you start having some designers and engineers.

[00:27:42] Who are fully dedicated to growth, and the, growth engineering fundamentally, like the easiest way to describe it, it's, the engineers working on growth as opposed to like actually building the product or as opposed to working on the platform or infrastructure team, right? It is the engineers, who work with the growth product manager potentially directly with the head of growth, to move forward the business's ability, to improve on all the growth metrics, which is acquisition, retention, engagement, referrals, et cetera.

[00:28:15] Nima Gardideh: How is that different from people that just work on product itself? Like why aren't they focusing on those exact metrics?

[00:28:23] Alexey Komissarouk: , it's a great question. I think it's, if we use the restaurant metaphor, right, you have a chef and ultimately that chef's job is to make the best possible meal. And the difference between making the restaurant successful versus not successful is how good that food is. Right. , and you can go to that chef and you say, Hey, could you please go outside and stand with one of those like sandwich board signs saying, Hey, check out my restaurant.

[00:28:49] And it's like, you absolutely could use one of the cooks in the kitchen to do that. , but is that actually the best use of their time? Are they actually, like, are they making, the food better by doing that? , and I mean, if you have two, if you only have two people in the restaurant, they should probably be focused on the food.

[00:29:06] If you have 10, 20 people, okay. It makes sense to have somebody focused on customer acquisition. It could be the same people. They're not, there's no obvious reason why they would be good at it. , but maybe some of them will be, or you could hire people who are explicitly focused on, like promotion, customer acquisition, and you're likelier to get those people to be more motivated, more focused, more specialized, et cetera.

[00:29:30] Nima Gardideh: I appreciate your metaphor. Is it, I know you have another one, so I'm just gonna explicitly ask you to tell the other metaphor, which is 

[00:29:36] the Hollywood metaphor. 

[00:29:37] Alexey Komissarouk: Oh, the Hollywood metaphor. Yes. , l l let me take you one step, from we're here to there. So that the counter argent with, the restaurant metaphor is like, okay, yeah, you need somebody to acquire customers, but that's just called marketing. Like why do you need engineers?

[00:29:54] , and engineers are not like hooks. , engineers are a very, special kind of han being, fundamentally, especially in a company because, engineers make whatever process their impact, like that is fundamentally their job, right? Like engineers come to finance and they say, cool, all your reports are automated now.

[00:30:12] And like triple check engineers come to customer support and they say, cool, we've like done a bunch of autoresponders and ChatGPT is responding to your stuff. Now engineers come to marketing and say, cool, we've built you a lead score that's automatically bidding on, Google and Facebook or whatever, right?

[00:30:25] , so engineering is not just one department. Engineering can go to any function within the company and improve, the efficiency of that, business unit. , and to some extent you can do that off the shelf, right? , you do not need engineering to do any of that, but once you get an engineer's attention, like they can take you beyond where you can get to with purely off the shelf, no code tools.

[00:30:47] , so the superpower of engineering is, , in, in the Marc Andreessen world engineering, is eating the world. Like whatever, work function engineering gets involved with, they can take it to the next level. , so let's start with that and then let's talk about what's the difference between an engineer taking the product to the next level, where usually the product is just, for, a SaaS company, the product is what the engineering team is building.

[00:31:10] Like, there's not, in in the Opendoor case, that's not true. The product is, we buy your house, right? , but sometimes the product is engineering in the growth sense. , the product is how efficiently are we acquiring the customer now, and the ways of working are very different because in the core product sense, To go back to the restaurant metaphor, you have this, you have this head chef who has a very strong opinion.

[00:31:39] , his last, his last restaurant had a Michelin star and he'll talk about exactly, like, it's exactly what to do. And the role of the cooks is to execute beautifully. And occasionally they'll have input. They say, what if we add some more cilantro to this meal? , but by and large, their ability to work, is determined by, how effectively and how with with what level of precision they can follow the head Chef's recommendations.

[00:32:04] , to, to take this to the Hollywood metaphor, the engineers here are actors, right? There's a screenwriter, there's a showrunner, and they came up with the beautiful script. , Aaron Sorkin knows exactly what the screenplay should be. He needs you to say the lines exactly as they are written because he is a genius.

[00:32:21] , and that, that's a lot of our core product is like, like if you've ever used an iPhone app, that somebody really, really loves, it's beautifully polished. It's because the engineers working on that are executing, the thing that makes 'em great is how well, they're executing at their craft. And that is not nearly as true for growth engineering.

[00:32:40] , growth engineering is closer to reality TV where there's still a showrunner. There's still a broad, general direction about where you're going, but you kind of, there's a lot more autonomy in figuring out exactly what the right answers are exactly, what should be done, right. If you're spending a bunch of your time telling your reality tv, like cast members exactly what to say.

[00:33:01] Like, that's kind of a weird way to run a reality TV show. , growth, growth engineers have a lot more autonomy, to figure out what happens. And the reason that is, by the way, is because it goes a lot quicker, right? , the amount of experiments and work that a growth engineer is supposed to do on any given time period is much more right, because we don't know what's gonna work.

[00:33:21] It's not worth, executing everything. , To the point where John Gruber would be super happy about it. It's like, let's just try 10 landing pages. Let's see which one, let's see which one of them work. And then we'll make it nice if we are actually seeing that it has that impact. , and at that capacity, the product manager just cannot babysit every single project, right?

[00:33:41] , some of those projects have to be engineers where they're both A, coming up with the ideas and B, making all the interesting small decisions about what to do. And it's not all gonna make it right, like a, a good, like experiment success rate is 30 or 40%. And that's fine. And that's also how reality TV works, right?

[00:33:56] Like not all of the hours of film that you've got, on Survivor are making it too. It's just, 

[00:34:01] can we figure out which parts that work and leave them into a coherent narrative? And that's how we acquire the.

[00:34:06] Nima Gardideh: Yeah, there's a couple things that stood out in that. So, first of all, I think that's a pretty good metaphor, especially the reality TV stuff. , in this variation of how you think teams should beru structured for growth. Are there growth products managers, 

[00:34:26] or are you thinking, okay, so, and, and their role is to still dictate at least the structure in which you're running the experiments?

[00:34:36] And I asse hypothesis definition, but are the engineers coming up with the solutions? Like, walk me through how the sort of orchestration works there.

[00:34:46] Alexey Komissarouk: Yeah, absolutely. , so what I'm describing to this reality TV world is the ideal, right? , that is not how you start at the end of the day. , let's say on a quarterly level, the head of growth and the. The relevant, like product prime manager and the engineering manager get in a room and say, Hey, what is the bet we wanna make with this team?

[00:35:08] It's like, well, we think our checkout flow has too many sets, is a bet. Or we think our homepage doesn't explain the value add properly. Or we think our onboard, we think our onboarding process, is like, doesn't do well for people on mobile, right? Like, let's find the big theme of the problem that we want to tackle.

[00:35:27] , and then let's go have a big brainstorm involved all the engineers, all the designers, all the relevant people, and come up with a big backlog of ideas that might be interesting. , and then the product manager with other stakeholders, we'll go and prioritize that until like, here's, here's the actual bets, here's the actual idea that's seem interesting.

[00:35:46] Let's turn these into designs. Let's turn these into specs so we can actually execute on them. , so if you follow this process doesn't sound that different than a traditional, product engineering path. You have a bunch of ideas and you're executing on them. I think the distinction becomes, A, how many of the specific bets are coming from the engineers versus things that, the product manager, the head of growth, the engineering manager had like, basically already known they wanted to do.

[00:36:13] So, like what percent of the ideas are truly coming bottoms up? , and then over time a single product manager can handle more engineers because the engineer can take the project further, can do, they need instead of going and being the product manager being the person in charge, they're the person orchestrating and they're the person approving instead of the person doing.

[00:36:33] So the engineer can go partner with the designer, figure out, iterate on the designs, bring something to the PM to approve, the engineer can execute and then bring it to the to pm to say, Hey look, you wanted a spot. Like, does this seem right? And then the engineer can productionize, turn on the feature flag, wait sometime, look at the results and then say, Hey, product manager, here's, here's how I think this, this result.

[00:36:53] Like here's a short write up of like how I think this experiment went in recommended excess. What do you think? And over time, if you, as your growth engineers get mature enough, more of them can own that experiment full funnel in the sort of like mini PM way, and that allows the PM to do higher level work.

[00:37:12] , you don't start that way. You start, you start more or less, in, in a tops down, pm driven way the same you would in, in corporate.

[00:37:21] Nima Gardideh: It feels like you're not only talking about autonomy, which feels like a sort of a cultural difference, or almost like a personality difference, but you're also talking about ownership, where the growth or the engineer in this scenario is less of an executioner and more of a thinker, and so they will own the solution to some extent.

[00:37:44] And, and you're saying the extent in which they would own the solution ideation and sort of like project management of it increases over time. , and is that sort of like the l seniority on a growth manager? Is it a seniority or, or like the longevity of the organization and the size of the organization?

[00:38:03] All of the above. 

[00:38:04] What is that like? Sorry, when you say In the beginning versus later.

[00:38:08] Alexey Komissarouk: Yeah, it's, it's a little bit of both. , I would say, right. A mature growth, growth organization is going to have more people operating in this capacity. , But also, and this is important to call out, like not every senior engineer is senior because they're really autonomous as, as a business problem solver.

[00:38:24] Like, some engineers sort of don't go that way and are still incredibly valuable because of their high ability to execute or, or their, their deep knowledge of the tech stack. So, there is not a monolith for what a senior engineer looks like in growth, but the kind that you will see a lot more in growth in core product is this sort of mini PM model where, they can just, if we're aligned on what the big idea is, they can own experiments and they're coming up with a lot of the experiments at the end of running.

[00:38:51] This is sort of the mold of IC that I ended up working on, working as at open guard.

[00:38:55] Nima Gardideh: Yeah, that just makes a lot of sense. What, so I asse, and correct me if I'm wrong, cuz I've been in that position you're talking about, but I've never hired for it. 

[00:39:06] How do you hire for these types of folks? They feel like very hard to find,

[00:39:10] Alexey Komissarouk: I mean, just, just look for YC companies 

[00:39:12] winding down. ,

[00:39:14] Nima Gardideh: ex founders,

[00:39:15] Alexey Komissarouk: So ex-founders is the easiest way, honestly. Yeah. Like actually, yes. , because they are most excited about it. , the two other ways are, I mean, one, people who have done this job elsewhere, like people who are coming from, growth engineering, growth companies with, like strong engineering reputation, just bringing somebody el into, in to, share the culture they had at Pinterest or somewhere like that, is valuable.

[00:39:42] , or you can train, right? Like the, other thing that, works relatively well is, bringing in, new software engineers, either if they're straight outta college or if they had a mid-career switch at either at a bootcamp, wherever. , they aren't, they aren't broken yet. , and what I mean by that is, after working at a couple of years, in a traditional product engineering role, engineers learn a nber of bad practices.

[00:40:12] , not, not bad globally, but certainly bad for growth engineering, right? They learned that in order to get a promotion, they should have, they should drive a complicated project with lots of stakeholders. , and they should be able to say how effectively they did this complicated thing, right? It's like, why does Google have four systems that do everything?

[00:40:29] Because every, some, everyone needed that promo. , and for growth, that's actually pretty bad. Like, I don't need the system to be complicated. I need it to make us money. Like I need it to be impactful. , and, engineers are respected in, product and platform engineering or for the beauty and elegance of their code.

[00:40:50] And, I will happily take shit code that made us millions of dollars. , and so, but that is a very hard pattern to unlearn because there is this understanding that is, what success looks like. , 

[00:41:02] And so you can either spend time, trying to change the mindset of existing engineers, or bring in engineers that don't have that mindset either because, they're former founders, so obviously they have the impact mindset.

[00:41:13] , or they're coming from a place where they already have that mindset, or they just don't mindset yet you have a chance to start scratch.

[00:41:21] Nima Gardideh: So it's interesting cuz at a high level I agree with you, but there's a difference between writing crappy code and writing overly complex code. Like I think there is code you can write that, has very little bugs. It's well-written, it's non-complex. Solves the business problem very well and it's written fast compared to like writing it in like, script kitty hacky way, right?

[00:41:55] Like, and I asse you don't want the script kitty hacky way. You, you want the sort of like elegant fast solutions. For the problem at hand, as opposed to the complex solutions that solves all the problems. And then 20 more.

[00:42:06] Alexey Komissarouk: Yeah, absolutely. And I think the reason that I say I'm fine with crappy code is because from the vantage point of product engineering, anything less than the bar that they're used to is script kitty code, even though it's perfectly reasonable, right? I think like, my job is to just like lower the bar, and say, okay, well do we need a hundred percent test coverage for this thing?

[00:42:27] But like, is 80% likely to be gone in two weeks? Like, are, are you sure? Like, can we not test it manually once for this release? , it's like, no, like, here's the code coverage, borrow at our company. Like, sorry, like we, we pride ourselves on our engineering excellence, right? So it's, I'm not trying to say there's like, the code should be completely gonzo.

[00:42:45] , no, no. There's definitely, engineering standards inside of growth as well, right? , we don't wanna bring the website down any more than than you do. , it's just that. The, like, the happy medi is, is much lower than where you are in a product world. , and so, I end up being the one advocating for scrappiness.

[00:43:07] But no, we're not, we're not, we're not hiring 12 year olds that have never worked on a production environment. Right.

[00:43:12] Nima Gardideh: , yeah, no, I just wanted you to clarify that because I asse that's not what you're, advocating. ,

[00:43:21] Alexey Komissarouk: Fair, fair.

[00:43:23] Nima Gardideh: All right. So the next area I wanted to focus on with you is just like even zooming back further. So, We, I like that we get, keep going in and out and that's kind of the vibe I like having. 

[00:43:38] If you were to at a high level, give me your sort of like, here are the six or seven or 10 or 20 things that need to exist in terms of like the best stack to have access to when you are trying to build a proper growth engineering infrastructure. And I asse you're gonna come back with something like, well, you don't need all of this in the beginning.

[00:44:03] You need like probably X and Y and then you're gonna add all of these things on top of it and , feel free to go that route. , but I given that now you've done this thing twice and you clearly have used different, Tools over the, over the seven years that you've been doing this type of stuff.

[00:44:24] What if you were to start from scratch, walking into a new team, new company, new product, what would be your ideal stack?

[00:44:33] Alexey Komissarouk: let's start, at the very beginning. , like the first problem that you have as a growth engineer, it's not what you can do, it's what other people can't do unless you help, right? , and that is the marketing automation platform or, or the lifecycle platform or the crm. Case like you need to, I think before you do anything, it is valuable to unblock entire organizations that can run autonomously once you have gotten them from zero to one.

[00:45:01] So the first thing that I would do, or I mean, not the first thing if you don't have a product, but the first thing now that you have a working product is I would set up a marketing automation platform like Custom Iterable, praise Clavio, any, any one of these. , because now marketing can hire a whole department of people just churning out or iterating on emails and push notifications and SMSs, which will ultimately like, and that they can ab test them and they can, they can improve them.

[00:45:25] That will have an impact on your business. , that is the single highest leverage thing you can do, because now all of these people can execute with minimal support from you. , versus if you were trying to ab test emails by yourself, I mean, yeah, you could do it, but it starts to get really annoying if you're the bottom link for every single email going to production.

[00:45:44] , now what do you need to, to set up that marketing automation platform? Well, you need to. You need two things, right? You need, when a user does a thing, you need to tell the platform out. So, hey, this user just entered their email. Oh, but then they didn't purchase. Okay, well cool, maybe we should send 'em an abandoned card email.

[00:46:02] So now somewhere in your production, you need an event that says, Hey, the user entered the email. We would like our analytics system to know about it. And then our analytics system will tell the marketing automation platform, or our event routing platform will tell the marketing automation platform, but also our event routing platform, will tell, the analytics system about it.

[00:46:18] So you can write a query that says, Hey, how many people entered their email today? What is our conversion rate from somebody landing on the homepage to entering their email? , and then finally, just on a very early on, you want an easy way to answer those questions. So you want an analytics tool. I mean, a lot of people use Google Analytics early on because it's free.

[00:46:39] , I think like, I've seen. Amplitude be really popular. I'm, I'm an Amplitude fan, as well. It's just a place where, in a self-serve way, people can say, Hey, show me the funnel between this and this, and oh, how does it differ for mobile versus desktop? Or how does it differ for paid versus organic traffic, et cetera.

[00:47:01] , so the three things I would start with is maybe the four things is a, some super easy way to send events. Hey, user did thing, outer to the event routing system. Then you want an event routing system, which, like for the, I mean by far the most commone is Segment, which does a really nice job of connecting everything.

[00:47:22] , but you can also just kind of hand roll something, early on. Then you want an analytic system, that just like lets you answer questions in a selfer manner. And then you want your marketing automation platform. , so that. People working in on the lifecycle side can be relatively autonomous and execute, on their own.

[00:47:41] Yeah. And there's a ton more we can add to that, but I think like if you're just 

[00:47:45] series, ab , like that, that will get you decently far. , and this is before, we haven't even talked about its experimentation or data warehouse at this stage.

[00:47:56] Nima Gardideh: And for this stage one, which is kind of like firing of the events before it gets to stage two, which is kind of the event routing, are you fan of those like products in that space like AVO where they're trying to like create some level of certainty about what properties you're sending, how often you're sending them and.

[00:48:19] , like code safety around sending events or have you not really explored that area?

[00:48:23] Alexey Komissarouk: I haven't heard of that tool in particular. I think it's really important, that the high fidelity events get sent accurately. Right. I think, the way we did this at Masterclass is there were. Five, five to 10 quote unquote gold standard events where these events absolutely, positively have to get out there because it's what Facebook and Google are optimizing on.

[00:48:41] , for example. And so we had much stricter monitoring and enforcement and integration tests around like those standard events than 

[00:48:49] everything else going on. 

[00:48:50] Nima Gardideh: So you're just writing tests for them? yeah, A ABO is basically like A tracking manager. It has observability and you can generate sort of SDKs, with type safety

[00:49:03] Alexey Komissarouk: Oh, that's very cool. Yeah. the equivalent that I've seen for this is Iteratively, which got bought by Amplitude and is now Amplitude data governance. , I haven't seen avo. , we ended up building a bunch of that in-house, I think. Yeah, if, if I were going from scratch today, I would look at some of those tools.

[00:49:18] , I think that's really nice to have. I think type safety is a really valuable way to. Reduce the likelihood that, you accidentally delete an event that like you're sending millions of traffic, of like spend dollar spend to, 

[00:49:32] , millions of dollars of, paid traffic through. , yeah, if it's easy enough to do it on day one, certainly do it on day one.

[00:49:43] , if not, then at least bring it in once you're really scaling up your paid traffic.

[00:49:48] Nima Gardideh: Yeah, that makes sense. And then step two is sort of the routing, step. I think you mentioned segment there is like, obviously nowadays Rudder Stack is like a hot player where there's like a million of them.

[00:49:59] Alexey Komissarouk: I mean, in a sense, Google Tag Manager is one of these, like, it's not, it's not my favorite, but the tag managers on the routing things are kind of, like kind of different versions of the same product. , and the whole purpose with these is, At the end of the day, you're gonna be sending data from a bunch of places.

[00:50:15] You'll be sending it from your front end, from your back end. You'll be sending it from Stripe or wherever. And you need that data to get into a bunch of places. You need it in your marketing automation platform, you need it to your ad partner, you need it in your data warehouse, need it to your analytics tool.

[00:50:27] , and you could write each of those integrations manually, but my God, you will drown over time. And so just having a single place where all the events go in here, and then we'll write some logic that says who deserves which events, and then they'll go out. , just having that standardized, makes your life so much easier.

[00:50:44] Nima Gardideh: Yeah, and unsurprisingly, I think they charge a lot of money for this, but it is so foundational to the stack that I think it's totally worth it.

[00:50:54] Alexey Komissarouk: Well, it depends on your, customer l tv, right? Like, if, if your customers are expensive, then , on a prevent basis, it's not a huge deal. , if you are, if, if your customers, if you are at, at a place where. , like your average LTV isn't that high, then maybe you need to handle, or maybe you need to find, an alternative.

[00:51:16] , I'll give you an example. , so at masterclass we use Segment. , one of the issues we ran into is a lot of our, pages or SEO pages. So masterclass is a decently, like large seo, footprint. , but the SEO traffic unsurprisingly, does not convert as well as the homepage traffic, right? Cause if you're Googling, how to cook an egg, like, I don't know that you're ready to pay $180 a year for masterclass quite yet, versus if you go to masterclass.com.

[00:51:50] , but we still have segment on there and like the conversion rate is sufficiently low enough that, that like segment on the SEO pages is like kind of a non-trivial expense. , and so one thing, some of the things that we've looked at are like, Well, what if we just sample this traffic? Or what if we, , for the SEO page, just go to Google Analytics, which is free.

[00:52:14] , and so if the rest of my business was similar where it was, SEO oriented, like probably I would not, I would not recommend Segment because it's, it just ends up being prohibitively expensive.

[00:52:26] Nima Gardideh: Yeah. And there are, there's certainly trade offs, but it just feels like such a hard thing to roll on your own. , 

[00:52:36] Having done that before, 

[00:52:38] I don't really wish it on anybody. 

[00:52:40] And, and so yeah, it is like something you should really, seriously consider. , okay. So that, that's the second step. And then the third step is effectively choosing the tools in which these routers are going to send data to.

[00:52:53] And you mentioned sort of like the first one being something like Iterable or basically like some form of a life cycle marketing toolkit and or like the, sorry, first one being an analytical self-serve analytical tool and then a life cycle marketing analytical toolkit. yeah, just Google Analytics being the free one that everyone uses.

[00:53:14] It's fortunately being sunset with a better version of it, which is much, much more akin to, amplitude. 

[00:53:24] Do you, how much of your time do you spend as a growth engineer, making these reports on these tools themselves? Like, let's talk about the sort of training part of this, cuz there are nontechnical people who don't understand how data stitches together and some of these concepts are new to them.

[00:53:41] Do you spend a good amount of time doing that? Whose job is that on the growth team? Does, should everyone know already? How, do you think about that?

[00:53:49] Alexey Komissarouk: Yeah. Yeah. So I think I have a bit of a hot take here. , I end up spending quite a bit of time, in Amplitude and in a, in something like Looker or in something like Mode, , or neracy where I'm, I'm rating raw sequel, right? Like, I think one of my superpowers, is like, I'll just, I can just go answer that question in five minutes, where a bunch of people have been debating and be like, lemme just check, like, like in the middle of a meeting I'll be like, actually, like, we've been arguing about this thing, but here's the query.

[00:54:20] And actually, like, it's pretty, it's pretty straightforward what the answer here is. Like, I'll be that, that asshole who does that. , and that comes from Open Door, where Open Door was an incredibly data driven culture. And so, there was this big push, like the data science was so important to Open Door.

[00:54:38] There was a lot of, Sql, fluency and competence. So I really picked that up there. , I think it's, so my hot take is, yeah, actually everybody should be in these tools because otherwise we're just guessing about the business. , and this, there's sort of, there's two levels of competency there, right?

[00:54:54] Level one of competency is can you do it in Amplitude or can you do it in the whiskey wig tool where, you basically you're auto completing event names and you're saying this event and then this event, and then this event, but for these users, what does it look like? , and there, and I've run these trainings for my engineers, at, at Open Door and at Masterclass.

[00:55:11] I think that's a really important, skill to have. I think like, I mean, the PMs are doing it. I think the should certainly spend some amount of time and amplitude. So they're not, because if they don't, then every time they have an interesting question, they need to go to their data, data analyst and say, Hey, actually, could you answer this question for me?

[00:55:28] And now it goes on a backlog of an overworked person, and it'll get answered in an order of magnitude of days, not, not like hours or minutes. ,

[00:55:36] And so being able to self-serve for the 80% easiest questions you're ever gonna ask, is a really, really important, is a really, really important tool for being autonomous, as you said.

[00:55:48] So I think PM should be doing it a ton. I think, en growth engineers as they become more mature, should be doing it more because that's how they develop intuition. That's how they can answer questions. , and the next level beyond that is, okay, but can you write actual sequel? Because at some point you'll run into something that is sufficiently complicated to answer an amplitude where the right answer is to just actually go into the data warehouse and, like ask the question there.

[00:56:12] And that's where the data analysts slip. Like the data analysts fundamentally have answers to like all of, they have access to all of the data, and so they can answer the more complicated questions. , and I will go in the data warehouse, like I don't care. Like, for me it's, it's super helpful and I don't mind like spending 15 minutes.

[00:56:31] Coming up with a, a quick poll, like I think my, my sequel has gotten good enough over the years, that I can do that. And I try to help my growth engineers do that as well when it comes up. But honestly, like I would settle for just people being comfortable going into the Google analytics, going into the mix panel, going into the amplitude and answering that first 80% of questions.

[00:56:50] Nima Gardideh: Yeah, and, and I think that there is a prerequisite for that to be sort of trustworthy and even if I know how to use Amplitude, the thing needs to work, which is kind of the naming of these things and events being understandable. 

[00:57:05] Have you over the years, sort of like decided on some naming legislature, some naming system that you follow whenever you are naming events?

[00:57:13] Like what is the process you generally recommend when it comes to tracking such that people can, do that auto complete in a safe way and think that they're looking at the right thing when they're looking at it?

[00:57:25] Alexey Komissarouk: There's two ways to approach this. , the way that most companies do it is they don't have anything and people just do stuff. And then over time it gets so painful that you like, spend a quarter or two cleaning it up, and then you bring in this thing that says, okay, like, here's how we name the things and here are the key events and , let's go delete all the events that aren't key.

[00:57:45] So the auto complete doesn't lie to you. Right? And let's go down to the actual 50 to a hundred events that are like real. , and that was, that was Q3 and Q4 at Masterclass. , it was a lot of fun. , but, if you want to avoid that, the thing you can try to do is early on try to establish the standards.

[00:58:05] Hey, here are the events. Here's, , using a or something. Here are the event. Re here's the event registry. Here is there's a review process to adding a new event. Like, all events need to be somewhat san. I mean, the challenge is it's a little bit heavy. , and early on if the company doesn't quote unquote, like, We don't know if this company is real yet.

[00:58:24] , it's a waste of time. , and so ideally you do that upfront, but in practice, like there will be, there will come as a time where you need to say, okay guys, guys, we gotta do a cleanup. , and the thing you can do before a cleanup in something like Amplitude is to say, Hey, here are the 10 events that are really important.

[00:58:42] We promise these 10 are legit. And we'll just name these, in amplitude as like, like gold or like, like, like, like, gold plated, like good, like standard, like, like trustworthy, right? And some just use these 10 events and everything else. Like we don't quite promise, but at least we can promise about these 10.

[00:58:59] And then when you have time, you can go back and do a full thorough cleanup and, a, a taxonomy and, and make that clear going head. But it's hard to do if you're not a real company, cuz it, that's not gonna make or break the business.

[00:59:11] Nima Gardideh: Yeah, totally. And like, so you write things like unit tests and integration tests for those, let's say gold events. , but the rest,

[00:59:22] who 

[00:59:23] Alexey Komissarouk: integration tests, 

[00:59:24] right? , integration tests, right? Like what you wanna do is at some level of maturity, you want to have a test that says a customer can sign up and all the right things happen, right? , you want to be able to say, okay, they go on the landing page, they hit the big red button, they pick a plan, they go through the checkout, they enter the fake credit card, and they purchase.

[00:59:46] And while that happened, here's all the events that needed to fire and when, and also at the end, the user needs to look like this, and they needed, they needed to have gotten this email, et cetera, et cetera. Like, I'm not a, I'm not a fan of having too many tests, especially in the growth surface area because, they'll tend to be fragile as like as you iterate on the process.

[01:00:08] But there are some nber of basic things that must be accomplishable and citing up is a pretty important one.

[01:00:13] Nima Gardideh: Yeah. And I think, to your earlier point, the importance of these events, getting to the ad networks is so big that the failure scenario, I think is worth the time to maintain some tests on these sort of like, events that are being optimized towards that. , it'll just save the company so much money, that they sort of like added, 

[01:00:36] let's say a, a friction to iteration is probably worth, the money that you'll 

[01:00:42] Alexey Komissarouk: yep. 

[01:00:42] yep. 

[01:00:43] Nima Gardideh: up these events.

[01:00:44] Yeah. 

[01:00:44] Alexey Komissarouk: exactly. Like, at the point that you are, sending non-trivial amounts of traffic at the point that, an hour of downtime is like non trivially pricey to you. , you need to put an, a integration test to reduce the likelihood, of breakage, but also b monitoring, right?

[01:01:01] , that says, Hey, like this event hasn't fired in like 35 minutes. Do you think nobody bought, or do you think something's gone wrong with the release? , and you want the ability to roll back quickly. , ideally you have a full stack monitoring tool that start, starts basically at the paid ad campaign all the way down to like, what's the CPA for it, like, where are users dropping off in the funnel so that when you learn that something is broken, where it broke, and that that investigation time can be shorter.

[01:01:28] Nima Gardideh: And is there like, sort of like a han level naming convention you like or you just, your main point is like, just come up with some standard, and you're good 

[01:01:37] enough? Cuz I know Segment has this sort of like object verb, . Approach or 

[01:01:42] Alexey Komissarouk: yeah. 

[01:01:43] Nima Gardideh: sort of naming system.

[01:01:45] Alexey Komissarouk: yeah. I was just about, I was just about to say, yeah. No. Yeah, we use the segment one. , I like the segment one.

[01:01:52] Nima Gardideh: Yeah, yeah.

[01:01:53] Alexey Komissarouk: I think, and then you have 

[01:01:54] to 

[01:01:54] figure out what your specifics areas 

[01:01:56] are 

[01:01:56] like. 

[01:01:57] Yep. Yep. If it's like, this object just served, right, this order is completed, and then over time, you, like, you can't get a little fancier, you can say like, oh, this was like from a landing page.

[01:02:11] Or you can say, oh, this was right. Like, like you, you can start to, to add more to your taxonomy, but, object verb is a really good start. 

[01:02:20] Nima Gardideh: Yeah. Yeah. 

[01:02:21] That's an interesting 

[01:02:21] one cuz I think it's like han readable. I like that a lot. 

[01:02:24] Alexey Komissarouk: The other reason you wanna get this right is, if you're ever gonna be on devices, right? , I think at Masterclass we had order completed and we had completed order at some point. And it's like, where is completed order? It's like, oh, it's firing at an old version of an i of the iPhone app that we can like never get rid of.

[01:02:39] Now. Now, right? Because it's just an old version and we're just gonna support it. , so especially if you're gonna be on devices, like you want to try to get that right up front. Cause that cleanup's gonna be costly.

[01:02:50] Nima Gardideh: No, that's so painful, I feel for you. , yeah, I, yeah. Some of these you can't even fix up. , so that's a very, very good point. , okay, so let's say you have all of this, you have a, a way to set events, to some router, to some analytics tool, to some lifecycle marketing, products. What's next? What are the one or two levels of abstraction that you can take us to before we wrap this up?

[01:03:20] Alexey Komissarouk: , so the next, few tools that I would add would be, one I would add, feature flags or AB testing, , experimentation, tooling, These days, there's, , EPO stats. , I mean, Optimizly full stack is sort of the, like one of the like fancier, like pricier players. , so I would add some sort of feature flag and experimentation tooling, cause that allows you to actually run your experiments and, and show your impact.

[01:03:50] , I would add a data warehouse, which is sort of implied here, but that's ultimately where all of the data is gonna sit. And that's where analysts will live and that's where a lot of your data science models, will work from. , and then I would add a monitoring solution, which is to say like, so now that you have the events flowing through, when they break, you want to know and you want to be, you want to be woken up, when something is broken.

[01:04:14] You, you want to have that level of comfort beyond just testing that, you are. You wanna have that level of comfort beyond just testing that when something breaks, you're gonna know. Like, but at the very least, if I haven't heard of a purchase in an hour, like I will know, and it will, if, if I'm on vacation or if I'm hiking, like my boss will, this will escalate in a safe way that we will catch this issue in a, in a timely manner.

[01:04:40] , so monitoring platform, either like Datadog or something more, marketing specific. , let's see. Yeah, experimentation, data warehouse and monitoring, I would say are the next three. , 

[01:04:52] Nima Gardideh: What are the, other 

[01:04:53] Alexey Komissarouk: ctl 

[01:04:55] Nima Gardideh: before

[01:04:55] we mo move away,

[01:04:57] Alexey Komissarouk: , which ones do you like?

[01:05:00] Nima Gardideh: I like Datadog.

[01:05:01] Alexey Komissarouk: Yeah, Datadog's good. It's a little pricey.

[01:05:04] , we were, we were using, new Relic. , new Relic has an offering for this. 

[01:05:09] , a masterclass. I also like, I mean, it's one of my investments, but I like this company called Funnel Guard, which specifically sits on top of like Facebook and Google one side, and then ingests your segment on the other side and gives you that, that full funnel view.

[01:05:22] Nima Gardideh: Mm-hmm. 

[01:05:23] Yeah, I like those guys too. We, we've ended up building a lot of that in house.

[01:05:26] Alexey Komissarouk: yeah. yeah. A lot of bigger places under building that house just cause it's so valuable. I think, the reason I ended up, using Funnel Guard was like, oh, great, this will save me a quarter worth of engineering. Fantastic. Yeah. Please plug it.

[01:05:39] Nima Gardideh: , and then, sorry, so you were going on, so data warehouse, I asse there was some like, tooling on top of that, that in itself feels kind of shared between product and growth. 

[01:05:51] Alexey Komissarouk: Oh yeah, absolutely. These, these like, yeah. 

[01:05:53] The data warehouse is ideally not owned by growth engineering, like the GR of engineering is a customer of the data warehouse. I mean, also, like of all of these things, like. Growth. Engineering's job is not to own tools to say they own tools, like ideally other teams can own them.

[01:06:08] , and growth engineering can simply be a customer. If it's a data org, an analytics org, like a platform org, that's always great. , yeah. But the da, the data warehouse, like as you're scaling up series day, series B, like, early on, what do you do? You just run a bunch of queries on production or like a production replica and it's fine.

[01:06:27] , but then, you bring in a like real analyst and they're disgusted by this and they say, no, no, no, no. We need to be running an ETL pipeline. That like creates a really nice view of the, of the customer that makes all these queries easy. And it's like, okay, sure, but now we need a data warehouse.

[01:06:41] Nima Gardideh: Yeah, that, that makes a lot of sense to me. And I think quite, often we are working with some data engineering and science team that just doesn't touch growth. And so we are effectively a customer to them. And then we end up producing some D B T models for them quite frequently and, and, and sit on top of their, infrastructure, which works quite well for us, I think.

[01:07:05] And, I love being a customer of, of a well run data team.

[01:07:10] , But sometimes you just don't have that team and you're effectively that team, but

[01:07:15] Alexey Komissarouk: Yep.

[01:07:16] Nima Gardideh: part of the work. , and then you just touched on, reverse ctl. You had mentioned census, just to be like a little completionist here. Did you look at other pro products

[01:07:27] Alexey Komissarouk: Not at the time. It was pretty early on. I think today I would take a look at high-Touch as well. , any others people should be looking at.

[01:07:34] Nima Gardideh: no, actually I only know of those two. I'm sure there are other players,

[01:07:37] but 

[01:07:38] they, 

[01:07:38] Alexey Komissarouk: they really own this market. Good for them.

[01:07:40] Nima Gardideh: Yeah. Yeah, they're both of them just

[01:07:42] really taking the market by store. 

[01:07:44] Alexey Komissarouk: et TL bubble.

[01:07:45] Nima Gardideh: We are. , yeah, I think that that stack really, really sort of fits for me. 

[01:07:51] , how do you think about these companies like Segment moving to become what they call themselves, sort of like customer data platforms instead of routers where they're trying to maintain, Effectively a copy of your customer, right?

[01:08:06] So like, I don't know if you've seen the profiles section of per persons personas. 

[01:08:11] Hmm. I'm forgetting the name of the personas. Yeah. They're, they're effectively trying to get you to do some of that reverse int, int l into segment itself. Keep an updated version of, of the user or the profile of, of your customers within segments such that they can enable all this sort of stuff.

[01:08:27] Like do you want these, to these companies to be your piping? Do you think they should solve this problem? , how do you feel about that? I'm, I'm, yeah, I'm just curious. I'll, I'll maybe

[01:08:37] voice my opinion afterwards. 

[01:08:39] Alexey Komissarouk: this, is one of my sore spots, man. Y you, you really got me on this one. All right, so, so first let's take a minute and define c D P because it's this term everybody throws around. , and it took me so long to understood what the hell it actually meant. All right. , so A C D P is a customer data platform.

[01:08:57] I wanna say, my best understanding of what A C D P is, is it's one sequel table that's very wide that basically says, okay, one row is a customer, and here's everything we know about this customer. And that might include which campaigns they've clicked. Like anything we know from third parties, like we know this is like, like a woman in their mid thirties living in Idaho, that's really into sports or something, right?

[01:09:18] Like maybe we, we've got that from a third party. We know all of their interesting activity on this platform. Ultimately we have this very wide thing that's like, Hey, here's everything useful. We might know about this customer that we could use for better targeting that we could use to send emails. , but it's fundamentally just one very wide table.

[01:09:33] , and early on, if you were to produce a table somewhere, the data warehouse makes a ton of sense cuz that's where you have all of the data. , and in my, like, in my experience, the way we've largely worked with these reverse ETLs is first we've written like whatever the reverse ETL view in our data warehouse, and then that's largely what we're sending out in the reverse etl.

[01:09:55] Like, that's the basis for what we're, for, what we're sending. , now separately, segment now claims it's a C D P, which is not true the way I have ever used them, right? Because to be a C D P. You would want data that we only have in our production data warehouse or in our production, and therefore sync to our data warehouse.

[01:10:14] , and there's just no way that data has gotten in ingested into Segment. , like, like Segment just has the, like a bunch of identifies and like events, 

[01:10:23] Nima Gardideh: Yeah. They, they have this now, by the way. They do, they are solving for this. ,

[01:10:27] Alexey Komissarouk: so Right, 

[01:10:28] right. So, so, so, as a, as a segment or now Twilio, like shareholder, yes, absolutely. This, this is a great, place to expand. Like, it, it's an adjacent market. Of course you should be going after it. , as a customer, it's like you do one thing well, like, please just like, can I use the tools that are good for everything?

[01:10:48] And, I think we played with Personas, a bit. I think at the time it wasn't mature enough. I haven't taken a look at maybe a couple years, if it's gotten better, maybe, but, like that's not what I want from Segment. I mean, segment also has a marketing like, A marketing automation tool now, which okay, great.

[01:11:07] But like, yeah, so Segment wants to be HubSpot basically, or wants to be one of these like, mega platforms that just does everything, which, worth trying. , but like, at least as of today, not what I want from them. 

[01:11:20] Nima Gardideh: Yeah, and I think this goes to this problem of like trying to grab a business and make more money out of it 

[01:11:29] and trying to expand across this sort of core team, which is the data 

[01:11:33] team. 

[01:11:33] Alexey Komissarouk: and Amplitude is doing a similar thing, right? Like 

[01:11:36] Nima Gardideh: Mm-hmm. 

[01:11:36] Alexey Komissarouk: be segmented now.

[01:11:38] Nima Gardideh: Yeah. Yeah. And I think, like, I don't love that as, as someone who has been able to like, actually love the separation of these things because they make me, just trust them more.

[01:11:49] It's like, okay, you're very good at this thing. I would like, I'm, that's lovely. I 

[01:11:52] want you to spend all your engineering power and efforts being very good at this one thing. , and it's how I felt with, mix Panel as a marketer and as a product person when they started doing everything. They added push and email and, so on and so forth, that it just diluted the value of the core thing, which was its analytical capabilities.

[01:12:11] And then Amplitude came and just ate its lunch. , and so I feel that that's happening to Segment. , and that's why they're weak and that's why, a company like Rodder Stack and, a couple others are, are competing well in the space. But, I'm glad I'm not alone in, in this view.

[01:12:30] Alexey Komissarouk: Yeah, I think my approach is like, that's nice. I'm just gonna use you for the basic stuff. And then, , like I'll give you one example. Before we, before we, pulled in monitoring, before we, we rolled out our own monitoring, we realized Amplitude now is a monitoring product

[01:12:45] where you can write a query and if that query changes by like enough, then they'll email you.

[01:12:51] But A, it's only hourly. B, it's only events you send. And then C, it's only email, so like, it's no Datadog. 

[01:12:59] Right. But I appreciated it. , we, we were beta customers of it for a couple of months. Like, like I think it was a decent stop gap while we were, while we actually like picked out our, like, longer term monitoring strategy.

[01:13:11] But yeah, like as a company, it makes sense to try to, to go after an adjacent market, as a purchaser of these tools. I'm just gonna use actually makes sense for each of the problems I have.

[01:13:22] Nima Gardideh: yeah. And I think it's a feature of the SaaS market, right? Because if you are in an enterprise, Already working with Amplitude, it's actually much easier for me to sell you up on this new future than it is for you to go after and find some new observability tool for your growth team. There is this procurement problem, so there is like systemic reasoning for this as well.

[01:13:45] Beyond just like, Hey, I wanna make more money as amplitude. It just tends to be harder for e people to come in and just sell you the one thing you want and we the best version of that one thing. And so there, there are sort of systemic, sort of tides against what, what we maybe as engineers want, but it just tends to work better and it's more effective, 

[01:14:09] Alexey Komissarouk: Absolutely. 

[01:14:10] Nima Gardideh: a tool to scale up companies.

[01:14:12] Alexey Komissarouk: , a as as we get towards the end, I think there's a couple of earlier points that you made that I want to comment on before we, before 

[01:14:18] we end up. Is that All right. 

[01:14:19] So I think you asked like, why the hell am I not a product manager? I think is, was like a very fair question. It's like a lot of former founders just go into product cause then they can drive, So that's, that's a great question.

[01:14:30] I think I'm just too misanthropic. I think I'm just, like, , I think I like, I don't know that I have the capacity for a product manager's, level of shareholder buy-in and like, like docent writing. Like I've done product management work. I've certainly like worn a product manager's shoes in all of the years, just, like filling in and like, it's a lot.

[01:14:56] , and I have so much more respect. I work so much harder as a product manager than I ever did as an engineering manager, just in terms of like, oh, and every week we have to write a report for this executive, and then every other week we have to write a report for this executive, and then we have to go to the cross-functional meeting where we present the thing at.

[01:15:13] I can do it, but like, ah, I'm so happy to have my product counterparts, doing that. I think the trade off there is like, you don't own the roadmap. You get to contribute to it, but you don't own it. , and. I well, the two things to that is one, I think like if you have a strong product engineering partnership like you, that isn't a real issue.

[01:15:35] Like I think, like if you, if you are, if you get to a relationship between a product manager and engineering manager, that there's that mutual respect there. , like I end up being pretty happy with the roadmaps that, we end up executing, one and then two, just on an IC level, like I think the best growth engineers are effectively mini PMs, and some of them do end up going into product manager roles.

[01:16:02] , but some of them say, Hey, like, I actually quite like just being able to run these experiments and my PM gives me and my em gives me enough autonomy that I connect, execute on this stuff. And I'm quite happy with it. Like, I didn't become a manager for a nber of years at, at Open Door because like, honestly it wasn't an issue until the company got larger.

[01:16:20] Right? Like I think, again, JD was, my product manager for a while and. , we had a weekly one-on-one, but there, there was just a lot of trust there, and so long as we were aligned on the priorities, I was, I was good to execute.

[01:16:33] Nima Gardideh: Was there some part of it that you just, you liked the building aspect, like, oh, I built this thing and I did so well and, and brought so many customers and made millions of dollars. Like, I know some 

[01:16:44] product managers are get that dopamine hit, but I tend to get it much more when I have like contributed, at least when a line of code or two.

[01:16:52] Alexey Komissarouk: yeah, so, so absolutely. Like just knowing about the win and being part of the win, is a great feeling. The other part of that is like, some of those wins could have only happened if it were you, right? Like, like a bunch of the ways we end up doing things, like in the process, like it's not like I foresaw ahead of time exactly how we would solve this.

[01:17:11] It's, it's. , like while figuring out the exact problem, I end up fighting the solution. But I don't know that a, a generic like cookie cutter engineer would've found. , and as a product manager, like, that's much harder for me to do. Cause it's like, well I guess they built it like a, if they say it's hard, I guess it's hard.

[01:17:26] Right. , but like, as an engineer in the weeds, like you, you can find what the elegant solution is. ,

[01:17:35] Nima Gardideh: Yeah, and, and it's interesting cuz I think I also struggled with this sort of decision earlier in my career. I was like CT up my own company and then some early stage startup and then I was done like for with startups for a little bit and I was like, okay, I'm gonna go into product. And so I actually did that, I did that for like quite a few years and it was a product 

[01:17:56] manager and there was a head of pro 

[01:17:57] Alexey Komissarouk: more presentable than I 

[01:17:58] Nima Gardideh: company. 

[01:17:59] Alexey Komissarouk: You're you're just a nice guy,

[01:18:01] Nima Gardideh: It's the shirts. 

[01:18:02] Alexey Komissarouk: I think. Yeah, exactly. , I 

[01:18:05] I get to 

[01:18:06] Nima Gardideh: I actually, In going back, 

[01:18:08] Alexey Komissarouk: with it.

[01:18:11] Nima Gardideh: yeah, I what's funny is that like the first, first role I was very bad at, I would say I was in everyone's face, and quite rude, but somehow effective, which is why I kind of was able to be still in that role by the end of it. But by the end of the day when I was at the head, Linux was head of product role, I actually asked them to still be able to code, and I was able to do both of those jobs, but just not, well, obviously it was an early company and it was like sub 20 people.

[01:18:44] , but there is something special about. Yeah, let me backtrack. Cause I think the reason I was talking about this like line of code commit thing to, and asking you that is because the thing that I find very interesting is just like, is code, code feels very, like, for lack of a better word, beautiful to me. And that is like what gets me excited.

[01:19:13] , and then I have to put that in juxtaposition to like my love of business effectively. , and so that, 

[01:19:21] that's kind of where I like, get stuck. It's like, I want to just like do business, but like code is pretty, and so then I get 

[01:19:28] Alexey Komissarouk: Yep. 

[01:19:29] Nima Gardideh: this tension exists. So I really feel it and I think, I just also happen to be really social, like social dynamics, which it sounds like you are not as into.

[01:19:42] , so I think I both paths are pretty clear paths for me, 

[01:19:46] Alexey Komissarouk: Yeah, 

[01:19:46] Nima Gardideh: , I think

[01:19:47] Alexey Komissarouk: I, think, I think for like, for burnt out startup CTOs going into product is an absolutely reasonable solution. Going into growth engineering is also an absolutely reasonable solution. And, 

[01:19:56] , like also from growth engineering, you can go become a product manager. Like that is, that is not an uncommon path.

[01:20:03] , 

[01:20:04] So you you skipped that step. I think it sounds like you would've enjoyed it quite a bit. , and it sounds like the work you do now has gotten you like closer to it as well. , but yeah, it, it certainly, it was a great path for me and I would, I would encourage others. , I'm a, I'm about to, a hard stop here.

[01:20:23] , do you mind if I do a little plug?

[01:20:26] Nima Gardideh: go for it. Go for it. Yeah. We're 

[01:20:27] going to, I want to talk about that last anyway.

[01:20:30] What are you, what are you off to now out these days? Tell us 

[01:20:33] Alexey Komissarouk: so, so, I'm doing two things. I, moved to Japan and so, wound down my time at Masterclass in March. And I'm doing two things. One, I'm writing a book about growth engineering. , I think there's not a ton of stuff out there that's, there's a ton of stuff out there, for growth.

[01:20:47] There's a ton of stuff out there for engineering. But as I was building curriculs for training, in my organization, there's not a ton that I found that was growth engineering specific. So I'm gonna try to write, that first handbook. , if you wanna follow along, Alex and k.com. , Come, come hang out.

[01:21:05] I'm, I'm putting early drafts up. , and then second, I'm doing fractional consulting, which is basically, if you don't have a head of growth engineering yet, but you're starting to spin up that org, I can spend some time at a company and I'll, I'm, I think I might be booked up. , let, let by the time this goes up.

[01:21:21] But, always happy to chat with people, if they're curious.

[01:21:26] Nima Gardideh: Yeah, that just means you're gonna be even more expensive by the time they get to you. , Alexa, thank you so much for coming on. I really appreciate this. I don't have a lot of people to talk to about this stuff, and I think you've, you've definitely been one, over the years and I really, really appreciate, having in mind like yours around.

[01:21:46] I'm excited for the book and I really feel quite grateful for your time, 

[01:21:50] Alexey Komissarouk: Thank you very much. 

[01:21:52] Nima Gardideh: waking up so early, 

[01:21:53] Alexey Komissarouk: Yeah. 

[01:21:54] Nima Gardideh: Japan.

[01:21:55] Alexey Komissarouk: And likewise, you've been a great mentor to me and I really appreciate, anytime I can come to you. , what the cool new thing that isn't census or high touch is

[01:22:06] Nima Gardideh: Exactly. All right. Thank you so

[01:22:08] Alexey Komissarouk: all right down.

[MUSIC FADES IN]

[01:22:10] Nima Gardideh: All right, and that's a wrap. Thank you LexI for coming on such early hours and waking up early to do this podcast with me. I'm a big fan of his work. I think you really should be listening to this podcast and thinking, now how do I hire these types of folks or build the process that he talked about in the episode.

[01:22:31] and having the different standards for growth engineering work compared to product engineering work, I think is a very hard thing to come to terms with, and he has very strong opinions there. Opinions that I agree with. So, I hope you've, you've taken this away and, and, and are going to have some, conversations internally about it.

[01:22:49] up next we're gonna have Rodriguez Hutt. He's the VP and GM at Viome Life Sciences is a company that does quite interest interesting work. I learned a lot about the biotech space through that conversation. but also having someone who's sort of been a founder and, and, and. Now leading growth at, a company where he understands the sort of full dynamics of, of an industry was, honestly humbling, to say the least.

[01:23:17] So I'm excited for you to listen to that episode as well. it's coming up soon. See you on the next one. 

[MUSIC FADES OUT]

‍