<MUSIC PLAYS>

[00:00:45] Nima Gardideh: Hey everyone, this is Nima Gardideh, your host for The Hypergrowth Experience. We've got another episode here with Leon Sasson, the Co-founder and CTO of Rise Science. It's a sleep tracking app that launched during the pandemic after a pivot from helping folks in sports teams improve their sleep and performance, and going direct to consumer afterwards.

[00:01:12] So it was great to speak to him. We had a lot in common mostly around building cool stuff, robotics, physics, and even we touched on economics for a little bit, on the podcast. It was really great to hear his story and Rise Sciences’ story throughout the pandemic with this big pivot.

[00:01:31] And what they had to do to reallocate some of the resources of the company and go through a big change and survive through it. So, I'll have a lot of respect for him and his co-founder for going through that process. We also touched on iOS 14.5 and the big changes to the standards of privacy and the ecosystem and how it affected their growth and their approach to marketing.

[00:01:59] And I'm glad to be able to talk to someone who is a little bit more technical than the average marketer, who understands sort of how these things connect. It was great to talk to him. So I hope you enjoyed the chat. Here's Leon.

[00:02:12] Leon Sasson: I have always been a builder. That's kind of, I'm still a builder. That's my main thing since I was a kid. Kinda just building robots and learning how to program since I was  10, kinda that kind of stuff. So it's super, that was always super fun. I was kind of a, that guy that was always trying to just build games and build electronics. 

[00:02:32] Nima Gardideh: Was there a first thing that got you into it? Was there a specific thing you built or what? I guess what was the curiosity starting from? Where was it coming from?

[00:02:43] Leon Sasson: I don't really know, but my mom always said that I would tear apart my toys to get the electronics. I don't know why I would do that, but apparently as a kid, as a five year old, six year old, I would do that. But no, I think I was really into gaming as a lot of people wear and I just want, I was, how do I just build my own game?

[00:03:05] And then it's, oh, okay, you can program things and then you start to figure out how to program. There was a small kind of computer lab at my school, at my K12 school. This was only for seniors, but I would go in and try to just use basic, I don't know why they had basic in those computers.

[00:03:25] Cause it was 2000, I don't know, 2000 the year, 2000 probably or something and I would just start building games and then you'll realize, Holy cow, you can just kind of make computers, do whatever you want and that just becomes kind of an unfathomable thing that it's you can, whatever you can think of, you can just make the computer do.

[00:03:48] And that just ends up opening up this rabbit hole of, all right, you can just do things. And that led me, I don't know how I made the jump, but then I was like, what's  below the computer? I type these things in the computer and then it does something. I make a little game and you can move your character in the game.

[00:04:09] And it's like, why? When I press a key does he move the character? And then you realize, okay, there's electronics and there's something that your code ends up being into distinct old chips and semiconductors and all these layers in between. 

[00:04:24] Nima Gardideh: So you went from, I'm curious about the software thing because I like games to, Oh, how does this thing work? And then you got into this hardware layer. That's interesting. Cause this gaming path is so common for engineers that I meet and I'm quite grateful for them. I think I got into software fruit because of gaming as well, but I, at that point, I had gotten into hardware already and I was building stuff.

[00:04:49] And then started playing PlayStation Portable, and then I was, I wanna hack this thing to get free games on it and started learning engineering. So yeah, this gaming path is interesting. So you go down to the hardware level and then you start building robots that, So I actually also did that, but I'm curious how, was there a thing in your school or you were just like, I want to build stuff.

[00:05:10] Leon Sasson: No, no. My school had none of the, I was in a small school in Panama, so  it, I think now my school has a bunch of STEM programs for kids. But, I was still, the main activity was  going to play soccer with my friends. So this was after, or something, but, Yeah, I don't know how it, when you go into electronics and there's, Oh crap,

[00:05:33] you can learn that electrons flow and you go from analog to data and you can  start, figure out how the building blocks work. And that was maybe a two, three year period where I did in the software and I would a longer story, but I would, I have this  imaginary company and you can, the large semiconductor suppliers in the world, Fairchild, semiconductor mouse distribution, they're large B2B companies, massive organizations, and you can actually order samples.

[00:06:07] And they would overnight them for you. If you need to transistor, most customers are very large kind of R&D department. So if you needed a single chip, you can just place an order for free and they would overnight it to you anywhere in the world. 

[00:06:21] Nima Gardideh: That's amazing. [Laughs]

[00:06:21] Leon Sasson: A sample and then, you can only do that five times a month or something. So I would just get parts for free for this imaginary company, I had to build robots cause you, I couldn't find how to get  parts in Panama. So you start ordering chips and ordering transistors and high powered electronics. So that was kinda a fun little, just building stuff.

[00:06:46] And I don't know how I then just kept going back into software and in school. Got a lot in college, got a lot back into it. And I gets here we are doing a lot of software these days. 

[00:06:58] Nima Gardideh: And so how, so you chose, eventually go to, you studied computer science, right?

[00:07:04] Leon Sasson: I then went to Chicago for university and I actually started as a physics major, which is another rabbit hole was if you go deeper from the games to software, then electronics and it was , how do the electronics work so you can spend two years trying to figure how the physics work? So I went very deep.

[00:07:22] I realized I'm not good at math at that level. I don't really do theoretical physics and that's just not for me. But I then kind of got into, I designed my own major because I was really into industrial engineering and computer science, which are kind of, have a lot of similarities where they're still very different programs.

[00:07:41] So, spending a lot of time thinking about the industrial engineering side of the house too, which is kind of , how do you optimize processes and systems and everything in the world is, tends to be a statistical systems. Not very few things are deterministic. They're kind of statistical processes.

[00:08:00] You can call them sarcastic process analysis and I think that was just really interesting to blend that with computer science, which was kind of more theoretical, just  how to design computers and algorithms and systems on that front.

[00:08:18] Nima Gardideh: That's cool. Do you feel  that helped you on the industrial design side, think improbabilities and less in binary outcomes? 

[00:08:25] Leon Sasson: Oh, I'm, yeah, I think the, and no, by no means I'm great at statistics, but at least a program that gives you pretty good foundation of the basics. So you can just at least understand what you don't know. And you, there's a lot of  counterintuitive nature to statistics and  our brains are really bad at it.

[00:08:46] I was just chatting with my wife I think last weekend and I was telling her the birthday paradox, which is  a very common stats 101 problem of, if you're in a group of 20 people, what are the odds that two people in that group share the same birthday?

[00:09:06] And, if you think about it, most people say, well, 20 out of 365, so it's maybe a little bit less than 10%, right? Because there's a lot of days in the year on the 365, so the odds are low. But no, it's actually almost probably higher than 60% because the way statistics works, you just have a lot of permutations of 20 people each against another 20 in a way.

[00:09:33] So, even some of that basic statistical foundation and at least understanding that it's counterintuitive and what at first, things might not actually look the way they are, I think just becomes pretty powerful down the line. Especially now that I work a lot in and, and even product, right?

[00:09:58] A lot is just trying to pick up patterns and know, and know when there are patterns and when there aren't patterns and just kind data or signal from all the sources that aren't actually real.

[00:10:11] Nima Gardideh: Do you feel your base intuition has changed or you just have the intellectual guardrails around these things? If when you look at a situation, do you  intuitively know that this is not a binary outcome? Or it's your brain starts running and you're, Okay, well this is probably probably a six situation, and then you start thinking through it.

[00:10:32] Leon Sasson: Yeah, I mean I think I rarely actually go and run the numbers and everything that I don't think you can live life that way. But, there's a lot of things that you, I think my intuition, it just makes me skeptical of most things out of the hand, which sometimes is annoying for everyone around me 

[00:10:48] because it's just very easy for there to be a pattern when there, for there to appear to be a pattern when there isn't, especially when having things that are very complicated systems. And that goes everywhere from when I'm reading books about nutrition and there's so much complicated science on  what actually works, because statistics are really hard, for example, Right?

[00:11:10] All the way to sort of  regular day to day stuff when the classic example you run into someone from your childhood at the grocery store and you're, Oh, what are the odds that you run into them? And it's, well if you actually

[00:11:25] wanna run the odds, it's pretty high that you're running to one of the thousand people you've ever met at the place that you were at your hometown, right?

[00:11:34] Nima Gardideh: Yeah. People have these, oh, fate or the world is so mysterious and, well no, it's not that mysterious that you ran into the friend that you went to this university with, and they're in New York, because you also live in New York. 

[00:11:49] Leon Sasson: It definitely makes me a lot more skeptical, but I think it's kind of a healthy dose of skepticism sometimes is good. 

[00:11:55] Nima Gardideh: Yeah. I think that's probably a generally good approach, especially when you're looking at, in my head, the way you say complex systems, the way I think about these things, it's systems that have a lot of dependencies, systems that are reliant on all these things that you don't understand yet. And so you look at one isolated incident or why one isolated metric and you just think you understand them all of a sudden.

[00:12:22] I think people in economics tend to have this problem. It's very hard to trust people in economics, in my opinion, just because their explanation is always way too simple. When you have  many billions of people interacting at scale, it's just really hard to actually understand their behavior. So you're trying to look at inflation right now and read about it and there's all these neo-economists talking about it. I just don't believe most of them, just because the explanations are just too simple.

[00:12:56] I want a five page essay about why you think there is inflation and I need math. I would like a math degree to understand it. Do what I mean? That's how hard it should be. Just give him the complexity of the system.

[00:13:07] Leon Sasson: So complex systems are interesting. I am definitely not an expert. Took a few kind of undergrad research classes, actually trying to understand, trying to model something, VC funding and complex systems. But there's a whole subfield called agent based modeling, which instead of trying to get back to your inflation point, instead of trying to model, Hey, what's the formula that models supply and demand and inflation and monetary policy into one nice equation.

[00:13:37] But you can also approach it as an agent based modeling that is, imagine you had a thousand kind of units agents, and each is kind of  a family or a household and they have very simple rules, right? They, as part of the rules, they do very simple things  they put in a unit of work and they spend X amount of money and they, and, and you get very complicated emerging behaviors emerging from very simple subset set of rules that often actually can converge with the regular, not regular, but traditional economic analysis.

[00:14:12] And that field is fascinating. I got a good friend of mine that works in that and doing a lot of research on how to use these new set of tools, that can be compute intensive to do that kind macro level modeling, which is totally different.

[00:14:26] Nima Gardideh: Which is kind, what's super cool though. I have, I like one of these economists actually, I think he talks about the three agents of capitalism and it's  the consumer, the bank, and the government and he talks through it and runs some models. It's quite interesting. I'm forgetting his name right now.

[00:14:42] But, I like those models a lot because I actually think that's the right way of thinking about it is just the atomic units that end up creating the more complex system. It's almost understanding the atoms better and then you'll  get a sense of how the whole thing works. Although in physics that's not necessarily true. you keep going deeper, you don't understand how things work. So, cool. So you went to school, so it sounds like you were already into startups in school? You were doing research about venture capital? 

[00:15:13] Leon Sasson: Yeah. I just got into, I don't know, in school you're just exposed to a lot of things, which I think that's one of the best things about school. There's a lot of the, a lot of the conversation these days is pretty anti university, which I'm squarely contrarian against, I think going to school as being for someone from Panama, we kind of have a very different point of view in the world.

[00:15:36] Very different access to things. He just really opens up and exposes you to things that you wouldn't otherwise. But, very quickly, are there extracurriculars and your friends are doing interesting things and my friend was like, Oh, look at this startup thing, and get an internship and you just get exposed to, oh wow, there's this whole new world of things happening.

[00:15:57] And, it helps that I like programming and software, so that just fits right into the tech and startup world. But, yeah, just doing a bunch of research and, and one of my, that's how I got into what we do, which is one of my best friends back then who's now my business partner. He was very into sleep.

[00:16:21] He would just tell me get better sleep. I used to be the person that slept five hours. I thought sleep was a waste of time. I'll sleep when I'm dead. [Laughs] I wanna

[00:16:32] have fun. Yeah, sleep is for the weak. I wanna have fun. But then also engineering school, so I have dumb problem sets until 4:00 AM and class at 9:00 AM So  you're just not sleeping.

[00:16:43] And then my friend Jeff, who was like, you should stop doing that. You're  literally functioning like a drunk person based on all these interesting papers and you're killing yourself. And he would just start sharing the data and the scientific literature and studies. And long story short, I started changing my habits there.

[00:17:04] But then he also got very into doing research, not only learning about it, but also doing some research with a couple professors. Northwestern had very good sleep scientists and for some reason back then, a lot of, even now, but back then more, a lot of sleep performance studies, a very good outcome for performance and ends up being athletic performance.

[00:17:33] because it's very often very quantifiable and people care a lot about it. So we just did a lot of research with football teams and lacrosse and soccer, just a lot of the kind of, college, division sports teams on how sleep affects, how fast they're on accuracy, strength training, all that stuff.

[00:18:00] And long story short, I guess their research became kind of a program that we could then deliver to any sports team, and that's when we  first launched as a small company when I was still in college just working with kind of higher end sports teams to try getting sleep as part of their just very intense set of discipline training they do.

[00:18:25] Right? If you're a football player, you're doing all sort of both from  your strength training to get stronger to your kind of actual function if you're a quarterback versus a line versus a kicker. You do a lot of functional training, but then you have nutrition and mental health and sleep was a big hole.

[00:18:45] It still is to some extent, but now there's a lot more awareness. But back then you're talking 2012, 2013, it was not that clear. And even tracking sleep was a nightmare, right? I spent probably three years trying to figure out how to track sleep reliably and issues. But back then, when we first started this stuff, Fitbit was the, the Fitbit we spent didn't exist.

[00:19:09] So they had the, it was cool. I think the Ultra or the Ulta, it was kind of a clip that would go on your belt or your pocket, not an actual race. So it wouldn't even track sleep or activity.

[00:19:20] Nima Gardideh: Wow. Yeah. Was Jawbone out then? I think right around that type Jawbone was launching maybe. 

[00:19:25] Leon Sasson: Jawbone launched a year, 2014. I think that's when we started trying to get a partnership with them and you had a few other players.

[00:19:33] It's not that we invented tracking, but tracking wasn't an easy thing to do. Right? Bluetooth wasn't reliable. So most trackers you still needed to connect to USB and try getting a hundred football players to connect something. We actually had one that would go over your arm, not even wrist?

[00:19:52] Nima Gardideh: Did you end up building your own at some point, or no?

[00:19:55] Leon Sasson: We never built our own full hardware, but at some point we did work very closely with an OEM out of Finland. For our own under mattress sleep sensor. So it was kind of a device that goes under the mattress, which works really well. And the beauty is that it doesn't need a user to do anything after the first time you plug it in. So it's very kind of frictionless versus all the bands and wearables that you have to plug in every day and charge and connect and all this stuff. 

[00:20:25] Nima Gardideh: Were you working with the Northwestern football team and the college level, or were you immediately going out there and working with pro teams?

[00:20:34] Leon Sasson: I think both, I mean, the college level. The college level tends to also be fairly financially lucrative. Cause they, at least football teams and basketball teams tend to have budgets for stuff like that. I mean, it's surprising you would think it's more funded than it is even at the pro level.

[00:20:55] When you look at the budget of a large NFL team, you would be surprised that they actually don't have a lot of cash, except for player salaries and marketing. Right? But point aside, yeah, one thing led to another and we got very deep into not only sleep, but the whole kind of performance world.

[00:21:17] Athletic performance, how do you measure injury rates and, and the load for a player. So all that world is pretty sophisticated, but we spent quite a bit of time there and, and yeah, it's kind of one thing led to another, just pulling a thread and it was very nice that we were in school back then, so you could kind of intertwine business ideas with classes and you could, every class, our project would be something related to the business or we would take a bunch of design research classes at the Design Institute and it's all right, let's just do, use the research for our customers. And so you can do a lot if you intertwined the two. And that was very proactive. 

[00:21:53] Nima Gardideh: So you started this company, so it is more of a B2B because these  companies were paying, You guys do that. When did you become a consumer business? 

[00:22:02] Leon Sasson: It was fully B2B, so it was fully B2B and very large enterprise. So working with a NFL team, you're talking about six figure contracts and long sales cycle. So we had to learn the B2B side and we got decently good at sales and more my founder than myself. But still, it's hardcore sales.

[00:22:25] You were talking about, some of these college football teams are actually part of the public domain, University of Alabama, these are technically part of the state budget, University of Tennessee, these are state budgets. So you're even negotiating with  the state, right? You get a check from the state of Texas or something, right? But no, it was all B2B and we always knew that we care a lot more about helping people improve sleep and understand that sleep is so impactful. Because even, I think now in 2022, it's a lot more, there's more awareness of sleep and its impact not fully, but there's more than you go back.

[00:23:09] I think at this point, 2015 or so people were already talking a lot about nutrition and fitness obviously, but sleep was still not really in the space of health and wellness. Right? It wasn't really talked about too much, but, we knew the impact was there, right?  Everyone should need to be understanding why sleep and circadian rhythms matter and how you should be taking advantage of them for your life.

[00:23:35] Cause everything you care about in life, whether that's your cognitive abilities, your performance, your physical abilities, your promotional levels, weight loss, everything it just related to at some point sleep has a huge impact. So we knew we wanted to have impact on the sleep side and athletics was a very clear entry point and we kind of, it was obvious that we could do a business, but the idea was to.

[00:24:02] Figure out how to bring this to the mass market, right? We were never sort of athletic first. It was never our idea to be, Oh, we love athletics and we wanna just optimize, the world's best athletes a little more, we wanted to, how do you then bring that to millions of people? And we didn't know that was a problem.

[00:24:23] And part of it at one point ended up being, a lot of companies started reaching out to us through our network. And we had, one of the great things about working with athletes is that you get a really good product because you're working with large teams and you just very kind of good for stories and PR and marketing.

[00:24:43] So these stories would come out of us working with football teams and NBA teams and companies would reach out and say, Hey, why don't you do this same program with our sales team? Why don't you do this program for our team of a hundred people? So, we started exploring, okay, how would that look if it turns out a sales team at a large company is kind of  the same thing as a sports team.

[00:25:08] It's just that maybe the budgets are smaller and the performance maybe matters a little less for day to day, and it's harder to measure. But a lot of the basics, they wanna perform, they have a job, but they have a lot of personal conflicts and trying to help someone still improve their sleep and habits.

[00:25:26] It's very similar. So we started doing that and that was going well. B2B sales can be tricky. And then honestly, kind of to close the loop back to how we got into the consumer side is when COVID hit. The athletic world shut down. I mean, everything shut down, but the athletic world totally shut down, right?

[00:25:50] Nima Gardideh: That was still your major point of revenue.

[00:25:52] Leon Sasson: That was a major point of revenue. Even though we knew that was in the future, that was still a major point of revenue that shut down. B2B kind of shut down because no companies had any spare budget. If you think about March 2020, April 2020,  it was kind of dark times.

[00:26:07] No one knows how long it's gonna take. You don't have a lot of, we didn't have a lot of cash left. So it was right, we know our product works well because we give this product to people in sports teams. We give this product to people in regular companies and the end users are just kind of regular people and retention is good.

[00:26:27] Our core product is good. So we decided, right, how about we spend some time just going, putting out a subscription offering and try to figure out if we can grow this. because that might be the only bully that we have. So we kind of split and my co-founder was trying to figure out if B2B could scale. And I was like, right, I'm gonna go figure out if the consumer side could scale. Cause we need to make a decision and we need to get something working before we run out of cash.

[00:26:59] Nima Gardideh: How many years in were you at this point? By 2020?

[00:27:04] Leon Sasson: Probably five or six from a very different company, right? It started as almost a consulting, tech service for sports teams. But at this point it's still the same entity, the same legal entity and company. But it definitely has more over time. And so roughly five, six years.

[00:27:25] Nima Gardideh: And then you had fundraised? What form of fundraising did you do? Angels? Did you go down the VC route?

[00:27:29] Leon Sasson: We had some funding before then that now you would call this pre-seed. Mostly Angel, a small fund joined. And that was still when we were athletic, trying to figure out the mass market and at that point when Covid hit, we already had proper seed funding from True Ventures and Freestyle and High Alphas and really great funds out there.

[00:27:58] So, we definitely had funding to take it to the mass market. That's when we raised our seed round in 2018 to figure out, how do we bring this to the mass market? And we were exploring… 

[00:28:10] Nima Gardideh: And that included sort of B2B SaaS teams or… 

[00:28:13] Leon Sasson: B2B Saas was one of the main of the main hypothesis because we had so much inbound of companies trying to bring this to their employees. 

[00:28:22] Nima Gardideh: This totally makes sense. I would totally, I talk a lot about sleep with my whole team and it's funny enough as today I had awful sleep and we were talking about this earlier, but generally speaking, I'm very aware of the power of it. And then we pay for mental health stuff. So, I can understand that I would totally pay for sleep training as well. 

[00:28:45] Leon Sasson: We should get you, we still have a few customers, so we should get you on, but, it's, yeah, we can talk about why the sales side is stuff, but it's, we have some very interesting anecdotes and insights there, but ultimately you're either going kind to health, HR or insurance to really scale up, rather, you're talking about get, how do you get to 10, a hundred million in revenue, Right? You need to go either hard HR or kind of the insurance or even health provider side. And these are very different cycles of sales and go-to-market. 

[00:29:19] Nima Gardideh: I really dislike those paths because they're, the money's coming from someone different than you're solving for.

[00:29:26] Leon Sasson: The problem with that, and this is now two years in the past, it's not what we do now, but you're then trying to sell something that no one is necessarily actively buying. So you first need to, there's very few people that have under their list of priorities, I need to get my team sleeping better.

[00:29:45] If you convince, if you educate them on why the reason their cells are lower is because your salespeople are tired and they're literally over the phone when they're trying to close a meeting. They sound less energetic. And there's great studies on how with higher sleep debt and sleep probation, you sound less emphatic.

[00:30:07] There's a lot of really good stuff, but, no one, the head of sales, their priority is to hire people and increase sales efficiency and close more deals. And so you need to figure out how to link sleep improvement to their main priorities versus directly solving a problem that they're actively trying to solve, right?

[00:30:30] If they're trying to buy a, improve sales efficiency, they know they need a better CRM and that just kind of a direct sale. So, it become, it's doable, but it was only really working well for more progressive leaders. 

[00:30:47] Nima Gardideh: You have to figure out how to tell the story. Right. And it has to be even the right people listening to the story.

[00:30:51] Leon Sasson: It was already for people like you, that are, Yeah, I know sleep is important. I own the budget. I can make decisions and, yes, I want this. But, there's few people out there, a very large companies that think that way and already have a budget secured for things like that. So it was doable, but it was a tough sale. I think we're gonna at some point get back, because it's so obvious.

[00:31:15] Nima Gardideh: So one last question before I wanna jump into this whole consumer thing. So at this point in time you had some sales from both of these sort of  general markets, this sports world and B2B SaaS teams or just businesses. What was the structure? Was there an app already and you were selling it and then people got access to this, to this app? What were you exactly selling at that point? 

[00:31:37] Leon Sasson: Yeah, for most it was a fixed per user, per year contract. And it had access to, when you sign up, you would get a physical box with a sleep kit that had  a hardware device that would go under the mattress. You would get a sleep mask, orange glasses to help you with melatonin production at night.

[00:32:00] And earplugs that you could, in case you're traveling and you need soundproof stuff. And the app right, the main app ultimately designed to help you improve your habits for sleep and feeling better during the day. So that was kind of the collective thing.

[00:32:17] Obviously on the B2B and even athletic side, you had the coach side of the business too, which is the analytics, right? Sports teams. You needed to have, the analytics are so important on how their, the coaches are making decisions on the nutrition are, and all the analytics and reporting stack was very interesting and very useful cause you'll, and very tricky cause you needed to be very privacy aware and privacy sensitive. You, one of our promises was, we will never tell your head coach that you didn't sleep last night, right? So that, you cannot do that because you would lose trust immediately. So then you would need to figure out how do you aggregate data and minimize it, summarize it in ways that makes it useful.

[00:33:06] Without sort of  pointing fingers at what's happening. So, that was kind of the whole kind of suite of, and I mean, for athletes you had kind of human coaches assigned to you. So an athlete could just message us and have a coach respond the same day on whatever problem that we're having.

[00:33:27] And you would, it's fun times working with high end athletes of all sort of problems you can have and how to, you have to be creative. Right? I remember this small anecdote. This quarterback that shall not be named. Pretty high level. He had a baby during the season.

[00:33:50] His wife had a baby I think on Wednesday night and he had a game on Sunday and it's, what do you do, right? If you watch the baby, you won't sleep for two nights and if you don't sleep, you're gonna lose the game. So, how do you help navigate that situation where it's not even a habit.

[00:34:09] It's, well, you don't play or you figure out how to, you move out of your house for a couple nights, and you're not with your wife and your baby. Eo even stuff that wasn't sort of…

[00:34:21] Nima Gardideh: And then they reached out to you about this problem?

[00:34:24] Leon Sasson: Yeah, they're like, what should I do for my sleep? And, we're like, well, you're just not gonna sleep. We should talk to your coach and figure out either you're gonna miss the game or you're just gonna move outta, you're gonna sleep somewhere else to make sure you get you at least sleep.

[00:34:38] So stuff like that somehow ended up in our plate. because, that stuff was fun. So you hadmore regular, obviously problems with people not able to fall asleep and kind of the more usual sleep guidance that we would do.

[00:34:54] Nima Gardideh: At this point you're like, Okay, we're gonna grab this exact service, cut out a few things. I assume you don't assign a single coach to every person anymore. And, so you've created a core version of that service and that offering and then you're like, I'm gonna take it to the consumer. 

[00:35:10] Leon Sasson: We cut out two major things, and that was the big bet. We cut out hardware and we cut out the human in the loop, which we had and we could afford, obviously would have to be higher prices we couldn't afford on the consumer side. So, that was the large bet. How do we make an experience that is still great, and is valuable for people?

[00:35:32] It helps you improve your habits, and you wanna use it without both of those things, The hardware and the human in the loop. So the hardware, there's a bunch to do there and that was a fun, a fun problem to solve, but. We take sort of a hardware agnostic point of view where if you have any wearable, we can connect to it. But if not, it also works fairly well. And it depends at what level of, kind of fidelity you want. 

[00:36:02] Nima Gardideh: That makes sense. So you went direct-to-consumer. So, okay, I also believe in this thing where you're trying to bet on very hard things in a company. And then one founder focuses on one bet and the other founder focuses on the other bet. So you said, I'm going to take what we built out and see if I can sell it directly to consumers using some trial subscription model.

[00:36:25] How did you structure the team at that point? What was the divide? Was it the same engineering team and you were just prioritizing things differently and putting some flag for this? This is the B2B version, this is the consumer version. Walk us through the structure both on the team side and then just the product side.

[00:36:44] Leon Sasson: I mean, at this point we're a small company. We're, I mean, we're still about the same size, so I think we were maybe 10 people total. So you don't have a lot of organizational structure problems to solve at that point. But, it's interesting we had even before that point, because we, on the B2B side, one of the problems of that business is that you cannot iterate on product fast enough.

[00:37:10] Because your volume is very low, right? Even if you onboard a new client every week, you're talking to a handful of people or dozen of people that you can learn from the product perspective. So one of the things we would do that I'm a huge believer on is we would still drive new cohorts of consumers through regular marketing channels.

[00:37:32] Very small, maybe 50 to a hundred people a week. Purely to test product and to know what's working, what's not, to change things in onboarding. So we already had that almost systematic approach for our product development cycle. We have an incredible product and designers.

[00:37:53] So we were designing a product that was consumer great, that didn't really have to shift, if you, we take a lot of pride in that work. So the end user experience felt consumer since day one. Because that was important for us and to ultimately create a good product for end users.

[00:38:11] And we had kind of a little bit of the mentality of how do you always figure out you having a volume of users to iterate on your core product value prop and your core product onboarding loops and all that stuff. So when we wanted to actually test it out, it was less about, Hey, how do we test out the core product?

[00:38:35] Because I don't think you, I don't think we could have changed core product value prop in a couple months, right? That's just really hard to do. You're talking about years of research and experience and  figure out what works and whatnot, and technology. So we couldn't really iterate on that as much, but we could iterate on, well, how do you monetize it?

[00:38:55] How do you onboard users and how do you market? Which are things that we've never really done before in a systematic way. And those things tend to be a little easier and faster to iterate on than saying, Oh, what is our core value prop? And we need to reinvent the product from scratch.

[00:39:13] So we did it at that point, thinking about consumer versus B2B, it's the same product. We just change how people get into it, how you onboard them and how they either do a subscription or free trial or this sort of  monetization mechanics that we had to, we had no experience and had to learn.

[00:39:32] But, luckily I think that's, it's one of my favorite things about just having conversations this and with other, in the VC world, just being able to share ideas with other founders where you can just get very smart very quickly with people working on similar domains, even though their core actual industry very different, right?

[00:39:54] I’m very close to someone that became a very close mentor and consultant for us. It was someone that worked on a music app, a music learning app and had nothing to do with sleep or health and fitness or anything. But turns out that onboarding best practices and marketing best practices are the same, because it's an app on the App Store and a lot of the similarities are the same.

[00:40:18] So you can get up to best practices very quickly, the infamous 20% effort gets you to 80% of the way. And you could do that very quickly when we were starting from literally zero. So, I remember it was a three month sprint trying to figure out how to scale up marketing to non-trivial levels. In the $10k-20k of revenue a week to try just seeing, I'm sorry, a month to try saying, All right, this is not a hundred dollars, right? There's something here that at least we can glance at and can get to millions in the future. And it was just fast iteration there. I think, on the team organization

[00:41:05] back to your initial question, it's kind of, I think being able to have some of the culture shipping fast, especially when a lot of the setup, it's very kind of infrastructure driven, where you need to just getting off the ground with consumer stuff. You just need really good instrumentation, really good analytics, really good, even onboarding, funnel activation, all that stuff is a lot of infrastructure work.

[00:41:37] And some of it is just kind of blocking and tackling and being able to just, being also a CTA, being able to move very quickly and knowing how that can be a high priority for the business, even if it's not a core value prop, right? We just need to instrument things better so we understand if marketing is working or not. 

[00:41:55] Nima Gardideh: So that came directly from you then. So you were, I need to understand these things, so we gotta build this stuff in there. It wasn't some marketing person or…. 

[00:42:05] Leon Sasson: Yeah, no, at this point we didn't even have a marketing person on the consumer side. That took us a bit to get there. We had a marketing person for B2B and a marketing person on the content side. We do a lot of sleep related and health related content. But not in any way kind of on the, why you would call  traditional user acquisition.

[00:42:26] That is probably more what we were trying to figure out to do, but again, through our network, we found a consultant that helped us just get the basic stuff done. What do you need to just even start testing out ads? Right? I had no clue. But there's not, it's not that complicated.

[00:42:43] Nima Gardideh: And then you were lucky and unlucky. I think you were lucky that you started before iOS 14.5 because there was a beauty in knowing when you have all the data and then you have to shift, right? So tell me about that shift. I think you already knew this world. You had just started spending maybe for a few months, I assume at that point. And then iOS 14.5 comes around and pulls a rock from underneath you. So tell me how, what went on, how prepared were you? 

[00:43:16] Leon Sasson: I still have nightmare of that. I still have nightmares. [Laughs] I remember when Apple announced our stuff, it was September, 2020. We had six months of non massive scale, but at that point we were, Right, we're gonna go full on B2C. We're gonna kill the B2B business, we're gonna kill it. We're gonna sunset all the existing customers there. And I was getting married that month and going on a short honeymoon in the middle of Covid and Apple announced, next week we're releasing SCAN and we're gonna break everything. And it was like, holy crap, Right on my honeymoon, Apple is gonna release iOS, 14.5 and break all marketing. Turns out they delayed it. 

[00:43:56] Nima Gardideh: Can explain what SCAN is just for the audience?

[00:43:59] Leon Sasson: Traditionally, every single iPhone had effectively an identifier that has a number. So my phone to every single app used to have a number that was the same across app. So no one knew it was my phone, but they knew that it was phone 321. So when you would have an ad on, let's say a Facebook property, so the Instagram app, Instagram knew that it was showing an ad to phone 321. And then you had us on the other side, the advertiser saying , Oh, the phone 321 started a trial. He's a great user, he's gonna, he made a purchase.

[00:44:39] So it's a good quality user that enjoys a product. So then Instagram could be very good at finding people that are similar to 321 that phone and Apple just single handedly kind of, for a bunch of reasons, kill that, what's called the IDFA. The IDFA, I don't even know what it stands for. The idea of every phone, effectively, and they replace it with something called SK Ad Network, the store kit advertising network where he tries creating some of these attribution technologies so you can still connect users to ads while preserving privacy. But now it's getting a little better.

[00:45:24] Two years in and the next version is gonna be better. But back then he was very, very unclear and he basically shut down what the last five years of how the marketing world worked, especially for apps, but also for e-commerce. He just kind of moved the clock back 10 years. Right. 

[00:45:44] Nima Gardideh: It really felt like they had not done their work.  I actually, the thesis was great. I was like, Okay, you're coming in here and you're making it better for consumers and their data's not going to be as prevalently easy to capture and things like that. But then, you looked at the APIs and you were like, Did you talk to a single marketer building this thing? Because it just looked so bad and so limiting just broke so many things. 

[00:46:10] Leon Sasson: Yeah. And it's the kind of thing that it's independent point of view, whether you wanna have the point of view of is it malice or is it incompetence? It's hard to know exactly which way it was. The fundamental point of view is very strong. And I'm confident they're gonna get it to a point that it's ultimately way better for the industry.

[00:46:30] But they could have done it in a way that it was actually more useful than what we had before. While fully perceiving privacy and not creating this sort of  cottage industry of data sharing and data piping. If they had, would've done it from the ground up, actually work well. I think now they're rolling out changes.

[00:46:55] And I don't know if you're gonna get into details, so they're rolling out changes over the next few months that hopefully it's gonna make it even better. And I think it's, It just be a win-win for everyone, both marketers, kind of user privacy. Cause it's really gonna, there's gonna be no need to find other complicated ways of doing attribution.

[00:47:13] If you can trust Apple Central Service that also preserves privacy, but still gives enough data to be able to find users that are gonna be good customers. 

[00:47:24] Nima Gardideh: There's only two areas of it right there is, do I know which of these campaigns and ads that I'm running are bringing the best users? nd then there's the other problem, which is am I giving these signals back to the network so they can do their own work there? I think they're pretty against that second one still. I don't think they're willing to give you enough signals on Facebook or Instagram. Even with the new one, unless you disagree with me, but I… 

[00:47:55] Leon Sasson: Yeah, no, I probably disagree with you there because I think those two things sound like two different things for people like me and maybe you, when we're on the advertising side, but they're actually the same thing, right? By doing the second you kind of need to solve for the first and getting visibility.

[00:48:15] And you cannot really solve the first without the second. You can try with things  make mix modeling and statistical analysis, but it ultimately doesn't work that well. 

[00:48:26] Nima Gardideh: It doesn't work for you and me. It works for the ad networks though. Facebook's gone way better over the last six months. 

[00:48:35] Leon Sasson: Yeah. So with the new SCAN, and SCAN is kind of the scan network. It's called 4.0, the fourth version. I think right now we're basically the release, release version two is when they really shut off everything. And then with scan four, they're finally giving us a bunch more tools. So they have this constraint of only 99 campaigns and basically, ultimately the network, so Facebook or Google have to do some sort of mapping between campaigns from what we think as campaigns as advertisers, and actual what they call campaigns to Apple. Just to try to find more signal and it's fun to explore that data set when you look at the raw data, how Facebook is doing it under the hood.

[00:49:27] But, that's basically expanding to campaign and creative. So effectively giving 99 slots of information, but giving, I believe, close to a thousand slots so you can have a lot more fidelity of what's working once not, you can break down by a lot more groups, which still it's gonna give you a signal.

[00:49:51] And then they're doing another thing, which was the other big problem that was the privacy threshold. So, if you don't hit the minimum number of insults per campaign per day, which by the way, it's a number that is indicated by Apple, which creates a lot of other pain cause no one knows what that number is.

[00:50:08] You get basically zero signal. You either get full signal of how many purchases or events that you make per campaign per day, or you get with the new system. Apple is introducing the, I forget the exact terminology, but it’s, so if you don't need the thresholds at the highest level, you're still gonna be able to get some data. But instead of having eight values, you are only gonna have,

[00:50:37] so, you can do high, medium, low value users instead of your more granular data. And I think combined with that, and I think all the work that networks are doing,  Facebook and everyone else, I think is gonna get a lot better and we're still gonna be able to have the signal that we need in many ways.

[00:50:59] It depends on Apple executing on their promise here and yeah, it's gonna be very interesting to see how, how that whole thing plays out. 

[00:51:08] I still think it creates, I couldn't have imagined doing what we had to do in 2020 when you had kind of full data attribution doing that now because you could, now you cannot test with small budgets, right? You need to be hitting, I don't know, now you need to get a hundred users a day minimum. Right? 

[00:51:37] For anything to test anything, right? So when we were trying to start out spending, I don't know, $500 a day, just felt like, holy crap, we're just throwing so much money out. Is it gonna work? Is it not? And now you just need to say, well, that you need to start there and you need to leave it on for seven days and even 14 days because the other side of scanning is that it has all this random timers and data delays and, and all that stuff. So it basically, I think, hires the bar where  you cannot, Yeah. If you're a small app developer, it becomes really hard to try testing marketing from scratch at small budgets. Like it he becomes… 

[00:52:20] Nima Gardideh: And when it came out, Facebook's rhetoric was, Hey, this is going to hurt small businesses and things like that. But no one understood why. And then so the press looking at it, well, Facebook is just trying to care about their revenue, I guess is gonna make it harder. But it's actually true.

[00:52:34] It's making it harder for small businesses to acquire customers. It's pretty bad. And we now have, we've always had some limits on ad spend, but we have it way higher for app companies. If you come to us and you wanna work with us, well, if you're not spending a few hundred thousand dollars a month, you're stuck in this world where I can't actually know what's working.

[00:52:55] So, I can't possibly help you just because the structure of our company is such that we care so much about the inputs and the outputs of the work and we were much less sitting around thinking that this could work. We're literally looking at the data every day. So, this structure of our process doesn't work and it made me sad because, we started in startups and working super early stage companies. 

[00:53:22] Leon Sasson: And it's probably getting a lot better a little better now that privacy thresholds are gonna be more flexible with SCAN4 for. 

[00:53:28] Nima Gardideh: And also, Facebook's gotten better at their own prediction, right? So at the beginning it was pretty bad and then Facebook changed their models and you don't need the 50 conversions a week on each asset anymore. On mobile, they figured out how to do with 20. So that's actually…. 

[00:53:43] Leon Sasson: Yeah. It' way better. And I think me being, CTO that I had this, , we had to solve the marketing problem, but, , and we didn't have a marketer back then, so he was , All right, someone's gotta do it. , I happen to enjoy that stuff, so I, I'll take it on and figure it out because I'm, I really believe I'm  co-founder first, , and  CTO second as sort of my job, so to speak.

[00:54:09] Meaning, company priorities are way more important than whatever data they actual job title I have. , and being able to just, During that time it was so unclear,  what's changing, what's not, what the APIs and the SDKs are changing, even going to the source of reading the Apple doc, trying to figure out what changes, reading the Facebook doc. So I think that really shortened the loop because I could do a lot of those changes myself. I could go in and, and update the 

[00:54:43] SDK and see what would happen. And I think one of the advantages of happens that I was doing the marketing side and I'm technical and I knew the engineering side pretty well, so we didn't have to have all these delays in trying to get scan working.

[00:55:01] And I remember when we first went live I think in March of 2021, cause Apple deleted all that stuff. But it wasn't enforced until June. So that period of March through June, For some reason it was incredibly good. We think that Facebook had their own inventory as they were testing out the system. And because most advertisers hadn't put the new scan SDKs on we were getting really good performance. 

[00:55:32] Nima Gardideh: It was a different auction before everyone came up. 

[00:55:35] Leon Sasson: Being able to just be, as soon SCAN launched we had to get ready. Cause we did, we didn't have to be like, I knew it was generally easy to do it myself. We didn't have to go through our usual product development cycle that's a lot more structured and we have sprints and we have plans and we don't wanna be hijacking engineering time.

[00:55:53] Nima Gardideh: So what you're saying is that you commit code to production without going through process? That's great. I'm sure your engineers love hearing that. [Laughs]

[00:55:59] Leon Sasson: No, you still have process. But, yeah, the flexibility is important. I think that's why, I'm still trying to figure out what's next, how do you keep scaling teams? 

[00:56:13] Like the next layer. And something that is becoming a lot more common is having sort of marketing analytics engineering or marketing ops, sub technical that is embedded within marketing to help facilitate all this stuff.

[00:56:28] And because it just takes a lot of cycles to figure out how everything connects to everything and some things are really minor change and some things aren't. So the minor change you can just get done quickly and you don't need to go through  your usual sort of longer term prioritization process that usually most part things are gonna have. 

[00:56:52] Nima Gardideh: Yeah, most product instrumentation things are very fast and most teams run it through the same reg, sort of prioritization loop, and it doesn't make sense to me.

[00:57:01] Leon Sasson: And it's a balance because you also don't wanna be distracting engineers every single day. Hey, can you make this small change and release? And can you make that small change, that's also very improductive. So trying to figure out what the path forward is is gonna be very interesting. I think it should really be someone  an analytics engineer or an analytics ops person being embedded in marketing at some point.

[00:57:27] Nima Gardideh: Yeah, I think a growth engineer is what I've called it in the past, where it's an engineering minded person is they're trying to get very good at growth in general, kind of what you ended up having to do, right? You're an engineer, you think growth problems are interesting and so you're willing to sort of spend the time learning all these systems.

[00:57:45] And in this scenario, you're the founder of the company, so you pretty much care about growth. But I think if I didn't have the founder part in my head, you could have probably sold me early on in being  a very good growth engineer. Andrew Chen and I spoke a lot about this, and at some point he tried,

[00:58:07] I tried to join his company actually. He thought I was not enough a good enough engineer, which was great. [Laughs] I ended up doing product management for years after that. And I was a little scared of San Francisco, so I remember going and interviewing with his team. He was doing some, I would say an intellectual exercise of how to create growth loops that are not related to the product themselves.

[00:58:33] So they had launched a series of products that had millions of users, but had awful churn just to show that they're able to sort of grow products very well and I was entering with their team and remember thinking, this is just not a city I want to be in yet and I was very scared of it.

[00:58:50] I was very young. I was  19 at this point. But I think you could have sold me on, if the growth engineering paradigm that exists now. I would've been happy being in  different companies doing that type of work. And I think more and more you'll see them being separate teams, having their own  discipline and caring about things. I tend to think of our team where we have engineers and, and data scientists and I dall those roles, growth engineering roles when I was hired.

[00:59:22] Leon Sasson: Yeah. No, it's very interesting because when we grew a little more last, we raised our series A beginning of last year, so we needed to add a few more people and it was time to split our kind of product pods into two and it's always strict of how do you go through a split, right?

[00:59:44] You can, there's a lot of different playbooks you can run, but we ended up sort of choosing, right? We're gonna have a growth team. A growth product team and a product team sort of in the more modern sense. Maybe we are Marty Cagan and kind of “Empowered” approach where you really have a PM designer and whatever functional engineers you need on that team to be empowered and autonomous, right?

[01:00:10] The team can set the priorities and they can execute on everything without needing anyone else. You can have a team that doesn't have the functional skills to execute on their priorities. Right. Which is very common where you have teams that like…. 

[01:00:22] Nima Gardideh: This is the Spotify pod model as well. Yeah. 

[01:00:25] Leon Sasson: So we split into growth and retention kind of core product work and that helps. That certainly is part of it. Growth is in charge of, tends to be a lot of onboarding related acquisition loops, monetization, a lot of that work. You still end up having….

[01:00:45] Nima Gardideh: Are there marketers on that team or is it just shipping things?

[01:00:47] Leon Sasson: No, it's still primarily shipping things in our core product, in a core app. Our main product as a company is an app that you can download in the app stores, and they're primarily shipping product, software on that app, sometimes on the website, but that's more rare.

[01:01:03] And, it doesn't bleed into marketing that much. And I think that's something that we're still trying to figure out what the right move is. We're not big enough to have that dedicated engineering, supporting marketing. So the growth team usually is gonna jump in and help. You have a lot of just instrumentation

[01:01:24] even if you go outside of pure advertising, you go through analytics and email marketing, and you wanna have the right events. And then even all the revenue stack that goes, connects to the app store, revenue hooks, and the stripe and whatever billing system you use. How do you wrangle all of that?

[01:01:45] We tend to have that under the growth team, but it's something that is not obvious. And as you grow, many companies end up splitting that into their own pods with ownership on that. But, it's gonna be interesting…

[01:01:59] Nima Gardideh: What's their, so I guess, two questions. One, do you use some OKR sort of apparatus to run things? Okay. And then what does that team's OKRs look like? 

[01:02:12] Leon Sasson: Yeah. By and large, you're gonna have things improve trial start, which is a percent of people that start a free trial or kind of revenue per user in many ways. But a lot of the kind of hidden things about the OKR systems and some of the flaws is that you do have a lot of kind of business as usual things of you will never have a clear, hey, Figure out why scan data is going through as part of your OKRs.

[01:02:45] Maybe that's a marketing OKR. So we tried having shared OKRs between that team and marketing, and it works to some extent. But yeah, by and large, a team is sort of on things once you get a user in the app, how do you get them to realize the value and be convinced enough to start a free trial?

[01:03:03] So, we tend to ship them every few months. It can be more focused on the conversion or on a different user segment, but for the most part it tend to be around that angle. Trial, start revenue, that sort of stuff. 

[01:03:17] Nima Gardideh:  Is that, is the rate of those things or is the volume of those things, are you saying you need to get 10,000 trial starts, or you're talking about the conversion rate, from trial to subscription?

[01:03:28] Leon Sasson: Conversion rate, right? It needs to be conversion rate because one of the challenges of that team, it's not accountable for the marketing budget, right?  When marketing performance went down and we had to cut budgets or scale down volume, it's not that team's fault in a way, right? The team and the way we measure most of these OKRs need to be on all on A/B testing, right on.

[01:03:54] It cannot only be, oh, blended trial star rates moved from five to 10%, or whatever metric it is. It needs to be, because the marketing channel change tends to affect your core product numbers, a lot, especially in times like this that you have very drastic changes in the underlying, sort of which research you were able to talk to.

[01:04:17] So everything needs to be about the rates, right? Which is back to probability.  What's the probability of someone going into the user or what's the expected value of a user when they sign up the app and can use it systematically. Do the right, prove with A/B testing at some level of significance that you're increasing that problem of your time.And if done well, you should then see global, actually fully aggregated blend the numbers going in the right direction too. But sometimes they don’t. 

[01:04:49] Nima Gardideh: Yeah, Let's talk about that exact thing you just said. Sometimes they don't. Right. And , it's because of the combination, the combination part is the hard part to get done, right?

[01:05:00] The marketing team could be bringing low quality people and the growth team could be doing all they can to increase the conversion rate, but the quality of traffic is low and does the marketing team, how will you judge them? What are their OKRs? Are they caring about ultimate revenue or are you saying, I want these many installs? How do you talk to that team? 

[01:05:21] Leon Sasson: Yeah, I mean, marketing, and right now it's a very small team and I'm still heavily involved. 

[01:05:26] Nima Gardideh: Is it you and an agency? 

[01:05:28] Leon Sasson: And one person, right? So it's, by no means, we have this sort of  organizational bureaucracy how to communicate important, but you're not going through levels and levels of goal setting here.

[01:05:39] It's, I mean, revenue and marketing efficiency, which is back to the company kind of cash flow financial, what do we need from the business viability point of view? And I think, we sometimes, after doing it for a bit, you start realizing what the constraints are and what levers we have.

[01:06:02] Things looking at very basic segmentation aspects that the marketing team can have visibility into, right? For us, it's the age, right? Age matters a lot for monetization for us, right? And it tends to be very easy for marketing to drive more traffic of, let's say, way younger users that tend to be “cheaper” to acquire.

[01:06:32] But they don't convert. So very quickly you start having all right, different targets for different age and try sort of  saying, we really don't want to target users under X years old unless it's extremely kind of below this levels, way cheaper than, kind of more. 

[01:06:55] Nima Gardideh: Oh yeah. So, the economics of that cohort doesn't just make sense. So you have to make it make sense or stop doing it. 

[01:07:00] Leon Sasson: That's harder to trickle into OKR in a systematic way, so to speak. But, we're talking about it all the time and we're sort of working with a growth team. And I think your flexibility is key to be, all right, we realize that insight, that segmentation factor matters a lot. How do we run a product sprint to try moving one of those cohorts the other direction?

[01:07:22] Or should we try making the users are on converting, well convert more or should you just try ignoring them and you go through an exercise, Right, first let's try so, and we can move their conversion. 

[01:07:34] Nima Gardideh: Oh, interesting. I would've gone, let's just ignore them.

[01:07:37] Leon Sasson: Yeah. unless there's a cheap way you run through, Right. Is there a way to make this user available? Cause they have, let's say great retention or all these good properties but they just don't have a credit card or whatever that is. So, I think that's where the rigidity of OKR is getting in the way and it's good for high level company business goals, but sometimes the flexibility of iterating on sort of at the weak level is where it really kind of shines and having the same talking a lot I think is important.

[01:08:09] Nima Gardideh: I've done OKRs now twice at different companies. We don't do them at Pearmill because we go through a different process. But the thing that I like about OKRs, and I think he talks about this, if you read “High Output Management” and his growth book, or he reveals OKRs in there and then later on he talked about it, right?

[01:08:29] He talks about how it's basically the process in itself that's useful, not the thing at the end, the process of going through who owns what, what are we trying to get done, what are the overall goals, how are we gonna track these things? That's what actually matters. The rest is kind of, okay, you can keep it as a tracking system, but it's not that. But getting people to go through the ringer of being introspective about their role in the company, getting ownership assigned, that's what actually matters, 

[01:09:00] Leon Sasson: No, I fully agree. And we still do it. I still struggle with part of the framework, for example, where I think for kind of true product innovation and new product development, OKRs is a bad system. Because often you don't even know what metric. Yeah, you're moving revenue at some point, but  there's miles and miles of stuff between revenue and what is the product we're building and to what user segment and what audience, right? Those things become a lot and there's ways in to account it, but it just becomes cumbersome where hacking a system that was created for these is  companies that were scaling super fast, it's a different world. 

[01:09:42] Leon Sasson: And finding the right level of metric is tricky, right? Where if it's too high level, move retention or move revenue, it becomes a little less useful because your actions cannot move the metric at all. At least rarely, until you actually know the levers very well and you're very clearly scaling, right? But you also don't wanna go too detailed, too low level in the metrics because then you're, I don't know, optimizing the people that sign up for feature because you think that's highly correlated to everything else.

[01:10:17] But then it's, well what if you discover because you're still early? That feature wise is not the main priority. Do you just scrap everything? So, finding the right level of metric is probably half of the job of trying, having good goal setting and I think that could be pretty hard.

[01:10:37] So what we still do them, I find the planning process is very useful to at least know the end stage and what could be. But, our product process is sometimes a little detached from the OKR and we run our own product cycles that ultimately, generally kind of fold into it, but it's not exactly the same.

[01:11:00] Nima Gardideh: It's a one-to-one. What's the cadence? How often are you doing it? 

[01:11:03] Leon Sasson: OKRs quarterly and our product cycles are quarterly, but we do six week cycles during the quarter. So, two of those cycles per quarter seem to be pretty healthy. And in six weeks, you can generally have enough time to push a big rock. So,  that amount of time to set sort of the bets that we wanna place for.

[01:11:34] We think a lot about our bets. Which bet we're putting, how much appetite we have for a given bet and that getting sense to work fairly well. That doesn't work that well for marketing, right? You need to figure out analytics is right. It might be too slow. So, that's kind of a, depending the needs for marketing.

[01:11:54] Nima Gardideh: So are you launching a new app version every six weeks? 

[01:11:56] Leon Sasson: Oh no, we're launching a version I mean, couple times a week maybe. Maybe once a week. Yeah, yeah, definitely more.

[01:12:03] Nima Gardideh: It's not your launch cycles, it's more how you prioritize?

[01:12:06] Leon Sasson: It's how you prioritize and what are the larger projects you're working on. Right. And that's where you also have, sometimes you're gonna have discrepancies between the kind of growth and core product or retention teams where growth teams can often place smaller bets that are more kind of encapsulated.

[01:12:26] If you're doing a pricing test where you're doing a new subscription type or kinda these things that may be or kind of an improvement in the onboarding flow to communicate more clearly, the set of features that we offer or the set of problems that this is solving for a user, you might be able to get those done a lot faster.

[01:12:50] So, that the frequency sometimes does matter between the teams in terms of one team maybe focus on two large things, whereas the other one maybe focus on one large thematic project that has six different components or tests that can be released in isolation and gathering, kind of learning more quickly than just waiting six weeks.

[01:13:15] So, it's, and that goes back to the volume and the statistics of things. When you're testing the higher and the funnel you tend to be able to, because your volume is gonna be higher. Or you tend to be able to get statistical significance on things faster and it's not always true. Sometimes you have to wait until you get…

[01:13:37] Nima Gardideh: Well, if it depends on what the goal is, right? If you're trying to still out retention based on an upper funnel change.

[01:13:43] Leon Sasson: Yeah, exactly. You still have to wait unless you have a very good proxy metric for alonger, but you still have to wait. But, generally speaking, the higher in the funnel, you're gonna have more users. So you can detect whether a change you made is actually impactful or not, way faster. And as you scale, that amount of time tends to get fast, shorter to, so it becomes a nice kind of feedback loop where, it's nice, it can be dangerous. I think I've been playing with idea of having kind of hurdle rates for our investment, just kind of in the finance world that you have. Minimum hurdle rates for investments and kind of financial projects.

[01:14:31] Wow to have that in product teams where really anything that doesn't move numbers, I don't know, less than 5%, it doesn't matter, right? Cause you can spend a lot of time optimizing things one, 2% when you need to be working on things that are gonna move things 20% right.

[01:14:48] Nima Gardideh: And, in your stage, yes. I think there is a point in time where you are just caring about the one to 5% improvements in a product cycle but you're probably not there yet. 

[01:14:58] Leon Sasson: Yeah, at some point. Yeah. And it's interesting because ultimately you can get that 20% by stacking a lot of 1% changes. But, it can also become kind of a large operational burden on a small team when you're trying to do 10 tests and each of them trying to get significance. 

[01:15:18] Nima Gardideh: And you're at best gonna find a local maximum when you're doing that, right? You're not going to find the top…

[01:15:22] Leon Sasson: I don't think that process works well for sort of the more innovation new product work. I think. In fact, I think it works pretty terrible for that. And you A/B test is never gonna give you true insight into what you should be doing, Right? It's gonna, I view it more as it helps validate a hypothesis of what people want or when you're unsure the level of impact it can have, but it's not directly what gonna actually tell you what you should do.

[01:15:58] You need to, one, talk to customers, which we do a lot, right? Actually getting Zoom meetings and surveys and everything to actually understand which people, what are they doing? That they find you're available or not and what are the obstacles they have and circumstances in their way of either not realizing the value in your product or realizing it.

[01:16:23] And so you need that qualitative work to actually figure out what to do. And then A/B test really, I think, useful to then figure out if your hypothesis is correct, right? Is it, is, that's harder on the, on more product, I'm sorry, growth level kind of optimization work, where very little  qualitative work is gonna be really hard to figure out if a seven day free trial or a 14 day free trial is better. Right?

[01:17:00]I mean, you can have your hypothesis, oh, people want a longer trial because, it takes a bit to change your habits and you wanna have the time to really play with the product and understand it. And, one day is too little. Seven is the middle. 14 is so you can have right, why you think is right, but then until you have enough people and in a volume, you just won't know. Because it is kind of fundamentally a quantitative question where your answer is a number in many ways and the qualitative feedback doesn't help that much for that small improvement. 

[01:17:38] Nima Gardideh: Yeah. You're not changing the core of the product. It's just how you're presenting it. It's a very different portion of the thing. I have one last question for you, and maybe we won't have time to really cover it, but you talked about the hurdle rate, what do you use as a framework of prioritization right now? How does it work in your head and then how does it operationalize? 

[01:18:01] Leon Sasson: One of the questions that's one of the hardest and I feel it's always changing. You'll have your basic kind of product management ideas of impact, reach, estimated effort. Often we try refining things, so what's actually the metric that matters for the business.

[01:18:22] In the last six months, marketing efficiency and payback has taken a lot of focus from the kind of general business world. We need to get the business in a more kind of cash efficient way. Maybe grow a little slower, but more efficiently. So we're even trying to map things, All right, can we back out? Project estimated impact straight up to marketing to efficiency at the business level, at the payback or revenue level. 

[01:19:00] Nima Gardideh: How are you doing that? Net present value stuff? What are you doing there? 

[01:19:03] Leon Sasson: Not quite, but we are trying to map most things to how does it change our month one or month six in economics to at least make a decision on impact. There are things that you cannot map it that clearly. I'm saying, a large kind of fundamental product better on a totally new user segment is gonna be hard to map.

[01:19:28] Cause it might be more long term, but something in the shorter term can definitely should be mapped. And you can very quickly understand, all right, there's no way that thing gets you more than 5% here. Let's not even consider that. So at least that tends to encourage conversation of, all right, maybe we're debating between these three larger projects that we think under, kind of back of the envelope assumptions can, if things pan out properly, give us a 10% boost. So then you become, that's what kind of the art comes in and be right, which one do we think is lower effort? Which one maybe is more aligned with long term plans?

[01:20:06] And kind of just going through that and I think that's a process that our product leads focus on with a lot of input from myself and my co-founder. From that we try bringing more of the business level to objective financial objectives. Our product leads are gonna be a lot closer to what users actually want and need every single day. So trying to really bridge between the two. So I don't think that gives you the answer that you wanna know. This is the magical priority decision and exercise. 

[01:20:40] Nima Gardideh: No. Okay. Actually this is exactly what I wanted because I think people write about magical versions of these and…

[01:20:46] Leon Sasson: I mean, it's not gonna be realistic when you go and sometimes you want, hey, this is a spreadsheet and you have this perfect weighted estimation of how much effort and time something's gonna take and the impact, and you have confidence bar, and then it's, all right, PM and maybe an analytics data science person spent two weeks working on this stuff and  half of it is just total guesses.

[01:21:10] So it just hard. So some of that work is useful, right? You should have a point of view on, all right, if you launch that feature, do you think you're gonna get 10% of people to use it? Or 80% of people, right? If you don't even  now where, which order of magnitude you are, all right, that's a problem. But, you just don't know if it's gonna be 10 or 20%. and that's right there… 

[01:21:32] Nima Gardideh: Yeah, and I think it comes down to what we were talking about earlier, the process of going through that type of prioritization is what actually matters. It's, you're gonna be very wrong about the numbers, but you go through the rigor of actually evaluating the ideas. And sometimes they're quite obvious, Oh, of course we have to do this one.

[01:21:52] Or you said, there's three that we have, some level of conviction that are gonna be good. And now let's talk about which, which one or the three we're gonna do first. But it's the process of caring about priority that actually matters. It's similar to the OKR problem, it's not the the outcome itself that matters, it's the process that you go through to vet the ideas thoroughly before you put effort, money, and engineering and product effort behind building them out and launching them. 

[01:22:23] Leon Sasson: And I think one of the things that I love. Our team does is that we tend to cluster them by kind of thematic or strategic angle. So you have sort of, you tend to have, alright, let's say things around or organic referral loops are one theme, right? How do you get people to share word of mouth referral loops?

[01:22:47] Or you have more kind of pure pricing optimization and pricing testing and pricing mix, right? Where you wanna have a point of view at. So when you're discussing priorities, are we in a mode of just doubling down on one theme and placing all of our bets on one theme? 

[01:23:07] Or do we wanna say, Hey, this cycle we wanna have one bet, one project in each of the three teams that we know over the long term we have to work on and I don't think there's the right answer because you can just back out the numbers anyway. But sometimes what we have a sentence that organic referral loops are very, very impactful. We have good learnings there. We're gonna spend, because we're gonna be in the head space, we're gonna place a lot of bets here.

[01:23:34] So you can think about the, kind of exploit or explore different themes and I think, that ends up being a useful conversation to know, are we doubling down on a lot of similar projects because it's looking so fruitful? Or are we just gathering more information about different, totally different themes so the next cycle maybe you double down on one. So that's a level that sometimes comes into play and I find it useful to think about. 

[01:24:01] Nima Gardideh: This is why also sometimes you need different levels of people thinking about these things because you're gonna think at a high level. And that's literally  you're thinking about a portfolio of bets, where maybe your marketer is thinking about, I just need to move this number right now, so let me run these five tests around pricing or whatever.

[01:24:19] And that might be an interesting thing of having other people look at this whole apparatus from time to time or having people step back and think through the problem from scratch over and over again is very helpful and trying to get the right bets in place. 

[01:24:37] Leon, thank you so much. I know we went a little over. Thank you so much for doing this. This is super fun. I probably have you on again every few months just because I think it'll be really interesting to see the growth. [Laughs]

[01:24:50] Leon Sasson: Yeah, I feel we haven't even started talking about the marketing side, but maybe that can be next time. There's a lot of fun stuff everywhere. But, thanks for having me. It's been a fun chat and good to know that you have your electronics and, and physics background over there. 

[01:25:07] Nima Gardideh: Oh yeah. Super into it. I build a Burning Man camp, I was telling you earlier every year. And the hardware part of my brain loves that because we build solar panels there and I have our own lithion battery apparatus and it's super fun.

[01:25:23] Leon Sasson: That's fun, I love it. 

[01:25:28] Nima Gardideh: So glad to have you man. Thank you so much. 

[01:25:29] Leon Sasson: Yeah. Thanks so much. 

[01:25:31] Nima Gardideh: All right. And that's a wrap. I'm so grateful for Leon and the time you spent with me on the pod. We've actually gone back and forth a little bit more about economics since we recorded a pod. He's quite a curious man] and it's been really fun to get to know him better. Follow and subscribe on whatever platform you're on and listening to this show. We're trying to increase our footprint a little bit just to be able to get a better sense of how we can modify these and make them better.

[01:26:08] So if you have feedback, please send them to nima@pearmill.com. I'm trying to make these as useful as possible. I'm obviously very curious, [laughs] and spend some time talking to these founders about their past and trying to understand them and sort of building this thing for myself. But, I'd love to get feedback on it and get better.

[01:26:31] Producing a type of content that you all want to watch and listen to. On our next episode, I'm bringing on my co-founder to talk about our overall growth process. So if you are running a growth team or starting one in your company or founder, it would be quite useful for you to go through that.

[01:26:49] So I'm pretty excited about that episode. Subscribing, you should be able to get it the next week or so. And we're gonna go through our general process and the standards in which we enforce our team to follow. Anyway, thank you so much for listening. This was the Hypergrowth Experience. Have a good one.

[MUSIC FADES OUT] 

‍