Building Futures: AI, Careers & the Rails Ahead with Avi Flombaum

Valentino Stoll (00:01)
Hey everybody, welcome to another episode of the Ruby AI podcast. I'm one of your hosts today, Valentino Stoll, and joined by Joe.

Joe Leo (00:10)
Hey everybody, I'm Joe Leo. I'm the other host and we are both here joined by Avi Flombaum. Welcome Avi.

Avi Flombaum (00:18)
Hey, thank you for having me. Yeah, excited to talk.

Joe Leo (00:21)
And thank you

for coming in ⁓ an equal to this episode ⁓ flannel shirt. I am wearing mine as well. ⁓ The denim is looking okay up, but the t-shirt is poking out from the... ⁓ But anyway, and Evil Martians, and even bringing a competitor t-shirt. ⁓

Valentino Stoll (00:39)
I have my evil Martians fanfare on.

Joe Leo (00:47)
into this podcast. I don't know how I feel about that. I know I'm not supposed to be representing deaf method while I'm here, but you know, I kind of always am. It's kind of my thing. But that's OK. We love evil Martians and everybody had evil Martians on this show. ⁓ So I'll be welcome. ⁓ So we're curious. I guess I'm curious right at the top. You are the founder of Flatiron School.

It had this great run of success that was started in 2012. Is that correct? And, um, and now you're a few years removed from it. And I guess my, like the first thing I want to know is what is it that from that, like, what, what do you think that that has said about you and about your career and what do you want it to say about you and, your career at this point?

Avi Flombaum (01:42)
Yeah.

I'm not sure what I wanted to say or what it says about me, but ⁓ I will tell you sort of how I ended up starting it. ⁓ I'd never taken a programming class in my life and I dropped out of college. ⁓ And it was very funny to find myself like teaching people how to code and starting a school. Cause it's the last thing that I think I expected for myself in my career. But you know, in 2012, 2011, know, New York city was still recovering from the financial crisis and you know, people were

Joe Leo (02:06)
Hmm.

Avi Flombaum (02:15)
out of jobs and kind of not able to find like, you know, some of their careers just were downsized, you know, tremendously. And my friend started Skillshare and needed someone to teach programming classes. And, you know, I was I just left my first startup and I started teaching them. And, you know, people really loved them. And I had this one student that took it super seriously and, you know, I was offered a job at a company at and ⁓

I turned it down, but I told them that there was this one guy I've been teaching for the last like six weeks and you he's obviously not, you know, a staff level engineer, but he's going to be a great junior and you should interview him. And they interviewed him and like four weeks later, I get an email from him that he got the job and he's making twice as much money and he's got like this whole career he sees in front of him and can support his family in ways he never could imagine. And I changed his life. And I'm sitting there looking at this email and I'm like, this is insane. You know, like one of the reasons I

Joe Leo (03:07)
Hmm.

Avi Flombaum (03:12)
Loved programming and got into startups because I wanted to make like a dent in the universe. I wanted to create some sort of value and meaning. And I never expected it to be through teaching. Like I always thought it was going to be through building stuff and starting companies. But it seems like I could change people's lives teaching people how to code. And I got really into it. And I started learning a lot about how to teach and pedagogy and mentoring students and getting them jobs one at a time.

for around like six, seven months. And one of the people that took my classes was a venture capitalist. And after the class, he emailed me and was like, you know, let's get coffee. And he was like, why are you doing this? Like, you you could be like the CTO of a company. And I said, you know, told them that story. And he was like, let's just start a school that just does that. That just gets people jobs. And I didn't know if it was going to work. You know, I thought it was an interesting thing to try. And I met Sharif from Dev Bootcamp.

Joe Leo (04:00)
Hmm.

Avi Flombaum (04:08)
that weekend. He was thinking about starting that. And we both sort of came up with this format of what the boot camps ended up looking like. And I just ran with it. And, know, there was so much skepticism. Like when I was telling people that was going to teach people how to code in 12 weeks, you know, I have a pretty good network in New York. I run NYC on Rails, which is one of the largest Rails meetups. And, you everyone I was talking to was like, I'm never going to hire someone that's only been coding for 12 weeks. And I just thought,

Joe Leo (04:09)
Yep.

Avi Flombaum (04:38)
They would, you know, like I saw what you could learn in 12 weeks and why not try it? And it worked, you know, it took me like a month to get the first 20 students jobs and you know, the second cohort, the same thing. And then as these people were performing really well, you know, it became a thing ⁓ and it just grew and grew and I really loved it. You know, it's an amazing feeling to watch people learn and change their lives and

That's sort of how it snowballs, you know? And I'm proud of it. Like, it's the best thing I ever did in my life. You know, I hope I get to do something that's impactful. So, and yeah, I mean, there's over like 10,000 grads that have gotten jobs. It's not, you know, a small amount. you know, especially...

Joe Leo (05:20)
Yeah. Yeah.

And some deaf method, current and alumni came from flat art school, we're proud to say. Yeah.

Avi Flombaum (05:31)
Yeah. ⁓ so yeah, you know, I think that the skepticism getting over that was really interesting. ⁓ it took, you know, really like two years for this to become accepted. And then, you know, probably around like four or five years in, there was like such a proliferation of boot camps of varying quality that the bootcamp grad sort of got tarnished. But luckily we had really established ourselves as like at least one of the tops. So it wasn't.

Joe Leo (05:49)
Mm-hmm.

Avi Flombaum (05:59)
that bad for us. And we were also never like scaling, you know, like there were boot camps that were enrolling thousands of people doing essentially MOOCs and, you know, we still had like a very high bar for admissions. So we, you we never wanted to enroll someone that we didn't think could learn it in this timeframe in this format. So yeah, you know, then after like, I guess eight years I left, it was COVID had been a long, it had been a good run and I just needed a break.

Joe Leo (06:12)
Mm-hmm.

Mm.

Avi Flombaum (06:29)
But yeah, I know that's been my career. And I've gotten out of education now for a variety of reasons. It's a hard industry. And then also the market has very much changed. But it's interesting. There was like six months ago, I was really thinking about this question of what would a modern boot camp in the AI world look like? And one of the things you hear is that

Joe Leo (06:54)
Mm-hmm.

Avi Flombaum (07:00)
There's no, are no careers for juniors. Like no one is, no one is hiring entry level devs because you know, ⁓ they're going, they're going to be as good as the most modern models. And, know, if I can get a model to do it, why should I get a junior? ⁓ and I think that's just, I think that's such a funny thing to say because these ideas, these words like junior, you know, staff level senior, they're measurements, they're arbitrary, right? So, ⁓ you know, I think what's it.

Joe Leo (07:13)
Mm-hmm.

Mm-hmm.

Avi Flombaum (07:28)
I think what it takes today to be an entry level developer is a way higher bar than it was six years ago before AI. I think you have to know a lot more to qualify for being an entry. ⁓ But that doesn't mean it's not possible. It just means that we have to define what would someone hire if the person had no experience?

Joe Leo (07:55)
Well, you have

you bring up something interesting there with.

you know, with the bar for whatever we decide collectively, but sort of arbitrarily as junior, right? Like what is, what does that really mean? Now, speaking of somebody who just hired a junior developer a month ago, I also think it's ridiculous to say that, nobody's going to hire juniors. My personal thought, and I'd like to get your opinion on this, is that the, the junior engineers, whether they be young people out of college or people that are making a career change, ⁓ actually have,

the most incentive to sort of to ⁓ plunge into AI and use it to inform their development and leverage it in their development ⁓ because they are just learning and because they have no ego to get over. They have no ⁓ habits to unlearn. Right. And so to me, that feels like an advantage. And of course, this is a very, very small

cohort here at a deaf method, it seems to have been proven out. And I'm curious to know what that looks like ⁓ from your perspective. And as somebody who is, you you're writing about this and you are also thinking about what an AI kind of, know, curriculum for lack of a better word would actually look like.

Avi Flombaum (09:16)
Yeah. ⁓ so I guess the first thing that I think about is, ⁓ like the workflow, like having a really nailed down AI workflow is non-trivial. You know, like I think about like Kirian at every, ⁓ and his AI workflow is just so beyond mine, you know, and I invest a lot of time into how, how I'm going to use AI, how I'm going to leverage it. Like, how do I get it to, you know, not write slops?

Joe Leo (09:26)
Mm-hmm.

Avi Flombaum (09:44)
And how do I create a review process and a testing process that's efficient? So if I was starting out in code and looking to break into it, the first thing I would really think about is not necessarily the depth of my programming knowledge, but rather my workflow. Because if I can demonstrate that I can very much leverage these tools and at least not get slop at my level, I think that's a really, that's a competitive advantage.

I think of AI as really a force multiplier. So if you're like a 10x engineer, AI is going to improve your your efficiency, let's say by 50x. So whatever level engineer you are, it's going to be that level times 50. So first, if you're a zero level engineer, you just don't know how to code at all. You're just totally vibing it. I don't think you're getting a lot of gain from AI. So if I was at least a two or three,

Joe Leo (10:12)
Mm-hmm.

Mm-hmm.

Avi Flombaum (10:41)
I think you're actually, as long as you're really leveraging AI correctly, you're still getting that force multiplier, which is going to make you look better than devs that aren't leveraging AI. And on the last team I managed, there were six devs and two of them just refused to really invest in the AI workflow. Like they were, they were still using, you know, cursor on like spotlight edits, right? Like, and then writing a lot of code.

I refuse to write code by hand. I don't care how trivial it is. I'm prompting it. ⁓ so I think there's a large cohort of devs that, you know, are stuck in their old ways. And I think that unfortunately they're going to go the way of the dinosaur. Like I think about the devs I knew that would refuse to embrace the web as a platform. Just refuse to think that like, you know, writing web apps was a real thing.

of their caliber and they stuck to the compiled Java desktop apps. Those guys are done. They're just out. So I do think there's going to be a cohort of like really senior experienced devs that just won't embrace this workflow. It's just too much for them to adapt. I do think if you're trying to break into this, you got to really, you can't be one-shotting stuff. Your prompts cannot be these vague end-to-end

Joe Leo (11:40)
Mm-hmm.

Avi Flombaum (12:06)
like descriptions of what you want, right? You've got to have a sense of what good code looks like and have read a lot of code and be able to prompt in very specific ways. Like I want a class that does this, right? That works like that, right? That has these associations, right? As opposed to saying like, build me a login system. I would say something like, you know, create a user class, right? Or use the Rails authentication generator, right? ⁓

Things like that I think are really important for, and then also just having like, know, did I write, did I have the AI write tests, right? Did I make the pull requests small enough so it could actually be reviewed, right? Am I writing, you know, granular commits? I think those are really important things. I think any code you generate needs to be read. Like you need to know like the depth of code, even if you're not writing it by hand. And again,

One way I learned and what I love doing is reading code. Like I love reading, you know, when 37 signals like open-sourced right book, you know, and I bought campfire. I never intended to use those as apps. I just wanted to know what those, what those guys code looked like. And it's insane. I could never write that code. Like I just don't think on that level of abstraction and I don't hold myself to that kind of standard, but it was incredible to read.

Joe Leo (13:20)
Yeah. Yeah.

Avi Flombaum (13:33)
⁓ so I think, you know, if I was starting out, like I would not, I would still want to know how this stuff works in detail. Like I would, so I would not cheat myself and rob myself of that learning. ⁓ but I would very much focus on my workflow and. No. Being willing to drop down that level of abstraction. I think that's my biggest fear, ⁓ about how the world is changing is that, you know,

People that want to be in code or in product and whatever you want to call it, ⁓ just aren't going to get that they still need to learn this stuff. They don't need to learn it all upfront, but they need to learn it. Like you need to get to a really deep understanding of how these things work. Otherwise you won't know what to ask for, you know?

Joe Leo (14:17)
Mm-hmm.

Yeah,

Valentino, Penny for your thoughts. Are you writing any code right now these days? Are you just prompting?

Valentino Stoll (14:35)
am writing code, but like maybe 10 to 20%, depending on the day. ⁓ Most of it is like ⁓ referential where I have like obvious, code that I like for specific purposes. like if it's a singleton class or a service object or some kind of like ⁓ form encapsulation.

Joe Leo (14:36)
You're right,

Yeah. Yeah.

Mm-hmm.

Valentino Stoll (15:03)
or all these things that web developers use day to day. You know, I have reference points and I could say, ⁓ I'll generate a thing like this, but like, you know, for something, some other purpose. But I do also like, ⁓ you know, from a, because I do have that depth of knowledge, I also use it for a different purpose. And that's more like architectural where I want to think about the abstractions only. And I am worried about the long-term.

Joe Leo (15:15)
Mm-hmm.

Valentino Stoll (15:32)
you know, ⁓ growth of a specific kind of class and how it acts with other classes and ⁓ how that actually works within a system. And so there is like a large variety of ways to use AI that I think people ignoring are, ⁓ you know, I would hire a developer that knows that doesn't know the depth, but knows how to use the AI tools over somebody who has the depth and doesn't use AI tools ⁓ is what it's come to.

Joe Leo (15:38)
Mm-hmm.

Yeah.

Yeah.

Valentino Stoll (16:02)
⁓ so I guess like

You know, Avi, when you're like, because when I was learning Rails, you know, I learned Ruby through Rails. I did, you know, I like many I think do, right. And then later on you learn, oh, like, you know, if you, you know, if you create a Ruby only class, like you can work with the system much better and cleaner, right, than otherwise. And you've kind of learned that over time. And so I'm curious, like, where do you see that parallel growing? Like.

Joe Leo (16:27)
you

Valentino Stoll (16:35)
If somebody's new to Rails ⁓ and they just like jump into an AI coding agent, know, like, I guess where do you see the focal points, right? Like where do people try and learn while they are getting these examples generated?

Avi Flombaum (16:55)
Yeah, that's good. Yeah. I remember learning. ⁓ Also, I started with Rails and, you know, ⁓ I didn't, it took me a while to know where Rails ended and Ruby began, you know, like, ⁓ and, like, I remember when I, when I realized that like has many was just a class, it was just a class method on the Mac, you know, basically essentially a macro and how like meta programming worked. And

Joe Leo (17:09)
yeah.

Mm-hmm.

Avi Flombaum (17:21)
You the one way we taught a flat iron that I was really effective was we always moved up levels of abstraction. So we didn't get into rails until like week four. You know, the first thing I taught them was basically object oriented Ruby, like how to build classes. And, you know, we would build like a class, like a post and they would build like a command line blog that had no, like, you know, it had no memory, right? You build up the program, you write a post, you close the program and

the next time you boot it up, there's no old posts. Everything was being stored in memory at the runtime. And then we took those classes and we added SQL to them. And then they learned CreateTable, Insert, like a post class now that when you call save, instead of popping it into a class variable, it was writing it to the database calling Insert. And then we metaprogrammed another class like Author.

Joe Leo (18:01)
Right.

Avi Flombaum (18:13)
And I showed them that, okay, they're writing the exact same SQL, like insert into posts is exactly like insert into authors. How do we abstract that? And that's how we learned like modules, right? So you can mix something in. And basically they were building a little version of active record. Right. Perfect. With the exact same API, they had no idea. Right. And then we moved into like, we spent like maybe half a day on rack, just so they could see that building like a basic web server and integrating it with those, you know, active record like classes. And then we moved into Sinatra.

Joe Leo (18:26)
Mm-hmm.

Avi Flombaum (18:42)
And I had sort of hacked together like a somewhat rails looking, you know, MVC sort of structure to Sinatra apps. So they would understand MVC outside of the, you know, entirely packaged version of it in Rails. Right. And then we'd finally go into Rails so that by the time they got to Rails, like they knew so much about the API, right. And suddenly all that was just abstracted away by the framework, but they understood that. Right. And they were never intimidated.

by like Ruby or like making poros, you know? And I think that's what really helped us get to that depth ⁓ of really understanding. ⁓ So I would one, you know, if I would, what I would tell juniors or I hate that word, what I would tell entry level devs is that one, ⁓ they should be asking the AI wise, right? Why does that work? How does that work? You know, in fact, fire up, you know,

Fire up two sessions of Claude. Have one writing the code and have the other one where while it's writing, all you're doing is asking it questions about other code it's already written so that you can be learning and understanding it. Right. I always like to ask it for pros and cons. Like when I'm approaching like a architectural decision, I might have an opinion and I'll say, what do you think about this? Feel free to disagree, list out the pros and cons, what other patterns might apply. One, because

Again, just, those, those choices matter to me, especially like complex systems. Like I know I want, I'm going to need to maintain them. ⁓ but I also want to understand, know, like I still learn from the AI on that level. ⁓ so I would make sure to be doing that. Like, ⁓ you know, I would ask a lot about why and how does that work? and in that I would learn, ⁓ I would also again, like read code, right? Like I would still read the books.

You know, like I love reading books. ⁓ you know, blog articles, like I would always tell our, like, you know, students like, you know, they need to have a favorite blog, right? Tell me who your favorite programmer is. Right. I made them all blog. They had to write posts. I would tell all entry level developers to be writing at least a blog post a week. What did they learn this week? Right. ⁓ yeah, you know, I, I just think that all those practices still matter and it's a discipline. ⁓

Joe Leo (20:53)
Mm-hmm.

Avi Flombaum (21:08)
You know, we always had script kiddies and you could always tell, right? Even at rails, right? You always could tell the people that like had no idea how this stuff was working and no interest in that. Right. You can't have that. Like you have to love this. Like you have to love the craft and I need to come out. ⁓ so. Yeah. If I was building a curriculum today to try to get people, you know, entry level jobs, I would really focus on, ⁓ the workflow, right?

how to learn and patterns.

Joe Leo (21:41)
Well, so let's

let's get into that. Let's let's imagine that you've got a six week curriculum, right? And it's the same folks. And let's just assume that this is still a marketable enterprise. You've got let's say you've got the same 20 people right that you had in your first cohort. ⁓ And your goal at the end of that six weeks was to be able to, you know, go, you know, with a clean conscience to some companies and say, hey, I think these I think these folks are ready to ready to rock. ⁓ What would you do?

How would you structure it?

Avi Flombaum (22:12)
Yeah. The way I thought about it was basically I would want them starting ⁓ like AI workflow first, right? ⁓ I wouldn't make them really write a lot of code by hand, but what I would want them to do and what I would stop them from doing is they're not using the AI workflow to generate ⁓ real code yet. They're using it to learn, right? So they're, asking it to, you know, and I would tell them like, you know, you want to build a class that does this.

Right. The only prompts I would allow them to use and I would, you know, I would basically build a chat app, like a tutor. that, ⁓ the only things that would answer is basically, ⁓ like, ⁓ here's what I would do. And like, here's how it works and why, but they would have to hand copy that code into the file. Right. So that like they're, they're somewhat writing it themselves, right. And knowing what maintaining it looks like, right. It would mean that they're not allowed to just.

have it generate thousands of lines, not read any of it and continue. Right. ⁓ so the things they would generate would be on a way more granular level, right? It still would be methods, classes at a time, things like that. ⁓ and then again, just move up that level of abstraction, right. and slowly, you know, allow them to use the AI to actually generate like that, that command line blog app. Right. But the way they would have to approach it was first, you know, create a post-class.

Joe Leo (23:14)
Mm-hmm.

Avi Flombaum (23:39)
Right. Create the CLI that initializes a post or asks me, know, ⁓ you know, what the title is going to be and calls that method correctly. Right. It would be on a way more granular level. ⁓ and that's, yeah, that, that is basically what I'd go for. I also, ⁓ they can't just be an engineer. They have to be a product engineer. Like, they have to understand like UI, like, you know, some like design, like they just need to be at that product level.

Joe Leo (23:49)
Mm.

Avi Flombaum (24:07)
Right. They have to understand how to write a spec, how to take requirements, how to describe those to the AI. ⁓ I just don't. I wouldn't hire just a programmer at this point. I would want to know that they can really be end to end.

Joe Leo (24:22)
And how would you be able

to tell the difference?

Avi Flombaum (24:24)
Basically, did they write a spec to the product level spec before they wrote the code spec? When I'm building a feature, the first thing I start with is basically what I would write as a product manager. Then I have it create a plan of how I would implement that. Then I read that markdown file for what its implementation looks like. I review it. I decide if I like that architecture. I basically edit the processes. The product manager writes the...

know, product spec, the engineering manager and engineering team that's going to work on it, breaks it down into, you know, sprints and issues, right, that are engineering centric, same exact process, right? They have to do that.

Joe Leo (25:01)
Well, but for

our listeners, what distinguishes that product spec from just an engineering plan, right? A plan of attack that does not take into account the product.

Avi Flombaum (25:13)
Yeah.

Yeah. It's what to build and not just how to build it. Like what is the feature that's going to deliver this value and solve the problem for the end user and the business goal? And then how am I going to implement it? Right. Like if you ask an engineer to, you know, build a user registration system, they might build, you know, like that feature that generates weird, you know, usernames.

and like, you know, like the way GitHub does, and then also build like, you know, can't use like bad words. So I have to build like a, you know, a whitelist or blacklist dictionary. ⁓ That is not needed. Right. Engineers will yak shave features a ton. And, you know, it's not their fault, right? They just don't think in terms of like, what is the real value and what's the feature that's going to solve this? You know, their UIs and UXs are just, pretty random. ⁓

Joe Leo (25:51)
Hmm.

Avi Flombaum (26:08)
So that's like sort of the difference, right? And yeah, I think there's a big difference in knowing what to build and then how to.

Valentino Stoll (26:16)
This reminds me a lot of like the, what was formerly behavior driven development, right? With cucumber and everything. and so, like so much of, of this kind of thinking reminds me of that time, ⁓ where people literally just like, you know, used Gherkin, which was like, ⁓ as like very English looking thing that just said, Hey, you know, go to this page and then click on this button and then type this thing. And it was almost like.

Joe Leo (26:24)
Mm-hmm. ⁓

Valentino Stoll (26:46)
you know, behavioral instructions of what was wanted to be built. And then eventually people just hooked it up to like an automation suite and like had something visit a page and click on a button and all of these things that you like, right. And so I'm wondering if there's some corollary here of like, even like junior developers learning, right? Like, ⁓ you know, should people learning early in their career?

think about that behavioral first, or should they focus on the structural, getting the foundational depth of programming, right? Like, where should that entry point be? Because I'm kind of, like, confused myself, like, telling people and being like, you know, really, everybody keeps saying communication is key, it's king, like, how you communicate with these AI things is most important. And with that, like, it seems to me like...

Clearly defining a specification seems more important. What do you think about that?

Joe Leo (27:47)
Mm-hmm.

Avi Flombaum (27:47)
Yeah. Yeah.

Yeah. So, you know, I think back to like, ⁓ the authors of the Agile Manifesto and, you know, the mythical man month. And there was a time, I think, before, let's say the product manager was born where, ⁓ the engineers would have to take specs, right? You'd be talking to the client, the customer about what is the problem, right? What's the solution you had to take, you know, scope the spec. Otherwise you're in a land war of Asia.

Right. And, you know, they were very much thinking about, you know, how do we build complex software? How do we really understand like what the client needs? And we, you know, codify that in like, you know, like the test suite to begin with. Right. I don't know when the product skill came out when we said that, okay, we need actually specialists on that. Like, what are the requirements and what's the solution that can pass it off to the engineers? Because.

It is its own skill. And I guess also some engineers are simply just bad at it, right? They just.

Joe Leo (28:52)
So are some product managers.

Avi Flombaum (28:54)
Right, exactly. Totally. ⁓ So ⁓ yeah, I just think as a product person, ⁓ I know a lot of engineers that look down at that role. ⁓ And they just don't get why it's useful. They just take for granted that the spec they see in front of them, the product spec, was like, of course I could come up with that. But it's, ⁓

Joe Leo (29:18)
Well, I think

that, you know, to use the to go back to the conversation we were having earlier about the different layers of abstraction, I do think that that the the engineering work it takes to solve an engineering problem is a layer of abstraction above or below. However you want to look at it, the the software as a whole and how it solves a business.

problem, right? How it makes money for for most of these software programs. And I do think just as somebody who runs a business who ⁓ is or was an engineer, however you want to look at me, ⁓ you know, it is actually hard to plunge down into how I'm going to solve this problem and pick my head up and say, right, it's not just about solving this problem, which is a really cool problem to solve. It's also how that how solving that problem is going to make the company money.

And I think it's true exactly what you said. ⁓ Some engineers don't want to think it's such crass terms, right? ⁓ But that's what we've got. That's why we get paid. ⁓ And I think it is also the case that ⁓ some product managers or just people who call themselves product people, they tend to focus on either ⁓ the communication between

various teams or in just writing the perfect spec and they also do not connect it to, right, there's somebody that's paying us to do this and that money comes from whether or not this thing is successful. And I think that mindset, you've nailed it to say that it's a skill. ⁓ I don't know when it became separate from the engineers, but I actually think it's okay that it has, but I also don't think that it's, I don't think it's easy to marry those two.

Avi Flombaum (31:07)
yeah.

Yeah, no, definitely is. there are certain, I mean, there are, like, I've worked with amazing engineers that I would never, ever want to, I would never want them to think about the product level. It's really important that they are working with, you know, the product manager, the engineering manager, and so that they just need to think about the implementation and how. ⁓ You know, one of the things I would see a lot ⁓ is, you know, product people don't necessarily know what's hard and like where the dragons in the code is. And, you know, they might write a

Joe Leo (31:21)
Hmm.

Avi Flombaum (31:41)
like fantastical product spec, I, this is what I wanted to do. And, you know, you pass it off to engineers and they're just like, yo, this is going to take like years. Right. And, you know, sometimes they don't push back and they just start trying to implement that. And, you know, it just, goes haywired. Like there needs to be a really deep collaboration between product and engineering. Like I, every product, every, I've always only, my departments are always product and engineering.

Joe Leo (31:45)
Mm-hmm.

Right. Yeah. Right.

Mm-hmm.

Avi Flombaum (32:10)
Right? Like it was, never had two separate teams. was one team because I just don't understand how you can have like this silos where like one is throwing something over the wall waiting for the other one to throw it back. So, you know, when, when the product person write the spec, the next thing that would happen is they would sit with the engineering manager and the engineers so that the engineers can be like, Hey, this part is going to be really difficult. Right? So the product person could scope and say, Oh, I didn't know that. Let's try to think of a different solution.

Joe Leo (32:21)
Mm-hmm. Yeah, I hear you.

Avi Flombaum (32:39)
Right. Let's create, know, what would be easy. Right. That collaboration is so important. And I think it also gives everyone buy-in and understanding, like understanding what's going on. ⁓ So, you know, yeah, I think that's, you know, there's a subtlety to that. Right. And again, I don't think that's how it works everywhere. ⁓ But I do think that's really important to success. Yeah, it's just product people just don't appreciate what's hard. And if they knew it,

they would make different choices and you just never, you have to give them opportunity to make that choice.

Joe Leo (33:15)
Yeah. So, OK. So then how do we tie this back to AI? Because AI is also not good at figuring out what's going to make you money. I mean, maybe it's it's gotten maybe it's getting better. Maybe the latest models are better at it. But, you know, last time I asked it to make me a bunch of money, it failed. And so I'm just going to assume it's not great at it. but what how are we supposed to leverage these tools when it comes to sort of that that

step before or that planning phase before we go in and write a bunch of code. Because the thing is, now we can write a bunch of code. And that product spec that the product manager from five years ago brought to us that would have taken us a year to build, well, now it only takes two months. So doesn't feel as bad, which doesn't mean that it's a good thing, because probably it's not all needed.

Avi Flombaum (34:06)
Yeah. Yeah. So, ⁓ what I, my workflow and I'm starting, let's say at the end of like, what should I be building is I am creating, you know, canned like UI demos. Like I'm using Claude first to create like a react, a little react app of the, of what I want the app to look like, ⁓ you know, so I can iterate and be like, ⁓ I don't like that feature. That doesn't really seem like, like it's going to really help users. let me see a different version.

of it, I'll ask again, I'll ask Claude, know, or Codex, I'll say like, you know, create three options for this screen, right? What are three ways in which we could, you know, ⁓ help the user see, like, ⁓ why this VC is a good VC to pitch, you know, for their startup? Like, what information could be on that page? And yeah, it's just so quick, you know, like on one level, you know, when I'm working in my product mode,

Joe Leo (34:55)
Mm-hmm.

Avi Flombaum (35:04)
My specs are no longer words. Like it is like, hey, look at this demo, right? And look at the prompts I use. Like at the end of it, once I have the demo I like, I have, you know, basically the AI write the product spec for me. I have a template of what sections I want it to fill in, you know, examples of what good and bad look like, you know, it's a slash command at this point. ⁓ and yeah, that's kind of the way I would approach product at this point also, right? Know how to use AI and know, you know, what ShadCN looks like and like,

Joe Leo (35:08)
Mm-hmm.

Avi Flombaum (35:33)
Create a canned demo, like fill it in with fake data, right? Look at it. You get to do with different options. And that gives so much clarity to like what I'm looking at now in the engineering level, like what am I trying to build? Right. ⁓ so again, that's part of the workflow. Now, how do you teach you the product also as a salty, right? But there are books on it and there are blog posts, right? Understanding like, you know, what entropy is, what information architecture is, you know, what

Joe Leo (35:47)
Hmm.

Avi Flombaum (36:00)
the difference between user experience and user interfaces, You know, understanding how to ask questions, like understanding even what like business value means, like how do you define KPIs, right? Like what is the goal? How are you going to know if it actually worked or not? Like, what is your target? How are you going to iterate on it? Like that's teachable. I just, didn't do that at Flatiron to, you know, in any kind of depth. Like they had to understand it on some level because they were building their own apps.

Joe Leo (36:19)
Okay.

Avi Flombaum (36:31)
But I would very much constrain what they're allowed to do. One, you're not allowed to design. You have to be using standard component libraries like Tailwinds or whatever. You cannot create new user interface elements. Your apps don't need to look like that. So do not waste time on the design. But you still need to create a lot of like, here's what my app does on a high level. I want to see a flowchart. I want to see descriptions of stuff. Otherwise, I just don't know what you're going to build, and neither do you.

Joe Leo (36:44)
Mm.

Hehe.

Avi Flombaum (37:01)
⁓

but I never made them actually like, you know, define KPIs or write the spec or, know, but yeah, so now I would, that's what I mean by like the bar has been raised, right? Like I'd want to see the AI workflow of like, you know, like I love that new feature they built on Twitter or X about how they're dealing with like links, right? You know, we, Twitter knew or X knew that having a link in a tweet meant that the user was getting bounced off Twitter.

which is not something they want, right? They punished people for putting links in the first tweet, right? And basically didn't know how to solve that problem. But obviously people need to link to stuff like that's part of the fun of Twitter, right? And, know, Nikita, when you joined as, you know, uh, the head of product at, at X, like now when you click on a link, right, you get that like scroll up of the embedded browser. So you're not bounced off the Twitter app.

Joe Leo (37:40)
Mm-hmm.

Right.

Avi Flombaum (37:57)
You can still read that page and then minimize it really quickly and go back to the tweet, right? It's whatever you call them now. That is freaking brilliant. It is so cleanly implemented. maintains their use. It maintains both their goals of we don't want people bouncing off of X, but we need to allow people to share content that is on the web. What Twitter has been around since I don't know how long they, no one else thought of that.

Joe Leo (38:07)
Yeah.

Yeah, LinkedIn's been around longer. They haven't figured it out yet. You still get dinked for putting for putting links in your posts.

Avi Flombaum (38:29)
Right, that is insane.

That is an insane, like you have to understand the problem in both senses so clearly to understand that solution. And now that that solution is there, it seems crazy obvious. Right. At no point, don't think that Nikita think, well, how are they going to implement that? Right. Cause that's such a, obviously that's not, you know, it's not like rocket surgery to build it, but like I would want to, they don't mean that that's again, that's, would say like some of the best product work I've

seen recently. ⁓ But I need to teach them to think on that level, right? To really dig for what the problem is. know, ⁓ like, I drive people crazy when, like, a department would come to me with a problem. ⁓ They would also propose a solution. And I have to get them in a conference room and just ask them, well, why? What are you trying to solve? You know, just the most pedantic amount of questions because their solution is not right. Off the bat.

Joe Leo (39:32)
Hmm.

Avi Flombaum (39:33)
Like, everyone, whenever all the solutions are essentially, I want you to build me a spreadsheet because that's the interface people know. Right. ⁓ So yeah, you just got to ask questions like what is the problem you're trying to solve? Why? Right. Where's the entropy right now? What's the workflow? Right. ⁓ And then I can understand like, okay, here's ways I would solve it. Here's the magic. Right. Here's what's possible with good product design. ⁓ And then I would write, you know, again, take the spec.

Valentino Stoll (39:40)
Hehehehe

Avi Flombaum (40:01)
talk to the product managers, make sure they understand it, do that meeting with the product manager and the engineering team and kick it off that way. Yeah, I would, again, I'd want them to go through that process, right? At some point in the curriculum, I would be that end user and say like, ⁓ we're building an LMS and ⁓ the grading process is really slow. I have this spreadsheet with all the students in it.

And there are columns for every assignment. And I mark in the spreadsheet when they handed it in and if they handed it in and whatever. now we're doing, we have so many more. And I've got hundreds of different spreadsheets for every single cohort and running reports on the progress on cohorts is impossible. Can you just build it into the app so the app now just has all the spreadsheets in one place? No, I can't.

Joe Leo (40:56)
Yeah. Yeah.

Avi Flombaum (40:59)
So yeah, that's the first, you know, that's sort of like the way I'd want, like I would bring in that problem and see what they do with it. Right. Like I would love like to recreate that like link problem at Twitter. Right. You know, I think a lot of like, I think Airbnb also is just amazing like product in UX design. Right. I want them to recreate that, you know, I, of the questions, interview questions I would ask my product people was, you know, everybody books in Uber.

to the airport, right? How do you get them to book the Uber, both from the airport to the destination, and then from their destination, eventually back to the airport and from the airport to their homes? I want to ensure that if you're taking an Uber to the airport, you book all those four trips. What do you do? Right? Yeah, like, you know, like,

Joe Leo (41:42)
Hmm.

Okay.

Avi Flombaum (41:55)
Sometimes I would get answers. Like I put advertising, you know, in the Uber pickup about like, you know, use Uber to book, right. There's a hundred different solutions, right. But the first question you have to ask is, well, why aren't people doing that? Right. you know, is it laziness? Is it like, they think they're going to find a different route, right? Like, do they have more flexible travel plans? So maybe they don't have the return flight booked like what, right. ⁓ you know,

Yeah, I want to understand that a lot. I'd want to understand like where the, am I allowed to offer discounts for booking the airport to destination, right? At the same time you're booking the airport trip, right? Am I allowed to send push notifications, you know, a day into their trip that, you know, offers them a discount, right? ⁓ Like things like that, right? Am I allowed, how would I possibly integrate with like their, ⁓ you know, calendar

Joe Leo (42:45)
No.

Avi Flombaum (42:53)
or like they're, you know, ⁓ taking like airline basically to even know like, what are, what is the end date? Where are they going? You know, things like that. So yeah, like I don't understand that. And I come up with different solutions and, know, then figure out like, yeah, what are we going to try first and how are we going to measure it? Right. ⁓ so yeah, I, I want

Valentino Stoll (43:16)
Yeah, you

you, come up with like two really interesting ⁓ ideas here ⁓ that I think are like super important for engineering that just like maybe is ⁓ not looked at ⁓ as much as it should be. ⁓ The first is like, ⁓ like you mentioned, just like requirement deconstruction. Like, do we need this? Why do we need this? Right? Like people just don't ask that question. Right.

And like, sometimes, you know, like, Avdi Grimm has this like fantastic talk, I forget what conference it was at, where it was like no code. And like, often, like the best solution is just not writing anything. like, maybe just start with like a spreadsheet, right? And see if like, you can get away with that before like, you know, like, do you even need a database table, right? Like, sometimes the processes just like, don't require the complexity. And like that.

Joe Leo (43:54)
Mm-hmm.

Heh.

Valentino Stoll (44:14)
is like not really taught, right? Like that's supposed to be like an on-job thing of, you like discover the complexity and when you need it and don't. And then like after five years or so of developing stuff, you realize, I didn't need to create like, you know, three chains of abstractions to just like create a blog post, right? Like maybe I could have just used the Git repo, right? And then let it do its thing, right? ⁓ Which some people do, right? And you can do now. ⁓

But at the same time, ⁓ the second thing is like this learn by example of, this is how I learned Rails really well, is like there used to be a site called Open Source Rails and they had like a bunch of Open Source Rails apps for very specific purposes, like a bug tracking tool, right? Like a project management, like they had all the different things that you would use day to day in your software engineering ⁓ skillset.

Avi Flombaum (45:02)
yeah.

Valentino Stoll (45:10)
And they had a Rails app and you go and you look how they built the bug tracker, right? And like, we almost need tools like that, have this like, you know, like you mentioned, like an example set up already done and you ask an AI tool, how does this work? Right? Like. ⁓

Avi Flombaum (45:28)
Yep. Yep.

Joe Leo (45:28)
That's an interesting

point, right? Because all we are all we are persisting right now is the end result, but not the means to get there. That's interesting.

Avi Flombaum (45:35)
Yeah, yeah.

Yeah, we actually had a code reading club at Flatiron where we do exactly that. Like I would pick an open source project and every week we would read about, like read the code and discuss it. And in Saran, Yiprak started Code Newbies. Like one of the events they had weekly was she took a code reading club, right? Like it's amazing. The good job source code, by the way, site, open source Rails still exists and there's a whole bunch of like

Joe Leo (45:57)
Hmm.

Avi Flombaum (46:04)
know, awesome open source rails that GitHub like, you know, repo. Yeah, I think it's great. Like that's that that is exactly what how I approach, right. And again, like I would have my like essentially my AI tutor, right, it would, you know, know that code base. And, know, they basically have like a chat session where the AI would like basically help prompt them about like, you know, what like what part of this code you want to read first, you know, and show the source, you know, the GitHub lines like side by side, right.

where it can really walk through, right? ⁓ I'd love them to be able to like mouse over a line on that code and see like a quick AI explanation, right? So that, you know, even that user experience is better, right? Than them having to open the code and navigate to it, right? ⁓ But yeah, I think it's such an amazing way to learn, right? The same way with like the product stuff. Like there are a lot of great websites that list like, here are the amazing product solutions, right? Let's look at them, right? Let's understand why they're so coveted and amazing.

⁓ Yeah.

Valentino Stoll (47:02)
Yeah,

you know, I think of, you know, the Rails guides, Chris Oliver contributing all these example apps, right? Like you said, it's like so brilliant. Like that is ultimately, you know, what a great use of the funds for her, to be honest, from the foundation, right? Like, you know, that's ultimately what the guides needed this whole time. Now that you see it, right.

Joe Leo (47:09)
Yeah, I was thinking the same thing. Yep.

Avi Flombaum (47:16)
my God. Yeah. It's yeah. Yeah. It's so

Joe Leo (47:17)
I know.

Avi Flombaum (47:22)
Yeah. Yeah. Yeah.

Yeah. I mean, it's so good. mean, look, that's also like Agile Upton and Ruby on Rails. It's such a great book because that is how it approaches it. Like the entire, it's, there's this entire narrative, you know, it's totally different than like, you know, the Rails three way or Pooter. Right. ⁓ I don't know how many of the books in the Rails world are like that, but I think that was, DHH in general is just so good at that. Like, you know, the five minute blog, right. Same, same way of approaching, right.

Valentino Stoll (47:31)
Yeah.

Joe Leo (47:48)
Mm-hmm.

Avi Flombaum (47:51)
⁓ like, yeah. so again, like I think these are, I think those are a lot of elements that I brought to teaching that made a big difference in like the outcome and also the experience of what it felt like to learn. ⁓ yeah, like, but yeah, I think that that's, that's how I approach it. And again, I, I w they can't, they just can't use chat GPT out of the box. They would rob themselves of learning.

⁓ you know, I built a sample of this tutor. ⁓ it's actually a line called like Socratic tutor dot AI or something where, ⁓ it just won't give you an, it doesn't give you an answer. Right. Like you ask it what one plus one is and it will be like, well, what do you think one plus one is? Right. Like this really pedantic. And I would tell students also that like, when you ask me a question, I am just going to ask you one back. You're going to think I'm an ass and I'm just being pedantic and annoying.

Joe Leo (48:34)
Hmm.

Avi Flombaum (48:46)
But I will ask you some leading questions. I just won't give you answers. You'll discover the answer yourself. So I would very much build that tutor and give it different modes so that I would understand, are they building right now? Are they learning? Are they debugging and constrain the way it's going to interact with them based on that? Yeah.

Joe Leo (49:08)
Yeah, yeah, that's interesting.

Valentino Stoll (49:10)
Yeah, you know,

I've seen some similar ones out there where they generate incorrect answers and inject bugs on purpose. Prove that same learning.

Avi Flombaum (49:18)
that's so,

Joe Leo (49:19)
yeah?

Avi Flombaum (49:22)
I love that. That is so brilliant. Yeah, I would, my God, I love that. I love that. That's such a good idea. Yeah. That is some good product thinking.

Valentino Stoll (49:30)
You know, especially like I think of like,

I think of like the Rails parameters, right? Like it's so easy to just like, you know, have a params permit that just isn't including something. And then you submit the form and like, Oh, why isn't this like, you know, saving this like one field I have in my form? And like, you yeah, just drop that and let them find out what it is, right? Like.

Joe Leo (49:49)
Mm-hmm.

Avi Flombaum (49:49)
Yeah, yeah, yeah.

Joe Leo (49:54)
Right.

Avi Flombaum (49:55)
Yeah,

Joe Leo (49:56)
Yeah, that's a good idea.

Avi Flombaum (49:56)
yeah, yeah, I love that. That's such a good idea. Yeah. All the curriculum at Flatiron was test driven. like all the, every, every lesson was essentially a repo on GitHub. And, know, I would give them a test suite and everything would fail, right? And they would basically, you need to make it pass. And that was also one way in which I would give them the spec of what they needed to build. And to some extent, like the architecture, right?

⁓ Like whether, you know, sometimes it would be like a Selenium suite so they could pick it themselves, but you know, they know the elements that need to exist and the end result. But sometimes I'm literally giving them, you know, like, you know, describe this class and hear the instance methods. And, you know, they're learning test during development, you know, they're learning how to debug. They're, not being scared by broken code. Right. ⁓ But yeah, I think it's a great idea. That is some awesome product thinking, right? Like have the AI inject bugs on purpose, you know, script it.

Valentino Stoll (50:42)
You

You know, sometimes I do that to myself just to stop from generating AI slop. As I'll generate a test that fails and then have it purposefully, you know, not work appropriately. I'm like, yeah, maybe I shouldn't be generating code today.

Joe Leo (50:55)
Right.

Avi Flombaum (51:01)
Yeah. Yeah.

Yeah. Yeah. Yeah. Yeah.

Joe Leo (51:05)
Yeah.

Avi Flombaum (51:08)
Yeah. Yeah. When I'd give lectures and they were all so scripted, like I would, you know, I would do something wrong on purpose and like the bug would come up and be like, that's weird. And then what do you guys think it's not working? Like, how should we debug it? What's the approach? You know, and it also showed me that like, it's okay to have to make mistakes. Like it's literally all your code is broken by definition at first, and it's not your fault.

Joe Leo (51:18)
Mm-hmm.

Mm-hmm.

Avi Flombaum (51:32)
Right? Like when you get the code working, you're no longer programming. You're done for the day. You know?

Valentino Stoll (51:32)
Hehehehehe

Right.

Joe Leo (51:37)
I did the

same thing when I was teaching, but I did not have to plan it in advance. was guaranteed to have at least one to two failures in front of the class every time.

Avi Flombaum (51:42)
Yeah. Yeah.

Yeah. Yeah. Yeah. Exactly. just, I just, that's how I came up with them. I actually happened. This is, this is the one like, this is now built into this lecture.

Joe Leo (51:50)
Yeah, right, yeah.

Valentino Stoll (51:59)
That's great.

Joe Leo (51:59)
Well, we're

coming up on time. So should we get to our final segment?

Valentino Stoll (52:05)
Yeah, I think so. ⁓ I did. Yeah, I mean, we didn't get to dive into all of your philosophical ⁓ thinking about creativity and how the muse plays into all this AI stuff. Because you have a lot of points on many articles that I thought were pretty insightful. ⁓ So ⁓ if you're interested in that, folks, I would recommend checking out Avdi's articles. He's got some great stuff.

Joe Leo (52:07)
Yeah.

Valentino Stoll (52:35)
⁓ And I don't think, you know, we're going anywhere as engineers anytime soon.

Avi Flombaum (52:36)
Thanks

Oh

yeah, no, I mean, look, more software has, it's counterintuitive, but more software has only created the demand for more engineers. And we're, we're going to get so much more software in this world now. And like the demand for engineers is just going to skyrocket. Again, I think what entry level looks like is going to change. But, you know, I remember like early on in my career with like the WYSIWYG error, you know, of like Dreamweaver and Flash and to work, you don't need to like, just everyone's going to do this. You know, and I was like, nah.

Joe Leo (52:40)
hehe

Mm-hmm.

Yeah.

Avi Flombaum (53:07)
No, you know, that's not, that's not going to happen. You know? then, then when like the low code tools came out, like retool and stuff, we're not going to need more programs. Every, every person is going be able to build their own app. I was like, Nope. You know, ⁓ now like, the vibe coding thing, like one, you know, people that are vibe coding stuff and like the app gen is what I call like lovable bolts. Man, those things are not even close to approaching the complexity of the stuff we work on. It's just not, you know, like.

Joe Leo (53:18)
Mm-hmm.

Mm-hmm.

Valentino Stoll (53:32)
Yeah. Although to

your mention of Flash, like creativity has truly dropped off since I feel like. It used to be so easy to animate stuff and like, you know, frame by frame. Personally, you know, I felt a huge loss there.

Joe Leo (53:44)
hehe

Yeah.

Avi Flombaum (53:50)
Yeah,

yeah, those things are really necessary, right? All that, you know, crap like, yeah, yeah, yeah. But yeah, so, you know, I think

Valentino Stoll (53:52)
Absolutely necessary.

Well, it's so-

Yeah, we're getting up on time. I wanted to make sure that we hit this. We started this new segment at the end of the show where we share an AI prompt or tool that we're using that we found very useful, super helpful if it's day to day, but could be anything where you're just like, wow, and I found this.

Avi Flombaum (54:02)
Yeah, I just.

⁓ yeah, I guess, ⁓ the thing I've been really interested in right now is this thing that Kyrian and every team call compounding engineering, right? Which is like, what is the automated workflow look like so that, you know, ⁓ when you're, when you're starting something, creates the GitHub issue, right? And it could read from that and then create like whatever your spec is, right? The markdown files or whatever that you can review.

Right. And then it can create the solution and put in the PR. And then you have another agent being able to review that PR, right. And leave comments and give you a summary of like what the refactor should be. And, you know, based on all that at every step, you're basically changing your prompts, right. So that like when you're reviewing, here are the things I want you to pay attention to here, the things not to do. Right. And then you're basically, you know, constantly on this end to end cycle, automating

all like smaller, smaller parts and, and giving each part the ability to reinforce and learn over time. So it doesn't make the same mistake. Like the amount of times when I've had it write a spec where it estimates like in week in human time, this part, this sprint will take one to two weeks. Right? What you mean, man, you're going to get it all done today. you know, um, the amount of time that it writes,

Joe Leo (55:42)
Mm-hmm.

Avi Flombaum (55:47)
testing like tests for performance, like these weird edge case, like, okay, let me make sure that this that this can, you know, get called 100,000 times. You know, like, what are you doing? Don't don't write performance tests. Here's how to constrain the edge cases that you're writing. Right. ⁓ And, you know, I haven't gotten that full workflow yet. Actually, next week, me and Nate and Kiran are having like a zoom call where Kiran is going to break it down for us.

Right. Because ⁓ Nate doesn't understand how that even works. And I haven't gotten it. I just, you know, I just won because I don't, work on a small team alone a lot. Like, I don't know if I need that end to end stuff, but I think that's a really interesting thing. And again, that's what you mean about workflow. Right. ⁓ Like understanding how to have the AI consistently learn from its mistakes.

and codify that into the process so you don't have to constantly hit them. At first we were using like cursor rules and things like that. That doesn't work because the AI just doesn't, it can't get the context. As much as I try to be like, have an active record agent, here's how he's an expert. If I forget to tell it to use that agent, it won't. And then I just have the general AI writing complex active record stuff. it's just, it makes the same kind of crap that,

If it read that agent file and my rules for how to use AR, like active record, it wouldn't have done that. ⁓ So that solution, think, is still very much needed. How do you prevent the AI in every part of your workflow from making the same mistakes over and over?

Joe Leo (57:32)
Yeah. Can you send us a link to maybe to the compounding engineering or?

Avi Flombaum (57:37)
Yeah, I'll you a... Yeah,

he's written a few articles about it. ⁓ you know, yeah, like I am in awe of how they build in their workflows, you know.

Joe Leo (57:49)
Yeah, we'll add it to the show notes. ⁓ As for me, I, so what's been making the rounds at deaf method over the last week is ⁓ Peter Steinberg is just talk to it. The no BS way of agentic engineering. And I'll add this to the, to the chat here. So this is, this is like long form, you know, experiences with the different models. ⁓

Avi Flombaum (57:51)
Yep. Yeah.

Joe Leo (58:17)
You know, like what we were talking about, it is it is all workflow and and it's really been excellent. I've I've started implementing some of the things that that he talks about there. It just it seems like a very seasoned experience report. They made me want to get Peter on the show, so maybe we'll reach out.

Avi Flombaum (58:36)
Yeah, yeah. Yeah. I again, like I think workflow is very much taken for granted. And I think it's like ⁓ you have to focus on it, right? Like you have to make that investment. And yeah, I think also it'd be really cool to do studies of prompts like, ⁓ you know, both. Yeah, I think it's a really cool thing. Like I would love to see more people publish the prompts they used and like even chats. And I would read those a lot. I think it'd be so interesting.

Joe Leo (59:04)
Yeah,

I do too. And I think to your point, it could be that across the industry workflow is being taken for granted or not being discussed enough. But on this show, every single person who comes on is talking about workflow. I think, you know, and we get, ⁓ of course, very smart, very driven, very successful engineers on this show. So I think that's that is providing an example for for all of our listeners for sure, certainly for me.

Lee, how about you?

Valentino Stoll (59:41)
Yeah, so ⁓ I've just listened to a ⁓ Latent Space podcast episode. They had Lance Martin on who does the Open Deep Research Project. ⁓ And he gave this, ⁓ it was a really great talk on breaking down ⁓ agentic structures and communication layers and what works and doesn't. And he's tried all kinds of different things. ⁓ But one thing really stood out.

Joe Leo (59:53)
Hmm.

Valentino Stoll (1:00:10)
And it was this project he made called LLM's Text Architect. And LLM's.Text is like a emerging standard for communicating website contents to LLM's. And what he found ⁓ just ⁓ trying to generate a ton of content and consuming it for agents ⁓ is that often if you just give it a ton of information, it doesn't use it very well.

Joe Leo (1:00:38)
Yeah.

Valentino Stoll (1:00:38)
and so

what this does is kind of just break it down into like simple, ⁓ really workable descriptions with URLs that link to the content. ⁓ and he's found that performance wise, you know, these co these agents that consume these, this informat these own text files, they perform much better if you just give it those descriptions and then have them go consume it on their own. ⁓ then if you try and.

generate a full stack that ingests the data yourself. And it's really wild. And so he has like kind of a script that lets you generate that from like a documentation website or something like that. And just like makes it really easy to do. So I've been playing with that. It's really fun. And then he also has some great insight into kind of like learnings from the bitter lesson.

Joe Leo (1:01:09)
yeah, that's interesting.

Valentino Stoll (1:01:35)
on ⁓ how to set up structure for your ⁓ AI related workflows or ⁓ services in a way that allows you to easily remove that structure. Because what is learned over time is that the models get better at specific things. ⁓ if you kind of like, you know, code yourself into a certain structure, it could, you know, ⁓ kind of

kind of force yourself to get worse over time because the models get better in different ways. And so it's really interesting insight. So I'll link the article.

Joe Leo (1:02:11)
I see. Yeah, that is.

Avi Flombaum (1:02:12)
Yeah.

Joe Leo (1:02:15)
This is with successive releases of

the model or well, I'll check it out. I'll listen to the episode.

Valentino Stoll (1:02:21)
Yeah, that pretty good.

Joe Leo (1:02:22)
Okay, cool.

All right. ⁓ Well, I want to thank our guest, Avi, for joining us today. ⁓ Avi, ⁓ where can we find you, find your work?

Avi Flombaum (1:02:42)
Yeah, I'm really active on X. my username is Avi Flumbaum. My blog is code.avi.nyc. I haven't published anything in a little bit, but yeah, know, yeah, X is a great way. I love it. I love the Ruby community and this is the engineering community on X. I think it's great. Like I love my feed. don't know, know, people complain about their feeds a lot. I'm like, I love mine. Like I just hit not interested in all the crap, you know?

Joe Leo (1:03:04)
You

Avi Flombaum (1:03:11)
But yeah, awesome.

Joe Leo (1:03:14)
All right.

Yeah. Well, thanks again for joining us. And, uh, um, and we will, oh, I almost forgot the, the, the Manning early access of the well-grounded Rubyist edition four is out and there'll be a link for you in the show notes. Uh, can we get 50 % off? Um, and, uh, and check it out. Um, all right. Almost forgot that, but I slid it there at the end. All right. Thanks guys. And, uh,

Avi Flombaum (1:03:18)
My pleasure. Thank you for having me.

Valentino Stoll (1:03:38)
Awesome.

Avi Flombaum (1:03:40)
Yeah. Thank you.

Joe Leo (1:03:42)
and

everybody have a great day. Thanks for listening. Bye bye.

Avi Flombaum (1:03:45)
Thank you

guys so much, bye.