In episode 11, Ron is joined by Michael Wharton, Vice President of Engineering at KUNGFU.AI, to unravel the art and science behind building effective AI teams. They discuss optimal hiring strategies, crucial technical skills, and the distinctive challenges AI teams face compared to traditional software development.
Check out the book Michael co-authored called, "How to Put AI Models Into Production", written for NVIDIA and published through Manning
Ron Green: Welcome to Hidden Layers, where we explore the people and the tech behind artificial intelligence. I'm your host, Ron Green, and I'm very happy to be joined today by my colleague, Michael Wharton, to talk about the art and science of building AI teams. We'll talk about optimal hiring strategies, essential technical skills, and how AI teams and projects differ significantly from traditional software. Michael Wharton is Vice President of Engineering at KUNGFU.AI AI and is an expert in computer vision with a particular focus on remote sensing. Michael studied aerospace engineering and computer science at the University of Texas at Austin, after which he spent five years at SpaceX working on test systems for flight hardware. More recently, he worked at Applied Research Laboratories, where he developed deep learning algorithms for processing data from terrestrial, airborne, and space -based sensing platforms. Over the last five years, Michael has contributed to a vast array of production -grade AI systems and focused on computer vision, NLP, and predictive analytics. Welcome Michael.
Michael Wharton: Thanks for having me.
Ron Green: Let's kick off and talk a little bit about how you got into AI in the first place.
Michael Wharton: That's a can of worms. Yeah, so back in the day when I went to school, I was focused a lot on engineering, aerospace engineering specifically, and an aerospace degree is a lot like a mechanical degree with a couple extra bells and whistles, and you learn about orbital mechanics and some extra stuff, but it's really fundamentally a mechanical degree. And while I was there, I also got exposure to computer science, I ended up getting a minor there, and I really loved what I found there, and it kind of pulled me into this algorithm development, software kind of life, where I got to spend a lot of time at applied research labs doing work with deep learning systems, and it started out as classical computer vision, but in the time I was there, it really translated over to deep learning because you had the Alex moment, sorry, you had the Alex net moment back in 2012, and around this time period, all those advances were coming out while I was working on these computer vision problems. So they just naturally were a really great fit with space-based imagery, and then took a little detour and worked at SpaceX for a while, but now I'm here, and I've loved AI since I just got that little taste, and I couldn't let go of it.
Ron Green: I love it when I get a chance to talk to people who have experience in computer vision, sort of pre -deep learning because you kind of have experience with the pain of feature engineering and all that, you know, entire world that's kind of gone away now. Let's jump in and talk a little bit about how, you know, building AI systems and building AI teams differ from traditional software, you know, we kind of hinted in the intro. Maybe just kind of a high level, give me your thoughts on that.
Michael Wharton: Yeah, it is fundamentally different. And I think a lot of it's really psychological, and you get to have to get used to probabilistic outcomes. And historically with software, you expect determinism, you expect I put in an input and you get the same output every time. And you know, roughly what you can expect, you know, with AI, you got to flip the script and completely throw that out. And a lot of time with clients, you know, that's the biggest hurdle to get them over is that you're taking a risk putting these production systems online, and it's taking in data that's never seen before. And there are going to be probabilistic outcomes. So you can de risk that. But it's really just a big mind shift that translates to every other facet of the engineering process.
Ron Green: Right. Right. And to be clear for maybe folks who may be scared about the sort of unpredictable element of this, it's not so much that it's just completely random, but it's much more complex. You're dealing with probabilistic outcomes because you are dealing with systems that have been trained on a bunch of data and that have hopefully learned to generalize to a broad class of inputs and the challenges you can see some inputs the model didn't expect and then kind of all bets are off. Right? Given these differences, how have you found it best to organize AI teams?
Michael Wharton: That is a great question. You know, I think the at this state, at least right now, the industry is still very volatile. There's still a lot of best practices that are getting established, people kind of testing things, seeing how they work. So I think you can't really bet on a single playbook working all the time. And instead, I think it's great to optimize for flexibility for, you know, a relatively general skill set, not necessarily a Swiss army knife do it all kind of person. But you really want to have your team made up with a lot of diverse backgrounds and a lot of, you know, you want a lot of creative throughput. And a great way to do that is to have a bunch of people with different backgrounds, put them in the same room and the kind of outputs you get are just incredible.
Ron Green: And when you say different backgrounds, do you mean like different experiences with an AI or even more broadly than that?
Michael Wharton: I would say even more broadly than that, you know, certainly we want some people that have experience in computer vision, NLP, you want some recommender system folks and that like you need a mix of all that if you're doing the kind of work that we do, which is just AI professional services. But in addition to that, I love to see people that have backgrounds that go beyond that, people that kind of had a developed a love for science or engineering, irrespective of this, you know, new toolkit that we get from all these AI systems, you know, at KUNGFU.AI we have people that are economists, you know, neuroscientists, linguists, I'm trying to think of all the others.
Ron Green: Yeah, mechanical engineers. Yeah.
Michael Wharton: Yeah, I got the aerospace thing covered. It's across the board.
Ron Green: chemistry, it's really amazing. I couldn't agree more. I think the diversity of backgrounds that we have is actually a competitive advantage because people are coming to the table with just such diverse skill sets, mathematics, statistical backgrounds, et cetera. Are there any mistakes that you think you've noticed other companies make when putting together AI teams?
Michael Wharton: Yeah, I think organizing around discipline and baking that directly into the structure of the team, rather than, like I said, optimizing a little bit more for flexibility, can really stop you dead in the water. Because, you know, you end up with these silos of specialists that they do this amazing work in their own domain. But it sort of limits that cross pollination, that sharing that, you know, group growth mindset that's really important for adapting and listening to the market as quickly as it's changing.
Ron Green: I totally agree. I've seen silos in a bunch of different ways. Sometimes it'll be, well, you know, this is the computer vision group, or then this is the NLP group. And that might have made sense 10 years ago when the techniques and the lingo and the we're just radically different with the convergence. With the convergence now, it really doesn't make sense. But I've also seen, and I think this is a mistake, is this idea of having like a dedicated data science team or machine learning team that is completely separate from a software team because it's just so much more effective to have everybody working together and not have these fences where you feel like things are just thrown over without context.
Michael Wharton: Absolutely. 100 percent.
Ron Green: Well, given all that, what type of background do you look for when you're hiring somebody for a machine learning engineering role at KUNGFU.AI AI?
Michael Wharton: Yeah, I think there's some non-negotiables, you know, you, you want people to obviously have some experience with deep learning, you know, you, you need some core skills that are relevant to the job at hand, but by no means does it stop there. You know, you need people to have this holistic, you know, systems level, high, high level thinking about things that they can't really be narrow. This is, this is very open-ended creative type stuff, at least now, and you need someone who really cares about software engineering, about best practices, you know, these inevitably are systems that you build that you need to be maintainable in production. And you know, as, as tough as it is, a lot of the machine learning talent that's out there, you know, they're taught to just stay in a Jupyter Notebook or a CoLab Notebook, everything that they, and it's no fault of theirs, but that's just how it's kind of taught. But at the end of the day, you still need those traditional software engineering best practices or else you can't deploy production systems.
Ron Green: I don't want anybody to get mad at me about this, but I couldn't agree more. And as much as I love Jupyter Notebooks, it is really, in my opinion, best suited for prototyping during discovery and analysis phase. Writing production code in Jupyter Notebooks is just so, so, so challenging. What about diversity of skill sets? One of the things you mentioned is the production focus. How much software experience do you typically look for when you're hiring for an AI position?
Michael Wharton: Yeah, you know, we we typically stay away from the years of experience side of things. And instead of that, just really focus on the core skills that are necessary here. And, you know, that covers things like someone's got to be familiar with CICD, you know, unit testing, object oriented programming. You know, all the all that stuff is really vitally important. And even in addition to that, some familiarity with different hardware nuances becomes incredibly important with with artificial intelligence, just because, you know, GPUs are kind of all but a necessary ingredient in anything that you do.
Ron Green: Well, we were talking just recently this week about an instance where a system that was working great in training setting actually was underperforming in a production setting just because of floating point precision errors that differed within the GPU and CPU environment. So a little in the weeds, but it is kind of crazy how much software you really do have to understand. How do you foster a sense of trust and an environment where there's sort of continual learning in a field like AI where it seems like, you know, there are like groundbreaking advances almost daily?
Michael Wharton: Yeah, it's a little tough. I think you just kind of have to bake it into the DNA of everything that you do. There's no just one stop shop for an answer to fix it. But we have a few different techniques that we use. One of them is we invest a lot in personal growth to the point where we dedicate an entire day every week toward continual growth and learning. On Fridays, we call it lab day. But we have sessions that are dedicated for people to come share what they had been learning on a client project. They share a paper that they read recently that they want to work with the team on. Or sometimes people just have something that they want to test out on their own, and they want to just have three, four hours of just uninterrupted time where they can learn and get some experience with some new technique that just came out. But it doesn't stop there. I mean, we are really, really firm believers in pair programming. This is definitely not a place for someone who likes to just stay in a silo and go off in a closet and build something and then come back and say, hey, here, no, I built this thing. That's great for that isolated incident, but that high bandwidth connection that you get between people on project teams is indispensable when the field's moving as quickly as it is.
Ron Green: Yeah. Man, I couldn't agree more with that. The ability to stay informed and connected through pretty frequent pairing, I think, is transformational, especially when, like you said, the techniques are moving so quickly. And when you're dealing with scenarios which we're dealing with almost all the time, which is we're building supervised learning models. So it's so data dependent, so we don't really know project to project necessarily where the complexities are going to lie. Take me through, you know, maybe at a high level, a typical day in the life of an AI engineer.
Michael Wharton: Usually starts with a cup of coffee, maybe two or three. But no, you really, it's, it's relatively unstructured. Like we don't, we don't force a lot of structure on people, but a lot of that structure naturally emerges through projects. You know, not every client, I should add that we're professional services. So that's got its own nuances compared to something like a product company. But client projects typically have, you know, a couple of standups a week, maybe one or two, depending on the stage of the project, there'd be a lot of collaboration. Usually someone will spend hour, two hours of pair programming with people that are working on these projects with them.
Ron Green: Is that, is that daily?
Michael Wharton: Oh, if you can, I mean, from my perspective, the more the merrier.
Ron Green: I agree.
Michael Wharton: But you know, everyone's got their own personal preference on that spectrum. Right. You know, that's an important part of it. And really the, the key there I think is this, you know, we live in a day and age where context switching is just rampant and happens all the time. And to the extent that we can encourage people to try to block in really deep focus on uninterrupted times where they can just work on the same thing for a couple hours at a time. I think that's also important to put some structure around because context switching is just, it's terrible for mental mental health. It's terrible for efficiency and productivity and all that stuff. So that kind of stuff's important.
Ron Green: What about what comes to my mind next is like ethics. So we're dealing with a field that's moving as fast as anything in the world right now. But there are added challenges in this domain because, as you mentioned, the experimental and probabilistic nature of the output of these types of models, I think ethical considerations really have to be elevated more so than in traditional software development. Maybe talk about that a little bit and how you factor that into team development and growth and structure.
Michael Wharton: Yeah. I think leaving space for having ongoing discussions about it is crucial more than anything just because there's so much that you can't really foresee. Luckily, I think the industry is kind of waking up to the importance of ethics and legal risk. It's getting to that point as well. Right. It's a pretty frothy space and the legal side of things too. You have examples that are coming out frequently now that we can point to and say, look, this is very important. This kind of thing can happen. I'll give you an example. There was a guy in California that managed to purchase a car for $1 because they opened this chatbot up to the world without thinking about a lot of the consequences that might come from that. Although that's not necessarily an ethical issue, I think it points to how incredibly unpredictable these systems are and then you have these other higher risk scenarios where a great example is the housing industry. You'd be surprised at how much is tied to just a zip code, how much is implied there in terms of racial makeup, economic disparity, all that stuff's baked in there. You can have these really anonymized systems, but unless you have familiar with some of these ethical issues, you might end up building a discriminatory system without even realizing that you built it.
Ron Green: yeah, without even intending to do so, yeah. Let's talk a little bit about work-life balance. This is something that you and I have talked about offline before many times. How do you manage all of the different things we've talked about but then also have a life? Like how do you unplug at the end of the day and how important is that?
Michael Wharton: It's incredibly important. You know, you end up with these, you end up in these scenarios, I think, where creativity is one of the most paramount resources that you have at your disposal. And if you got someone that slept for three hours the night before, and they're just, you know, working day in and day out, just doing the same thing, working really hard, a lot of times that kind of puts up horse blinders and keeps you from seeing these opportunities for improvement, where, you know, a lot of times, something that normally would take two days, you can come up with some clever trick to help save you almost all that time and get it done in a half an hour. And I think the fact that creativity is such a deep part of that, we need people to be the best version of themselves. We need them to be happy, healthy. You know, we don't want to be on the bad side of their families, we don't want them hating us. All that stuff's incredibly important so that the time we get from people is so well spent.
Ron Green: Yeah, I couldn't agree more. In fact, I remember when I was, you know, earlier in my career, I used to think that it was more psychological. Like, you know, I could work 12, 13, 14 hours and do good work. And as I got older, I realized I was just wasting time. Just being at your best, being rested, and in a good state of mind, you can get more work done in an hour than you can in a day when you're just off your game. And I know that's something that we take very, very seriously at KUNGFU.AI. What about soft skills? So, KUNGFU.AI is a services company, so we are not building a product. We're interacting with people and clients daily. What about the soft skills as an aspect of building and managing teams?
Michael Wharton: Yeah, we like to focus on, we have this trifecta of hungry, humble, and people smart that we throw around all the time. Those are these three core traits that we think are vital to the success of the team. I think that came from Pat Lincione originally. I think so. But that is something that also needs to be baked in our DNA because we talk to clients. A lot of times we'll realize that you can have the most incredible result that's a 0.99 F1 score on a challenging problem that no one's ever even touched before. But when you go talk to someone in the oil and gas space that doesn't know what an F1 score is, and you present this incredible result and you're happy and wanna hang your hat on it, they might say, what does that mean? And it just falls flat. And I think the need for having these soft skills to really be able to communicate the incredible things that you do on a day-to-day basis, it's just as important, if not more important, than being able to just blow it out of the water in terms of technical success.
Ron Green: Man, I couldn't agree more. And again, I think if I go back to early in my career and some of the naiveté I had, I thought if you just did good work, right, just do good work and it will speak for itself. But being able to communicate and understand the nuances of your client's needs are so critical. And having complete transparency and trust are paramount. It's unbelievable. You can build amazing things, but if you don't have that trust, they really won't be successful. Let's talk about maybe challenges on projects where things didn't go straight. You didn't hit the ground running and achieve success. You had major setbacks or there were other difficulties, whether technical or otherwise. How do you handle that? Are there strategies that you use to make sure that you can break through and achieve success on projects?
Michael Wharton: Yeah, I think that strength in numbers and, you know, pulling in all the diversity of the team that we've been talking about this whole time is, I mean, that's your most valuable resource in these situations. And if people on the team are vulnerable, and they have, you know, I would say the courage to raise their hand and say, hey, I'm stuck, I'm having a really tough time with this, can you help me out? You know, that helps you detect those issues early on and then correct for them. So I can think of an example, you know, with a particular client where the team was really focused on the modeling, they were really focused on hyper parameter tuning and model architectures. And they kind of hit a rut where, you know, the benchmark for achieving business value was a little bit higher than their best model to date. And then we had a team brainstorming session, I mean, that's really what it came down to. It's simple, but if you if someone raised their hand and says, hey, you know, we're stuck over here, you bring in this collective, I mean, something special happens when you get so many diverse and talented people in one room and just give them all the same problem to work on. It's kind of fun to watch. And you know, in this case, we did that. And it turned out just a little tweak on the data processing side of things to clean up the data that was going into the model to begin with. That was really all that mattered at the end of the day. And it, you know, blew the metrics out of the water.
Ron Green: I couldn't agree more on the need for that vulnerability. The idea that what we're doing is so complex, and there are so many different technologies and area of expertise. Nobody can be an expert in all of those. And you just really have to trust your team and feel comfortable asking for help. And what I love to see is when even like principal engineers who may have decades of experience are willing to raise their hand and ask for help because that really fosters that team growth because they see the leaders being willing to express vulnerabilities. It's just magical. Okay, well, I wanna wrap this up as we usually do here with just kind of a fun question, which is, Michael, if you could have AI automate anything in your life, what would you go for?
Michael Wharton: Oh man, that's a toughie. I think, I'm sure someone's already automated a schedule somewhere and there's some product that I haven't found, I would say my schedule.
Ron Green: I've seen your schedule. You need it.
Michael Wharton: I need someone that can just automate cleaning. Cleaning drives me crazy. It's my least favorite thing. It's a source of tension around the house, as you might imagine. I need a robot to just come in and clean without me there. It's gotta be quiet. It's gotta come in when I'm not there and just take care of everything.
Ron Green: You're the first person to say that and you're channeling me because that's exactly what I want.
Michael Wharton: All right. Well, there you go.
Ron Green: I love it. Michael, thank you so much for coming on. I really enjoyed this.
Michael Wharton: Yeah, same. Thanks for having me.
