Your AI Pilots Won't Move the P&L. Here's What Will - Adam Nahari, Apollo

[music] >> Hi everybody. Good morning. Um, my name is Adam Nahari. I'm going to share a few thoughts today on driving a successful AI transformation and I want to start somewhere specific, a scheduling desk for a large cargo airline. An aircraft experiences a failure, the scheduler has 15 minutes to re-crew the flight. The process is entirely manual and the scheduler has to consider rest rules, qualifications, where each pilot is. They identify one legal solution and it's not necessarily the cheapest one. Often, they identify no legal solutions, in which case the release valve is premium pay, paying more to make the problem go away.

This happens every day across the entire network and crew is the second largest cost after fuel. Remember the scheduler? I'm going to come back to him. Let's zoom out. AI tools work. Everybody in this room knows that they work. Over 55% of US adults have used generative AI within the last 3 years. At the same point in the curve, the PC was at 20% and the internet was at 30%. Almost 90% of businesses report using AI in at least one business function. And yet, only 6% report significant EBIT impact. Time savings is real. People report it. If you add it up, it amounts to about 2% of total US working hours, and yet almost none of that shows up in the P&L.

Robert Solow famously said, "You can see the computer age everywhere but in the productivity statistics." He said that in 1987. We are seeing the same thing happen again today on a larger scale and faster speeds. Why is that? And what do the 6% that see material EBIT impact do differently? Here's my answer. For every AI transformation that I've led or watched up close, engineering was the smaller part. For every five units of change management, there's only one unit of engineering. That's my mental model. And this ratio is moving. As models get better and better, the engineering share is only shrinking.

Picking the right problem, creating buy-in, driving adoption, measuring impact, that's where the real time and effort goes, and that's where failures tend to happen. So, for the rest of this talk, I'm going to focus on those five units of change management. So, why doesn't the P&L move even though the tools work? I like to think of AI in businesses in these three layers. The first layer is general productivity, the copilots and chatbots that we all use. Because everyone has access to them, gains tend to show up in price but not in margin. They get competed away. The second layer is domain-specific AI. You can think of these as um business function-specific co-pilots or retrieval over your business documents.

These tools allow for faster retrieval and better and deeper use of your enterprise knowledge. And finally, the third layer, core operational AI, these are the vertical systems embedded in the business processes that make your business what it is. So, crew scheduling for airlines is a great example of this. And because you can't buy these tools, this change management off the shelf, these gains tend to remain in margin. Most businesses are investing in layer one, but the businesses that see material EBIT impact are three times as [clears throat] likely to redesign an entire workflow end-to-end rather than just adopting a new tool. They're work They're investing in layer three in addition to investing in layer one.

I'd like to go through the six moves that in my mind are key to driving a successful AI transformation and moving the P&L. Move number one. Pick one or two problems, not 40. Let's start with a P&L and work our way backwards. So, for an airline, that North Star metric might be EBITDA. The four drivers of that metric are routes, which is demand, tails, which is supply, maintenance, which dictate constraints, and crew, which is the cost driver. It's such a significant cost driver that we decided to start and focus there. As I mentioned, it's the second highest cost driver after fuel.

We did not work on 40 pilots at this company. We chose this one initiative that, if we got right, would be truly needle moving. So, on the right, you can see the five things that I look for in a good use case or problem. The first is it moves a line the CFO recognizes. It's measurable with a baseline agreed upfront, not in retrospect. The tech and data infrastructure is ready for the transformation, and the company owns the data. A named user wants it and can champion the initiative, a person, not a department. And this is one of the CEO's top five priorities. Okay, move number two. Reimagine the process end to end. These are illustrative numbers, but the idea is that when you take an existing process and just try to automate it, the gains are incremental.

Whereas, when you redesign an entire process around the new technology, the gains can be truly transformational. So, in airlines, the 10% play would be taking that existing process, scheduling process, that I described to you, and giving the scheduler a tool to make them somewhat faster. On the other hand, the 10x play was building an optimization agent engine that took all of these constraints into account, produced the entire list of possible solutions, and ranked them by cost. And this system was able to do all of that in under a minute. In addition to building that system, we also redesigned the process, the workflow that the scheduler went through.

This took consolidating all of the company data, including flights, crew, maintenance, building this optimization age engine that I described to you, and building a decision support app for the schedulers. All of this engineering work was still the smaller part of this project. The bigger job was getting it used. Move number three, start with the CEO. Before I put any money at risk, the one question I ask is where is the CEO? Have they committed publicly that this initiative is a priority to the company? If they haven't, I don't fund the initiative. The correlation with success is that strong. Without it, the company doesn't necessarily have to say no, but you experience scheduling conflict after scheduling conflict. Data requests take weeks and weeks. Pilots get rescheduled and delayed.

Everybody takes their cue from the CEO. The P&L owner staffs it, the CFO who needs to fund and track it, the CTO who needs to prioritize engineering and technical resources for it. And this alignment wears off over time, so we need to keep renewing it. Move number four, building with the business. Significant transformation requires the P&L owner to drive change. In the airline, the VP of crew scheduling was the co-sponsor of this transformation. We also identified AI champions, crew schedulers, that were embedded in the solution design from day one. They met with the engineering team every other day. They designed the interface together with the engineering team. They were part of AB testing, and they drove this change to their peers.

So, they adopted it because they had a hand in building it. And likewise, talent, we recruited new leadership to build the platform and the AI at the point of decision-making. We hired a chief information officer, a VP of data and AI, and a senior director of data science. These folks had both the technical expertise and the domain expertise in airlines, so they understood how schedulers worked and could work with them to design a tool that fit their needs. So, the takeaway here is that the organizational investment exceeded the techno- the technology investment. Move five, put the tools in everyone's hands. I was in a AI workshop with 30 or so other executives, and we were asked to think how we might redesign our business with AI.

But three out of four people in that room had never used any AI tool until that week. It's impossible to redesign a workflow with AI if you've never used AI. The intuition on where value lies and which use cases can be enabled with AI only comes if you're using the tools. Experimentation with the tools is the absolute minimum requirement to drive this type of transformation. And move number six, set the kill metrics first and let the CFO keep keep score. Before we start a pilot, we write down what are the kill metrics that will determine if a pilot continues or stops.

Three types of gates that we used in this initiative. The first is a time gate. We gave each stage 90 days and if we didn't see value in those 90 days, we would stop the initiative. The second is a metric gate. AB testing at each stage relative to a baseline that we decided on in advance. And the third is an adoption gate. Were the crew schedulers using this tool at over 50%? Over 50% of crew schedulers. The other important test that we put in place, working hand-in-hand with the CFO, was calculating whether the cost of this initiative at scale, including compute and token and the individuals involved with this work, was significantly lower than the impact that the initiative would generate. Many pilots die at this stage because they don't pass this test.

So, let's recap the six moves that I just described. First, you want to be thoughtful about choosing your problem. You want to pick one or two things to tackle, not 40, and you want to reimagine the entire process end-to-end, rather than building on processes that may already be obsolete. Second, you want to choose the owners, starting with the CEO and building with the business. Third, you want to make it stick. Make sure that relevant stakeholders are experimenting with the tools. Set kill metrics in advance before starting a pilot and work with the CFO on the unit economic scalability of your projects. While the tools are available to everybody, models are available to all of us, the ability to absorb these models is a competitive differentiator, and the businesses that are able to do that and truly take advantage of the models as they continue getting better and better, are going to thrive.

So, before it took 15 minutes to find one answer, now it takes less than a minute to find the answer. Thank you. >> [applause]