#282 Tech Mastery with Gene Kim: Simplify, Slow Down, Innovate

0:00:01 - Mehmet
Hello and welcome back to any episodes of the CTO show with Mehmet Today I'm very pleased, very honored, to have with me joining me from the US, gene Kim Gene, I know that many people know who you are. You're a famous guy, I know this, but still can you tell us who is Gene and what you do? 

0:00:20 - Gene
Yeah for sure, and thank you so much for having me on. So my name is Gene Kim. I've been studying high performing technology organizations for 24 years 25 years now and that was a journey that started back when I was a CTO and the technical founder of a company called Tripwire in the information security and compliance space. And so we had we called these high performers those organizations that had the best project due date, performance and development, the best operational reliability and stability in ops, as well as the best posture security and compliance. So we want to understand, you know, what are those amazing organizations doing differently, so we can figure out how other organizations can replicate those amazing outcomes. And so, as you can imagine, in a 25 year journey there are many surprises, but the biggest surprise was how it took me into the middle of the DevOps movement, which I think is just so urgent and important. I mean, in a time of unprecedented disruption that all of our industry are facing these days, nothing matters more than being able to adapt quickly, deliver value quickly and learn quickly. 

So what I've been working on for the last three years is working with Dr Steven Spear at the MIT Sloan School of Business, really trying to understand this question of what is in common between DevOps and Agile and the Toyota production system and lean resilience engineering, and our conclusion is that there are all incomplete expressions of a far greater whole, and it was the most intellectually challenging thing I've ever worked on, but it was also one of the most rewarding, because what we found was that there's actually only three mechanisms of performance that allow organizations to go from good to great, or maybe not so good to great, and so it's just very much validated so much of what I've learned in the DevOps journey, including the state of DevOps research, which is something I did with the, with Dr Nicole Forestman and Jess Humble, and that was a cross population study that spanned over 36,000 respondents over six years, and we find that there are some things that really matter for performance, and we call them cultural norms, technical practices and architectural practices, and all of those came out in this person monist theory performance that we put into this book called, as you mentioned, wiring, the Winning Organization that just came out a couple of months ago. 

0:02:31 - Mehmet
Fantastic and, of course, we're a little bit deep dive as much as we can, because the topic and I like the concept honestly, because it's like mentioning few, I would say, terms that for some people looks like complex and what we need to do about this, but it ends up something much easier than this. But let's start because I know and thank you for sharing with me, the team shared with me the ebook and I had the chance to just get the important parts. So there are mainly three themes, as you mentioned. I want to start with the first one, which is you talk about slowification, right so, and you mentioned that it's a key, deliberate problem solving. I want to understand more about this because I think we saw before like slow down, to go fast and these kinds of concept, but I want to understand from, of course, how we can leverage it in technology and all the topics you mentioned. What is it and how we can use it and if you can give us, like, maybe a real use case about it. 

0:03:51 - Gene
Yeah, for sure. Yeah, and I love what you suggested, which is yeah, I very much agree with you. I think anyone can take something simple, make it appear complex, and it is a lot harder to take something complex and make it genuinely simple. And so our observation is that whenever you see an organization transform, there are really only three mechanisms of performance. The first one is a slowification, and our observation was that a surprising kind of conclusion that we came to was that there's no single word in English that really embodies this concept of slowing down to speed up. I love that you picked up on that phrase. 

You know the notion of making a short term investment for a longer term gain, and so there are many adages that suggest this, like stop sawing to sharpen the saw, or slow as smooth as fast. And the root of this concept is that when you are in the worst time to solve problems is when you're under a lot of time pressure, when you can't undo, when every mistake that you make is highly consequential and potentially causes catastrophe. And this makes problem solving very difficult, because if you can't undo, if you can't get multiple trials at bat, that's a problem, because learning is inherently experimental, it is iterative, and so the worst time to solve really difficult problems is in production, and so the notion of slowification is that you move that problem solving backwards in time so you can do it in planning and preparation, and so whenever you see someone doing something incredible under when there's incredible danger, when they're having to deal with novel situations, you just know that they had to have invested time in planning and preparation. So I think in our DevOps space, one of the famous examples was probably the first AWS massive outage in 2011, when ADOS East went down, which is still probably one of the largest availability zones, and so, because they put most of their big customers there when they went down, their big customers went down, and one of the big surprises was that Netflix stayed up, and so they were famously running entirely in the AWS cloud even back in 2011. 

And so the question is how? And so the answer was real in a blog post that they put out some days later, which described Chaos Monkey, which described this audacious set of design decisions they had made some years back, which said we can have no single point of failures, and their biggest single point of failure was AWS, and so to make sure that they could actually survive even massive disruption. In AWS, they had something called Chaos Monkey that would randomly kill VMs in production during the middle of the day, and so, as you can imagine, that's happening all the time, and developers are responsible for not just developing the features, but also running in production. They became very, very good at making sure their services that they created were indeed resilient, and so this explains this. Massive investment in planning and preparation is what allowed them to survive these outages in production that affected everyone else. So this is obviously now very, very common in the cloud engineering space as an architectural pattern, but that's not possible without slowification. How am I doing, mehmet? 

0:07:26 - Mehmet
Fantastic. And just to the point, back in the days, if you tell someone you're going to kill a machine in production, they will be like, oh no. And funny enough, we still talk about site reliability, we talk about how we can make sure services are up and running, and it's an interesting approach about the slowification. 

0:08:00 - Gene
But I think it's related to the next one also as well, which is more of a simplified Mehmet, yeah, so you bring up a great point right In bed within site reliability engineering, sre, is this notion about error budgets and the notion that you can't basically it boils down to developers can't do the work on features. They have to allocate some time for the non-functional requirements. And so I think where so many engineering organizations let's not call it engineering so many organizations, engineering and products get themselves in trouble when they are not investing enough in slowification activities, so that could be automation or testing or refactoring. I love Kent Beck's new book called Tidy First, the notion of just making sure that things are tidy so that you can build more features, not just now but forward in time, and so the notion of technical debt reduction. I think that too is also a phenomenal example of slowification right Investing in activities that may not be immediately visible or customers may value, but is absolutely critical for long-term health of the organization. 

0:09:15 - Mehmet
Absolutely. Now the second one, which is interesting also as well, because with slowification we create the conditions as you were describing, but now with simplification, you talk about shaping problems to be simple, controllable and easy to iterate on. If I want to elaborate from your side on that, because some people they say agility is more valued than simplicity, so how we can really keep things simple while staying agile. So what you can tell us about this? 

0:10:04 - Gene
Yeah, absolutely so. The first domain of slowification, that's shifting activities so that we're not solving our most difficult problems in production but we're doing instead in planning and practice. The second one, as you mentioned, is actually changing the nature of the problem so that they are easier to solve. And so, if you can imagine the worst case problems to solve, not only are you doing them in production, but the problems that you're trying to solve are highly intertwined so that you can't solve just one piece of the problem without affecting everything else, and so one small mistake in one region causes a massive mistake everywhere else, because everything is so tightly coupled together. Another mark of non-simple problems is that you don't have independence of action, that no one. To get even small things done, requires communicating, coordinating and synchronizing with scores or even thousands of other people. And so this is how organizations get stuck, because even if you know what to do, you can't do it because you have to coordinate with so many other people in the organization, which actually means you probably have to escalate up many levels in the organization. So the simplification mechanisms are let's see here there's one to make more incremental. So incrementalism should probably resonate with anyone doing agile. We want to do things not in large batches, but in small batches, small incremental pieces. The second and then the next two are just so amazing to me. So the second one is modularization. We actually partition the problem so they can be worked on independently, and so the most amazing example of this is the big Amazon API rearchitecture that happened in the early 2000s, and so the problem that they had to solve was that, as they grew from just selling books and music to 35 other product categories, their ability to ship software slowed down. They went from being able to do potentially hundreds of deployments per year to only doing tens of deployments per year. In fact, most deployments in the early 2000s did not finish just because something would go wrong in the testing or deploying process and they would have to abort the deployment. And so, with thousands of developers, this was a huge problem. 

In fact, one of the things that Dr Werner Vogels and current CTO of Amazon, he said everything required a ridiculous amount of communication, coordination and synchronization. He described how the Amazon digital team so that was Kindle, video, music for them to be able to fulfill their products to the customers, they would have to provide a shipping address for a digital product. That just didn't make much sense. And so he described how they would go to the other you know, say, 50 different ordering teams and ask that they change requirement for a physical shipping address, and they would say we didn't budget for it, and so now they're stuck. 

And so I think we've all been in this situation where everything, even small changes, require just incredible amounts of coordination cost, and so the the countermeasure for this was a famous two-peat the team where Jeff Bezos said that we want every one of these pizzas so every one of these teams no larger than can be fed by two pizzas to be able to be independently deployed, to be able to independently solve Amazon's largest problems on behalf of their customers. And so that's what resulted in the need to re architect their services so that there were hard modular boundaries between them, so that every team could independently Develop tests and deploy value to their customers independently, with no need to communicate and coordinate. And so the result of this is astounding. But you know they went from you know a handful of services to hundreds of services to, eventually, you know thousands of services, and you can see this in their deployments per day where? But you know, they shocked the world by 2011 2011, when they announced that they were doing 15,000 deployments per day, and in 2014, they described how they're doing a hundred thirty six thousand deployments per day. 

And all of this was made possible by modularizing their systems, and that create independence of action to make sure that small changes remain small, as both are rippling out into the entirety of Amazon e-commerce, and what's astonishing is that this leads to the third mechanism of simplification, which is Linearization. And so linearization does for sequential processes what modularization does for parallel processes. And so, if you take a look at the Toyota production system, or even an assembly line, that creates the same sort of independence of action. And so, in our world, that would be, like you know, continuous integration and deployment pipelines that not only just automate a series of tasks, but it creates independence of action in between, say, the testing engineers and the build engineers and the security engineers, because they can now work independently of each other. Mehmet, how am I doing? 

0:15:05 - Mehmet
You're doing fantastic team, amazing. So so to put back and I like to do this from time to time so first we needed to Do the slowification to create, as we said, as you explain, kind of a Testing right, so to see how we'll act in real time right. And then now we talked about simplifications, to, you know, dissect them into like smaller pieces, so we can see them simple, controllable and easy to iterate on. Now, and which I think the third one would be like very obvious, but it's how we ensure that this problem actually would be Reacting to it and not ignored. And I think this is where you know the third concept you talk about in the book the amplifications come into the picture. So want you to explain that as well. 

0:16:04 - Gene
Yeah, absolutely. And so amplification is all about how do you create a System? Or even weak signals of failure or weak cries for help are amplified so they can be decisively and quickly acted upon, so that those problems can be, you know, detected and corrected quickly, but also prevented in the future. And and so I would put this on a spectrum, we can imagine that, so we can that has a very strong amplification system where, sitting, even weak signals are generated, transmitted, received, acted upon and those corrective actions are verified. Or we can imagine the opposite system where even, you know, the most urgent cries for help are Suppressed or not heard or even extinguished entirely. And I think many of us know we have, you know, we've been in situations where we are in a system where, you know People were afraid to ask for help, or ask for help, well, we're never responded to. So this is not a great, this is also not a great situation to be solving problems in. And so amplification is this, this it describes how, after we're using information theory and signal theory to show that, you know, in this, in well managed systems and in a well wired organization, now only our leaders creating the Space for slow vacation, they are changing the system so that they're simpler, but also this all signals in the system are Amplified. And I think in our world yeah, I think that's very much shows up in the resilience engineering community, in safety culture. 

You know, famously, at Google and so many great engineering organizations, they have this Culture of blameless postmortems, the notion of you know. Whenever something terrible happens, engineers are trained to lead and participate in these practices that allow the creation of Chronology of what actually happened. What did people see? What were the mental models that caused people to react in the way they do? Because you know it, within there are the answer of you know, how do you make it easier and safer to work in in the future? And so my friend Randy shop he was an engineering director for Google App Engine for many years at Google and he described how, you know, they would have these post-incident reviews that they would share Sternly, and he described the value of them, that that Googlers you know everyone loves these Outer stories. You know, whenever there's a customer impacting incident, people would be waiting breathlessly for the post-incident review to be published, and it wasn't just for entertainment purposes, but it's also for education. And he described this amazing dynamic that happened, which is that, you know, by doing this rigorously for every customer impacting incident, they eventually reduced the number of customer impacting incidents to the point where they didn't have enough to learn from, and so they they changed the threshold so they would do this for not just customer impacting incident, but team impacting incidents. 

You know, places like where you know we had seven safeguards to prevent a customer impact. Six of them failed. So how do we Detect when that happens and, better, prevent it so that we can never be in that situation? And so this shows another one of those amazing dynamics that happen, you know, when there is genuine amplification, and so the combination of slowification, simplification and amplification is what, in the book, we're calling this layer three, wiring, that it is. Layer one is kind of the work in front of us. So this, in technology, this could be the code that we're working on, or could be the Binary that's actually running in production. Layer two is the tooling and instrumentation. So there are often teams, platform teams, supporting those. And layer three is how do teams actually communicate and coordinate with each other? 

What are the processes and the architectures that dictate how organizations you know how teams actually interact within the organization, and I love this quote from Winston Churchill, who said we shape our buildings and thereafter they shape us. You know so too. It is with this organizational wiring, this combination of the extent which creates simplification, slowification, amplification we shape the structures and thereafter they shape us. And so now the lesson we're really trying to create in the book is that we're saying that leaders are responsible for the wiring of their organizations and that so much dictates. 

In fact, I will. Our assertion is that this dominates how the organization Performs, and this is very much validates, you know, my six years of involvement in the state of DevOps research, where, you know, we call them technical practices, architectural practices and cultural norms. That maps so cleanly to the notion of simplification, slowification and amplification, the notion that architecture dictates the degree of which teams can work independently and can work safely with each other. Amplification is is there enough signals being generated that we can act upon to prevent, detect and correct? And all those create and require slowification Because they're not free right, we actually have to create them. And so we think there's such a personal expression of what every engineering leader has gone through, and hopefully this gives a common language that they can communicate with their business counterparts to their leadership. You know, to get from here to there. 

0:21:31 - Mehmet
That's great. Now you mentioned something about leadership gene and I'm curious to know how they can start so. So, so what's the best approach to start implementing these concepts into the organization? And does it matter if the organization is still small? Maybe they are start up, scale up, or maybe a very well established organization? So are there like different, I would say, portions for each kind of of of organization size? And you know what's our, again, the best practices to start implementing that? 

0:22:10 - Gene
Yeah, what a great question. In fact, let me start with the second one is you know, does this is relevant for both large and small organizations, and I would say absolutely yes. I mean, I think everyone has probably experienced the challenges of growing. But even managing smaller organizations is very challenging, because the predominant variable is not just size but the number of functional specialties. So I think life was a lot easier when, you know, we didn't have dev and ops led alone. Dev ops and information security versus dev and QA and operation security. You got the business and now you have generative AI right. You have this ever increasing number of functional silos. I mean, I loved how you're focused on healthcare. 

One of the things I one of the many things I learned, but one of the things that just blew my mind working with Dr Steven Spear, was his observation that healthcare systems are so difficult to manage now compared to, say, the 1950s, where it was actually a much simpler system. The 1950s hospital essentially had only two functional specialties, you know, doctors and nurses and there was almost no technology, which means that you don't have these teams supporting these different technology systems within the organization. Now, in the modern hospital system, you have scores of functional specialties, even within the clinicians, you have, you know, scores of functional specialties, plus nurses, plus pharmacy, plus supply chain. And then if you have technology, and the result is that often if a nurse on the someone on the nursing floor has a problem with, say, the number of pills side, the number of pills at the bedside or torn gloves and supply chain, they often have to escalate up eight levels in the organization, you know, from nursing to the chief operating officer, down to pharmacy or down to supply chain it involves technologies up eight over one to the CIO and down to the technical team. And so as you increase the number of functional specialties, the sophistication and responsibilities as layer three increases. And so the same thing has happened in technology. Right, we didn't. You know there was a day when you just had, you know, developers who owned operations. Now you have scores of functional functional specialties. You know you've had on your show, you know, not just information security specialists but AI specialists, you know. So the job of the engineering leader is far more difficult now than it was, say, 20, 30 years ago. And so so my claim would be you don't have to be running a very large organization before you are. 

You can find yourself into very difficult problems, whether it's architecture you call it architecture, what would you call it? Call it wiring. So the where to start, boy, what a great question. So I'll just go one by one through the three. So you know, you have a slowification problem when you are running into problems in the live production environment and it's bringing up questions like oh my gosh, I wish we had spent more time planning and preparing. We need to do more time doing kind of drills and risk analysis and tabletop exercises so that we can better build those routines, to build those habits and those playbooks so that they're available to us when we are in the fast moving production environment. 

So, listening to those signals or from the development perspective, you know we find ourselves constantly running into problems. Why we're trying to do our development when these are things that we should pause in production right, and production doesn't mean live in production. But, you know, take time out of our daily work right to make the daily work easier. The same thing with simplification. If we find ourselves spending a lot of time having to communicate and coordinate and synchronize and prioritize together across different functional specialties, this is a great signal that there's more need for specifically simple, you know, modularization, or linearization. So these are what gave rise to the famous rearchitectures, whether it was at Amazon or at Google, or the creation of CICD pipelines. These are all created, you know, by listening to those sorts of signals and amplification spans. You know, the entire domain is the leader has to be able to be able to hear those signals right, which actually must be generated right, and so and also decisively act upon them. 

I love this. I was talking to Dr Ron Westrom, who created the Western Organization on Typology Model, which so much influenced the state of DevOps research, and he had this notion of the socio-technical maestro. He said there's five characteristics of great leaders. They're great in the large, great in the small. Let's see, actually, let's start again. They're high energy, high standards, great in the small, great in the large. They love walking the floor and when he said that, it just it struck me it's so beautiful because I recognize that. I think that is the mark of a great leader who can, is very highly attuned to what's going on in the organization. They love talking to people on the front line doing work right, and they are. They hold themselves responsible for creating the organizational wiring you know. So the everyone in the organization can do their work easily and well. Mehmet, how am I doing? 

0:27:31 - Mehmet
Fantastic. Now I have a little bit loaded question for you, jean, and feel free to answer it the way you want. So, how we can, because you know, when we bring any new kind of processes to the organization, people will start to ask okay, how are we going to measure this? Because some forces will come and say, do we really need that? Right, and then people will say, but hey, guess what? More processes, more procedures means we are slowing down. And how are we going to be like innovative and agile? So I know I say a lot of the question, but how we measure, how they can overcome the objections and how we can prove that it doesn't conflict with, you know, innovation and agile and all the beautiful stuff that happens. 

0:28:26 - Gene
Yeah, I love that and I would say the big learning. For me, having studied the DevOps movement for nearly a decade now, it comes out to a very simple question that leaders should be asking, which is can everyone in my organization do their work easily and well? And you know so. Is there a lot of toil? Is it difficult or dangerous to employ in production? Is it hard for me to get feedback on my work quickly? Is it difficult for me to actually see whether I did my work well? Is it difficult for me to do small things where even small things require a tremendous amount of communication, coordination, or that's dominating my effort, and I think when people express that it's the leaders responsibility to do something about it. And we're saying there's really only three things you can do you can slow-fi, simplify or amplify. And so the objection. If the objection is coming from within the organization, there should be nothing more appealing right than having the leaders say there's no more important signal for me as a leader to hear the problems that you are facing in your daily work, and my job is to remove obstacles to enable you to do your work well. So this is something that I found in every one of the case studies that I studied, leading up to the DevOps handbook, which, along with my fellow co-authors. This comes up in the 19 conferences called the DevOps Enterprise Summit that I've done since 2014, which we've instantly renamed to the Enterprise Technology Leadership Summit, and it also shows up in every other of the 25 case studies that we did in Wiring, the Winning Organization. Is this notion of the leader enabling? In fact, there's even a movement called frontline empowerment that's been around for decades, which is not about the leader being nice. It's about leaders being able to do things for people, actually doing the work right that they cannot do alone, either because they don't have the authority or they don't have the scope of responsibility. That spanned multiple silos, anyway. So then, I think the other objection that might hear is from their peers or from their leadership. From the perspective of technology leader, it's like we don't want you creating more processes, creating bureaucracies. It's absolutely not about that. The goal is to create organizational performance. 

In the state of DevOps research, we found that you can measure performance from the technology side in terms of deployment frequency, deployment, lead times, meantime, repair, repair and change success rate, but it was highly correlated and predicted organizational performance to what extent these organizations were exceeding profitability, market share and productivity goals. To what extent were they achieving their organizational and mission goals? But we also found that those organizations had the highest employee net promoter scores. What percentage. They were twice likely to recommend their organization as a great place to work to their colleagues and friends. So I think that the evidence is so decisive that says you know, these things are not just about leaders being nice. It's about enabling people to do their best work, which actually helps organizations achieve their objectives, regardless of what industry they're in, regardless of what company size they're in, whether it's profit or not for profit. And so for me it's such a satisfying set of tools to be able to sell not just within their organization but across and up as well. 

0:32:11 - Mehmet
Amazing. Just quick one, Jean. Like who owns it usually? I mean, who can own this initiative? Is it like the CTO, the VP of engineering, from your point of view? 

0:32:23 - Gene
Yeah, you know. So you know we use the term organizational wiring, or we also call it social circuitry, and so we don't we don't mean this just figuratively, but really metaphorically we're saying that the organization is set of social circuits, and so you know circuits, whether it's hydraulic or mechanical, or electrical or software, you know they are hierarchical, and so this is a responsibility of all leaders and leaders at all levels, and so this could be an architect or team lead, it could be a mid-level manager, it could be. You know, every leader is responsible for some portion of the organizational circuitry, and so, really, you know, I think the there are two ways I would stress that are important for this. One is that there are leaders at all levels, and so just because you don't have the highest level title does not abdicate you from your responsibility to help you. People in your organization or your team do their work easily and well. 

But especially for the most senior leaders, there should be no mistake they are ultimately responsible for the wiring of the organization, and so one of the things I'm just so proud of is that the forward of the book is written by Admiral John Richardson, so he was formerly the chief of naval operations for the entire US Navy. He's currently on the board of Boeing and Exxon. These are safety critical organizations and he described how the book resonated with him and I'm hoping that that Ford describes to every leader, whether it's the CEO, the CFO, their responsibility of the wiring of their organization for highly consequential organizations doing important work that affect hundreds of thousands of people. You know that this is something that they are responsible for, but also I'm hoping that when people read this book that it will resonate with them that the problems we throw in the book are relevant to them. You know whether their teams are 10s, hundreds or even 10s of thousands of people. 

0:34:33 - Mehmet
Amazing. I hope the same. So, but yeah, like it's fantastic because actually you're giving you know the tools and, let's say, the know how of how they can. And you mentioned something you know which is close to my heart. So when I hear you know customer satisfaction, nps score, you know this is what it's all about. This is why we do what we do, whatever is our position in the organization we are the CEO or we are a site reliability engineer, doesn't matter. Indeed, so, of course, we touch base very little bit even before we start the recording today, when we were talking about the AI, so kind of let's have like future predictions, right? So, based on the research you've done, and you know, of course, the trend that you follow within how organization performance is being you know heading today. So how do you see you know the integration of AI on an automation within the business processes, helping you know actually to leverage you know these simplification, amplification and slowification on having better performance. So what is, what is your take on that? 

0:35:52 - Gene
Yeah, yeah, yeah, I think it's for me. I find it somewhat amusing. Well, first off, let me just say what's happening in technology today with gender via AI. I make no claims that I can predict the future. I make no claims that being able to prognosticate what's actually going to happen, but I do find it funny that some people say that engineering and software people will go away. Right, I think that the job of managing complex software systems and building them will somehow get simpler, because to me that just does not resonate at all. 

You know, if we can increase the productivity and help developers do work and bring down the cost of production, what generally happens is that, if we can, say, have the cost of producing software, we're going to have twice as much software to build because we can now do them for a problem that could not have justified the investment before. Secondly, the fact that we have such disruption in the way that is that software is produced means that the job of the technology leader has just gotten more difficult. And so, just how, just like how information security being integrated into technology value stream made life a little bit more difficult for actually a lot more difficult for engineering leaders, so too does the generation of AI capability, because now we have this enormous rearrangement of work that has to happen Before we had, say, the data and ML people working at the very left. We have to have development shift left there, and we actually have to shift all of ML ops all the way to the right so they can detect and respond to things actually happening in a live production environment when customers are directly interacting with these models. Right, you know that means that they have to. These people have to be on the front lines, and so I think that will be very challenging. Just like developers were, all you know might have been all for DevOps until they learned that they might potentially have to be on page rotation, the same thing can happen, you know, with the AI and ML people, and there's actually probably going to be another silo created, you know, called AI engineers, who are able to bridge the gap between traditional development and you know all of the things that you know AI is making possible. So you know anyone, so the opportunities in front of us are so vast. 

Personally speaking, I haven't had this much fun and technology in decades, and it's just been so much fun learning to see what gender AI can and can't do and it is my, I guess I'll make a prediction is that, in fact, one of the things that you'll see in the that you saw in the DevOps Enterprise Summit presentations and enterprise technology leadership presentations, is we're having a third of the programming reserved for the gender AI, just to see how leaders are responding to the gender AI challenge. Now we're going to be just like in the early days of DevOps, where we brought in these kind of unicorns to show what, just shows what's possible. What does the frontier look like? You know? Blow us away with the theories of. You know, we're doing 136,000 opponents a day. 

What's the equivalent in the AI space? Just to help technology, to help us as a technology leadership community, see what is possible right, so that we can see, you know, which of these patterns are applicable, you know, to helping the generation that went in the marketplace. So sorry for that jumble of concepts. So if I boil it down is not getting easier, it's getting more challenging, and so the wiring actually has to do more than it did before. 

0:39:36 - Mehmet
No, absolutely, and I think one of my guests multiple guests, they mentioned the same thing that of course, I will not replace developers, it will not replace software engineers, but you will have software engineer that leverage AI to their benefit and you will have the ones who would, you know, we'll have to do the double work to finish what they have to do. So, 100% on this, and you know, when I'm speaking with some people who have ideas and if you see how they are thinking about it, whether in DevOps or even like some other engineering related things, it's fascinating because you know, just imagine, you still need the human brain, right, so you still need this engine to come up with the idea. But instead of me doing and let's say, in the DevOps sense you know, I'm not a DevOps guy by any sense, but you know, I know the concept let's say, if I want to have a CI CD environment, right, and I need to go prepare it, blah, blah, blah and use maybe something like Terraform or any of these other tools, so now I can go to the AI, hey, go use the, go, find the best tool that can do for me, 123, and come back and spawn the environment to me, right? So I think this is where the power of AI it's like about reducing the time and maybe reducing the cost also as well to solve the problem that we have. So I'm excited to see what's hidden for us in the future. 

So, yeah, 100% on that gene. And, as you know, we've come to and to the end. And because you know you had your, the required your, the startup that you started, you were the city also as well. So if you want to reflect back and you know, do you, do you wish that you had this research that you have done later? And what do you advise? Like fellow, you know, founder city owes that they might be on the verge of starting them. 

0:41:44 - Gene
Next big thing yeah for sure, and yes, I. That question deeply resonates with me. I cannot tell you how many times I, in the process of the research, where there was some sort of finding that you know it seemed academic and it was very, it was exhilarating and I was like, oh, this makes so much sense in terms of why this was impact performance that much. And then there were a couple of moments where then it really hit me. I was like, oh my gosh, it's like not only, not only does it make intellectual sense, it has this emotional impact on me saying, oh my gosh, I made the wrong decision at Tripwire, whereas that was from 1997 to 2010. Let me just share with you two stories. One of them was in 2004 when I watched with one of our lead engineers and one of our product managers. We were watching and our CTO, our chief architect, we're watching a customer use our product for the first time. There's a certain routine operation that we respect them to do weekly and it took them 62 clicks. The whole time they're apologizing, saying, oh, that's probably a better way to do this. I cannot describe how it felt for me. I felt like I was going to throw up. It was the worst professional experience I've ever had in my career just because you're watching someone struggle to use a product that you helped create, that we were ignorant of just how difficult we had made it for them Setup procedures took. In one person's case, it took them 3,000 steps and they had to do it all over again. Again, they were saying there's probably a better way to do this. We were thinking, no, there's not, because we're terrible people. This is actually what led us to all get trained in user interaction design. I got trained at the Cooper University. Anyway, there's no more important activities than watching your customers use your product. It was so important for us to fix those things because we had been building this product for years without any of its feedback. It's called that amplification. 

The second story that hit me. This was probably in 2013, when we were doing the state of DevOps research, which was the first year that I was doing it with Jezz Humble. We found that one of the key predictors was continuous integration and deployment processes. But specifically continuous integration. Do you have daily builds, daily tests or, I think, if you fast at the minimum, daily feedback on your work? I remember chuckling about this like, oh, that makes so much sense. We found this signal in the research. And then it hit me that around 2000 and about the same timeframe, 2003, I remember this problem that we had where our cruise control server broke at Tripwire we had to make a decision Do we hire a build engineer or do we hire another developer? I was on the side of hire another developer because it's all about working on features. 

In that moment in 2013, it hit me about how bad of a decision that was. It just made me recall the story that we started hearing about in 2005, 2006, is that we used to be able to ship back then, a CD or DVD every nine months. After we lost our build server, we were only able to ship once every year and a half because the cost of the code integration, the testing effort, was so high that we couldn't afford to ship every nine months anymore. We essentially have the release frequency because we had to amortize it over more time. All of that could be traced to our inability to get daily feedback on our work. I have to believe that this had a material impact, negative impact on our ability to compete in the marketplace. We filed to go public in 2010. We never made it public. I can point to many of our competitors who did. I know that they had better engineering processes than we did. The story in this case? It took me eight years to realize just how consequential and negatively that decision made. 

The better decision is that you have to have certain infrastructure there to help developers do their work well. That's the build system, the test system, the ability for everyone to do their work easily and well. Again, this is amplification. This is about linearization and simplification. This is about slowification. As leaders, I think I'll give you two unsolicited pieces of advice. One is watch your customers use the product. You will learn so much from doing that. Two is look at your internal practices and ask yourself, or ask your engineers To what extent is it possible for you to do your work easily and well and get fast feedback on your work so you can find problems early, fix them early, or are you letting them all pile up? You have to fix them all at once at the end of the release? Life is never good when you're doing that. I have to imagine this resonates with some of your own experiences in a moment. 

0:47:12 - Mehmet
Yeah, but I have two comments. One is fun, I'll keep it later. I think it's underrated Because I used to work as a solution consultant for some companies. I was telling the product management team we are the guys on the ground. You need to see how the customer used the product, because sometimes customers are smart, so sometimes they ask why they decided to design it this way. What was the logic behind taking the decision, let's say, to put this item before this item in the menu, or why, when I do this, it's behaving this way? Of course I'm not the guy who wrote the code, but I have to do some. Yeah, but 100%. And I've seen the companies who actually go and they ask the customer for feedback and even, as you said, maybe someone from the executive team on, whether it's the city or the product manager or someone even maybe a software engineer. It makes sense for them. Fun fact, maybe for the people who are maybe new generation then me and Gene, I said what you have to ship updates on CDs. Yes, guys. 

0:48:29 - Gene
I remember. 

0:48:30 - Mehmet
I used to have to go and wait for the Microsoft service patches coming to the city to do this. So this is before, because the internet was slow so you cannot download these things. So yeah, 100% yeah, but it's like the good old days. I would say Anything you wished. I have asked you, and did I miss anything? 

0:48:57 - Gene
No, I can't think of anything, but I would just love to describe. They say the goal of science is to explain the most amount of observable phenomenon with the fewest number of principles, confirm deeply held intuitions and reveal surprising insights. For me, working on this book for the last three and a half years with Dr Steven Spear has done that In fact, even in the general value space, they say that all LLMs are large language models. They're form of compression. I think the same concept like how much can you do express with the least? And for me, what came out of the work Wiring the Winning Organization. What I'm hoping it gives leaders is a small number of principles to span every domain of work. And it's not just software but technology, it's engineering, it's manufacturing, any system where you have to integrate the work of many functional specialties towards a common objective. You see these phenomena work. I'm hoping that this book not only confirms the intuitions of technology leaders but actually gives a common language for that technology leader to talk with their peers and even their bosses. 

0:50:07 - Mehmet
I have no doubt that they will benefit from it. It's available, I think, on Amazon right. 

0:50:14 - Gene
Yup at your favorite book retailers. 

0:50:17 - Mehmet
Favorite book retailers. So for the guy in the US, in Canada, you know where to go. For the people in the other part of the world, you can find it. I checked before we started so it's on Amazon. You can find it there. For people here in Dubai, in the Middle East, it's available even in the local Amazon as well, so you can grab your version. So, Dr Jean, how can people find about you and about the work you do? 

0:50:44 - Gene
Oh, the best way to reach me, I think, is these days it's probably LinkedIn. I'm also on Twitter at real-gene. Kim but I would go to LinkedIn and I'm available there and posting content more than I have in years Okay, great. 

0:51:00 - Mehmet
So I will make sure two things I will put your LinkedIn profile in the show notes and I will put also the link to the book. And you know it's one of my most enjoyable episodes I have ever done. I learned a lot from you today. It's fun. You know, I always like when I speak with someone like you, jean, who is able to take a very complex and this is where we started actually to take something that was not complex by any means, but make it fun and everyone can understand it. So, even if I don't have enough background, that was very useful to me and I think it will be useful for people in the domain. 

So thank you very much for sharing your knowledge and experience with us today, and it was my pleasure to have you on the podcast. And yeah, so this is the way we end. If you are first time listening to me on this podcast with Jean today, thank you for passing by. Make sure that you subscribe on any podcast app you use or available on Apple Podcasts, spotify, all the others, and if you are watching this on YouTube, so subscribe and don't forget to tell about this podcast to your friends and colleagues. They will find something useful for them and, as usual, thank you very much for the comments, for all your questions and all your suggestions and taking them into consideration. So thank you very much. So thank you again, stay tuned for the next episode and goodbye.