Total Network Operations with Scott Robohn

Philip Gervasi: The funny thing about networking is that pretty much everything we do with computers today relies on it. I mean, most of the applications that we use are delivered to us over a network. And, of course, all the online activity like banking, and scheduling, and entertainment, and shopping is all done over a network. Even the way that AI workloads run today relies completely on a really, really serious data center network. Now, the thing is that this network, this system is made up of a ton of many, many parts. So many different components, services, and even components that we don't usually even think of as part of the network. So how do we understand what's going on with this system? And how can we actually run this system and make sure that applications get to where they need to go? This is the idea of total network operations which is sort of a network operations framework being developed by Scott Robohn, a veteran network engineer, veteran trainer, and a co- founder of the Network Automation Forum. And in this episode we'll be unpacking what total network operations is and why it's pretty much necessary to run most networks today. My name is Philip Gervasi and this is Telemetry Now. Scott, thanks for joining today, really appreciate it. And it's really been great to get to know you over the last few months or so, especially with my interaction at the Network Automation Forum, having attended AutoCon in Denver recently, having some really good conversations in the hallways off to the side. Actually, wait, I take that back, we met prior to that a year ago.

Scott Robohn: I was going to say, do you remember, I actually reached out to you about producing podcasts, and that was our very first conversation I think just about a year ago.

Philip Gervasi: Yeah, absolutely, I do remember that very much actually. So and here we are. So thank you for joining. Before we get started though, I'd like to give you an opportunity to introduce yourself to the audience. I think folks are probably pretty familiar with who you are now because AutoCon was a huge success and it was all over tech news, tech media, the tech social media influencers, and all that. But in any case, maybe a little bit about your background as well and how you came to be where you are today.

Scott Robohn: Sure. Let me see. Right back at you in terms of getting to know you and spending a little more time, especially as we worked up to the NAF event in Denver and since then. All good. I've got 30- plus years in the industry, I hate to say that. We don't have video associated with this but you'd see the gray hair so you would know. And the grandchildren count keeps going up so things I'm very, very, very thankful for. I had a formal education in industrial engineering which many people like to joke about being imaginary engineering or the liberal arts of engineering.

Philip Gervasi: There you go.

Scott Robohn: But it gave me a systems view that has come back very helpfully recently, and we'll touch on that later. I came out of school bitten by the network engineering bug. I did some projects in grad school that made this networking thing really interesting. My first job out of university I basically found ways to do networky things as side projects in addition to my normal duties. That helped me land at Bell Atlantic, part of... One of the baby Bells as a precursor to Verizon. Went from there, not directly but spent a lot of time at Juniper Networks. There's probably some interesting conversation we could have there about Juniper and HPE today. After working for corporate employers for those 30- plus years I really wanted to break out and do my own consulting work, which I planned for 18 months ago and pulled the trigger on just about a year ago. That's what enabled me to have the flexibility to work with Chris Grundemann and co- found the Network Automation Forum. A white- knuckle ride of my career so far.

Philip Gervasi: You mean the last 18 months and then your work with NAF, right?

Scott Robohn: Correct.

Philip Gervasi: Very cool though. And I'm really glad that you and Chris chose to do that. Maybe we could talk a little bit about why you chose to do that as well. I think probably one of the biggest networking- focused events out there like specifically networking, right, network engineers. It's probably Cisco Live that, obviously, is very Cisco- focused because it's Cisco Live which is totally fine, I built my career on that. But I loved it and I still do. I haven't been in a couple of years. But there really aren't that many enterprise networking events out there, conferences, small or large. There haven't been aside from that. And now I'm starting to see with AutoCon focused so network automation for sure. But still, I remember going there and sitting there in that big hotel conference room, whatever they call it, the ballroom in Denver and I'm like, I know 1/ 3 of the people here, these are my people. And so that was awesome. And then, of course, the local network user groups. The( US) NUA is now just exploding in popularity. I know that you are leading the very inaugural Virginia Network user group, correct?

Scott Robohn: Correct. I'm sure it will have happened by the time this publishes but Thursday, January 18th which is one week from the time we're recording this. We're ready to kick it off in Northern Virginia and the DC suburbs. Very, very excited about it. I think we have 35, 40 folks registered so far. We'll see who shows up on that evening.

Philip Gervasi: That's fantastic, really glad to hear it. I think that along with what... We had NetDevOps Days, we had AutoCon. There does seem to be this new resurgence of definitely smaller but high- quality, very high value events, very specifically for networking, enterprise networking. And maybe again, more on the automation side or more on... I went to an SD-WAN thing that was, obviously, specifically about WAN routing and SD-WAN. And I'm starting to see those, and that's awesome because there are just so many on the service provider side. I mean, there's a DevOpsDays every other month or every month I think. So there's so many other things going on out there that it's cool to see our niche of technology, networking, in particular, really just growing... At least the community growing and coming together again, right?

Scott Robohn: No, I totally agree.

Philip Gervasi: Your background is mostly on the service provider side? Because you mentioned Bell Atlantic so I'm just taking a guess here.

Scott Robohn: Correct. Full disclosure, I actually worked building and operating the enterprise network within Bell Atlantic but, obviously, got exposed to... What is the SONET SDH thing, T-1s, T- 3s, things that no one recognizes anymore. Got a good start there, it was a great place to learn a lot.

Philip Gervasi: I learned about all those things but I admit I didn't use a lot of that.

Scott Robohn: Right.

Philip Gervasi: And you're right, this is an audio- only podcast so nobody can see the gray. But to be fair, nobody can see the gray in my beard either, and I have a little bit, it's salt and pepper now. Because I shave my head, right, the past few years. I have found, like Patrick Stewart in Star Trek, I get older you can't tell as much. I will claim that as a benefit of losing your hair and then shaving it literally daily. You can't quite tell how old I am.

Scott Robohn: I will resist the temptation to do a Jean- Luc Picard right here but I'll keep that in mind for future use.

Philip Gervasi: Yeah, yeah, okay, very good. And I also see on your hat you're wearing a Syracuse hat.

Scott Robohn: Absolutely.

Philip Gervasi: So you're from central New York originally or you live there?

Scott Robohn: Born and raised in Syracuse, went to school in Rochester. 172 inches of snow a year, baby, bring it on. I'm sure they don't even get that much anymore there due to weather changes. A good place to be from is the way I like to put it.

Philip Gervasi: Is it?

Scott Robohn: Yep.

Philip Gervasi: Okay. I'm from outside New York City but I live in the capital region now which is about two and a half hours from the Syracuse area. I live in the suburbs of Albany so about half an hour from Albany. And we don't get the same amount of snow. I find that the lake- effect snow from the Great Lakes sort of stops around Syracuse, Utica, it doesn't quite get to the Hudson Valley where I am. On the other hand though, we do get sometimes the last remnants or sometimes a big chunk of the nor'easters coming off the coast.

Scott Robohn: I was going to say.

Philip Gervasi: Especially off of the New England coast. So we'll get that and not necessarily the lake- effect snow. But we do get our snow here. It is cool to drive along 90 going westbound. I love it, it's beautiful. Central New York is beautiful for those that don't know. Especially in the summertime, it looks like you're in northern California, it's very, very pretty.

Scott Robohn: I miss that part of things for sure.

Philip Gervasi: Oh yeah, for sure. You'll see the giant plow parked on the side of the highway.

Scott Robohn: Absolutely.

Philip Gervasi: And a big sign that gives you this like date we had this many inches of snow. It's a lot. Not quite like Buffalo, I don't think. Rochester too, huh? Rochester has a lot of snow.

Scott Robohn: I would put it this way. Syracuse and Buffalo are situated perfectly to their respective Great Lakes where weather systems come across, soak up moisture, and dump it on Buffalo, and Syracuse, and Watertown, north of Syracuse. Rochester is lucky. Just like you don't get it off Ontario they don't get it off Erie as much but it's cold and windy in Rochester.

Philip Gervasi: I'm surprised to learn that because Rochester is right on the lake. I mean, close.

Scott Robohn: It's directly south, right, and it's that west- to- east motion that picks up the moisture and dumps it.

Philip Gervasi: Interesting. Okay, all right.

Scott Robohn: I'm not a forecaster and I don't play one on television. I've probably exhausted the depth of my meteorological knowledge here.

Philip Gervasi: Right. There you go. Okay, all right. You don't live there anymore, you moved out of the area, and that's when you began your career in Bell Atlantic. I think you worked for Juniper for some time or for-

Scott Robohn: Correct.

Philip Gervasi: Quite a bit of time, right?

Scott Robohn: Yep.

Philip Gervasi: And then you decided to go on your own which I respect and envy, to an extent. I think any hard- driving person in tech, we feel we sort of work for ourselves, right? Regardless of the fact that I'm a W- 2 employee, I'm building my own skill set which I take with me, it's in my own brain. So I've always felt sort of like I work for myself. I really respect that you did that. But what was the impetus to start the Network Automation Forum now?

Scott Robohn: This was not on my mind 18 months ago as an event or an organization. But as we went from late 22' into 2023 I became aware of Chris Grundemann's work in publishing, the State of Network Automation Survey, and I thought that's interesting, I would love to see those results and I would like to have a conversation with Chris. Long story short, we got to know each other a little bit, found that we had very complimentary views on the state of network automation, and why isn't it moving along faster than anybody thought it would. Why do you think that is? I don't know why do you think that is? Well, let's get some smarter people around a big, big table, and let's start talking about it. And that's what led us to the Denver event in November'23. We set a goal of 200 people. We thought if we can get 200 people to come to this that... We'll call that a success. We ended up having over 340 high- quality content, great presenters, and people that were just hungry to talk about this. I think the post- COVID effect and quarantine was still weighing on people, it was good to be in person, and just people had a lot to share about what they've actually done that other people can try in their own environments.

Philip Gervasi: I appreciated the fact that it was largely practitioners, as far as I could tell, I'm sure there was a mixture. And I know there were folks that were coming from the service provider side, folks from various types of enterprise, folks from various vendors as well. I know that everybody was represented. At least what I heard in my ears and saw with my eyes very practitioner- focused, very independent- focused, though there were vendors and sponsors and I get all that, but it felt very engineers talking to other engineers about what's going on. So I really appreciated that. That was probably my favorite part. That doesn't technically tie directly to network automation, because it could have been another topic, but still, that was very important to me as a former practicing engineer. But also the fact that you asked that question very explicitly, why aren't we thought we would be? Because I remember when every other blog post was talking about Python and Ansible, and network automation is going to take over the world and all these things. In 2000 maybe 13, 2014, it started to get a little bit more traction in the tech media, and social media, and things like that. And then it sort of didn't really progress exactly like you just said, which again caused you to ask the question, why aren't we where we are? Now, that's not necessarily what I wanted to talk to you in this podcast today. Sure, this is tough, I'm sorry to do this to you. Do you have a one or two- sentence answer to that question?

Scott Robohn: Culture change has more momentum than you think. A resistance to culture change.

Philip Gervasi: Okay.

Scott Robohn: That's a huge learning, I think, for me coming out of the last year and culminating in the Denver event. The technology and the tools matter, don't get me wrong. It's funny, others have asked the same question and, in particular, Terry Slattery who I worked for at Chesapeake Computer Consultants a long, long time ago, he heard one of our early podcasts and said, " Well, I don't think you really answered the question and I'd like to help you answer the question." The resistance to culture change was essentially his number one reason. And I basically said, " Huh," and started thinking about that. I'm pretty convinced that that's a key contributor to the friction against more widespread adoption of automation.

Philip Gervasi: So it's not necessarily that there's a technical stumbling block like all the devices out there are completely closed, we're not able to access them in a programmatic way and so it's just something that we can't do technically. Or, we don't have the tools. There's no such thing because Python hasn't been invented yet so we can't do anything or whatever you're using. Terraform doesn't exist. So we don't have a means, a technical means. Those aren't the problems. You're saying that it's really largely based in people.

Scott Robohn: Yeah. There are things we can tease out of that, right? One dominant factor today I think is the hype and fear around AI. If you think about it AI is another form of automation, or at least it's very, very adjacent, right, and maybe a superset depending on how you think about it. But there is the comparisons to Skynet, these things becoming sentient and taking on a life of their own. I don't want to be out here saying, " Hey folks, there's absolutely nothing to worry about." Of course, there are things to be concerned about and think about. You can drive to a much bigger question of AI helping us understand what it means to be human, that may be another podcast.

Philip Gervasi: That's a scary podcast.

Scott Robohn: Or it could be a very encouraging one actually.

Philip Gervasi: Okay.

Scott Robohn: Hold that thought. I don't think AI is going to come along and replace people anytime soon. A more likely scenario is to be replaced by somebody who knows how to leverage AI tools. I'll use my horribly overused analogy, the movie Aliens, Sigourney Weaver at the end in the big loading dock exoskeleton enabling her to defeat the super scary hybrid human alien. I think AI is more like that suit, not like the alien. It's what can give you superpowers to... If we go MCU on this conversation.

Philip Gervasi: That makes sense. I've always looked at these technologies as augmenting an engineer, adding to not taking away, certainly not taking away, and not replacing. There may be a day when we have a commander data like AI, right, with a positronic brain. I like to allude to Star Trek a lot, Scott, just so you know. If you come back as a returning guest, which I hope you do, we'll be discussing more and more Star Trek as we talk more and more about AI.

Scott Robohn: This is not a problem.

Philip Gervasi: That's cool stuff. And I think that a lot of people look at AI and other technology, in general, in the light of fiction. Literally, they say, " Oh, well I'm scared because I saw those movies." It's like "You know that those are" ... " That's Hollywood, right, that's not real. You don't need to have that fear, you got to detach those two things." But if that's all you know that's all you know. I mean, what else does popular culture have? Not everybody's out there with a computer science degree and learning N- gram models and things like that. But what I'd like to do is talk to you about something specific. Yes, we could definitely talk about why automation isn't where it is or where we think it should be, excuse me, but I want to talk to you about an idea that you had, a concept, called total network operations, and how that ties into automation, of course. How that ties into this drive towards programmability, programmable networks, maybe tied into AI, I don't know. And, of course, how networking as an industry and as a practice is changing. Can you give me a quick definition? What does total network operations mean? I know what network operations is, I think, and I know what the word total means, but what does that mean combined together?

Scott Robohn: As a result of work that we've done in the last year, very focused on automation, I took it and zoomed out and had a broader view of okay, automation is one discipline or set of disciplines that we need in operating networks. There's a lot of change happening in the tools coming to bear for automation. The way I put this together, point one and point two, point one, zoom out of the silo and look at the system as a whole, really engage in systems thinking. How does automation need to work with visibility, with multi- cloud networking, with collaboration tools? All operating on network infrastructure. That's not a complete list of all the disciplines that we need to worry about in operations. But have a systems viewpoint number one. Point number two, things are changing so fast technologically you need to allocate and budget time to proactively investigate new tech and tools that you're going to use from an operations perspective. Not just what new version of segment routing am I going to turn on in the network? What's the new tech and tools coming online that will actually help me operate again? All those silos, now with permeable membranes, as a whole system.

Philip Gervasi: Okay. So then when you say network operations and then you use the term system, are you talking about this entire system that we call the network? And, of course, I know there's many components and it's changing all that. And not necessarily the entire technology stack. Am I right? Because you are saying network operations/

Scott Robohn: So, Phil, that's an excellent question. And no, I did not stage this question with you. I do believe there's a zoom out here where you can apply that same principle to viewing network, and storage, and compute, and applications as a system as well. So I think the principle applies as you zoom out to the entire IT tech stack. How do you eat an elephant? One bite at a time.

Philip Gervasi: Sure.

Scott Robohn: And because I'm really grounded more in networking than any of those other technology areas, that's where I'm going to try and make an impact on that.

Philip Gervasi: And I mean, you know who I work for and what I do for a living, I work in observability. And then we take the approach I can take looking at the network as a system, or the system of the network, the substrate that all applications are delivered on. And then we use the term network observability to mean the same thing. We look at the system holistically. And the system is comprised of a lot of different components and also network adjacent components like DNS, for example, right? But I got to say, the reality is, when you look at what's going on in a network, the delivery mechanism for the application or for the data or whatever, and ultimately we don't care that much about it. We care about the human being, or the one computer, getting access to the data on the other side. And because most of the applications are delivered via network... I'm looking at my computer, and most of the applications that I use regularly are I don't know... 80% are delivered over a network from somewhere else. So to me, so much of the stuff we care about... We say we don't care about the network, but the stuff that we care about is completely reliant on the network as far as reliability, performance, all of that. You can even, I've found, infer what's going on outside the bounds of the quote- unquote network by looking at the network activity simply because all of that application data is going over the network.

Scott Robohn: For sure.

Philip Gervasi: So you can see the delays are not happening here between these hops. There's no latency, there's no whatever. We can see that the actual time that we got our response from that server, all the way on the other end, inbound into my switch was at this time. So we could see the delays certainly, clearly happening on the server side. Now what's happening there we don't know so you can actually infer quite a bit. And that's the whole point of observability is determining the health of an entire system by looking at its components and its outputs. And so when you say total network operations, is that where we're headed, where we're looking... I'm talking about observability, of course, determining the health of the system. You're talking about going even beyond that. Not just understanding its health, which is visibility on steroids, right, but then also in its daily operations with actual, of course, understanding what's going on, determining its health, and then maybe pushing config, making adjustments in order to then again, ensure performant reliable delivery mechanism, right? So that's where the whole part and parcel comes in.

Scott Robohn: You asked the question, is that where things are going? My response would be, I'm not so sure but I think they should.

Philip Gervasi: Okay.

Scott Robohn: I'm just one individual, and I haven't been in an active operations role for some time, putting these thoughts together, calling it a framework, and bringing it out here to get some feedback on, right? I would love to get more people involved in a conversation on, " Scott, this is crazy. This is already being done and you shouldn't waste your time." " Okay, great. Thank you, that's great information" or the other, no, I haven't thought of it this way before. And I'm very focused on the optical piece and optical interaction with IP, I'd love to help there. Or I'm a network firewall person, I would love to bring that element into the conversation. I'm looking for other people that want to collaborate and flush this out if we see utility in it.

Philip Gervasi: When I started my career in tech that is, specifically in tech, working at a help desk, then eventually I got into more of a systems administrator role and I had to touch everything because my customers were SMB customers.

Scott Robohn: Exactly.

Philip Gervasi: It was total network operations-

Scott Robohn: Sure.

Philip Gervasi: I had no choice. Now as I got into more sophisticated engineering, that was very difficult to do simply because the depth of knowledge and expertise required for each domain was such that it was very difficult to just be an expert in all those areas. So I think what might happen is that as you have complexity and scale, you have almost no choice but to silo your skillsets and your different operational practices. You're calling for a reversal of that. How does a human being do that? How do I become an expert in every single operational domain so that way I have this holistic view that you were talking about?

Scott Robohn: And so good for clarification, right? I'm not implying everyone needs to know everything, we all have our human cognitive limits, right, and people are going to specialize, right? But there needs to be someone, and I would say there needs to be a function that intentionally understands the end to end, how those things fit together. One of the ways I've thought about this is maybe we create the role of operations architect. Somebody who's not just creating diagrams in Vizio, or not just talking at conferences but has had some real hands- on time in at least some of those disciplines. Has got to the point and has enough of a systems thinking perspective that can help organize okay, how do we play Moneyball, right? I'm not a baseball guy but I love that movie, right? I don't need to replace person A for person B exactly, I need to figure out what the mix of skills are on the team regardless of where those skills might sit. That's more of my... What I'm trying to imply here.

Philip Gervasi: All right. Again, it seems like what you're trying to imply is implementing an operational framework that governs all of these domains whether it be network, network adjacent, whatever.

Scott Robohn: Correct.

Philip Gervasi: And so maybe there's an individual or a small team that has a more than cursory, but not necessarily expert level knowledge in these domains, but really the focus on how they interact with each other.

Scott Robohn: Correct.

Philip Gervasi: And so the framework is not a person necessarily it's... Again, it's an operational mechanism that all of the ... What we used to keep in very disparate, distinct silos will then operate under. What's the result? You think that's just going to make things better as far as... People always say, " Oh, breaking down silos." Well, who cares everything's working. I checked my Gmail just now and it worked just fine. Do we really need to break down all these silos? What's the result? What are you looking to do?

Scott Robohn: Make life better for people who are operators.

Philip Gervasi: That's always a good thing to do, yeah.

Scott Robohn: But also bring higher availability and resiliency at lower cost to network operators, whether it's an enterprise network, a carrier network, a cloud network, or something else. There's almost always room for improvement, right? And when's the last time you ran into a very well- rested operations person?

Philip Gervasi: Okay. That's a good point.

Scott Robohn: I feel pretty strong... Confident in making that statement, right? I made an illusion earlier in the conversation to some things from early in my career, actually, my academic career coming back around 30- plus years later. At the Denver NAF event, one of our keynotes, John Willis, who comes out of the DevOps arena primarily but has done a lot of work applying those concepts to networking, he's where I got the challenge to really start thinking outside of your own silo. And so the way I took that was okay, I've been really hyper- focused on automation for the last 12 to 18 months, what else should we be stitching together here? He actually referenced a 1940s,'50s through the'80s statistical process control luminary named W. Edwards Deming, D- E- M- I- N- G. Dr. Deming was someone who post- World War II really worked to help improve manufacturing quality in the United States. Didn't get a lot of traction here. Actually went to Japan and helped Japan accelerate and really grow their manufacturing, and, in particular, auto manufacturing businesses, and leapfrog quality in the US. It was really eye- opening. This is a guy I studied as an undergrad industrial engineer. Never ever thought about applying any of his concepts to IT disciplines let alone networking. Now my eyes were opened and I went and I read Willis's book, Deming's Journey to Profound Knowledge, which sounds hoity- toity, it's actually a really good book if you're grounded in the manufacturing world. And then a follow- up he recommended called The Phoenix Project. Have you ever heard of that?

Philip Gervasi: Oh, yeah.

Scott Robohn: Okay.

Philip Gervasi: I remember reading that. inaudible.

Scott Robohn: I'm in the middle of it right now actually.

Philip Gervasi: Are you? Okay.

Scott Robohn: There's a lot of talk of process and observing bottlenecks. Work in process or WIP, being a killer for any system. If you have a function or a workstation where WIP is piling up, inventory is piling up, that's the weak point, that's the constraint for the whole system.

Philip Gervasi: Right, yeah.

Scott Robohn: We can think about that in terms of lots of network operations or other operations roles where there are people in certain functions that just get consistently slammed. How do we think about flow control? I'm not talking about TCP windowing.

Philip Gervasi: Right, right, yeah.

Scott Robohn: Across the whole system, right? Not just with this Python script, or not just with these ethernet interface errors, or not just with my Slack integration not working.

Philip Gervasi: Some of the things that you said reminds me of when I learned about lean and when I learned about control theory from years and years ago and how now that's being... That's basically the fundamental idea behind observability so the same idea. Again, looking at the many components that exist within a system, and how they are operating, and how a component could cause the health of the entire system to degrade which is what you're talking about right now.

Scott Robohn: Exactly.

Philip Gervasi: How there is a bottleneck in one area. In an operational context so there's some config that's pushed or some script that's running or some activity. In this sense, we're talking about the network, or in this conversation we're talking about the network.

Scott Robohn: Right.

Philip Gervasi: The thing is that the network is just so much. So I really want to make sure we understand, as far as the audience is concerned, that when we're talking about the network. Think about everything involved with delivering an application to you... Right down to your phone or the computer that you're sitting in front of. It's a tremendous variety of devices. And devices that really... That a lot of people don't associate with the network. They're in the cloud, you don't own them, you can't really do much with them except just trust that they work. Again, going back to what I do for a living then understanding... It's the difference between seeing and understanding. Yeah, I can see more stuff on a graph, but how does this particular component relate to the performance of this component? Why is it that when I swing B2B peers from data center A to data center B to do some routine work, and I want to swing my traffic over, why is it that all my DNS resolution times go in the toilet? That makes no sense, what's going on. Understanding how components fit together, and where there's a bottleneck you said. Ultimately to me, will then help me determine why applications are performing the way they are, which is the whole point of this thing. Some of those applications are mundane, right, you're using Office 365 to write out your shopping list. Or, the 911 services in your city running over an IP network. So they could be mundane or the operation, literal people operations in your hospital, they could be very, very mission- critical. That's how I see the result here. Yes, I agree with you making my life better as an engineer, right, or whoever's out there still doing engineering, not me anymore. Making life better, making your job more efficient, and things like that. But ultimately it's to make applications more performant, more reliable, and to this system to work better because we do rely on it, it's life and death in a lot of situations.

Scott Robohn: Absolutely.

Philip Gervasi: You might disagree with me here. I got two things I want to say, I'm going to play the devil's advocate. I hate that term, by the way, but I don't know another term. One, do you remember when folks were advocating for network automation 2015, 2014 and they were saying, " Ah, it's a good way to" ... " You can automate away the mundane tasks so you have more time to do more interesting work." And I was always like what? You're going to help me do this stuff better so I can go do other work that I don't want to do? That's not a compelling reason to me for network automation or for programmability or for what we're talking about now, total network operations. What's more compelling is making the system work better. So yeah, my life is better true because there are fewer faults and the end users, in our case, right, our customers quote- unquote are happier.

Scott Robohn: Sure.

Philip Gervasi: That was never a real compelling thing. The second thing you also might disagree with me on is, I don't know is networking really changing that much. You go talk to folks in operation and they're like"I had to configure my root bridge today. Oh, yeah, I had to configure" ... " I had to troubleshoot this thing and I was doing a MAC trace on Cisco switches." Okay. Or like" Yeah, I set up OSPF, I left the defaults because it's easy" and blah, blah, blah. And you talk to people and they're like" Yeah." Granted, maybe it's an SD- WAN box instead of an ISR, ASR.

Scott Robohn: Right.

Philip Gervasi: I don't know, I wonder is it really that different?

Scott Robohn: Let's talk about both of those-

Philip Gervasi: Okay.

Scott Robohn: And briefly on your first point. Automating away mundane tasks so I can do more interesting work not being a compelling reason. And I actually would agree with you on that. By itself, that's not enough of a reason to engage in structural wholesale change with automation but there doesn't have to be mutual exclusion here.

Philip Gervasi: Sure, okay.

Scott Robohn: People can get benefit like we just talked about there and also make the entire system work better. It's a meta point from Deming where in his process analysis he very intentionally took responsibility off individual performance and put it on the design of the system and associated processes. Now, I'm not sure I'm fully on board with him because individual performance does matter, but you really do need to show that your processes are designed well and functioning well. And I would say my experience in three decades of network engineering, man, it was the wild, wild west for a long time and it was so cool. Your pedigree didn't matter. Piece of paper, no piece of paper. If you could configure that AGS +, or insert favorite network device name here, if you could make it work, you're in and you could. On one hand that's awesome. On the other hand, it does inspire hero syndrome and really making people like to be the hero all the time. The technology has moved so fast that we haven't put as much time and effort, I think, into designing process and making sure that's working well. Some shops have done a much better job of it than others, I don't want to make it sound like a blanket indictment, but there's room for more focus there. Remind me what your second point was, I'm sorry.

Philip Gervasi: No. My second point was really all about how... Has the networking industry really changed that much? We're still passing packets and now I have SD- WAN instead of an ISR. Sure, I'm using the cloud but it's still routers, and AWS, and switches, isn't it?

Scott Robohn: There have been some other interesting podcasts and other information out there addressing that point. There was a great podcast late last year from Packet Pushers between Greg Farrow and Johna, I cannot remember her last name. But they often do, I think, the heavy networking podcast together and they ask the very same question. And I would highly recommend your listeners go check that out. I can't remember the exact episode right now. But the answers were basically yes and no. Is network really changing? Greg said, " No, it's not really changing." Johna said, " Yes, it absolutely is." And so let's peel that apart a little bit. If it's not really changing that much, and you've got a compelling case therefore we're still pushing packets, and frames, et cetera, what an opportunity to catch up and operate it better. If the technology isn't really moving that fast right now let's catch up on the operational side-

Philip Gervasi: Good point.

Scott Robohn: And bring that in line. So I think there is an element of truth to that thread. The other side is, well, AI and AI- enabled tools are bringing some interesting things both to changing the network infrastructure and the way we can operate on the network infrastructure. And I would say the emphasis on GPU clusters for AI training and operations, it's going to be a pretty promising growth area for folks that want to be involved in that networking for probably the next five years or longer. It doesn't mean it's necessarily super complex. You see the emphasis in the last year on how Broadcom is trying to do more here, play a role. But other network equipment vendors too really, really focusing on that use case. That's interesting. But then to go back to Sigourney Weaver, Ripley in Aliens, now I've got new tools that are empowered by this new technology to get over my human cognitive load limits, right? We had the little snippet of conversation earlier, we're not asking everybody to understand everything but these tools embedded in products exactly like yours to do data reduction, to connect dots that a human may not see readily, and to be able to get more meaningful information out of lots and lots of data. I'll pause and breathe there. Sorry for the rant.

Philip Gervasi: No, not a rant. In fact, I have a roadshow presentation called Finding Meaning in the Data which is just that, using these tools allows us to analyze data faster, more efficiently, and be able to find insight in ways that we just weren't able to unless you got a whole team of PhDs from MIT and even then you're still waiting for people to do the work. Being able to do that faster because of the complexity of networking. I was just arguing that it hasn't really changed that much. It's not that I think that networking hasn't changed, I mean, we... It's gotten complex in the amount of devices that we use, the types of devices. Sure, now I got these ephemeral Linux networking stacks inside of containers I got to take into account. I's still TCP/ IP, it's still stuff. So that's what I mean by it's more complex but it hasn't changed at the same time, right? I got load balancers I don't own so I got to get this dribble of flow that the provider will give me, I've got to incorporate that in there. So you got all this information and now do something meaningful with it to make the system better. So finding meaning in the data. What would you say then are the domains or the disciplines that we're talking about within networking? Are we just talking about basically network engineers and cloud engineers? What are the domains that you're talking about under this umbrella of total network operations?

Scott Robohn: The ones that come to mind for me, and I don't count this a comprehensive list, right, but just the seven or eight buckets.

Philip Gervasi: Sure. It's a work in progress.

Scott Robohn: First of all, the operating the infrastructure and still using the CLI for different flavors of network gear, that's a discipline in and of itself, right? It's mature, it's been with us for a long time. Knowing how to set BGP Next- Hop in iOS, in Junos, in EOS, et cetera that matters, right? All that chunk of stuff matters. Then there's automating it, right? And again, this is tip of the iceberg descriptions, right, there's so much buried into the automation bucket. I would put telemetry, visibility, and observability as a bucket. The whole AIOps arena is another bucket that can include AI models embedded in tools. And the chat tools that we're starting to more than play with today. ChatGPT, Bard, et cetera. Different use cases for different tools there, obviously. Multi- cloud networking is another bucket. So to your point, are we just talking about network engineers and cloud engineers? Well, this puts one foot on each side, right? And maybe I need four or five feet if I'm going to do my network connected to cloud A, and then cloud B, and then cloud C. And what does the market want for workloads to be highly mobile, right? Multi- cloud networking is definitely going to play a key role in that. I'm really interested in the whole digital twin discussion. Again, some of my academic background is in modeling and simulation. And I scratched my head for the first 10 years in the business. Why don't we do more modeling of networks? The short answer to that is because we won't believe it unless we see it work in the lab. And having a model, or even just VMs running on x86, or Kubernetes- based, container- based models it's not the same as working on real hardware. And the last bucket I'd call out here is security and particular network security. We do a really poor job of integrating that into the bigger picture and I think there's better room for that. Lots of other things we could talk about there. Again, what collaboration tools have to do on top of this? What about optical and IP? There are other adjacent technologies I think that are all in scope here.

Philip Gervasi: It really feels like you're talking about an operational framework. Whether there's this one person who's got this understanding... More than cursory but not expert level understanding of how these domains operate loosely and then... But really focus on how do they interact, how do they fit together? And then you get the experts in a room. So it's not that you're advocating for somebody to know all these areas and be an expert in all these areas. And it's great if you can be... Have a good solid knowledge in all these areas, fine, and a lot of people do, right, but it's really having the operational framework that combines those areas under... Maybe there's some hierarchy there to again, ultimately then produce a better operating system. Having an automation specialist, having somebody who's really familiar with ML models and how to apply them, and how to tweak an LLM to produce fewer hallucinations and things like that. And how to train models in the first place. Your security engineer and having that... So you asked the question, is this anything new? I mean, it's what we've been doing for a long time, sort of, isn't it? I mean, back in the day I didn't... We didn't have Zoom we'd get on a Webex, right?

Scott Robohn: Even before that, we got on an audio- only bridge.

Philip Gervasi: I remember those too, a conference call, right? And you'd have the 12 people responsible for each domain, and then usually a project manager that knew very little about any of them, right, and you're trying to troubleshoot a problem and figure it out or whatever it was. You are sort of advocating a more formalized structured version of that, right?

Scott Robohn: Correct. Again, to my preamble, right? I'm not saying this is necessarily anything brand spanking new, but if there's a little revival and a re- emphasis on the concepts needed, let's do that let's put a finer point on it. But combined with the fact that key technologies are moving very fast today, maybe make this different this time around, maybe. I'm not 100% sure in that statement but AI is certainly a candidate for that.

Philip Gervasi: Sure. And ultimately I don't think it matters if it's something that we've done for years and years and years even prior to technology existing the way it does today, it's you're trying to create the framework and formalize it. Like you said, right, putting a finer point on it so that way we can implement it in such a way where we really are doing it. And it's not just ad hoc random conference calls and we're literally winging it half the time. And then me, as the engineer, as soon as I'm like" Yeah, ping's work," I go back on mute, I'm not participating, and there's nobody that really knows that holistic view of how all of this stuff fits together we're all just sort of checking the boxes of all right, that domain works, this domain works, this domain works. All right, well, and what else could be the problem? Are we missing any domains? Where's the server guy? Yeah, we got all the same people in the same ... In the room but it's a different approach. You are talking about a very, very explicit framework of understanding how those things fit together. That's I think would be a differentiator between just people getting together and collaborating over the years, right?

Scott Robohn: For sure. And I would even take that one step further, maybe it's a baby step further. Intentionally allocating time and resources to look at the new stuff that's coming down the pike, that almost never-

Philip Gervasi: Oh, I see.

Scott Robohn: Happens in an operations team. Look, there's a lot about operations that's the bottom rung on the ladder. Everything you know what flows downhill, and people in ops end up catching lots of stuff that people let slip that way down. Being more intentional and investing in the operations layer I think it's a very good thing, and it's a very Deming- esque way of approaching it.

Philip Gervasi: So today, it's safe to say 2024. Sometimes I don't like to say the date or the year because you don't know when the pod... But it's the beginning of 2024. You mentioned things flowing downhill. What about within operations? Is there a hierarchy of domains and disciplines there? Is it the network engineer that gets the brunt of it but the AI ML person, they're the rockstar today. What would you say is sort of the preeminent domain discipline that we should be focusing on? Maybe to improve? I don't know, maybe I'm asking the question the wrong way.

Scott Robohn: No, it is a great question. There's certainly a risk of what you're saying. Look, as people in tech, right, we know what it's like to gravitate towards the shiny objects, right? I want to work on the new cool thing.

Philip Gervasi: Oh, yeah.

Scott Robohn: And that's not a bad impulse, right? I think you need strong leadership to be watching out for things like that and helping balance when imbalances like that come into play. It's a horribly generic answer, I'm sorry. Okay, I'm the new AI person on the ops team, and therefore since I'm on the most new important technology I get to call all the shots. It's like well, no, you need somebody dampening that behavior. BGP dampening as a paradigm for life is not a bad thing.

Philip Gervasi: Very good.

Scott Robohn: Something to be watched for sure. This is out of the packets and circuits. It takes real people watching these dynamics, knowing about their people, and actually caring about them. What are we trying to accomplish for our employer?

Philip Gervasi: Right.

Scott Robohn: Both matter.

Philip Gervasi: It always comes back to people, it really does. It sounds like then this fits right in line with the work that you've been doing with specifically network automation, and dare I say orchestration if we can differentiate the two.

Scott Robohn: We can.

Philip Gervasi: Again, it goes right in line with that considering that to automate a system rather than just automate pushing like interface descriptions. Going into any complexity, which was probably transcending into the orchestration realm, you have no choice but to have an understanding of the entire system as a whole.

Scott Robohn: Correct.

Philip Gervasi: I mean, the way I see it is having a completely programmable infrastructure. I mean, let's go back to something very mundane. I wiggle the wrong wire and I have a BGP adjacency go down from literally physically playing with a wire where some twisted pair was messed up. That layer one, right? BGP adjacency goes down, it kicks off an entire convergence, re- convergence, right? Some application is now being hosted of a different data center. All sorts of things are happening as a result of something that you can maybe say is very... Now, should that happen, that's a different question, right? There should be some SRE there that's like" All right, that's a silly thing. We should not have all of that stuff cascade from a wire being touched." In any case, that's how I look at it. You have all of these individual components operating within the system that you're trying to orchestrate and make programmable, and if any one of them isn't well understood it can affect the health of the entire system.

Scott Robohn: It almost certainly will. We love the cloud term blast radius, right?

Philip Gervasi: Oh, yeah.

Scott Robohn: It could be a small blast radius but it could be much bigger than you think, right?

Philip Gervasi: Yeah. So to wrap up this conversation so far, Scott, what would you say are the biggest takeaways as far as how one should start thinking about this entire network, this system that we call the network?

Scott Robohn: I would recommend starting in the place that I started on this, think about the bigger picture, think about the system. It's not hard, right, it just takes stepping back a little bit and looking at all the piece parts associated that you have in view. That would be my first recommendation. The second recommendation is really go out of your way to make time to investigate new tools. All the buckets are important. Some of the more important ones I think are AI automation and multi- cloud networking. Some technologies are more equal than others. Apologies to George Orwell. Do your homework, right? Go back to a time when you really love learning about new stuff in tech. Try and revitalize that. And then the last thing I would just say is fear not just try. Just start small. Start playing with ChatGPT. Whether it's for jetting up configs. Sorry, even using that language dates me. Or, asking it, " Hey, what else should I be thinking about in this problem as a tool for what else can I go do my own human- centered investigating on?" Don't wait until you get the perfect idea start playing with tools now.

Philip Gervasi: Those are good recommendation. And I'll just piggyback off the last one. When I was a teacher taught my students... I taught basically the equivalent of ICD one and two. I'm like" Listen, just start." You have this idea of oh, but there's this CCIE out there. It's like fine and maybe one day you'll get there. Just start with what's in front of you.

Scott Robohn: Exactly.

Philip Gervasi: With network automation, when I brought that to leadership at a pharmaceutical company I worked for, they kept looking at the end and they were like" Well, we don't want to do" ... I'm like"No, no, no, no, no how about we just start with IDF two down the hall and gathering information so we're not even pushing config. We're just making life a little easier there." Just taking those first few steps. You can apply that to really anything in life, right?

Scott Robohn: For sure.

Philip Gervasi: If you want a new exercise routine, well, I'm not going to ... You start to look at some bodybuilder's routine. How about you just start? Get in the gym for five minutes and then you build. And I think that's great advice, Scott, of that last one. Just don't be so scared just start somewhere and you can build from there. Really great talking to you today, Scott, I really appreciate it. This is something that we can explore certainly, especially because we've been talking about culture and people so much, that's a never- ending discussion.

Scott Robohn: For sure.

Philip Gervasi: And certainly, I'd love to talk to you more in- depth about what is going on with the Network Automation Forum, with future upcoming AutoCons, but more specifically your take on what's going on in the industry with regard to network automation and programmability. For anyone that would like to reach out to you to ask a question or maybe to disagree with you about something, how can they find you online?

Scott Robohn: LinkedIn is the best way to get ahold of me. I'm Robohn, it's like putting a robe on, and don't let the H fool you, R- O- B- O- H- N. And I'm Scott Robohn on Twitter as well, or X, or Twit X, or whatever we're calling it.

Philip Gervasi: Very good.

Scott Robohn: I'm still finding good value there. I've got a pretty curated set of accounts that I watch so I'm there. I'm trying the other things too but I haven't found anything that still provides the value that X does.

Philip Gervasi: Absolutely, absolutely. I'm experiencing the same. And you could find me still on Twitter, I'm active there at Network_Phil. You could search my name, Philip Gervasi on LinkedIn, also very active there. And my blog network phil. com. I also encourage you to go over to Networkautomation. forum to learn more about the Network Automation Forum of which Scott is co- founder. There's a couple of future events that you could check out as well, there's AutoCon one in the end of May and then AutoCon two in November of this year. Now, if you are interested in being a guest on Telemetry Now, or if you have an idea for an episode I'd love to hear from you, reach out at TelemetryNow @ kentic. com. So for now thanks very much for listening, bye- bye.