Looking back and peering ahead

Phillip Gervasi: The other day I was listening to the radio, the actual terrestrial radio was in the car and a Pearl Jam song came on. Nothing weird. But at the end of the song, the DJ said something like, " Hey, you're listening to the Rock of New York, where we play classic rock hits all day every day." And I was like, " What? That's weird. Pearl Jam is a pretty new band. That song came out only a few years ago." And then I did the math, and it was certainly not a few years ago. So that's basically how fast time has been going for me lately, especially over the past year, 2023. And especially in tech. We've had internet outages, we've had security breaches, huge acquisitions. We've had a nonstop buzz around AI. And on the local level, we've seen job changes, companies pivoting to new markets, new technology, and on a personal level, I can say that I've had a re- evaluation of what my whole role in the networking community is all about. So with me today is the entire Technical Marketing Evangelism Team at Kentik. My colleagues, Leon Adato, Nina Bargisen, and Doug Madory. And this episode is supposed to be just some year- end review necessarily, but I do want to hear from some serious experts in the field about their own thoughts on what's been going on this past year and what they expect to see in the coming year. My name is Phillip Gervasi and this is Telemetry Now. Leon, Doug, and Nina, it is really great to have you all on the podcast today. Honestly, I've had you all on at various times, various topics, so it's pretty cool to me to have all of us together talking for the audience's sake. We pretty much talk all the time anyway in Slack. Sometimes there is an explicit component. Sometimes there isn't. Sometimes it's profound in the statements that I hear, especially from Doug and his analysis of what's going on in the internet, and sometimes it's just a disagreement on what the best band is right now. Because I only listen to the same four bands. Apparently, I'm stuck in the'90s. But it's really great to have you on. What I want to do is do a quick roundtable or go around the table, the virtual table, just do a one or two- sentence introduction of yourself real quick. So why don't we go in alphabetical order? Leon, we'll start with you.

Leon Adato: I am Leon Adato. I did graduate second in my class alphabetically behind Lisa Abrams. I have been in tech for 35 years. I remember one of the first things was that Windows came for free on 12, 5 1/4 inch floppies when you bought a copy of Excel 1.0, which no one did because everyone was still converting over from VisiCalc to Lotus 1- 2- 3. So that gives you a sense of just how much older than dirt I am. I've been working in monitoring for about 25 years using almost every tool out there, and I am still the newest member of the Evangelist Team here, and possibly one of the newest members of Kentik overall, and it is amazing to be here.

Phillip Gervasi: Thanks, Leon. Nina.

Nina Bargisen: I'm Nina Bargisen, and 25 years, I think, is my time in the industry now, which is a little bit scary. I'm not really sure what happened. I used to be one of the people building the internet, and I'm pretending that I still do and know something about it. That's why Phil has been kind enough to invite me here. So I still pay attention to CDNs, service providers, optical C cables, what's going on. But since I live in Europe, it's a little bit boring because I think all the fun stuff is actually happening in APAC and Africa right now, and so we have to find a good way of expanding in that direction, I think, because it's more interesting. Here, it's just consolidation and, " Oh, let's try to split the companies in different inaudible."

Phillip Gervasi: All right. Thank you, Nina. And honestly, that actually is pretty interesting. It's not something I would've thought of or I did think of what's going on in APAC and other parts of the world outside of North America and Europe is more interesting. I want to hear more about that. Before we do though, Doug. Last but not least.

Doug Madory: My name's Doug Madory. I'm the director of internet analysis for Kentik and I do internet measurement, internet analysis. The main buckets of stuff that I produce are either looking at BGP analysis, things involving RPKI, route leaks, hijacks, or big outage autopsies and then the last stuff is the geopolitical geographic submarine cable category that we may touch on today.

Phillip Gervasi: You said submarine cable and I think it goes without saying that we're absolutely going to be touching on that. That's one of the coolest topics that I don't have anything to do with yet want to talk about all the time.

Leon Adato: It is catnip for network engineers. You mentioned undersea submarine cables and everyone's like, " I want to hear more. I want to know about the sharks."

Phillip Gervasi: Yeah, the sharks, which are a myth, we learned this past year. If you remember Doug, we had Alan on from TeleGeography and he talked about how the last incident of marine life in any kind of meaningful and serious way, damaging the submarine telecommunications cable, was in the mid- late'80s. And since then, of course, the armor and shielding, all that, has been improved and there has been no issue with marine life damaging a telecommunications cable since then. There are videos and pictures of a shark or some other animal trying, grabbing the cable, but we have the protection now on the physical infrastructure under the water to prevent any kind of serious damage. That was a really cool podcast, Doug. I remember that we've been talking about submarine cables in other contexts. Of course, today, we're going to talk about some outages as a result of cable cuts and some new parts of the world that are turned up, so that'll be interesting. One of the first things I was thinking of as you all introduced yourselves, and I've been in the tech industry for a little less than you all just because I changed my career in my mid- late'20s, so I've been in tech for about 15 years. As we're going around talking about it, I'm thinking, when I first got my CCNA day, and I was working on a help desk at the time. I get my CCNA. I walk into the office after I pass it. I had to take it four times, something like that, and I felt like I was king of the world, and I walked in and I'm like, " Hello, peasants. I know all things networking." And now as I get older, in a technical sense, as I get older as far as years in the industry, my goodness, that's become the inverse. And so, Nina, you're like, " I don't know what's going on now the older I get in this business." And Doug and Leon, you sort of echo the same thing. Isn't that weird? It's like the more you learn and the more you know... And I know you all personally. You have very deep and broad knowledge of tech, the industry of networking, so much so that I lean on you for a lot of content that I produce. Yet here we are where the opposite happens over time, where we become more aware of what we don't know and how things are so volatile and changing and maybe there's a little element of being jaded by the whole industry seeing so many changes. I don't know.

Leon Adato: I think part of it is, yeah, knowledge acquisition, that is a normal thing regardless of the domain of knowledge, is that you start off thinking, " All right, I can figure this out in a weekend." Maybe you have a little bit of success with it. It's that xkcd cartoon of that one weekend you screwed around with Perl, and then you learn a little bit more and you realize that there's just this vast body of knowledge and you sort of dive into it. But also, I think that there is a difference in the pace of change. I think that the switch from completely on- premises installations where the cadence of updates, it was quarterly if not yearly because pushing out a new update meant pushing out some sort of media. And now, with mostly cloud- based apps, you've got this continuous deployment concept, and so the technology itself and the capabilities itself change far more rapidly than any of the four of us were used to back in the day.

Phillip Gervasi: But do you really think that's true? All right. So I agree with you that the industry has been changing fast. Fine, quickly, at a rapid rate. But hasn't it really always been, if we think about it, going from the mid-'90s to 2000? I mean remember when everybody started doing PTVs and that was groundbreaking technology. That happened really quick. And then in two years, we went from a completely physical infrastructure to a lot of VMs everywhere. The advent of going from regular routing to SD-WAN, that took about three years. Well, maybe three years is a long time now I think about it.

Leon Adato: We're back to Pearl Jam.

Phillip Gervasi: We're back to Pearl Jam,

Doug Madory: Phil, you're touching on is something that's universal in that when they look at scientist study, help young people read, they get caught up on a lot of details, maybe miss the big themes, and as you're older and you read a book, you forego the little details that turned out to be inconsequential and you are better at grasping major themes and stuff. You see this in a tech space like software development where a software engineer will move on to being either management, product management, those kinds of things where now, you're kind of getting away from particular details and now understanding the broad themes and how the bigger picture works. That's where we are. I think, maybe we're less implementing the latest tech and then we're looking at the themes and how the big picture works.

Phillip Gervasi: Yeah, do you miss implementing the tech? This is not exactly where I wanted to go in this podcast. Being in the field, turning a physical and virtual wrench from time to time, I mean I know I do.

Nina Bargisen: I do. I miss building stuff.

Phillip Gervasi: Yeah.

Nina Bargisen: I had a friend once ask me a little while ago when I was trying to explain what I'm doing and he is like, " But when do you do real work?" I was shocked not because he said it, but because I kind of agreed. At least when you come from many, many years of off- building things and making things happen, it's interesting to be in a completely different role, and then yeah, you miss the details because you don't need the details anymore. There's another thing. I mean why...

Leon Adato: I'm going to say though that there's also, it's not, not that, what Doug and we're talking about. But it's also that as you become more sophisticated a reader, we'll go back to that analogy, you also begin to understand a lot of the nuance of what you're reading. When you were learning to read in those first couple of books, you were just trying to grind out the words themselves, and reading a book meant getting through 10 pages or whatever it was and you completed it. But now, you read a book and you're lovingly savoring the nuances and the moments and the layers and the relationship of these words to other words that you've written, these ideas to other ideas that you've written. I think that along with not getting bogged down by details that you realize later don't matter, there's also the fact that you can appreciate so much more richness and depth to what you're doing. And I think the same is true professionally. That when you're just, again, getting your CCNA, it's like, " I just want to be able to configure EIGRP. I just want to be able to get the routing to work.' That's it. And now you're thinking about redundancy and you're thinking about there's so much more nuance to a packet capture. " Have I set this up so that it is scalable and maintainable? What have I done with it?" And Nina, to your point, that's the work that I think sometimes can feel the most rewarding when you really set up an architecture well and you have minimal, I won't say no, minimal redos and minimal mistakes that you have to fix later on, that feels like an accomplishment. And I think sometimes our work is missing that if we don't work really hard to find ways to put that back in.

Phillip Gervasi: And when you say our work, I think if you're talking about the context of what we do on this team, our work is more subjective and qualitative in nature and I miss, like Nina said, just building and fixing stuff. There is for me an intrinsic satisfaction that comes from something being broken and then it not being broken anymore because I fixed it. For example, I had my hot water heater go over the summer, and I don't fix things much anymore. I mean I do at home, but you know what I mean as far as my day job, and I got my daughter to help me because I wanted her to learn how to do these things for her adult life and we replaced the water heater. It took 45 minutes. It wasn't a big deal, but I just remember thinking to myself, " I fixed that. It's working now. We have hot water in the house." And I had that feeling and experience when I was building the... Even designing networks because I had to be eyeball- deep in the technology in order to design it properly and interface with customers about it, and I missed that. But I think I agree with what all of you were saying though, taking a step back and now being able to understand because we've been in the weeds, being able to understand the macro perspective in a way that we weren't able to before. Maybe that's why looking back over 2023 and making prediction about 2024 is maybe as kind of tried as that is. Everybody's doing at this time of year. Maybe that does make sense for folks that have been in the business for a couple of decades. I know when I was first in tech, I wouldn't have been able to do that. I would've just been like, " I know what the administrative distance is of OSPF. I can tell you all about LSAs and BGP communities." I would be able to rattle off stuff. And now, I realize, " Who cares? I can look that up, but how does this all work together and why is the internet doing what it's doing right now and how can we make that, that kind of thing." Which is really more, in my opinion, Doug and Nina's realm because I'm still stuck in enterprise land with campus and wireless and people complaining about connecting things.

Doug Madory: I like the role of a storyteller now. I like the putting together interesting stories and telling those stories, and I feel like that's our role these days, and it's an important role. It's kind of fun.

Leon Adato: There's a quote I remember from a Rabbi that science takes things apart to tell you how they work, and religion puts things together to tell you what they mean. And I think that as you want to say developer relations, tech evangelists, thought media, whatever, storytellers, Doug, is a great thing because I think what we do is we take that technical experience of taking things apart to see how they work and then putting them back together to make them work. And then now we put ideas together to help people understand what those technologies mean and what they can do.

Phillip Gervasi: I never really looked at it that way and I'm glad, especially Doug, that you brought it up because that helps me a lot. This has been like a counseling session for me thus far. That's one thing that I've learned. I'm in my mid- 40s, so for the audience, I'm not super young or super old. I feel like I'm right in the middle. But I can say, as I get older, not just in tech but just in general, as a human being, I am more comfortable reserving the right to change my mind and to say, " Eh, let's do this now and this is better," and not worrying about it. Because you know how folks will say like, " Well, I don't have any regrets in life because the regrets make you." I have a list of regrets as long as my arm where I'm like, " I wish I never said that." That was objectively and subjectively stupid. I regret it. But in these kinds of things where it's like I teach that to my kids. My oldest is almost 17 and she's been thinking about school, college, what she wants to do. She has a lot of different ideas, several of which she's very passionate about. I'm like, " Sweetheart, you could start and change your major. You could change your career eight years down the road, five years... You're allowed to change your mind and it's okay. Don't worry about it. Just whatever you're doing that's in front of you, you do it at a hundred percent. If it's school, get straight As. Whatever your major is, get straight As, and then you can change your major." By the way, just to throw it out there, I introduced her... You know how you could take all the classes for free online at Harvard, MIT, and Stanford and stuff. So she's taking Harvard's CS50, which you may be familiar with. It's a very, very famous class. It's available both on YouTube and on Harvard's website. I introduced her to that because I've been sort of gently nudging her toward the tech field without pushing her. And yeah, she was reluctant. And then when she started to take that class, it was just a recommendation. I said, " Check it out. You have some time here during Christmas and New Year's. You have time off from school." She fell in love with it. She keeps coming downstairs to show me her program that she made for the class, and she submitted it and got an 8/ 8 or a 10/10. She's really, really into it and as her dad, it's making me really happy.

Leon Adato: Nice.

Doug Madory: Nice.

Phillip Gervasi: They're not talking about AI in that class. They are talking about object- oriented programming and for loops and things like that, but AI was definitely one of the big things in 2023, I think. And I know ChatGPT technically, as far as when it came out publicly, the very end of 2022, whatever. 2023 was certainly the year where we were talking about it constantly, whether it be ChatGPT or AI or LLMs or anything in that realm. Certainly, we were at the peak of the hype cycle at some point. Where do you think we are in that right now and where we're headed?

Nina Bargisen: I think you're right. We're in the hype cycle, but we're also in a place where... I mean we've been using... AI or some sort of machine learning has been used in our industry for a long time already, but then the large language model is kind of what is in the hype right now, and that's basically just that we can talk. I mean we can talk to something and then shit comes out, where before you used to be more savvy, but basically, we just got an easier interface into something that maybe already existed. And then one thing that really, really just blows me away in the bad way is how, at least here, I hear journalists and other folks who are like they want to run on the IA hype and they make radio programs and they start using ChatGPT and the likes as search machines and they start believing things that come out and sort of like, "Let's have a discussion about ChatGPT said this and this and this is going to happen. It's making predictions," and completely ignoring where those predictions come from. Informing people, which I think is one of the most important things to be informed about when you want to use ChatGPT, is that you cannot trust what the fuck is coming out of there. If you don't know what you're asking it about or asking it to write about or draw about, you might be making a complete fool of yourself because things that come out might be wrong because it's writing based on what is learning and on probability and not on facts. I just don't understand, now in more the public way, why do people not talk about that, so the big warning sign, " Do not trust the facts. This is not facts." Is it just me being in my little stupid corner and this was the reality six months ago and now it's all fixed?

Leon Adato: No, no, no. I am nodding like I'm a heavy metal lead guitarist shredding on the guitar. You are absolutely 100% right. Really, this was generated with LLM should be a warning like the surgeon general's warning, " The information may harm you. This information is not for pregnant women or not pregnant women or anyone really." Going back to the prediction part, the buzz about AI is simply not going to die down. It's still too heavy in the hype cycle and too many people see some sort of angle. They can push their career forward by jumping on this bandwagon. We're going to see AI or machine learning or LLM or 3 IF statements in a trench coat, whatever you want to call it. Because, Nina, to your point, a thing, a natural language interface has been around for a really long time. For those people who remember there was an old database product called Q& A, and that was its whole thing was that it hid all the internals of the database behind it, and you could ask natural language English queries to it and it would convert it into some form of SQL. And that was back in 1989, 1990. So something like this has been around for a really long time. And we're going to see this AI interface baked into a metric Aston that is a technical measurement of products and some of them have no business having AI integrated into it, but they're all going to have it because everyone feels like they have to have it. They'll all be rushed to market. And what we're going to see over the next year, I believe, is companies coming to terms with how costly it's going to be both in terms of money because a lot of people will integrate Chat Gippety, and that is how I prefer to pronounce it. Thank you, Corey Quinn. That integrating Chat Gippety into it, and they don't realize that there is a charge that they have to pay themselves for every query that their users end up using, so there's a dollar cost. But there's also a cost in terms of distraction integrating it is a non- zero prospect for your engineers. And they're going to realize by the end of this year, 2024, we could have built all this other stuff and we didn't because we jumped on the Chat Gippety or LLM bandwagon. On the other hand, I think that there's going to be a few vendors who realize that it's much easier to develop internal LLM, that there are tools now that will let you develop your own self- hosted large language model that is domain- specific. Oh, I can have my users ask meaningful questions about my stuff in a way that really helps them, and they're going to build those and integrate it, and it will become, by the end of 2024, it'll be very, very obvious which a vendor chose to do and which one people prefer to use. That's my take on AI. We're never going to get rid of the hype cycle though this year. It's going to keep on going, and we're going to all be even more sick of it than we are now.

Phillip Gervasi: Yeah, the entire idea of natural language processing, if you think about it, really started with the advent of computers. I mean in the 1940s and'50s especially the 1950s when we saw the first experiments with translation from English to Russian and vice versa and putting in a human input and then getting an output also in natural language, and the input was in natural language. Isn't that how we've always interfaced with computers, even our code? Some of it's less intuitive and less natural than others, but there was always this endeavor to make it more natural. And then ultimately, I look at this as our interface with data. So I agree with you, Nina, that LLM is within the realm of neural networks and deep learning and there's an element of correlation and probability and probabilistic, deterministic causality, all these kind of things, and machine learning behind the scenes. However, the LLM, like you said, it's an interface between us and the other activity that's going on. The other AI, the other ML, or just the data, whatever kind of very, very mundane activity going on underneath. And in that sense, the hype cycle is totally going to... We're going to go past that trough of disillusionment and get into, all right, what are the real practical applications of being able to interface with data in a much friendlier and faster way? And I don't know the answer to that. I do know that for what we do at work, we ingest so much information and so many different types of data that I see the benefit there. You can ask questions and then the platform itself can munch the data for you, how it's going to be applied across other industries. I don't know. I don't know. But I do agree with you, Nina. It's kind of this thing that's been conflated. ChatGPT and other LLMs that is the SkyNet that is the intelligence, and not really, not exactly. It's our interface between us and the machines and the computers. I like saying machines a lot because that's in the literature, especially the old literature. Yeah.

Nina Bargisen: I'm kind of thinking I was wrong actually. Now, when you're agreeing with me.

Phillip Gervasi: Well, how so? What do you mean?

Nina Bargisen: It's because of my experience, or just again, I'm thinking about ChatGPT, the fact that you get the wrong things out of it. It's trained on data and basically, you ask it to write something. It's just bringing out the text with the highest probability of being around what you're asking it. It's like a very, very high level understanding. So what you get back can be factual wrong. So that's why the interface to data, to me, that doesn't really ring well because you actually could get the wrong data back. So again, actually, I'm not even sure I understand what it is. But it's not going away. That's a fact.

Phillip Gervasi: To be fair, when you're on the ChatGPT interface, there's a warning on the bottom. It says, " ChatGPT can make mistakes. Consider checking important information." The mistakes as we know are mistakes in math, but also the mistakes that manifest themselves from a limited data set. We're looking at, what is it, the internet from 2001 and beyond and newer. I don't remember if that's the exact data set that it's using to scour to produce an answer for you, but whatever the answer is-

Nina Bargisen: Well, I think that's one of the things that keep really tight is exactly what data they're using to train the big model. But I guess the thing that Leon was talking about that for your special purposes, you can now build your own model, the music technology, and then the things you'll get out that, it's probably more correct because you're training it on specific amounts of data. But then again, yeah, I don't know.

Phillip Gervasi: Right. But that's why I think the future is from that macro level that we started talking about a few minutes ago. The future is this technology. It's not necessarily the specifics of how ChatGPT works or how Google's doing their thing, but how is this technology then going to be applied in the broader sense and where are people going to go with it? How can it actually be helpful to IT operations or to other industries in general? And I have a friend who... She's not my friend. She's my sister- in- law now. My brother just got married recently and so... No, no, she's my friend, but my point is, for a long time they were going out, and I didn't call her my sister- in- law and now she is. So I'm in the adjustment period. Anyway, what they're using some of this technology for in her industry is completely different. I mean, sometimes I wonder if there's a bunch of plumbers or lawyers or choose a trade and they're having a podcast right now talking about the changes in their industry and how things are changing and the thought leaders of electricians just like we do. I do think that this is just the progress moving forward that we've seen. Machine learning being around for literally decades, college- level statistical analysis, that kind of math that we use even at Kentik to solve very easy, or not easy but basic problems. We apply models and algorithms to produce a result. The result isn't that good. It's not what we're looking for. We try something else. We try to start with what's simple because it's less taxing on the actual gear that we're running the algorithms on. I mean, that's what we've always done. I agree with you, Nina. I mean we're just going to now as the ability to use more compute as the ability of networking connect to connect GPUs to do more advanced workloads develops. We could do cooler stuff. Cool. So it's just kind of a natural progression. I don't see this as this giant, everything is new and different now. I don't see it like that. I just see it as a natural progression. But that's AI. That's AI and LLMs. We've had a lot of other stuff going on this past year. So for example, do you all remember the Azure outage last year? That was a big deal. We've had some outages this year. Our cloud service provider, CSPs that are supposed to be invincible, and they don't go down because of the incredible resiliency and redundancy and fault tolerance that they have built- in, well, they went down. Well, at least one of them went down. Doug, do you remember that one? The Azure one?

Doug Madory: Yeah. So I wrote up in January this year an analysis of that outage. And so it was a networking problem with Azure that seemed to focus mostly in the Asia Pacific area. I can't think we maintain a full mesh of tests between every cloud region and every other cloud region of every cloud, so there's like 10,000 combinations there of continuous measurements, and from that wide net, then you could draw around a circle of what are the links that went down there. In the end, Microsoft put out a post- mortem blaming a misconfigured router command. It sounded very similar to the Facebook outage in October 2021 where you had a seemingly innocuous, just a routine test that ended up leading to arguably the largest internet service outage in the history of the internet. In that case, in the Facebook case, this obviously was not as impactful. But in 2021, we had a series of these cloud outages, content providers going down. It seemed like the following year we didn't see so many. I think 2021 was a banner year for that kind of thing. I think there were some lessons learned that came out of those that hopefully, maybe people tightened up and understand, gained a healthy respect for the complexity that's involved, and understanding, there need to be a higher degree of testing of some of these change modifications because the consequences can be catastrophic. But in that case, yeah, the Azure outage, it's ancient history now. It's a month ago. But at the time, it was a big deal. We wrote something up about that. Another little angle too. I feel like on any of these outages, I feel obliged to point out, especially as a BGP analyst, is that people who are in the space of internet monitoring, BGP is one of those things that's a data type that's fairly easily accessible and a lot of people have got ways to go look at this. There's public tools you can look at, and so when there's some evidence of the outage in the BGP data, then sometimes people jump to the conclusion that BGP was the cause of the outage as opposed to the opposite. This was another case where we can see that by virtue of the net flow that we have, I can see routes that went down obviously are no longer passing traffic, but then routes that stayed up also no longer passing traffic. So that means that it's not a question of reachability. That was a conclusion of the Rogers outage last year, but it was true again for Azure.

Phillip Gervasi: Ultimately, your prediction for 2024 and beyond is that the network still matters.

Doug Madory: Yeah. And it's super tricky. I mean, Azure, on the grand scheme of things, this is an outfit like AWS was in 2021 or Facebook. These are organizations with relatively infinite resources. That's not the missing ingredient. It is just these are such huge operations, and the complexity is your enemy. Sometimes you thought you understood what's going to happen and then it ends up, you're down for hours.

Phillip Gervasi: Yeah. And certainly, we could talk about how AI in the future can be a part of the whole operational aspect.

Doug Madory: That would be interesting. I could come in and solve that problem. I don't know that we're there yet, but maybe that's a possibility by the time we're all retiring.

Phillip Gervasi: I got a couple of decades, my friend. I got a couple of decades.

Leon Adato: I just want to pull out that it is emblematic to me. The Azure outage of'23 was less impactful than the Facebook outage of'21. That the Facebook outage was measurably more impactful. That was an internet outage, whereas Azure going out was not. Now, I could take that as a throwing shade at Azure like, " It didn't really matter." But I also think that, again, I mean back to the network matters, it does. But at the end of the day, when you think about what impacts end users, what impacts people, it is the loss of the ability to get work done. And I recognize that saying getting work done in Facebook is an oxymoron. But if Facebook represents the application that people are actually using and Azure represents the foundational infrastructure that it runs on, a lot of times, not always, but a lot of times it's the lack of ability to get to that tool that matters, and I think it's worth remembering as people who love the network and have come up through the ranks and the network side, that it's still very much, the network matters, but it's still about the application.

Doug Madory: I would clarify also that the Azure outage was a partial outage. It was more regional than a total thing. And then, with Facebook, when they lost authoritative DNS, that meant everything. There's nothing anywhere it would work. So if Azure were to experience a total global outage, lasting whatever, seven or eight hours, it would be up there, maybe not quite at the Facebook outage level. It might be number two if it were that same scale.

Nina Bargisen: That's partly because... I mean, Facebook have billions in the American billions understanding of uses. And a lot of it is entertainment and people actually do care a lot about their entertainment. I don't know if you guys are aware of that, but in certain areas of the world, Facebook is actually selling services to businesses for authentication and all that kind of stuff. So the Facebook outage actually did mean that some companies could not get their work done. A lot of people selling stuff on Facebook couldn't sell their stuff, so a lot of businesses were affected by that outage. It wasn't just entertainment being down or for some people the internet being down.

Doug Madory: I just want to point out, I appreciate Nina your clarification of which billion you're using. You think numbers are the same and math is the same everywhere in the world. It is not. The word billion has a different meaning in Europe than it does in the United States, and in Europe, it's a million million, which would be our trillion. A couple of years ago I learned this and it continues to blow my mind that the word billion has different meanings depending where we are.

Nina Bargisen: Yeah. And somehow you guys missed out how to count.

Phillip Gervasi: There was another major outage, the continent of Africa, and I don't remember specifically which nations suffered a pretty severe internet outage. So connectivity, not just Facebook, due to a pair of submarine cable breaks. Doug, can you tell us a little bit about that?

Doug Madory: This was kind of fascinating. I know I too am a person who's fascinated by submarine cables, but this also brings in some geological stuff as well. So the Congo River is a major river of the continent of Africa, and it comes out in the South Atlantic and it's been flowing for, I don't know, millions of years. And in that time, it's carved out this canyon that goes underneath the seabed, and this is the world's largest undersea canyon, and it is a geographic feature that submarine cables have to navigate as they go down the West Coast of Africa. And so one consequence of this geographic feature in the river is that there are regularly landslides, and we're talking just massive, massive events happening under the sea, these landslides triggered by the river. And so every couple of years there's a landslide that's big enough that it catches a couple of the submarine cables and breaks them. And so that's what happened in August. We had another major undersea landslide. It took out SAT- 3 and WACS, two of the major submarine cables going down the East Coast. The latest one is Google's Equiano cable, and that I guess was laid in a way. They were able to avoid this undersea canyon. Anyway, so I wrote up a post on this and went back and looked at some of the previous events of undersea seismic activity. There was a big event in 2008 around Taiwan where there was a lot of cables that were taken out due to an earthquake. The major earthquake that hit Japan in March 2011, triggering landslides, and I remember going to a submarine cable conference a couple of years later where they had a panel of folks from Japan talking about the experiences of reconnecting those cables and it was fascinating. Everybody was just completely in wrapped attention listening to this. And one of the details that stood out, I put it in the blog post, was how normally in these cases where there's multiple cables, there's a negotiated agreement of what's the priority list and different cables will pay a little premium to get bumped up on the priority list to get fixed before another cable based on which companies are going to come sail a cable repair ship over to fix them. That priority listing just went out the window because there was this cloud of radiation coming from Fukushima that they needed to navigate around. The priority list was, where can they sail where they're not in a cloud of radiation? That's what gets fixed first. But one of these cables, when they went to try to find it, had been dragged kilometers under the seafloor. Again, that's something. I don't have a lot of firsthand experience on the ocean seafloor, but to have a landslide that drags the submarine cable kilometers, it's just amazing to think about what takes place down there. It can be a dangerous place for submarine cables, the seabed.

Phillip Gervasi: Sure. Yeah. The 2Africa cable is largely lit. I know that they made the first landing in Europe.

Doug Madory: Oh, yeah.

Phillip Gervasi: I think in Italy was where it starts, and it goes around the entire continent of Africa. That's currently the longest submarine telecom cable in the world. I don't know if many of the spurs are lit because they were going to use that.

Doug Madory: I'm just looking it up now. It looks like it's scheduled as RFS, so ready for service in 2024. So they're not even claiming that it's up yet. Usually, what ends up happening with these things, and this is something I learned over a decade ago studying these, is that there's a date when the cable is RFS, ready for service, and then subsequent to that, maybe months later, it's actually carrying traffic. And that's not to say that someone's lying. Sometimes there's another process of negotiation between the telecoms that are going to use the cable extends that time. Usually, when the cable is built, there's a big press release and photos of guys cutting ribbons on a beach somewhere saying the cable is ready for service. But that usually doesn't mean that's carrying traffic just yet because they've still got some other contractual agreements that often... I don't know. Yeah, they want to see that thing finished, and then they'll sit down and start the negotiation. I don't know. But it's quite common. Months take place before actually a single packet goes down that ready- for- service cable.

Phillip Gervasi: Yeah. And that particular cable, the 2Africa cable is... I mean that's important going into 2024. We are talking about an incredible amount of bandwidth being available. Obviously, there already are cables servicing the continent of Africa with other continents, of course. But this is bringing it to another level. The spurs are very numerous, creating connectivity where there otherwise wouldn't have been, or it was just mediocre connection quality. That's amazing that that's happening. And maybe that speaks back to what you said, Nina, that there are interesting things happening in other parts of the world outside of my zone of immediate perspective and influence in the United States and North America. But also, there was a new turn- up with St. Helena Island, correct? Staying on the topic of submarine cables.

Doug Madory: Yeah. I'll tell this one last story here. Yeah, so this was one I've been following for a while. And I've kind of made it a practice for, I don't know, over a decade now of trying to spot the activation of submarine cables, like when it's actually carrying traffic. We've got internet measurement data confirming that traffic's getting passed, and that goes back to maybe the ALBA- 1 submarine cable in Cuba. Now, next month it'll be 11 years ago, I spotted that one. I knew about the situation. I'm close friends with the German telecom expert that's been advocating for this for many, many years. And so here's the situation. St. Helena is a very small, extremely remote British overseas territory in the South Atlantic, and this is the place, if you've ever heard of it, maybe you've heard of it because it is the final place of exile of Napoleon. This is where he was finally placed to live out his last couple of years of life. Presently, there's about 5, 000 or more British citizens that live on this British overseas territory. But because it is a British overseas territory, it is ineligible for the other types of funding that would sources like the African Development Bank or something that would be in that region that would potentially pay for a submarine cable connection. This is what's actually happening in the Pacific to a lot of island nations, that it's a humanitarian or development gift to connect an island because it really can't have a modern economy without a modern internet connection. And so St. Helena was kind of caught there where they couldn't get any help, and then the UK government, it just wasn't a priority for them. To shorten the story, my friend founded this NGO to advocate. He went to the UK government, ended not getting anywhere. Ultimately, they got the EU to pay to build a spur to the Google's Equiano cable. What's interesting is that took place in 2018. So this is two years after Brexit. The EU is paying for the UK's spur, which might be one of the last direct benefits the EU gave to the UK before they left. It's interesting what's possible here. So they had to build that spur at 20 million euro, land it in St. Helena, run it to the east towards the coast of Africa, and then just drop it off unterminated at the bottom of the seafloor because the Equiano cable was not there yet. And so when they finally were coming down the coast and laying that cable, they were able to snag this off the bottom of the seafloor, then reconnect it. And that actually occurred maybe in May of this year that they actually connected those things. But again, back to the comment a minute ago that the RFS doesn't always when the traffic gets passed, so that occurred in May. It wasn't actually passing traffic until the 1st of October this year. So, again, a few months there, even though the physical connection is all ready to go, it took a couple of months before it was carrying traffic. And it was a new day for people in St. Helena because they all of a sudden have a submarine cable to split among 5, 000 people, which the amount of bandwidth per capita now in that small remote place went from almost zero to nearly infinite. I know we posted a graphic showing the spike in the amount of traffic we could see coming in and out of St. Helena. Cloudflare posted something that was similar. It's like 100x, I mean just the volume of traffic, the capacity is just day and night.

Phillip Gervasi: Yeah. So it's important to remember for the audience's sake that there's a very big difference between satellite connectivity and then a hard wire under the water satellite connectivity provides maybe a percent of global internet connectivity, largely for remote connectivity, specialized use cases. Obviously, if you're in St. Helena Island prior to the submarine cable, that's what you use because that's what you got. But ultimately, there was a study from the World Bank, 2013, 2014, or'15, something like that. I'm not quoting it exactly word for word, but a 10% increase in connectivity, the internet, broadband penetration, I think is the word that they use, the term. It results in about a 1. 3, 1. 4% increase in GDP for most, I guess lower- middle income countries. That's a big deal. And so I think into 2024, with the two Africa cable with other subsequent cables, we're already about 1.5 million kilometers of cable under the water for telecoms specifically. We're not talking about power cables. As that continues to increase, we're going to see a direct impact on the global economy, like you mentioned, Doug, and I think that's a big deal. Yeah, some of the stuff that's happening with communication satellites, Starlink, and others is really neat and interesting, but I think we need to pay attention to what's going on under the waves from a technical perspective because it's fascinating to me and how that works. You're just talking about spurs. How does a spur work? The different types of submarine cables that connect into this octopus- looking thing and how that gets dredged up in the plows that they use to re- submerge it under a meter or two meters of mud, really interesting stuff, but directly impactful to people's lives, to human beings lives from an economic perspective, from a access to information perspective, really, really cool. We could do podcasts on submarine cables all day long as you know. It's an interesting topic. But I do want to talk about some acquisitions, mergers, things like that. Now, I saw in the show notes, a bunch. Leon, you put a whole bunch in there. I got to admit. From a networking person's perspective, there were two that stood out. One being Cisco, acquiring Splunk probably also because that's related to what we do at Kentik, so we were very aware. And also the VMware acquisition because that was a big deal, and everybody's like, " Oh, so VMware is going away. Interesting. I wonder what's going to be done with them."

Leon Adato: Well, and the announcement that the CEO has made subsequent... It validates that suspicion. Things are not going to be stable or calm or static at VMware.

Phillip Gervasi: Stable or static. Those are both negative connotation words.

Leon Adato: Well, when people are told that if you live less than 60 miles away from the office, you better learn to walk on water or get in there. When they ask about ERGs, employee resource groups, and that the CEO says, " That's a foreign concept to me." That doesn't sound particularly warm or comforting as an employee.

Phillip Gervasi: They had their heyday. Obviously, Hyper- V wasn't as successful as they wanted. Their virtualization as far as vCenter, vSphere, all that, that was the de facto virtualization technology for a decade or something like that. And it could be that they are going off into the sunset, but there was a time when they were just printing money. So it is interesting to see such a large mainstay of technology go in this direction, and the acquisition just showing how that's playing out now. Why do you think Broadcom was so interested in purchasing them though, a chip manufacturer?

Leon Adato: I don't know. I mean, VMware has been part of a series of acquisitions over the last, broadly say a decade. There was EMC, VMware, Dell, all sort of acquiring and unacquiring and trying to absorb, and then that didn't work, and stuff like that. So Broadcom just continued the weirdness of that whole space. And I think it speaks to, first of all, the volatility of that space. And some of it was that we're printing money, but also some of it is we don't know what this is all supposed to be or do now. And I'll say also, especially in the age of cloud. Once cloud came on the scene and became firmly entrenched, VMware, EMC, how much on- prem virtualization do we really need or want? And I'm not implying it's zero. It is a non- zero percentage. But I don't know how much they need, and I think that the companies themselves aren't sure what they need to do.

Phillip Gervasi: Yeah, there was some disruption in that entire industry, especially with the advent of containerized microservices, Kubernetes, that sort of thing, which some enterprises are not really embracing, and so they're still in VMware land. It was also over a $ 60 billion purchase. So obviously, there's going to be some kind of investment and pivoting. And so whether that's going into like, " Let's focus on how we can enable enterprise multi- cloud and focusing on that realm and less on more P2Vs," I don't know. But that was a big one. Another one was Cisco acquiring Splunk. That meant a lot of course to us at Kentik being a visibility company. That was also a very expensive acquisition. And we know how Cisco operates. They're very good at growing and developing their portfolio through acquisition, and sometimes, those acquisitions are, they buy a company and then it gets put on the shelf. Sometimes the acquisition, like I'm thinking Cisco, when they purchased Viptela. That was an amazing acquisition. They slowly and surely turned it into Cisco SD-WAN. For a time, it was Cisco Viptela. That was one. But I can remember many acquisitions where they purchased some company and it was set out to pasture.

Leon Adato: Yeah. So everyone's wondering whether this is going to be another ThousandEyes, which was not 2023. Just to clarify, it was a while back. But it doesn't seem like a lot has come of that. Obviously, time will tell. Splunk seems to be too big an asset to simply ignore or bury. I think for 2024, since we're looking at 2024, the Splunk acquisition will mean nothing to the people at Splunk and nothing to the people using Splunk. It's just going to remain Splunk, a Cisco company, and that's it. It's just going to be a cash generator for them. Whether or not that capability gets integrated into other tools, I think is an interesting thing. The running joke, I have to say it because it was so good, was that Cisco really just wanted to buy the license of Splunk, but buying the company was cheaper. There we go. But 2024, I don't think we're going to see any change. And to be honest, again, using ThousandEyes as a model, that first year after Cisco acquired ThousandEyes, ThousandEyes remained ThousandEyes. It's just nobody has seen a Cisco- branded, Cisco- leveraged observability or monitoring tool arise out of that. It just is. And that's the part that confuses people about a lot of Cisco's acquisitions.

Phillip Gervasi: Well, it depends on where they want to put them in the portfolio. If it's going to be a security first kind of a tool or they're going to incorporate it, like you said, into their platforms like they did with ThousandEyes where it's just on boxes. The same thing happened with Meraki. Meraki was very much its own thing for some time, for actually some years. It wasn't a short time. And now, we're seeing how it's integrated and now it very much is part of Cisco and things are being rebranded all over the place. Obviously, Catalyst, the whole phone thing with Unified Communications again decades ago, when Selsius was purchased. That was integrated over some time. I think there are some examples of really successful integrations where it does take some time to see where this is going to fit into the portfolio, what problems it can solve, and then, we have those examples where nobody knows what to do with it and it just slowly dies and goes away. The technology, and maybe that's also a business decision as far as we need to get these guys out of the market. I don't know. So those are a couple of major acquisitions. I don't know if you want to bring any other ones up, but as far as networking, those are the most top of mind for me.

Leon Adato: The only thing I want to highlight because I did throw a bunch into the show notes. But I think I want to highlight just the sheer volume of companies that acquired something that had that AI component. Back to AI, you know it's the hype cycle. AMD acquired Nod. ai for an undisclosed amount. Akeneo acquired Unifai. I don't know if there's any sort of legal implication of me getting the pronunciation wrong. But here we go. ServiceNow acquired G2K, which makes Parsifal, which is a data platform. Databricks acquired MosaicML. Snowflake acquired Neeva. McKinsey acquired Iguazio. Hewlett- Packard, I will still call them that, not HPE. Hewlett- Packard acquired Pachyderm, which is, again, a data platform. AI is on everyone's mind. But once again, I am skeptical about whether any of this turns into something meaningful or useful for the actual users. The only other one I wanted to talk about in terms of things that I would've said look at until yesterday was Adobe buying Figma. That created a lot of conversation, especially among my friends who were in UI/ UX, and now, it's not because the merger was not permitted, and so it's dead.

Doug Madory: With the downturn of the tech industry in the last couple of years and the increase of interest rates, there's been, I think a slowing of the rate of acquisitions that we've seen in the last couple of years. These exceptions that we've talked about. Noted. If the interest rates start coming down next year, maybe on stealing from people's predictions for next year, but there could be some pent- up demand if capital is more easily attained and there's some targets on the horizon, but maybe we'll see a more active space in the M& A sector next year.

Phillip Gervasi: Could be. Yeah, as money gets cheaper for sure, and people are more comfortable with investing. Certainly, the trend in the cost of labor, in the cost of just the entire service chain, but the cost of doing business, I don't see that slowing down. So I don't know how much that's going to offset the cheapness of money as interest rates come down like you said. I know that the Fed, we're still looking at increasing rates, but that's more in the context of the housing market, I think. I do see what you're saying, and it's not something I thought of, Doug. That's interesting. So as money gets cheaper, getting into 2024, if it does, will that spur some more mergers in that direction?

Doug Madory: Well, it better because it's already been priced into the market at this point. So the recent announcement by the DronePal, it'll be applying. There'll be cuts next year. Marketers already reacted to that. We're already seeing the benefit. So if it doesn't then...

Phillip Gervasi: Yeah, and I wonder if that's going to be more on the smaller and medium- sized tech companies too. Not necessarily on the very largest tech companies that seem to be flush with cash. I mean just because that's the kind of business that they do, and I think some of those are the more interesting acquisitions too, when you have two medium companies that go through some sort of a merger and come out as this new thing as opposed to Juniper Cisco on the non- commercial side like Apple, things like that, buying a smaller company. A huge company, buying a little company, those are less interesting to me. But yeah, we'll see. That's an interesting prediction I hadn't thought of Doug, so yeah, we'll see in the next coming year. What about from a technical perspective? Obviously, we've been talking about AI quite a bit. What do you foresee 2024 in that context?

Leon Adato: Okay, I'm going to get all the ones that we have to say like everyone's obligated. I'm going to get them all out of the way. 2024 is going to be the year of VDI. It's going to be the year of Linux on the desktop. It's going to be the year of flying cars. It's going to be the year that IP version 6 finally takes over. That one's for you, Doug. And finally, this one's mine and only mine. This is the year that the Cleveland Browns go all the way. Okay. These are all the predictions we make every year and none of them come true.

Phillip Gervasi: The VDI one, I'm going to make a point though, every year there is an attempt to make it the year of VDI. But every year it's like, " Now we have thin clients or now we have these things in the cloud and I'm just logging into my machine and AWS." It's like every year it's this sort of flirting with the idea that maybe it really is the year of VDI, and then it's not.

Nina Bargisen: A lot of people are still storing their data on their laptops, which to me is like, " Wow, really? I stopped doing that 15 years ago."

Phillip Gervasi: Yeah, I store a little bit, certain files. A lot of the time, it's stuff that I forget to upload. I don't have an automatic synchronization going on.

Doug Madory: It's still screenshots for me.

Phillip Gervasi: Oh my goodness. You're right.

Leon Adato: I have some spicy predictions if we want them.

Phillip Gervasi: Go ahead. That's what we're here for.

Leon Adato: Okay. The first one is possibly the spiciest. 2024 is when Twitter, the platform called X by no one except Elon Musk, will die completely, and nothing will rise from the ashes because the ground on which it stood has already been salted. As much as it hurts me to say it because I think that we are poorer for having lost the platform that it was before April 2022 in perfect as it was. I think that it is dead in all, but name only, and 2024 is when it becomes dead in name as well. That's my first spiciest prediction. The next one is that unless my daughter gets married and moves the bakery that she runs in my basement out of my house, I will never lose the 10 pounds that I've been trying to lose for a long time. So I'm just putting that one out there. And the last one, this is... I live in Ohio, which for those people who aren't in the US, and many people who are in the US, it's on the Northern part of the country, sort of near the middle by the Great Lakes, and Ohio has a bill to lock in at daylight savings time. To not keep changing times with the seasons. As someone who maintains an application that is very, very time- sensitive, it has me terrified. I don't know how the people who maintain libraries are going to update when states start willy- nilly changing how they handle their time zones and how they handle their daylight savings time, and things like that. So 2024, maybe the year that the US completely screws up the entire concept of time zones by making it a state- by- state thing.

Doug Madory: Maybe UTC. We'll all just go to UTC.

Leon Adato: Please. I'll take Swatch time. I'll take anything as long as it's consistent. But oh my gosh, yeah. So those are some of the spicier...

Phillip Gervasi: Agree. I want to lose 10 pounds as well. The second one though about Twitter, I am going to disagree. I don't think Twitter is going to die. I do think that there was an exodus of a lot of people and the dynamic changed very much, but I still get a tremendous amount of activity. I haven't had any negative issues in my usage of Twitter, and I see other people really upset about it, going around saying, " Oh, everything is toxic constantly." I don't see any of it. I don't see it. Now obviously, I don't see it because I curate my feed, so I don't have things in my feed that I don't want to see. So there's that. That's an easy one. But as far as it dying, I still see a lot of activity. I don't see the activity from a certain group of people that I interacted with a year ago. That is true. Honestly, I don't see them anywhere. That's the other thing. So I'm like, " Okay, let me go hang out with them on Bluesky." They're not active on Bluesky. They're there with accounts and they barely use it. Remember the Mastodon blip for about a minute and a half, almost no engagement, and that sort of died. I think Bluesky is probably going to be the main alternative for a lot of folks. A lot of people went into private Slack groups, not public. Some public, but a lot of private Slack groups, Discord groups, which changed the entire nature, I think, of our community, and people interacting and just throwing stuff out there like, " Hey, I'm working on this config." I remember those days. I hate to say this, but Reddit and LinkedIn seem to be pretty good lately. Reddit is like a war zone. Sometimes you have no idea. That could be dangerous, but there is actual engagement. So I will say that. And then LinkedIn, which was a surprise because that's the platform that I hate to love and love to hate. I'll put something out there and I'll get a ton of engagement, whether it's a question or a silly meme, so I don't know if Twitter is going to outright die.

Doug Madory: The business of Twitter X is at risk for sure, but it still has a value and there isn't a replacement. I mean I'm mostly just doing kind of the BGP internet community discussion, which I just haven't seen that replicated anywhere else. Definitely, we may see something like a bankruptcy or something, but as we all know, you can have a bankruptcy and the thing still has a lot of value, and so it'll continue on in some form even if the owner is running the advertising business into the ground.

Phillip Gervasi: Yeah. I have a spicy prediction, we're going to have some major security breaches, and as usual, nobody's going to care about security.

Leon Adato: No.

Phillip Gervasi: No. We're going to have a couple of big ones, whatever, maybe more. And I do believe that they're going to be all over the place, but the ones that are going to really hit the news and that we're going to care about a lot just because they're top of mind are going to be with cloud service providers. So the big CSPs, we're going to see, because that's the hot attack vector moving forward right now, whether it's from a networking perspective or some other perspective. Resources in the cloud, that's a big attack vector moving forward. It always has been, but I think we're going to see more of that. And at least on the enterprise IT side, nobody's going to care about security still. That's been a thing for me that people say, " Oh yeah, we care about security," with a little wink, and then nobody puts that much money into it. We hired a CISO, I get it. We have a network security team, but it's minimally staffed and like, " No, stop it. We need full access. Every packet goes everywhere, whatever. Whatever it is." And then you have a security breach, you pay some lip- services security, you increase your budget, you buy a tool, Security Onion for everybody, and then, it goes away and people stop caring. I think, and this is my curmudgeon opinion based on my years in enterprise IT. People don't really really care about security. You might disagree, but I think that's what's going to happen. We're going to have a couple of big outages or rather... Well, it could be outages, but we're going to have a couple of big breaches and people are still not going to care.

Leon Adato: I will only disagree in the quantity. I think we're going to have one to two a month.

Phillip Gervasi: Okay.

Leon Adato: Big breaches.

Phillip Gervasi: That's a lot.

Leon Adato: Big news catching breaches. And yeah, nothing will change for the individual companies, the industry as a whole. It pains me to say it, but you're right. People ultimately don't care. I think companies have learned that they can insurance their way out of security breaches. They can just pay whether it is malware or ransomware insurance, and they can, okay, so they pay a little bit of a penalty, a million dollars here, a million dollars there, eventually, it adds up to real money. But they don't care. Ultimately, they just keep on moving along.

Phillip Gervasi: Yeah, I mean there was some talk some years ago about whether it's with the United States or in a consortium of countries or however it was going to be done, where there's legislation so that there are consequences to not doing security best practices, and then that result ultimately in a breach so that there's harm being done to customers, to people, and so the company's going to be held accountable. And we do see companies having to deal with fines today. So there is a little bit of that element. But I think it was Greg Farrow. We were at Networking Field A,'11 or '13, so this was, I don't know, 10 years ago, eight years ago. We were talking about, I think, it was Target. Target got hacked. Credit card issue and there was a certain cost to that. They just had insurance, and the insurance, the premium, and the cost of getting everything back online and everything was cheaper than how much they would've had to spend on a yearly security budget. From a business perspective, they were like, " It's just cheaper for us to deal with the issue. We have backups. Backups are our security measure." And I remember everybody at the roundtable was like, " Yeah, but the reputation, the reputation." And Greg was like, " Hold on, let me look it up." He's like, " Yeah, their stock dipped for about 1. 7 seconds and then it went right back to normal." And the same thing happened with the Bank of America, Facebook, yeah, there's a little bit of a blip there and then it goes right back to normal. People have short memories and attention span.

Leon Adato: The target breach is emblematic because that was the one... If you remember, yeah, because the way that they got in it was they hacked the point of sale system, the POS, which is point of sale, not the other POS. But the way they got in was that they had actually just compromised a HVAC heating and ventilation HVAC technician's password to the HVAC system, which was on the same network. It was all one network. It was on the same network as the store point of sale systems. So they hacked the HVAC system and just moved laterally into the point of sale and stuff like that. So this was a network breach. I mean this was our space, the four of us, and still, I was like, " Yep, whatever. It happens. Moving on."

Phillip Gervasi: Yeah, talk about best practices. We both know, we all know, the four of us, that it was number one, they were all on VLAN 1 and there was no access list or VLAN access list and the password to that HVAC system was admin, admin, or admin, no password almost certainly. And it was on the 192 network. So all are like a perfect storm, I guarantee. I don't know if that's true or not, but that's my guess. How about from a connectivity perspective? What do you see as the future of connectivity in 2024? I already made my comment that we're going to see more activity with the submarine cables. But what about, let's say with satellite connectivity? I mean that's been a hot thing for the past few years, especially with Starlink. What do you think?

Doug Madory: Maybe we'll see more constellations come active. Amazon's got the Project Kuiper, these Chinese constellations out there in the low Earth orbit space. Obviously, Starlink is the leader in the space. The point I thought I'd bring up, that I just brought up in my keynote at a digital rights conference just recently was, when there is an outage, whether it's a shutdown or you'll see on Twitter X or other social media people trying to get Elon Musk to turn the internet on and Country X using Starlink. There's some complications to actually doing that. One is that these services have to have spectrum allocation. They have to have local authorization to operate in these countries or it's not a technical matter. The satellites are there already and if someone's got a ground terminal, then they just need to activate not black out the country. But in general, nobody's done this so far, and it seemed like it was unlikely to ever happen. However, last fall, Starlink activated in Iran without authorization from Iran, and following the internet outages, there was a lot of protests in Iran last year after the death of a woman in police custody, and Starlink turned on the service. It's unclear how many people, if any, actually were able to make use of it, but Iran took this to the ITU, which is the UN's body for ironing out international telecom issues, mostly around spectrum allocation. And maybe it's two months now ago, the ITU published their findings where they kind of sided with Iran that Starlink had no authorization to operate and that they're directing the US and Norway to have Starlink deactivate service in the country. ITU has no enforcement mechanism, so it's unclear what's actually going to happen with that. But I don't know that we're going to see more of those. It would be really a game- changer if these satellite services were able to restore service in a country that was experiencing a shutdown and it's less of a technical issue. The other issue is the ground equipment. It's often illegal to bring it into the country. So a man named Alan Gross spent years in prison in Cuba for bringing in satellite equipment. This is long before Starlink. This is 29... I don't know how many years ago this was. And then radiating the equipment, if someone were really determined, they'd be able to find someone connecting, so you may be imprisoned or worse depending on what country you're in by using this stuff. I don't see that being a solution going forward as much as it sounds like you should be able to just say, " Bring that satellite over and just beam down internet into this country." It gets a lot more complicated than that.

Phillip Gervasi: Yeah well, in spite of those complications, I do think that we're going to still see advancements in both the proliferation of the technology among various companies, not just Starlink. We're going to see some more, and then we're just going to see more activity of satellites being put in orbit and the connectivity becoming more available. Hopefully, that also means cheaper, more ubiquitous, especially in, I guess you could call the residential market, which is kind of where you see a lot now anyway. Folks that are just sticking it on their RV or I know with Starlink, for example, you can just purchase the hardware and then turn up the service as you want it and then turn it down when you don't want it, and just pay like five bucks a month to keep it going.

Doug Madory: There's been a lot of satellite internet projects over the years, whether it's Iridium or things in the past that have been bankrupt, and it's still not clear that this is a business that's going to make money. I know I was just having lunch a couple of months ago with a good friend of mine who's in the satellite business, and he was like, " The fact that Starlink owns their own delivery mechanism, the company also can put its own satellites in space, that could be the thing that makes that one service successful where all the other ones have to use an external service to put the satellites in space." That may be the one defining characteristic that makes it financially viable than the other ones. We don't know that much about the Chinese projects. Those may be state- sponsored, so profitability may not be the top concern.

Phillip Gervasi: Yeah. Well, that's interesting. On a more local level, not outer space necessarily. I think that we're going to see continued increase in performance from chip manufacturers, especially as we continue to build bespoke data centers for crunching AI workloads and doing more long- term jobs like machine learning jobs that take three months to do the networking that supports that requires extremely, extremely high bandwidth, 400, 800 gig connectivity at the endpoint. And in 2024, we're going to see that become more common. And then as you know with bandwidth, it's going to keep going up and we're going to consume it and we're going to want more because that's how we're in networking, more bandwidth. So I do see that happening and just the quality of connectivity specifically in the data center. But I always liken it to like how the technology in F- one racing eventually makes it into my Toyota Camry. At least some of it. Not all of it. But some of it, you have the development in the highest end realms. In tech, maybe it's in the web- scale or these very specific purpose- built corner cases for AI. And then some of the technology adopted then and changed a little bit to fit the more everyday networking. So I think we're going to see a lot of that happening as those chip manufacturers then can redeploy some of that technology into their everyday router switches, things like that. So we're going to see some increases there, which just makes sense, considering the increase of the number of people on the internet, the amount of connectivity to services that are not sitting in the server down the hall, but in the cloud. Which kind of leads me to my last one and this I'm going to put out there as a question. What do you think about enterprises continuing their lift and shift in their adoption, a continued adoption of cloud, and then services in the cloud, microservices, or this kind of realization that maybe we need to take a more hybrid approach and bring some of these things back on- prem. We're seeing some companies bring most of their stuff back on- prem and look at this differently for a variety of reasons. And that's been a new trend this past year, just lifting and shifting everything I got and sticking it in AWS or Azure, that wasn't necessarily the right thing to do.

Doug Madory: That was mostly balking at the prices. I think people are looking at their bill and being like, " Holy cow is this is a..."

Leon Adato: Yeah, I've been watching this for the better part of 10 years. I don't remember how far back. There's been various state of the internet, state of the cloud type things. And yes, Doug, to your point, the cost is one reason where people lift and shift. Also, my phrase for lift and shift is lift and shit. That is not what cloud is for. And you don't do that. Unless you have a Brink's truck full of money, you don't do that. You re- architect your environment to...

Phillip Gervasi: Yeah, right. In a perfect world-

Leon Adato: But if you want to leverage it.

Phillip Gervasi: But come on, enterprises like Bob's House of whatever in upstate New York, they're just lifting and shifting, man. I agree with you.

Leon Adato: Go for inaudible. I mean, there's other options. But that's-

Phillip Gervasi: Yeah, that presupposes that, one, people know what they're doing, and we're talking about as far as enterprise IT, a lot of the time, it's just like, " I got 74 different hats on. Let's just get this stupid server up in AWS, spin it up, DCPROMO. I don't..." We don't DCPROMO anymore, but you know what I mean.

Leon Adato: Right. Send it. Yeah, beam it up into the sky. I have seen companies lift and shift and then shift back. I think some people do it because either security or performance or some other aspect isn't what they thought it was going to be, or they can't make it be what they thought it was going to be. There's a lot of reasons for the shift back, and that shift back has always been on the order of 10 to 20% per year, 20% at the outside, at the very beginning when people were like, " Let's go to the cloud. Oh my gosh, it sucks," and back. But there's always been a movement back on- prem for certain things. I don't think on- prem is going away. What I do think is that the term multi- cloud, there's the stupid multi- cloud, and the not stupid multi- cloud. And the stupid multi- cloud is, " I'm really worried about the stability of this. So I will put the same things in multiple clouds just in case one fails." My friend if US East fails, there are bigger problems than the pull my finger app not working for a hundred users or whatever it... And I'm obviously being hyperbolic in both directions on that. But if the cloud fails, nobody is really paying attention to your app or your capability not working. So putting it into the other cloud is probably not a good use of your money. That's the stupid multi- cloud. However, if you say multi- cloud and you mean, " I want to use the best that each cloud has to offer, whether it is the processing..." And I'm not saying this is true. I do not want... This is not an endorsement of any kind, and the Kentik legal department is now happy. But if you're going to use the processing capabilities of GCP and you're going to use the SQL database capabilities of Azure, and then you're going to use the ephemeral elastic compute abilities of AWS, let's say, and that's your multi- cloud strategy, and you've architected in a way that the movement of data across those multiple clouds does not cost you, once again, a Brink's truck full of cash, then yeah, that is multi- cloud and that is a way to do things. And as long as you like the ROI of it, that's a good multi- cloud. And I think we're seeing more of that. In fact, I know we're seeing more of that. We're seeing more people be more thoughtful. Back to my comment, lift and shift is lift and shit re- architect. Monitor and manage is a different way, but if you've built it for that environment and it leverages what that environment is good at, we're seeing more of that and we will see more and more of that in 2024.

Phillip Gervasi: Well, I certainly agree that there's going to be a reevaluation of strategy, whether that means bringing certain resources back on- prem or what does multi- cloud mean, whether I got a front- end in AWS and a back- end in Azure, whatever. I do think that there's going to be a much more critical look at how people approach cloud specifically within the enterprise, not necessarily in web- scale and some global enterprise that have a different... They live in a different world. I think the last thing, and I'll just throw this out there, is for a long time I was doing SD-WAN deployments and design, and I'm starting to hear through the grapevine because I'm not doing deployments and design for that anymore, that there are a lot of folks. They're putting in the SD- WAN boxes just as their endpoints. They're just the routers really. And then everything is just heading out to a CASB, some SASE service. There's no real point to the SD- WAN overlay other than just, it's my new router and it gets me out to whatever cloud provider. And I'm seeing that as almost like a shift where SD-WAN was the big shift from traditional WAN routing and networking for the enterprise, and it only lasted a few years. And now we're starting to see a shift towards this. And I don't know if that's a naive perspective because I'm not eyeball- deep into the technology anymore, but I'm starting to see that a little bit. I wonder if that's going to be the thing in 2024. Anyway, I think this is a good place to stop simply because by far the longest podcast we've ever had. That's okay. So I'm going to take a quick minute here. Go around. Have everybody give an opportunity to just give our audience email, social media, whatever you want for folks to reach out if they want to make a comment, question, concern. Again, we're going to go in alphabetical order. So, Leon, why don't you go first?

Leon Adato: Okay. So you can find me as Leonadato all one word in almost every platform. I'm most active right now on Bluesky. I'm also on Mastodon, which existed for far more than a minute. It is a thing. But LinkedIn, you can find me there. And you can also find my personal blog at adatosystems.com. And of course, you can find me over on Kentik. com as well.

Phillip Gervasi: And Nina.

Nina Bargisen: I think the most likely place to find me would be LinkedIn these days. And then I'm also on some obscure old RSC channels and you know what they are if you know me.

Phillip Gervasi: There you go. Great. And Doug, lastly.

Doug Madory: LinkedIn and Twitter X. DougMadory is my handle.

Phillip Gervasi: Yeah. And you can still find me on Twitter network_phil. My blog, networkphil.com. You can search my name on LinkedIn. I'm very active there as well. I do have a link tree so you can take a look at where I am active all over the place. Some of it's personal. Some of it's professional. Now, if you have an idea for an episode or if you'd like to be a guest on Telemetry now, I'd love to hear it from you. Please send us an email at telemetrynow@kentik. com now. For now, thanks for listening. We'll see you in the new year. Bye-Bye.