Episode 93: Live at 5x5: Navigating a Hurricane

Narrator: You're listening to Public Safety First, a podcast to help you learn about the First Responder Network Authority and how you can be part of the future of public safety technology.

And now, your host.

Jennifer McIntyre: Welcome to the podcast. I’m Jennifer McIntyre, Senior Public Safety Advisor for 911 and Emergency Communications at the First Responder Network Authority. The episode you are about to hear was recorded live at 5x5: The Public Safety Innovation Summit held in June 2025 in Bellevue, Washington. I had the opportunity to plan this session and pull together key leaders in emergency communications centers to discuss a topic that I am personally passionate about — Next Generation 9-1-1. For more than 20 years, I worked in Emergency Communication as a telecommunicator and supervisor. The FirstNet Authority’s Acting Board Chair Renee Gordon introduced the discussion with Pokey Harris, the Executive Director for the North Carolina 911 Board, Tom Rogers, NextGen 911 Network Engineer and Program Manager to the North Carolina 911 Board, and Joshua Briggs, the State of North Carolina’s Next Generation 911 Program Manager for AT&T Consulting. It was a great conversation about North Carolina’s statewide ESinet and the transition to NextGen 9-1-1 and how FirstNet has become a gamechanger for the state. I hope you enjoy!

Renee Gordon: Welcome to Navigating a Hurricane! ESInet with FirstNet in North Carolina. I'm Renee Gordon, director of the Department of Emergency and Customer Communications for the city of Alexandria, Virginia. I'm also the acting chair of the FirstNet Authority Board. I'm thrilled to be here and be able to introduce Pokey Harris and the team to talk about North Carolina's statewide ESInet, and their transition to NextGen 9-1-1. 

Pokey Harris: Good afternoon, everyone. I'm Pokey Harris, the executive director for the North Carolina 9-1-1 Board. I have the distinct pleasure of serving as the state's 9-1-1 director. With me today, I have two gentlemen that I picked up at the airport. I offered them $100 each if they would come. I'm going to let these guys introduce themselves to you.

Tom Rogers: Good afternoon, everybody. I'm Tom Rogers, I'm the network engineer and program manager, and I do whatever Pokey tells me to do. Most days.

Pokey Harris: You're not getting more than $100. I'm telling you now.

Josh Briggs: Hey, I'm Josh Briggs, I'm with AT&T, and I am the dedicated program manager for North Carolina, and I do what these two tell me to do.

Pokey Harris: This is not our entire team, because behind what we do in North Carolina, there's a tremendous amount of people that include our staff. We're very fortunate, very blessed in our 9-1-1 office, we have a total of 22 employees, staff that do everything from operations, the technical side, our 24-hour NOC [network operations center], finance, grants, the whole gamut. But then also our partnership with AT&T, that evolved into our partnership with AT&T FirstNet. So, it's those partnerships that we're going to talk about.

So, let me start because we are talking about Hurricane Helene. We don't come here with a celebratory attitude in any manner. We had success, but we're not celebrating that success because it's still too raw and too real for a lot of people in our state. They lost loved ones. They lost businesses. They lost property. Property on mountains no longer exist. Mountains no longer exist. Rivers change direction. Everything people own floated away. So, we're not here in a celebratory manner. We come to you all humbly, just to share our story in North Carolina of what transpired with the death, destruction, obliteration, and devastation that we were able to assist in a way that we had planned, not knowing it would be an event such as a hurricane that we would be facing. 

So, with that, what we're going to share with you today is the story with Hurricane Helene, but it's also the story of our Next Generation 9-1-1 program in North Carolina. I will denote that North Carolina is somewhat different than a lot of states with our Next Generation 9-1-1 program, our 17-member 9-1-1 Board. It is a policy making board, not an advisory board. Funding is provided through legislative vehicle. All of the PSAPs in North Carolina, by legislative mandate must connect to our statewide ESInet. The 9-1-1 board does have governance and authority that some states do not have.

So, some key factors we're going to be talking about. We have 191 physical locations that are PSAPS, public safety answering points, or EECs, emergency communication centers. So, it's 191 physical locations. But then also with this we have 22 partners. Those are organizations, entities that are not funded by the 9-1-1 Board, but they have connectivity to our ESInet. That includes Seymour Johnson Air Force Base, a couple of air institutions. UNC [University of North Carolina], we've got a couple of smaller 9-1-1 centers that are not funded by the Board. 

We have a hosted call handling solution, specialized equipment for answering 9-1-1 calls. And in North Carolina, we offer the PSAPs the opportunity to be part of our hosted call handling solution. The equipment is provided through our contract with AT&T. They don't have to pay for anything. They don't have to be responsible for the maintenance and the equipment is provided to them. With that comes along, what we have called a network and monitoring assistance center, NMAC, defined in state legislation for the 24-hour monitoring of that NOC. So, a PSAP has an issue, they pick up the phone, they dial 800 number, and they get a NMAC technician 24 hours a day.

The network that we have in North Carolina for the sharing of 9-1-1 calls, if another state has the same network behind them, the same core services, we can send calls there. We can send them up to Virginia. We had the offer during the hurricane to send the calls up to Virginia. 

But then the key to this and why we have a statewide ESInet. Sure, it's the antiquated, telecommunication infrastructure that's going away. The copper wires are going to be a thing of the past. But when you look at it from 9-1-1 calls, you have a statewide ESInet that the closed private circuit network for our PSAPS. When you look at that, you can share that call data with anybody or that call with anybody. You can either do it through alt routing or abandonment routing. 

We're also looking at that tertiary communication options for interoperability. One PSAP we're sending the call to, let's say, Hurricane Helene. 19 PSAPs in western North Carolina at the onslaught and aftermath of the hurricane. We're sending calls to 23 PSAPS in the other parts of the state. Now by sending calls, we're talking about if a caller could get dial tone. That was the key. Wireless or wireline. But then we realized how do we get that call information back. That was not necessarily a task of the 9-1-1 Board. Legislation doesn't speak to that. But when you have legal counsel that is very engaged with what you're doing and believes in the program, you begin to look at ways that you can help the PSAPs. And right now, we're looking at some of that interoperability with the PSAPs for getting that information back.

So, early on we were talking with the PSAPs. "Hey, you've connected to this statewide ESInet. You have the power of this to share calls with your friends when those situations hit, when those emergencies come." Months leading up to September of 2024 the regional coordinators, we have four regional coordinators. They are liaisons with anywhere from 26 to 32 PSAPs in the state. They coordinate with them between our office and all that they're doing. They were beating the bushes. You've got to get a backup plan, you know, find out another PSAP that can take your calls, another PSAP that wants to take your calls.

"Hey, if you're if you're too large, we'll load balance your calls to another PSAPs." But when we started realizing what was going to happen with Hurricane Helene, we started working with the PSAPs, and one of the regional coordinators said, "You got to find some friends that are going to take your calls." And it stuck. So, that's been the secret to this, finding who your friends are, getting those MOUs in place, getting those agreements in place. So, what does that entail? In an event of, for alt routing, if all of our telecommunicators are busy, the calls are going to come to us and this is what we're going to do. If we have to abandonment route, which means, something terrible has happened. And it's not necessarily abandoning your facility, but it's that you can't take those calls. One PSAP, two PSAPs, four PSAPs are going to take your calls. You're going to work on your MOU, your arrangements to get that information back.

We encourage them to have four; one in their region and one in the other three regions. So, if you're in western North Carolina, where now we do get hurricanes, and you're seeing the aftermath even into central North Carolina with that, it's good to have folks over on the eastern part of the state taking your calls. What if you are in the eastern part of the state and you always get hurricanes? It's good to have some folks in the central part of the state. 

You have to coordinate to do this. It doesn't happen magically. You have to sit down and talk with the leadership and the powers that be. So, folks know they're getting calls from someone else. So, that was very paramount and very key because as the hurricane was hitting, we literally had PSAP managers talking with PSAP managers, county managers, sheriffs on the phone with our coordinator saying, "it's going to happen, you're going to get hit." We didn't know it was going to be the devastation and destruction that it was, that would last for months. So, folks became believers, if you will. 

So, it was catastrophic flooding. Infrastructure was destroyed. Homes, businesses, hundreds of deceased. To this day, we do not know the actual total call count or number for the deceased. Businesses are starting to reopen. There's ribbon cuttings going on every day, but PSAPs, they're processing calls, and they look down through the valley and they see what once was part of their operations.

So, our wireless phones in North Carolina, specifically, 87% of our call volume to the PSAPs, is wireless. I think that pretty much holds true for everyone that that's what we do. But with the devastation in North Carolina, infrastructure literally ripped out of the ground. When that's gone, first of all, folks can't get that dial tone. But then when they start getting that dial tone, your PSAP could have been impacted because of this. And that's where the capability of what we were able to provide to our citizens and visitors comes into play. Service, for the most part, has been restored temporarily. And why do we say temporarily? Because as they go back in and they're going to start doing that permanent restoration, we're going to see impacts again and we're going to see service that is not up.

So, I do want to say that, you know, with the destruction, the being able to route the calls, there is some sadness to this. And that's what I was talking about with, we come humbly to talk about this and we're not celebratory. What we're here to tell the story of is our operations, the technology behind that. So, I'm going to turn it over to these guys, and they're going to talk to you more about the technology and primarily our partnership with not only AT&T, but with FirstNet. And how that has been a huge benefit to the citizens of North Carolina. Tom.

Tom Rogers: So, Pokey's been talking about our network and how great it is. And it is. It's tremendously great. And it's worked really, really well for us. But like anything, there can be improvements, right? So, it's our last mile is where we see some of those improvements could be made. So, way before Hurricane Helene, a couple years ago, we started working on we need to provide a tertiary backup solution. Every one of our PSAPs, they have two, what we call AVPN [AT&T virtual private network] connections. And it's just the connection feeding the ESInet to the PSAPs from the AT&T network. AVPN is their word for the VPN connection going back over an MPLS [multiprotocol label switching] network. So, we knew we needed some additional help with that.

So, we have unplanned, unscheduled things that occur, things that we know are going to occur. We've got man-made disasters like Pokey just said, we got backhoe Bob's going to come outside your building and pull it up. So, in addition to our primary PSAPs and the 113 primaries, and then we've got the backup sites. The backups only have one connection. We feed them one connection. So, we're like, we can give them two and give them real redundancy at their site when they're working out of those. So, we knew we needed to add some of these additional features and give some more resiliency. 

One of the other projects that we're kind of working on is SIP admin. As most of you know, copper is kind of going the way of the dodo. So, we've got majority of our PSAPs have switched over to SIP admin for their administrative services. So, we let those ride that same last mile. They don't touch the ESInet, but it does ride that last mile on our hosted environment. So, they can utilize administrative phone and actually move those calls similar to how they do 9-1-1. So, we came and we said, “okay, we're going to use a FirstNet solution.” 

FirstNet is all you can use on the AT&T ESInet. It's secure and it's made robust for that. So, we knew we were going to do that. So, what we've agreed to do is we'll put a Cradlepoint at each site. All these routers are configured to see the FirstNet connection. The routers at the site. Nothing else sees them. There's no broadcast. Phones can't touch them. Nothing else. It only works when there are AVPNs don't work. It utilizes the FirstNet route and goes over that back to connect in. 

Still getting the same priority and preemption, just like you would expect with FirstNet. 

Josh Briggs: Yeah. As Tom mentioned the APN that we have with our FirstNet architecture is again, it's a dedicated, secure connection between our hosted call handling network, obviously back through ESInet and FirstNet. So, it's all traversing, through that private network together. It's never traversing any sort of a third-party VPN or a third party using the public internet. It's all using that private network provided by AT&T. 

Tom Rogers: So, through Hurricane Helene, you know, we had 19 PSAPs impacted in the actual hurricane zone where the flooding and such devastation. 17 of those were completely offline. They had no infrastructure, by and large. Madison County was probably one of the most impacted from this particular standpoint. So, they were gone down completely. They were not operational at all. They could not take their 9-1-1 calls. So, after a couple of weeks, we're trying to work with the different last mile carriers out there. And we realized, they're having to completely lay fiber in the ground. I mean, they're starting from scratch out there. So, we knew it was going to be a long process. So, we worked with AT&T. We had already started this project. It had just gotten funding in the budget. So, we had just really started. So, they were able to expedite. On October 25th, we were able to bring them up running solely on FirstNet to actually take their 9-1-1 calls. So, we're very happy about that. Originally they were abandonment routed and they're still alternate routed over to a sister PSAP a few counties over inland who took their calls for that month and were able to take their calls and send them back to them so they could dispatch those out. We also had a SatCOLT dispatched out in their parking lot, just to be sure.

Josh Briggs: I was going to add just a little bit commentary. So, the Cradlepoints that we have, they're not broadcasting any sort of SSID for the telecommunicators or the members of the public to be able to access that. That's strictly provisioned to connect to any FirstNet device. The SatCOLT was there. And for actually a third connection, we shipped a microcell out to the PSAP. They were able to use what I would call their commercial backhaul for that microcell. So, it was connected to that as the infrastructure was being rebuilt. So, cell sites were coming up and down. They're still running on a FirstNet connection between the tower site that serves that area and the microcell. They're still running solely on FirstNet today.

Tom Rogers: Like I said, we had 17 PSAPs that were abandonment routed, and we'll get to what that is in just a moment. And then we had two that were what we call policy routing. We're talking about an alternate and abandonment routing. Those are forms of policy routing. And if you get into networks it's policy routing on a network essentially is what it is. At one particular point, we had 19 PSAPs routed to 23. So, math doesn't work there, does it? It's not 1 to 1. So, keep that in the back of your mind as we talk about it when we go a little further here. Like I said, one PSAP that was abandoned for a month, and it was restored with FirstNet, and then of course, our SIP admin, the ones that were on that were able to complement their operations by moving some of their administrative calls further inland as well, so those could get answered. 

So, we talk about alternate abandonment routing. This means that we have manually switched them to say your calls are going here. Once we set it in motion, those calls that are designated for that location go to the other one. But the reason I wanted to make sure we used the word policy routing is we also use load balancing. So, not only were they taking their own calls, they were able to spread some of that load with some additional PSAPs who could help them take that volume of calls. Because like we talked about a while ago, when Pokey mentioned who can take somebody else's calls. So, they can't take all those calls coming in and somebody else’s. It's challenging. And y'all know the staffing needs of 9-1-1 centers across the nation.

Josh Briggs: And we talked about having the four friends and the pre-planning that we had in the days leading up to Helene. We knew we were going to get some impact. So, we had multiple calls with regional coordinators. We set up the different routes that some of these might not have had. 

Tom Rogers: The alternate routing there, that is used every day. We use that every day. Stuff happens, right? So, this is always in place. Now when somebody goes “not up,” we sit around and it's like, "okay, well calls are going here. We're going to get them over here. We're good." So, it makes it so much better. 

Josh Briggs: Asheville, their PSAPs stayed up the entire time. Their last mile connection with diversity. They fortunately one of those stayed in place, with all the devastation. But by design, for instance, we were sending calls to Raleigh, which is up in the north center there. And we were also sending calls to Wilmington and load balancing that. And at one point, at the height of the storm, both Raleigh and Wilmington, through the regional coordinator, got on the bridge and said, "listen, we can't take any more of their calls. We can't take any more." So, then we were on the fly, dynamically starting to move those calls. We were sending the calls from Wilmington, which is New Hanover County, to two other PSAPs to help out with the flow of the calls that were coming in. And we were doing that on-the-fly, real time. And we can do that with ESInet. 

Tom Rogers: So, some of the math that we're talking about. On average just for the impacted counties, 586% above their normal operations on the impact day. So, that's an average of just the impact area. That's not what the total is. 

Pokey Harris: Let me interject here because yeah, we do see on impact day the 586% increase. But there was one PSAP on day of impact that stood out tremendously. And that PSAP had an 11,300% increase in their call volume. So, you might say, "oh, well, you know, they're just a three-seat PSAP." To that PSAP, that was relevant, that was very pertinent to having 11,000% increase of their calls. But the ESInet was able to send that call elsewhere. 

Tom Rogers: So, everybody hears Asheville on the news. So, I figured I would use Asheville because most everybody is familiar with that. The day before, they had 443 9-1-1 calls. That's actually a little bit slow day for them. They're usually around 5 to 600 a day is typical what they run into. So, day of impact, they had just over 5,000 calls. So, significant increase, right? The day after it's dropped down, they’re still at 159% of what they typically would be. So, the calls have come down a lot. But, people couldn't call in. There was just utter destruction. There's literally fiber just laying out on the road that's not connected to anything. Copper everywhere. It was kind of a mess. So, I'm honestly shocked that that many people were able to still make connection and call in.

Pokey Harris: So, the moving forward to this, it's lessons learned. We're creating a resiliency compendium that we're looking at. How can we invest in radios to assist the PSAPs? How can we invest in CAD-to-CAD solutions and applications to assist the PSAP? So, we're currently building a catalog, if you will, that we can offer to the PSAPs at cost to the 9-1-1 board to assist them with getting those calls back when something does take place. 

One of the other things that I thought was most appropriate for us to share here, because we are with our friends from FirstNet and we're here with our friends from NIST, is to really talk about FirstNet as our safety net. So today, right now, the things that we as a staff and with the PSAPs that we're utilizing that has our relationship and connectivity with FirstNet, you'll see the Sonim radios. We're going to offer this as an opportunity for the PSAPs to have at each console that they can talk to their friends with. Also our phones. I have eSIM in my phone. Tom's carrying two phones. Other staff, they have opportunities. But AT&T, the FirstNet network, that's one of our resiliency plans. The AT&T MiFis, everybody on staff, we started transitioning. Now we carry AT&T FirstNet MiFis for our wireless connectivity, and even the iPads. So, I thought it was very appropriate that we do go ahead and share with you what a partnership that we have with them. Very organic came about out of out of need. So, thank you all very much. Appreciate your attentiveness.

Narrator: Thanks for listening today. We're excited to have you join our podcast community. Make sure to subscribe on iTunes, SoundCloud, and YouTube. You can learn more about the First Responder Network Authority at FirstNet.gov and learn about FirstNet products and services at FirstNet.com.