Field Notes — IT Director Series

On the evening of July 28, 2026, American Airlines grounded every departure at every airport it operates. According to reporting from multiple outlets, the FAA’s Air Traffic Control System Command Center issued a nationwide ground stop at approximately 6:30 p.m. ET after a connectivity failure froze dispatch, weight-and-balance, and crew scheduling systems across all hubs. The core outage lasted roughly 48 minutes. The cascading delays, missed connections, and scrambled crew positions rippled through the night. This was not weather. It was not a third-party cyberattack. It was a connectivity failure in the airline’s own operational technology — and according to industry analysts, it was the third time in under two years that the same underlying architecture gap produced the same result.

American Airlines is not an SMB. But the failure mode is identical to what I observe regularly in small-to-medium-sized business environments — and the consequences, scaled to organizational size, are proportionally just as severe. A connectivity failure that grounds 40 aircraft at Dallas/Fort Worth during peak departure hours is the enterprise equivalent of an SMB losing access to its point-of-sale system, ERP platform, and cloud-hosted productivity applications simultaneously. The physics of the failure do not change with the size of the organization. What changes is who is available to diagnose and resolve it — and how quickly. I have seen that difference play out firsthand.

The Undocumented Switch Nobody Knew Was Thereh2

In one SMB environment I supported, a subset of users on a specific floor began experiencing intermittent connectivity. The symptoms were non-deterministic — packet loss and latency spikes that occurred irregularly and could not be consistently reproduced during active troubleshooting sessions. Software and configuration issues were ruled out early. Physical layer investigation eventually revealed what software-layer diagnostics could not: a cascade of unmanaged switches had been installed by a non-IT staff member without documentation. One of those switches had a failing uplink port that degraded under thermal load during peak business hours — invisible during off-hours diagnostics, present and damaging during the hours that mattered most.

Resolution required systematic physical tracing of all cable runs and device substitution. It consumed nearly two full business days.

The contributing factors were not exotic. No advanced threat actor was involved. No sophisticated failure mode required specialized expertise to understand in retrospect. What extended the resolution time was the absence of a network topology map, zero visibility into unmanaged devices, no environmental monitoring, and a thermal trigger that could not be observed from a remote console. The fault was physically present in the environment the entire time. The documentation that would have led a technician directly to it did not exist.

This is the most consistent finding across 27 years of network infrastructure work: the single most reliable predictor of extended resolution time is not the complexity of the fault — it is the absence of accurate, maintained network documentation. That finding is not unique to my practice. Academic research is beginning to quantify it.

What the Research Confirmsh2

Mason Ned, doctoral candidate at Colorado Technical University, is currently conducting dissertation research on physical network troubleshooting skill shortages and their relationship to LAN and WLAN reliability in SMB environments — research that speaks directly to the pattern the field finding above illustrates. His work targets IT professionals with direct, hands-on responsibility for physical network troubleshooting in organizations with fewer than 500 employees — precisely the environments where the gap between documented best practice and operational reality is widest.

The findings emerging from that research align with what practitioners observe in the field. The industry-wide shift toward cloud computing, software-defined networking, virtualization, and cloud architecture has reoriented training and certification programs away from physical layer competency. Many early-career IT professionals enter the workforce without having terminated a cable, configured a managed switch from the CLI, or used a cable tester in a production environment. The commoditization of networking hardware has created an expectation among organizational leadership that physical networks are self-managing — reducing investment in dedicated network support roles, particularly in SMBs where IT staff are expected to be generalists.

The workforce pipeline is not producing the physical layer competency that SMB environments require. And the organizations that discover this during an active outage are paying a significantly higher price than those who address it before one occurs. Nowhere is that price higher than in environments where the physical network supports operations that cannot stop — and where a technician without physical layer fundamentals is the only resource available when something goes wrong.

When the Rig Goes Darkh2

A retired colleague who spent decades working on offshore oil and gas infrastructure describes a recurring pattern that the physical network skills gap produces at the most consequential scale imaginable. Vendors dispatched to offshore rigs to install firewalls, routers, and switches arrived unprepared — without foundational knowledge of physical networking or the cable types the environment required. On an offshore platform operating 150 miles from the nearest port, there is no opportunity for a parts run. There is no senior engineer two offices away who can walk over and take a look. When a vendor connected unauthorized equipment to the production network without documentation or change management oversight, the consequences were immediate and the cost ran into the millions of dollars in downtime before the fault was isolated and the unauthorized hardware removed.

The offshore environment amplifies every consequence of undocumented physical network changes. But the underlying failure — a technician without physical layer fundamentals making an unauthorized infrastructure change — is not an offshore problem. It is a workforce problem. The same vendor, with the same skill gap, is walking into SMB server rooms every day. And the skill set required to prevent that outcome — or diagnose it quickly when it occurs — is increasingly difficult to find.

The Diagnostic Skill Set That Is Disappearingh2

What that skill set actually requires is worth stating precisely, because the gap is not abstract. Effective physical network troubleshooting begins with a deep understanding of the OSI model — particularly Layers 1 and 2. Proficiency with physical media means the ability to identify cable categories, inspect terminations, and interpret results from cable certification and qualification testers. It means reading and interpreting link-layer statistics — CRC errors, runts, giants, and interface error counters — because these are often the first indicators of a physical fault, visible in switch interface statistics long before the symptoms become obvious to end users.

It means being competent with diagnostic tooling: TDRs (time-domain reflectometers), optical power meters, and Wi-Fi spectrum analyzers for wireless environments. In hybrid and SD-WAN environments, it means the ability to correlate physical symptoms with logical overlay behavior — because physical degradation in a hybrid environment may not manifest as an obvious outage but as suboptimal path selection, quality-of-service violations, or application timeouts that look deceptively like a routing or policy misconfiguration.

Documentation skills — specifically the ability to maintain and interpret accurate network topology diagrams — are consistently undervalued but prove essential when tracing undocumented or legacy infrastructure. The technician who cannot read a topology diagram they did not draw is functionally blind in an environment they did not build.

The industry is not producing enough technicians who possess this complete skill set. And as that gap widens, the business consequences move well beyond extended MTTR — they reshape how the entire organization operates under and between outages.

The Business Impact Is Not Contained to the IT Departmenth2

The business impact of physical network troubleshooting failures extends well beyond the IT department. In SMB environments, core business processes — point-of-sale systems, ERP platforms, VoIP communications, cloud-hosted productivity applications, and remote access infrastructure — depend on continuous network availability. Brief connectivity disruptions translate directly into lost revenue, missed customer commitments, and degraded service delivery.

SMBs have lower tolerance for downtime than enterprise organizations because there is less operational redundancy. A network outage may simultaneously disable order processing, internal communication, and access to business-critical data. Recovery time — during which employees reconnect, resynchronize data, and reconstruct lost work — extends the effective disruption window significantly beyond the outage itself.

The subtler consequence is the erosion of confidence in IT leadership. When network instability is chronic and unresolved, employees develop workarounds: personal hotspots, unsanctioned cloud storage, informal communication channels. Each workaround introduces shadow IT risk that compounds the organization’s security and compliance exposure. The organization that cannot resolve a physical layer fault in a reasonable timeframe is not just losing productivity during the outage — it is accumulating security and governance debt between outages. The real-world consequences of connectivity failures in a multi-location environment — and what proactive monitoring actually prevents — are documented in When the Internet Goes Dark: A Real-World Look at Connectivity, Continuity, and Proactive Network Management. The sectors where those consequences are most acute share a common characteristic: operations that cannot afford to stop.

Sector Implicationsh2

The physical network skills gap carries specific implications for sectors where 4TH AND BAILEY operates — and in each case, the failure mode is the same even when the consequences differ.

Energy and Oil and Gas — the offshore incident described above is not an edge case. Production environments, remote monitoring infrastructure, and operational technology networks in this sector depend on physical layer reliability in environments where replacement hardware cannot be expedited and qualified technicians cannot be on-site within hours. Managed network infrastructure with documented topology, change control, and remote monitoring is not optional in these environments — it is the operational baseline.

Healthcare — clinical systems, medical device connectivity, and EHR platform access depend on LAN and WLAN reliability. Physical layer failures that interrupt clinical workflows carry patient safety implications that extend well beyond the cost of the downtime itself.

Logistics and Distribution — warehouse management systems, barcode scanning infrastructure, and real-time inventory platforms depend on wireless network reliability. Marginal access point placement and unmanaged switching equipment introduced without site surveys are common root causes of performance degradation in distribution environments.

Financial Services and Legal — compliance obligations in these sectors often require documented change control and audit trails for network infrastructure modifications. Undocumented physical changes — the single most common root cause of extended resolution time in SMB environments — represent both an operational risk and a regulatory exposure.

Across all four sectors, the path forward is the same: documented infrastructure, managed equipment, and a physical layer governance discipline that does not depend on institutional memory held by a single staff member.

What Organizations Should Do Before the Next Outageh2

The path forward is not complicated, but it requires treating network documentation as a first-class operational discipline rather than an administrative afterthought. Every managed device on the network should appear on a topology diagram. Every cable run should be labeled. Every change to the physical infrastructure should go through a documented change management process — regardless of who is making the change or how minor it appears.

Procurement policy matters here too. Standardizing on managed network infrastructure rather than commodity unmanaged devices provides the visibility necessary for effective troubleshooting and reduces the diagnostic complexity that skill-deficient teams are least equipped to handle. An unmanaged switch introduced by a well-intentioned staff member to add a few ports in a conference room is a future multi-day troubleshooting engagement waiting to happen.

For organizations that cannot support a dedicated network engineer, managed network services with structured knowledge transfer — not purely transactional break-fix support — provide the physical layer governance that internal generalist staff cannot maintain alone. 4TH AND BAILEY works with organizations across Houston and nationwide to build the managed network foundation that makes this governance sustainable at any organizational size.

The Infrastructure Placement Framework’s self-assessment at framework.4thandbailey.com covers physical layer governance, change control, and hybrid estate documentation as part of Module 03 — Hybrid Estate Optimization.

→ Schedule a Guided Assessment

If you are ready to go deeper — a structured review of your network infrastructure governance, documentation posture, and change control process — the Infrastructure Governance Briefing is the right starting point. It is a focused engagement designed for IT Directors and CIOs who want a practitioner’s assessment of where their organization stands before the next outage makes that question urgent.

→ Book the Infrastructure Governance Briefing

This post references ongoing dissertation research by Mason Ned, doctoral candidate at Colorado Technical University, on physical network troubleshooting skill shortages and their relationship to LAN and WLAN reliability in SMB environments. The research represents a timely and necessary effort to quantify a workforce gap that practitioners have observed for years but that remains underrepresented in industry data.