c/enterprise-isp-discussions

Found out my enterprise fiber SLA was a lie after one outage

Signed a 3-year deal with a major carrier in Phoenix last spring, they promised 99.99% uptime in writing. Our link dropped for 6 hours during a grid maintenance and their credit was only 2% of my monthly bill, way less than what I lost in downtime. Has anyone actually gotten a carrier to honor an SLA penalty that covered real business losses?

Our 24/7 monitoring went silent for 40 minutes before anyone noticed

Last Thursday I was on call for a client in Columbus and their core router dropped every BGP peer right at 2:15 AM. I only found out because the client's own internal alarm pinged me, our vendor's so-called 24/7 NOC never logged a single ticket during that window. Has anyone else dealt with an ISP who advertises proactive monitoring but you end up being the one catching outages first?

That Tuesday when our backup circuit actually saved a hospital floor

We had a scheduled maintenance window for a client in Columbus, cutting over to their secondary fiber while we swapped a switch. Everything looked clean on paper, but at 11:47 AM their primary circuit took a lightning hit near the building and dropped hard. The backup path we tested a week earlier held the link for 3 hours until the provider patched the fiber, and the only person who noticed was their network manager who emailed us a thumbs up. We always preach redundancy to customers, but that was the first time I saw it pay off in real time. Anyone else had a failover moment that made the extra monthly cost feel dumb at first but brilliant later?

One client's feedback changed our whole monitoring setup

A logistics company in Dallas told us their latency graph looked like a lie because we only checked every 5 minutes. We switched to 30-second probes and found a packet loss spike that happened every Tuesday at 2am, turned out their backup window ate the network. Anyone else get pushback on sampling intervals from clients?

That $1,800 fiber circuit quote made me rethink my whole backup setup

I run a small logistics office in Columbus and we had a fiber outage that lasted 9 hours last spring. My old T1 line as a failover was basically useless, everyone was on their phones in the parking lot. So I got a quote from a local carrier for a dedicated fiber backup with a 4 hour SLA. $1,800 a month just for redundancy felt insane, but we lost about $6k in that single outage between delayed shipments and missed customer calls. I ended up negotiating a shared burstable connection for $950 a month instead, still fiber but lower guaranteed speed. It's been 7 months and we've only used it twice, but both times it paid for itself in one afternoon. Has anyone else tried the cheaper burstable route or is that just asking for trouble when the main link dies?

Our redundant fiber plan was a lie, one cable cut proved it

We signed a 5-year contract with a big carrier for dual diverse paths into our data center in Phoenix. I never bothered to trace the actual physical routes. Then a backhoe took out a main conduit last Tuesday, and both links went dark at the same time. Turns out they ran parallel in the same trench for the last 3 miles. My whole disaster recovery plan assumed zero downtime, but we were down for 6 hours. The carrier's SLA refunded us 4 grand, but our own clients saw the outage. What tipped me off was a field tech's photo of the cut, both cables side by side. Now I ask every ISP for a route map and a site survey before signing anything. Has anyone else found their 'redundant' setup was really just one point of failure?

SD-WAN is not the silver bullet everyone claims it is

We ran a 14-site rollout through Fortinet boxes and honestly, the MPLS circuit we kept as backup saved us twice when the public links hiccuped. Has anyone else kept legacy routes alive just to sleep better at night?

Drove past Level 3's old PoP in Fresno, whole block is dark now

Stopped for gas near the old building on Shaw Ave and noticed the generator yard is totally silent, no hum at all. That used to be a major hub for central valley peering, so is a silent PoP just normal decommissioning or should carriers be worried about redundancy in that region?

Our Chicago office lost internet for 6 hours because of a single mislabeled cable

IT guy at our HQ in Chicago pulled what he thought was a dead patch cable from the rack last Tuesday. Turned out it was the only link to our primary fiber provider. Whole building went dark at 10am and we didn't get back online until 4pm. Sales calls dropped, our VoIP phone system went silent, and the backup line we paid for never kicked in because the config pointed to the wrong VLAN. Anyone else had a failover setup that only works on paper? How do you actually test that thing without causing a real outage?

Why does nobody talk about Lumen's fiber rollout stalling in the Midwest

I pulled up their Q3 earnings report and noticed they only added 23,000 new fiber passings in the whole region. Meanwhile AT&T did over 100,000 in Texas alone. Has anyone else seen this slowdown or is it just me?

Vent: Comcast Business vs. a local fiber coop for our warehouse

I manage the network for a small distribution center just outside Austin, and we switched from Comcast Business to a local fiber cooperative about 6 months ago. The difference is night and day. Comcast gave us 'up to 1 gig' but we'd see drops every afternoon around 2 PM when the kids got home from school and traffic spiked. We were paying $800 a month for that unreliability. The fiber coop charges $650 for a dedicated 1 gig symmetric line with a 99.9% SLA, and I haven't seen a single outage since day one. Their techs actually answered the phone when I had a config question on the router handoff too. Has anyone else ditched a big carrier for a local provider and seen this big of a jump? I'm thinking about moving our office over next.

PSA: Stop ignoring the latency spikes during off-peak hours...

Warning: That local ISP in Phoenix promised 99.9% uptime

I went to check out a new office space in downtown Phoenix last Thursday and the property manager bragged about their "enterprise grade" fiber from a small regional ISP. He said we'd get 99.9% uptime guaranteed. I pulled up my phone and showed him the last 3 months of outages reported on Downdetector for that provider - it was down 12 times in just 2 months. The longest outage was almost 4 hours during a Tuesday afternoon. I told him I'd need to see the SLA in writing before signing anything. Has anyone else had a local ISP overpromise reliability like this?

Half the enterprise quotes I see are missing proper QoS markings because sales teams don't understand the difference between CoS and DSCP.

Saw a bid for a 10 gig fiber line last week where the account manager just slapped in 'QoS enabled' without any actual traffic shaping details, and when I asked the customer what their voice traffic looks like they had no idea - has anyone else had to redo whole contracts because the pre-sales folks glossed over these tagging rules?

Pro tip: ditch the static IP blocks for BGP if you have multiple sites

Was running 3 offices in Phoenix on separate /29 blocks from 2 different providers. Nightmare for routing. Tried getting a /24 from ARIN and announcing via BGP to both ISPs. Took about 2 weeks to get the paperwork done. Latency dropped 40% and failover is instant now. Anyone else find BGP worth the extra complexity for small multi-site setups?

That "bargain" fiber handoff from a reseller cost me 3 days of downtime

I signed up with a small regional reseller for fiber at my office because they were $200 a month cheaper than the big guys. Big mistake. Their handoff was this janky old media converter that kept dropping packets, and when I called for support they took 8 hours just to pick up the phone. I lost a full day of business (about $1,200 in billable work) before I finally pulled the plug and went with the established provider. The reseller promised the same backhaul but they clearly had zero redundancy or real NOC staff. Has anyone else been burned by going with a cheap reseller instead of direct from the carrier?

Remember when ISPs actually had local techs? Now it's all outsourcing.

Last week I had to call our enterprise ISP about a static route that kept dropping. Three years ago, I could call the same local guy I'd known for over a decade. Now I'm routed through a call center in somewhere, and the new guy tried to sell me on fiber before even reading my ticket. Has anyone else noticed their service quality tank since the local offices closed down?

Our SLA got tested hard after a fiber cut at the Dallas colo last month

We had a backhoe take out a main fiber trunk near our data center in Dallas around 3 AM on a Tuesday. The ISP claimed a 4 hour response time in the SLA but it took them almost 11 hours to get a crew on site. I ended up having to spin up a failover link with a secondary provider we kept around for emergencies. Has anyone else had luck enforcing SLA credits with the big carriers when they miss their window?

Rant: Took 8 weeks to fix a BGP peering issue that was a single typo

I spent two months chasing a flapping BGP session between our colo in Dallas and our main data center. Turns out an intern fat-fingered the AS number back in my notes from 2021. Has anyone else burned crazy time on something this dumb?

Just had a major outage at our Chicago office because of a DNS misconfiguration

Our main ISP went down for 3 hours last Thursday after I accidentally pushed a bad DNS record to our secondary provider, and it took forever to figure out why traffic was getting blackholed. Has anyone else dealt with multi-homing issues when one provider's DNS starts fighting with the other?

Chat with a datacenter guy about fiber SLAs changed my view

I was talking to a network engineer at a colo in Dallas last week, and he said most enterprise ISPs oversubscribe their fiber by 400% or more on local loops. He showed me a circuit he monitors where latency jumps from 5ms to 80ms every evening at 7pm. Made me wonder how many of you actually verify your SLA metrics with third-party tools, or do you just trust the paperwork?

TIL fiber handoffs can be a total scam if you don't check the SLA

I was paying $1200 a month for a 1Gbps fiber circuit from a major carrier for our Chicago office. Latency was all over the place and we kept dropping packets during video calls. After 6 months of fighting support I asked for a copy of the actual SLA and turns out they only guaranteed 99.5% uptime with no latency targets. Switched to a smaller local ISP that offered 99.99% with a 10ms latency guarantee and it cost $900. Has anyone else gotten burned by hidden SLA terms?

Why does nobody talk about the congestion on Level 3's backbone in Chicago?

I was over at the 350 East Cermak data center last Tuesday for a routine circuit handoff, and I noticed something off. The latency on a test traceroute jumped from 12ms to 68ms for no clear reason around 2 PM. I asked the on-site NOC guy about it, and he said that peering point has been oversubscribed by about 30% for three months now. My own monitoring tools later showed packet loss hitting 2% during peak hours. Has anyone else seen this kind of lag coming out of that hub lately?

Fell for a fiber upgrade pitch from a sales rep and paid double for 6 months

I signed a two year contract with a regional ISP in Ohio because the sales guy swore their new fiber would save me money, but after 6 months of $180 bills I did the math and realized I was paying $40 more per month than my old cable plan for speeds I never even used - anyone else get burned by a slick upgrade offer?

Hot take: BGP communities are way more powerful than most enterprise engineers realize

I was chatting with a network architect from a regional ISP in Ohio last week and he showed me how he uses custom BGP communities to dynamically route traffic based on latency thresholds instead of just prefix filtering, and it made me realize I've been leaving 80% of the value on the table with my own setups who else has found creative uses for communities that go beyond the basics?