Rant: Our main fiber cut at 3am and nobody knew for 4 hours

Had our primary ISP link go down Tuesday morning at 3am for our Chicago office. The monitoring system showed green the whole time because the backup circuit was carrying traffic. I didn't find out until I got in at 7 and noticed latency was junk. Called the ISP and they said the fiber was cut by a construction crew at 2:45am and nobody bothered to check the backup was saturated. Has anyone else had a monitoring stack fail to flag a failover situation?

2 comments

2 Comments

jennifer_murphy537d ago

Well, isn't that just the perfect way to start a Tuesday - a secret outage nobody knew about.

davidwright6d ago

Yeah that's shady. I just check the status page first thing now.