|
|
发表于 2020-9-1 17:27:04
|
显示全部楼层
Summary of Incident:
3 w( \" Y3 A; y———————————————
+ {1 l6 ?: t$ i& c2 ?9 `
8 m* p# p2 n3 I+ E
3 A* ] Y# [9 D: A; |7 C& r5 j4 X. i
Yesterday, August 30, 2020 at 6:04am EST, the Hivelocity NOC received alerts regarding a network connectivity issue tied to our Century Link/Level3 (ASN 3356) transit. At this time, our Network Engineering Team noticed BGP sessions with ASN 3356 were flapping and immediately shut them down across our network. However, after we disabled these BGP sessions, effectively de-peering CenturyLink/Level3, they continued announcing our IP space which caused our traffic to be blacked-holed within their backbone network. In laymen’s terms, if your provider was using this backbone, it could not reach our network. We quickly found that this issue was a major outage affecting the entire AS3356 backbone. This included several common Internet Service Providers who lean heavily on this backbone, such as Spectrum and other local ISPs. Due to the nature and scope of this issue, including the fact CenturyLink/Level3 was improperly announcing our routes, the typical solutions we would implement were ineffective. While working with our other transit providers to work around the problem, at roughly 11:00am the Hivelocity network as well as networks across the globe began to normalize as CenturyLink/Level3 began resolving the issue internally8 u; y' ]% J$ W- t$ |8 d$ i
- d# j. R8 R3 ZBelow is an excerpt from the official CenturyLink RFO we received this morning: : j8 ^ `) ]' S& \
2 x; P z" {* V _* I2 a7 Q& ~: e: y
Cause" K# l% e6 V5 N& z6 b+ T
An offending flowspec announcement prevented Border Gateway Protocol (BGP) from establishing correctly, impacting client services.: U5 R' f, U& I1 U. O
; Q2 p! }4 X/ S; }. D
Resolution1 o; `; H6 g+ {$ o1 J* s
The IP NOC deployed a configuration change to block the offending flowspec announcement, thus restoring services to a stable state., g9 o2 h" @1 e5 V( a$ j. B2 a6 ~
( |% } j9 p* q3 c
Summary
4 X0 r6 D; o3 a" L1 _On August 30, 2020 10:04 GMT, CenturyLink identified an issue to be affecting users across multiple markets. The IP Network Operations Center (NOC) was engaged, and initial research identified that an offending flowspec announcement prevented Border Gateway Protocol (BGP) from establishing across multiple elements throughout the CenturyLink Network. The IP NOC deployed a global configuration change to block the offending flowspec announcement, which allowed BGP to begin to correctly establish. As the change propagated through the network, the IP NOC observed all associated service affecting alarms clearing and services returning to a stable state.& _: b) e2 _$ f* s. q, p: q
3 G8 J" H5 _9 d$ g2 o 9 V, S8 z0 x- |' X8 c, m. d
1 l( s# x, ?& h% R% O, \& wService Impact Times:0 Z) T% ~3 h! o( I: d( o9 A. ~3 `0 K
———————————————
+ O4 }: e" f* J6 i& g) ?* }/ y* e5 P: \% i6 ]) j! l1 j1 i' _
August 30th, 6:04am EST - 11:30am EST
7 v [6 e, n* i+ M& e- i# p
6 l: c3 g) z( e C- l1 \1 E) W- k7 e3 w; y Z
Additional Information:9 w. j3 ^' x" D; [
———————————————$ w+ q7 @* H; ^1 [) I
% s' f4 i+ l) `You can find further information regarding the global impact of yesterday's event at the following links.
- t3 U: n/ q1 V" a3 {3 }: m, r1 ~# R
https://www.theverge.com/2020/8/ ... hulu-feedly-discord% z1 J1 B0 l/ u7 n2 x
https://9to5mac.com/2020/08/30/centurylink-outage/ e4 }! L ]# B! z3 F5 ^$ D
https://www.bleepingcomputer.com ... steam-discord-more/
9 h+ b0 X# g: f
 |
|
发表于 2020-8-30 22:43:06
发表于 2020-8-31 00:17:50
发表于 2020-8-31 09:01:38
