AWS的验尸报告~

V2EX = way to explore

V2EX 是一个关于分享和探索的地方

现在注册

已注册用户请登录

这是一个创建于 4955 天前的主题，其中的信息可能已经有所发展或是发生改变。

http://aws.amazon.com/message/65648/

AWS这次严重的故障，影响到了Quora和Foursquare等多家客户，不知道有看了官方验尸报告的吗？
我花了2个小时看完，整个事故的过程分析起来惊心动魄啊！

AWS

验尸

我花

6 条回复 • 1970-01-01 08:00:00 +08:00

MarkFull

2011-05-02 06:11:40 +08:00

这里有个简版。

http://internet.solidot.org/internet/11/04/30/0915200.shtml

ry_wang

2011-05-02 11:24:24 +08:00

发现异常为啥不能及时回滚呢?其实我好奇这个

xatest

2011-05-02 11:48:30 +08:00

@ry_wang 他们错误的回滚策略造成了雪崩，引起更大范围的错误~

ry_wang

2011-05-02 12:10:05 +08:00

@xatest 哦?有详细点的说明么?官方文档看不懂呀

xatest

2011-05-02 14:52:17 +08:00

@ry_wang 这还不够详细？5700字了~

Sidney

2011-05-02 22:02:25 +08:00

看着有种心惊肉跳的感觉

两点感受
1. 人为的操作容易出现错误。之前碰到过hosting的人员在配置Firewall的时候误操作，Firewall Hang住，导致网站无法访问。
The traffic shift was executed incorrectly and rather than routing the traffic to the other router on the primary network, the traffic was routed onto the lower capacity redundant EBS network.

2. 对回滚操作的后果估计不足，带来连锁雪崩效应。除非对系统各个模块和它们之间的交互有足够的理解，说不定就会踩中地雷。
Incorrect traffic shift was rolled back and network connectivity was restored. This quickly led to a “re-mirroring storm,” where a large number of volumes were effectively “stuck” while the nodes searched the cluster for the storage space it needed for its new replica.