Expect the unexpected... even if you are Google. Sometimes as software developers, regardless of how much you've done to mitigate risk, code goes out the door that has unexpected results.
Over the weekend Google's Gmail team "...released a storage software update that introduced the unexpected bug, which caused 0.02% of Gmail users to temporarily lose access to their email." The trouble is that even with all of the redundancy built into their data centers, the bug deleted all online copies of the affected users data, which meant that the team was forced to restore from backup tapes. This is a time consuming process, especially when dealing with 0.02% of mail held by Gmail.
While those of us who have had to deal with these hair pulling ordeals can understand and sympathize with the development team, most users will not. Rest assured that the backup and recovery strategy has accomplished what it was intended to do. If anything, this is a valuable lesson in backing up data. What's your strategy? If your laptop took a tumble would you be able to recover?
