由 2017/02/01 GitLab 资料库事件来检讨备份 / 回复机制

事由：提供 Git 版控的付费 Hosting 网站 GitLab.com，发生资料库异常而下线，恢复上线后，有六小时的资料完全救不回来。

参考资料：官方说明

以下是针对备份／回复的官方说明节录

Problems Encountered

LVM snapshots are by default only taken once every 24 hours. Team-member-1 happened to run one manually about 6 hours prior to the outage because he was working in load balancing for the database.

Regular backups seem to also only be taken once per 24 hours, though team-member-1 has not yet been able to figure out where they are stored. According to team-member-2 these don’t appear to be working, producing files only a few bytes in size.

Team-member-3: It looks like pg_dump may be failing because PostgreSQL 9.2 binaries are being run instead of 9.6 binaries. This happens because omnibus only uses Pg 9.6 if data/PG_VERSION is set to 9.6, but on workers this file does not exist. As a result it defaults to 9.2, failing silently. No SQL dumps were made as a result. Fog gem may have cleaned out older backups.

Disk snapshots in Azure are enabled for the NFS server, but not for the DB servers.

The synchronisation process removes webhooks once it has synchronised data to staging. Unless we can pull these from a regular backup from the past 24 hours they will be lost

The replication procedure is super fragile, prone to error, relies on a handful of random shell scripts, and is badly documented

Our backups to S3 apparently don’t work either: the bucket is empty

So in other words, out of 5 backup/replication techniques deployed none are working reliably or set up in the first place. We ended up restoring a 6 hours old backup.

像 GitLab 这种等级的网站
用了 5 种备份机制
发生问题时
备份也是派不上用场
掉了 6 小时的资料

回头看看自己公司的备份机制
是否该做些调整
或者
如我所建议的
平时不演练，出状况就搞笑