xfs_repair an LVM on centos or other distros

For the first time I have started using LVM on my home server and things have been alright for an year. I am able to resize, replace disks , expand size of my mount points or move mount to a new disk without any trouble at all.

While its not relevant I am using CentOS 7.

I have had to force reboot the server a couple of times over the year and woefully XFS complained of corruption 24 hours ago and it was time to figure this out. I kept dropping into emergency mode upon reboot. There is also another reason for this problem. I have bad memory, I bought some refurbished FB-DIMM DDR2 with ECC and ironically not all of it is good.

Anyway TL;DR

Filesystem : XFS

Boot with rescue CD/USB whatever and don’t mount anything

Are you using LVM? yes: Try with lvscan to confirm. The output will show you your lvm volumes but they might be INACTIVE which is ok, we will fix it later.

Then if I run xfs_repair /dev/sdx1 on the failed disk it complains of bad superblock and it also fails to find a good secondary superblock. Something like `Sorry, could not find valid secondary superblock Exiting now.` Ofcourse I am using the wrong method which I am familiar with because of the old non-LVM way of doing things to repair a drive because the drive was the volume itself.

Wait! I need to repair from the Virtual Volume created by LVM.

start with

vgscan -v --mknodes

This will find and create /dev/xxx/ nodes for the lvm. But it is not mounted so it’s good.

Now activate the volumes, it still does not actually mount them, so still good.

vgchange -a y

Now see what volumes are active with another lvscan

I have /dev/centos/root which is what I need to fix. It is an XFS volume so this time I do

xfs_repair /dev/centos/root

bunch of output follows… repair is done. Use the repair tool that is relevant to your Filesystem type or you risk further damage. Repeat the repair for other volumes as needed.

Lets finally mount and test

mkdir /mnt/test && mount /dev/centos/root /mnt/test

Looks good?

Unmount and reboot without rescue CD. You should try to verify the LVM once again in rescue mode for all the volumes since multiple drives can span a volume.

Update 2018: Tested on CentOS and Ubuntu, including failed drives from another system attached as external drive for recovery.