2015-06-05

Resizing Online Disks in Linux with LVM and No Reboots

When we set this up, we only had a 16GB primary disk... Plenty of space. Until you start to write lots of logs and data... and then it fills up quick. So let's talk about how to resize an LVM based partition on a live server without reboots... Reboots are for Windows! This system is a CentOS 6.x machine running in VMWare 5.x that's currently got a 16GiB VMDK based drive. Let's see what we've got to work with:
$ df -h
Filesystem                                         Size  Used Avail Use% Mounted on
/dev/mapper/myfulldisk--vg-root                     12G   11G     0 100% /
none                                               4.0K     0  4.0K   0% /sys/fs/cgroup
udev                                               7.4G  4.0K  7.4G   1% /dev
tmpfs                                              1.5G  572K  1.5G   1% /run
none                                               5.0M     0  5.0M   0% /run/lock
none                                               7.4G     0  7.4G   0% /run/shm
none                                               100M     0  100M   0% /run/user
/dev/sda1                                          236M   37M  187M  17% /boot
/dev/sdc1                                          246G   44G  190G  19% /data

Hmm... time to get on this then. Now, luckily we're running in VMWare. A quick edit to our VM to enlarge the VMDK (not covered in this how-to) will fix this... First, what device are we talking about?

$ dmesg|grep sd
[    1.562363] sd 2:0:0:0: Attached scsi generic sg1 type 0
[    1.562384] sd 2:0:0:0: [sda] 33554432 512-byte logical blocks: (17.1 GB/16.0 GiB)
[    1.562425] sd 2:0:0:0: [sda] Write Protect is off
[    1.562426] sd 2:0:0:0: [sda] Mode Sense: 61 00 00 00
[    1.562460] sd 2:0:0:0: [sda] Cache data unavailable
[    1.562461] sd 2:0:0:0: [sda] Assuming drive cache: write through
[    1.563331] sd 2:0:0:0: [sda] Cache data unavailable
[    1.563451] sd 2:0:1:0: Attached scsi generic sg2 type 0
[    1.563452] sd 2:0:1:0: [sdb] 8388608 512-byte logical blocks: (4.29 GB/4.00 GiB)
[    1.563479] sd 2:0:1:0: [sdb] Write Protect is off
[    1.563481] sd 2:0:1:0: [sdb] Mode Sense: 61 00 00 00
[    1.563507] sd 2:0:1:0: [sdb] Cache data unavailable
[    1.563508] sd 2:0:1:0: [sdb] Assuming drive cache: write through
[    1.563755] sd 2:0:2:0: Attached scsi generic sg3 type 0
[    1.563881] sd 2:0:2:0: [sdc] 524288000 512-byte logical blocks: (268 GB/250 GiB)
[    1.563942] sd 2:0:2:0: [sdc] Write Protect is off
[    1.563944] sd 2:0:2:0: [sdc] Mode Sense: 61 00 00 00
[    1.564008] sd 2:0:2:0: [sdc] Cache data unavailable
[    1.564010] sd 2:0:2:0: [sdc] Assuming drive cache: write through
[    1.564282] sd 2:0:2:0: [sdc] Cache data unavailable
[    1.564283] sd 2:0:2:0: [sdc] Assuming drive cache: write through
[    1.564360] sd 2:0:1:0: [sdb] Cache data unavailable
[    1.564362] sd 2:0:1:0: [sdb] Assuming drive cache: write through
[    1.564989] sd 2:0:0:0: [sda] Assuming drive cache: write through
[    1.571010]  sdb: sdb1
[    1.571426] sd 2:0:1:0: [sdb] Cache data unavailable
[    1.571514] sd 2:0:1:0: [sdb] Assuming drive cache: write through
[    1.571626] sd 2:0:1:0: [sdb] Attached SCSI disk
[    1.574181]  sda: sda1 sda2 < sda5 >
[    1.574797] sd 2:0:0:0: [sda] Cache data unavailable
[    1.574888] sd 2:0:0:0: [sda] Assuming drive cache: write through
[    1.575003] sd 2:0:0:0: [sda] Attached SCSI disk
[    1.579250]  sdc: sdc1
[    1.579805] sd 2:0:2:0: [sdc] Cache data unavailable
[    1.579944] sd 2:0:2:0: [sdc] Assuming drive cache: write through
[    1.580141] sd 2:0:2:0: [sdc] Attached SCSI disk
[    6.922330] Adding 4193276k swap on /dev/sdb1.  Priority:-1 extents:1 across:4193276k FS
[    7.137134] EXT4-fs (sda1): mounting ext2 file system using the ext4 subsystem
[    7.142419] EXT4-fs (sda1): mounted filesystem without journal. Opts: (null)
[    7.218150] EXT4-fs (sdc1): mounted filesystem with ordered data mode. Opts: (null)
[    7.384566] Installing knfsd (copyright (C) 1996 okir@monad.swb.de).

The first one is the 16GB drive in question. Take the number on that line and use it in the next step:

$ echo 1 > /sys/class/scsi_device/2\:0\:0\:0/device/rescan
$ dmesg |tail
[1918441.322362] sd 2:0:0:0: [sda] 209715200 512-byte logical blocks: (107 GB/100 GiB)
[1918441.322596] sd 2:0:0:0: [sda] Cache data unavailable
[1918441.330685] sd 2:0:0:0: [sda] Assuming drive cache: write through
[1918441.489622] sda: detected capacity change from 17179869184 to 1073741824

So, that's good, it sees our increased size. Now, lets enlarge that Volume Group. First we get info about the volume group.

$ vgdisplay
  --- Volume group ---
  VG Name               myfulldisk-vg
  System ID             
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  3
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                2
  Open LV               2
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               15.76 GiB
  PE Size               4.00 MiB
  Total PE              4034
  Alloc PE / Size       4028 / 15.73 GiB
  Free  PE / Size       6 / 24.00 MiB
  VG UUID               dv3URd-EVvz-oTwY-WiDW-RPt1-4rbD-FnPxxM

That '6' is only 24MiB. It doesn't see our new space yet. In order to get it to, we need to make a new partition of the right type, then add it to the volume group. We'll then end up with more Free PE's. Here we go:
$ fdisk /dev/sda
Command (m for help): p

Disk /dev/sda: 107.4 GB, 107374182400 bytes
255 heads, 63 sectors/track, 13054 cylinders, total 209715200 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000ade37

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *        2048      499711      248832   83  Linux
/dev/sda2          501758    33552383    16525313    5  Extended
/dev/sda5          501760    33552383    16525312   8e  Linux LVM

Command (m for help): n
Partition type:
   p   primary (1 primary, 1 extended, 2 free)
   l   logical (numbered from 5)
Select (default p): p
Partition number (1-4, default 3): 3
First sector (499712-209715199, default 499712): 33552384
Last sector, +sectors or +size{K,M,G} (33552384-209715199, default 209715199): 
Using default value 209715199

Command (m for help): t
Partition number (1-5): 3
Hex code (type L to list codes): 8e
Changed system type of partition 3 to 8e (Linux LVM)

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.

WARNING: Re-reading the partition table failed with error 16: Device or resource busy.
The kernel still uses the old table. The new table will be used at
the next reboot or after you run partprobe(8) or kpartx(8)
Syncing disks.

At this point you could reboot... but we're not going to. Even though this is our root drive which makes this a little trickier, it's nothing we can't fix:
$ partprobe /dev/sda

Hopefully, partprobe has found your new partition for you and enlightened the kernel with it's wisdom (or at least fresh load of zeros). Now we need to make it an available volume to use for expanding the disk. This consists of making it a 'Physical volume', and then adding that physical volume to the Volume Group containing the disk we want to expand.

$ pvcreate /dev/sda3
  Physical volume "/dev/sda3" successfully created
$ pvdisplay
  --- Physical volume ---
  PV Name               /dev/sda5
  VG Name               myfulldisk-vg
  PV Size               15.76 GiB / not usable 2.00 MiB
  Allocatable           yes 
  PE Size               4.00 MiB
  Total PE              4034
  Free PE               6
  Allocated PE          4028
  PV UUID               a3mhvZ-ogyk-ao4y-2JSM-KVfL-i9no-q0LAUk
   
  "/dev/sda3" is a new physical volume of "84.00 GiB"
  --- NEW Physical volume ---
  PV Name               /dev/sda3
  VG Name               
  PV Size               84.00 GiB
  Allocatable           NO
  PE Size               0   
  Total PE              0
  Free PE               0
  Allocated PE          0
  PV UUID               IpyjOU-1GDy-bTLL-U9kE-iSGP-BYg1-a25LIm
   
$ vgextend /dev/myfulldisk-vg /dev/sda3
  Volume group "myfulldisk-vg" successfully extended
$ vgdisplay
  --- Volume group ---
  VG Name               myfulldisk-vg
  System ID             
  Format                lvm2
  Metadata Areas        2
  Metadata Sequence No  4
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                2
  Open LV               2
  Max PV                0
  Cur PV                2
  Act PV                2
  VG Size               99.76 GiB
  PE Size               4.00 MiB
  Total PE              25538
  Alloc PE / Size       4028 / 15.73 GiB
  Free  PE / Size       21510 / 84.02 GiB
  VG UUID               dv3URd-EVvz-oTwY-WiDW-RPt1-4rbD-FnPxxM
Awesome, we now have 21510 free PE's that we can use... That's, apparently, 84.02GB in this case. Next up, we'll need to know what portion of the VG we need to extend. Looking back up at a 'df' output, and knowing our system, it says "root" in there. doing a quick ls of /dev/myfulldisk-vg/ shows that there's only really a choice between root" and "swap". So, knowing it's root we move on with:
$ lvextend -L95G /dev/myfulldisk-vg/root
  Extending logical volume root to 95.00 GiB
  Logical volume root successfully resized
$ df -h
Filesystem                                         Size  Used Avail Use% Mounted on
/dev/mapper/myfulldisk--vg-root                     12G   11G  114M  99% /
none                                               4.0K     0  4.0K   0% /sys/fs/cgroup
udev                                               7.4G  4.0K  7.4G   1% /dev
tmpfs                                              1.5G  576K  1.5G   1% /run
none                                               5.0M     0  5.0M   0% /run/lock
none                                               7.4G     0  7.4G   0% /run/shm
none                                               100M     0  100M   0% /run/user
/dev/sda1                                          236M   37M  187M  17% /boot
/dev/sdc1                                          246G   44G  190G  19% /data
Okay, the VG might be bigger, but no one else knows that. Because the filesystem ON the VG is still the same size, luckily there's a command for that too!
$ resize2fs /dev/myfulldisk-vg/root
resize2fs 1.42.9 (4-Feb-2014)
Filesystem at /dev/myfulldisk-vg/root is mounted on /; on-line resizing required
old_desc_blocks = 1, new_desc_blocks = 6
The filesystem on /dev/myfulldisk-vg/root is now 24903680 blocks long.

$ df -h
Filesystem                                         Size  Used Avail Use% Mounted on
/dev/mapper/myfulldisk--vg-root                     94G   11G   79G  12% /
none                                               4.0K     0  4.0K   0% /sys/fs/cgroup
udev                                               7.4G  4.0K  7.4G   1% /dev
tmpfs                                              1.5G  576K  1.5G   1% /run
none                                               5.0M     0  5.0M   0% /run/lock
none                                               7.4G     0  7.4G   0% /run/shm
none                                               100M     0  100M   0% /run/user
/dev/sda1                                          236M   37M  187M  17% /boot
/dev/sdc1                                          246G   44G  190G  19% /data


Ha, there we go! 79GB available, enjoy!

No comments:

Post a Comment