Page tree
Skip to end of metadata
Go to start of metadata

Warning

Be extremely careful when increasing volume sizes. It's possible that you may fatally corrupt your filesystem if you're not extremely cautious.

ALWAYS PERFORM A BACKUP BEFORE ATTEMPTING TO INCREASE VOLUME SIZES

YOU MUST HAVE BOTH NODES ONLINE, AND ALL DISKS IN 'UPTODATE' STATE BEFORE CONTINUING

 

With that said, if you're careful and patient you can do an on-line hot upgrade of your filesystems without an outage.

Stage 1: Increase block device

First, verify that you have free space in your LVM ON BOTH MACHINES

[root@freepbx-a ~]# vgdisplay
  --- Volume group ---
  VG Name               vg_ha
  System ID
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  7
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                6
  Open LV               6
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               465.14 GiB
  PE Size               4.00 MiB
  Total PE              119077
  Alloc PE / Size       36576 / 142.88 GiB
  Free  PE / Size       82501 / 322.27 GiB
  VG UUID               WemjWb-sskA-31xv-H5bu-CKj0-UktP-1T97WZ

The second line from the bottom - 'Free PE' - says that this volume group has 322G of unused space that we can allocate to any volume.  The other machine says a similar amount, so I'm certain that I can add space on both machines. 

Run the exact same command on both machines. In this case, we're extending the 'asterisk' filesystem (or, to start with, the bottom level block device).

lvextend -L250G /dev/vg_ha/drbd_asterisk

This sets the volume size, rather than abstractly increasing it. As the volume sizes may have been created slightly differently on install (depending on disk sizes), you must explicitly set a size.  

After running this command, verify on both nodes that both volumes are now exactly the same size.

[root@freepbx-b ~]# lvdisplay /dev/vg_ha/drbd_asterisk
  --- Logical volume ---
  LV Path                /dev/vg_ha/drbd_asterisk
  LV Name                drbd_asterisk
  VG Name                vg_ha
  LV UUID                sBnghY-YHLh-03De-jard-CVgy-fNYV-TRt23Y
  LV Write Access        read/write
  LV Creation host, time freepbx-a, 2014-01-09 23:02:57 -0600
  LV Status              available
  # open                 2
  LV Size                250.00 GiB
  Current LE             64000
  Segments               2
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:2

The 'Current LE' (Current Logical Extents) is the number that should match. If they don't, you must resolve this before continuing. Attempting to continue if they don't match is guaranteed to corrupt your filesystem, and will also probably crash the machine. In our case, they matched (and they will, if you don't try to use the 'add' command to lvextend, and use the 'set' option as above).

[root@freepbx-a ~]# lvdisplay /dev/vg_ha/drbd_asterisk
  --- Logical volume ---
  LV Path                /dev/vg_ha/drbd_asterisk
  LV Name                drbd_asterisk
  VG Name                vg_ha
  LV UUID                4CbvFI-pit7-WTMi-deUI-Xhvh-eXg5-PhqBZb
  LV Write Access        read/write
  LV Creation host, time freepbx-a, 2014-01-02 18:08:16 -0500
  LV Status              available
  # open                 2
  LV Size                250.00 GiB
  Current LE             64000
  Segments               2
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:2

Both nodes have 64000 Logical Extends, so we can proceed.

Stage 2: Tell DRBD to update its internal counters and resync.

You only need to perform this command on one node.

drbdadm resize asterisk

Replace 'asterisk' with whatever volume you're increasing the size of, obviously. This will update DRBD on both nodes, you do not need to do anything on the other one. Checking via the Web UI will show that the volume is now resyncing. This will perform a complete resync, and attempt to read and write to every block of available space to ensure that the previous stages have been completed. There is no way to abort this, nor do you want to. 

Stage 3: Wait

No. Really. Wait for the resync to complete. Whilst you can, if you REALLY MUST, perform the resize now, don't. The server is doing a complete resync now to verify that the next stage isn't going to have problems. Let it finish.

Stage 4: Increase filesystem size

This is the final and - hopefully - anti-climatic part. After all the previous replcation has completed, resize the filesystem on the active node. You need the block device - in our case, it's /dev/drbd1.

[root@freepbx-a ~]# mount
/dev/mapper/vg_ha-slash on / type ext4 (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
tmpfs on /dev/shm type tmpfs (rw)
/dev/md0 on /boot type ext4 (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
none on /sys/kernel/config type configfs (rw)
/dev/drbd3 on /drbd/httpd type ext4 (rw)
/dev/drbd1 on /drbd/asterisk type ext4 (rw)
/dev/drbd4 on /drbd/spare type ext4 (rw)
/dev/drbd2 on /drbd/mysql type ext4 (rw)
[root@freepbx-a ~]#

You can see that /dev/drbd1 is mounted on /drbd/asterisk, so that is the volume we're going to resize.

[root@freepbx-a ~]# resize2fs /dev/drbd1
resize2fs 1.41.12 (17-May-2010)
Filesystem at /dev/drbd1 is mounted on /drbd/asterisk; on-line resizing required
old desc_blocks = 3, new_desc_blocks = 16
Performing an on-line resize of /dev/drbd1 to 65533991 (4k) blocks.
THIS TAKES TIME! DO NOT ABORT!

 

Stage 5: Wait, again

While the filesystem is growing, the machine is fully accessible, and everything will be fine. You can watch the filesystem grow in a different window, by running the 'df' or 'df -h' command.

[root@freepbx-a ~]# df
Filesystem              1K-blocks     Used Available Use% Mounted on
/dev/mapper/vg_ha-slash  48380792  2177960  43745232   5% /
tmpfs                     1993480    58712   1934768   3% /dev/shm
/dev/md0                   495780    32390    437794   7% /boot
/dev/drbd3               20641752   237416  19355796   2% /drbd/httpd
/dev/drbd1              116758940 29531216  81297720  27% /drbd/asterisk
/dev/drbd4               15481324   169456  14525464   2% /drbd/spare
/dev/drbd2               20641752   869796  18723416   5% /drbd/mysql
[root@freepbx-a ~]#
 ... I paused here for a minute ...
[root@freepbx-a ~]# df
Filesystem              1K-blocks     Used Available Use% Mounted on
/dev/mapper/vg_ha-slash  48380792  2178012  43745180   5% /
tmpfs                     1993480    58712   1934768   3% /dev/shm
/dev/md0                   495780    32390    437794   7% /boot
/dev/drbd3               20641752   237464  19355748   2% /drbd/httpd
/dev/drbd1              151722156 29557460 114459100  21% /drbd/asterisk
/dev/drbd4               15481324   169456  14525464   2% /drbd/spare
/dev/drbd2               20641752   869972  18723240   5% /drbd/mysql
[root@freepbx-a ~]#

Note that the '1K-blocks' has increased, which has lead to an increase in 'Available' and the respective shrinking of 'Use %'.

You can also run 'df -h' which will give you a human readable output:

[root@freepbx-a ~]# df -h
Filesystem               Size  Used Avail Use% Mounted on
/dev/mapper/vg_ha-slash   47G  2.1G   42G   5% /
tmpfs                    2.0G   58M  1.9G   3% /dev/shm
/dev/md0                 485M   32M  428M   7% /boot
/dev/drbd3                20G  232M   19G   2% /drbd/httpd
/dev/drbd1               165G   29G  128G  19% /drbd/asterisk
/dev/drbd4                15G  166M   14G   2% /drbd/spare
/dev/drbd2                20G  850M   18G   5% /drbd/mysql
[root@freepbx-a ~]#

Stage 6: You're almost done!

When the resize has finished, it should exit without errors.

[root@freepbx-a ~]# resize2fs /dev/drbd1
resize2fs 1.41.12 (17-May-2010)
Filesystem at /dev/drbd1 is mounted on /drbd/asterisk; on-line resizing required
old desc_blocks = 3, new_desc_blocks = 16
Performing an on-line resize of /dev/drbd1 to 65533991 (4k) blocks.
The filesystem on /dev/drbd1 is now 65533991 blocks long.
[root@freepbx-a ~]# df -h
Filesystem               Size  Used Avail Use% Mounted on
/dev/mapper/vg_ha-slash   47G  2.1G   42G   5% /
tmpfs                    2.0G   58M  1.9G   3% /dev/shm
/dev/md0                 485M   32M  428M   7% /boot
/dev/drbd3                20G  232M   19G   2% /drbd/httpd
/dev/drbd1               247G   29G  206G  13% /drbd/asterisk
/dev/drbd4                15G  166M   14G   2% /drbd/spare
/dev/drbd2                20G  851M   18G   5% /drbd/mysql
[root@freepbx-a ~]#

Your final thing to do is to update the database with the size of the volume so that when you replace a node, the new volume is created at the correct size.

amportal a m
select * from freepbxha where `key`='SIZE_asterisk';

This will show you the OLD size.

+---------------+-------+
| key           | value |
+---------------+-------+
| SIZE_asterisk | 40    |
+---------------+-------+
1 row in set (0.00 sec)

You now update it with the new size. USE THE SIZE YOU USED ON THE LVEXTEND COMMAND, NOT THE SIZE REPORTED BY 'df.'

mysql> update freepbxha set `value`='250' where `key`='SIZE_asterisk';
Query OK, 1 row affected (0.41 sec)
Rows matched: 1  Changed: 1  Warnings: 0
mysql>

 

 

  • No labels