There comes a time when your FreeBSD root partition (or slice) is just too small to be of any use. You've already moved /var, /usr, and /tmp to separate disks, you've tried cleaning up some logs, clearing cache, and even run pkg clean. There just isn't anything else you can delete. The problem is it was initially created too small when FreeBSD installed.
Updated: 2020-12-05
Filesystem is Full
It's a day just like any other. You run freebsd-update upgrade -r to get on the latest FreeBSD release. You follow the onscreen instructions, reboot, and continue with the upgrade. It all seems to be going well... until you run pkg upgrade -fy and see something like:
pkg: Not enough space in /var/cache/pkg, needed 43 MiB available -148 MiBOh no! how did that happen? You thought you had plenty of free space!
As you begin to accept the reality of the situation you find yourself in, you become ever so grateful for that 10% space reservation FreeBSD keeps for the root user by default.
This is just one of the possible circumstances that lead to the conclusion that it's time to resize the root partition or slice. Traditionally you would have had to backup and start over. Many guides would suggest you use a dump and restore method as an alternative. All that is too complicated, too much work for us lazy system administrators, and more importantly introduces a lot of downtime.
Fortunately in FreeBSD 9.0 and above there are handy little tools called gpart. and growfs. In FreeBSD 11 and above (in most circumstances) you can resize a running system without rebooting.
Process Overview
The goal is to free a contiguous block of space directly following the root partition (or slice) of your boot volume. This will allow it to expand into whatever additional storage capacity you add to your system. The amount of effort involved in getting to that state depends entirely on your hardware, and how FreeBSD initially partitioned the volume when it was installed. The final step is to expand the filesystem to match the size of the root partition or slice.
This guide can be followed from start to end or you can use the links below to skip to a specific task of interest.
- Increase Storage: Adding additional storage capacity to your system.
- Verify Changes: Ensure FreeBSD is recognizing the storage changes.
- Initialize New Volumes: Make the new storage usable for FreeBSD.
- UFS Dump and Restore: How to use built-in tools to move around partitions or slices.
- Remove Old Partitions or Slices: Cleanup old partitions and slices after they are moved.
- Move /tmp and Swap: Recreate and delete the existing /tmp and Swap.
- Re-size the Root Partition/Slice: Grow the partition or slice.
- Re-size the Root Filesystem: Grow the filesystem.
Requirements
This guide assumes you are using a single non-RAID IDE/SATA or SCSI volumes. Device names may be different and extra drivers may need to be loaded for other types of volumes. Some volume types may not support online resize due to legacy hardware.
- This guide is for UFS only. ZFS on Root is an entirely different process.
- FreeBSD 9.0 or later (if you have a lesser version, you might be able to use a live CD of a later version)
- FreeBSD 9.x or later boot-only media. (ISO or flash image)
- FreeBSD 11.x or later for online resizing.
- Extra or unused space on the drive where the root partition is stored.
- A backup of your most precious files stored on this system
Note: The boot media you select should be equal to or later than the version of FreeBSD you are running.
Do a Backup
Step 0 is to create a backup of every important item stored on this system. Don't skip this step! The extra time and effort you spend creating a backup will be worth it when something goes wrong. If this system is a virtual machine take a snapshot. Just make sure you have a backup of your stuff before you continue.
Expand or Add Volume
There are generally two ways to add storage capacity, and that depends on your hardware configuration and needs. You will either grow an existing volume or add an additional volume. Some hardware supports the ability to add or expand volumes while the system is running. That's usually the case with virtualization. While I have not come across it yet, I'm sure there are physical machines that would let you do the same. If you have not already done so go ahead and add the additional capacity as required.
If you are using a virtual machine this could be done on a running VM without having to shutdown or reboot. Consult your hypervisor's documentation on how to perform that action (if possible).
VMWare's ESXi Server 5.0 and above lets you increase the size of a virtual disk or add new disks while the VM is running!
Check Capacity
Use gpart show to see if FreeBSD recognized the additional free space. If you didn't reboot there is a good chance the operating system is not yet aware of the changes.
# gpart show
=> 40 20971440 da0 GPT (10G)
40 1024 1 freebsd-boot (512K)
1064 18874368 2 freebsd-ufs (9.0G)
18875432 2096048 3 freebsd-swap (1.0G)If you run into this situation use the camcontrol command to reprobe the device. Substitute da0 with your boot volume's device identifier.
# camcontrol reprobe da0
# gpart show
=> 40 31457200 da0 GPT (15G)
40 1024 1 freebsd-boot (512K)
1064 18874368 2 freebsd-ufs (9.0G)
18875432 2096048 3 freebsd-swap (1.0G)
20971480 10485760 - free - (5.0G)The CAM subsystem will be updated to reflect to change in capacity.
Initialize New Volumes
New volumes will have to be partitioned first. In this example, the new disk has been identified as /dev/da1. You can look at /var/run/dmesg.boot for information about what devices have been detected.
Issue the following command to initialize the disk as a GPT type disk.
gpart create -s gpt da1The result should be something like "da1 created". There is another command camcontrol rescan all that can be used when a new device was added but fails to show up.
# gpart create -s gpt da1
gpart: arg0 'da1': Invalid argument
# camcontrol rescan all
Re-scan of bus 0 was successful
Re-scan of bus 1 was successful
Re-scan of bus 2 was successful
# gpart create -s gpt da1
da1 createdReorganize the Volume for Contiguous Free Space
Before we can expand the root partition (or slice) on our FreeBSD system we need some room to grow. Things may need to be moved around depending on the layout. The goal is to have a contiguous block of free space directly after the root partition or slice.
What you do next varies by how the extra capacity was added and how the boot volume was initially partitioned or sliced. Older versions of the FreeBSD install process (prior to bsdinstall), would by default break up /usr, /var, and /tmp into separate slices on a single partition. Lets refer to that as "multiple mounts points". Modern FreeBSD installations that default to using the GPT partition scheme will have a root partition that takes up the majority of the volume, using a single mount point.
If your volume has a small freebsd-boot partition it can be ignored. The boot partition is always at the beginning of the volume and shouldn't need moving. In fact you can't move it without rendering your system un-bootable. Swap partitions/slices are easily moved and thus pose no significant issue.
Layouts with "multiple mounts points" will require data migration in order to get the contiguous block of free space that is needed. Layouts with a single mount point can skip the data and /tmp migration tasks. In all cases swap will probably need to be moved.
Example of a volume only needing swap to be moved:
# gpart show
=> 40 31457200 da0 GPT (15G)
40 1024 1 freebsd-boot (512K)
1064 18874368 2 freebsd-ufs (9.0G)
18875432 2096048 3 freebsd-swap (1.0G)
20971480 10485760 - free - (5.0G)In layouts like the one below where a new volume was added, data will need migration in addition to moving both /tmp (the last 512M slice) and swap:
# gpart show ada0
=> 63 71119692 ada0 MBR (33G)
63 71119755 1 freebsd [active] (33G)
# gpart show ada0s1
=> 0 71119692 ada0s1 BSD (33G)
0 1048576 1 freebsd-ufs (512M)
1048576 2029408 2 freebsd-swap (990M)
3077984 3110912 4 freebsd-ufs (1.5G)
6188896 1048576 5 freebsd-ufs (512M)
7237472 63882220 6 freebsd-ufs (30G)
# gpart show da1
=> 40 41942960 da1 GPT (20G)
40 41942960 - free - (20G)In this layout, the boot volume was expanded. Extra care and thought will need to be taken in order to properly shift things around. The partition on the volume would need to be expanded before you can add slices to the partition.
# gpart show ada0
=> 63 71119692 ada0 MBR (33G)
63 71119755 1 freebsd [active] (33G)
71119818 251658162 - free - (120G)
# gpart show ada0s1
=> 0 71119692 ada0s1 BSD (33G)
0 1048576 1 freebsd-ufs (512M)
1048576 2029408 2 freebsd-swap (990M)
3077984 3110912 4 freebsd-ufs (1.5G)
6188896 1048576 5 freebsd-ufs (512M)
7237472 63882220 6 freebsd-ufs (30G)UFS Dump and Restore
It's time to do some destructive re-arranging. You did backup, right?
Moving your UFS data is done using the dump and restore tools.
An example of how to move a single partition or slice to a new device. Replace device names, options, and paths accordingly.
gpart add -t freebsd-ufs -s 80G da1
newfs -U /dev/da1p1
mkdir /usr.new
mount /dev/da1p1 /usr.new
cd /usr.new
dump -0Lauf - /dev/ada0s1f | restore -rf -
cd /
umount /usr.new
umount /usr
rm /usr.new
edit /etc/fstab
mount /dev/da1p1 /usrThe same tooling can be used to move data within the same volume to a new slice after the last. Replace device names, options, and paths accordingly.
gpart resize -i 1 ada0
gpart add -t freebsd-ufs -s 2G ada0s1
newfs -U /dev/ada0s1g
mkdir /var.new
mount /dev/ada0s1g /var.new
cd /var.new
dump -0Lauf - /dev/ada0s1d | restore -rf -
cd /
umount /var.new
umount /var
rm /var.new
edit /etc/fstab
mount /dev/ada0s1g /varThe above are very specific examples of how to move your data. It's critical that you verify the device names before you perform any actions. Adjust the commands according to your setup and pay attention to the disk format in use. An MBR can only hold 4 primary partitions, which is why FreeBSD uses slices instead. UFS dump and restore should work fine on a live filesystem (hence the -L flag), but in some circumstances you may need to stop any services from accessing the filesystem. In extreme cases a reboot into single user mode show do the trick.
Adjust the device names in /etc/fstab after your dump/restore is done before rebooting. This process is illustrated here.
Remove Old Partitions or Slices
Again, "Did you make a backup like I asked? We are going to delete the old slices and/or partitions from the volume.
Take for example the following partition:
gpart show ada0s1
=> 0 71119692 ada0s1 BSD (33G)
0 1048576 1 freebsd-ufs (512M)
1048576 2029408 2 freebsd-swap (990M)
3077984 3110912 4 freebsd-ufs (1.5G)
6188896 1048576 5 freebsd-ufs (512M)
7237472 63882220 6 freebsd-ufs (30G)Our OLD /var and /usr slices used to be on 4 and 6. Currently /tmp is on 5 and our swap space is on 2. That means we can delete slices 4 and 6 because they are no longer in use, that data has been moved to another volume.
Before you do anything, confirm it:
df -h
Filesystem Size Used Avail Capacity Mounted on
/dev/ada0s1a 512M 256M 256M 50% /
devfs 1.0k 1.0k 0B 100% /dev
/dev/ada0s1e 512M 2.1M 510M 0% /tmp
/dev/da0s1d 193G 31G 147G 17% /usr
/dev/ada1s1d 116G 15G 91G 14% /varNotice that in this example slices 4(d) and 6(f) on the block device "ada0s1" are not mounted and therefore not in use. Make sure you check /etc/fstab too just to be sure.
Go ahead and delete the un-used slices:
gpart delete -i 4 ada0s1
gpart delete -i 6 ada0s1The above commands should return a conformation stating that the selected slice was deleted (for example "ada0s1p4 deleted"). Looking at the gpart output again, we will have two areas of free space:
gpart show ada0s1
=> 0 71119692 ada0s1 BSD (33G)
0 1048576 1 freebsd-ufs (512M)
1048576 2029408 2 freebsd-swap (990M)
3077984 3110912 - free - (1.5G)
6188896 1048576 5 freebsd-ufs (512M)
7237472 63882220 - free - (30G)Move /tmp and Swap
Even though we have some free space, it's not very useful in it's current location and layout. Notice how our swap and /tmp slices are blocking the way to grow the root slice.
gpart show ada0s1
=> 0 71119692 ada0s1 BSD (33G)
0 1048576 1 freebsd-ufs (512M)
1048576 2029408 2 freebsd-swap (990M)
3077984 3110912 - free - (1.5G)
6188896 1048576 5 freebsd-ufs (512M)
7237472 63882220 - free - (30G)We need to do some additional re-arranging. Ideally I would move the /tmp and swap to a separate volume. You could also create a new /tmp and swap at the end, in which case you'd need to would need to specify the starting block using the "-b" parameter. Check the gpart man page for usage instructions.
Create and format a new partition for /tmp:
# gpart add -s 5G -t freebsd-ufs da1
da1p1 added
# newfs /dev/da1p1Make sure it's mountable and finally set the correct permissions:
mkdir /mnt/tmpnew
mount /dev/da1p1 /mnt/tmpnew
chmod 1777 /mnt/tmpnewUse the remaining space on the volume for swap by omitting the "-s" parameter.
HINT: If you plan on keeping swap on the same volume, it's probably easier to do this step after you've resized the root partition/slice.
gpart add -t freebsd-swap da1The above commands should have return something like "da1p1 added".
Update /etc/fstab
Modify /etc/fstab and update the device names for swap and /tmp. Using FreeBSD's easy editor: edit /etc/fstab, open up the file and make the changes according to your setup. In the example below I simply commented out the old devices and added in the new ones.
# Device Mountpoint FStype Options Dump Pass#
/dev/da1p2 none swap sw 0 0
#/dev/ad0s1b none swap sw 0 0
/dev/ad0s1a / ufs rw 1 1
/dev/da1p1 /tmp ufs rw 2 2
#/dev/ad0s1e /tmp ufs rw 2 2
/dev/da0s1d /usr ufs rw 2 2
/dev/ad1s1d /var ufs rw 2 2
/dev/acd0 /cdrom cd9660 ro,noauto 0 0You can manually unmount each of the modified filesystems using umount -f, then do a mount -a. Just be sure the old filesystems are indeed unmounted before re-mounting.
Example: unmount /tmp
umount -f /tmpUse the swapoff and swapon commands to switch to the new swap partition.
swapoff /dev/ad0s1b
swapon /dev/da1p2The alternative is to reboot your system and hope it comes back (this is why we make backups!). Unless you goofed up big time, the system will boot up normally without any problem. If it does not, boot into single user mode and review the /etc/fstab file. Revert back to the old values and start over if necessary.
Delete Old /tmp and Swap
The /tmp directory should not contain anything of value and it's safe to delete. If for whatever reason you want/need to keep it's contents, do a UFS dump/restore.
Delete the old swap and /tmp slices. In our case they are slices 2 and 5:
gpart delete -i 2 ada0s1
gpart delete -i 5 ada0s1Now we have lots of free contiguous space, giving us room to grow.
Re-size the Root Partition/Slice
With enough contiguous free space after the root slice, it's possible to expand it:
gpart show ada0s1
=> 0 71119692 ada0s1 BSD (33G)
0 1048576 1 freebsd-ufs (512M)
1048576 63882220 - free - (32.5G)With FreeBSD versions less than 11 you need to boot into a FreeBSD Live CD environment in order to re-size the root partition/slice. Create your live media using the image you downloaded and boot the system to it. Select "Live CD" when asked and you'll be dropped into a shell prompt. Otherwise you can continue with what's called an "online" resize.
Use the gpart tool to re-size the root partition to the maximum size by omitting the "-s" parameter.
IMPORTANT: If you plan on keeping swap on the same volume you must leave the appropriate amount of space at the end. Therefore you must specify a new size using the "-s" parameter.
gpart resize -i 1 ada0s1If successful you'll see a message saying it has been resized
ada0s1a resizedA failure means you can't do an online resize. You'll need to boot off a live CD.
The root slice has now been expanded to fill the whole partition:
gpart show ada0s1
=> 0 71119692 ada0s1 BSD (33G)
0 71119692 1 freebsd-ufs (33G)Re-size the Root Filesystem
The filesystem needs to be resized to use the larger partition/slice. Use growfs to expand the UFS filesystem. Again you can omit the size parameter to have it automatically use all available space:
# growfs /dev/ada0s1
Device is mounted read-write; resizing will result in temporary write suspension for /.
It's strongly recommended to make a backup before growing the file system.
OK to grow filesystem on /dev/ada0s1, mounted on /, from 512MB to 33GB? [yes/no]Type "Yes" to proceed when asked, after which you'll see some output stating the locations of some new super block backups.
super-block backups (for fsck_ffs -b #) at:
19233792, 20516032, 21798272, 23080512, 24362752, 25644992, 26927232A failure means you can't do an online resize. You'll need to boot into single user mode or off a live CD.
Mission Complete
If you did an online resize there is no need to reboot.
If you had to reboot, your system should (hopefully) come back with no problem. Use gpart show to verify the new partition layouts and sizes. You'll see that your root file system is larger and has more free space:
gpart show
=> 63 419430337 da0 MBR (200G)
63 419424957 1 freebsd [active] (200G)
419425020 5380 - free - (2.6M)
=> 0 419424957 da0s1 BSD (200G)
0 419424957 4 freebsd-ufs (200G)
=> 34 20971453 da1 GPT (10G)
34 10485760 1 freebsd-ufs (5.0G)
10485794 10485693 2 freebsd-swap (5G)
=> 63 71184249 ada0 MBR (33G)
63 71184249 1 freebsd [active] (33G)
20971440 80 - free - (40k)
=> 0 71184249 ada0s1 BSD (33G)
0 71184249 1 freebsd-ufs (33G)More free space.
df -h
Filesystem Size Used Avail Capacity Mounted on
/dev/ad0s1a 33G 575M 32.5G 1% /
devfs 1.0k 1.0k 0B 100% /dev
/dev/da1p1 4.9G 2.1M 4.5G 0% /tmp
/dev/da0s1d 193G 31G 147G 17% /usr
/dev/ad1s1d 116G 15G 91G 14% /var
.png)