SystemMen - In this article, I will share with you a case where the df command shows the disk is full but it is not.
Case description
I am using a MariaDB cluster. On the master server number 1, I received a warning that the folder /var/log is full.
I login to the server, checking with the df command, the result that the folder using 99-100%.
[root@db-01 danie]# df -Th Filesystem Type Size Used Avail Use% Mounted on /dev/mapper/centos-root ext4 30G 2,6G 26G 10% / devtmpfs devtmpfs 3,9G 0 3,9G 0% /dev tmpfs tmpfs 3,9G 0 3,9G 0% /dev/shm tmpfs tmpfs 3,9G 425M 3,5G 11% /run tmpfs tmpfs 3,9G 0 3,9G 0% /sys/fs/cgroup /dev/sda1 xfs 497M 162M 336M 33% /boot /dev/mapper/centos-tmp ext4 22G 45M 20G 1% /tmp /dev/mapper/centos-var_log ext4 30G 28G 494M 99% /var/log /dev/mapper/centos-home ext4 9,8G 37M 9,2G 1% /home /dev/mapper/centos-var_lib_mysql ext4 197G 24G 164G 13% /var/lib/mysql tmpfs tmpfs 783M 0 783M 0% /run/user/1000
You can see that this folder is 30GB in size and it is using 28GB.
Next, I use the du command to display the current size of the /var/log directory. What surprised me was that it was only using 1.1GB.
[root@db-01 danie]# du -sh /var/log 1,1G /var/log
So what’s going on?
Find the cause of the df command shows the full is disk
With my experience, I have encountered the case of files that have been deleted but it is not actually deleted. These files are labeled deleted but it is still on the disk.
This causes the df command to shows an incorrect result.
I use the lsof command to find deleted files. And I found out that there is a slow log file of about 27-28GB in size labeled deleted.
[root@db-01 danie]# lsof +L1 COMMAND PID USER FD TYPE DEVICE SIZE/OFF NLINK NODE NAME systemd-j 523 root txt REG 253,0 274752 0 660140 /usr/lib/systemd/systemd-journald;5bad8321 (deleted) systemd-l 776 root txt REG 253,0 572320 0 660142 /usr/lib/systemd/systemd-logind;5bad8321 (deleted) dbus-daem 784 dbus txt REG 253,0 442080 0 660381 /usr/bin/dbus-daemon;5bad8321 (deleted) agetty 799 root txt REG 253,0 36864 0 659369 /usr/sbin/agetty;5bad8321 (deleted) NetworkMa 811 root txt REG 253,0 2618600 0 660555 /usr/sbin/NetworkManager;5bad8321 (deleted) mysqld 9700 mysql 20u REG 253,5 9083 0 12 /tmp/ibWC769x (deleted) mysqld 9700 mysql 21u REG 253,5 0 0 13 /tmp/ibQKzxPO (deleted) mysqld 9700 mysql 22u REG 253,5 8551 0 14 /tmp/ibgmSBbm (deleted) mysqld 9700 mysql 26u REG 253,5 0 0 15 /tmp/ibqJ15SC (deleted) mysqld 9700 mysql 30w REG 253,4 28315029916 0 1048578 /var/log/mariadb/slow.log-20181223 (deleted)
Immediately at this step, I knew why the notification disk was full but not really.
My job now is to delete this file. I have written a script to delete these files.
Recommended Reading: Script free disk space from deleted files
Specifically in this case. I will do the following steps.
Step 1: Find files with deleted labels.
[root@db-01 danie]# find /proc/*/fd -ls | grep '(deleted)' 815211450 0 lrwx------ 1 mysql mysql 64 Mar 20 13:52 /proc/9700/fd/20 -> /tmp/ibWC769x\ (deleted) 815211451 0 lrwx------ 1 mysql mysql 64 Mar 20 13:52 /proc/9700/fd/21 -> /tmp/ibQKzxPO\ (deleted) 815211452 0 lrwx------ 1 mysql mysql 64 Mar 20 13:52 /proc/9700/fd/22 -> /tmp/ibgmSBbm\ (deleted) 815211456 0 lrwx------ 1 mysql mysql 64 Mar 20 13:52 /proc/9700/fd/26 -> /tmp/ibqJ15SC\ (deleted) 724035709 0 l-wx------ 1 mysql mysql 64 Mar 5 18:08 /proc/9700/fd/30 -> /var/log/mariadb/slow.log-20181223\ (deleted)
Step 2: Get the file name to delete.
[root@db-01 danie]# find /proc/*/fd -ls | grep '(deleted)' | awk '{print $11}'
/proc/9700/fd/20
/proc/9700/fd/21
/proc/9700/fd/22
/proc/9700/fd/26
/proc/9700/fd/30
In this case, I need to delete the slow log file with the deleted label. Therefore, , I will only need to delete the file /proc/9700/fd/30.
Step 3: delete files.
rm -f /proc/9700/fd/30
Alternatively, you can use the df command with the -i option to display the disk capacity more accurately.
[root@db-01 danie]# df -i Filesystem Inodes IUsed IFree IUse% Mounted on /dev/mapper/centos-root 1966080 43052 1923028 3% / devtmpfs 998417 402 998015 1% /dev tmpfs 1001256 1 1001255 1% /dev/shm tmpfs 1001256 542 1000714 1% /run tmpfs 1001256 16 1001240 1% /sys/fs/cgroup /dev/sda1 256000 334 255666 1% /boot /dev/mapper/centos-tmp 1411680 24 1411656 1% /tmp /dev/mapper/centos-var_log 1966080 108 1965972 1% /var/log /dev/mapper/centos-home 655360 29 655331 1% /home /dev/mapper/centos-var_lib_mysql 13107200 449 13106751 1% /var/lib/mysql tmpfs 1001256 1 1001255 1% /run/user/1000
You see the above result is completely different, it has displayed the correct capacity of the disk.
What does this parameter -i mean?
-i, --inodes
list inode information instead of block usage
Conclusion
This is one of the experiences I have experienced. If in the process of working with Linux servers, you encounter it. Hope this article will be useful to you, don’t be too worried if you don’t understand why the disk is always full.
«« How to create read-only user in MySQLIntroducing free hosting panel VestaCP »»
