Today i had a person who had an interesting problem. They were getting the message ‘disk is full’ despite having plenty of free space. Luckily for him, my first thought was ‘inodes?’
I logged in and checked his inode usage
root@askdev:# df -i Filesystem Inodes IUsed IFree IUse% Mounted on /dev/xvda1 525312 524844 468 100% / varrun 65579 27 65552 1% /var/run varlock 65579 2 65577 1% /var/lock udev 65579 2696 62883 5% /dev devshm 65579 1 65578 1% /dev/shm |
This shows that all the inodes on the disk itself are full.
I used the following script to determine where the inode usage was most
root@askdev:/# for i in `ls -1A`; do echo “`find $i | sort -u | wc -l` $i”; done | sort -rn | head -5 468388 var 49844 usr 18741 proc 5187 sys 5026 root |
I tracked it down to /var/lib/php5/ and all the session files in there.
I used a find to then find any that were older than 10 days
root@askdev:/var/lib/php5# find ./ -type f -mtime +10 | wc -l 111041 |
High inode usage is usually caused by a massive number of small files. In this case the session files are normally stored somewhere temporary and removed when not in use. Either there could have been a bug in the code not removing them or it was a higher traffic website.
You can delete files older than 10 days if you want with the following command
cd /dir/of/inodes find ./ -type f -mtime +10 | xargs rm |