一、问题起源
在Linux系统中，通过rm删除文件将会从文件系统的目录结构上解除链接(unlink)，如果文件是被打开的（有一个进程正在使用），那么进程将仍然可以读取该文件磁盘空间也一直被占用
这样就会导致我们明明删除了文件，但是磁盘空间却未被释放
二、问题分析
1、首先获得一个已经被删除但是仍然被应用程序占用的文件列表
root@instance-zo0fkfng:~# lsof|grep deleted init 1 root 10w REG 253,1 283 155958 /var/log/upstart/network-manager.log.1 (deleted) init 1 root 12w REG 253,1 314 155934 /var/log/upstart/mountall.log.1 (deleted) init 1 root 13w REG 253,1 508 131203 /var/log/upstart/modemmanager.log.1 (deleted) init 1 root 17w REG 253,1 881 155967 /var/log/upstart/mysql.log.1 (deleted) init 1 root 24w REG 253,1 63 155949 /var/log/upstart/dbus.log.1 (deleted) mysqld 1159 mysql 4u REG 253,1 0 374 /tmp/ibkjeh3Z (deleted) mysqld 1159 mysql 5u REG 253,1 0 3458 /tmp/ibdbHetZ (deleted) mysqld 1159 mysql 6u REG 253,1 0 29466 /tmp/ibQSxcTY (deleted) mysqld 1159 mysql 7u REG 253,1 0 282 /tmp/ibxSgZNX (deleted) mysqld 1159 mysql 11u REG 253,1 0 29469 /tmp/ibXTzqnX (deleted) mysqld 1159 1185 mysql 4u REG 253,1 0 374 /tmp/ibkjeh3Z (deleted) mysqld 1159 17965 mysql 4u REG 253,1 0 374 /tmp/ibkjeh3Z (deleted) mysqld 1159 17965 mysql 5u REG 253,1 0 3458 /tmp/ibdbHetZ (deleted) mysqld 1159 17965 mysql 6u REG 253,1 0 29466 /tmp/ibQSxcTY (deleted) mysqld 1159 17965 mysql 7u REG 253,1 0 282 /tmp/ibxSgZNX (deleted) mysqld 1159 17965 mysql 11u REG 253,1 0 29469 /tmp/ibXTzqnX (deleted) php5-fpm 3724 www-data 3ur REG 253,1 0 29468 /tmp/.ZendSem.gfHFuR (deleted) php5-fpm 3853 www-data 3ur REG 253,1 0 29468 /tmp/.ZendSem.gfHFuR (deleted) php5-fpm 7741 root 3u REG 253,1 0 29468 /tmp/.ZendSem.gfHFuR (deleted) php5-fpm 9900 www-data 3u REG 253,1 0 29468 /tmp/.ZendSem.gfHFuR (deleted) php5-fpm 9901 www-data 3u REG 253,1 0 29468 /tmp/.ZendSem.gfHFuR (deleted) php5-fpm 9930 www-data 3u REG 253,1 0 29468 /tmp/.ZendSem.gfHFuR (deleted) php5-fpm 9961 www-data 3u REG 253,1 0 29468 /tmp/.ZendSem.gfHFuR (deleted) php5-fpm 9965 www-data 3u REG 253,1 0 29468 /tmp/.ZendSem.gfHFuR (deleted) php5-fpm 9966 www-data 3u REG 253,1 0 29468 /tmp/.ZendSem.gfHFuR (deleted) php5-fpm 10126 www-data 3u REG 253,1 0 29468 /tmp/.ZendSem.gfHFuR (deleted) php5-fpm 10127 www-data 3u REG 253,1 0 29468 /tmp/.ZendSem.gfHFuR (deleted) php5-fpm 10128 www-data 3u REG 253,1 0 29468 /tmp/.ZendSem.gfHFuR (deleted) php5-fpm 10174 www-data 3u REG 253,1 0 29468 /tmp/.ZendSem.gfHFuR (deleted) php5-fpm 10196 www-data 3u REG 253,1 0 29468 /tmp/.ZendSem.gfHFuR (deleted) php5-fpm 10248 www-data 3u REG 253,1 0 29468 /tmp/.ZendSem.gfHFuR (deleted) php5-fpm 10316 www-data 3u REG 253,1 0 29468 /tmp/.ZendSem.gfHFuR (deleted) php5-fpm 10375 www-data 3u REG 253,1 0 29468 /tmp/.ZendSem.gfHFuR (deleted) php5-fpm 10396 www-data 3u REG 253,1 0 29468 /tmp/.ZendSem.gfHFuR (deleted) php5-fpm 10400 www-data 3u REG 253,1 0 29468 /tmp/.ZendSem.gfHFuR (deleted) php5-fpm 10401 www-data 3u REG 253,1 0 29468 /tmp/.ZendSem.gfHFuR (deleted) php5-fpm 10438 www-data 3u REG 253,1 0 29468 /tmp/.ZendSem.gfHFuR (deleted) php5-fpm 10479 www-data 3u REG 253,1 0 29468 /tmp/.ZendSem.gfHFuR (deleted) php5-fpm 10511 www-data 3u REG 253,1 0 29468 /tmp/.ZendSem.gfHFuR (deleted) php5-fpm 10531 www-data 3u REG 253,1 0 29468 /tmp/.ZendSem.gfHFuR (deleted) php5-fpm 11150 www-data 3u REG 253,1 0 29468 /tmp/.ZendSem.gfHFuR (deleted) php5-fpm 11638 www-data 3u REG 253,1 0 29468 /tmp/.ZendSem.gfHFuR (deleted) php5-fpm 11670 www-data 3u REG 253,1 0 29468 /tmp/.ZendSem.gfHFuR (deleted) php5-fpm 12210 www-data 3u REG 253,1 0 29468 /tmp/.ZendSem.gfHFuR (deleted) php5-fpm 14974 www-data 3u REG 253,1 0 29468 /tmp/.ZendSem.gfHFuR (deleted) php5-fpm 17101 www-data 3u REG 253,1 0 29468 /tmp/.ZendSem.gfHFuR (deleted) php5-fpm 17161 www-data 3u REG 253,1 0 29468 /tmp/.ZendSem.gfHFuR (deleted) php5-fpm 17261 www-data 3u REG 253,1 0 29468 /tmp/.ZendSem.gfHFuR (deleted) php5-fpm 17470 www-data 3u REG 253,1 0 29468 /tmp/.ZendSem.gfHFuR (deleted) |
从输出结果可以看到哪些文件还被使用，未被释放空间
或者
root@instance-zo0fkfng:~# find /proc/*/fd -ls | grep '(deleted)' 71253 0 lrwx------ 1 root root 64 Jun 10 22:09 /proc/1159/fd/4 -> /tmp/ibkjeh3Z\ (deleted) 71254 0 lrwx------ 1 root root 64 Jun 10 22:09 /proc/1159/fd/5 -> /tmp/ibdbHetZ\ (deleted) 71255 0 lrwx------ 1 root root 64 Jun 10 22:09 /proc/1159/fd/6 -> /tmp/ibQSxcTY\ (deleted) 71256 0 lrwx------ 1 root root 64 Jun 10 22:09 /proc/1159/fd/7 -> /tmp/ibxSgZNX\ (deleted) 71260 0 lrwx------ 1 root root 64 Jun 10 22:09 /proc/1159/fd/11 -> /tmp/ibXTzqnX\ (deleted) 221991074 0 lrwx------ 1 root root 64 Aug 27 19:43 /proc/18132/fd/3 -> /tmp/.ZendSem.hIpOFr\ (deleted) 221991101 0 lrwx------ 1 www-data www-data 64 Aug 27 19:43 /proc/18135/fd/3 -> /tmp/.ZendSem.hIpOFr\ (deleted) 221991122 0 lrwx------ 1 www-data www-data 64 Aug 27 19:43 /proc/18136/fd/3 -> /tmp/.ZendSem.hIpOFr\ (deleted) 221991147 0 lrwx------ 1 www-data www-data 64 Aug 27 19:43 /proc/18137/fd/3 -> /tmp/.ZendSem.hIpOFr\ (deleted) 221991168 0 lrwx------ 1 www-data www-data 64 Aug 27 19:43 /proc/18138/fd/3 -> /tmp/.ZendSem.hIpOFr\ (deleted) 221991189 0 lrwx------ 1 www-data www-data 64 Aug 27 19:43 /proc/18139/fd/3 -> /tmp/.ZendSem.hIpOFr\ (deleted) 221991210 0 lrwx------ 1 www-data www-data 64 Aug 27 19:43 /proc/18140/fd/3 -> /tmp/.ZendSem.hIpOFr\ (deleted) 221991231 0 lrwx------ 1 www-data www-data 64 Aug 27 19:43 /proc/18141/fd/3 -> /tmp/.ZendSem.hIpOFr\ (deleted) 221991252 0 lrwx------ 1 www-data www-data 64 Aug 27 19:43 /proc/18142/fd/3 -> /tmp/.ZendSem.hIpOFr\ (deleted) 221991273 0 lrwx------ 1 www-data www-data 64 Aug 27 19:43 /proc/18143/fd/3 -> /tmp/.ZendSem.hIpOFr\ (deleted) 221991294 0 lrwx------ 1 www-data www-data 64 Aug 27 19:43 /proc/18144/fd/3 -> /tmp/.ZendSem.hIpOFr\ (deleted) 221991315 0 lrwx------ 1 www-data www-data 64 Aug 27 19:43 /proc/18147/fd/3 -> /tmp/.ZendSem.hIpOFr\ (deleted) 222001888 0 lrwx------ 1 www-data www-data 64 Aug 27 19:44 /proc/18157/fd/3 -> /tmp/.ZendSem.hIpOFr\ (deleted) 222001909 0 lrwx------ 1 www-data www-data 64 Aug 27 19:44 /proc/18161/fd/3 -> /tmp/.ZendSem.hIpOFr\ (deleted) 222001938 0 lrwx------ 1 www-data www-data 64 Aug 27 19:44 /proc/18170/fd/3 -> /tmp/.ZendSem.hIpOFr\ (deleted) 222001959 0 lrwx------ 1 www-data www-data 64 Aug 27 19:44 /proc/18178/fd/3 -> /tmp/.ZendSem.hIpOFr\ (deleted) find: `/proc/18182/fd/5': No such file or directory 55130 0 l-wx------ 1 root root 64 Jun 10 22:09 /proc/1/fd/10 -> /var/log/upstart/network-manager.log.1\ (deleted) 55132 0 l-wx------ 1 root root 64 Jun 10 22:09 /proc/1/fd/12 -> /var/log/upstart/mountall.log.1\ (deleted) 55133 0 l-wx------ 1 root root 64 Jun 10 22:09 /proc/1/fd/13 -> /var/log/upstart/modemmanager.log.1\ (deleted) 55137 0 l-wx------ 1 root root 64 Jun 10 22:09 /proc/1/fd/17 -> /var/log/upstart/mysql.log.1\ (deleted) 55142 0 l-wx------ 1 root root 64 Jun 10 22:09 /proc/1/fd/24 -> /var/log/upstart/dbus.log.1\ (deleted) find: `/proc/self/fd/5': No such file or directory |
2、如何让进程释放文件，进而释放磁盘空间
kill掉相应的进程或者重启该进程，让os自动回收磁盘空间
当linux打开一个文件的时候,Linux内核会为每一个进程在/proc/ 『/proc/nnnn/fd/目录（nnnn为pid）』建立一个以其pid为名的目录用来保存进程的相关信息，而其子目录fd保存的是该进程打开的所有文件的fd（fd：file descriptor）
kill进程是通过截断proc文件系统中的文件可以强制要求系统回收分配给正在使用的的文件
root@instance-zo0fkfng:~# service php5-fpm restart root@instance-zo0fkfng:~# lsof|grep deleted init 1 root 10w REG 253,1 283 155958 /var/log/upstart/network-manager.log.1 (deleted) init 1 root 12w REG 253,1 314 155934 /var/log/upstart/mountall.log.1 (deleted) init 1 root 13w REG 253,1 508 131203 /var/log/upstart/modemmanager.log.1 (deleted) init 1 root 17w REG 253,1 881 155967 /var/log/upstart/mysql.log.1 (deleted) init 1 root 24w REG 253,1 63 155949 /var/log/upstart/dbus.log.1 (deleted) mysqld 1159 mysql 4u REG 253,1 0 374 /tmp/ibkjeh3Z (deleted) mysqld 1159 mysql 5u REG 253,1 0 3458 /tmp/ibdbHetZ (deleted) mysqld 1159 mysql 6u REG 253,1 0 29466 /tmp/ibQSxcTY (deleted) mysqld 1159 mysql 7u REG 253,1 0 282 /tmp/ibxSgZNX (deleted) mysqld 1159 mysql 11u REG 253,1 0 29469 /tmp/ibXTzqnX (deleted) mysqld 1159 1185 mysql 4u REG 253,1 0 374 /tmp/ibkjeh3Z (deleted) mysqld 1159 1185 mysql 5u REG 253,1 0 3458 /tmp/ibdbHetZ (deleted) mysqld 1159 1185 mysql 6u REG 253,1 0 29466 /tmp/ibQSxcTY (deleted) mysqld 1159 1185 mysql 7u REG 253,1 0 282 /tmp/ibxSgZNX (deleted) mysqld 1159 1185 mysql 11u REG 253,1 0 29469 /tmp/ibXTzqnX (deleted) mysqld 1159 1186 mysql 4u REG 253,1 0 374 /tmp/ibkjeh3Z (deleted) mysqld 1159 1186 mysql 5u REG 253,1 0 3458 /tmp/ibdbHetZ (deleted) mysqld 1159 18149 mysql 4u REG 253,1 0 374 /tmp/ibkjeh3Z (deleted) mysqld 1159 18149 mysql 5u REG 253,1 0 3458 /tmp/ibdbHetZ (deleted) mysqld 1159 18149 mysql 6u REG 253,1 0 29466 /tmp/ibQSxcTY (deleted) mysqld 1159 18149 mysql 7u REG 253,1 0 282 /tmp/ibxSgZNX (deleted) mysqld 1159 18149 mysql 11u REG 253,1 0 29469 /tmp/ibXTzqnX (deleted) mysqld 1159 18150 mysql 4u REG 253,1 0 374 /tmp/ibkjeh3Z (deleted) mysqld 1159 18150 mysql 5u REG 253,1 0 3458 /tmp/ibdbHetZ (deleted) mysqld 1159 18150 mysql 6u REG 253,1 0 29466 /tmp/ibQSxcTY (deleted) mysqld 1159 18150 mysql 7u REG 253,1 0 282 /tmp/ibxSgZNX (deleted) mysqld 1159 18150 mysql 11u REG 253,1 0 29469 /tmp/ibXTzqnX (deleted) php5-fpm 18132 root 3u REG 253,1 0 28407 /tmp/.ZendSem.hIpOFr (deleted) php5-fpm 18135 www-data 3u REG 253,1 0 28407 /tmp/.ZendSem.hIpOFr (deleted) php5-fpm 18136 www-data 3u REG 253,1 0 28407 /tmp/.ZendSem.hIpOFr (deleted) php5-fpm 18137 www-data 3u REG 253,1 0 28407 /tmp/.ZendSem.hIpOFr (deleted) php5-fpm 18138 www-data 3u REG 253,1 0 28407 /tmp/.ZendSem.hIpOFr (deleted) php5-fpm 18139 www-data 3u REG 253,1 0 28407 /tmp/.ZendSem.hIpOFr (deleted) php5-fpm 18140 www-data 3u REG 253,1 0 28407 /tmp/.ZendSem.hIpOFr (deleted) php5-fpm 18141 www-data 3u REG 253,1 0 28407 /tmp/.ZendSem.hIpOFr (deleted) php5-fpm 18142 www-data 3u REG 253,1 0 28407 /tmp/.ZendSem.hIpOFr (deleted) php5-fpm 18143 www-data 3u REG 253,1 0 28407 /tmp/.ZendSem.hIpOFr (deleted) php5-fpm 18144 www-data 3u REG 253,1 0 28407 /tmp/.ZendSem.hIpOFr (deleted) php5-fpm 18147 www-data 3u REG 253,1 0 28407 /tmp/.ZendSem.hIpOFr (deleted) |
可以看到php5-fpm变少了
还有一种方式，清除文件
: > /path/to/the/file.log
: > "/proc/$pid/fd/$fd"
