New Zealand Linux

Sometimes files gets deleted accidentally, whilst they are still running. This also applies to things like flash videos or other things, usually temporary files or even when a server is exploited.
Sometimes you want to keep those files right? but they are deleted! but how can they be running when they are deleted?
\
This is what /proc is all about. Its a neat way of keeping track of information like file descriptors for each PID.
So, everyone knows what ps does, it shows you whats running .. like this

start
www-data 4146 2.0 2.9 62088 28292 ? S 00:00 0:02 \_ /usr/sbin/apache2 -k start
www-data 5287 0.0 0.4 42072 4536 ? S 00:01 0:00 \_ /usr/sbin/apache2 -k start
1005 27387 1.9 0.5 7940 5576 ? S Nov28 9:31 perl

Oh wait, what is that thing called ‘perl’ ? This is an example from a hacked box. I knew the application was not called ‘perl’, and since it had been sending spam i knew it was probably a bad file.
So, i wanted to find what files were open by the pid 27387 – i installed and used ‘lsof’ which gives an ls of open files.

# lsof -p 27387
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
perl 27387 ausername cwd DIR 202,1 4096 476065 /tmp
perl 27387 ausername rtd DIR 202,1 4096 2 /
perl 27387 ausername txt REG 202,1 1254016 100305 /usr/bin/perl
perl 27387 ausername mem REG 202,1 75472 344738 /lib/i686/nosegneg/libresolv-2.9.so
perl 27387 ausername mem REG 202,1 21976 344867 /lib/i686/nosegneg/libnss_dns-2.9.so
perl 27387 ausername mem REG 202,1 42504 344774 /lib/i686/nosegneg/libnss_files-2.9.so
perl 27387 ausername mem REG 202,1 38444 345027 /lib/i686/nosegneg/libnss_nis-2.9.so
perl 27387 ausername mem REG 202,1 87804 344872 /lib/i686/nosegneg/libnsl-2.9.so
perl 27387 ausername mem REG 202,1 30436 344758 /lib/i686/nosegneg/libnss_compat-2.9.so
perl 27387 ausername mem REG 202,1 19816 181707 /usr/lib/perl/5.10.0/auto/Socket/Socket.so
perl 27387 ausername mem REG 202,1 38296 345032 /lib/i686/nosegneg/libcrypt-2.9.so
perl 27387 ausername mem REG 202,1 1450372 344746 /lib/i686/nosegneg/libc-2.9.so
perl 27387 ausername mem REG 202,1 116294 345030 /lib/i686/nosegneg/libpthread-2.9.so
perl 27387 ausername mem REG 202,1 149328 344754 /lib/i686/nosegneg/libm-2.9.so
perl 27387 ausername mem REG 202,1 9676 345034 /lib/i686/nosegneg/libdl-2.9.so
perl 27387 ausername mem REG 202,1 117348 344891 /lib/ld-2.9.so
perl 27387 ausername 0r CHR 1,3 0t0 197031 /dev/null
perl 27387 ausername 1w CHR 1,3 0t0 197031 /dev/null
perl 27387 ausername 2w CHR 1,3 0t0 197031 /dev/null
perl 27387 ausername 3r REG 202,1 14782 2872816 /tmp/ (deleted)
perl 27387 ausername 4wW REG 202,1 0 2872817 /tmp/…

Edit: You can use lsof +L1 -p pid/27387 to only show deleted items – thanks Jeffrey Caughel

Sorry for verbosity there, but its required. Ok, now you can see the line that has (deleted) – this is what we want. The 4th line over tells me its using the file descriptor 3 (ignore the r after it for now)

So to access that file, i look at /proc/27387/fd/3 – often its best to copy that file elsewhere before it gets deleted (ie if program is closed). The first number is the pid, the fd is the file descriptor , and the last is the script itself which was deleted – in this case a spaming perl script.
Now i knew what that script did, it was not touching the filesystem just sending spam, so i knew it was safe to kill it.
This is also handy for saving youtube videos or when you accidentally rm -rf /etc or /bin when things are still running 🙂