Posts Tagged deleted

Finding deleted files in Linux (when an app is still running holding them open)

Posted by on Wednesday, 30 November, 2011

Sometimes files gets deleted accidentally, whilst they are still running. This also applies to things like flash videos or other things, usually temporary files or even when a server is exploited.
Sometimes you want to keep those files right? but they are deleted! but how can they be running when they are deleted?
\
This is what /proc is all about. Its a neat way of keeping track of information like file descriptors for each PID.
So, everyone knows what ps does, it shows you whats running .. like this

start
www-data 4146 2.0 2.9 62088 28292 ? S 00:00 0:02 \_ /usr/sbin/apache2 -k start
www-data 5287 0.0 0.4 42072 4536 ? S 00:01 0:00 \_ /usr/sbin/apache2 -k start
1005 27387 1.9 0.5 7940 5576 ? S Nov28 9:31 perl

Oh wait, what is that thing called ‘perl’ ? This is an example from a hacked box. I knew the application was not called ‘perl’, and since it had been sending spam i knew it was probably a bad file.
So, i wanted to find what files were open by the pid 27387 – i installed and used ‘lsof’ which gives an ls of open files.

# lsof -p 27387
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
perl 27387 ausername cwd DIR 202,1 4096 476065 /tmp
perl 27387 ausername rtd DIR 202,1 4096 2 /
perl 27387 ausername txt REG 202,1 1254016 100305 /usr/bin/perl
perl 27387 ausername mem REG 202,1 75472 344738 /lib/i686/nosegneg/libresolv-2.9.so
perl 27387 ausername mem REG 202,1 21976 344867 /lib/i686/nosegneg/libnss_dns-2.9.so
perl 27387 ausername mem REG 202,1 42504 344774 /lib/i686/nosegneg/libnss_files-2.9.so
perl 27387 ausername mem REG 202,1 38444 345027 /lib/i686/nosegneg/libnss_nis-2.9.so
perl 27387 ausername mem REG 202,1 87804 344872 /lib/i686/nosegneg/libnsl-2.9.so
perl 27387 ausername mem REG 202,1 30436 344758 /lib/i686/nosegneg/libnss_compat-2.9.so
perl 27387 ausername mem REG 202,1 19816 181707 /usr/lib/perl/5.10.0/auto/Socket/Socket.so
perl 27387 ausername mem REG 202,1 38296 345032 /lib/i686/nosegneg/libcrypt-2.9.so
perl 27387 ausername mem REG 202,1 1450372 344746 /lib/i686/nosegneg/libc-2.9.so
perl 27387 ausername mem REG 202,1 116294 345030 /lib/i686/nosegneg/libpthread-2.9.so
perl 27387 ausername mem REG 202,1 149328 344754 /lib/i686/nosegneg/libm-2.9.so
perl 27387 ausername mem REG 202,1 9676 345034 /lib/i686/nosegneg/libdl-2.9.so
perl 27387 ausername mem REG 202,1 117348 344891 /lib/ld-2.9.so
perl 27387 ausername 0r CHR 1,3 0t0 197031 /dev/null
perl 27387 ausername 1w CHR 1,3 0t0 197031 /dev/null
perl 27387 ausername 2w CHR 1,3 0t0 197031 /dev/null
perl 27387 ausername 3r REG 202,1 14782 2872816 /tmp/ (deleted)
perl 27387 ausername 4wW REG 202,1 0 2872817 /tmp/…

Edit: You can use lsof +L1 -p pid/27387 to only show deleted items – thanks Jeffrey Caughel

Sorry for verbosity there, but its required. Ok, now you can see the line that has (deleted) – this is what we want. The 4th line over tells me its using the file descriptor 3 (ignore the r after it for now)

So to access that file, i look at /proc/27387/fd/3 – often its best to copy that file elsewhere before it gets deleted (ie if program is closed). The first number is the pid, the fd is the file descriptor , and the last is the script itself which was deleted – in this case a spaming perl script.
Now i knew what that script did, it was not touching the filesystem just sending spam, so i knew it was safe to kill it.
This is also handy for saving youtube videos or when you accidentally rm -rf /etc or /bin when things are still running 🙂