Does your shell script use temporary files? Does it clean up the temporaries after it dies? What if the user aborts your script early by using ^C? What if he feels mean and uses kill -9 instead?
Not using temporary files is the easiest way out of this mess, but unfortunately there are situations where this is unavoidable. Let's say, for example, that you want to diff the output of two programs. Using temporary filenames, this can be done as follows.
bash
progA > outA.tmp
bash
progB > outB.tmp
bash
diff outA.tmp outB.tmp | progC
bash
rm outA.tmp
bash
rm outB.tmp
It's possible to use pipes to eliminate outA.tmp
bash
progB > outB.tmp
bash
progA | diff - outB.tmp | progC
bash
rm outB.tmp
It's also possible to eliminate outB.tmp
bash
progA > outA.tmp
bash
progB | diff outA.tmp - | progC
bash
rm outA.tmp
But with this technique it isn't possible to eliminate both temporaries at once. To see why this is is a problem, consider the case where progB is stuck in an infinite loop. In that case, the user is justified to kill your script using the magic CTRL-C combination; and even if the action isn't totally justified, the user can do whatever he wants and it's your job to keep your program from fooling up even when the user does. In any case, ^C will kill both progB and your script, which won't have the opportunity to delete its temporary file. It's not a very critical flaw, but I'd feel better if the script was robust enough to withstand this.
A naive solution could be to disable ^C.
bash
trap "" SIGINT
bash
progA > outA.tmp
bash
progB > outB.tmp
bash
diff outA.tmp outB.tmp | progC
bash
rm outA.tmp
bash
rm outB.tmp
But this will only annoy the user, who will just kill your script using kill -9 instead of ^C. And you can't disable kill -9. So what would be a better solution?
The solution I've been advocating until today is as follows.
bash
trap "rm -f outA.tmp outB.tmp" EXIT
bash
progA > outA.tmp
bash
progB > outB.tmp
bash
diff outA.tmp outB.tmp | progC
This will erase the temporaries on exit, whether the exit is normal or due to ^C. It still doesn't work against kill -9, but what does?
Well, I'll tell you what does. If user-mode code can't clean up on SIGKILL (the signal which -9 selects), then we'll use kernel-mode code. Incidentally, the kernel already contains code to clean up unclosed file descriptors, and we can use this to our advantage.
When we used "-" as an argument for diff, it really was just a short name for /proc/self/fd/0 (that is, standard input). This path names a virtual file which is deleted by the kernel when grep terminates. These virtual files correspond to file descriptors, so if we can create additional file descriptors, then we can create auto-cleaning temporary files.
read_fd3.rb
fd3, fd4 = IO.pipe
read_fd3.rb
fd3.getc # wait until fd4 is opened by an outside process
read_fd3.rb
fd4.close
read_fd3.rb
exec(*ARGV)
write_fd3.rb
$stdout.close
write_fd3.rb
$stdout = File.new "/proc/#{ARGV.shift}/fd/4", 'w'
write_fd3.rb
$stdout.write 'x'
write_fd3.rb
$stdout.flush
write_fd3.rb
exec(*ARGV)
bash
(
bash
progA | ruby read_fd3.rb diff /proc/self/fd/0 /proc/self/fd/3 &
bash
ruby write_fd3.rb $! progB
bash
wait $!
bash
) | progC
The first ruby program creates two linked descriptors and leaves them open, letting the exec'd program use them. In this case, the exec'd program is diff, and it is told to compare /proc/self/fd/0 against /proc/self/fd/3. The first is just standard input, which comes from progA. The second is one of the linked file descriptors which were left open.
The second ruby program writes to the other linked file descriptor. This effectively sends the output of progB to the virtual file which diff is reading, as needed.
write_fd3.rb also writes an additional byte before the transfering the control to progB. This byte is needed because the end-of-file only occurs once all programs have closed all of their handles on the virtual file. read_fd3.rb has one of those handles, but it cannot close it until write_fd3.rb has secured its own handle, or the end-of-file will come too soon. So write_fd3.rb sends one byte to tell read_fd3.rb that it is now safe to close its handle.
You'll notice that this virtual file strategy is a bit awkward to use. And now that you have learned how to do it by hand, I can reveal you a secret: this strategy was built into bash all along! The syntax is as follows.
bash
diff <(progA) <(progB) | progC
Concise, isn't it? However, this isn't the end of the story. There are some programs, unfortunately, which look at their input multiple times. You can't do that. Closing /proc/self/fd/3 is just as irreversible as closing stdin! And you can't use fseek(), either. So sorry, virtual files freed by the kernel. I tried to make this work, I really did, but... you're ordered, I'm random. We're just not made for each other.
Here's a more flexible solution. If the dying process cannot cleanup, then another process should take care of it. Preferably, a process which always runs, a daemon process. Whose job is to allocate and delete temporary files. A given client would reveal its process-id to the daemon, which will allocate a file for the client and delete it once the process with the given process-id has disappeared from the system. It could have disappeared for different reasons, including normal termination, ^C, and even kill -9.
If you think implementing a daemon would be going through a lot of trouble just for a few temporary files, then you'll be happy to hear that I've already went through this trouble for you. I called my daemon filenamed, and the corresponding client, filename, can be used as follows.
bash
diff $(progA | filename) $(filename $(progB)) | progC
I also provide a wrapper called "<", which can be used as a drop-in replacement for the bash syntax <(...) which I introduced earlier. My version, of course, allows fseek().
bash
diff $(\< progA) $(\< progB) | progC
There are a few aspects of my daemon which I'm not that proud of, but which I don't intend to fix. First, it depends on ruby, even though it's use is not ruby-specific. I'm just too lazy to reimplement it in C. Second, it's not very secure. Clients shouldn't be able to take control of the daemon, but they could easily request it to keep files until process 1 dies, that is, forever. Clients shouldn't be able to modify the files of other clients, but they could easily read them.
If you can live with that, enjoy!
Posts
