Subject: Re: intercepting system calls

From: [email-masked] (David Wagner)
Date: 12 Dec 1999 20:23:27 -0800

In article <[email-masked]>, Pat Gunn  <[email-masked]> wrote:
> David Wagner wrote:
> > In Solaris, use the /proc filesystem.  It is far better than ptrace.
> 
> I don't understand how it's better than ptrace --
> it seems to me that you'd do an awful lot of polling of things in /proc
> and very likely would miss events.

You won't "miss events" with the /proc filesystem: they are queued up in
the kernel until you receive them.  The tracing process will typically
issue a blocking call to /proc that only returns when some event is
available.  (select() and polling functionality are also available.)

The problems with ptrace are subtle and hard to explain, but they are
there.  Go read the source code of strace sometime if you want to see
some examples: it is amazingly convoluted.

For example, race conditions abound with ptrace.  Just try to reliably
trace all children of a given application, without missing events --
there are two or three different race conditions here to watch out for.

As another example, because ptrace uses signals to deliver events,
it changes some of the semantics of signals and wait() for traced
applications.  (ptrace also slightly modifies the parent-child
relationship of Unix processes, which may have strange consequences.)
These effects require arcane workarounds in the tracing process to restore
transparency, so applications don't break when they are being traced.

Finally, the Solaris /proc filesystem offers many pieces of important
functionality which are unavailable with ptrace.  One may select just a
subset of the system calls to watch; one may easily follow all children
of the traced application if it forks; one may prevent selected syscalls
from being executed; and so on.

If you want a disgusting amount of detail on some esoteric and undesirable
quirks of ptrace, see Chapter 12 of the following technical report:
  ``Janus: an approach for confinement of untrusted applications''
  http://www.cs.berkeley.edu/~daw/papers/janus-masters.ps
I report on a security tool that uses ptrace, and on some of the pitfalls
I encountered in building the tool.