Kernel Markers in Linux

I feel like I’m parroting LWN a bit recently, but I like to point out items that are of particular interest from a RAS perspective. To that end, I thought I’d mention that Kernel Markers are making good headway in the 2.6.24 kernel.

These markers allow probe points to be specified in the kernel. Each probe point can be active, meaning that a probe is attached to that point, or inactive, meaning that the probe point is not currently “of interest.” When a marked point is encountered, if a probe is attached to the marker, that probe will be invoked with a set of parameters that are specified by the marker. If there is no probe attached to the marker, execution continues as normal. Therefore, unused marker points are mostly dormant.

“Mostly” dormant? That means that there is a very tiny bit of execution overhead each time a marker with no attached probe is encountered. Instrumentation with a moderate number of markers would have a small impact on execution performance and a somewhat larger impact on space (due to the extra data structures needed for markers).

This is certainly a boon for SystemTap; presumably, these markers will be used to make it easier to use and more maintainable.

What does this mean from the user’s perspective? A better instrumented kernel means richer, more interesting monitoring and analysis tools. Performance analysis tools like filemon, tprof, curt, and splat are available on AIX because of its lightweight trace instrumentation, in which each tracepoint can be activated or deactivated on the fly. Kernel markers provide an equivalent mechanism in Linux to enable similar higher-level tools.

I think this also opens up some interesting opportunities for aspect-oriented programming. For example, instead of using printk for logging, markers could be placed at points of interest for logging applications. That way, any (potentially 3rd party) logging application could attach to those points, and then be invoked with a severity flag and a log message each time the kernel wishes to log some information. All kinds of data aggregation and mining could occur; one could even apply something like machine learning to do some more complex processing. This can currently be done with syslog, but I think this kind of AOP would make these tools easier to develop.

One disadvantage, though, unless I’m misreading the documentation, is that only one probe can be attached to a marker. This limits the utility of markers for robust AOP, and could potentially limit is use for advanced performance tools (or at least make it harder to implement those performance tools).

For details on using these trace markers in the Linux kernel, see Documentation/Markers.txt in the kernel source.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s