SystemTap has been sometimes plagued with “solution in search of a problem” complaints. It was interesting to run across an example of SystemTap being used to solve a real-world problem. A developer discovered an OOM (Out Of Memory) condition in the upstream kernel. In the quest to obtain additional information regarding the issue, the kernel refused to boot with additional debug printk()’s added to quicklist.c. The developer, being familiar with SystemTap, used the following script:

probe kernel.statement("quicklist_trim@mm/quicklist.c:56")
{
        printf(" q->nr_pages is %d, min_pages is %d ----> %s\n",
               $q->nr_pages, $min_pages, execname());
}

The SystemTap scripting language takes a little getting used to, but the intent here is clear: each time quicklist_trim() is run in quicklist.c, a message should be printed that displays some kernel data.

This sort of usage is interesting for two reasons:

  1. Kernel developers understand that there are places in the kernel that cannot be debugged by the printk() method; kprobes (simplified via SystemTap) provide a method to extract debug data from many of those locations, and this is an example of that in action.
  2. When using SystemTap to gather debug data, the system doesn’t need to be rebooted with a new kernel. In this particular scenario, that didn’t matter so much, as the problem was discovered by a developer who could modify his own kernel and reboot the system as needed. However, if an issue is discovered on a production system, this allows for root cause analysis to proceed without the need to take the system out of production.