I needed to add a custom System Request (Sys Req or SysRq) to a linux kernel some time ago. While doing so, I dug deep into how it works and I thought I’d make a quick post about it. Here is a good SuperUser answer about what a SysRq is. You may also know about SysRq via REISUB. This post has three parts: how to raise a SysRq, how SysRq works (looking into kernel code), and how to add your own SysRq.
Raising a SysRq
A SysRq can be raised in a few ways. One of these is by triggering the “Alt +
SysRq + CHAR” combination, where CHAR represents a character (like ‘h’ or ‘r’).
The other is by writing CHAR to /proc/sysrq-trigger. For example, echo h > /proc/sysrq-trigger.
In either case, doing this will start “triggering” code. The output of “Alt + SysRq + h” spits out a help message in your tty (or check dmesg) as shown in Figure 1 below:
Figure 1. The output of echo h > /proc/sysrq-trigger. The kernel replies with "sysrq: HELP" with many options; some of them are reboot (b), crash (c), poweroff (o), and unraw (r).
The Functioning of SysRq
In this section, I’ll be posting a lot of code from the Linux 6.9.3 kernel. If you don’t understand much of the code, that’s okay because the code is complicated unless you have a big picture. A simplified picture is shown in the form of a flowchart below in Figure 2.
Figure 2. A simplified flowchart for how SysRq works.
Any one of the two types of inputs, echo CHAR > /proc/sysrq-trigger or “Alt +
SysRq + CHAR” trigger the __handle_sysrq(CHAR) function. This function then
looks through the sysrq_key_table to see what to run, the result of which is a
handler, which I depict in Figure 2 as sysrq_CHAR_op.handler. This handler is
executed after the lookup.
Let’s take a look at an excerpt of SysRq setup when the kernel boots. This code is from /drivers/tty/sysrq.c and is shown in Listing 1 below.
/* /drivers/tty/sysrq.c: Line numbers correspond to kernel v6.9.3 */
#ifdef CONFIG_PROC_FS
/*
* writing 'C' to /proc/sysrq-trigger is like sysrq-C
* Normally, only the first character written is processed.
* However, if the first character is an underscore,
* all characters are processed.
*/
static ssize_t write_sysrq_trigger(struct file *file, const char __user *buf,
size_t count, loff_t *ppos)
{
bool bulk = false;
size_t i;
for (i = 0; i < count; i++) {
char c;
if (get_user(c, buf + i))
return -EFAULT;
if (c == '_')
bulk = true;
else
__handle_sysrq(c, false);
if (!bulk)
break;
}
return count;
}
static const struct proc_ops sysrq_trigger_proc_ops = {
.proc_write = write_sysrq_trigger,
.proc_lseek = noop_llseek,
};
static void sysrq_init_procfs(void)
{
if (!proc_create("sysrq-trigger", S_IWUSR, NULL,
&sysrq_trigger_proc_ops))
pr_err("Failed to register proc interface\n");
}
#else
static inline void sysrq_init_procfs(void)
{
}
#endif /* CONFIG_PROC_FS */
static int __init sysrq_init(void)
{
sysrq_init_procfs();
if (sysrq_on())
sysrq_register_handler();
return 0;
}
device_initcall(sysrq_init);
Listing 1. SysRq setup in the kernel.
In line 1210, we have device_initcall1. Initcalls are used by the kernel to call
functions when the kernel boots. The device_initcall (line 1210) instructs the
kernel to run sysrq_init() (line 1201). sysrq_init() calls two functions.
First, sysrq_init() calls sysrq_init_procfs()2 (line 1186) which creates /proc/sysrq-trigger
via proc_create() (line 1188). One of the arguments of proc_create() is
sysrq_trigger_proc_ops, a structure variable of type proc_ops (line 1181).
The .proc_write field lets the kernel know that all data written to
/proc/sysrq-trigger is to be directed to write_sysrq_trigger() (line 1157).
The main gist of write_sysrq_trigger() is that it calls __handle_sysrq()
(line 1172), which we’ll get to in a minute.
The second function sysrq_init() calls is sysrq_register_handler()3, whose code at line
1033 in Listing 2 below.
/* /drivers/tty/sysrq.c: Line numbers correspond to kernel v6.9.3 */
static struct input_handler sysrq_handler = {
.filter = sysrq_filter,
.connect = sysrq_connect,
.disconnect = sysrq_disconnect,
.name = "sysrq",
.id_table = sysrq_ids,
};
static inline void sysrq_register_handler(void)
{
int error;
sysrq_of_get_keyreset_config();
error = input_register_handler(&sysrq_handler);
if (error)
pr_err("Failed to register input handler, error %d", error);
}
static inline void sysrq_unregister_handler(void)
{
input_unregister_handler(&sysrq_handler);
}
Listing 2. Registering sysrq_handler as an input_handler
Notice in line 1039, sysrq_handler (from line 1025) is being registered as an
input_handler. The specifics of input_handler will not be looked into4, but a simplified
explanation is that sysrq_filter checks if the SysRq key combination is being
pressed via the keyboard and if so, it also calls __handle_sysrq(), similar to
write_sysrq_trigger() from Listing 1.
Now, we won’t take too deep of a look into __handle_sysrq() because the code
is somewhat chonky. The gist of it is pretty simple: __handle_sysrq() takes
two arguments, the first of which is key and the second of which is
check_mask. We won’t go into too much detail about the mask5 as the important
argument is key — this is character that’s pressed or the character sent to
/proc/sysrq-trigger. If you pressed “Alt + SysRq + h” (for help), the key is
“h”. __handle_sysrq() looks through sysrq_key_table to determine what must
be executed when “h” is pressed. The sysrq_key_table is shown in Listing 3
below. It defines what happens for every character. The character it’s linked to
is in the comment to the right of each member of the table.
/* /drivers/tty/sysrq.c: Line numbers correspond to kernel v6.9.3 */
static const struct sysrq_key_op *sysrq_key_table[62] = {
&sysrq_loglevel_op, /* 0 */
&sysrq_loglevel_op, /* 1 */
&sysrq_loglevel_op, /* 2 */
&sysrq_loglevel_op, /* 3 */
&sysrq_loglevel_op, /* 4 */
&sysrq_loglevel_op, /* 5 */
&sysrq_loglevel_op, /* 6 */
&sysrq_loglevel_op, /* 7 */
&sysrq_loglevel_op, /* 8 */
&sysrq_loglevel_op, /* 9 */
/*
* a: Don't use for system provided sysrqs, it is handled specially on
* sparc and will never arrive.
*/
NULL, /* a */
&sysrq_reboot_op, /* b */
&sysrq_crash_op, /* c */
&sysrq_showlocks_op, /* d */
&sysrq_term_op, /* e */
&sysrq_moom_op, /* f */
/* g: May be registered for the kernel debugger */
NULL, /* g */
NULL, /* h - reserved for help */
&sysrq_kill_op, /* i */
&sysrq_thaw_op, /* j */
&sysrq_SAK_op, /* k */
&sysrq_showallcpus_op, /* l */
&sysrq_showmem_op, /* m */
&sysrq_unrt_op, /* n */
/* o: This will often be registered as 'Off' at init time */
NULL, /* o */
&sysrq_showregs_op, /* p */
&sysrq_show_timers_op, /* q */
&sysrq_unraw_op, /* r */
&sysrq_sync_op, /* s */
&sysrq_showstate_op, /* t */
&sysrq_mountro_op, /* u */
/* v: May be registered for frame buffer console restore */
NULL, /* v */
&sysrq_showstate_blocked_op, /* w */
/* x: May be registered on mips for TLB dump */
/* x: May be registered on ppc/powerpc for xmon */
/* x: May be registered on sparc64 for global PMU dump */
NULL, /* x */
/* y: May be registered on sparc64 for global register dump */
NULL, /* y */
&sysrq_ftrace_dump_op, /* z */
NULL, /* A */
NULL, /* B */
NULL, /* C */
NULL, /* D */
NULL, /* E */
NULL, /* F */
NULL, /* G */
NULL, /* H */
NULL, /* I */
NULL, /* J */
NULL, /* K */
NULL, /* L */
NULL, /* M */
NULL, /* N */
NULL, /* O */
NULL, /* P */
NULL, /* Q */
NULL, /* R */
NULL, /* S */
NULL, /* T */
NULL, /* U */
NULL, /* V */
NULL, /* W */
NULL, /* X */
NULL, /* Y */
NULL, /* Z */
};
Listing 3. The sysrq_key_table is a table that maps a case-sensitive character to a sysrq_key_op structure variable (shown below in Listing 4).
Each of these entries is a structure variable of type sysrq_key_op. The
sysrq_key_op for the character “b” (reboot) is shown in Listing 4 below. The
structure variable sysrq_reboot_op (line 169) has four fields: handler,
help_msg, action_msg, and enable_mask. The handler is the function that
does the “running” of the SysRq combination code. In sysrq_reboot_op, the
handler is set to sysrq_handle_reboot() (line 163). This is the function
that’s executed when the sysrq key combination “Alt + SysRq + b” is pressed or
echo b > /proc/sysrq-trigger is executed. The help_msg is shown when SysRq-h
is called, the action_msg is shown when the SysRq is executed, and the
enable_mask is a mask that determines whether that SysRq combination is
enabled6.
/* /drivers/tty/sysrq.c: Line numbers correspond to kernel v6.9.3 */
static void sysrq_handle_reboot(u8 key)
{
lockdep_off();
local_irq_enable();
emergency_restart();
}
static const struct sysrq_key_op sysrq_reboot_op = {
.handler = sysrq_handle_reboot,
.help_msg = "reboot(b)",
.action_msg = "Resetting",
.enable_mask = SYSRQ_ENABLE_BOOT,
};
Listing 4. The structure variable sysrq_reboot_op sets sysrq_handle_reboot as its handler.
Adding Your Own SysRq
Now that we’ve gone through how SysRq functions, you can guess that adding your
own SysRq function requires modifying the sysrq_key_table to include a
sysrq_key_op structure variable. This would mean recompiling the kernel…
which everybody LoVeS doing. When I was working on adding my own SysRq function,
I was working on a custom kernel that was constantly being compiled anyway. I
simply added my code to the /drivers/tty/sysrq.c and called it a day.
You probably won’t want to do this. The much more appropriate method is to add your own custom SysRq function via a kernel module — you can find the instructions for that in the kernel documentation.
Here is a simple kernel module that reminds you to drink coffee when you hit “Alt + Shift + D” (capital D for DRINK). The result of the module is shown in Figure 3, below the code.
#include <linux/kernel.h>
#include <linux/init.h>
#include <linux/module.h>
#include <linux/sysrq.h>
MODULE_DESCRIPTION("An official SysRq mandate to remind you to drink coffee");
MODULE_AUTHOR("A drinker of coffee");
MODULE_LICENSE("GPL");
#define MOD_PREFIX "[Drink Coffee] "
static int is_sysrq_registered = 0;
static void sysrq_handle_drink_coffee(int key)
{
// Don't use pr_emerg unless it's an emergency. In our case, drinking
// coffee is an emergency.
pr_emerg(MOD_PREFIX "The kernel suggests you drink some coffee\n");
pr_emerg(MOD_PREFIX "P.S.: Don't drink tea, it's ew\n");
}
static const struct sysrq_key_op sysrq_drink_coffee_op = {
.handler = sysrq_handle_drink_coffee,
.help_msg = "drink coffee(D)",
.action_msg = "Drink Coffee"
};
static int drink_coffee_sysrq_init(void)
{
pr_debug(MOD_PREFIX " Initializing\n");
#ifdef CONFIG_MAGIC_SYSRQ
is_sysrq_registered = !register_sysrq_key('D', &sysrq_drink_coffee_op);
pr_debug(MOD_PREFIX "SysRq Registered: %d\n", is_sysrq_registered);
#endif
return 0;
}
static void drink_coffee_sysrq_exit(void)
{
#ifdef CONFIG_MAGIC_SYSRQ
pr_debug(MOD_PREFIX "Unregistering SysRq\n");
if (is_sysrq_registered)
unregister_sysrq_key('D', &sysrq_drink_coffee_op);
#endif
pr_debug(MOD_PREFIX "Completed!\n");
}
module_init(drink_coffee_sysrq_init);
module_exit(drink_coffee_sysrq_exit);
After compiling this module and insmod-ing it, this is our output when we run echo D > /proc/sysrq-trigger:
Figure 3. SysRq D reminds you to drink some coffee.
This YouTube video is a nice talk: Using Serial kdb / kgdb to Debug the Linux Kernel - Douglas Anderson, Google.
1 Initcalls are fascinating. Some great resources about them are YouTube: Demystifying Linux Kernel Initcalls! | ELCE 2020 by Mylène Josserand whose corresponding webpage is Collabora: An introduction to Linux kernel initcalls, and Linux-Insides: The initcall mechanism by 0xAX (Alex Kuleshov). ↩
2 We assume that CONFIG_PROC_FS is set (corresponding
to the first #IFDEF). cat /boot/config-$(uname -r) | grep CONFIG_PROC_FS
should result as set. ↩
3 Don’t worry about the if (sysrq_on()). This only
checks if SysRq is
enabled.
↩
4 See Linux Input Subsystem kernel API and struct input_handler for more info. ↩
5 SysRq has a bitmask of allowed SysRq functions. You
can read more about it on the kernel documentation page: How do I enable the
magic SysRq
key?.
You can check your machine’s value by running cat /proc/sys/kernel/sysrq and
converting the decimal number to bits
(https://www.tfxsoft.com/playground/calc.php). ↩
6 Check footnote 5 above ↩