MattCASmith

During any incident investigation on a Linux system, one of the most valuable things for responders and forensicators to establish is which commands were run. This is key to finding out what an attacker or malicious user was attempting to do, and what remediation activities are required.

The .bash_history file, located in each user’s home directory, is usually the investigator’s first stop for this information. This file contains a list of Bash commands recently run by the user and may appear relatively simple at first glance, but there are many ins and outs to its behaviours.

Most importantly, commands entered in a running Bash terminal are stored in memory and are only written to the file when the terminal is closed. As you can imagine, this means many caveats can be applied to the file’s contents, and in this post I aim to cover some of the most common scenarios, how they affect .bash_history, and alternatives when it does not contain the activity you are looking for.

Contents

1. .bash_history behaviours
          a. The $HISTCONTROL variable
          b. The history command
          c. Terminal windows and command order
          d. Logical AND (&&)
          e. Commands within scripts
          f. The kill command
          g. The -SIGKILL switch
          h. SSH sessions
2. Finding Bash commands in memory
3. Further reading

.bash_history behaviours

To test exactly when commands are and aren’t recorded to the user’s .bash_history file, I ran a series of tests covering common scenarios in which commands might be run. All of these tests were run on a clean installation of CentOS. Here’s what I found…

First, a note on the $HISTCONTROL variable…

It is important to check the contents of the $HISTCONTROL variable on the subject system (after taking images and preserving evidence, so you’re not overwriting .bash_history with your own commands) as this has a potentially significant bearing on which commands will be written to the file.

echo $HISTCONTROL

The command above will return one of the following strings. The table below details which commands .bash_history will ignore when each of the possible values is present.

Therefore if the system has one of these values in the $HISTCONTROL variable, there may be commands missing from .bash_history. If it is set to ignorespace and the attacker is aware, they could even just slip in a space before each of their commands and write nothing to the history file at all!

The CentOS system where I performed the testing for this post had $HISTCONTROL set to ignoredups. However, I also performed checks across other Linux distros I had available (namely Kali, Tsurugi, and Windows Subsystem for Linux) and it was set to ignoreboth for them all. So it’s important to check what you’re dealing with on the particular system you’re investigating.

It’s also trivial to change the contents of the $HISTCONTROL variable to exclude more commands from .bash_history, so watch out for evidence it has changed, which will look something like the following:

export HISTCONTROL=ignoreboth

The history command outputs a combined history from .bash_history and memory.

On a live Linux system, you might run the history command in a terminal window to review recently executed commands. The output of this command combines both the contents of .bash_history and any commands held in memory from the current session.

In the example below, we can see that the test command doesthiscommandshowinhistory? is returned by the history command on the left, but has not yet been written to disk as shown on the right.

Bear in mind that history will show commands from the memory of only the current Bash terminal - it won’t have access to those belonging to other processes. Since you won’t have access to the attacker’s terminal (unless the incident response team directly interrupted a hands-on-keyboard session), this means it won’t usually be much more useful than .bash_history itself.

Commands are written to .bash_history in the order terminals are closed, not the order they are run.

For this test, I opened two Bash terminals and ran example commands containing text stating which terminal they belonged to and in which order they were run. As I closed the first terminal before the second, you can see that both of its commands appear first in .bash_history, even though some of the second terminal’s commands were run first. It is the order the Bash processes end that matters.

As was discussed before, note that if we had run the history command in one of these terminals, it would only have returned its own commands because each process has its own space in memory.

Commands run with logical AND (&&) appear as a single entry in .bash_history.

If we string several commands together using && - a technique often used to save time (or in less innocent circumstances to avoid an attacker having to submit commands multiple times and risk attracting unwanted attention) - they will still only appear as one line in .bash_history.

This is an important detail as investigators and forensicators, as it preserves the original context in which these commands were run. If they were run back-to-back in this manner, this invites the question: “Why?”

Commands run within scripts are not reflected in .bash_history.

To see whether commands in scripts are added to .bash_history or not, I created a script called folder_test.sh that would create a new folder, navigate into it, list the contents, navigate back up a level, and then delete the folder. You can see its contents in the left-hand window below.

On the right, I ran the script with ./folder_test.sh. We know it executed successfully because the contents (i.e. nothing) are printed to the terminal. However, when we check the output of the history command we can see that although the command that ran the script was recorded, commands run within the script were not - an important distinction to remember when investigating Linux systems.

This means that our attacker could potentially download or create a script named something innocuous like file_cleanup.sh, execute it, and we would be none the wiser as to what it did based on the contents of .bash_history. There would likely be artefacts elsewhere on the system to give us a clue as to what they were trying to do, but that’s a different topic for another day.

If a terminal is stopped with the kill command, commands are still written to .bash_history.

This one was simple enough to test. I ran the command doesthiscommandappearinhistory? in a terminal, then opened a new terminal and killed it using its process ID, then checked .bash_history.

As you can see below, despite the Bash instance being killed, the command was still written to the log (the following echo $$ command was the one I used to identify the process ID of that terminal).

However, if the -SIGKILL switch is used, commands are not written to .bash_history.

More useful for red teamers - and something to bear in mind for blue teamers - is that .bash_history is not written if the terminal is killed with the -SIGKILL switch. Repeating the same experiment as above but with the additional switch meant no commands were written to disk.

This is because by default kill sends the SIGTERM signal, which gracefully kills the process and allows Bash to write to .bash_history as it is closing down. SIGKILL, on the other hand, kills the process immediately before the commands can be written to the file.

SSH sessions behave in a similar way to standard terminals.

If we run commands within an SSH session and then quit with exit or by closing the window, .bash_history is written in much the same way as it usually would be. The same also applies if we close the Command Prompt running the SSH session with Task Manager - presumably because Windows gives it time to tear down the connection in the background.

Interestingly, if connecting via SSH from a Linux terminal, running a standard kill command against the terminal will not close it until the user has ended the SSH session themselves. Using the -SIGKILL switch ends the terminal process and SSH session immediately, but still writes the commands run during the session to the user’s .bash_history file.

This is likely because this is the work of sshd, which sees its client kill the connection but itself closes gracefully. If we use kill directly on sshd itself - even with the -SIGKILL switch - commands are still written to the file, which makes me wonder whether there is actually a way around this for SSH sessions.

Finding Bash commands in memory

All the tests above showed when commands are and are not written to the on-disk .bash_history file. Now let’s see what we can do when an attacker has an active session (and therefore commands in memory) but either .bash_history has not been written yet or they have cleared its contents. This may also work for recently closed Bash terminal sessions that did not end gracefully.

The first thing we need to do is take a memory image of the Linux system - for this we can use a tool called Linux Memory Extractor (LiME). In a live incident scenario there are mechanisms to push the image to external storage or across the network, but I ran a basic command to create it locally.

sudo insmod lime-4.18.0-240.22.1.el8_3.x86_64.ko “path=/mem-img.mem format=lime”

To analyse our new memory image, we’ll use Volatility, which is currently considered the pinnacle of memory forensics toolkits. It’s not quite as simple to run Volatility against Linux memory images as it is for Windows images. I won’t go into the full process here, but you need to create your own profile for the specific Linux distro and kernel version from which the image was captured.

Once the profile is in place and ready to go, we can run Volatility’s linux_bash module with the following command to search for Bash commands held in memory.

python2 vol.py -f mem-img.mem –profile=LinuxCentOSx64 linux_bash

In the screenshot below, you can see a small extract from the results, are ordered first by the process ID of the Bash terminal to which the commands belonged and then by the time they were run. This includes commands that were run too recently to have been written to .bash_history, and towards the bottom you’ll even see the commands I ran to set up and run LiME.

One small word of warning regarding timestamps… Everything from the ls command at 15:59:18 downwards appears to be correct, but you’ll probably notice that all the commands above that allegedly ran at exactly the same second, which is obviously not right. Further investigation is needed to work out why exactly that is, but it’s likely that there is some limit to the number of Bash command timestamps stored in memory, or that Volatility cannot always read them accurately.

Further reading

The links below lead to pages that either inspired this post or provided useful information to compile it, including some more in-depth technical information on various features discussed above.

• Bash history manual page (gnu.org)
• $HISTCONTROL command in Linux with examples (geeksforgeeks.org)
• SIGINT vs SIGKILL (linuxhandbook.com)
• Volatility Linux documentation (github.com)
• How to extract memory information to spot Linux malware (crowdstrike.com)

Updated 25/02/2022 to add section on the $HISTCONTROL variable.