Introduction
The kernel is the primary part of any operating system and has complete control over all processes and memory. Replacing kernel code may lead to drastic outcomes, including a crash and a state, where it is impossible to restore the system. Despite these risks, antivirus designers often use kernel-mode mechanisms to enable self-defense. On 64-bit versions, however, kernel patching was disabled due to security concerns, and antivirus vendors have to look for alternative methods.
Overview
Windows supports a wide variety of software, and providing all applications with full control over the system would not be safe. Therefore, the developers of the product designed two access levels to separate user applications from system software. In user mode, software programs are given virtual address space, which is the only address array they can manipulate (User mode and kernel mode, 2017). Under such restrictions, applications do not have the capacity to affect the functioning of other software, including critical operating system processes. When a user-mode program crashes, only the process associated with this application stops (User mode and kernel mode, 2017). In kernel mode, however, there is only one virtual address space that is shared by all kernel software, including drivers and the kernel itself (User mode and kernel mode, 2017). Such an elevated control over the operating system may be necessary for some circumstances, but there is an implication to consider. A crash of kernel-mode application leads to the stop of the whole operating system, an event that is usually called the Blue Screen of Death (BSOD).
Benefits of Kernel Mode
An antivirus’s primary task is to protect the computer system from external threats. It can prevent malicious software from being installed by scanning each file that is downloaded from the internet. Modern malware, however, is sophisticated, and before initiating any attacks and concealing itself from the antivirus, it first tries to disable the security component of the computer (Mohanta & Saldanha, 2020). Therefore, it is essential for antivirus software to be able to protect itself, or it will be rendered useless. Designers of security software often write software that runs in kernel mode; that is, it calls kernel commands directly and has access to the whole memory of the system (Mohanta & Saldanha, 2020). While it is dangerous to operate in kernel mode as it may lead to the crash of the whole system, it is often necessary to make the antivirus a kernel software. Malware could also be running in kernel mode and may try to stop the antivirus process (Mohanta & Saldanha, 2020). After successfully halting the security software, viruses may even try to uninstall it from the computer.
A specific example is the usage of WinAPI calls to terminate a process. In user-mode, the command NtTerminateProcess is called when an application process is asked to be stopped (Mohanta & Saldanha, 2020). If an antivirus is requested to be stopped using this method, generally, callers receive a denied access status. Kernel-mode rootkits, however, do not use user-mode WinAPI commands; instead, they have privileged access and can stop any process without any restriction using ZwTerminateProcess (Mohanta & Saldanha, 2020). For an antivirus to be able to protect itself from such attacks, it has to operate at a kernel-level too. It has to continuously scan what commands are being executed and what memory addresses are being accessed. In case a program wants to halt the antivirus process, the security software immediately detects what component initiated the call and can label it as malware and proceed with either deleting it or quarantining it. This method is called WinAPI hooking through kernel patching and had been supported by many antivirus vendors until Microsoft decided to implement kernel patch protection (Mohanta & Saldanha, 2020). Hooking user-mode calls are currently the only way of providing some level of self-defense without affecting the kernel.
Alternative Methods
With the advent of 64-bit Windows versions, there has been a significant change in how antiviruses ensure self-protection. Microsoft has not ever supported kernel patching because the company believes that it leads to security flaws. When antiviruses inject their code into the kernel, the latter is essentially replaced by a third-party code that has not been tested to operate efficiently without causing any damage to the rest of the system. While there are many benefits of writing kernel-mode self-protection mechanisms, it often leads to BSOD.
With kernel patching protection developed by Microsoft, antivirus vendors can no longer make their software operate in kernel mode. An alternative way of ensuring security is hooking to user-mode WinAPI calls (Mohanta & Saldanha, 2020). However, this method is not effective because security researchers have established that patch protection mechanisms can be circumvented by malware developers (Bölük, 2019). In other words, it would be unethical for antivirus designers to make their programs avoid kernel patch protection even if they could, but hackers rarely wish to behave in an ethical manner. Therefore, despite providing some level of protection against common malicious software, user-mode hooking is not effective against kernel-mode rootkits.
Conclusion
Executing some of the antivirus components in kernel-mode can allow the security mechanism to prevent malicious software from tampering with antivirus files and processes. This self-protection, however, comes at a cost – kernel-mode execution increases the risk of system crashes and data loss. Microsoft has disallowed antivirus developers from patching the kernel since the emergence of 64-bit versions of its operating system. Today, antivirus designers may use user-mode hooking, but it is effective against kernel-mode rootkits.
References
Bölük, Can. (2019). ByePg: Defeating patchguard using exception-hooking. GitHub.
Mohanta, A., & Saldanha, A. (2020). Malware analysis and detection engineering. Apress.
User mode and kernel mode. (2017). Microsoft.