The CLIP OS kernel is based on Linux. It also integrates:
- existing hardening patches that are not upstream yet and that we consider relevant to our security model;
- developments made for previous CLIP OS versions that we have not upstreamed yet (or that cannot be);
- entirely new functionalities that have not been upstreamed yet (or that cannot be).
As the core of a hardened operating system, the CLIP OS kernel is particularly responsible for:
- providing robust security mechanisms to higher levels of the operating system, such as reliable isolation primitives;
- maintaining maximal trust in hardware resources;
- guaranteeing its own protection against various threats.
In this section we discuss our security-relevant configuration choices for the CLIP OS kernel. Before starting, it is worth mentioning that:
We do our best to limit the number of kernel modules.
In other words, as many modules as possible should be built-in. Modules are only used when needed either for the initramfs or to ease the automation of the deployment of CLIP OS on multiple different machines (for the moment, we only target a QEMU-KVM guest). This is particularly important as module loading is disabled after CLIP OS startup.
We focus on a secure configuration. The remaining of the configuration is minimal and it is your job to tune it for your machines and use cases.
CLIP OS only supports the x86-64 architecture for now.
Running 32-bit programs is voluntarily unsupported. Should you change that in your custom kernel, keep in mind that it requires further attention when configuring it (e.g., ensure that
Many options that are not useful to us are disabled in order to cut attack surface. As they are not all detailed below, please see
src/portage/clip/sys-kernel/clipos-kernel/files/config.d/blacklistfor an exhaustive list of the ones we explicitly disable.
CLIP OS will need the auditing infrastructure.
We do not need
.configto be available at runtime.
Symbols are only useful for debug and attack purposes.
This unlocks additional configuration options we need.
User namespaces can be useful for some use cases but even more to an attacker. We choose to disable them for the moment, but we could also enable them and use the
kernel.unprivileged_userns_clonesysctl provided by linux-hardened to disable their unprivileged use.
Allow allocator validation checking to be enabled.
Merging SLAB caches can make heap exploitation easier.
Randomize allocator freelists
Harden slab metadata
Add various little checks to harden the slab allocator. [linux-hardened]
Place canaries at the end of slab allocations. [linux-hardened]
Zero-fill slab allocations on free to reduce risks of information leaks and help mitigate use-after-free vulnerabilities. [linux-hardened]
Verify that newly allocated slab allocations are zeroed to detect write-after-free bugs. [linux-hardened]
Enabling this would disable brk ASLR.
Enable GCC plugins, some of which are security-relevant; GCC 4.7 at least is required.
Instrument some kernel code to gather additional (but not cryptographically secure) entropy at boot time.
Prevent potential information leakage by forcing initialization of structures containing userspace addresses. This is particularly important to prevent trivial bypassing of KASLR.
Extend forced initialization to all local structures that have their address taken at any point.
Randomize layout of sensitive kernel structures. Exploits targeting such structures then require an additional information leak vulnerability.
Do not weaken structure randomization
Use maximum number of randomized bits for the mmap base address on x86_64. Note that thanks to a linux-hardened patch, this also impacts the number of randomized bits for the stack base address.
-fstack-protector-strongfor best stack canary coverage; GCC 4.9 at least is required.
Virtually-mapped stacks benefit from guard pages, thus making kernel stack overflows harder to exploit.
Do extensive checks on reference counting to prevent use-after-free conditions. Without this option, on x86, there already is a fast assembly-based protection based on the PaX implementation but it does not cover all cases.
Enforce strict memory mappings permissions for loadable kernel modules.
Although CLIP OS stores kernel modules in a read-only rootfs whose integrity is guaranteed by dm-verity, we still enable and enforce module signing as an additional layer of security:
This option requires compiler support for
-fsanitize=local-init, which is only available in Clang. [linux-hardened]
Processor type and features¶
Retpolines are needed to protect against Spectre v2. GCC 7.3.0 or higher is required.
The vsyscall table is not required anymore by libc and is a fixed-position potential source of ROP gadgets.
Needed to benefit from microcode updates and thus security fixes (e.g., additional Intel pseudo-MSRs to be used by the kernel as a mitigation for various speculative execution vulnerabilities).
See above explanation about
Enabling this feature can make cache side-channel attacks such as FLUSH+RELOAD much easier to carry out.
This should in particular be non-zero to prevent the exploitation of kernel NULL pointer bugs.
Memory Type Range Registers can make speculative execution bugs a bit harder to exploit.
Page Attribute Tables are the modern equivalents of MTRRs, which we described above.
Enable the RDRAND instruction to benefit from a secure hardware RNG if supported. See
CONFIG_RANDOM_TRUST_CPUfor warnings about that.
Enable Supervisor Mode Access Prevention to prevent ret2usr exploitation techniques.
Enable User Mode Instruction Prevention. Note that hardware supporting this feature is not common yet.
Intel Memory Protection Extensions add hardware assistance to memory protection. Compiler support is required but is deprecated in GCC 8 and will probably be dropped in GCC 9.
Memory Protection Keys are a promising feature but they are still not supported on current hardware.
Enable the seccomp BPF userspace API for syscall attack surface reduction:
While this may be seen as a controversial feature, it makes sense for CLIP OS. Indeed, KASLR may be defeated thanks to the kernel interfaces that are available to an attacker, or through attacks leveraging hardware vulnerabilities such as speculative and out-of-order execution ones. However, CLIP OS follows the defense in depth principle and an attack surface reduction approach. Thus, the following points make KASLR relevant in the CLIP OS kernel:
- KASLR was initially designed to counter remote attacks but the strong security model of CLIP OS (e.g., no sysfs mounts in most containers, minimal procfs, no arbitrary code execution) makes a local attack more complex to carry out.
- STRUCTLEAK, STACKLEAK, kptr_restrict and
CONFIG_SECURITY_DMESG_RESTRICTare enabled in CLIP OS.
- The CLIP OS kernel is custom-compiled (at least for a given deployment), its image is unreadable to all users including privileged ones and updates are end-to-end encrypted. This makes both the content and addresses of the kernel image secret. Note that, however, the production kernel image is currently part of an EFI binary and is not encrypted, causing it to be accessible to a physical attacker. This will change in the future as we will only use the kernel included in the EFI binary to boot and then kexec to the real production kernel whose image will be located on an encrypted disk partition.
- We enable
CONFIG_PANIC_ON_OOPSby default so that the kernel cannot recover from failed exploit attempts, thus preventing any brute forcing.
- We enable Kernel Page Table Isolation, mitigating Meltdown and potential other hardware information leakage. Variante 3a (Rogue System Register Read) however remains an important threat to KASLR.
Most of the above explanations stand for that feature.
kexec()system call to prevent an already-root attacker from rebooting on an untrusted kernel.
A crash dump can potentially provide an attacker with useful information. However we disabled
kexec()syscalls above thus this configuration option should have no impact anyway.
This is not supposed to be needed by userspace applications and only increases the kernel attack surface.
Power management and ACPI options¶
The CLIP OS swap partition is encrypted with an ephemeral key and thus cannot support suspend to disk.
Executable file formats / Emulations¶
We do not want our kernel to support miscellaneous binary classes. ELF binaries and interpreted scripts starting with a shebang are enough.
Core dumps can provide an attacker with useful information.
Enable TCP syncookies.
TPM use is not supported by CLIP OS yet.
/dev/memdevice should not be required by any user application nowadays.
If you must enable it, at least enable
CONFIG_IO_STRICT_DEVMEMto restrict at best access to this device.
This virtual device is only useful for debug purposes and is very dangerous as it allows direct kernel memory writing (particularly useful for rootkits).
Use the modern PTY interface only.
/dev/portdevice should not be used anymore by userspace, and it could increase the kernel attack surface.
Do not rely exclusively on the hardware RNG provided by the CPU manufacturer to initialize Linux’s CRNG, as we do not mind blocking a bit more at boot time while additional entropy sources are mixed in.
The IOMMU allows for protecting the system’s main memory from arbitrary accesses from devices (e.g., DMA attacks). Note that this is related to hardware features. On a recent Intel machine, we enable the following:
Enabling this would provide an attacker with precious information on the running kernel.
This should only be needed for debugging.
This is useful even in a production kernel to enable further configuration options that have security benefits.
Enable sanity checks in virtual to page code.
Ensure kernel page tables have strict permissions.
Check and report any dangerous memory mapping permissions, i.e., both writable and executable kernel pages.
The debugfs virtual file system is only useful for debugging and protecting it would require additional work.
slub_debugcommand line parameter provides more fine grained control.
Prevent potential further exploitation of a bug by immediately panicking the kernel.
The following options add additional checks and validation for various commonly targeted kernel structures:
Note that linux-hardened patches add more places where this configuration option has an impact.
We choose to poison pages with zeroes and thus prefer using the simpler PaX-based implementation provided by linux-hardened (see
Prevent unprivileged users from gathering information from the kernel log buffer via
dmesg(8). Note that this still can be overridden through the
Enable KPTI to prevent Meltdown attacks and, more generally, reduce the number of hardware side channels.
CLIP OS does not use Intel Trusted Execution Technology.
Harden data copies between kernel and user spaces, preventing classes of heap overflow exploits and information leaks.
Use strict whitelisting mode, i.e., do not
Leverage compiler to detect buffer overflows.
FORTIFY_SOURCEto intra-object overflow checking. It is useful to find bugs but not recommended for a production kernel yet. [linux-hardened]
This feature requires userspace support that is not currently present in CLIP OS.
Enable us to choose different security modules.
CLIP OS intends to leverage SELinux in its security model.
We do not need SELinux to be disableable.
We do not want SELinux to be disabled. In addition, this would prevent LSM structures such as security hooks from being marked as read-only.
For now, but will eventually be
The default security module will be changed to SELinux once CLIP OS fully uses it.
The Yama LSM currently provides ptrace scope restriction (which might be redundant with CLIP-LSM in the future).
The integrity subsystem provides several components, the security benefits of which are already enforced by CLIP OS (e.g., read-only mounts for all parts of the system containing executable programs).
See documentation about the
kernel.perf_event_paranoidsysctl below. [linux-hardened]
Zero-fill page allocations on free to reduce risks of information leaks and help mitigate a subset of use-after-free vulnerabilities. This is a simpler equivalent to upstream’s
Verify that newly allocated pages are zeroed to detect write-after-free bugs. [linux-hardened]
This prevents unprivileged users from using the TIOCSTI ioctl to inject commands into other processes which share a tty session. [linux-hardened]
We incorporated most of the Lockdown patch series into the CLIP OS kernel, though it may be merged into the mainline kernel in the near future. Basically, Lockdown tries to disable many mechanisms that could allow the superuser to eventually run untrusted code in kernel mode (note that a significant portion of them are already disabled in the CLIP OS kernel due to our custom configuration). This is an interesting work for CLIP OS as we want to avoid persistence on a compromised machine even in the case of an already-root attacker. Among the several configuration options brought by Lockdown, we enable the following ones:
Similarly, we incorporated the STACKLEAK feature ported from grsecurity/PaX
by Alexander Popov and which should be merged upstream ultimately. STACKLEAK
erases the kernel stack before returning from system calls in order to reduce
the information which kernel stack leak bugs can reveal. It also blocks kernel
stack depth overflows caused by
alloca(), such as Stack Clash attacks.
GCC version 7.3.0 or higher is required to fully benefit from retpolines
Sysctl Security Tuning¶
Many sysctls are not security-relevant or only play a role if some kernel configuration options are enabled/disabled. In other words, the following is tightly related to the CLIP OS kernel configuration detailed above.
kernel.kptr_restrict = 2
Hide kernel addresses in
/procand other interfaces, even to privileged users.
kernel.yama.ptrace_scope = 3
Enable the strictest ptrace scope restriction provided by the Yama LSM.
kernel.perf_event_paranoid = 3
This completely disallows unprivileged access to the
perf_event_open()system call. Note that this requires a patch included in linux-hardened (see here for the reason why it is not upstream), otherwise it is the same as setting this sysctl to
2. This is actually not needed as we already enable
kernel.tiocsti_restrict = 1
This is already forced by the
CONFIG_SECURITY_TIOCSTI_RESTRICTkernel configuration option that we enable.
The following two sysctls help mitigating TOCTOU vulnerabilities by preventing users from creating symbolic or hard links to files they do not own or have read/write access to:
fs.protected_symlinks = 1
fs.protected_hardlinks = 1
In addition, the following other two sysctls impose restrictions on the opening of FIFOs and regular files in order to make similar spoofing attacks harder:
fs.protected_fifos = 2
fs.protected_regular = 2
We do not simply disable the BPF Just in Time compiler as CLIP OS plans on using it:
kernel.unprivileged_bpf_disabled = 1
Prevent unprivileged users from using BPF.
net.core.bpf_jit_harden = 2
Trades off performance but helps mitigate JIT spraying.
kernel.deny_new_usb = 0
The management of USB devices is handled at a higher level by CLIP OS. [linux-hardened]
kernel.device_sidechannel_restrict = 1
Restrict device timing side channels. [linux-hardened]
fs.suid_dumpable = 0
Do not create core dumps of setuid executables. Note that we already disable all core dumps by setting
kernel.pid_max = 65536
Increase the space for PID values.
kernel.modules_disabled = 1
Disable module loading once systemd has loaded the ones required for the running machine according to a profile (i.e., a predefined and hardware-specific list of modules).
Pure network sysctls (
net.ipv6.*) will be detailed in a
Command line parameters¶
We pass the following command line parameters to the kernel:
This parameter provided by a linux-hardened patch (based on the PaX implementation) enables a very simple form of latent entropy extracted during system start-up and added to the entropy obtained with
This force-enables KPTI even on CPUs claiming to be safe from Meltdown.
Same reasoning as above but for the Spectre v2 vulnerability.
Same reasoning as above but for the Spectre v4 vulnerability. Note that this mitigation requires updated microcode for Intel processors.
Even if we correctly enable the IOMMU in the kernel configuration, the kernel can still decide for various reasons to not initialize it at boot. Therefore, we force it with this parameter. Note that with some Intel chipsets, you may need to add
intel_iommu=igfx_offto allow your GPU to access the physical memory directly without going through the DMA Remapping.
Foption adds many sanity checks to various slab operations. Other interesting options that we considered but eventually chose to not use are:
Poption, which enables poisoning on slab cache allocations, disables the
SLAB_SANITIZE_VERIFYfeatures from linux-hardened. As they respectively poison with zeroes on object freeing and check the zeroing on object allocations, we prefer enabling them instead of using
Zoption enables red zoning, i.e., it adds extra areas around slab objects that detect when one is overwritten past its real size. This can help detect overflows but we already rely on
SLAB_CANARYprovided by linux-hardened. A canary is much better than a simple red zone as it is supposed to be random.
Also, note that:
slub_nomergeis not used as we already set
CONFIG_SLAB_MERGE_DEFAULT=nin the kernel configuration.
page_poisonis not needed by the page poisoning implementation provided by linux-hardened patches.
l1tf: The built-in PTE Inversion mitigation is sufficient to mitigate the L1TF vulnerability as long as CLIP OS is not used as an hypervisor with untrusted guest VMs. If it were to be someday,
l1tf=full,forceshould be used to force-enable VMX unconditional cache flushes and force-disable SMT (note that an Intel microcode update is not required for this mitigation to work but improves performance by providing a way to invalidate caches with a finer granularity).
Citations and origin of some items
|[linux-hardened]||(1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12) This item is provided by the |