Memory locking behavior issues

This is nothing new, and it’s strictly what the POSIX specification warns about mlock() & munlock(). As you may already know, mlock() locks memory to prevent it from being swapped to disk (for example, if you require cryptographic secrets such as encryption keys to be memory resident during system stress, preventing resilience on disk and other media). munlock() does exactly the opposite: it unlocks memory.

The problem is that both functions don’t necessarily work in the same manner across different implementations. The address parameter to both might be required to be page-aligned (rounded up to the size of a memory page, for example 4096 bytes in x86).

What happens if we supply a non-page aligned memory address? If the implementation rounds up by default, we will be either locking a whole page or unlocking it, if we use mlock() or munlock() respectively. That means all the memory contents within the same page will be affected. This might not be an issue during locking, but when you are unlocking, it’s a different situation… we might expose data that was supposed to remain locked and compromise other secrets.

The only solution to this issue is to have a consensus between vendors and implement the same behavior. That, or design and develop your own secure memory pool !

To illustrate this post, you can see below the implementation for Mac OS X Leopard (10.5):

972 int
973 mlock(__unused proc_t p, struct mlock_args *uap, __unused register_t *retvalval)
974 {
975         vm_map_t user_map;
976         vm_map_offset_t addr;
977         vm_map_size_t size, pageoff;
978         kern_return_t   result;
979
--
982
983         addr = (vm_map_offset_t) uap->addr;
984         size = (vm_map_size_t)uap->len;
985
986         /* disable wrap around */
987         if (addr + size < addr)
988                 return (EINVAL);
989
990         if (size == 0)
991                 return (0);
992
993         pageoff = (addr & PAGE_MASK);
994         addr -= pageoff;
995         size = vm_map_round_page(size+pageoff);
996         user_map = current_map();