Scope
These are some considerations about the memfile lease backend and multi-threading.
Introduction
make memfile thread safe issue (https://gitlab.isc.org/isc-projects/kea/-/issues/894 closed) added multi-threading support (aka safety) to the memfile backend code. Performance reports as https://kb.isc.org/docs/kea-performance-tests-17-multithreading show that the introduction of multi-threading does not always bring a large speedup to the memfile lease backend.
This document tries to explain why and how this could be improved. Note the performance of the memfile backend is already high and with multi-threading is very high but perhaps there is an easy way to get even better.
As usual for performance to have profiling results is critical so Perform profiling for MT Kea issue (https://gitlab.isc.org/isc-projects/kea/-/issues/1146) should help.
Memfile design
The memfile lease backend can be divided into:
- in memory lease storage using a boost multi-index container
- stable lease storage using a CSV file
The multi-threading safety is performed by a global C++11 std::mutex at the lease manager implementation level and a C++11 std::lock_guard using the global mutex for each method of the lease manager API doing a concurrent access to lease storage when the multi-threading is enabled so:
- when multi-threading is not enabled the cost is the check of the multi-threading mode
- when multi-threading is enabled there is an addition cost to acquire and release the lock and of course to wait the mutex to be available when it was taken by another thread
Possible improvements
From the performance results it is clear there is contention i.e. threads spend a lot of time to acquire the global lock.
The first idea is cut the critical section (part of the code where the lock is taken) into a part for the in-memory storage and another part for stable storage. Profiling should indicate the relative time spent in each part so where to put optimization efforts.
For the in-memory storage the boost multi-index container is not thread safe for updates. The idea to move to fine grain locks (or lock free thread safe container) is not so easy to implement: some indexes are ordered and concurrent balanced binary search tree implementation is still an active research topic according to the number of papers recently published about this (vs. unordered indexes using hash tables which is far easier to implement in a thread safe way).
The stable storage should be simpler: IMHO it is enough to make the output part of the CSVFile class thread safe. Note to protect I/O against concurrent write is far to be uncommon: not only this is done at the kernel level with the O_APPEND open flag and in the C stdio API by the flockfile function explicitly or implicitly called (and implemented with a pthread mutex).
Comparison with database lease backends
In fact what is needed is to replace the home made memfile backend by an in-memory database (aka IMDB) with persistent/stable storage. If an IMDB provides ACID properties (Kea can live with weaker but ACID / transaction is a standard for databases and includes thread safety) and good concurrent performances the solution could be to write a new lease backend for this IMDB...