OpenSource

High-performance embedded DBMS libmdbx 0.10.4 and libfpta 0.3.9 released

Pinterest LinkedIn Tumblr

The libmdbx 0.10.4 (MDBX) libraries have been released with the implementation of a high-performance compact embedded database of the key-value class, and the linked library libfpta 0.3.9 (FPTA), which implements a table view of data with secondary and composite indexes on top of MDBX. Both libraries are distributed under OSI approved licenses . All current operating systems and architectures are supported, as well as the Russian Elbrus 2000.

Historically, libmdbx is a deep reworking of the LMDB DBMS and surpasses its predecessor in reliability, feature set, and performance. Compared to LMDB, libmdbx places a lot of emphasis on code quality, API stability, testing, and automated checks. A utility for checking the integrity of the database structure is supplied with some recovery options.

Technologically libmdbx offers ACID, strict change serialization, and non-blocking reads with linear scaling across CPU cores. It supports auto-compactification, automatic database size control, and range query estimation. Since 2016, projects have been funded by Positive Technologies and have been used in its products since 2017 .

For libmdbx, a C ++ API is offered , as well as bindings supported by enthusiasts for Rust, Haskell, Python, NodeJS, Ruby, Go, Nim. For libfpta, only the API description in the form of a C / C ++ header file is publicly available .

The main innovations, improvements and fixes added after the previous news from May 9:

  • Provided the ability for reproducible assemblies.
  • Fixed a bug due to which, under a very rare set of circumstances, a loop / hang could occur during the commit of a transaction. The problem was identified by Positive Technologies specialists during internal testing of their own products.
  • Tests have been improved and test scripts have been extended to check all reachable non-isomorphic states of the page tree and GC content inside the database.
  • In the C ++ API corrected once “noexcept”, added the additional overload for method “cursor :: erase ()”, the implementation of the buffers is relieved by the use of “std :: string” to ensure alignment (actual for CLANG libstdc ++).
  • Fixed regression in the dirty page spilling algorithm (selective eviction of modified database pages) that manifested itself in a rare unexpected error MDBX_PROBLEM when changing data in huge transactions.
  • A phasing check was carried out with the addition of a number of checks to ensure stability in case of deliberate damage to the database.
  • Fixed minor UndefinedBehaviorSanitizer and Coverity Scan issues.
  • Fixed checking of outdated and already unused internal flag “P_DIRTY” in nested pages inside database images created by old versions of the library.
  • In CMake scripts, the search for compiler components required for LTO (link-time optimization) has been improved.
  • The maximum number of concurrent readers has been increased to 32,767.
  • Improved performance when using Valgrind and AddressSanitizer.
  • Fixed recursive use of SRW-lock on Windows when working in MDBX_NOTLS mode (without using thread local storage), fixed bootid generation in case of system time change, improved detection of WSL1 and WSL2, added the ability to open a Plan 9 database mounted using DrvFS.
  • In total, more than 160 changes were made to 57 files, ~ 5000 lines added, ~ 2500 removed.

I would also like to thank the Erigon project team (Ethereum ecosystem) for their help in testing in extreme use cases . It is significant that in the five months since the release of libmdbx v0.10.0, with a database volume of 1-2 TB in each Erigon installation (used by 7% of Ethereum nodes), only three reports of database damage have been received, all were due to external reasons, and not software errors: in two cases, the cause was RAM failures, in the third, a data zeroing error in a specific storage subsystem configuration using BTRFS.

Write A Comment