C++ STL Map and HashMap Benchmark vs. CrossDB
Overview
This benchmark test aims to demonstrate that the performance of the CrossDB in-memory database is sufficient to serve as an alternative to STL Map/HashMap or Boost Multi-Index.
When dealing with complex relational data, significant effort may be required to design and optimize your complex lookup strategy using maps or hashmaps. Databases excel in effective query planning and indexing. If you create efficient indexes and query plans for your maps, you can achieve high performance. However, if you have complex relationships involving numerous tables, do you really want to handle all the query planning yourself?
One of the key advantages of a database is its flexibility. While bespoke solutions for known and stable data relations or structures can match or even surpass database performance, any changes would necessitate rewriting your complex lookup logic. In contrast, with a RDBMS, you can simply use UML to model and adjust your SQL queries.
This tool will use auto-commit transaction to test single CRUD performance for CrossDB Prepared STMT.
Bench Tool
Bench Test Framework
STL Map&HashMap Bench Driver
CrossDB Bench Driver
The test scripts will conduct five rounds of testing and select the average value.
crossdb/bench/basic$ ./bench-stlmap.bin -q -r 5 -n 1000
crossdb/bench/basic$ ./bench-stlmap.bin -q -r 5 -n 10000
crossdb/bench/basic$ ./bench-stlmap.bin -q -r 5 -n 100000
crossdb/bench/basic$ ./bench-stlmap.bin -q -r 5 -n 1000000
crossdb/bench/basic$ ./bench-stlmap.bin -q -r 5 -n 10000000
crossdb/bench/basic$ ./bench-crossdb.bin -q -r 5 -n 1000
crossdb/bench/basic$ ./bench-crossdb.bin -q -r 5 -n 10000
crossdb/bench/basic$ ./bench-crossdb.bin -q -r 5 -n 100000
crossdb/bench/basic$ ./bench-crossdb.bin -q -r 5 -n 1000000
crossdb/bench/basic$ ./bench-crossdb.bin -q -r 5 -n 10000000
Note
- To ensure fairness in testing, both STL Map and HashMap (unordered_map) use pthread read-write locks, as CrossDB is thread-safe with read-write locks by default.
std::shared_mutex is not used because, in a single-threaded context, the compiler optimizes the code and omits the lock.- CrossDB will support a lockless mode in the future, and the benchmark tool will offer an option to configure the lock mode.
Note
Test results will vary depending on the CPU, OS, compiler, and system load. Even on the same server, the results will differ each time.
Test Server
CPU : 11th Gen Intel(R) Core(TM) i7-11700 @ 2.50GHz, Cache size 16384 KB
OS : Ubuntu 24.04
CrossDB : 0.8.0
Compiler : gcc 13.2.0 with -O2 optimization
c++ std : c++17
In-Memory DB Test with INT
1,000 Rows
Sequential Access Test
| DB | Access | Insert QPS | Query QPS | Update QPS | Delete QPS |
| STL | Map | 10,442,591 | 21,776,205 | 27,627,912 | 19,777,677 |
| CrossDB | STMT | 17,773,584 | 28,868,027 | 22,767,213 | 23,593,441 |
| STL | HashMap | 14,284,640 | 44,081,055 | 46,764,986 | 38,358,974 |
Random Access Test
| DB | Access | Insert QPS | Query QPS | Update QPS | Delete QPS |
| STL | Map | 9,841,660 | 15,349,472 | 17,730,115 | 9,580,807 |
| CrossDB | STMT | 20,171,189 | 28,834,741 | 22,076,943 | 21,973,314 |
| STL | HashMap | 14,975,036 | 42,500,933 | 48,155,156 | 36,146,519 |
10,000 Rows
Sequential Access Test
| DB | Access | Insert QPS | Query QPS | Update QPS | Delete QPS |
| STL | Map | 8,113,054 | 16,625,624 | 19,813,725 | 22,953,185 |
| CrossDB | STMT | 14,752,474 | 28,119,037 | 22,175,662 | 22,469,934 |
| STL | HashMap | 18,570,206 | 44,608,531 | 46,079,259 | 39,167,902 |
Random Access Test
| DB | Access | Insert QPS | Query QPS | Update QPS | Delete QPS |
| STL | Map | 8,629,238 | 8,951,170 | 9,888,694 | 6,962,080 |
| CrossDB | STMT | 16,550,826 | 26,314,983 | 21,294,530 | 21,212,114 |
| STL | HashMap | 19,087,491 | 30,989,837 | 43,464,636 | 29,057,718 |
100,000 Rows
Sequential Access Test
| DB | Access | Insert QPS | Query QPS | Update QPS | Delete QPS |
| STL | Map | 5,145,771 | 15,298,169 | 17,967,132 | 22,018,252 |
| CrossDB | STMT | 19,700,788 | 28,480,655 | 21,548,692 | 23,305,607 |
| STL | HashMap | 20,788,054 | 49,810,428 | 53,449,199 | 46,489,166 |
Random Access Test
| DB | Access | Insert QPS | Query QPS | Update QPS | Delete QPS |
| STL | Map | 5,815,344 | 5,163,883 | 5,531,051 | 4,941,685 |
| CrossDB | STMT | 20,411,675 | 22,748,379 | 18,789,522 | 19,783,731 |
| STL | HashMap | 21,936,850 | 26,063,150 | 37,778,169 | 23,640,037 |
1,000,000 Rows
Sequential Access Test
| DB | Access | Insert QPS | Query QPS | Update QPS | Delete QPS |
| STL | Map | 2,674,488 | 12,008,803 | 13,194,472 | 14,606,515 |
| CrossDB | STMT | 18,477,686 | 27,987,661 | 21,844,773 | 23,105,578 |
| STL | HashMap | 20,543,816 | 48,752,370 | 49,362,932 | 45,854,590 |
Random Access Test
| DB | Access | Insert QPS | Query QPS | Update QPS | Delete QPS |
| STL | Map | 3,655,302 | 1,868,468 | 2,006,574 | 1,829,826 |
| CrossDB | STMT | 18,610,416 | 6,568,603 | 6,631,045 | 6,341,988 |
| STL | HashMap | 22,427,027 | 7,474,584 | 10,213,001 | 6,075,652 |
10,000,000 Rows
Sequential Access Test
| DB | Access | Insert QPS | Query QPS | Update QPS | Delete QPS |
| STL | Map | 1,722,853 | 7,860,464 | 7,879,262 | 9,947,688 |
| CrossDB | STMT | 17,671,148 | 26,329,139 | 20,146,554 | 20,992,326 |
| STL | HashMap | 20,558,213 | 48,890,089 | 49,051,956 | 45,776,292 |
Random Access Test
| DB | Access | Insert QPS | Query QPS | Update QPS | Delete QPS |
| STL | Map | 2,154,541 | 1,035,817 | 1,041,939 | 1,094,936 |
| CrossDB | STMT | 18,768,313 | 4,923,951 | 4,299,394 | 4,622,262 |
| STL | HashMap | 22,269,633 | 6,651,748 | 8,840,238 | 4,838,402 |