Software and Computational Systems Lab

Publications about Benchmarks

1. Po-Chun Chien, Nian-Ze Lee, and Zhengyang Lu. Performance Dataset for Hardware Model Checking on Btor2 Benchmarks. arXiv/CoRR, May 2025. Keyword(s): Btor2, Benchmarking, Benchmarks, Dataset PDF Supplement
   Abstract

   This technical report presents a performance evaluation of several hardware model-checking tools on a collection of benchmark tasks in the Btor2 format. The resulting dataset is intended to support machine-learning research for hardware model checking, particularly in areas such as algorithm selection, performance prediction, and automated tool configuration. It has been used, for example, in the development and evaluation of Btor2-Select, a machine-learning-based algorithm selection framework for hardware model checking. To construct the dataset, we benchmarked a diverse set of model-checking tools and algorithmic configurations. Each verification engine was evaluated on a common set of Btor2 tasks and the performance measurements, including CPU time, wall time, and memory usage, were collected. All data, including scripts and files required to reproduce the experiments, are publicly available at: https://gitlab.com/sosy-lab/research/data/perf-eval-hwmc.
   BibTeX Entry

   @techreport{HWMCPerfDataset-TR, author = {Po-Chun Chien and Nian-Ze Lee and Zhengyang Lu}, title = {Performance Dataset for Hardware Model Checking on Btor2 Benchmarks}, number = {}, year = {2025}, doi = {}, url = {https://gitlab.com/sosy-lab/research/data/perf-eval-hwmc}, pdf = {https://www.sosy-lab.org/research/pub/2025-TR.Performance_Dataset_for_Hardware_Model_Checking_on_Btor2_Benchmarks.pdf}, abstract = {This technical report presents a performance evaluation of several hardware model-checking tools on a collection of benchmark tasks in the Btor2 format. The resulting dataset is intended to support machine-learning research for hardware model checking, particularly in areas such as algorithm selection, performance prediction, and automated tool configuration. It has been used, for example, in the development and evaluation of Btor2-Select, a machine-learning-based algorithm selection framework for hardware model checking. To construct the dataset, we benchmarked a diverse set of model-checking tools and algorithmic configurations. Each verification engine was evaluated on a common set of Btor2 tasks and the performance measurements, including CPU time, wall time, and memory usage, were collected. All data, including scripts and files required to reproduce the experiments, are publicly available at: <a href="https://gitlab.com/sosy-lab/research/data/perf-eval-hwmc">https://gitlab.com/sosy-lab/research/data/perf-eval-hwmc</a>.}, keyword = {Btor2, Benchmarking, Benchmarks, Dataset}, artifact = {}, institution = {arXiv/CoRR}, month = {May}, }

2. Moritz Bierwirth. Designing and Assessing a Benchmark Set for Fault Localization Using Fault Injection. Bachelor's Thesis, LMU Munich, Software Systems Lab, 2023. Keyword(s): Benchmarks PDF
   BibTeX Entry

   @misc{BierwirthFLBenchmarks, author = {Moritz Bierwirth}, title = {Designing and Assessing a Benchmark Set for Fault Localization Using Fault Injection}, year = {2023}, pdf = {https://www.sosy-lab.org/research/bsc/2023.Bierwirth.Designing_and_Assessing_a_Benchmark_Set_for_Fault_Localization_Using_Fault_Injection.pdf}, keyword = {Benchmarks}, field = {Computer Science}, howpublished = {Bachelor's Thesis, LMU Munich, Software Systems Lab}, }
3. Korab Zogu. SV-COMP Benchmarks for Weak Memory Models. Bachelor's Thesis, LMU Munich, Software Systems Lab, 2021. Keyword(s): Benchmarks, Weak Memory Models
   BibTeX Entry

   @misc{ZoguBenchmarksWeakMemoryModel, author = {Korab Zogu}, title = {SV-COMP Benchmarks for Weak Memory Models}, year = {2021}, keyword = {Benchmarks, Weak Memory Models}, field = {Computer Science}, howpublished = {Bachelor's Thesis, LMU Munich, Software Systems Lab}, }