Reproduction Package for FMSD Article `Construction of Verifier Combinations From Off-the-Shelf Components'

This file describes the contents of the reproduction package for the article “Construction of Verifier Combinations From Off-the-Shelf Components”, submitted to FMSD. It contains the experimental data for the evaluation done for the article.

Contents of this Archive

    .
    |-- benchdefs/                       # generated benchmark definition files
    |-- benchexec/                       # submodule of the BenchExec project: https://github.com/sosy-lab/benchexec
    |-- coveriteam/                      # submodule of the CoVeriTeam project: https://gitlab.com/sosy-lab/software/coveriteam
        |-- cache/                       # directory containing all the downloaded tools that are used in the experiments      
    |-- coveriteam-input/                # input data to  CoVeriTeam programs used in evaluation
        |-- actor-definitions/           # actor definitions used in the experiments; some of it are generated
        |-- cvt-programs/                # CoVeriTeam programs used in the experiments
    |-- hardness_models/                 # hardness models
        |-- checkpoints/default/         # The default selection checkpoint used in evaluation
        |-- data/                        # Precomputed embeddings and labels
        |-- train_hardness_model.ipynb   # notebook used for description and replication of the training process
    |-- results/                         # results of the execution
        |-- raw/                         # result files produced by execution of the benchmarks set
        |-- html-tables/                 # html-tables produced by the `table-generator`
        |-- index.html                   # web page describing experiment results and containing links to result tables.
    |-- LICENSE.txt                      # file containing license information
    |-- Makefile                         # Makefile containing targets to create the additional input files, benchmark templates, table definition file and html tables.

Additional directories created when preparing the execution of experiments:

    .
    |-- benchdefs/                       # benchmarks used for the evaluation. Cloned from https://github.com/sosy-lab/sv-benchmarks

Setup the TACAS ’22 Virtual Machine to Run the Examples of the Artifact

This section describes the steps needed to setup the TACAS ’22 virtual machine.

• Configure the system settings for the virtual to the following. Otherwise the experiments won’t start:

  • 8 CPU Cores
  • 16 GB RAM

• Start the virtual machine, and login using the provided credentials.

• Copy the provided artifact archive (named Construction-of-Verifier-Combinations-artifact-FMSD-submission.zip) to the home directory of the virtual machine, and unzip it there. It should create a directory named Construction-of-Verifier-Combinations-artifact-FMSD-submission in the home directory. > NOTE: PLEASE COPY. The tool WILL NOT EXECUTE from the mounted device.

• go to the directory “~/Construction-of-Verifier-Combinations-artifact-FMSD-submission” bash cd ~/Construction-of-Verifier-Combinations-artifact-FMSD-submission

• update package information on the system: bash sudo apt-get update

• install required packages by running the following command: bash sudo apt-get install -y \ git \ php \ libgomp1 \ make \ clang \ clang-10 \ clang-12 \ libmpfr6 \ libssl1.1 \ libz3-4 \ llvm-10 \ llvm-12 \ openjdk-11-jdk-headless \ python3-setuptools \ python3-numpy \ python2

• download the benchmark set for the experiments with the following command: bash make init

• Configure cgroups: execute the script cgroups_setup.sh to configure cgroup for container execution and measurements bash ./configure_cgroups.sh

  NOTE: You will need to run the script “configure_cgroups.sh” each time the VM is rebooted

Testing an Example

Execute Tool Combinations (Sequential Portfolio, Parallel Portfolio, and Algorithm Selection) Used in the Article

The purpose of this step is to execute all the combinations in the paper on an example.

The article reports on evaluation of three types of tool combinations: sequential portfolio, parallel portfolio, and algorithm selection; of different sizes. Each of these combinations act as a verification tool.

The script run_all_tool_combinations.sh contains the commands to execute all these combinations. A user can either execute the script or copy the individual commands from there and execute it.

We now give example of one such command:

  coveriteam/bin/coveriteam \
      coveriteam-input/cvt-programs/portfolio-sequence-2.cvt \
      --cache-dir coveriteam/cache \
      --no-cache-update \
      --allow-cgroup-access-to-actors \
      --input data_model=ILP32 \
      --input specification_path=sv-benchmarks/c/properties/unreach-call.prp \
      --input program_path=sv-benchmarks/c/bitvector-regression/signextension-1.c 

The above command will execute the sequential portfolio of 2 validated verifiers.

Each command on execution will produce an output on the console showing the result of the verification. Additionally, it also produces a directory containing the artifacts generated by the execution in the directory cvt-output/, and a symbolic link lastexecution pointing to the results of the last execution.

We now show an example of such an output information: The tool will output a few warnings when executing a parallel portfolio (explained below) and then the following information:

{'verdict': 'false(unreach-call)', 'witness': 'cvt-output/3474edca-d813-4f69-90c1-bc7ca92992d7/validating-portfolio-cpachecker/bd8d8223-41b7-429a-8a66-77663ca18381/lastexecution/cpachecker-default/a35fad0c-a826-45ea-bee9-48f3a71e1774/output/witness.graphml'}

The above console outputs at first prints some warnings in case the composition was a parallel portfolio. These warnings are harmless and indicate the termination of validated verifiers. The last line prints the verdict and the location of the produced witness.

More information about CoVeriTeam can be found in the README of the tool present in the directory coveriteam/.

Smoke Test

The purpose of this step is to execute three small verification tasks with the benchmarking tool BenchExec on all the combinations. > NOTE: Executing make local-smoke-test should take less than 10 minutes.

Please execute the following:

make local-smoke-test

This step will execute a benchmark definition with three tasks for all the 12 combinations, and run table generator to create HTML tables. The xml files containing measurements is generated in results/smoke-test/, and the HTML files are generated in results/html-tables.

Execution of the above command should produce a series of output (12 times) similar to the following:

executing run set 'SV-COMP22_unreach-call'     (3 files)
17:59:36   eca-rers2012/Problem02_label16.yml                       false(unreach-call)         21.42    9.21
17:59:45   bitvector-regression/signextension-2.yml                 true                        18.10    6.32
17:59:52   aws-c-common/aws_round_up_to_power_of_two_harness.yml    true                        27.60    8.84

It shows the timestamp when the task was triggered, the task, verdict, CPU-time, and wall-time. Since we execute 12 benchmark definitions (one for each composition), an output similar to the above should appear 12 times.

NOTE: Please execute make clean-smoke-test if you want to execute this smoke test again.

Reproducing the Results

The command make local-complete reproduces all benchmark results used in the paper. Please note that the complete benchmark set contains more than 5000 tasks. We used a compute cluster of 168 machines to execute the benchmarks and it took about half a day. On a single PC this would probably take more than a month.

Inspect the Generated Results and HTML Tables

Complete results of our experiments are present in the directory results/. In addition to the XML result files (compressed as .bz2) generated by BenchExec we also provide HTML tables generated by table-generator of BenchExec.

The directory results/ contains the following: - raw/ containing the XML result files. It has 2 sub-directories: smoke-test and complete corresponding to the two described configurations. - html-tables/ containing the HTML files produced by the table-generator. - index.html: this is an HTML file describing the experiments and their results. It can be opened simply in a browser, but in this case the viewer won’t be able to access the logs. It is better to simply start a php server by executing php -S 127.0.0.1:8000 and browse http://127.0.0.1:8000. This way one can view the logs in the HTML tables.

    cd results/
    # Start the php server
    php -S 127.0.0.1:8000   

After the server is running either browse http://127.0.0.1:8000 or run firefox index.html.