Reproduction Package for ASE 2023 submission `LIV: Invariant Validation using Straight-Line Programs'

This is an artifact for reproducing results from the paper “LIV: Invariant Validation Using Straight-Line Programs” that can also be used to showcase how the tool works.

Artifact VM

This artifact is set to use 10 GB of RAM and 2 CPU cores. The username and password for the virtual machine are both vagrant.

Demonstration

After importing and starting the virtual machine, you should find a folder named liv in the user’s home directory, i.e., at /home/vagrant/liv.

Please switch into this folder:

cd liv

Here you will find a copy of this README, a Makefile, and a folder named liv that contains the tool LIV packaged with all requirements.

Simple Example

In order to see whether everything works as intended, you can run the following command:

make example

This will change into the tool directory and execute LIV on a small example:

bin/liv --witness test/programs/simple/example.yml.yaml test/programs/simple/example.i --property test/properties/unreach-call.prp

The example program example.i looks as follows:

int z = 0;
  
void reach_error(void);
void reach_error(void) {__assert_fail("0", "example.i", 4, "reach_error");}
void __VERIFIER_assert(int cond){
  if (!cond) reach_error();
}

void incr(int amount) {
  z += amount;
}

unsigned int a = 42;

int main(int argc, char ** argv)
{
  int x = 1;
  int y = 0;
  while (y<10) {
    y++;
    while (x*10<y) {
      x++;
    }
    __VERIFIER_assert(x*10>=y);
  }
  __VERIFIER_assert(x*10>=y);
}

The witness that is used encodes invariants for each of the loops in the program, namely y<=10 for the inner loop and x*10>=y && y<=10 for the outer loop. The property specification unreach-call.prp is the regular reachability specification used by tools participating in the Competition on Software Verification (SV-COMP). It encodes that calls of the function reach_error shall not be reachable.

The output of LIV is shown in the next listing:

Generated 5 programs
Using verifier "cpachecker-default" from /home/vagrant/liv/liv/actors/cpachecker.yml which claims to be tool version "CPAchecker 2.2"
Verification result of output/program_1.c: true
Verification result of output/program_2.c: true
Verification result of output/program_3.c: true
Verification result of output/program_4.c: true
Verification result of output/program_5.c: true
Overall result: true

We can observe that LIV generated 5 straight-line programs from the original program using the provided invariant information. It then verified each of them using CPAchecker in version 2.2. As all 5 programs are found to adhere to the provided specification, the overall result is true. This means that the witness was validated and the invariant information is sufficient for a successful proof.

We can also have a look at the generated programs, which are conveniently put into the output direcotry as reported by the tool. For example, here is the main function of output/program_3.c:

int main()
{
  int z = __VERIFIER_nondet_int();
  unsigned int a = __VERIFIER_nondet_int();
  int argc = __VERIFIER_nondet_int();
  char **argv = __VERIFIER_nondet_pointer();
  int x = __VERIFIER_nondet_int();
  int y = __VERIFIER_nondet_int();
  LIV_assume(y <= 10);
  LIV_assume((x * 10) < y);
  {
    x++;
  }
  LIV_assert(y <= 10);
}

We can observe that this assumes the invariant of the inner loop, assumes we enter that loop, executes the loop body, and asserts that the invariant of the inner loop still holds. As such, this program corresponds to the check whether the inner loop invariant is inductive.

There is also an option --verbose that will print all generated programs to standard output for convenience, you can see how this output looks like by running make exampleverbose if you want to, or just append --verbose to your call to LIV.

We can also use a different backend for verification by using the parameters --verifier pointing to a different CoVeriTeam actor definition and --verifierversion specifying the actor version.

For this, either use the Makefile with make examplecbmc or run the following command from the tool’s folder yourself:

bin/liv --witness test/programs/simple/example.yml.yaml test/programs/simple/example.i --property test/properties/unreach-call.prp --verifier actors/cbmc.yml --verifierversion default

The output is largely the same as above, only the information about the backend verifier changes. Also note that cbmc is much faster than CPAchecker, as CPAchecker has a significant startup time. If you want to run the CPAchecker configuration with linear arithmetic mentioned in the paper, you can have a look at the Makefile target examplecpalia.

Another feature of LIV that is currently in development is to provide some statistics. This can be done by adding the commandline argument --stats. This will print some statistics e.g. about the invariants found in the witness, or the verification status of the generated programs. These statistics are machine-readable by benchexec and can be aggregated together with the validation results in order to do some statistics over the benchmark set that is executed.

An example of this statistics output can be seen when running the Makefile target make examplestats:

bin/liv --witness test/programs/simple/example.yml.yaml test/programs/simple/example.i --property test/properties/unreach-call.prp --verifier actors/cbmc.yml --verifierversion default --stats
Generated 5 programs
Using verifier "cbmc-default" from actors/cbmc.yml which claims to be tool version "CBMC 5.43.0 (cbmc-5.43.0-18-gb83f0e1cc3)"
Verification result of output/program_1.c: true
Verification result of output/program_2.c: true
Verification result of output/program_3.c: true
Verification result of output/program_4.c: true
Verification result of output/program_5.c: true
##########BEGIN OF STATISTICS##########
  CFA:
  WITNESS:
    number of invariants in witness:           2
    number of parseable invariants in witness: 2
  SLICING:
  VERIFICATION:
    total count of conditions:           5
    number of proved conditions:         5
    number of violated conditions:       0
    number of unclear conditions:        0
    number of conditions causing errors: 0
##########END OF STATISTICS##########
Overall result: true

Example for Experiment 1

We can also execute LIV for one of the benchmarks from experiment 1. For that, either run make exampleexperiment1 from ~/liv, or the following command from the tool directory ~/liv/liv:

bin/liv --witness test/programs/loop-zilu/benchmark25_linear.yml.yaml test/programs/loop-zilu/benchmark25_linear.i --verifier actors/cbmc.yml --verifierversion default --property test/properties/unreach-call.prp

This runs LIV with CBMC as backend on the task benchmark25_linear.i with the invariants taken from ACSL Benchmarks in the file benchmark25_linear.yml.yaml

The output is should be as follows:

Generated 3 programs
Using verifier "cbmc-default" from actors/cbmc.yml which claims to be tool version "CBMC 5.43.0 (cbmc-5.43.0-18-gb83f0e1cc3)"
Verification result of output/program_1.c: true
Verification result of output/program_2.c: true
Verification result of output/program_3.c: true
Overall result: true

Example for Experiment 2

Likewise, we can execute one of the benchmarks from experiment 2 by either running make examplexperiment2 or the following command from within the tool directory:

bin/liv --witness test/programs/exp/uautomizer/loop-zilu/benchmark25_linear.yml.graphml test/programs/exp/uautomizer/loop-zilu/benchmark25_linear.i --verifier actors/cbmc.yml --verifierversion default --property test/properties/unreach-call.prp

The output will look identical to the example for experiment 1, as in both cases 3 programs are generated, which are found to fulfill the specification.

Reproduction of Results

Experiment 1

For reproducing the results for experient 1, run make experiment1 from within ~/liv. This will execute LIV on the benchmark set using benchexec. The results are then stored in ~/liv/results_experiment1. A table can be generated with benchexec’s table-generator:

liv/lib/benchexec/bin/table-generator results_experiment1/zilu.*.results.*.xml.bz2 -n experiment1

This will generate a HTML result table at liv/results_experiment1/experiment1.table.html and also make the data in the table available for further processing at liv/results_experiment1/experiment1.table.csv

Experiment 2

For reproducing the results for experient 2, run make experiment2 from within ~/liv. his will execute LIV on the benchmark set using benchexec. The results are then stored in ~/liv/results_experiment1. A table can be generated with benchexec’s table-generator:

liv/lib/benchexec/bin/table-generator results_experiment2/experiments.*.results.cbmc.*.xml.bz2 -n experiment2

This will generate a HTML result table at liv/results_experiment2/experiment2.table.html and also make the data in the table available for further processing at liv/results_experiment2/experiment2.table.csv