FLAME: a formal framework for the automated analysis of software product lines validated by automated specification testing

Abstract

In a literature review on the last 20 years of automated analysis of feature models, the formalization of analysis operations was identified as the most relevant challenge in the field. This formalization could provide very valuable assets for tool developers such as a precise definition of the analysis operations and, what is more, a reference implementation, i.e., a trustworthy, not necessarily efficient implementation to compare different tools outputs. In this article, we present the FLAME framework as the result of facing this challenge. FLAME is a formal framework that can be used to formally specify not only feature models, but other variability modeling languages (VML s) as well. This reusability is achieved by its two-layered architecture. The abstract foundation layer is the bottom layer in which all VML-independent analysis operations and concepts are specified. On top of the foundation layer, a family of characteristic model layers—one for each VML to be formally specified—can be developed by redefining some abstract types and relations. The verification and validation of FLAME has followed a process in which formal verification has been performed traditionally by manual theorem proving, but validation has been performed by integrating our experience on metamorphic testing of variability analysis tools, something that has shown to be much more effective than manually designed test cases. To follow this automated, test-based validation approach, the specification of FLAME, written in Z, was translated into Prolog and 20,000 random tests were automatically generated and executed. Tests results helped to discover some inconsistencies not only in the formal specification, but also in the previous informal definitions of the analysis operations and in current analysis tools. After this process, the Prolog implementation of FLAME is being used as a reference implementation for some tool developers, some analysis operations have been formally specified for the first time with more generic semantics, and more VML s are being formally specified using FLAME.

Appendices

Appendix 1: CML preview for OVM

This appendix contains the metamodel and the corresponding abstract syntax for the Orthogonal Variability Modeling (OVM) notation [51]. The complete CML specification, including the (features-in-a-model) function and the (is-instance-of) predicate, has not been included in order to avoid an excessive length of the article.

The main concepts in OVM models are variation points, variants and constraints. Their graphical representation is shown in Fig. 15, borrowed from [54]. The corresponding metamodel is shown in Fig. 16. For a thorough description of the OVM notation, the interested reader can consult [51].

Fig. 15

OVM notation summary

Fig. 16

OVM metamodel

The translation of the metamodel into an abstract syntax specification in Z is the following. First, an OVM model is defined as a nonempty set of variation points and a set of constraints.

Then, variation points are defined as mandatory and optional. In both cases, they are formed by a feature name and a nonempty set of relationships.

The relationships between variation points and its variants are described as follows. Notice that the alternative relationship includes two natural numbers for the maximum and minimum cardinalities. Also, variants are described as containers of one feature name.

A generalization of variation points and variants, variation element, is needed to specify constraints, which are represented in a similar way as CTCs in BFM, except that in this case they can be set between any pair of variation elements, i.e., variation points and variants.

Appendix 2: CML preview for CUDF

In a similar way to Appendix 1, this appendix contains a preview of the CML for a simplified version of Common Upgradeability Description Format (CUDF) documents [69], a format for describing variability in package-based Free and Open-Source Software (FOSS) distributions. A sample fragment of a CUDF document is shown in Fig. 17.

Packages, attributed with name and version, are the main concept in CUDF documents, equivalent to features in BFM or OVM. They can be related to each other by conflict and dependency relationships. Dependency relationships can be grouped conjunctively—all dependencies must be satisfied—or disjunctively—at least one dependency must be satisfied. All relationships are version dependant, both in depender and dependee packages. The corresponding metamodel is shown in Fig. 18.

Fig. 17

Sample CUDF document fragment

Fig. 18

CUDF metamodel

Before specifying the abstract syntax for CUDF documents, some preliminary definitions are needed. Assuming some type for package IDs (usually character strings), version numbers are defined as natural numbers, version comparators are defined as relations between pairs of version numbers, and features are redefined as $(PackageID,Version)$ pairs:

Having defined previous concepts, a CUDF model can be defined as a set of package relationships:

Relationships, which can be conflicts, conjunctive dependencies, or disjunctive dependencies, are defined over constraints as follows:

Finally, constraints are defined as 5-tuples $(p,v,q,k,\theta )$, where p and q are the identifiers of the depender and dependee packages, respectively, v and k are literal version values, and $\theta$ is a comparison operator.

For example, a constraint such as $(arduino, 2, JDK, 6, {\ge })$ in a conjunctive dependency indicates that version 2 of the arduino package depends on the JDK package version 6 or higher.

Appendix 3: Theorem proofs

This appendix contains the proof of theorems included in Sect. 3.

Proof of theorem 1

(the number of products of a void SPL is 0)

This theorem is proved by the substitution of $void$ and $\mathcal {N}$ by their definitions:

Since $\#\emptyset = 0$ by definition, the theorem is proved. $\square$

Proof of theorem 2

(there not exists any valid configuration for a void SPL)

This theorem is proved by the substitution of by its definition:

Since spl is void, by definition is empty and therefore no valid product with respect to any configuration exists. $\square$

Proof of theorem 3

(any filtering on a void SPL results in an empty set of products)

This theorem is proved by the substitution of by its definition:

Since spl is void, by definition is empty and therefore is also empty for any c. $\square$

Proof of theorem 4

(any pair of void SPL s are equivalent)

This theorem is proved by the substitution of by its definition:

Since $spl_1$ and $spl_2$ are void, by definition and are empty and therefore equal. $\square$

Proof of theorem 5

(the set of core features of a void SPL is empty)

This theorem is proved by the substitution of by its definition:

Since spl is void, by definition is empty and therefore is also empty. $\square$

Proof of theorem 6

(all features of a void SPL are dead)

This theorem is proved by the substitution of by its definition:

Since spl is void, by definition is empty and therefore is also empty. $\square$

Proof of theorem 7

(the set of variant features of a void SPL is empty)

This theorem is proved by the substitution of by its definition:

We know by Theorems 5 and 6 that the set of core features of a void SPL is empty and that all its features are dead. Substituting in the subtraction expression of the three sets, the theorem is proved. $\square$

Proof of theorem 8

(the core, variant and dead features of an SPL partition its features)

In order to prove this theorem, first we substitute the expression by its definition, and then, the four resulting lemmas are proved:

$\square$

Lemma 1

(core and variant features are disjoint)

This lemma is proved by the substitution of by its definition:

is subtracted from spl.features in the right-hand side of the intersection expression; therefore, the intersection is empty.

Lemma 2

(core and dead features are disjoint)

This lemma is proved by the substitution of and by their definitions:

Since the distributed intersection of a set of sets is always a subset of the distributed union of the same set of sets, i.e., , the set difference does not contain any feature in , and therefore, the result of the intersection is empty.

Lemma 3

(variant and dead features are disjoint)

This lemma is proved by the substitution of by its definition:

is subtracted from spl.features in the left-hand side of the intersection expression; therefore, the intersection is empty.

Lemma 4

(the core, variant and dead features are all the features)

This lemma is proved by the substitution of by their definition:

Subtracting and adding the same set to another set leave the latter unmodified, i.e., . In the union expression, and are subtracted and added to spl.features, resulting in spl.features and therefore making both sides of the equality expression the same.

Once Lemmas 1, 2, 3, and 4 are proved, Theorem 8 gets proved too.

Proof of theorem 9

(the set of unique features of a void SPL is empty)

This theorem is proved by the substitution of by its definition:

Since spl is void, by definition is empty, and therefore, is false, making empty. $\square$

Proof of theorems 10 & 11

(in SPL s with more than one product, unique features are variant features & in SPL s with only one product, unique features are core features)

These theorems are proved together using the definition of :

Considering the definition of as , the definition of set subtraction implies that variant features cannot be neither core nor dead features, i.e.:

On the other hand, we know that unique features cannot be dead by definition, i.e.:

Since we know by Theorem 8 that core, variant and dead features form a partition over the set of features of an SPL, unique features must then be core or variant.

If a unique feature is core, that means that is present in all products. The only way of being present only in one product (unique) and in all products (core) at the same time is when there is only one product in the SPL.

By elimination, if there are more than one product in an SPL, unique features cannot be core and must therefore be variant. $\square$

Proof of theorem 12

(the core features, if any, are always one of the atomic sets)

This theorem is proved by the substitution of and by their definitions:

Since core features are included in all products, is true for all products, and therefore, core features are potential atomic sets. On the other hand, they are maximal by definition (), i.e., if a bigger potential atomic set existed, the core features would not be the core features but a proper subset of themselves. $\square$

Proof of theorem 13

(the dead features, if any, are always one of the atomic sets)

This theorem is proved by the substitution of and by their definitions:

Since dead features are not included in any product, is true for all products, and therefore, dead features are potential atomic sets. On the other hand, they are maximal by definition (), i.e., if a bigger potential atomic set existed, the dead features would not be the dead features but a proper subset of themselves. $\square$

Proof of theorem 14

(void SPL s only have one atomic set, its features)

This theorem is proved by applying the results of Theorems 6 and 13:

We know by theorems 6 and 13 that all the features of a void SPL are dead and that dead features are always an atomic set:

Since spl.features is not empty by definition, we can conclude that in void SPL s, . Obviously, if spl.features is an atomic set, no other atomic sets can exist. $\square$

Proof of theorem 15

(the total variability of a void SPL is 0)

This theorem is proved by the substitution of by its definition:

Since spl is void, we know by theorem 1 that . Therefore, . $\square$

Proof of theorem 16

(the partial variability of a void SPL is 0)

This theorem is proved by applying the results of theorem 7:

Since spl is void, we know by theorem 7 that .

Because of the definition of (see Sect. 3.5.2), implies that . $\square$

Proof of theorem 17

(The old homogeneity of a void SPL is 100 %)

This theorem is proved by the substitution of by its definition:

Since spl is void, we know by theorem 9 that . Therefore, . $\square$

Proof of theorem 18

(The new homogeneity of a void SPL is 0 %)

This theorem is proved by the substitution of by its definition:

Because of the definition of (see Sect. 3.5.1), the commonality of a void SPL is always 0. Therefore, . $\square$

Appendix 4: Prolog code of the reference implementation

This appendix contains the translation guidelines applied to the translation of the Z specification into Prolog, and an example of use of the Prolog reference implementation, which can be downloaded from http://www.isa.us.es/flame, together with the 20,000 metamorphic tests.

Fig. 19

Output of the execution of the analysis of a sample SPL in the FLAME reference implementation

1.1 Z-to-Prolog translation guidelines

The main guidelines followed during the manual translation of the Z specification into Prolog are described below.

• Z sets are represented as Prolog lists without duplicates, something common in the animation of Z specifications in Prolog [35, 73]. A small toolkit for those set operations not present in the SWI Prolog distribution was developed for that purpose.

• The SPL schema type was represented as the functor spl(F,M), where F is the SPL feature set and M is the SPL characteristic model. Functors are the usual way of representing compound objects in Prolog (see [20] for details).

• The Configuration type is represented as the functor configuration(S,R), where S is the set of selected features and R is the set of removed features.

• The relation (is-instance-of) is represented as the instance_of(P,M) predicate, where P is a product and M is a characteristic model.

• The function is represented as the features(M,F) predicate, where M is a characteristic model and F is the set of features used in the model.

• As a general pattern, when some elements in a set must be selected by satisfying a predicate, i.e.:

  

  this is translated into Prolog using the standard predicate findall(X,G,L) [20], which returns a list L with all the values of X that satisfy the, possibly compound, goal G. In this pattern, the goal is formed by the conjunction of the membership of X to X_S and the satisfaction of predicate P on X:

  

• Another pattern was applied for translating expressions using the universal quantifier over the elements of a set, i.e.:

  

  This is translated into Prolog using the common predicate forall (C,P), which succeeds if all solutions of C satisfy predicate P. In this case, the condition is the membership of X to X_S, and the predicate is any predicate P on X:

  

1.2 Sample use of the FLAME framework

If an SPL designer would like to use FLAME to analyze her FMs, she should represent them in the Prolog format for the FLAME abstract syntax. For example,

Then, she could use a predicate like this for analyzing her SPL:

That would produce the output in Fig. 19 after calling analyze( survey_spl ) from the Prolog prompt.