A risk assessment model for selecting cloud service providers

Abstract

The Cloud Adoption Risk Assessment Model is designed to help cloud customers in assessing the risks that they face by selecting a specific cloud service provider. It evaluates background information obtained from cloud customers and cloud service providers to analyze various risk scenarios. This facilitates decision making an selecting the cloud service provider with the most preferable risk profile based on aggregated risks to security, privacy, and service delivery. Based on this model we developed a prototype using machine learning to automatically analyze the risks of representative cloud service providers from the Cloud Security Alliance Security, Trust & Assurance Registry.

Introduction

Moving business processes to the cloud is associated with a change in the risk landscape to an organization [1]. Cloud Security Alliance (CSA) [2] has found that insufficient due diligence was among the top threats in cloud computing in 2013. This threat is linked to the fact that organizations which strive to adopt cloud computing often do not understand well the resulting risks.

Regulations related to data protection, financial reporting, etc. put certain requirements that should be complied with even when outsourcing business processes to 3rd parties, like cloud service providers (CSPs). For example, EU Data Protection Directive, in particular Article 29 Data Protection Working Party [3] recommends that all data controllers (usually corporate cloud customers) perform an impact assessment of moving personal data of their clients to the cloud.

However, most of the cloud service customers (CSCs), especially Small-Medium Businesses (SMBs), may not have enough knowledge in performing such assessments at a good level, because they may not necessarily employ IT specialists and the lack of transparency is intrinsic to the operations of the CSPs. This makes difficult to choose an appropriate CSP based on CSC’s security requirements, especially considering the abundance of similar cloud offerings [4].

This work proposes a methodology, cloud adoption risk assessment model (CARAM), to help in assessing the various risks to business, security and privacy that CSCs face when moving to the cloud by leveraging information from CSCs, CSPs and several public sources. CARAM consists of the following tools that complement the various recommendations from European Network and Information Security Agency (ENISA) [1], and Cloud Security Alliance (CSA) for a complete risk assessment framework:

• A questionnaire for CSCs

• A tool and an algorithm to classify the answers to Cloud Assessment Initiative Questionnaire (CAIQ) to discrete values

• A model that maps the answers to both questionnaires to risk values

• A multi-criteria decision approach with posterior articulation of CSC preferences for relative risk analysis, using a few parameters for security, privacy and quality of service, allowing to to quickly and reliably compare multiple CSPs

This paper extends our work in [5] with experimental results - we devised profiles representing realistic customer categories to classify providers according to the CSC needs. We also used a more precise risk scale for comparing CSPs, allowing one to visualize the differences in security practices of the most representative players in the cloud services landscape. Therefore the current version brings significative improvements with respect to the previous paper [5].

In Section “Related work” we elaborate on the literature related to the risk assessment for adoption of cloud computing: we focus on the work carried out by ENISA and CSA because CARAM is based on them; In Section “Risk levels computation” we introduce CARAM, and then a multi-criteria risk assessment approach with posterior articulation of the CSCs; In Section “Experimental results” we demonstrate experimental results from using CARAM on a case study; In Section “Limitations” we outline some limitations of the approach; We conclude our paper in Section “Conclusion”.

Related work

Several large standardization bodies such as International Organization for Standardization (ISO), International Electrotechnical Commission (IEC) and National Institute of Standards and Technology (NIST) and Information Technology (IT) Governance Institute and the Information Systems Audit and Control Association (ISACA) published standards on IT risk management and risk assessment: ISO 31000 [6], ISO/IEC 31010 [7], ISO/IEC 27005 [8], NIST SP 800-30 [9], SP 800-37 [10] and COBIT [11]. All these standards are generic i.e. not specific to cloud deployments, and while possible to use them for evaluating different cloud solutions, it will require a considerable amount of effort and expert knowledge, which SMBs cannot always afford.

Some adaptations of these standards were developed specifically for cloud deployments. E.g. Microsoft proposed a Cloud Decision Framework [12] based on ISO 31000. It provides guidance for risk assessment to be performed by potential CSCs when choosing a cloud solution. The risk profiles of different cloud solutions are constructed based on a predefined set of risks from four categories: compliance, strategic, operational and market and finance. The authors suggest using CSA Cloud Control Matrix (CCM) [13] as guidance for evaluating mitigating controls. While this approach could be more practical than other generic risk assessment frameworks for evaluating different cloud solutions it is still quite abstract and largely rely on experts’ opinions for estimating and evaluating the risks and mitigations. In contrast, we propose a concrete step by step approach to automate the estimation and evaluation of risks of adopting different cloud solutions. This could be more suitable for smaller organizations that lack sufficient resources for a full-scale risk assessment.

ENISA [1] provided recommendations and a framework for generic qualitative inductive risk assessment for cloud computing. Their recommendations include extensive lists of possible incident scenarios, assets and vulnerabilities in cloud computing deployments. It suggests estimating risk levels on the basis of likelihood of a risk scenario mapped against the estimated negative impact, which is the essence of the risk formulation by also many others in the literature [7, 11, 14, 15]. Although ENISA’s recommendations are specific for cloud computing, it is a generic framework that does not provide an approach to map the specifics of CSPs and CSCs to the 35 risk scenarios listed in the report [1]. In Section “Risk levels computation” we describe how CARAM fine-tunes this approach to estimate risk values based on some known information about CSCs and CSPs.

Another qualitative inductive scheme was published by “The Commission nationale de l’informatique et des libertés” (CNIL) or in English: The French National Commission on Informatics and Liberty [16] more recently. CNIL’s methodology is similar to the ENISA’s framework with the following difference: it is a risk assessment focused on privacy risks in cloud computing. It also recommends measures to reduce the risks and assess the residual privacy risks after the application of these measures. However, it is still generic and does not account for specific requirements of CSPs or CSCs.

The CSA Cloud Assessment Initiative Questionnaire (CAIQ) [17] is a questionnaire prepared for CSPs to document the implemented security measures. It is based on the CSA Cloud Control Matrix (CCM) taxonomy of security controls [13] and is aimed to help CSCs understand the security coverage of specific cloud offerings in relation to popular security standards, control frameworks and regulations. The questionnaires answered by many CSPs are publicly available in the CSA Security, Trust and Assurance Registry (STAR) [4]. We propose a methodology that uses the data extracted from STAR to evaluate the implementation of various controls provided by cloud solutions (see Subsection “The vulnerability parameter for a CSP”).

Luna et al. introduced in [18] Cloud Security Level Agreements (SecLA) and proposed a methodology to benchmark SecLA of CSPs with respect to CSCs’ requirements [19]. Both CSP SecLA provisions and user requirements are expressed using a special data structure: Quantitative Policy Trees, allowing expressing controls with different granularity: CCM control areas, control groups, and controls (corresponding to CAIQ answers). The authors demonstrate their approach using data on several CSPs from STAR, by calculating security levels for respective controls and control groups. While similar in the intent CARAM is a model for risk assessment, while [19] proposes a ranking algorithm for matching CSC requirements vs. CSP provisions. In [19] CSCs need a certain level of security expertise to specify their requirements, while in CARAM this is not necessary: CSCs only need to specify acceptable risk levels for security, privacy and service categories, while still allowing a more fine grained specification. Another major difference is that [19] assumes the existence of a mapping from provisions to quantitative Local Security Levels to allow further analysis. Given a high number of potential CSPs and controls for each CSP creating this mapping would require significant manual work. In CARAM we propose a way to automatically construct such a mapping (see Subsection “The vulnerability parameter for a CSP”).

Habib et al. proposed a multi-faceted Trust Management system architecture for a cloud computing marketplace [20]. The system evaluates the trustworthiness of CSPs in terms of different SLA attributes assessed using information collected from multiple sources. This is done by evaluating opinions related to SLA attributes and aggregating them into a trust score for a CSP. The authors mention CAIQ answers as a source of information, however they do not specify how exactly the CSP trust score is computed from the answers, especially considering that the answers are in free text form.

In [21], the EU funded project SECCRIT enumerated very relevant cloud risk scenarios systematically, in a similar fashion to what ENISA [1] did a few years earlier, but this time they evaluated the risk perception from the users point of view. They used a survey-based methodology to ask respondents to rank risks in a standard way - assessing probability and impact. CARAM is innovative when compared with this approach in two ways: it is using real information disclosed by cloud providers to estimate the likelihood of threats affecting the cloud would become concrete. Second, it allows the user of the methodology to focus on what they know best, their assets: the methodology only requires to assign priorities to assets in order to provide quantifiable risk information.

Other EU projects such as SPECS [22] and Escudo-Cloud [23] also looked at quantifying the capacity of a given cloud provider to satisfy security Servive Level Agreements (SLAs) using information published in the CSA STAR registry. It also has a focus on usability, but the endeavour is different in essence. Here we are assessing risk scenarios in an automated way, whereas in that work [23] does not provide an automated processing for the answers given by the CSPs to the CAIQ, making it difficult to compare more than three providers quickly.

Joint Risk and Trust Model (JRTM) [24] was developed by Accountability for Cloud and Other Future Internet Services Project (A4Cloud). JRTM is a quantitative risk assessment model that assesses the cloud service security and privacy risks for a specific CSP and CSC. It counts on a third party (i.e., a Trust as a Service Provider) to accumulate statistical data (i.e., evidence) on the trustworthiness of CSPs. These evidences include the number of security, privacy and service events that a CSP was subject to and the percentage of the events that the CSP recovered from before they become an incident (i.e., they impact on CSC). However, such detailed statistical data is not always available and even if available they will not be normally shared with (potential) CSCs—hence the need for a trusted third party. In this work, instead, we rely on public information already provided by CSPs regarding their implementation of various security controls for a qualitative risk assessment of their solutions. This can be considered as an extension of JRTM when statistical data is available or a substitution otherwise.

One of the main advantages of CARAM is that it is easy to make it evolve to cover further risk scenarios and to make it evolve if the security control frameworks evolve. As we will introduce in the next section, the risk level computation is parametrized by mappings between threats and security controls that help to mitigate them, making the approach relevant over time.

Risk levels computation

ENISA [1] identified 35 incident scenarios that fall in one of the following four categories: policy and organizational, technical, legal and the other scenarios not specific to cloud computing (see Table 1). The likelihood of each of these scenarios and their business impact are determined in consultation with an expert group. The scale of probability and impact has five discrete classes between very low and very high. For example, the probability and impact of Incident Scenario P1 in “Policy and Organizational Scenarios” category (i.e., lock-in) are given as HIGH and MEDIUM relatively.

Table 1 ENISA’s list of risk scenarios and their categories

ENISA also provides a list of 53 vulnerabilities (i.e., 31 cloud specific and 22 not cloud specific vulnerabilities, see Table 2) and 23 classes of CSC assets (see Table 3) that may be affected by the cloud adoption. Each of 35 incident scenarios is related with a subset of vulnerabilities and assets. For example, the Incident Scenario P1 is related to Vulnerabilities V13 (lack of standard technologies and solutions), V31 (lack of completeness and transparency in terms of use), V46 (poor provider selection), V47 (lack of supplier redundancy) and Assets A1 (company reputation), A5 (personal sensitive data), A6 (personal data), A7 (personal data critical), A9 (service delivery - real time services), A10 (service delivery).

Table 2 ENISA’s list of vulnerabilities

Table 3 ENISA’s list of assets

The likelihood and business impact values that are determined by the experts are converted to the risk levels for each incident scenario based on a risk matrix with a scale between 0 and 8 as shown in Fig. 1. Then, the risk levels are mapped to a qualitative scale as follows:

• Low → 0-2

  Fig. 1

  

  ENISA definition of risk levels

• Medium → 3-5

• High → 6-8

Hence, a CSC can assess the risk level related to an incident scenario qualitatively and understands what kind of vulnerabilities and assets are related to each scenario by examining [1]. These values represent educated guesses over a wide range of common cloud deployments and do not have a precise semantics. In practice, the risk levels are related to many factors such as the security controls that CSPs implement and the concerned assets of the specific users. Therefore, a generic value cannot be applied to all CSPs and CSCs. Although vulnerabilities and assets for each incident scenario are given by the ENISA framework, it does not describe how those values can be adapted for a specific CSP and CSC pair. CARAM fills this gap. For that, first the qualitative scale used by ENISA as probability and impact values are mapped to a quantitative scale as follows:

• Very low → 1

• Low → 2

• Medium → 3

• High → 4

• Very high → 5

For example, probability P ₁ and impact I ₁ values for the first scenario (i.e., lock in) is HIGH and MEDIUM respectively. We map these values as follows: P ₁=4 and I ₁=3.

To compute the risk levels considering particular vulnerabilities and assets involved CARAM adjusts the values from ENISA, taken as a baseline, taking into account additional information about the cloud service. For that, we use Eqs. 1 and 2:

$$\begin{array}{*{20}l} \beta_{i} &= P_{i} \times \nu_{i} \end{array} $$

(1)

$$\begin{array}{*{20}l} \delta_{i} &= I_{i} \times \alpha_{i} \end{array} $$

(2)

In Eq. 1, for the risk scenario i, β _i is the adjusted probability, ν _i is the vulnerability index of a given CSP, δ _i is the adjusted impact and α _i is the asset index for a given CSC. Here we assume that probability and impact of an incident are proportional to the number of non-addressed vulnerabilities by a CSP; and impact is proportional to the number of CSC assets related to a risk scenario.

Note that vulnerability index of a CSP is the same for all CSCs and the asset index of a CSC is the same for all CSPs. Vulnerability and asset indices are calculated as given in Eqs. 3 and 4 respectively, where ν _ki =1 if vulnerability k is in the list of vulnerabilities [1] for risk scenario i, and 0 otherwise. Similarly, α _ki =1 if asset k is in the list of assets [1] for risk scenario i. Please note again that there are 53 vulnerabilities (see Table 2) and 23 assets (see Table 3) listed in [1]. The vulnerability related parameter ε _k in Eq. 3 is derived from the answers to CAIQ, and elaborated on later in Subsection “The vulnerability parameter for a CSP”. The asset related parameter γ _k in Eq. 4 is given value 0 if the CSC’s answer to the question “Does the service that you seek will involve any asset of yours that fall in the same category as asset k?” is “No”, and value 1 otherwise.

$$\begin{array}{*{20}l} \nu_{i} &= \frac{\sum_{k=1}^{53}{\nu_{ki} \times \epsilon_{k}}}{\sum_{k=1}^{53}{\nu_{ki}}} \end{array} $$

(3)

$$\begin{array}{*{20}l} \alpha_{i} &= \frac{\sum_{k=1}^{53}{\alpha_{ki}\times \gamma_{k}}}{\sum_{k=1}^{53}{\alpha_{ki}}} \end{array} $$

(4)

We would like to highlight that CARAM is independent from the number of incident scenarios and probability, impact, vulnerability and assets assigned to the incident scenarios. Moreover, it is possible to assign weight values for each of assets and vulnerabilities if some of them are assumed as of higher importance comparing to the others.

The vulnerability parameter for a CSP

We use CSPs’ responses to CAIQ from [4] to assign a value to the vulnerability related parameter ε _k . Most of the entries in STAR are using the CAIQ v1.1 template [25] that provides 148 questions grouped into the control areas shown in Table 4 covering the state of implementation of security controls.

Table 4 The control areas in CAIQ

The v1.1 template accepts a free text answer in contrast to the newer v3.0.1 template where the CSPs are expected to choose their answers between “Yes”, “No” and “Not applicable”. This makes the answers unsuitable for automated analysis. We proposed a mechanism to map the answers given to the questions in CAIQ to one of the control implementation status categories in Table 5.

Table 5 The categorization of the answers given to the questions in CAIQ

The category Implemented has a positive meaning (the control is in place), but the answer “Yes” to a CAIQ question does not always imply a more secure system. For example, the “Yes” answer to CAIQ Question RS06-01 “Are any of your data centers located in places which have a high probability/occurrence of high-impact environmental risks (floods, tornadoes, earthquakes, hurricanes, etc.)?” implies a negative outcome, which means the control is not implemented. In such cases we classify the “Yes” answer as Not Implemented and the “No” answer as Implemented.

We analyzed 44 out of 70 CSPs from the mentioned registry providing answers to about 200 questions each. To be included in the selection a provider had to fill in the CAIQ questionnaire in the format provided by CSA (to enable automated processing). The responses of some of the big CSPs (e.g. Amazon, HP, Microsoft, RedHat, or SAP¹) who provided answers in other forms, though, were processed manually to ensure the consideration of the major players. Given the workload, we decided to automate the classification of the free text answers to CAIQ questions using the supervised machine learning algorithms (sequential minimal optimization and string vectorization) provided by the WEKA tool [26]. We created a training set from a random sampling of around 300 manually classified answers out of overall circa 9000 answers and used it to classify the other remaining answers. The 10-fold cross-validation provided an accuracy of around 84 % of correctly classified instances, which we consider enough for our purpose.

After classification of the answers to one of the categories in Table 5, the implementation value q _m is assigned for each of the controls identified by question m:

• Implemented → q _m =0: the related vulnerabilities are mitigated

• Not Applicable → q _m =0: these controls do not impact the risk value

• No → q _m =1: the related vulnerabilities are not mitigated

• Conditionally Implemented → the CSC needs to clarify with the CSP if the control can be implemented. If yes, q _m =0. Otherwise, q _m =1.

When q _m is known for a CSP and a CSC, Eq. 5 gives the vulnerability related parameter ε _k for the CSP and the CSC. Please note that this value is for a specific CSP and CSC pair.

$$ \epsilon_{k} = \frac{\sum_{m=1}^{n}{r_{mk} \times q_{m}}}{\sum_{m=1}^{n}{r_{mk} \times b_{m}}} $$

(5)

In Eq. 5, n is the number of questions in CAIQ. r _mk is the mapping of the CAIQ questions to vulnerabilities: it is 1 if the control representing the question m mitigates the vulnerability k, and 0 otherwise (see Table 6 for an example mapping excerpt and Additional file 1 — Mapping CAIQ questions to vulnerabilities.xlsx for the full mapping).

Table 6 Mapping r _mk of CAIQ questions to ENISA vulnerabilities (excerpt)

Finally, b _m =0 if the answer to the question m is “Not Applicable” and 1 otherwise. This allows discarding the unrelated questions to avoid wrongly penalizing the CSPs.

In Eq. 5 ε _k receives a minimum value 0 if all the controls related to the vulnerability k are implemented and hence the vulnerability does not impact negatively the risk values. The more controls related to the vulnerability k are not implemented, the higher ε _k is. Its maximum value is 1, which means the CSP has no measures against the vulnerability k.

Relative risk assessment-based CSP selection

ENISA’s risk assessment model is based on 35 incident scenarios. These numerous risks make it difficult for customers to meaningfully compare multiple providers. Therefore, we first reduce the number of criteria from these 35 incident scenarios to three categories of cloud risks: security, privacy and service [24]. For that, we compute the probability that a privacy (β _r ), a security (β _s ) and a service (β _e ) incident can occur and the impact of a privacy (δ _r ), a security (δ _s ) and a service (δ _e ) incident by applying Eqs. 6 to 11. In Eqs. 6 and 9, r _i =1 if ENISA incident scenario i is related to privacy, and 0 otherwise (see Table 7 for an example mapping). ω _ri and α _ri are real numbers between 0 and 1. They are the weight factors for probability and impact respectively. The significance of every scenario may not be the same when calculating an aggregated value for privacy, security and service incidents. Moreover, the scenarios may need to be treated differently for each CSC especially when calculating the aggregated impact values. The weight factors are for making these adjustments. If the significance of each scenario is the same, then the weight factors can be assigned 1. Similar to r _i , s _i and e _i are the mapping values for security and service risks respectively (see Table 7). ω _si and α _si are the weight factors for security scenarios, and ω _ei and α _ei are the weight factors for service scenarios.

$$\begin{array}{*{20}l} \beta_{r} &= \frac{\sum_{i=1}^{35}{\beta_{i} \times r_{i} \times \omega_{ri}}}{\sum_{i=1}^{35}{r_{i} \times \omega_{ri}}} \end{array} $$

(6)

Table 7 Mapping (r _i ,s _i ,e _i ) of ENISA risk scenarios to risk categories

$$\begin{array}{*{20}l} \beta_{s} &= \frac{\sum_{i=1}^{35}{\beta_{i} \times s_{i} \times \omega_{si}}}{\sum_{i=1}^{35}{s_{i} \times \omega_{si}}} \end{array} $$

(7)

$$\begin{array}{*{20}l} \beta_{e} &= \frac{\sum_{i=1}^{35}{\beta_{i} \times e_{i} \times \omega_{ei}}}{\sum_{i=1}^{35}{e_{i} \times \omega_{ei}}} \end{array} $$

(8)

$$\begin{array}{*{20}l} \delta_{r} &= \frac{\sum_{i=1}^{35}{\delta_{i} \times r_{i} \times \alpha_{ri}}}{\sum_{i=1}^{35}{r_{i} \times \alpha_{ri}}} \end{array} $$

(9)

$$\begin{array}{*{20}l} \delta_{s} &= \frac{\sum_{i=1}^{35}{\delta_{i} \times s_{i} \times \alpha_{si}}}{\sum_{i=1}^{35}{s_{i} \times \alpha_{si}}} \end{array} $$

(10)

$$\begin{array}{*{20}l} \delta_{e} &= \frac{\sum_{i=1}^{35}{\delta_{i} \times e_{i} \times \alpha_{ei}}}{\sum_{i=1}^{35}{e_{i} \times \alpha_{ei}}} \end{array} $$

(11)

When probability (i.e., β) and impact (i.e., δ) values are calculated, they are mapped to the qualitative scale as follows:

• [0,0.5)→ Negligible

• [0.5,1)→ Extremely Low

• [1,1.5)→ Very Low

• [1.5,2)→ Low

• [2,2.5)→ Below Average

• [2.5,3)→ Above Average

• [3,3.5)→ High

• [3.5,4)→ Very High

• [4,4.5)→ Extremely High

• [4.5,5]→ Not Recommended

Please note that this is a higher resolution scale with 10 values comparing to ENISA’s original five value qualitative scale. We need a higher resolution scale to differentiate CSPs because the adjusted probabilities of risk scenarios are mostly below the average for the CSPs that answer CAIQ. Those CSPs are clearly aware of the incident scenarios and implement at least a subset of the controls, which are subject in CAIQ. Finally, by using the matrix in Fig. 2—which is similar to but has a higher resolution than the matrix in Fig. 1—the risk values for privacy R _r , security R _s and service R _e are obtained. Please note that these values are calculated for each CSP-CSC pair. Although, the color codes in Fig. 2 are only for the three value qualitative scale (i.e., Low < Medium < High) as in the ENISA’s assessment, the quantative scale for the overall risk assessment is also higher resolution (between 0 and 18 instead of 0 and 10) which is the result of selecting higher resolution scales for probability and impact.

Fig. 2

CARAM definition of risk levels

At this stage, the CSC provides CARAM with the maximum acceptable levels of risks for privacy R _rmax , security R _smax and service R _smax . The CSC may also provide a set U={p ₁,…,p _n } of CSPs that should be excluded from the assessment due to reasons like business relations, politics, past experience, etc. When this information is available, CARAM creates a set F of feasible CSPs out of the set S of all the CSPs available for assessment (i.e. CSPs that have a completed CAIQ in STAR) such that F⊂S using Eq. 12.

$$ \begin{aligned} p_{i} \in F \iff (p_{i} \notin U) &\wedge (R_{rmax} > R_{ri}) \wedge \\ (R_{smax}> R_{si}) &\wedge (R_{emax} > R_{ei}) \end{aligned} $$

(12)

where R _ri ,R _si and R _ei are the privacy, security and service risks for the CSP p _i .

F can be an empty set, a set with only one element or multiple elements. If F is an empty set, there is no feasible solution for the CSC. If F has only one element, that is the only feasible solution for the CSC under the given constraints. In both of these cases, CARAM informs the CSC directly with the result. If F has multiple elements, all the dominating CSPs are removed from F resulting in the set F ^′. Here we define the dominating relation ≻:C S P ₁≻C S P ₂ ⇔ R _ri ,R _si and R _ei for C S P ₁ are higher than those for C S P ₂. If the resulting F ^′ includes only one CSP, CARAM informs the CSC about the solution that fits the best to it. If there are multiple CSPs in F ^′, the CSC is given the complete F ^′ for the posterior articulation of the preferences.

Experimental results

We prototyped CARAM and run experiments the data in STAR [4] during the course of 2015. In this section we present the results of these experiments (see Additional file 2 — CARAM Experiments.xlsx for the detailed data). Figures 3 and 4 illustrate the vulnerability index ν _i and the adjusted probability β _i values for the 44 CSPs included in our analysis from STAR. These are calculated by the CARAM prototype as explained in Section “Risk levels computation”.

Fig. 3

Vulnerability index ν _i for 44 CSPs in STAR

Fig. 4

Adjusted probability β _i for 44 CSPs in STAR

The differences between the vulnerability indices and therefore the adjusted probabilities of various CSPs in STAR are in the order of magnitude. For example, the lowest vulnerability index is 0.011, which is the vulnerability index of a CSP called as Iguana (we call the CSPs in STAR database by using animal names to preserve their confidentiality). On the other hand, the vulnerability index for the highest risk CSP (i.e., Gazelle) is 0.491. Although the vulnerability index of Gazelle is more than 44 times higher than the Iguana’s, it is less than 0.5. This means that the probability value for the highest risk CSP in STAR will be reduced more than 50 %, and become “LOW” according to our higher resolution qualitative scale. This results from the fact that the CSPs from STAR declare to implement the majority of controls. That makes sense because the CSPs in STAR represent a subset of CSPs which are putting an effort to reduce their risk. Their submission of CAIQ is an indication for that.

Another interesting observation in Figs. 3 and 4 is about the differences among privacy, security and service vulnerability index and adjusted probability for each CSP. The differences among these three values are observable only for few CSPs. This is an indication that the CSPs in STAR do not focus on one of privacy, security or service, but typically treat them equally. This is coupled with the fact that the categories of privacy, security and service risks are overlapping, and the number of risks in the privacy, security and service categories by our selection are almost the same: 19, 22 and 22 respectively (see Table 7).

For our experiments, we considered five types of CSCs based on the nature of their assets. We calculated the percentage of the risk incident scenarios that each asset is related to the ENISA risk assessment. We noticed that the following three assets are not related to any scenarios: A15 management interface APIs, A18 physical buildings, and A19 cloud provider application (source code). We call any asset related to at least 10 % of the incident scenarios as highly exposed asset. Some assets are service-related and some are data assets. Finally, data assets can be personal or not personal. Based on these, we use the following five types of CSCs in our experiments:

• C S C ₁ have all the assets in the ENISA’s list.

• C S C ₂ have all highly exposed assets.

• C S C ₃ have all the data and service assets.

• C S C ₄ have only data assets.

• C S C ₅ have only personal data assets.

CARAM computes the privacy, security and service impact values adjusted for these five classes of CSCs as depicted in Fig. 5. As clearly shown, the privacy impact of risk scenarios is somewhat higher than the service and security impacts. This may be explained by the fact that most of the risks directly or indirectly may impact an asset related to privacy. Since the higher order CSCs (i.e., C S C ₅ is the highest order and C S C ₁ is the lowest order CSC) have only a subset of the assets owned by the lower order CSCs, their adjusted impact values are predictably lower.

Fig. 5

Adjusted impact δ _i for five classes of CSCs

In Figs. 6, 7 and 8 the risk levels for various CSC classes and three CSPs, which have the lowest, average and the highest adjusted probability, are illustrated. The darker part of the figures shows the CSC-CSP pairs with medium level of risk, and the lighter part shows the CSC-CSP pairs with low level of risk according to the ENISA’s qualitative risk scale. The vertical axis of the figures gives the risk level according to the CARAM’s higher resolution scale as in Fig. 2. We cannot observe any CSC class and CSP (the CSPs from STAR) pair that have high level of risk. This is as expected because all the CSPs in STAR apply at least half of the controls from the CSA CCM. The effects of measures are also assessed in the cloud risk assessment of CNIL [16]. Almost the same shift in risk levels is addressed in [16]. CARAM’s difference from CNIL’s risk assessment is that CARAM can take into account the effects of measures for specific CSC-CSP pairs. In Figs. 6, 7 and 8 it can be observed that privacy risk is higher than the security and the security risk is higher than the service risk. The privacy risk is low only for C S C ₄ and C S C ₅ when the CSPs with adjusted probability value below the average. For all the other types of CSC-CSP pairs, the privacy risk is medium.

Fig. 6

Privacy risk for CSP-CSC pairs

Fig. 7

Security risk for CSP-CSC pairs

Fig. 8

Service risk for CSP-CSC pairs

Figures 7 and 8 indicate that the security and service risk levels for almost all the CSPs in STAR is low for C S C ₃,C S C ₄ and C S C ₅. When the CSC uses cloud also for service assets (i.e., C S C ₃) the security and service risks for CSPs with average or above adjusted probability becomes medium. The security and service risks for C S C ₁ and C S C ₂ are always medium for the CSPs in STAR.

Limitations

There are a few limitations that may impact the accuracy of the results mainly stemming from the analyzed input data, some of which were also outlined in [5]:

• Vague formulation of the CAIQ answers provided by the analyzed CSPs: some CSPs avoid direct yes/no answers to the CAIQ questions and use generic wording instead;

• Possibility for deliberate misinformation in the CAIQ answers provided by the analyzed CSPs: CSA has a process of reporting misinformation in CSA STAR [4];

• Ineffective implementation of the security controls by the analyzed CSPs: only 3 of them have third party certification from CSA and CSA does not provide a detailed breakdown of scores for each control. To address this additional methods for evaluating control effectiveness are required, e.g. penetration testing or analysis of previous incidents (see [24, 27] for example approaches);

• In the CAIQ v1.1 [25] there is a misalignment between the description of the security controls and the actual questions that are querying their implementation; this seems to be addressed though by CSA in the newer version of CAIQ.

Notwithstanding, all results exposed in this paper are verifiable and reproducible. Variations can be introduced according to the interpretation of the cloud provider answers. We favoured an impartial, automated classification of the answers based on our training set. Smaller scale uses of CARAM can reach much better precision in the case of manual classification of the CSP answers to the CAIQ questionnaire - which is time consuming but doable and even advisable for some categories of CSCs. Such manual check of the CSPs security practices can sometimes be done in coordination between CSP and CSC, what would lead to more transparency. As big cloud players are unlikely to invest in such reviews, CSCs can perhaps increase the pressure for (automated) continuous monitoring of security controls, which would provide more visibility to them on the security operations of the providers.

Conclusion

CARAM is a qualitative and relative risk assessment model for assisting CSCs to select a CSP that fits their risk profile the best. It is based on the existing frameworks such as ENISA, CAIQ and CNIL and complements them to provide the CSC with a practical tool. It is a risk assessment approach such that evaluation is carried out for a specific CSC, which means assessment for each CSP-CSC pair is for that pair and not generic. Moreover, the model can be easily adapted to assess further risks scenarios and/or security control frameworks, might they change in the future. These multiple features make CARAM unique with respect to the state of the art in risk assessment techniques and also among other works proposing cloud security metrics.

We have implemented a Proof-of-Concept prototype as part of the Data Protection Impact Assessment tool developed in A4Cloud project [28, 29]. The tool asks a (potential) CSC to select a CSP from a given list of around 50 providers, which answered to the CAIQ and evaluates a risk landscape of 35 risks from Table 1 grouped into 3 categories: service, security and privacy using the described methodology. Also the tool allows the CSCs to compare the risk profiles of any two providers, thus helping to select the most suitable CSP from the security point. We performed the analysis of the risk profiles for 44 CSPs from STAR and 5 imaginary classes of CSCs illustrating the coverage of security controls by the different CSPs.

Endnote

¹ The answers concerning the SAP HANA Enterprise Cloud were only available to customers on demand.