Designing an integrated socio-technical behaviour change system for energy saving

Abstract

Stimulating households to save energy with behaviour change support systems is a challenge and an opportunity to support efforts towards more sustainable energy consumption. The approaches developed so far, often either; do not consider the underlying behaviour change process in a systematic way, or do not provide a systematic linking of design elements to findings from behaviour change literature and the design of persuasive systems. This paper discusses the design and evaluation of a holistic socio-technical behaviour change system for energy saving that combines insights from behavioural theories and the persuasive system design in a systematic way. The findings from these two streams of research are combined into an integrated socio-technical model for informing the design of a behaviour change system for energy saving, which is then implemented in a concrete system design. The developed system combines smart meter data with interactive visualisations of energy consumption and energy saving impact, gamified incentive mechanisms, energy saving recommendations and attention triggers. The system design distinguishes between a version with non-personalized energy saving tips and a version with personalized recommendations that are deployed and evaluated separately. In this paper, we present the design and evaluation results of the non-personalized system in a real-world pilot. Obtained results indicate reduced energy consumption compared to a control group and a positive change in energy knowledge in the treatment group using the system, as well as positive user feedback about the suitability of the designed system to encourage energy saving.

Introduction

Meeting the European targets for a reduction of CO2 emissions by 2030 (40% compared to 1990) and energy savings (27% compared to “business-as-usual”) (2030 Energy Strategy 2014) requires extensive changes in consumption behaviour of European citizens. Previous studies on the effect of behaviour change interventions on energy consumption report energy savings of 4%-12% on average (Nachreiner et al. 2015), but also point out a number of limitations and issues related to the persistence of behaviour change over time. Overall, the current body of knowledge on determinants and processes of behaviour change for environmentally conscious behaviour (Steg and Vlek 2009) and research from the design of persuasive systems (Oinas-Kukkonen 2013), provide good groundwork for designing systems to support behaviour change. However, the combined consideration of these findings in the development of technological solutions for stimulating energy saving has been rather limited and few approaches have systematically based their design on a theoretically-grounded model. In particular, none of the existing approaches and studies have been able to validate in a real world pilot a theoretically grounded design of a holistic behaviour change system for energy saving or provide specific recommendations for the design of such systems for various types of users.

While many approaches explore the use of consumption feedback or the use of game-like motivational elements, recent research both in energy and related fields (e.g. water saving) suggest that such individual elements alone are not suitable for inducing a durable change in behaviour (see (Nachreiner et al. 2015; Novak et al. 2018) for an overview). Recently, several European projects have started systematically investigating models where different types of elements are combined in an integrated behaviour change system (Tisov et al. 2018), although few results regarding their theoretical approach and empirically validated findings are currently published. In our view, energy consumption should be considered beyond an individual decision framework, as a complex socio-technical process that takes into account social norms, technologies and infrastructures. The demand for energy is therefore indirect, created by services such as comfort, which are in turn provided by devices and infrastructures (Shove 2003a, b), and is “systematically configured” over the long term (Van Vliet et al. 2012). When it comes to energy savings we propose that effective and sustainable behaviour change cannot be achieved by a single intervention impacting a specific attitudinal or behavioural variable, but requires a holistic socio-technical approach that uses a combination of individual enablers, mechanisms and techniques, and aligns technological enablers with suitable models of behaviour change. While such integrated socio-technical systems for behaviour change in energy saving are available in theory, they have not been validated in real-world pilots.

In this paper, we propose a theoretically-grounded design of a holistic socio-technical system supporting behaviour change for energy saving, which combines smart meter data with interactive consumption visualizations, gamified incentives, energy saving recommendations and attention triggers. We present the theoretical background, derive the socio-technical integrated behaviour change model for energy saving and then describe the design and technical implementation of the model in a concrete system that provides different types of incentives (virtual, physical) and is adaptive to the various types of user motivations. Finally, we discuss the results of its evaluation in a real-world pilot including impact on energy consumption, analysis of user activity and evaluation with the end users.

Behavioural change systems for household energy saving

One stream of research on behaviour change with respect to energy savings considers behavioural theories that aim to identify antecedents of pro-environmental attitudes and behaviour and/or describe an associated behaviour change process (Steg and Vlek 2009). Frequently cited behavioural theories exploring the adoption of new behaviours by users and applying them to energy saving include the Ajzen (1985) theory of planned behaviour and Schwartz (1977) norm activation model. Studies also recognize a wide range of factors determining individual household energy consumption behaviour, most frequently subdivided into socio-economic variables, psychological factors and external contextual and situational factors (see Frederiks et al. (2015) for a review). Although some factors have been found to be better predictors of energy savings than others, these findings are not consistent across time, context, and sample type of participants and studies (Frederiks et al. 2015). Additionally, very few studies have found the interventions to be effective or achieve substantial behavioural changes, mainly due to the fact that they have not appropriately tested causal relationships (Frederiks et al. 2015).

Another research stream deals with the actual design of behaviour change support systems that can aid in inducing users to change behaviour, drawing on generic design frameworks (Fogg 2009; Oinas-Kukkonen 2013). These systems have emerged in different domains (for a review, see Hamari et al. (2014)), including various kinds of pro-environmental behaviour, such as energy saving (Shih and Jheng 2017). Most authors agree that effective designs of behaviour change systems in the energy domain should incorporate different types of feedback and analysis options (e.g. allowing users to access historical series of consumption data or provide them with social comparisons). Feedback as a consequential intervention tool helps individuals associate their actions with outcomes (Fischer 2008) and shapes behaviour by breaking consumption habits (Abrahamse et al. 2005). Such feedback can be data-oriented (e.g. bar or pie charts (Monigatti et al. 2010)), connected to the real consumption context (e.g. floor plans (Monigatti et al. 2010)), metaphorical (Monigatti et al. 2010), playful and ambient (Be aware project 2008; Wemyss et al. 2016), or connected to nature or animal habitats (eco-visualization (Gustafsson et al. 2009a, b). Energy consumption feedback holds the promise of reducing energy consumption by 4%-12% on average, with peak savings going beyond 20% (Nachreiner et al. 2015; Tiefenbeck et al. 2019). However, feedback is regarded as a double-edged sword: some authors find that behaviour change is likely to be reversed once feedback is no longer provided (Allcott and Rogers 2014). Others found that feedback should be coupled with motivational techniques and energy saving advice (Nachreiner et al. 2015).

To be effective, incentives must consider different consumer types and needs, be presented at the right moment and provide actionable suggestions tailored to a given user and context (Novak et al. 2018). A new direction is the use of personalized recommendations employing machine learning techniques to identify different consumer classes, create user models and map appropriate actionable suggestions in a personalized manner (Novak et al. 2018). While the combination of consumption visualization and feedback with recommendations is promising, little evidence is available regarding the most effective combinations of feedback, or how these behavioural changes can be sustained over time.

Gamification approaches and social strategies are often employed in an attempt to induce energy-efficient behaviour with various degrees of success (Johnson et al. 2017; Abrahamse et al. 2005). By adding game-like elements to an otherwise pragmatic application, users may become more engaged with the application and thus more likely to change their behaviour (Geelen et al. 2013). Social rewards can be induced by allowing users to compare game achievements or energy saving achievements against other users (e.g. leaderboards reflecting users ranking (Johnson et al. 2017)). As an alternative to competition, cooperative approaches supported through e.g. virtual social rewards (Johnson et al. 2017) have also been explored. However, in most of these approaches, actual impact on energy use was not measured or unreported.

The majority of the described research has focused on influencing behavioural determinants (e.g. attitudes, behavioural intentions) to stimulate energy saving without considering the stages of behaviour change and how intervention strategies must be tailored to accommodate individuals as they fluctuate between these stages (van der Werff and Steg 2015). But there is growing awareness that interventions and incentives need to be provided based on an individual’s stage in the behaviour change process (He et al. 2010). While early attempts have been made to adopt this approach to feedback systems for natural resource consumption (He et al. 2010; Geelen et al. 2013; Novak et al. 2018), its systematic implementation into a real-world system and related empirical evaluation has yet to be done. The approach and model that we propose in the next section is an attempt to design a holistic behaviour change model for energy saving that integrates the insights from the trans-theoretical model of behaviour change (Prochaska and Velicer 1997), its adaptation to pro-environmental behaviour by (Bamberg 2013) and water saving (Novak et al. 2018), and the mapping of various incentive mechanisms (such as feedback and recommendations) to appropriate stages of the behaviour change process.

Socio-technical behaviour change model for energy saving

The underlying assumption of behaviour change systems (Oinas-Kukkonen 2013) is that behaviour change can be initiated by stimulating psychological factors that influence specific behaviours (Hamari et al. 2014). Accordingly, a change in energy consumption behaviour can be stimulated through a combination of specific incentive and persuasion strategies that address underlying psychological determinants. However, impacting the attitudes and the intention to change behaviour may not lead to actual change in behaviour (intention-behaviour gap). Individuals also have to develop skills and strategies needed to implement new behaviors (Bamberg 2013). Thus, behavioural change can be seen as a transition through a time-ordered sequence of several stages, such as in the trans-theoretical model of behaviour change (Prochaska and Velicer 1997) or the model of action phases (Heckhausen and Gollwitzer 1987). The trans-theoretical model of behaviour change, originally from the health domain, explains behaviour change through five consecutive phases spanning from raising awareness (’pre-contemplation’) to creating new habits. Each of the stages follows distinct goals, and as a result, requires different motivational drivers. The trans-theoretical model has been applied to water saving (Novak et al. 2018) and a similar model of action phases, the MAP model, has been applied to studying environmentally harmful behaviours (Bamberg 2013).

We model behaviour change as a multistage process using a slightly adapted trans-theoretical model of behaviour change. Following Novak et al. (2018), the first two phases, the pre-action and action phases are merged into one renamed phase, “pre-contemplation”, to reflect the limited cognitive effort usually invested in energy consumption behaviour. The process of behavior change is not always linear, as users can relapse to earlier stages. The resulting socio-technical behaviour change model is conceived to support all phases of the change process with an integrated incentive model for energy saving that combines different incentive mechanisms matching each phase. Although the original trans-theoretical model of behaviour change gives suggestions as to the motivational goals that need to be achieved in each stage, it does not specify in detail the social, cognitive, and affective factors and processes that impact the formation of attitudes and intentions in each of the stages (Bamberg 2013). But the knowledge about these factors is critical for developing the incentive mechanisms to impact the formation of attitudes and intentions in each stage. Therefore, building on Novak et al. (2018) we first specify what motivates an individual in each stage to invest cognitive effort into thinking about the behaviour and potentially changing it by using theories from environmental and cognitive psychology. We then use the input from the design of persuasive systems research to identify which incentives need to be provided to the users to increase their motivation to save energy in each stage. The resulting behaviour change model for energy saving is given in Table 1.

Table 1 Socio-technical behaviour change process model for energy saving

Integrated incentive model for household energy saving

Based on the presented socio-technical behaviour change model, we develop more detailed incentive elements for designing a concrete behaviour change system for stimulating household energy saving that addresses all stages of the behaviour change process. The designed elements include interactive energy consumption visualizations, energy saving tips, goal setting and gamified incentives (virtual, social, tangible) and are implemented in a concrete system.

Interactive feedback visualizations with goal-setting for energy saving

Multiple studies show that feedback has the potential of influencing underlying beliefs regarding energy consumption and attitudes towards energy saving (Steg et al. 2014; Tiefenbeck et al. 2016; Novak et al. 2018) at different stages of the behaviour change process. But the design of effective feedback visualizations for energy saving is not trivial. Energy consumption behaviour is abstract, non-sensory and of low personal relevance to most individuals (Karlin et al. 2015). The abstract units of W and kWH are difficult for household consumers to understand (Karjalainen 2011). Consumers have different environmental goals and values (Lindenberg and Steg 2007), as well as different needs regarding energy consumption feedback (Gölz and Hahnel 2016), at different stages of the behaviour change process (Bamberg 2013). To address these challenges, an adaptive metaphor-based visualization approach can be used, helping users interpret complex numerical, and abstract information (Monigatti et al. 2010; Froehlich et al. 2009; Hackenfort et al. 2018).

To set and monitor the achievement of an energy saving goal for a current month, a battery metaphor (Fig. 1a) is used. It communicates the notion of energy as a limited resource that should not be wasted. Users can set their monthly saving goal relative to their baseline consumption in the same month of the previous year (20% by default). The user can then monitor the achievement of the goal in real time: as energy is consumed, the battery depletes. Consumption alerts are provided as color-coded normative messages, which display the user’s progress in relation to the set goal (green motivational message when the user can still achieve their goal, orange warning message when the user is close to not meeting their goal, red message stating that the user cannot achieve their goal anymore). As consumption feedback should be actionable (Micheel et al. 2015), tapping on the ‘Learn how’ button leads directly to the tips page. This visualization is mainly intended for users in the pre-contemplation phase, to raise awareness of negative behaviour, and users in the contemplation phase, to stimulate action through goal-setting and monitoring.

Fig. 1

Metaphorical feedback visualizations. a battery metaphor to set and monitor an energy saving goal. b monetary impact visualization. c environmental impact visualization

To show the effect of energy savings in a way related to everyday life, the impact visualizations display achieved energy savings using three metaphors drawn from goal framing theory (Lindenberg and Steg 2007): monetary impact, environmental impact (CO ₂ emissions) and hedonic value. The monetary impact metaphor (Fig. 1b) is depicted as a piggy bank, the environmental impact metaphor (Fig. 1c) is represented by trees corresponding to saved CO ₂ emissions, and the hedonic value metaphor displays jars filled with balls and badges which are earned after a user saves energy (user’s gamified achievements). As users have different goals for using energy feedback systems (Gölz and Hahnel 2016), they may be more motivated by a specific impact view. Therefore, the impact view is personalized based on people’s main motivation for saving energy (solicited from a questionnaire). However, according to Goal Framing theory, individuals may have several competing motivations, therefore users are able switch between these three different visualizations. These visual metaphors are intended to target users in the pre-contemplation stage to illustrate the impact of energy saving, as well as users in the action stage to create a sense of achievement that motivates users to continue saving energy.

Raising awareness about the trade-off between energy saving and comfort and suggesting recommendations for energy saving actions that do not jeopardize the user’s comfort level, is a key challenge to improve user attitudes and acceptance of behaviour change systems. For this, we use a simple visualization displaying the user’s achieved savings alongside their measured comfort level (determined using temperature, humidity and luminance values) that was maintained during that month (see Fig. 2a). If savings were achieved and the comfort level was maintained, values are presented in a green box (alternatively, a red box provides a warning). An initial assessment of the comfort level is inferred automatically by the system (using sensors installed in the users’ homes) and is refined using explicit comfort feedback from the user. While some efforts have previously been made at visualizing comfort level for e.g. building managers in an office setting (Schiavon and Altomonte 2014), the consideration and visualization of comfort for households is under-investigated and is a novel element of our approach. By showing that energy savings can be achieved with only a small reduction of comfort, the display of the comfort level alongside energy savings can induce the users in the contemplation phase towards energy savings (cf. Table 1). Finally, the detailed consumption view provides data-affine users (Micheel et al. 2015) with a bar chart consumption visualization, allowing users to check their energy consumption at different time scales (e.g. weekly, monthly) and compare it against their historical average.

Fig. 2

Feedback visualizations and gamification. a comfort vs. energy savings visualization. b energy saving tips. c leaderboard element

Energy saving tips and recommendations

Educating users on how to perform the desired behaviour is a well-known behavioural change strategy (Oinas-Kukkonen 2013; Michie et al. 2011). Providing concrete actionable suggestions strengthens the sense of responsibility (Schwartz 1977), promotes self-efficacy and the perception of behavioural control (Ajzen 1985), and can also directly strengthen the intention to implement the desired behaviour (Bamberg 2013). Offering recommendations for energy saving actions allows users to learn how to save energy (Gölz and Hahnel 2016). Our system design foresees two types of recommendations (see Fig. 2b): a list of generic energy savings tips and personalized recommendations (derived from energy behaviour data, user-generated data, and sensors). In each case, users are asked to commit to execute the recommended action (’Ok, will do’), state that they are already doing it, or indicate that it is irrelevant to them. Please note that the system design presented and evaluated in this paper considers only the non-personalized version.

Gamified incentives and rewards

A review in Johson et al. (2017) found that gamified applications can positively influence energy behaviour, stimulate learning, and improve the user experience. Game-based strategies often motivate users through extrinsic rewards, but the removal of these rewards can result in the dissipation of the desired energy saving behaviour(Richter et al. 2015). However, motivation to allocate cognitive resources is not only driven by rational thoughts, but also by hedonic values (Steg et al. 2014). Appealing to the hedonic values with playful interaction can motivate user engagement beyond external rewards. In our approach, users are motivated to perform recommended actions via two types of gamified rewards: symbolic (points, badges) and tangible (external prizes). These are reinforced by game-mechanics such as goal setting (meeting personal savings goals are rewarded with bonus points), competition (vouchers are awarded monthly to the best performing households) and social comparison (e.g. collecting points for performing energy saving actions improves a user’s leaderboard rank). An example of the leaderboard is displayed in Fig. 2c. These features are intended to support user engagement in the action phase and reduce the risk of relapse by stimulating the need for competition and/or the need to demonstrate one’s achievements to others (Reiss 2014).

Notifications and attention triggering

Given the generally low involvement of users with energy consumption (Karlin et al. 2015), and the competition with other applications, there is a risk that users lose interest over time (Hargreaves et al. 2013). Therefore, push notifications are sent to users’ smartphones comprising tips for energy saving, warnings when they are close to missing their energy savings goal, announcing gamified achievements (e.g. earning a new badge), and reactivation messages when activity becomes low. Such notifications are especially helpful in the monitoring stage to re-activate users’ attention (e.g. after relapse to old behaviour). Repeatedly triggering user’s attention to an energy saving application is a difficult challenge, since a balance must be struck between keeping users engaged without irritating them. Consequently, users can limit the number of notifications per week by choosing a low (maximum of 2 per week) or high frequency setting (maximum of 5 per week), or completely disable them.The timing and type of notifications to be sent to the user in this non-personalized system is determined in a fixed manner based on input obtained from user requirements workshops (without considering different user types, actual user presence at home, or their current activity).

System implementation

The technical implementation of the system is based on a layered architecture, in which each layer is designed to address a specific task. The system architecture comprises three layers: the data layer, the business process layer, and the consumer layer. The data layer is responsible for the data acquisition, pre-process and storage of the smart meters and indoor climate sensor readings; it also exposes services for the upper-level layer to access the consolidated data. The business process layer is responsible for the execution of the business logic; it exposes business services for the Consumer layer. In the non-personalized system version it comprises:

• The Gamification Engine: a configurable gamification rule engine, designed to provide services to the Consumer layer, specifically to the end-user mobile application through a service API. The exposed services allow the user application to record user activity on the platform and assign points for completed tasks, assess energy saving goals, track user achievements, manage rewards, and notify users about relevant platform events.

• Inference Engine: a machine learning component that processes sensor and consumption disaggregated data to infer indicators like user comfort level and household occupancy.

• Notification Engine: a component that provides notification services to other Platform components. It schedules notifications pushed by other components and delivers them based on user-defined preferences; it exploits the Google Firebase Cloud Messaging service, for the delivery of mobile alerts.

• Service Integration and Orchestration Component: this component synchronizes the execution of the processes that the different components require, and provides services for the communication between components.

Finally, the consumer layer comprises the client-side application for publishing the services exposed by the Business Process Layer, and the mobile App for displaying the consumption and sensor data and for integrating the gamification services.

The end-user application is divided into two components: a web application with responsive web design, which handles the communication with the business layer and the rendering of the visualizations, as well as client applications for mobile devices (Android and iOS), which wraps the web application and enables access to the native features of the devices. The responsive web application was developed following a Model-Driven Engineering approach: models representing the organization of the interface, its content and the user interaction events were specified using the OMG Interaction Flow Modeling language (IFML) and WebRatio (WebRatio 2019), a model-driven development environment based on IFML, which automatically transforms these models into executable code using industry standard frameworks, such as Struts, JSP, CSS and jQuery. The client wrapper applications are developed using the native tools for each mobile platform, Java for Android and Swift for iOS; they consist mainly of a WebView, i.e., a simple web browser whose behaviour can be customized, that enables access to the end-user web application, enriched with native functions such as notifications, safe storage of user data and passwords etc.

Evaluation in a real-world pilot

Research Methodology

The system was designed with a user-centered design process, starting from eliciting the requirements of the users in a series of workshops, developing mock-ups, testing them in online crowd tests, as well as prototyping before final system development. To assess the proposed model and system design we evaluated behaviour change for a set of residential households in a real-world pilot in Switzerland¹. To this end, we applied an experimental design with a treatment and a control group. The treatment group used the developed system, whereas the control group wasn’t exposed to this intervention. People signed up on a self-interested basis to the control and the treatment groups. We then assessed changes in energy consumption between the treatment and the control group and the interaction of the treatment group with the system. Additionally, a survey-based approach has been applied to capture behaviour and awareness change due to the intervention. Specifically, we measure changes in attitudinal and behavioural measurements before and after exposure to the system. This includes changes in behaviour, knowledge and attitudes, as well as the user evaluation of the usefulness and motivational effect of the system.

Energy savings as a result of system usage

Sample characteristics

The experiment in this pilot has been organized by selecting a treatment group and a control group. The treatment group has been composed of 66 self-selected households from the district of Contone in the municipality of Gambarogno located in the Ticino Canton (southern Switzerland). A focused control group has been composed of 34 households in the districts of Gerra, Magadino, Piazzogna, Quartino and Vira, all of them in the same municipality of Gambarogno. The members of this focused control group agreed to take part in a series of questionnaires during the whole project. For the purpose of the consumption evaluation, we also have access to consumption data of a much larger control group of users (558 households) located in the same district, who are neither taking part in the project nor answering the questionnaires. This larger group is used only for a statistical assessment of the energy saving results.

The structure of the treatment group comprises 50% families with kids, 31% couples, and 9% reporting themselves as single. The control group comprises 28% families, 47% couples, and 25% reporting as single. The treatment group lives in apartments (21%), terraced houses (18%) and single homes (62%), while the control group distribution is as follows: 6% live in apartments, 35% live in terraced houses, and 59% live in single homes. The average monthly consumption for the treatment group over the baseline period spanning May 2017 until April 2018 was 574.5 kWh per month. The average monthly consumption for the focused control group in the same period was 625.6 kWh per month, and 450 kWh per month for the larger control group.

Results on energy saving

The results in this section cover the treatment period for the non-personalized system spanning June 2018 until February 2019. Therefore, we compare the consumption of the treatment period (June 2018 – February 2019) against the same sub-period during the baseline period (June 2017 – February 2018). The following formula is used to compute the consumption percentage change:

$$\begin{array}{*{20}l} savings = \frac{\sum_{i=1}^{N} \frac{x_{i}^{base}-x_{i}}{x_{i}^{base}}}{N} \end{array} $$

(1)

Where N is the total number of users, x_i is the consumption of the i^th user during the treatment period so far, while x_i^base is the consumption of the same user during the baseline period.

We find that for the treatment group, the average consumption has decreased on average by 5.81% (standard deviation 18.81%). The same analysis for the focused control group shows that the average consumption has increased by 1.33% (standard deviation 15.22%) after removing an outlier that increased the consumption by 168%. The percentage variation in consumption of the treatment group users can be seen in Fig. 3. Apart from two users who, respectively greatly increased and greatly decreased their consumption (users 86 and 87), the other users in general managed to reduce their consumption (positive increments mean savings).

Fig. 3

Percentage variation in consumption of treatment group users

An F-test shows that the variance of the data in the focused control group and in the treatment group can be assumed to be similar (P(F≤f)=0.09) so we performed a two sample t-Test which returned non-significance in the difference of the two means, as reported in Table 2. We thus accept the hypothesis that the average savings in the focused control group and in the treatment group are similar.

Table 2 Two sample t-Test between treatment and focused control group

Given the limited size of the focused control group, we also compared the average percentage savings of the treatment group against the average percentage savings of the larger control group from the Contone population. In this case, the variances of the two samples do not pass the F-test, therefore we employ a Welch t-Test to compare the average savings in the two samples. The average percentage savings in the larger control group are equal to 0.2% with a variance equal to 6.4%, which confirms the stability of the consumption of this group of households as expected, given that no major exogenous factors have affected the energy consumption in Contone over the study period. The results presented in Table 3 show that we can reject the hypothesis that average savings between the larger control group in Contone and the treatment group are similar.

Table 3 Welch t-Test between treatment and larger control group

We have then analyzed possible relationships between the usage of the system (user activity in the app) and the obtained energy savings. For this, we have gathered the average number of actions performed in the app per week. Actions can be diverse types of interactions within the app, such as checking current energy consumption, reviewing a tip, updating the monthly goal and so on. Figure 4 shows the average number of weekly actions on the y-axis with consumption variation on the x-axis. We have then performed a K-means clustering on the data, setting the number of clusters to 4. The result is depicted in Fig. 4, where a group of users (coloured green) who display a rather intense weekly activity on the app can be noticed. These users do not necessarily show higher savings compared to other clusters, but their savings are consistently positive, which is not the case for the other clusters (e.g. the larger purple cluster has a positive energy saving on average, but contains a subset of members who actually increased their energy consumption).

Fig. 4

K-means clustering of users by consumption variation & app activity

Evaluation of system usage

In the treatment group, 66 users downloaded and registered to the mobile app during the first two months of the pilot. Among these users, 52 logged in more than once and were therefore considered to be active users. This subgroup’s application usage data was the subject of further analysis. Figure 5 shows the frequency of new sign-ups per month, the number of active users each month (those who logged in more than once over the course of the pilot) and the number of users who logged in each month. From Fig. 5 we see that a majority of active users (in a range of 54%-100% depending on the month) are logging in to the application at least once every month. This is a healthy level of user engagement.

Fig. 5

Usage statistics: user growth and engagement by month

To estimate how frequently the users interacted with the main visualizations and pages of the application, we calculated the monthly average frequency of interaction per user per month of pilot membership². The results are shown in Fig. 6. The results show that the users interacted the most with the Savings and Goal page which they accessed on average over 14 times a month. The second most frequently used page was the Comfort page, with an average of 10.7 accesses per month. Similarly, the Tips page (10 accesses per month)³ and Achievements page (9.6 monthly accesses), were also frequently used. The Consumption page with the detailed time-based overview of the consumption and the Leaderboard were used less frequently, though still more than once a week on average. In contrast, the Impact and Comfort Feedback pages were accessed less, about once a week on average (3.9 monthly accesses).

Fig. 6

Average monthly app interactions per user and app page

To shed more light on the interaction with the three types of impact visualizations, we analyze how many users accessed the default impact visualizations in the application (determined from their stated motivation to save energy) and how many of them changed the default impact visualization to another metaphor. Among the users who indicated their preference to save energy, 88% were motivated by environmental considerations and 6% each by monetary or hedonic goals. Results show that most users accessed the impact visualization reflecting their stated preference at least once; for example, 95% of users who cited the environment as their preferred motivation for saving energy accessed the environmental visualization. Among users who had the environmental visualization set as the default, 50% changed to the monetary and 55% changed to the hedonic visualization at least once.

Evaluation of the change in antecedents of behaviour change and user acceptance

Measurement instruments and sample

In order to assess differences in the antecedents of behaviour change, we measured three constructs with validated measurement instruments, adapted to the context of energy saving. This was done using 3 items measuring intention to save energy (7pt Likert) in line with Ajzen (2002), 4 items measuring perceived behavioural control (7pt Likert) in line with Thøgersen & Grønhøj (2010), and 5 items measuring environmental knowledge (5pt Likert), with an adapted scale from Kaiser and Frick (2002)⁴. These items were measured both for the treatment and the control group, with the main hypothesis that due to the intervention with the developed system, these behavioural determinants would improve in the treatment group more than in the control group.

To evaluate user acceptance of the system, we used the items from the UTAUT model (Venkatesh et al. 2012). Specifically, we used 2 items measuring performance expectancy, 4 items measuring effort expectancy, 3 items measuring hedonic motivation, and 3 items measuring behavioural intention, all on a 5pt Likert scale (see Table 5). We also measured the motivation of users by the system in general and by specific elements (such as the battery, and the environmental and monetary impact views) in particular (these items were self-developed).

For the analysis of the changes in behavioural variables, we can only include users who filled out both the baseline questionnaire (given at the beginning of the pilot), and the assessment questionnaire after the pilot had been active for 4 months. Therefore, the results presented below come from 40 users in the treatment group, of whom 66% are male and 34% are female; 25% are between 31-40 years of age; 28% between 41-50, 25% between 51-60, 15% between 61-70, and 5% are over 70. In the control group, there are 26 users, of whom 72% are male and 28% are female, 18% are between 31-40 years old, 23% between 41-50%, 23% between 51-60, 14% between 61-70 and 23% are over 70.

Results of impact on antecedents of behaviour change

We measured three antecedents of behaviour change: energy knowledge, perceived behavioural control and behavioural intention. For behavioural control and behavioural intention, we computed averages for the items mentioned in the previous section, whereas for the energy knowledge we computed the sum, as the construct is formative. These constructs were measured at the beginning of the trial and after four months⁵ in order to assess any change resulting from the intervention. In Table 4 we show the means and standard deviations of the change in the behavioural determinants computed by subtracting the score after four months of trial from the score measured prior to the beginning of the trial. In this table the Shapiro-Wilk test, used for small sample used for small sample sizes (Shapiro and Wilk 1965) reveals that all of the scores are normally distributed, except for the scores of energy knowledge in the intervention group.

Table 4 Means, deviations and statistical test results for the change in attitudes, intention and knowledge due to the intervention

Table 5 Descriptive statistics of the main measurement constructs evaluating user acceptance of the enCOMPASS app (all measured on a scale from 1 to 5)

Scores measuring energy knowledge, show a change from 14.2 to 16.8 in the treatment, and from 18.30 to 16.81 in the control group. The scores provided by participants in the treatment group have increased, whereas the scores of the control group have decreased. As the scores of the treatment group are non-normally distributed, a Mann-Whitney test shows that people in the treatment group (Mdn=1) managed to increase their knowledge compared to the control group (Mdn=-0.5); U=735, p <0.01, r=0.35, which is considered a medium size effect (Cohen 1992).

As evident from Table 4, other behavioural determinants measured were not substantial; the independent samples t-test revealed that there are no significant differences in the mean changes in perceived behavioural control between the treatment and the control group (t (64)=1.18, n.s.) as well as no differences in behavioural intention (t (64)= -1.33, n.s).

Results regarding user acceptance and motivation

The results in Table 5 reveal that the application shows above average user acceptance for all measured constructs of UTAUT model: the application was considered moderately useful in daily life (average performance expectancy score of 3.6/5) and easy to use (average effort expectancy score of 3.75/5). Users also reported that the application is fairly fun to use (average score of 3.4/5 for hedonic quality) and that they intend to use it in the future (average score of 3.7/5). The average scores of user responses for the evaluation of the developed visualizations are presented in Fig. 7. They show that users evaluated positively all aspects of the developed visualizations.

Fig. 7

User evaluations of the visualizations

Discussion and conclusions

We have presented a socio-technical model and system for behaviour change for energy saving and evaluated it in a real-world pilot. The results suggest that the designed behavior change system is suitable to stimulate energy savings among consumers. The observed positive effects on energy consumption of the treatment compared to the larger control group and a change in user’s energy related knowledge suggest that such systems can be successfully applied to motivate users to save energy. Our results show that users who used the application frequently, achieved positive energy savings, compared to users who used the application less regularly. Moreover, the positive user evaluation of the system, the high motivation of users by the developed visualizations to save energy, and the actual active usage of the application suggest that the integrated incentive model motivated users to adopt the proposed system to start changing their behaviour. These results provide both theoretical and practical contributions.

On the theoretical side, we find that the underlying socio-technical model can be applied to study behavior change in the energy saving domain. Research suggests energy knowledge is one of the first antecedents of behaviour change towards energy saving and our results show energy knowledge improving in the treatment group after users participated for four months in the pilot. Proceeding behavioural determinants such as perceived behavioural control or behavioural intention, have yet to show significant change. This suggests that four months is not enough time for the behavioural determinants to change and a post-hoc analysis must be performed to assess whether these determinants show any change in the long-term.

Additionally, our results suggest that the developed interactive visualizations are well suited to motivate users to save energy. Specifically, the battery visualization allowing the user to set and monitor the achievement of the energy saving goal was reported by the users as easiest to understand and most useful. The environmental impact visualization had the most visual appeal and was perceived as the most motivating metaphorical visualization for saving energy. While most users were interested in the visualizations matching their primary motivation, some were also interested in the visualizations matching the other motivational drivers. This confirms that users might have different motivations for energy saving, as suggested by the goal framing theory (Lindenberg and Steg 2007).

Additionally, during the four month trial users were increasingly using the application, motivated by the incentive mechanisms proposed in our model. Specifically, users have been most frequently accessing such pages of the application as: the savings and goal, the comfort, the tips, and achievements (ordered by frequency). The interest in the savings and goal page is justified as this page allows users to monitor their consumption with respect to the energy saving goal - the main aim of the application. The lower frequency of access for Impact visualization pages is surprising, although one needs to consider that accessing these pages also requires more clicks since they follow after the higher-level visualization pages.

On the practical side, the developed incentive model can guide the designers of behaviour change systems in energy use and other sustainable domains to design their systems in such a way that they are easy to use, helpful and engaging for the users. For example, we show that the energy consumption and savings visualizations help users save energy, by e.g. allowing them to set and monitor their energy saving goals. Additionally, providing users with energy saving tips and competition-based achievements can help keep users engaged in the behaviour change process. An overall recommendation for designers is to use different incentive mechanisms (those for different behavior change stages or various motivation types) that can potentially motivate various types of users.