This paper is in the following e-collection/theme issue:

Mobile Health (mhealth) (2956) e-Learning and Medical Education (695)

Published on in Vol 16, No 3 (2014): March

Web-Based Apps for Reflection: A Longitudinal Study With Hospital Staff

Web-Based Apps for Reflection: A Longitudinal Study With Hospital Staff

Web-Based Apps for Reflection: A Longitudinal Study With Hospital Staff

Authors of this article:

Bettina Renner1 ;   Joachim Kimmerle1 ;   Dominik Cavael2 ;   Volker Ziegler2 ;   Lisa Reinmann2 ;   Ulrike Cress1, 3
Article Authors Cited by (13) Tweetations (7) Metrics

    Original Paper

    1Knowledge Media Research Center (KMRC) , Tuebingen, Germany

    2Neurological Clinic Bad Neustadt, Bad Neustadt, Germany

    3University of Tuebingen, Tuebingen, Germany

    *these authors contributed equally

    Corresponding Author:

    Joachim Kimmerle, Dr rer nat

    Knowledge Media Research Center (KMRC) 

    Schleichstr 6

    Tuebingen, 72076

    Germany

    Phone: 49 7071979363

    Fax:49 7071979100

    Email: [email protected]


    Background: Reflection is an important cognitive process in workplace learning; however, it occurs only rarely on its own and therefore needs additional support.

    Objective: In this study, we investigated the effect of software applications (apps) that aim to support reflection on hospital staff’s actual reflection behavior. In doing so, we also analyzed the relationship between reflection and the job satisfaction of health care professionals.

    Methods: Reflective learning was introduced in the ward of a neurological hospital by providing apps that aimed to foster particular aspects of individual and collaborative reflection. Data were collected repeatedly: once before the introduction of the apps and again 2 years after the initial measure. We used a questionnaire with subjective ratings of reflection and job satisfaction. Response rates were 34.4% (167/485) for the first and 40.6% (210/517) for the second measure.

    Results: Collaborative reflection was increased (P=.047) after the provision of the apps (2010: mean 2.84, SD 0.72; 2012: mean 3.06, SD 0.63) in contrast to a control group of other wards of the same hospital (2010: mean 2.68, SD 0.67; 2012: mean 2.63, SD 0.68). In addition, we revealed a positive correlation between collaborative reflection and job satisfaction (r=.61, P<.001).

    Conclusions: The findings provide evidence for an effect of the apps on hospital employees’ reflection behavior. Apps that foster reflective learning can increase health care professionals’ reflection about work experiences and support them in discussing experiences in teams or with their supervisors. The relationship between collaborative reflection and job satisfaction suggests that opportunities for joint reflection on work experiences in a hospital have further impact over and above fostering reflective learning per se. We discuss the limitations of our study and provide suggestions for both future research and the development of Web-based apps.

    J Med Internet Res 2014;16(3):e85

    doi:10.2196/jmir.3040

    Keywords

    computer applications software; mobile applications; learning; education, continuing; job satisfaction; hospitals; longitudinal studies; self report 


    Introduction

    Background

    Learning and the continuing development of employees are important issues in hospitals. Confronted with complex problems and stressful situations, nurses and physicians always need to keep their skills up to date and develop them further [1-3]. This happens not only through formal training but also informally in the daily routine of health care professionals [4-6]. Hospital management needs not only skilled and competent employees but also contented employees, since dissatisfied nurses and physicians are more likely to resign [7,8]. Job satisfaction of health care staff is related to patient satisfaction [8] and patient care (for an overview see [7]) and therefore plays an important role when it comes to the quality of medical care.

    The aim of the present study is to examine an easy and efficient way of supporting reflective learning in hospital staff. For this purpose, we tested the suitability of application software (apps) that support reflection. Our aim was also to examine the relationship between reflection and job satisfaction of hospital staff.

    We present theoretical and empirical considerations of reflection processes in the workplace and of job satisfaction, with each presentation leading to a concrete hypothesis. Then we describe the methods of this longitudinal study and its results. Concluding, we discuss our findings with respect to their practical implications and suggestions for future research in medical informatics.

    Reflection

    One informal way of learning at work is through reflecting on work situations that employees themselves or colleagues have experienced [9,10]. Reflection is a process “in which people recapture their experience, think about it, mull it over and evaluate it” [11]. Reflection is considered to be an important process in organizations, for example, for innovation [12] or for organizational learning [13,14]. Although reflection is quite a natural process [9], people often do not use it consciously or intentionally as a skill to improve learning and performance [15]. Moreover, there are several other preconditions for the successful application of reflection in the workplace. On the part of the individual, not only a lack of awareness of the benefits of reflection, but also an absence of motivation for reflective thinking, as well as a lack of metacognitive skills like self-monitoring and self-evaluating might hinder reflective learning [16]. On the part of the organization, there are also certain conditions necessary to foster reflective thinking. An organization that wants employees to engage in reflection needs to give them opportunities to participate in shaping their work environment, at least in some areas. If an organization is characterized by strong hierarchies and employees are not involved in decision-making processes, the potential for change through reflective thinking is quite limited [16]. In sum, reflection, whether carried out individually or in teams, does not tend to occur by itself [12], rather, it needs support [13].

    One area where reflection and reflective thinking are broadly researched topics is in the health care sector [3,17-20]. A special focus of previous research has been on the education of nurses [17-21] and physicians [1,22-26]. In hospitals, reflection helps the staff combine theory and practice [3,20] and enables them to turn experiences into learning opportunities [27]. In this way, reflection and reflective work behavior foster continuing professional development [28]. Research findings in this field suggest, however, that reflectivity is indeed not an easily applied skill. Instead, guidance and support are needed to achieve higher levels of reflection [21].

    Since reflection is rare, but important and valuable for health care professionals, research needs to find approaches that can support this process [23]. One option that is relevant from a medical informatics perspective is the development of Web-based or mobile apps [29-31] to assist individuals and teams in their reflection processes in hospitals. These apps are supposed to support staff in exercising different aspects of reflection. Apps can remind people to reflect on their experiences and therefore compensate for low awareness or motivation. They can also extend the range of situations and experiences a single person faces, for example, by making additional experiences possible through virtual worlds or computer simulations. Moreover, apps can support collaborative reflection. Through collaborative reflection, people can learn from others and benefit from their experiences [32]. Since memories of experiences can become imprecise over time, it is important to capture them in such a way that they can be used for further reflection [33]. Apps can foster reflection in different ways, by providing experiences (eg, in virtual settings), documenting experiences, triggering reflection about experiences, and supporting communication and cooperation in collaborative reflection. So the first hypothesis of the current study is that (1) apps designed to foster reflection of health care professionals have a positive impact on the reflection activity of hospital staff.

    Job Satisfaction

    One reason why people learn for their job is that the tasks of the job require it. But this is not the only motive. People also learn because it helps them in their career and because they have certain personal goals [6]. These goals comprise, among others, a need for personal achievement and development, and a feeling of autonomy [6]. Job satisfaction can be interpreted as the sum of the job expectations employees have that are met [34]. This means not only monetary expectations but also the wish to reach personal goals of development and achievement.

    Consequently, we expect that job satisfaction is increased by good learning conditions. Some empirical evidence already exists for the potential influence of learning on job satisfaction. In different studies, significant correlations between organizational learning and job satisfaction [35], as well as between workplace learning and job satisfaction [36-38] have been found. Other authors have identified organizational learning culture as a predictor for general job satisfaction [39,40]. There is also support for the assumption of a relationship between learning and job satisfaction in the health sector. A strong correlation has been identified between professional growth and job satisfaction among nurse practitioners [41].

    Because reflective learning is one type of informal learning that enables employees to develop and shape their working environment to a certain degree, we hypothesize that fostering reflection leads to an increase in job satisfaction. So the second hypothesis to be tested in this study is that (2) reflection is positively related to the job satisfaction of hospital staff.


    Study Design and Setting

    A longitudinal study was designed with the staff of a neurological hospital as participants. The study represented a 2 × 2 factorial design with reflection apps and time of measurement as between-subject factors. Due to privacy protection regulations, it was not possible to connect the data of the two measurement points on an individual level. Therefore, time of measurement could not be treated as a within-subject factor. Accordingly, we cannot completely rule out that the samples at the two points of measurement included slightly different individuals. But there was a low turnover rate in this hospital (around 3%); during the study, only two members left the unit that served as the experimental group. Reflection and job satisfaction were measured both before (2010) and after the introduction of the apps (2012).

    The staff in one ward of the hospital participated as the experimental group. Participants in this group had the opportunity to use the apps. The staff in all the other wards of the hospital served as the control group. Participants in this group were not acquainted with the apps. The experimental group were ward personnel caring for patients suffering from acute strokes and other neurological emergencies (“Stroke Unit”). The staff of this ward consisted mainly of nurses and physicians. Participants of the control group were staff either of a ward treating all different kinds of neurological injuries, of a neurological intensive care unit, of a rehabilitation unit, or interdivisional staff.

    Procedure and Material

    In the period between the two data collections, participants were involved in user studies and workshops in order to examine the reflection behavior and needs of hospital staff. Those findings were accounted for in the development of the apps. After the development stage, participants in the experimental group tested the apps in workshops and used them during their everyday work. The developing and testing phases together covered the entire time span between the two points of measurement. During this time, several 1-day workshops as well as two user studies were conducted in order to give the entire staff the chance to participate. The participants were provided with devices to make sure everyone could use the apps and nobody was prevented from participation. The apps addressed the different aspects of reflection as described above.

    DocTrain is an app for mobile phones as well as a Web application. It was developed to support physicians during their specialist training. It reminds its users to reflect upon their experiences and helps them document their tasks and capture and share with their mentor their progress in training elements. The app provides a list of tasks, and the users can connect them via the barcode of a patient case with a particular patient. Users can also add personal notes. Data can then be used to reflect about a case individually or with a mentor [42].

    CLinIC-The Virtual Tutor is a serious game. It was created to help nursing staff and physicians reflect on and prepare for difficult situations with patients. With the help of the game, users are provided with virtual experiences in handling difficult conversations with patients in order to be better prepared for corresponding situations in real life. A virtual tutor provides feedback and advice. Users can also take notes to reflect on them later. The set of tasks in the game is complemented by so-called mini games. These mini games are integrated into the branching stories of the game as a whole. They allow users to practice skills that are required in the health sector, such as recognizing emotional facial expressions and reflecting on them [43,44].

    Finally, the Talk Reflection App allows users to reflect collaboratively on difficult situations with patients and relatives [45]. It consists of a forum where users can contribute experiences with difficult conversations and reflect on them asynchronously together with colleagues. Users can document problematic experiences, share them with others, and add their own thoughts about the situations privately or publicly. Others can then contribute to this topic. In addition, users can document their reflection outcomes [46]. So the app facilitates reflection by capturing data and provides support for coordination and communication in collaborative reflection.

    Data Collection

    A survey with the hospital staff was conducted twice during the study. The first data collection took place in November and December 2010. The second measurement point was October/November 2012. The response rate was 34.4% in 2010 and 40.6% in 2012.

    Participants

    In 2010, 167 employees of the hospital participated in the survey: 21 individuals worked in the Stroke Unit, 132 individuals in other wards of the hospital, and 14 did not indicate their ward affiliation. In 2012, 210 employees answered the questionnaire: 17 members of the Stroke Unit, 164 staff members of other wards, and 29 participants did not indicate their ward. The difference in group sizes between the experimental and the control group is due to the fact that the participants of the Stroke Unit were compared to the staff of all the other wards of the hospital; however, response rates did not differ between conditions (all Ps>.05). Staff consisted of nurses, different kinds of therapists, physicians, and administration staff. Participants not indicating their ward were excluded from further analyses, so 334 individuals were considered for the investigation.

    Instruments

    A 10-item questionnaire was used to measure reflection. Items were created during the research project MIRROR [47] on the basis of those aspects of reflection specified in the literature that were relevant to this study. Particularly relevant are the needs to reflect and to develop through reflection [48]. These aspects concern the motivational side of reflection, since motivation is considered an important precondition for the occurrence of reflection [10]. Another important requirement is the absence of barriers such as time pressure [4,10]. The positive consequences of collaborative reflection in terms of sharing and discussing experiences and different perspectives are described by several authors (eg, [4,9,10,13,49]). Collaborative reflection includes team reasoning, that is, the collective analysis of problems [9,12,50]. Collaborative reflection also includes support by a person with greater experience, such as a mentor or a supervisor [9]. To make sure that the items were comprehensible to the participants, their content was discussed with various hospital representatives beforehand, and some revisions were made [47]. The items that resulted from this procedure are listed in Table 1. Participants rated these items on 4-point Likert scales from 1=totally disagree to 4=totally agree.

    In addition, participants were also asked 8 questions regarding general job satisfaction (Table 2). Participants rated these items also on 4-point Likert scales from 1=totally disagree to 4=totally agree. Moreover, they were asked to indicate their occupation group and the ward they worked in.

    Table 1. Reflection items.
    No.Item
    1I often have the need to think about my work experiences.
    2I am provided with enough time to reflect upon my experiences at work.
    3It helps me for my development to think about my work experiences.
    4We discuss on a regular basis if we work successfully together as a team.
    5If things do not work out as they should, we try as a team to find the reason.
    6We as a team think about how we can learn from past experiences.
    7When I think I did not do a good job, I discuss with colleagues how I could improve.
    8When I think I did not do a good job, I discuss with my supervisor how I could improve.
    9I often think about how we can improve things in our organization.
    10In our organization we are encouraged to reflect upon our experiences at work.
    Table 2. Job satisfaction items.
    No.Item

    		In general, I am satisfied with...
    1…my work situation.
    2…the advanced and professional training.
    3…the work climate.
    4…the communication.
    5…the operational procedures.
    6…the quality of care.
    7…the leadership of my supervisor.
    8…the neurological hospital as employer.

    Statistical Analysis

    To assess the internal consistencies of the scales, we calculated Cronbach alpha values and conducted an exploratory factor analysis for the reflection scale. For testing hypothesis 1, we conducted a contrast analysis [51]. The second time of measurement (2012) in the Stroke Unit ward was tested against the three other conditions (Stroke Unit 2010; other wards 2010 and 2012). For conducting the contrast analysis, we treated the 2 (2010 vs 2012) × 2 (Stroke Unit vs other wards) factors as a 1 × 4 design. The hypothesis that there was a higher level of reflection in the Stroke Unit in 2012 than in all other conditions (A>B=C=D) is represented in the focal contrast with the coefficients 3 -1 -1 -1. We also conducted an analysis of single items using t tests. For testing hypothesis 2, we calculated Pearson product-moment correlations for both the complete scales and for single items. Statistical analyses were conducted using IBM SPSS Statistics 20.

    Ethical Considerations

    Data collection was approved by the hospital’s works council and the data protection officer. Participants signed a consent form that was also approved by the works council and the data protection officer.


    Internal Consistency and Factor Analysis

    The complete reflection scale had an internal consistency of alpha=.84. To further explore the structure of the scale, a factor analysis was conducted extracting two factors, using a Varimax rotation. The two factors explained 56% of the variance that emerged (factor 1: 37%, factor 2: 19%). After rotation, all items had factor loadings of at least .50 on one of the two factors and not more than .40 on the respective other factor. The first factor was labeled collaborative reflection and the second factor labeled individual reflection. The collaborative reflection subscale consisted of 7 items (items 2, 4-8, and 10 in Table 1) and had an internal consistency of alpha=.85. The corrected item-total correlations of these items are presented in Table 3. The individual reflection subscale consisted of 3 items (items 1, 3, and 9 in Table 1) and had only an internal consistency of alpha=.59. Therefore, we had to exclude this subscale from further analyses. The internal consistency of the job satisfaction scale was alpha=.89.

    Table 3. Corrected item-total correlations of the collaborative reflection items.
    ItemCorrected item-total correlations
    I am provided with enough time to reflect upon my experiences at work..38
    We discuss on a regular basis if we work successfully together as a team..66
    If things do not work out as they should, we try as a team to find the reason..68
    We as a team think about how we can learn from past experiences..75
    When I think I did not do a good job, I discuss with colleagues how I could improve myself..59
    When I think I did not do a good job, I discuss with my supervisor how I could improve myself..64
    In our organization we are encouraged to reflect upon our experiences at work..62

    Reflection

    To test whether there was more reflection in the Stroke Unit in 2012 compared to 2010 and to the other wards (hypothesis 1), we conducted a contrast analysis with collaborative reflection as a dependent variable. To check whether there was systematic variance besides the variance of the focal contrast, we computed additional orthogonal contrasts. If the hypothesis of the focal contrast was correct, only the focal contrast but not the residual contrasts should be significant.

    Regarding collaborative reflection, the hypothesis was supported by the data. There was a significant effect of the focal contrast: F(1, 327)=3.97, P=.047, eta squared=.01. But there was no effect of the associated orthogonal effects: F(1, 327)=1.90, P=.17. So the contrast analysis shows that collaborative reflection increased in the Stroke Unit from 2010 (mean 2.84, SD 0.72) to 2012 (mean 3.06, SD 0.63) but not in the other wards (2010: mean 2.68, SD 0.67; 2012: mean 2.63, SD 0.68).

    In order to provide more information regarding their construct validity [52], we also considered the particular items of the scale and compared the data of the Stroke Unit in 2012 with the combined other conditions on item level. Results are presented in Table 4. For five of the seven collaborative reflection items, we found significantly higher means for the Stroke Unit in 2012 compared to the other conditions.

    Table 4. Collaborative reflection items: means, standard deviations, t, and P values.
    ItemStroke Unit 2012, mean (SD)Other conditions, mean (SD)t (df)P (one-tailed)
    I am provided with enough time to reflect upon my experiences at work.2.81 (0.91)2.38 (0.79)2.14 (323).02
    We discuss on a regular basis if we work successfully together as a team.3.35 (0.70)2.53 (0.99)4.59 (19.65)<.001
    If things do not work out as they should, we try as a team to find the reason.3.41 (0.71)2.96 (0.93)2.00 (328).02
    We as a team think about how we can learn from past experiences.3.35 (0.61)2.90 (0.91)2.00 (329).02
    When I think I did not do a good job, I discuss with colleagues how I could improve myself.3.00 (0.87)2.98 (0.93)0.08 (326).47
    When I think I did not do a good job, I discuss with my supervisor how I could improve myself.2.88 (1.17)2.37 (1.03)1.98 (326).03
    In our organization we are encouraged to reflect upon our experiences at work.2.65 (1.00)2.50 (0.91)0.67 (322).25

    Job Satisfaction

    In order to test hypothesis 2, we computed a Pearson product-moment correlation to analyze the relationship between collaborative reflection and job satisfaction measurements. As expected, the collaborative reflection scale was positively correlated to the job satisfaction scale (r=.61, P<.001).

    Analogous to the presentation of the reflection results, we considered the single items of the collaborative reflection scale in order to provide more information regarding their construct validity. For this purpose, we computed Pearson product-moment correlations for job satisfaction with the individual items of the collaborative reflection scale. As shown in Table 5, all items of the collaborative reflection scale were positively correlated to job satisfaction (all Ps<.001).

    Table 5. Significant correlations of the collaborative reflection items with job satisfaction.
    Itemr
    I am provided with enough time to reflect upon my experiences at work..43
    We discuss on a regular basis if we work successfully together as a team..38
    If things do not work out as they should, we try as a team to find the reason..49
    We as a team think about how we can learn from past experiences..45
    When I think I did not do a good job, I discuss with colleagues how I could improve..37
    When I think I did not do a good job, I discuss with my supervisor how I could improve..42
    In our organization we are encouraged to reflect upon our experiences at work..58

    Main Findings

    The first goal of this study was to test whether the introduction of apps that support reflection would affect hospital staff’s actual reflection behavior at the workplace. We found an increase in collaborative reflection for those hospital employees who had the opportunity to use the apps. With regard to the contents of the reflection items, our findings illustrate which aspects of reflection the apps were particularly able to support. The apps facilitated the conjoint aspects of reflection. In 2012, participants in the Stroke Unit indicated to a higher degree that they discuss on a regular basis their work as a team. They also stated an increased effort to collaboratively learn from past experiences and to find reasons if something did not work out as it was intended. App users also indicated they increased communication with their supervisors. These are aspects that were explicitly addressed by the apps, since they aimed directly at collaborative reflection. The Talk Reflection app, for instance, was designed with the ambition of fostering collaborative reflection. DocTrain was not only designed for individual reflection but also for reflecting with a mentor. Finally, CLinIC simulated colleagues in the game and allowed for collaborative reflection as an opportunity to improve performance.

    As expected, there was a positive relationship between reflection behavior and job satisfaction, indicating that higher levels of collaborative reflection processes go along with enhanced job satisfaction among participants in the hospital. This further supports the findings of other authors regarding a relationship between informal learning and job satisfaction as introduced above. The content of our collaborative reflection scale suggests that being provided with sufficient time to reflect, reflecting collaboratively in a team, and discussing work with a supervisor go along with a higher level of job satisfaction.

    Limitations and Future Work

    A limitation of this study is that there were three different reflection apps. Thus, it is not possible to say whether it was the co-actions of all the apps or only a particular app with certain features that led to the increase in reflection. The DocTrain app, for instance, mainly captures data for reflection. CLinIC provides users with additional experiences in a virtual world where they can test different reactions to a situation without doing any harm. But even a single app might itself foster reflection in diverse ways. In the case of the Talk Reflection app, for example, it could be that just writing down their problems helps users to think about them in a different way. But it could also be that the comments of their colleagues help them to acquire new insights. Future studies with single specific reflection apps should be designed to shed light on these questions.

    Another issue is the topic of collaborative reflection. All apps aimed at fostering collaborative reflection. And in fact, participants in the experimental condition indicated that there was more reflection in the team after the introduction of the apps. However, it remains unclear whether this increase in reflective activity was directly affected by the apps, or whether it just happened that people started to talk about the experiences they had using the apps, or a combination of both. Our data do not ultimately indicate how often and in which way participants used the apps during their everyday work. Future studies should be designed in a way that allows capturing pertinent data about usage. In addition, we did not entirely evaluate the extent to which each individual item of the collaborative reflection scale represented typical reflection behavior from the perspective of the participants. Further research using qualitative methods should assess the representativeness of these items with respect to collaborative reflection.

    Another limitation of the study is the character of the analysis of the relationship between reflection and job satisfaction. As the relationship could only be shown here correlatively, future research should study this relationship in more detail in order to find out under which conditions reflection apps increase general job satisfaction. Finally, it might be worthwhile for future research to examine the connection between app-induced reflection and other socio-emotional experiences of health care professionals over and above job satisfaction.

    Conclusions

    The current study shows that collaborative technology is relevant for health care professionals [53] and that apps that support reflection have an impact on the collaborative reflection behavior of hospital staff. App designers should therefore have a special eye on the issue of collaborative reflection. Developers should also keep in mind that apps might preferably be used by the whole team of a hospital unit. In order to accommodate this preference, developers should consider the different occupation groups in their app design in order to facilitate multidisciplinary collaboration. Reflection apps should provide opportunities for simple and self-evident interactions with colleagues.

    We also found that collaborative reflection is related to the job satisfaction of hospital staff, further indicating the high relevance of reflection for health care professionals. This provides an additional reason for designers of medical apps to look into this topic. Reflection apps seem to have strong potential in various respects. They may not only facilitate users’ cognitive processes, such as improving their ways of learning and working. Apps that foster reflection also seem to have the potential to support health care professionals on a socio-emotional level.

    Acknowledgments

    This study was funded by a grant of the European Commission (grant number: 257617). For their important contributions to the development of the apps that were applied in this study, we would like to thank: Michael Prilla, Martin Degeling, Kada Benadjemia, and Jan Nierhoff (Information and Technology Management at the Institute for Applied Work Science, Ruhr University of Bochum, Germany); Lars Müller and Andreas Übelhör (FZI Research Center for Information Technologies, Karlsruhe, Germany); and the team of imaginary srl (Milan, Italy).

    Conflicts of Interest

    None declared.

    1. Aronson L, Niehaus B, Lindow J, Robertson PA, O'Sullivan PS. Development and pilot testing of a reflective learning guide for medical education. Med Teach 2011;33(10):e515-e521. [CrossRef] [Medline]
    2. ABIM Foundation. American Board of Internal Medicine, ACP-ASIM Foundation. American College of Physicians-American Society of Internal Medicine, European Federation of Internal Medicine. Medical professionalism in the new millennium: a physician charter. Ann Intern Med 2002;136(3):243-246. [Medline]
    3. Mann K, Gordon J, MacLeod A. Reflection and reflective practice in health professions education: a systematic review. Adv Health Sci Educ Theory Pract 2009 Oct;14(4):595-621. [CrossRef] [Medline]
    4. Eraut M. Informal learning in the workplace. Studies in Continuing Education 2004 Jul;26(2):247-273. [CrossRef]
    5. Lave J, Wenger E. Situated learning: legitimate peripheral participation. Cambridge [England]: Cambridge University Press; 1991.
    6. Leslie B, Aring MK, Brand B. Informal learning: The new frontier of employee & organizational development. Economic Development Review 1997;15(4):12-18.
    7. Bovier PA, Perneger TV. Predictors of work satisfaction among physicians. Eur J Public Health 2003 Dec;13(4):299-305 [FREE Full text] [Medline]
    8. Irvine DM, Evans MG. Job satisfaction and turnover among nurses: integrating research findings across studies. Nurs Res 1995;44(4):246-253. [Medline]
    9. Wood Daudelin M. Learning from experience through reflection. Organizational Dynamics 1996 Dec;24(3):36-48. [CrossRef]
    10. Knipfer K, Kump B, Wessel D, Cress U. Reflection as a catalyst for organisational learning. Studies in Continuing Education 2013 Mar;35(1):30-48. [CrossRef]
    11. Boud D, Keogh R, Walker D. Promoting reflection in learning: a model. In: Reflection: turning experience into learning. London: Kogan Page; 1985:18-40.
    12. West M. Reflexivity, revolution, and innovation in work teams. In: Beyerlein MM, Johnson DA, editors. Product development teams. Stamford: JAI Press; 2000:1-29.
    13. Høyrup S. Reflection as a core process in organisational learning. Journal of Workplace Learning 2004;16(8):442-454. [CrossRef]
    14. Kump B, Knipfer K, Pammer V, Schmidt A, Maier R, Kunzmann C, et al. The role of reflection in maturing organizational know-how. In: Proceedings of the 1st European Workshop on awareness and reflection in learning networks. 2011 Presented at: 1st European Workshop on awareness and reflection in learning networks; 2011; Palermo, Italy.
    15. Boyd EM, Fales AW. Reflective Learning: Key to Learning from Experience. Journal of Humanistic Psychology 1983 Apr 01;23(2):99-117. [CrossRef]
    16. Marsick VJ. Learning in the Workplace: The Case for Reflectivity and Critical Reflectivity. Adult Education Quarterly 1988 Dec 01;38(4):187-198. [CrossRef]
    17. Liimatainen L, Poskiparta M, Karhila P, Sjögren A. The development of reflective learning in the context of health counselling and health promotion during nurse education. J Adv Nurs 2001 Jun;34(5):648-658. [Medline]
    18. Richardson G, Maltby H. Reflection-on-practice: enhancing student learning. J Adv Nurs 1995 Aug;22(2):235-242. [Medline]
    19. Wong FK, Loke AY, Wong M, Tse H, Kan E, Kember D. An action research study into the development of nurses as reflective practitioners. J Nurs Educ 1997 Dec;36(10):476-481. [Medline]
    20. Wong FK, Kember D, Chung LY, Yan L. Assessing the level of student reflection from reflective journals. J Adv Nurs 1995 Jul;22(1):48-57. [Medline]
    21. Kuiper RA, Pesut DJ. Promoting cognitive and metacognitive reflective reasoning skills in nursing practice: self-regulated learning theory. J Adv Nurs 2004 Feb;45(4):381-391. [Medline]
    22. Shumway JM, Harden RM, Association for Medical Education in Europe. AMEE Guide No. 25: The assessment of learning outcomes for the competent and reflective physician. Med Teach 2003 Nov;25(6):569-584. [CrossRef] [Medline]
    23. Aronson L, Niehaus B, Hill-Sakurai L, Lai C, O'Sullivan PS. A comparison of two methods of teaching reflective ability in Year 3 medical students. Med Educ 2012 Aug;46(8):807-814. [CrossRef] [Medline]
    24. Aronson L, Niehaus B, DeVries CD, Siegel JR, O'Sullivan PS. Do writing and storytelling skill influence assessment of reflective ability in medical students' written reflections? Acad Med 2010 Oct;85(10 Suppl):S29-S32. [CrossRef] [Medline]
    25. Aronson L. Twelve tips for teaching reflection at all levels of medical education. Med Teach 2011;33(3):200-205. [CrossRef] [Medline]
    26. Aronson L, Chittenden E, O'Sullivan P. A faculty development workshop in teaching reflection. Med Educ 2009 May;43(5):499. [CrossRef] [Medline]
    27. Jarvis P. Reflective practice and nursing. Nurse Educ Today 1992 Jun;12(3):174-181. [Medline]
    28. Groot E, Jaarsma D, Endedijk M, Mainhard T, Lam I, Simons RJ, et al. Critically reflective work behavior of health care professionals. J Contin Educ Health Prof 2012;32(1):48-57. [CrossRef] [Medline]
    29. Cafazzo JA, Casselman M, Hamming N, Katzman DK, Palmert MR. Design of an mHealth app for the self-management of adolescent type 1 diabetes: a pilot study. J Med Internet Res 2012;14(3):e70 [FREE Full text] [CrossRef] [Medline]
    30. West JH, Hall PC, Hanson CL, Barnes MD, Giraud-Carrier C, Barrett J. There's an app for that: content analysis of paid health and fitness apps. J Med Internet Res 2012;14(3):e72 [FREE Full text] [CrossRef] [Medline]
    31. Albrecht UV, von Jan U, Sedlacek L, Groos S, Suerbaum S, Vonberg RP. Standardized, App-based disinfection of iPads in a clinical and nonclinical setting: comparative analysis. J Med Internet Res 2013;15(8):e176 [FREE Full text] [CrossRef] [Medline]
    32. Bokhorst F, Moskaliuk J, Cress U. How patterns support computer-mediated exchange of knowledge-in-use. Computers & Education 2014 Feb;71:153-164. [CrossRef]
    33. Prilla M, Degeling M, Herrmann T. Collaborative reflection at work: supporting informal learning at a healthcare workplace. In: Proceedings of the 17th ACM international conference on supporting group work. 2012 Presented at: 17th ACM international conference on supporting group work; 2012; New York p. 55-64.
    34. Porter LW, Steers RM. Organizational, work, and personal factors in employee turnover and absenteeism. Psychological Bulletin 1973;80(2):151-176. [CrossRef]
    35. Rose R, Kumar N, Pak OG. The effect of organizational learning on organizational commitment, job satisfaction and work performance. Journal of Applied Business Research 2009;25(6):55-66.
    36. Rowden RW. The relationship between workplace learning and job satisfaction in U.S. small to midsize businesses. Human Resource Development Quarterly 2002 Dec 02;13(4):407-425. [CrossRef]
    37. Rowden RW, Jr CTC. The impact of workplace learning on job satisfaction in small US commercial banks. Journal of Workplace Learning 2005;17(4):215-230. [CrossRef]
    38. Rowden RW. Workplace Learning: Principles and Practice (The Professional Practices in Adult Education and Lifelong Learning Series). Melbourne, FL: Krieger Publishing Company; 2007.
    39. Egan TM, Yang B, Bartlett KR. The effects of organizational learning culture and job satisfaction on motivation to transfer learning and turnover intention. Human Resource Development Quarterly 2004 Sep;15(3):279-301. [CrossRef]
    40. Hsu HY. Organizational learning culture's influence on job satisfaction, organizational commitment, and turnover intention among R&D professionals in Taiwan during an economic downturn. Ph.D. Thesis. Minnesota: University of Minnesota; 2009.
    41. Pasarón R. Nurse practitioner job satisfaction: looking for successful outcomes. J Clin Nurs 2013 Sep;22(17-18):2593-2604. [CrossRef] [Medline]
    42. Doc Train.   URL: http://www.mirror-project.eu/showroom-a-publications/mirror-apps-status/92-doctrain [accessed 2013-09-19] [WebCite Cache]
    43. 3D Serious games: CLinIC-The Virtual Tutor and Think better CARE-The Virtual Tutor.   URL: http://www.mirror-project.eu/showroom-a-publications/mirror-apps-status/171-3d-serious-games [accessed 2013-09-19] [WebCite Cache]
    44. 3D serious games for the Health and Care sector MIRROR project. 2012.   URL: http://www.youtube.com/watch?v=mqUC2dcllWA&feature=g-all-u [accessed 2013-09-19] [WebCite Cache]
    45. Prilla M, DeLeeuw K, Cress U, Herrmann T. Individual and collaborative reflection at work: support for workplace learning in healthcare. In: To see the world and a grain of sand: Learning across levels of space, time, and scale: CSCL 2013 Conference Proceedings Volume 2 — Short Papers, Panels, Posters, Demos & Community Events. 2013 Presented at: CSCL; 2013; Madison, Wisconsin p. 55-64.
    46. Patient/Relative Talk App.   URL: http://www.mirror-project.eu/showroom-a-publications/mirror-apps-status/90-prepapp [accessed 2013-09-19] [WebCite Cache]
    47. Kump B, Knipfer K, Wessel D. Specification of research methodology and research tooling (Deliverable 1.1). 2010.   URL: http:/​/www.​mirror-project.eu/​Jdownloads/​public%20documents/​deliverables/​D1_1_Research_methodology_and_tooling.​pdf [accessed 2014-02-14] [WebCite Cache]
    48. Woerkom MV, Croon M. Operationalising critically reflective work behaviour. Personnel Review 2008;37(3):317-331. [CrossRef]
    49. Raelin JA. Public Reflection as the Basis of Learning. Management Learning 2001 Mar 01;32(1):11-30. [CrossRef]
    50. Schippers MC, Den Hartog DN, Koopman PL. Reflexivity in Teams: A Measure and Correlates. Applied Psychology 2007 Apr;56(2):189-211. [CrossRef]
    51. Abelson RP, Prentice DA. Contrast tests of interaction hypothesis. Psychological Methods 1997;2(4):315-328. [CrossRef]
    52. Messick S. Validity of psychological assessment: Validation of inferences from persons' responses and performances as scientific inquiry into score meaning. American Psychologist 1995;50(9):741-749. [CrossRef]
    53. Bientzle M, Cress U, Kimmerle J. How students deal with inconsistencies in health knowledge. Med Educ 2013 Jul;47(7):683-690. [CrossRef] [Medline]

    Edited by G Eysenbach; submitted 21.10.13; peer-reviewed by A Miner, R Grad; comments to author 09.01.14; revised version received 25.01.14; accepted 19.02.14; published 17.03.14

    Copyright

    ©Bettina Renner, Joachim Kimmerle, Dominik Cavael, Volker Ziegler, Lisa Reinmann, Ulrike Cress. Originally published in the Journal of Medical Internet Research (http://www.jmir.org), 17.03.2014.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.