Perceptions of the effectiveness of system dynamics-based interactive learning environments: An empirical study. more

Computers & Education 55 (2010) 1277–1286 Contents lists available at ScienceDirect Computers & Education journal homepage: www.elsevier.com/locate/compedu Perceptions of the effectiveness of system dynamics-based interactive learning environments: An empirical study Hassan Qudrat-Ullah* York University, School of Administrative Studies, 4300 Keele Street, Toronto, ON, Canada M3J 1P3 a r t i c l e i n f o Article history: Received 23 December 2009 Received in revised form 22 May 2010 Accepted 25 May 2010 Keywords: Simulation Improving classroom teaching Interactive learning environments Post-secondary education a b s t r a c t The use of simulations in general and of system dynamics simulation based interactive learning environments (SDILEs) in particular is well recognized as an effective way of improving users’ decision making and learning in complex, dynamic tasks. However, the effectiveness of SDILEs in classrooms has rarely been evaluated. This article describes the construction, integration, and evaluation of an interactive learning environment in two educational settings. Subsequently, it explores how undergraduate business students perceive SDILEs and SDILEs-based course approach. This research draws on data obtained from two courses in undergraduate business program, over a period of three years. Results of this study suggest that students enrolled in the SDILE-based courses do indeed perceive important learning benefits and educational value. Further more, introduction of SDILE-bases courses at higher level are more beneficial than at the lower level introductory courses. However, there is need of more resources to be developed and deployed to harness fully the benefits of experiential learning provided through SDILE-integrated course approach. Ó 2010 Elsevier Ltd. All rights reserved. 1. Introduction The use of simulations for teaching and leaning is becoming increasingly popular (Adobor & Daneshfar, 2006; Faria, 1987; Holweg & Bicheno, 2002; Moratis, Hoff, & Reul, 2006; Tao, Cheng, & Sun, 2009). Specifically, computer simulation-based interactive learning environments (ILEs) are often developed and used to improve people’s decision making in the context of the dynamic complexity of business settings by facilitating user learning (Bakken, 1993; Issacs & Senge, 1994; Lainema & Nurmi, 2006; Lane, 1995; Qudrat-Ullah & Karakul, 2007; Romme, 2003). People learn from the experience. Learning is the process whereby knowledge is constructed by the transformation of experience (Adobor & Daneshfar, 2006; Kolb, 1984; Wall & Ahmed, 2008). Simulations in general and ILEs in particular are one form of experiential learning. In an ILE session subjects make a series of decisions and have access to the instantaneous feedback. Subjects also have the opportunity to evaluate and reflect on their performance in the after-the-simulation debriefing session. ILEs are found helpful in training people on decision making in complex tasks in several domains (e.g., The Business Networking Game in mass customization (Hoogeweegen, van Liere, Vervest, vand der Meijden, & de Lepper, 2006), The MERIT in construction industry (Wall & Ahmed, 2008), INTOPIA in decision support systems (Ben-Zvi, 2010), Realgame (Lainema & Nurmi, 2006) in manufacturing decision making). Prior studies have demonstrated some notable advantages and benefits of the use of traditional business simulations in science and management education and training (e.g., Adobor and Daneshfar (2006) and Anderson and Lawton (2009) provide excellent reviews on this topic). However, developing skills in dynamic decision making, a raison d’etre of business education, is a challenge. Fragmented subjects taught in each functional area (e.g., finance & accounting, strategy, marketing, operations management, human resource management) hardly prepare our students to develop coherent mental models and strategies about the business world (Lainema & Nurmi, 2006; Sterman, 2000). Developing expertise in dynamic decision making requires the decision maker to develop the structural understanding of the task system i.e., how do the decisions and the consequences from them are causally related over time? (Bravo, van Joolingen, & de Jong, 2009; Spector, 2000; Sterman, 1989a; Yasarcan, 2009). System dynamics based ILEs purport to rise to this challenge (Forrester, 1961; Lane, 1995; Sterman, 2000). * Tel.: þ1 416 7362100x33849; fax: þ1 416 7365963. E-mail address: hassanq@yorku.ca 0360-1315/$ – see front matter Ó 2010 Elsevier Ltd. All rights reserved. doi:10.1016/j.compedu.2010.05.025 1278 H. Qudrat-Ullah / Computers & Education 55 (2010) 1277–1286 The basic premise of system dynamics is that “structure drives its behavior” (Forrester, 1961). An ILE based on a system dynamics simulation model, therefore, allows the users to “see” how their decisions lead to the particular outcomes, over time. However, majority of the evaluation research on the promise of system dynamics based simulations, including those with the explicit purpose of improving dynamic decision making skills (e.g., Maani & Maharaj, 2004; Tan, Anderson, Dyer, & Parker, 2010; Yasarcan, 2009) are based either on experimental studies (Moxnes, 2004; Parush, Hamm, & Shtub, 2002; Spector, 2000; Tan et al., 2010; Wheat, 2007) or on professional training and workshop settings (Gröbler, 2004; Gröbler, Maier, & Milling, 2000; Lane, 1995; Qudrat-Ullah & Karakul, 2007). Classroom use of system dynamics based ILEs has been limited to teaching of system dynamics modeling skills (Bravo et al., 2009; Plate, 2010; Wheat, 2007). Thus, despite the promising potential and an increasing interest, system dynamics based ILEs benefit to the business education learning process in the classroom has rarely been investigated empirically. The main purpose of this study, therefore, is to evaluate whether students in post-secondary classrooms perceive any learning benefits from system dynamics based ILEs. Is it more beneficial to introduce such an ILE-based course to junior or senior undergraduate students? In this respect, the implementation of a system dynamics based ILE in several undergraduate courses in business education will be explored. This implies a focus on integrating ILE in standard undergraduate courses (e.g., introduction to operations management, and advanced decision making), without an explicit purpose in the area of systems thinking and system dynamics. From a methodological point of view, this study draws on data gathered based on action research in authentic educational settings (Romme, 2004). Data captured by the ILE’s programmed module, questionnaire and other qualitative data obtained in these settings will serve to explore whether ILEs integrated in classroom settings generate any learning benefits. We begin by discussing some background concepts related to system dynamics based ILEs and how do they support learning and decision making in dynamic tasks and by briefly presenting the FishBanklILE. In Section 3, we describe the settings, methods, and measurements used in this study. We then report results from two educational settings. Finally, we offer some reflections on the role of human facilitator support, utility of system dynamics based ILEs in experiential learning, need for additional instructional resources in ILEs, and role of background education and experience in improving dynamics decision making skills through these ILEs. 2. Background literature 2.1. Background concepts We use “ILEs” as a term sufficiently general to include microworlds, management flight simulators, DSS, learning laboratories, and any other computer simulation-based environments – the domain of these terms is all forms of action for the facilitation of learning in complex, dynamic environments. In an ILE, the learning goals are clearly made explicit. Therefore, the computer games played for fun will not count as ILEs. An ILE consists of three components (i) a computer simulation model to adequately represent the domain or issue on hand with which the decision makers can experience and induce real world-like responses (Davidsen, 2000; Homer & Hirsch, 2006; Kriz, 2003), (ii) a user interface capable of allowing the decision makers to make decisions and access the feedback on interactive basis, and (iii) a human facilitator or a coach responsible for conducting briefing and debriefing sessions (Davidsen, 2000; Davidsen & Spector, 1997; Ledrman, 1992; Zydney, 2010). When an ILE’s underlying simulation model is based on system dynamics methodology (Forrester, 1961), we call that ILE as SDILE. Examples are People Express (Sterman, 1988), FishBankILE (Qudrat-Ullah, Saleh, & Bahaa, 1997) and Healthcare Microworld (Hirsch, Immediato, & Kemeny, 1997). Complex, dynamic decision-making situations differ from those traditionally studied in static decision theory in at least three ways: a number of decisions are required rather than a single decision, decisions are interdependent, and the environment changes either as a result of decisions are made or independently of them or both (Edwards, 1962). Researchers in system dynamics have characterized such tasks by feedback processes, time delays, and non-linearities in the relationships between decision task variables (Moxnes, 2004; Sterman, 1989a). Driving a car, managing a firm, and controlling money supply are all dynamic tasks (Diehl & Sterman, 1995). In these dynamic tasks, contrary to static tasks such as lottery type gambling, locating a park on a city map, and counting money, multiple and interactive decisions are made over several periods whereby these decisions change the environment, giving rise to new information and leading to new decisions (Forrester, 1961; Sterman, 1989b). 2.2. Learning with SDILEs Learning about complex, dynamic tasks does not happen easily. There are some fundamental barriers to developing expertise in dynamic tasks (Sterman, 2000): (1) dynamic complexity: our limited ability to understand the impact of time delays between our actions and their consequences coupled with the interactions between feedback loops that are multiple and non-linear in character and are ever present in the most of the real world managerial tasks, (2) information availability and quality limitations: information we estimate, receive, and communicate is often oversimplified, distorted, delayed, biased, and ambiguous, (3) information processing limitations: when it comes to decision making people generally adopt an event-based, open-loop view of causality, ignore feedback processes, fail to appreciate time delays and are insensitive to non-linearities present in the feedback loop structures of the task system, perceive flawed cognitive maps of the causal structure of the systems, make erroneous inferences even about the simplest possible feedback systems, fall prey to judgmental errors and biases, defensive routines (Sterman, 1994, 1989a, 1989b). The effective SDILE, therefore, should allow the users to overcome such impediments to decision making and learning in dynamic tasks. SDILEs meet this challenge through the provisions of (1) a representative simulation model of the task system, (2) powerful interface, and (3) human tutor support-the three fundamental components of any SDILE. 2.2.1. Learning and decision support through system dynamics simulation model The primary premise of system dynamics methodology is ‘the structure of the system drives its behavior’. This structure consists of feedback loops, stocks and flows, time delays arising from accumulation processes, and non-linearities arising from the interaction of these basic structures (Sterman, 1989b). The core of SDILE is a system dynamics based simulation model (Forrester, 1961). System dynamics based H. Qudrat-Ullah / Computers & Education 55 (2010) 1277–1286 1279 simulation models have strengths to map (i) the multiple stakeholders’ interests, (ii) available but limited resources, and (iii) decisions at different levels of the organization - a general characterization of the most of the task systems e.g., health care system, education system, energy system etc. A typical system dynamics simulation model allows: i. Feedback processes and interaction between the system variables in and across various functional entities (e.g., human resource sector, demand sector, supply sector, finance sector etc.) be explicitly represented (see examples in the work of Bun and Diner (1996) and/or Qudrat-Ullah (2005a)). ii. In contrast to the traditional econometric models, the disequilibrium framework for modeling be established, where the adjustments, say in the need for variable ‘A’ in response to the changes in the variable ‘B’ to new equilibria typically create imbalances and transient behavior. iii. Delays arising from the accumulation processes and other distortions in perceiving the true value of the variables are explicitly modeled. iv. Desired sate and actual sate variables (e.g., sales) are explicitly distinguished in the model of the task system. v. Non-linear relationships between the ‘actions’ and the ‘responses’ are explicitly represented. The significance of the modeling capabilities of system dynamics methodology lies in its contribution to our understanding of the structure and behavior of complex, dynamic systems. An understanding of the relationship between the structure(s) and behavior(s) leads to better understanding of the task system (Conant & Ashby, 1970) and improved decision making (Sengupta & Abdel-Hamid, 1993). 2.2.2. Learning and decision support through the user interface design For effective decision making in complex, dynamic tasks, decisions makers must acquire some reasonably precise notions of relationships among key task variables and develop an understanding of the most influential delays and feedback loops in the task system (Dörner (1980). System dynamics methodology provides powerful tools such as causal loop diagrams and stock and flow structures, to represent qualitatively the link between structure and behavior of the task. Utilizing these tools powerful interface, whereby references to the underlying simulation model are facilitated interactively, can be constructed for an SDILE (for excellent illustrations please see, Davidsen (1996)). In this way, SDILEs aid learning in complex tasks by allowing the learners to examine the structure–behavior relationship as and when needed in an SDILE session (Babin, Tricot, & Marine, 2009; Puustinen & Rouet, 2009). 2.2.3. Learning and decision support through human support Human tutor support constitutes the distinguishing and fundamental component of an SDILE model. In an SDILE session, decisional aids can be provided at three levels: pre-task, in-task, and post-task levels. Pre-task level decisional aids can be conceptualized as information provided by the human tutor to a decision maker regarding the model of the task prior to performing the task (Davidsen & Spector, 1997; Gröbler et al., 2000). In-task decisional aids attempt to improve the individuals’ decision-making performance by (i) making the task goals explicit at early stages of learning, (ii) helping them keep track of goals during the task, and (iii) providing them with diagnostic information (Beauchamp & Kennewell, 2010). Post-task level decisional aids such as the ‘debriefing sessions’ aim at improving performance by providing the decision makers an opportunity to reflect on their experiences with task (Buda, 2009; Davidsen, 2000; Davidsen & Spector, 1997). Thus, an SDILE could support-the user’s understanding of dynamic tasks by offering the opportunity to experimentally design, test, and evaluate their decision strategies. 2.3. FishBankILE: an SD ILE for teaching, learning, and research The main purpose of FishBankILE is to provide the users an opportunity, through a gaming session, to improve their understanding about “The Tragedy of the Commons”; a common phenomenon often encountered in managing the complex, dynamical systems whereby the competitive behavior of individuals, sharing a common limited resource pool, eventually, destroys that resource pool. To achieve this objective of learning and understanding of ‘the tragedy of the commons”, FishBankILE provides a computer simulated environment whereby the learners can try out their decisions in a safe and friendly manner. The tragedy of the commons situation has been realized in the form of FishBankILE, the stock of fish being the common resource and the harvesting the conflict partners’ actions. In this SDILE, the learner plays the role of a manager of a fishing company. Each company begins with the same initial assets such as bank balance and fishing fleet. The measure for performance is the cumulative profits the learners will accrue by the end of game trial run plus the fish stock value in the final year. A system dynamics based simulation model, an information system, and a decision panel constitute the key components of FishBankILE. The learners are told to ask “the facilitator” any question they might have prior, during, and after-thesimulation task session. 2.3.1. The simulation model of FishBankILE Fig. 1 portrays the overall structure of the underlying simulation model of FishBankILE that includes ship system, fish system and financial system1. The simulation model will generate dynamics as the user makes decisions. The dynamic behavior in the model arises from the link between two fundamental accumulation processes i.e., accumulation of ships and accumulation of the common resource - fish. Fish regeneration adds to the fish stock, while fish harvesting depletes the stock. Fish regeneration depends on the existing level of fishery stock. However, fish depletion is determined by two factors: fish harvesting and natural death of fish. The fleet capacity (i.e., total number of ships), fleet capacity utilization, and catch per ship determine fish harvesting. Catch per ship is dependent on fish density, while the current stock of fish determines the fish density. Catch per ship drives the profitability of each firm. The relationship between fish catch per ship and fish density is very important to the behavior of the system. 1 Mathematical equations of this model are available from the author. 1280 H. Qudrat-Ullah / Computers & Education 55 (2010) 1277–1286 Ship System Total Ships New Ships Fleet Utilization Fleet Operating Cost Fleet Harbor Cost Financial System Total Profits / Losses Yearly Profits/ Losses Fish Stock Fish Regeneration Fish Depletion Carrying Capacity Fish Density Catch Per Ship Fish System Fig. 1. Overall view of FishBankILE’ simulation model. Fish death rate is dependent upon the current stock of fish and the carrying capacity of the environment. The carrying capacity is an ecological system ratio and denotes the maximum number of fish, which the fishing environment can support. In purely natural conditions, fish population tries to maintain this equilibrium level. The fleet capacity increases by new ship orders (i.e., investments) and the fleet is reduced by scrapping after an average lifetime of 20 years. The two decision variables are (i) new ship orders, and (ii) ship utilization percentage. 2.3.2. The information system of FishBankILE FishBankILE contains an elegant information system which allows the users to monitor developments in all areas of the firm and the markets. Users will have access to reports which detail the current status of their fleet, fleet utilization, catch per ship, estimated resource, and financial status as well as historical development of these variables.Fig. 2 exhibits a screen-snapshot of “Financial Report”. 2.3.3. The decision system of FishBankILE The decision system of FishBankILE provides the users the opportunity to make decisions and achieve their goals. Each game trail runs for 30 years and the user will make two decisions per year which are: 1- Ship Purchase – how many new ships the user wants to buy? 2- Ship Utilization – what fraction of fleet the user wants to send out for fishing? The overall profit structure for each firm is given in Fig. 3. The details on these parameters (e.g., values and assumptions) are listed in the user manual of FishBankILE which is made available to the users well before the simulation sessions. 2.3.4. The data collection capability of FishBankILE FishBakILE is programmed to automatically capture a range of data that can be used for research purposes. For instance, besides tracking the performance data, data on several behavioral variables such as decision time for each decision, and information system access tracking are recorded and files can be exported in a spreadsheet. 3. Methods This study draws on data gathered through action experiments in authentic educational settings (Argyris, 1993; Klabbers, 2000; Romme, 2004). These experiments do not involve control groups. Data captured by the FishBankILE program, and questionnaire and qualitative data obtained in these settings serve the purpose to explore whether the integration of SDILEs in classroom settings generates learning benefits. All the data except task performance are self-evaluations. Action experiments approach was chosen to study the impact of integration of SDILEs in standard undergraduate courses (e.g., introduction to operations management, advanced decision making), without any explicit purpose in the areas of system dynamics. In addition, this approach provides comparison across two different settings and student populations. This study implements one SDILE in two courses and educational settings, one in the undergraduate junior level and the other in the undergraduate senior level. The first setting was an ‘Introduction to Operations Management’ course at the second year of an undergraduate bachelor of administrative studies program. The elective course, “Advanced Decision Making,” in the fourth year of an undergraduate bachelor (honors) of administrative studies program served as the second setting. Data collection occurred between March 2005 and March 2008. H. Qudrat-Ullah / Computers & Education 55 (2010) 1277–1286 1281 Fig. 2. A Screen-Snapshot of “Financial Report” of FishBankILE. 3.1. Sample and setting descriptions 3.1.1. Undergraduate junior level In this setting, FishBankILE was introduced in the course “Introduction to Operations Management”. The main objective of this course was to build an understanding of the dynamics and complexity of firm’s operations and strategies for sustainable growth and profitability. By means of lectures, case discussions, assignments in team settings (3 or 4 students each) and individual study, a basic understanding of the interdependencies between managerial decision making, business operations, and sustainable business growth and profitability was developed. FishBankILE was implemented in the week 9 of this 13 week course to create opportunities for experiencing the dynamics of these complex interdependencies. Student performed the first trial in week 10. In week 11, a comprehensive debriefing session took place in which several selected teams presented their simulation results and learning outcomes. The instructor facilitated the reviews of these results and learning of all teams. In week 12, all the teams completed the final trial. There were 306 students who took this course in three subsequent years. The course is offered two times per year. The average age of the students (both male and female) in this group was 22 years. Most of theses students have completed 30 credits of their 120 credit degree program. 3.1.2. Undergraduate senior level As an elective course, “Advanced Decision Making” in the fourth year of an undergraduate bachelor (honors) program in administrative studies served as the second setting to implement FishBankILE. The purpose of this, case-based course is to build awareness and an understanding of the complexities of real world decision making. The latest real world cases are debated, criticized, and reflected upon. Written assignments, case analysis reports, both individual and team-based, and with instructor’s feedback students develop skills in dynamic resource management area. FishBankILE was adopted to provide them the opportunity to hypothesize and test their decision strategies and build knowledge about the long-term dynamics of various decision strategies. Students worked in teams (3 or 4 members per team), on the assignments related to FishBankILE. FishBankILE was used near the end of the semester. A total of 176 students took this course over a period of three years. The age of the students (both male and female) in this group ranged from 21 to 29. Most students have completed 90 credits of their 120 credit degree program. 3.2. Measurements Subjects’ performance in the task was automatically recorded through a programmed module of FishBankILE. There were three variables: task performance, decision time, and information system access. The task performance metric for each learner was chosen in such a way to Profits = Revenue - Expenses Fish Sales Fish Catch per Ship Ship Utilized Ship Purchases Harbor Cost Fishing Cost Fig. 3. The Overall Profit Structure for Each Firm. 1282 H. Qudrat-Ullah / Computers & Education 55 (2010) 1277–1286 asses how well she/he did relative to a benchmark rule (please see the details of this benchmark rule in Qudrat-Ullah (2007). The task performance measure, TP, has the following formulation: Pny TP ¼ t¼1 PnT t ¼ 1 ðyit À bit Þ ny *nT where ny is number of task performance variables, nT is the number of trials, bn is the bench mark value of task performance variable i at time t, and ya is the empirical value of task performance variable i at time t. Every decision period, the benchmark’s performance variable’s values are subtracted from the learner’s. The learner’s final performance, TP, is the accumulation over 30 periods of this difference, given the identical conditions, averaged over of the number of task performance variables and number of trials. Information System Access (IS) is the frequency i.e., number of times Information System button was clicked (to see the performance status). Decision Time (DT) is the average time in seconds, per decision period. To explore the perceived value of SDILEs, a questionnaire was adopted from Romme’s work on the evaluation of the learning benefits of microworld simulation (Romme, 2003). This questionnaire was administered along with the usual course evaluations at the end of the semester. In addition to the demographics, the developed instruments included questions on a five-point Likert scale regarding background education and experience, previous exposure to ILEs, the user manual, user interface, the simulation-based assignments, briefing and debriefing by the instructor, team collaboration, added learning benefits, and course recommendations to others. 4. Results A total of 280 students out of 306 students, who took the undergraduate junior level course over a period of three years at a CanadianUniversity, responded with completed questionnaires. Consequently, the data of only these 280 is stored and used in this study. The response rate for the undergraduate senior level is 94% (166 participants out of 176 participants. None of the participants had prior experience with ILEs. The responses from both the groups are summarized in Table 1. Overall, both groups rated the integration of SDILE in classrooms approach high, with a mean score of 3.97 (for juniors) and 4.12 (for seniors) on a five-point scale based on the results of the responses (questions 19–24) in Table 1. These results suggest that the SDILE-based course work approach using the simulation task, that facilitates the teaching and learning of complexities of decision making for sustainable growth and profitability of business organizations, are interesting, exciting, innovative, and more value-adding than traditional textbook-based learning. In addition to this descriptive statistics in Table 1, the self evaluative/qualitative data, teams’ written reports on their task performance (simulation results) as well as what Table 1 Perceived effectiveness of the SDILE-based course by juniors and seniors. Constructs Juniors (n ¼ 280) Mean User interface: Using FishBankILE is 1. Fun 2. Pleasant 3. Exciting 4. Enjoyable 5. Is easy-to-use 6. Has user-friendly interface 7. Represents real business situation 8. Has effective on-line help 9. provides immediate & useful feedback User manual: The SDILE, FishBankILE 10. Has well written and self-explanatory user manual 3.12 3.15 3.10 3.10 4.10 3.80 2.98 4.10 3.66 3.66 SD 0.76 0.77 0.86 0.81 0.66 0.66 0.46 0.66 0.64 0.56 0.44 0.72 0.44 0.51 0.62 0.71 0.45 0.44 0.44 0.61 0.45 0.56 0.66 0.76 Seniors (n ¼ 166) Mean 3.56 3.18 3.12 3.26 4.66 3.96 2.90 4.23 3.88 3.87 3.98 3.88 3.88 3.97 3.78 3.56 3.24 3.66 4.12 4.06 4.22 4.44 4.34 3.45 SD 0.85 0.42 0.41 0.56 0.72 0.80 0.44 0.34 0.80 0.80 0.56 0.80 0.56 0.66 0.44 0.56 0.80 0.52 0.46 0.44 0.66 0.46 0.46 0.66 Difference between seniors and juniors t-value (p-value) À4.979(0.000) À0.368(0.712) À0.256(0.797) À1.95(0.051) À7.138(0.000) À4.231(0.000) 2.272 (0.023) À1.463(0.144) À3.334(0.000) À2.286(0.022) À0.311(0.751) À0.286(0.774) À9.182(0.000) À10.347(0.000) À2.012(0.044) À1.291(0.197) À0.393(0.694) À3.709(0.000) À2.499(0.013) À3.655(0.000) À2.349(0.019) À5.017(0.000) À7.33(0.000) 9.293 (0.000) Simulation-based assignments: The assignments related to FishBankILE were 11. Clear and relevant 3.88 12. Interesting 3.86 Team Collaboration: In our team, we collaborated successfully by 13. Sharing ideas 14. Appreciating ideas of fellow team members 15. Crafting decision strategy by consensus Briefing and Debriefing: Feedback by the instructor at 16. Pre-task level was useful and aroused my interest 17. In-task level was useful 18. Post-task level (debriefing) was very useful 3.12 3.10 3.55 3.44 3.02 3.44 Added Learning Benefits: Overall, the SDILE embedded course approach 19. Is more interesting than traditional textbook learning 3.88 20. Is more exciting than traditional textbook learning 3.91 21. Is more innovative than traditional textbook learning 4.01 22. Adds value for learning 4.22 23. adds value relative to real world experience 3.61 Recommendations: Overall, the SDILE embedded course approach 24. Has my strong recommendations 4.20 H. Qudrat-Ullah / Computers & Education 55 (2010) 1277–1286 1283 they had learned, support this conclusion further. We categorized all the written reports based on presence of some key task variables, for instance, “ship orders”, “ship utilization percentage”, “fish stock/level”, “ship construction delay”, “over utilization”, “under utilization”, and “time delay.” For example, one of the teams from undergraduate junior level, representative of those who developed some understanding of the dynamic task, wrote: “Working with FishBanklILE and doing assignments related to simulations runs were interesting for our team. Each of us [Team 3] decided to come up with some strategy to run the business enterprise (the simplified reality though) the best possible way. We discovered that our decisions in the previous years have huge influence on our next decisions and the outcomes. Only in our second trial with FishBankILE we found the impact of “ship construction delay” on our performance: you need to pay attention that ships ordered will not arrive that year but a year later.” On the other hand, a representative team of those undergraduate juniors who barely developed any understanding of the dynamic task, as per their written report, wrote: “Our team [Team 6] used [a]simple strategy: order maximum ships and send them all in the sea to catch the fish. More the ships we had, more the catch but still we were not having huge profits. May be there is a problem with the game [FishBankILE]. We had fun though!.” Further analysis was conducted to investigate whether a difference existed between both groups in the mean ratings of their perceptions about the effectiveness and usability of the SDILE. T-tests were used and the results are presented in Table 1. There were 24 questions. Questions 1–12 were used to assess “usability” category and the remaining questions assessed the “effectiveness” category. Seniors rated higher than juniors on “fun” (3.56 versus 3.12, p ¼ 0.000), “ease of use” (4.66 versus 4.10, p ¼ 0.000), and “availability and usefulness of feedback (3.88 versus 3.66, p ¼ 0.000) with the simulation learning environment, FishBankILE. There was no significant difference between the mean ratings of undergraduate juniors and undergraduate seniors on how “exiting” and “pleasant” they felt while using FishBankILE. On the other hand, there were significant differences in their mean ratings of the most items related to the “effectiveness” category. Only in the case of two items, “pre-task” and “in-task” level feedback, there were no significant differences in their mean ratings. Thus, the usability and effectiveness of the SDILE-based course approach has been ascertained by the positive feedback obtained from the evaluation forms. In Table 2, we present the actual task performance of both the groups. It is interesting to note that although both the groups rated the SDILE embedded course approach high, their actual performance on the dynamic task: FishbankILE differed significantly. On average, seniors performed relatively better on task performance. On the other hand, juniors frequently accessed the task information system (that would display the status of their performance over time) and spent more time in each of the 30 decision time periods. However, both the groups performed poorly compared with the benchmark. One could argue that improvement in performance in Trial 2 might have come as a result of practice of Trial 1. We analyzed the effects of practice separately utilizing the data obtained from both the groups. We found significant effects of practice on “decision time” of the subjects in both the groups (i.e., both the groups spent less time in Trial 2 than in Trial 1), but not so much as to represent a real threat to our main finding that both the groups rated the SDILE embedded course approach high. 5. Discussion Although the majority of university classrooms are still dominated by traditional teaching methods such as lectures and discussions, the use of computer simulations in instruction is on the rise (Faria, 1987; Holweg & Bicheno, 2002; Lane, 1995; Wolfe & Crookaal, 1998). The wide spread use of computer simulation-based learning environment for teaching and learning is concomitant with increasing interest with and attention to experiential learning methods. ILEs provide safe and relatively inexpensive opportunity for the decision makers and learners to test out their decision strategies before costly and often irreversible implementation follows (Qudrat-Ullah & Karakul, 2007; Tang & Austin, 2009). Still, we often see little improvement in decision making and learning performance of the users of ILEs (Sterman, 1989a, 1989b, 1994). A common criticism of ILEs is that users often fall into so called “vide arcade syndrome”: people can win the game but without having any clue about how (Gröbler et al., 2000). Moreover, learners in ILEs and in the absence of human facilitation face difficulties in assimilating the new knowledge with the existing mental models. 5.1. Human facilitator support It is clear that the potential to improve teaching and learning with ILEs is not without conditions. ILEs should be used with structured human facilitator support (Babin et al., 2009; Buda, 2009; Davidsen, 1996; Faria, 1987; Zydney, 2010). For instance, pre-task facilitation may reduce cognitive load of the learners because a substantial amount of the information the learners have to infer through the complex Table 2 Actual performance in the simulated dynamic task. Groups Juniors Seniors Juniors Seniors Juniors Seniors Performance TPa ¼ À0.60 TP ¼ À0.42 ISb ¼ 22 IS ¼ 12 DTc ¼ 61.45 DT ¼ 29.54 t-value 9.870 21.336 19.131 Critical t (two-tail)-value 1.987 1.987 1.994 p-value 0.000 0.000 0.000 a TP: Task Performance (TP) is the average performance (relative to the benchmark) in Trial 2. A score of 0 means that subjects performance at par with the benchmark rule. A score of >0 would mean better performance and a score of <0 means subject performed poorly as compared with the performance of the benchmark rule. b IS: Information System Access (IS) is the frequency i.e., number of times Information System button was clicked (to see the performance status) during Trial 2. c DT: Decision Time (DT) is the average time in seconds, per decision period, in Trail 2. 1284 H. Qudrat-Ullah / Computers & Education 55 (2010) 1277–1286 interactions with the ILE is already transmitted through prior instructions. In-task level facilitation can provide appropriate goal-directed and diagnostic information that can aid learners in ILEs to focus on the task objectives, enhance their ability to track the changes in the underlying system, and form an adequate model of the task system (Qudrat-Ullah & Karakul, 2007; Yaman, Nerdel, & Bayrhuber, 2008). Post-task facilitation or debriefing review aims at improving decision-making and learning performance by providing an opportunity to reflect on their experiences with the simulated task (Davidsen, 2000; Wolfe & Crookaal, 1998). Thus, we see no surprise in our finding that both the groups found human facilitator support at all the three levels useful (please refer to Briefing and Debriefing measures in Table 1) with the one exception that the seniors found post-task debriefing more useful than the juniors (3.66 versus 3.44, p ¼ 0.0). The seniors found debriefing more useful perhaps because of their access to what Schön (1938) called “reflective conversation with the situation” gave them greater adaptability in recognizing task system changes and updating their mental models about the task system. With relatively more work experience and diverse background education (i.e., more courses), they were able to reflect on and improve their understanding of the task. 5.2. SDILEs in support of experiential learning The present study was designed to gain a better understanding of the perceived effectiveness of SDILE-based course at the undergraduate level. The premise was the SDILEs with structured facilitator support which relates and motivates learners to embrace “experiential learning” and as a result would increase the perceived effectiveness of SDILE-based course. The results described in the previous sections support this proposition. That is, students, both juniors as well seniors, tend to perceive substantial “learning benefits” from SDILE-based course – in general as well as relative to the traditional textbook learning activities, as can be seen in measures pertaining to “Added Learning Benefits” construct in Table 1. The junior students with hardly any exposure to real world work appear to perceive less (3.61 versus 4.34, p ¼ 0.0) value of SDILE-based course relative to the real world experience. On the other hand, senior students with some work experience (with average of 1.2 years) and more exposure to higher education settings tend to perceive SDILE-based course approach more interesting (p ¼ 0.013), exciting (p ¼ 0.0), value adding (p ¼ 0.0), and innovative (p ¼ 0.019. In fact, when we look at their actual performance in Table 2, seniors did perform relatively better on both the measures: “task performance” (p ¼ 0.0) and “decision time-they used less time (29.54 s versus 61.45 s, p ¼ 0.0) in each decision period.” Therefore, one could argue in favor of the introduction of SDILEs at senior undergraduate level. 5.3. Learning with SDILEs and the need of resources Traditionally, SDILEs are used in workshop settings with only cursory level of briefing and debriefing support. However, the results described in the previous section suggest that to accrue the learning benefits from an SDILE-based course a number of resources have to be created and provided. These resources include a well-written and self-explanatory user manual, a user-friendly interface with easy-to-use help and information systems, a related business case-based simulation model, relevant simulation assignments that motivate students to design and test their decision strategies in the simulated environment, and effective feedback at all levels i.e., pre-task, in-task, and post-task level, by the facilitator. Other studies e.g., in a technology-enhanced learning environment (Puustinen & Rouet, 2009), in a computer supported learning environment (Holweg and Bicheno (2002)), and Lazakidou & Retalis, 2010 in a participative simulation modeling environment, have reported similar observations to the need of these resources. 5.4. Role of background education and work experience The difference with regard to perception of use of FishBankILE as a “fun” and “ease of use” can be explained in terms of the subjects’ background education course work, lab, and internship experiences. Senior students have accumulated substantial business knowledge through variety of courses, case studies, project work including substantial use of computing, and internships through the undergraduate business program. If the users of an ILE can relate the underlying simulation task system to some known concepts and situations then they are more excited, find it easy, and have more fun in exploring the simulation task (Nagi, 2007; Tao et al., 2009; Trundle & Bell, 2010). On the other hand, junior students may have approached SDILE-based work as any other assignment just to pass the course rather than an opportunity for experiential learning (Romme, 2004). Relatively more (61.45 s versus 29.54 s, p ¼ 0.0) time spent at each decision period by the juniors suggest that they might have not developed a decision strategy rather than focused at each individual decision. Such an approach rarely helps the learners to recognize patterns and develop a model of the underlying task systems (Moxnes, 2004). The junior undergraduate students showed greater willingness to share their experience and make recommendations to others for SDILE-based course approach. Perhaps with little or no prior work and internship related experience, these junior students find this ‘only’ opportunity of experiential learning worth sharing. 6. Conclusions 6.1. Limitations of this study A number of potential limitations to the current investigation exist. The findings reported in this article are based on only two educational settings and should be researched and tested further in some other educational settings, e.g., in the context of graduate and executive programs. This study used only one SDILE, FishBankILE, future studies should test whether the use of a combination of SDILEs is more beneficial for the learning outcomes. The generalizability of our findings may also be limited to simulations that are similar to FishBankILE, the type used in the study. The findings may also be biased as a result of the nature of action experiment and the self-reported data used. Possibility of novelty effect also exists. Future studies based on experimental approach therefore needs to validate the results of this study. The author is conducting such an investigation to follow-up on and extend the present study. H. Qudrat-Ullah / Computers & Education 55 (2010) 1277–1286 1285 6.2. Key findings Overall, this study finds that students have positive perceptions of the use of SDILE (i.e., FishBankILE) and value of SDILE-based course approach. By itself, this is sufficient incentive for college/university administrators and faculty to continue to emphasize the integration of SDILEs into the most facets of post-secondary business education. This study makes substantial contributions to the research in the area of educational simulations at least in two ways: (1) by incorporating structured and systematic debriefing between the trails, it introduces a new feature in the design of simulations, and (2) it is perhaps first study that have implemented and evaluated system-dynamics type simulations both at junior and senior level of undergraduate business classes. This study suggests that students enrolled in a SDILE-based course do indeed perceive important learning benefits and educational value. In the context of the findings of this study, there are several important implications for how we design and conduct our undergraduate simulation-integrated business classes:  At what level do we need to introduce a SDILE-integrated course? The difference with regard to perception of use of FishBankILE as a pleasant experience and relatively better task performance by the seniors suggest that to accrue learning benefits from a simulationbased course approach, upper level classes should be the primary candidate for such intervention.  Positive effects of practice suggest that more sessions with simulated task should become the core in-class learning activity in a SDILEintegrated course.  Prior research has shown the positive impact of debriefing on subjects performance in SDILE-based simulation tasks. Our study extends this research by finding the differential impacts of pre-task level (i.e., before the learners begin interaction with simulated task): while seniors should be encouraged to begin the task with minimal introductory information, juniors should be given detailed task related examples and overview of the key concepts related to the task. Overall, our findings suggest that there is value in exploring aspects and modes of ILE-based course approach in classrooms. Acknowledgement Author would like to thank the editor, Chin-Chung Tsai, and three anonymous reviewers for their helpful comments and suggestions throughout the revision process of this paper. An earlier draft of this work has been presented at the 16th International Conference on Computers in Education (proceedings on CD-ROM), (October 27–31, 2008), Taipei, Taiwan. References Adobor, H., & Daneshfar, A. (2006). Management simulations: determining their effectiveness. Journal of Management Development, 25(2), 151–168. Anderson, P., & Lawton, L. (2009). Business simulations and cognitive learning. Simulation & Gaming, 40(2), 193–216. Argyris, C. (1993). Knowledge for action. A guide to overcoming barriers to organizational change. San Francisco: Jossey Bass. Babin, L., Tricot, A., & Marine, C. (2009). Seeking and providing assistance while learning to use information systems. Computers & Education, 53, 1029–1039. Bakken, B. E. (1993). Learning and transfer of understanding in dynamic decision environments. Unpublished doctoral dissertation. Boston: MIT. Beauchamp, G., & Kennewell, S. (2010). Interactivity in the classroom and its impact on learning. Computers & Education, 54, 759–766. Ben-Zvi, T. (2010). The efficacy of business simulation games in creating decision support systems: an experimental investigation. Decision Support Systems, 49, 61–69. Bravo, C., van Joolingen, & de Jong, T. (2009). Using co-lab to build system dynamics models: students’ actions and on-line tutorial advice. Computers & Education, 53, 243–251. Buda, R. (2009). Learning-testing process in classroom: an empirical simulation model. Computers & Education, 52, 177–187. Bun, D., & Diner, I. (1996). System simulation to support integrated energy analysis and liberalized planning. International Transactions in Operational Research, 3(2), 105–115. Conant, R., & Ashby, W. (1970). Every good regulator of a system must be a model of the system. International Journal of System Science, 1, 89–97. Davidsen, P. I. (1996). Educational features of the system dynamics approach to modeling and simulation. Journal of Structural Learning, 12(4), 269–290. Davidsen, P. I. (2000). Issues in the design and use of system-dynamics-based interactive learning environments. Simulation & Gaming, 31(2), 170–177. Davidsen, P. I., & Spector, J. M. (1997). Cognitive complexity in system dynamics based learning environments. International system dynamics conference. Turkey: Bogacizi University Printing Office Istanbul. Diehl, E., & Sterman, J. D. (1995). Effects of feedback complexity on dynamic decision making. Organizational Behavior and Human Decision Processes, 62(2), 198–215. Dörner, D. (1980). On the difficulties people have in dealing with complexity. Simulations and Games, 11, 8–106. Edwards, W. (1962). Dynamic decision theory and probabilistic information processing. Human Factors, 4, 59–73. Faria, A. J. (1987). A survey of the use of business games in academia and business. Simulation & Gaming, 18(2), 207–224. Forrester, J. W. (1961). Industrial dynamics. In Gaming. Simulation & gaming, Vol. 29. Cambridge, MA: Productivity Press. Gröbler, A. (2004). Don’t let history repeat itself-methodological issues concerning the use of simulators in teaching and experimentation. System Dynamic Review, 20(3), 263–274. Gröbler, A., Maier, F. H., & Milling, P. M. (2000). Enhancing learning capabilities by providing transparency in transparency. Simulation & Gaming, 31(2), 257–278. Hirsch, G. B., Immediato, J. M., & Kemeny, M. (1997). Creating integrated care and healthier communities microworlds. Cambridge, Mass: Innovation Associates. Holweg, M., & Bicheno, J. (2002). Supply chain simulation – a tool for education, enhancement and endeavour. International Journal of Production Economics, 78, 163–175. Homer, J. B., & Hirsch, G. B. (2006). System dynamics modeling for public health: background and opportunities. American Journal of Public Health, 96(3), 452–458. Hoogeweegen, M., van Liere, D., Vervest, P., vand der Meijden, L., & de Lepper, I. (2006). Decision Support Systems, 42, 1402–1412. Issacs, W., & Senge, P. (1994). Overcoming limits to learning in computer-based learning environments. In J. Morecroft, & J. Sterman (Eds.), Modeling for learning organizations (pp. 267–287). Portland, Or: Productivity Press. Klabbers, J. H. G. (2000). Gaming & Simulation: Principles of a science of design. Simulation & Gaming, 34(4), 569–591. Kriz, W. C. (2003). Creating effective learning environments and learning organizations through gaming simulation design. Simulation & Gaming, 34(4), 495–511. Lane, D. C. (1995). On a resurgence of management simulations and games. Journal of the Operational Research Society, 46, 604–625. Lazakidou, G., & Retalis, S. (2010). Using computer supported collaborative learning strategies for helping students acquire self-regulated problem-solving skills in mathematics. Computers & Education, 54, 3–13. Ledrman, L. C. (1992). Debriefing: towards a systematic assessment of theory and practice. Simulation & Gaming, 23(2), 145–160. Maani, K., & Maharaj, V. (2004). Link between systems thinking and complex decision making. System Dynamics Review, 20(1), 21–48. Moratis, L., Hoff, J., & Reul, B. (2006). A dual challenge facing management education; simulation-based learning and learning about CSR. Journal of Management Development, 25(3), 213–231. Moxnes, E. (2004). Misperceptions of basic dynamics: the case of renewable resource management. System Dynamics Review, 20, 139–162. Nagi, E. W. T. (2007). Learning in introductory e-commerce: A project-based teamwork organizational change. San Francisco: Jossey-Bass. Parush, A., Hamm, H., & Shtub, A. (2002). Learning histories in simulation-based teaching: the effects on self-learning and transfer. Computers & Education, 39, 319–332. Plate, R. (2010). Assessing individuals’ understanding of nonlinear causal structures in complex systems. System Dynamic Review, 26(1), 19–33. 1286 H. Qudrat-Ullah / Computers & Education 55 (2010) 1277–1286 Puustinen, M., & Rouet, J. (2009). Learning with new technologies: help seeking and information searching revisited. Computers & Education, 53, 1014–1019. Qudrat-Ullah, H. (2005a). MDESRAP: a model for understanding the dynamics of electricity supply, resources, and pollution. International Journal of Global Energy Issues, 23(1), 1–14. Qudrat-Ullah, H. (2007). Debriefing can reduce misperceptions of feedback: the case of renewable resource management. Simulations & Gaming, 38(3), 382–397. Qudrat-Ullah, H., & Karakul, M. (2007). Decision making in interactive learning environments: towards an integrated model. Journal of Decision Systems, 16(1), 79–99. Qudrat-Ullah, H., Saleh, M. M., & Bahaa, E. A. (1997). FishbankILE: an interactive learning laboratory to improve understanding of ‘the tragedy of commons’; a common behaviour of complex dynamic systems. In Proceedings of 15th international system dynamics conference. Istanbul, Turkey. Romme, A. G. L. (2003). Learning outcomes of microworlds for management education. Management Learning, 34, 51–61. Romme, A. G. L. (2004). Perceptions of the value of microworld simulation: research note. Simulation & Gaming, 35(3), 51–61. Schön, D. (1938). The reflective practitioner. New York: Basic Books. Sengupta, K., & Abdel-Hamid, T. (1993). Alternative concepts of feedback in dynamic decision environments: an experimental investigation. Management Science, 39, 411–428. Spector, J. M. (2000). System dynamics and interactive learning environments: lessons learned and implication for the future. Simulation & Gaming, 31, 528–535. Sterman, J. D. (1988). People express management flight simulator. Available from. In J. D. Sterman (Ed.), Software and briefing book. Cambridge, MA 02142, U.S.A: Sloan School of Management, 50 Memorial Drive, MIT. Sterman, J. D. (1989a). Misperceptions of feedback in dynamic decision making. Organizational Behavior and Human Decision Processes, 43, 301–335. Sterman, J. D. (1989b). Modeling managerial behavior: misperceptions of feedback in a dynamic decision making experiment. Management Science, 35, 321–339. Sterman, J. D. (1994). Learning in and around complex systems. System Dynamics Review, 10(2–3), 291–323. Sterman, J. D. (2000). Business dynamics: Systems thinking and modeling for a complex world. New York: McGraw-Hill. Tan, B., Anderson, E., Dyer, J., & Parker, G. (2010). Evaluating system dynamics models of risky projects using decision trees: alternative energy projects as an illustrative example. System Dynamics Review, 26(1), 1–117. Tang, T., & Austin, M. (2009). Student’s perceptions of teaching technologies, application of technologies, and academic performance. Computers & Education, 53, 1241–1255. Tao, Y., Cheng, C., & Sun, S. (2009). What influences college students to continue using business simulation games? The Taiwan experience. Computers & Education, 53, 929–939. Trundle, K., & Bell, R. (2010). The use of computer simulation to promote conceptual change: a quasi-experimental study. Computers & Education, 55, 1078–1088. Wall, J., & Ahmed, V. (2008). Use of simulation game in delivering blended lifelong learning in the construction industry – opportunities and challenges. Computers & Education, 50, 1383–1393. Wheat, I. D. (2007). The feedback method of teaching macroeconomics: is it effective? System Dynamic Review, 23(4), 391–413. Wolfe, J., & Crookaal, D. (1998). Developing scientific knowledge of simulation/gaming. Simulation & Gaming, 29(1), 7–19. Yaman, M., Nerdel, C., & Bayrhuber, H. (2008). The effects of instructional support and learner interest when using computer simulations. Computers & Education, 51, 1784–1794. Yasarcan, H. (2009). Improving understanding, learning, and performance of novices in dynamic managerial simulation games. Complexity, 15(4), 31–42. Zydney, J. (2010). The effect of multiple scaffolding tools on students’ understanding, consideration of different perspectives, and misconceptions of complex problem. Computers & Education, 54, 360–370.