Evaluation of the Effectiveness of Systems Thinking and System Dynamics

Systems Thinking and System Dynamics (ST/SD) present a holistic framework (of concepts, methodologies, techniques, and tools) that can be used to support problem solving in dynamic complex systems. An ST/SD approach aims to untangle the complex interrelationships between system components, processes, boundaries, objectives, and stakeholders.

Whereas there are many claims in the literature about the potential of using an ST/SD approach to understand and learn about complexity, this is still limited to being a largely untested hypothesis with little empirical evidence, much of which is inconclusive. This status not only raises scepticism about the value of ST/SD, but also implies a wasted learning opportunity for the field to grow and improve. The need for rigorous ST/SD research is critical, especially at a time of increasingly tight resources where funds (in the research, education, and management sectors) tend to be directed towards high-impact research topics. Recently, the topic of evaluating ST/SD methods has gained increasing attention in the literature (Midgley et al., 2013; Scott et al., 2013). The design and conduct of rigorous evaluation studies involve addressing the following questions:

  • What are the requirements for designing and running a rigorous and pragmatic evaluation approach?
  • How can we mix qualitative and quantitative instruments in evaluation design?
  • How can we make use of established theories (e.g. a behavioural theory, learning theories, program evaluation theories) to support the design of evaluation experiments and to explain results?
  • What are the implications of web-based technologies and distance learning modes on the use and evaluation of ST/SD techniques and tools?
  • How can we transfer findings (of all the above) from an experimental setting to a real-life situation?

Scott, R. J., Cavana, R. Y., & Cameron, D. (2013). Evaluating immediate and long-term impacts of qualitative group model building workshops on participants' mental models. System Dynamics Review, 29(4), 216-236.

Midgley, G., Cavana, R. Y., Brocklesby, J., Foote, J. L., Wood, D. R., & Ahuriri-Driscoll, A. (2013). Towards a new framework for evaluating systemic problem structuring methods. European Journal of Operational Research, 229(1), 143-154.

Associated Projects

Project: Investigating decision making using stock and flow problems

This project aims to contribute to understanding dynamic decision making by exploring: (1) how people understand the dynamic models underlying a task with various levels of complexity, and (2) mechanisms or tools to improve people’s understanding and decision making. Using a class of problems known as the “Accumulation” or “Stock-and-Flow” problems, we conduct a series of laboratory experiments which allow us to examine individual’s dynamic decision making capabilities.

Findings from this study carry implications for research and education in the areas of decision making and complex problem solving. 

  • On the research side, the study will generate empirical knowledge on factors (i.e. individual-related, problem-related) that explain thinking and decision making capabilities in dynamic situations. This knowledge can be used to validate and extend existing theoretical dynamic decision making models and problem solving methodologies based on these theories.
  • In practice, this knowledge can be used to inform decision making, such as devising mechanisms to select people most well-suited to deal with specific dynamic problems.

On the professional education side, it contributes to the development of effective methods that can improve students’ systems and dynamic thinking ability. Part of this is examining the effects of individual differences in explaining effectiveness of these methods.