Defining Open-Ended Problem Solving Through Problem Typology Framework

Problem solving is central to engineering education. Yet, there is little agreement regarding what constitutes an exemplary design problem or case analysis problem for modeling undergraduate instruction after. There is even less agreement in engineering education literature regarding the best way to measure students’ ability or progress in learning to be better problem solvers in these discrete problem categories, especially with “open” problems. We describe the development of a research method toward accessing how students think about solving different types of engineering problems, like design and case analysis, what constitutes a measurable response, and how to compare through qualitative research methods pre and post student performance. The contribution of this paper is a discussion of our effort to develop an appropriate open problem statement, based on problem typology research of David Jonassen. The discussion draws from Jonassen’s framework, as well as cognitive learning frameworks, and metacognition as a theoretical basis that informs the problem formulation and planned approach for analysis. Keywords—Problem Typology, Engineering, Ill-structured problems, Problem based learning, research instrumentation 1 Engineering Pedagogy and Engineering Problems Engineers are known for defining themselves as problem solvers [1]. It is a common moniker of identity for engineers, one used far more often than “great teacher.” While much of engineering education is delivered through lecture [2]–[8], engineering education reform continues to call upon engineering instructors to increase their awareness of effective pedagogies informed by engineering education research [9]–[12]. In part this is motivated by large attrition rates. Despite the large investments by universities to recruit and graduate high achieving students, large numbers of engineering students leave the field to pursue other majors and professions. According to Chen and Solder, attrition rates of engineering students in higher education exceeded 50 percent between 2003 and 2009 [13]. According to Marra et al., undergraduate students who left engineering indicate three main factors: “poor teaching and advising, the difficulty of the iJEP ‒ Vol. 10, No. 1, 2020 7 Paper—Defining Open-Ended Problem Solving Through Problem Typology Framework engineering curriculum, lack of belonging in engineering” [14], which is also supported by a more recent survey of retention among STEM majors [15]. Active learning is a pedagogical framework that has been embraced by many in the engineering field [4], [6], [16] [21]. Active learning often refers to an instructional approach that encourages students to participate and be engaged in the learning process [4], [6], [22] [25]. In contrast to traditional lecture methods, active learning has been argued to offer a greater contribution in developing students’ motivation and analytical inquiry as a means to better understand science knowledge [26] [29]. Studies report that an active learning approach has increased students’ participation and achievement as compared to a lecturing approach in engineering [18], [19], [30] [32]. While there is no singular instructional method that defines active learning [24], one facet of active learning involves the collaborative work of students on solving open-ended or inquiry problems as a part of instructional practice. For purposes of this discussion, we define “openended” as problems for which there is no singular correct answer nor singular solution path. Jonassen argued that the most important disposition for a professional engineer to acquire is that of “problem solver” [33], [34], but his arguments point to a criticality toward pedagogy that must be adopted by engineering instructors to actually produce strong problem-solving students as a direct result of our instruction. He argued, “The discrepancy between what learners need and what formal education provides represents a problem that instructional design may be able to ameliorate. Why are we so inept at engaging learners in problem solving? A major reason is that we do not understand the breadth of problemsolving activities well enough to engage and support learners in them.” [35] Jonassen’s arguments lead to a framework for examining multiple forms of problem solving, but the types of problems students experience in academic contexts can be fundamentally different from those offered in professional contexts. The National Academy of Engineering makes the same essential argument. The origins of engineering lie in the trades with focus on producing something useful; therefore the formalization of engineering education has served to further disconnect engineers in practice and academic settings [36]. This disjunction has been the focus of much engineering education literature [37]–[39], especially as it pertains to student perceptions of engineering problem solving in academic and professional contexts [40]–[42]. Authors have called for a focus on nontechnical, non-calculative sides, ill-structured problems, conflicting and non-technical success measures, and varied solution strategies to be employed within the experiences of academic engineering learning [1], [33]. Despite recognition that technical engineering competence is strongly tied to professional competencies [39] engineering students often encounter technical and professional skills in disconnected contexts. That is, since much of engineering education is focused on theory, technical skills are encountered through coursework while professional skills become a focus in the context of employment. Some argue that significant improvement to engineering instruction can be found by making explicit technical and professional connections through Problem Based Learning (PBL) and Jonassen argues that PBL environments are important for getting engineering students beyond simple 8 http://www.i-jep.org Paper—Defining Open-Ended Problem Solving Through Problem Typology Framework computational assignments and introduced to a wider array of problem complexity they will face as professionals [33]. 2 Examining Student Performance on Open-Ended Problems But what is a good open-ended problem and what does it look like for students to be successful in solving such problems? To answer the first part of this question, we explored the literature and found no singular answer. The most specific and well-defined notion of how open-ended problems may be framed in the realm of engineering education could be Jonassen’s work on problem typology. Jonassen described 11 types of problems in developing a theory of problem solving [22]; furthermore, he argues that the foremost role of an engineer is that of “problem solver” with a specific focus on problems of design, selection, and troubleshooting [33], [34]. Of particular relevance to this work are ideas regarding the ways in which problem solving differs on dimensions of problem variation, representation, and individual differences of the problem solver. Considering and systematically investigating these dimensions is important to developing learning experiences and associated assessment methods (i.e. open-ended problems) that facilitate intraand inter-comparison among engineering students. We believe the second part of this question – i.e. student success on open-ended problems – resides in our ability to develop in students a way to think about their thinking while engaging in problem solving activities. Such an approach stems from an investigative trajectory where researchers aim to qualify students’ approaches to thinking tasks [43] [47]. In like fashion we are developing a preand post-assessment tool which will offer a more fine-grained analysis of student problem solving in collaborative settings. One vital application of qualitative educational research is to reframe old problems and reveal new approaches for innovative engineering instruction as informed through the exploration of learning theory [48], [49]. We are developing an instrument to explore the role of Jonassen’s problem typology theory in steering pedagogical innovation in experiential learning contexts. We believe that studying students’ responses to open-ended problems before and after the introduction of a problem typology framework [35] in an experiential learning environment can lead to the honing of assessment strategies to actually measure the progression of students’ effective use of theoretical frameworks. Furthermore, we speculate that our findings will add evidence for shifts in metacognition, or thinking about thinking, among engineering students. Metacognition has also been described as “knowing about knowing.” Research reveals that metacognition does not necessarily develop organically through normal educational practices; but it can and should be explicitly taught [50] [53]. Since Flavell [54] first introduced the term, several models and expanded definitions of metacognition along with complex hierarchies of cognitive skills have emerged from the fields of psychology and education [55] [57]. We hold to the definition from Kuhn and Dean that metacognition, “Entails awareness of one’s own thinking and reflection on the thinking of self and others as an object of cognition. Metacognition is defined in similar terms as awareness and management of one’s own thought Metacognitive functions can be procedural or iJEP ‒ Vol. 10, No. 1, 2020 9 Paper—Defining Open-Ended Problem Solving Through Problem Typology Framework declarative. The former invokes awareness and management of one’s own thinking. The latter involves one’s broader understanding of thinking and knowing in general [which has also] been studied under the heading of epistemological understanding.” [58] Most research recognizes two fundamental components of metacognition: cognitive knowledge and cognitive regulation [56], [59], [60]. In our study, Jonassen’s problem typology framework introduces knowledge about availab


Engineering Pedagogy and Engineering Problems
Engineers are known for defining themselves as problem solvers [1]. It is a common moniker of identity for engineers, one used far more often than "great teacher." While much of engineering education is delivered through lecture [2]- [8], engineering education reform continues to call upon engineering instructors to increase their awareness of effective pedagogies informed by engineering education research [9]- [12]. In part this is motivated by large attrition rates. Despite the large investments by universities to recruit and graduate high achieving students, large numbers of engineering students leave the field to pursue other majors and professions. According to Chen and Solder, attrition rates of engineering students in higher education exceeded 50 percent between 2003 and 2009 [13]. According to Marra et al., undergraduate students who left engineering indicate three main factors: "poor teaching and advising, the difficulty of the engineering curriculum, lack of belonging in engineering" [14], which is also supported by a more recent survey of retention among STEM majors [15].
Active learning is a pedagogical framework that has been embraced by many in the engineering field [4], [6], [16] [21]. Active learning often refers to an instructional approach that encourages students to participate and be engaged in the learning process [4], [6], [22] [25]. In contrast to traditional lecture methods, active learning has been argued to offer a greater contribution in developing students' motivation and analytical inquiry as a means to better understand science knowledge [26] [29]. Studies report that an active learning approach has increased students' participation and achievement as compared to a lecturing approach in engineering [18], [19], [30] [32]. While there is no singular instructional method that defines active learning [24], one facet of active learning involves the collaborative work of students on solving open-ended or inquiry problems as a part of instructional practice. For purposes of this discussion, we define "openended" as problems for which there is no singular correct answer nor singular solution path.
Jonassen argued that the most important disposition for a professional engineer to acquire is that of "problem solver" [33], [34], but his arguments point to a criticality toward pedagogy that must be adopted by engineering instructors to actually produce strong problem-solving students as a direct result of our instruction. He argued, "The discrepancy between what learners need and what formal education provides represents a problem that instructional design may be able to ameliorate. Why are we so inept at engaging learners in problem solving? A major reason is that we do not understand the breadth of problem-solving activities well enough to engage and support learners in them." [35] Jonassen's arguments lead to a framework for examining multiple forms of problem solving, but the types of problems students experience in academic contexts can be fundamentally different from those offered in professional contexts. The National Academy of Engineering makes the same essential argument. The origins of engineering lie in the trades with focus on producing something useful; therefore the formalization of engineering education has served to further disconnect engineers in practice and academic settings [36]. This disjunction has been the focus of much engineering education literature [37]- [39], especially as it pertains to student perceptions of engineering problem solving in academic and professional contexts [40]- [42]. Authors have called for a focus on nontechnical, non-calculative sides, ill-structured problems, conflicting and non-technical success measures, and varied solution strategies to be employed within the experiences of academic engineering learning [1], [33].
Despite recognition that technical engineering competence is strongly tied to professional competencies [39] engineering students often encounter technical and professional skills in disconnected contexts. That is, since much of engineering education is focused on theory, technical skills are encountered through coursework while professional skills become a focus in the context of employment. Some argue that significant improvement to engineering instruction can be found by making explicit technical and professional connections through Problem Based Learning (PBL) and Jonassen argues that PBL environments are important for getting engineering students beyond simple computational assignments and introduced to a wider array of problem complexity they will face as professionals [33].

Examining Student Performance on Open-Ended Problems
But what is a good open-ended problem and what does it look like for students to be successful in solving such problems? To answer the first part of this question, we explored the literature and found no singular answer. The most specific and well-defined notion of how open-ended problems may be framed in the realm of engineering education could be Jonassen's work on problem typology. Jonassen described 11 types of problems in developing a theory of problem solving [22]; furthermore, he argues that the foremost role of an engineer is that of "problem solver" with a specific focus on problems of design, selection, and troubleshooting [33], [34]. Of particular relevance to this work are ideas regarding the ways in which problem solving differs on dimensions of problem variation, representation, and individual differences of the problem solver. Considering and systematically investigating these dimensions is important to developing learning experiences and associated assessment methods (i.e. open-ended problems) that facilitate intra-and inter-comparison among engineering students.
We believe the second part of this question -i.e. student success on open-ended problems -resides in our ability to develop in students a way to think about their thinking while engaging in problem solving activities. Such an approach stems from an investigative trajectory where researchers aim to qualify students' approaches to thinking tasks [43] [47]. In like fashion we are developing a pre-and post-assessment tool which will offer a more fine-grained analysis of student problem solving in collaborative settings. One vital application of qualitative educational research is to reframe old problems and reveal new approaches for innovative engineering instruction as informed through the exploration of learning theory [48], [49]. We are developing an instrument to explore the role of Jonassen's problem typology theory in steering pedagogical innovation in experiential learning contexts. We believe that studying students' responses to open-ended problems before and after the introduction of a problem typology framework [35] in an experiential learning environment can lead to the honing of assessment strategies to actually measure the progression of students' effective use of theoretical frameworks. Furthermore, we speculate that our findings will add evidence for shifts in metacognition, or thinking about thinking, among engineering students.
Metacognition has also been described as "knowing about knowing." Research reveals that metacognition does not necessarily develop organically through normal educational practices; but it can and should be explicitly taught [50] [53]. Since Flavell [54] first introduced the term, several models and expanded definitions of metacognition along with complex hierarchies of cognitive skills have emerged from the fields of psychology and education [55] [57]. We hold to the definition from Kuhn and Dean that metacognition, "Entails awareness of one's own thinking and reflection on the thinking of self and others as an object of cognition. Metacognition is defined in similar terms as awareness and management of one's own thought Metacognitive functions can be procedural or declarative. The former invokes awareness and management of one's own thinking. The latter involves one's broader understanding of thinking and knowing in general [which has also] been studied under the heading of epistemological understanding." [58] Most research recognizes two fundamental components of metacognition: cognitive knowledge and cognitive regulation [56], [59], [60]. In our study, Jonassen's problem typology framework introduces knowledge about available strategies and when to use a particular strategy, thus it contributes to students' cognitive knowledge. Interventions and coaching sessions that demonstrate how to use problem typology in planning solutions, monitoring progress, and evaluating proposed solutions exemplify cognitive regulation [61], [62].
We hold to the notion that all learning is context-dependent and socially constructed [63] [66], therefore, rather than focusing on students' individual cognitive awareness, we see developing metacognition as an iterative process co-constructed by the students and the teacher/researcher in the experiential learning environment. When students are taught within the context of PBL with their peers and guided to think about their thinking through questions like, "How do you know you're right?" "Why is your solution better than your partner's?" and "What kind of problem do you think this is and what is your evidence?" they will appropriate this kind of discursive norm and become better at examining their own thinking. Kuhn also argues that students' fundamental epistemology towards knowledge can be impacted in such instructional contexts. For our study we look toward metacognition as the monitoring of one's own cognitive process and influences when they are focused on a specific task or goal [45], [54], [67]. In addition, metacognition deals with the awareness of how one learns, the ability to judge the difficulty of a task, the monitoring of understanding, the use of information to achieve a goal, and the assessment of the learning process [54].
Flavell [54] stressed metacognition as a major influence in cognitive performance. Further, he suggests that in order to monitor a student's cognitive processes it is necessary to consider four areas of metacognition: • Metacognitive knowledge • Metacognitive experiences • Goals (tasks) • Actions or strategies Two of the factors that are specifically utilized in our study is harnessing students' metacognitive knowledge, and metacognitive experience. Firstly, metacognitive knowledge engages students' beliefs toward their personal ways of knowing, and secondly, the development of experiences reflecting upon tasks and individual stances whether brief or extended [35], [54], [68], [69]. In our study we aim to draw out students' metacognitive experiences through problem discussions, reflections and interviews, assessed through a form of discourse analysis. Attaining metacognitive thinking at these specific moments are especially likely to occur as the activities "stimulate a lot of careful, highly conscious thinking" in the tasks we give to students, which represent "novel roles or situations, where every major step you take requires planning beforehand and evaluation afterwards" in solving different types of engineering problems [54].
There are researchers who have argued that failure to explicitly attend to metacognition strategies results in errant trajectories of learning which have to be "unlearned" before expertise in the field can be attained [44], [70] [72]. Steif, et al. explored metacognition strategies with pre-and post-assessments and found that experts pay considerable attention to the complete representation of the problem before selecting a strategy to proceed toward a solution. Their research showed novices, "…tend to immediately jump to some type of detailed analysis, often based on a poorly conceptualized and incomplete representation of the problem. As a result, they may omit or mischaracterize key aspects of the problem and fail to successfully solve it." [44] Researchers examining such aspects of student thinking suggest that students can become better learners [41], better self-correctors of their own thinking [51], better perceivers of core disciplinary concepts [44], and better problem solvers by developing more accurate and relevant approaches to internalizing and solving open-ended problems [61].
Adopting Jonassen's work for framing educational experiences for undergraduate engineers and Flavell [54], [73], [74], and Kuhn and Dean's [58] framework for metacognition, we propose that the problem typology framework can be studied and instrumental for informing, assessing, and guiding student problem solving in experiential learning contexts. Using mixed methodology we are exploring students' problem solving abilities, epistemological stances, recognition of typology in and out of an experiential learning project context, and students' ability to convey the essence of their experiential work in professional contexts. As part of the research we have particular interest in collecting data that reflects how engineering students think about different types of problems, including case analysis, as well as design problems. Design is considered the fundamental activity of engineering [75], [76], which often leads to emergence of other engineering problem types, like case analysis, selection (decision-making), troubleshooting, and planning. Jonassen himself directed researchers to explore the role and manifestation of metacognition in solving ill-structured problem solving arguing, "Research on the role of metacognition in problem solving has focused primarily on solving mathematical story problems, which are typically well-structured [however] almost no research on the role of metacognition in solving ill-structured problems exists" [35]. Pursuing this claim has led to a fundamental question of interest that is explored in this paper: How do we formulate open-ended engineering problems that fit a definition within a framework of problem typology to conduct comparative research on students' metacognitive thinking?
There would be no possibility to answer the question of what it looks like to succeed until we refine the nature of the problem to which students respond. Only then can we hope to compare students' responses and growth. To accomplish such refinement, we are working to apply Jonassen's framework to analyzing the thinking of engineering students engaged in PBL. We unpack our analysis, revision, and development of an instrument for examining problem solving in context as we aim to use this tool for examining student growth in recognizing and utilizing problem typology in engineering education.

Methodology
Overall, our research is concerned with the development of metacognitive strategies of students and how problem typology can be central in developing the connections between technical and professional competencies that are critical to the effectiveness of engineering practice [39]. At this time we consider problem types of design and engineering case analysis. Case analysis is represented in the process diagram of Figure  2, which is used as part of discussing engineering problem solving with students. Design and case analysis are the primary problem types encountered by students in the experiential learning program that serves as the basis for data collection. The development of these problem type frameworks is to provide a generalizable schema by which engineering students may consider problems and represents explication of problem types as discussed by Jonassen [33], [35]. Through such process diagrams, we seek to provide students from multiple disciplines a "discipline agnostic" framework to consider open problems, spanning multiple domains, that they are challenged to solve. In this paper only the engineering case analysis problem is considered. One goal of our study is to access student metacognition through discourse analysis. Because we recognize that having students discuss problem solving will be important to our study, we are currently collecting data by observing and recording students' discussion of design and engineering case analysis problems; however, we currently lack a framework for measuring students' metacognition. Lin [77], Lippmann and Group [78], and Black [79] all point to ways in which discourse can be utilized as a form of analysis. Through our current investigation of examples in the literature, we will determine which approach will best measure and evaluate metacognition through discourse analysis.

Engineering Case Analysis
We are currently gathering problem solving events and reflective interviews with students taking part in semester long extracurricular projects organized by the School of Engineering and Applied Sciences office of experiential learning. Before and after these extracurricular projects we engaged students in problem solving discussions wherein students are presented both a design and engineering case analysis problem statement. The case analysis statement is shown in Appendix 1. This data will inform our study on students' use and understanding of problem typology. Our mixed methods approach will use epistemological, professional, and experiential inventories to triangulate with interviews to qualitatively analyze student thinking and shifts in their use of typology as an organizing framework for engineering problem solving. Part of our analysis will use discourse analysis [80] [83] coded students' discussions as it mapped onto specific phases of the problem type process frameworks of Figure 1.
The development of a data collection instrument -i.e. the problem statement -is a fundamental consideration. Of particular relevance to this work are ideas regarding the ways in which problems differ on dimensions of variation, representation, and differences among individual problem solvers [35]. We pay particular attention to problem framing in light of Jonassen's problem solving characteristics. Our research considers these attributes as an important part of creating an engineering problem statement that will evoke detailed responses from students. It is through consideration of these attributes that we seek to refine the development of appropriate problem statements. The next section describes our efforts in developing appropriate problem statements for our research. We believe this to be an important aspect of engineering education deserving broader consideration as it relates to teaching and assessment of student learning.

Refinement of our evaluation instrument
We reviewed our initial instrument, an open-ended problem which has been assigned in past years to engineering students as a case analysis as a regular class activity. Although other better formulated and more widely distributed case analysis have been used under this specific problem typology, it was one which was familiar to the department. As we applied Jonassen's framework, we explored each of the problem facets for its meaning, clarity and embodiment within the instrument. Our task was to match as closely as possible all the rich criteria Jonassen has outlined as essential for defining open-ended problems for PBL instruction [35], [84].
According to Jonassen, problem variation is concerned with characteristics of problem structuredness (well-vs. ill-structured), complexity, and domain-specificity (abstract vs. situated). Well-structured problems are those that are found most commonly at the university level at the end of textbook chapters, with well-defined constraints and narrow context for relevant concepts, rules and principles. Ill-structured problems however, are more commonly encountered by engineers in professional practice and do not necessarily specify constraints or imply a solution path. Instead, ill-structured questions require the integration of multiple content domains and strategies to solve the problem [35]. Complexity considers characteristics like the number of issues, functions, or variables in a problem; connectivity of those characteristics; and the dynamic nature of the problem [35]. Problem complexity is fundamental to defining and making tradeoffs that are common in engineering contexts. Finally, domain-specificity considers the situatedness or context-specificity of a problem and the extent to which a problem crosses multiple disciplines, requiring domain specific solution principles.
With regard to problem representation, context, problem cues or clues, and modality are important characteristics. Context is valuable in helping problem solvers determine which information is important and which is irrelevant. Problem clues and modality (i.e. the way in which particular information is presented) are particularly important to the design of educational problems. Under the control of the instructional designer, these problem attributes may significantly impact the difficulty of the problem and/or the type of problem solving tasks the instructor wants to see from students.
There exist a number of differences between ill-structured problems and a range of characteristics which may impact problem solving performance. These may include but are not limited to include the problem's content domain, structure of canonical knowledge, requisite procedural and conceptual knowledge. Similarly the differences between problem solvers suggest a range of learners and general problem-solving strategies may be impacted by affective attributes like self-confidence, motivation, and perseverance.
In developing an appropriate data collection instrument (problem statement), a subset of these characteristics is considered to formulate the instrument. The following is the text from our original research instrument for assessing open-ended case-analysis problems. The original problem also contained a map showing the geographical relationship among the three cities, as well as a few charts related to the cost of specific desalination methods. Refinement of the instrument is motivated by the text, so we limit presentation to that information.
Line numbers serve as a guide to the identification of weaknesses and improvement in our initial research protocol. Since both Jonassen's criteria and the instrument are central to our study, we offer specific guidance to our readers as to how our instrument was refined along with justification for the revisions that resulted in the final instrument (Appendix 1). These criteria include: 1. Complexity 2. Familiarity 3. Attending to structural knowledge domains 4. Metacognition 5. Epistemological commitments 6. Attention to affective domains.
We describe our attention to each of these below (See Appendix A, facets 1-6).

Line 1 Water Desalination Plant Recommendation
Line 2 Water covers approximately 70 percent of the Earth's surface, but less than 1 percent of Line 3 that is available for human use. The world must share this small amount for agricultural, Line 4 domestic, commercial, industrial, and environmental needs. Across the globe, water Line 5 consumption has tripled in the last 50 years. Managing the supply and availability of Line 6 water is one of the most critical natural resource issues facing the United States and the Line 7 world. With water use in the United States increasing every year, many regions are Line 8 starting to feel the pressure. Given the need for more fresh water, and the increasing Line 9 prevalence of drought throughout the U.S., especially in California, the cities of San Line 10 Francisco, Oakland and San Jose are considering building desalination plants to Line 11 supplement current fresh-water resources. Assuming these municipalities move ahead Line 12 with building desalination plants, each must decide whether to build their own plant or Line 13 to collaborate with one or more of the other cities. The objective of each city is to Line 14 minimize the construction cost per million gallons of water per day (MGD) while Line 15 providing appropriate water quality for typical uses within the city (e.g. home use, Line 16 irrigation, industrial use). In addition, each city is looking to supplement at least 30% Line 17 of their water supply through 2050.Your team has been contracted to provide a Line 18 recommendation to these three cities on their most appropriate path forward.

4
Findings: Reformulating the Case Analysis Problem

Complexity and structuredness [lines 9-13]
Jonassen argues that problem difficulty is a function of problem complexity [35]. Our original problem asks students to consider a desalination project shared among three sister cities as a safeguard for drought and disaster. At first glance our notion was that our example was adequately complex and that students had relatively good success when engaging with this problem over the course of many weeks. Jonassen argued, "Complex problems involve more cognitive operations than simpler ones accommodating multiple factors during problem structuring and solution generation places a heavy burden on working memory. The more complex a problem, the more difficult it will be for the problem solver to actively process the components of the problem. [In addition] Complexity and structuredness overlap ill-structured problems tend to be more complex, well-structured problems, such as textbook math and science problems, tend to engage a constrained set of variables that behave in predictable ways." [35] When considering lines 13-19 our task did not represent highly structured problems typified by textbooks with predictability. Our ill-structured problem allowed for a vast array of approaches to which only a few may be fruitful. While the student is not required to select details as finely determined as pump capacity, filter rates, and cost, it also does not offer sufficient guidance to support the student toward a meaningful analysis with core constructs. For this reason, we revised the question toward a case analysis which offered a more structured problem, defining the concept further for water purification. In our case we defined a flatbed truck delivery system to analyze (See Appendix A, modification 1) and further narrowed the ill-structured problem by naming it a feasibility analysis. We argue that these modifications have limited the amount of working memory students need to maintain regarding unrelated concepts of desalination chemistry, distances from ocean sources, fluid dynamics, energy efficiencies, and civic politics of sharing costs for different cities. These modifications help students with a high school physics background to contribute to developing a model for this problem, which makes it ideal for our designed assessment, which can include undergraduate freshman and sophomore engineers, to measure and gauge the effectiveness of typology.

Familiarity (Domain knowledge) [lines 3-4, 9-11]
Presenting students with blind problems to solve as a process to assess their use of typology can completely undermine the study if the student has no context nor experiences in which to situate this knowledge. For over 40 years, educational researchers have emphasized background knowledge and experiences, as essential for accommodating, not assimilating new knowledge [85]- [88]. While assimilation is the process of fitting new knowledge into existing schema, accommodation is a different process. This kind of cognition requires the questioning and reexamination of existing schema, the creation of new categorical knowledge for sense making, and discarding less useful structures for accessing this new knowledge [89]. Jonassen argues, "Perhaps the strongest predictor of problem-solving ability is familiarity Experienced problem solvers have better developed problem schemas, which can be employed more automatically." [35] Failure to recognize the importance of background experiences and knowledge runs the risk of students not recognizing any usefulness of prior knowledge and applying entirely wrong schema to the problem they are presented [90], [91]. Problem solving requires knowing not only what to monitor but also how to monitor one's performance and sometimes unlearning misapplied strategies and poor problem solving habits [92]. For this reason, when we considered our research subjects all lived less than 10 minutes from two Great Lakes in the midwest, California cities seemed a problematic option. Though they experience water issues that attract more national attention, the familiarity of the Great Lakes Region warranted a local problem. By making this revision (Appendix A, facet b) we are able to measure and compare students' familiarity of problem types rather than familiarity with basic contextual knowledge.

Attention to structural knowledge domains [lines 13-17]
Presenting students with open problems is to put them in situations in which domain and structural knowledge are (understandably) limited. Such cases would make for challenging problem-solving scenarios and may put students in positions where problem solving progress is inefficient and perhaps even limited in success. Jonassen contends, "Domain knowledge and skills are very important in problem solving. Structural knowledge may be a stronger predictor of problem solving than familiarity. Robertson (1990) found that the extent to which think-aloud protocols contained relevant structural knowledge was a stronger predictor of how well learners would solve transfer problems in physics than either attitude or previous experience solving similar problems. Structural knowledge that connects formulas and important concepts in the knowledge base are important to understanding physics principles. Gordon and Gill (1989) found that the similarity of learners' graphs (reflective of underlying cognitive structure) to those of experts was highly predictive of total problem-solving scores (accounting for over 80% of the variance) as well as specific problem-solving activities. Well-integrated domain knowledge is essential to problem solving." [35].
However, these types of situations are important for students to encounter, given appropriate guidance and expectation, as they represent a reality of engineering practice. Namely, the types of open problems practicing engineers solve often require domain knowledge that is either shared among multiple collaborators or that must be acquired or developed as part of the problem-solving process. The ability to coordinate and develop domain knowledge through structural representations is important but needs to be attended in the problem description. In our refinement we reduced the use cases to purifying water for consumption alone and restricted the design concept to a single entity -a flatbed truck with a purification system -rather than a distributed system shared among cities that purifies water for different uses like irrigation, industry, and consumption (Appendix A, facet a). Further, we provided a prompt that suggest specific purification technologies, providing leads for researching appropriate domain knowledge that can be coordinated through a conceptual (structural) understanding of the intended purification platform in support of conducting a feasibility analysis with appropriate documentation (Appendix A, facets 3 and 4). Supporting learning experiences that require building and acquiring knowledge as part of engineering problem solving is an important aspect of inquiry-driven learning required in PBL environments.

Metacognition [lines 1-19]
Like Jonassen, we adopt the notion of metacognition from Flavell [54], and Kuhn and Dean [58] who described metacognition as an awareness of how one learns, how one monitors understanding, and how one self-assesses their learning progression. Jonassen further argues that, "The development of metacognitive skills enables students to strategically encode the nature of the problem by forming mental representations of the problems, select appropriate plans for solving the problem, and identify and overcome obstacles to the process." [35] According to Kuhn and Dean, metacognition is "awareness and management of one's own thought, or thinking about thinking." [58]. As students approach open-ended problems, part of their thinking should focus upon the intent of the problem, the actions and beliefs of the learner and their consequences, the nature of the ultimate solution and not the numerical value by itself.
As we examined the literature and Jonassen's original explanation for metacognition we found no supports within our 19 lines of our question that would prompt such thinking. Moreover, we could not measure growth in the students pre-and post-assessments to evaluating their metacognition with the problem as it stood. We therefore recrafted the initial analysis problem to reflect a process of revisiting their learning approaches and supports for thinking about their thinking (See Appendix 1, facets 3 and 5). The first modification was to focus the task which highlighted the purpose and contents of their solution. We indicated that their final solution [the report] needed to contain an ANALYSIS which prompts the student versed in typology with an appropriate approach to solving this open problem. Next, we encourage students to consider documentation of their research to support assumptions they have made. Our final modification for metacognition in our prompt was an offering of guidance to students towards the issues they may have in justifying their solutions and provision for recommendations. The bullets are the last words read out loud by the researcher administering the recorded problem-solving session. The students describe the nature of the solution and some potential facets of a solution which may be scrutinized by outsiders for the credibility, efficiency, and the accuracy of their proposed solution.
In addition to recording the problem-solving assessment to the scaffolded questions, we also conducted debriefing interviews to explore the metacognition, or the thinking about their thinking, immediately following their pre and post assessments. This is discussed in detail in the next section.

Epistemological commitments [lines 13-17]
Epistemological commitments refer to the students' awareness of and their ability to communicate how they know what they know. We consider epistemological commitments one of several aspects of metacognition which shape how students appropriate problem typology into their thinking. In order for us to explore their approaches to problem solving and their application of problem typology, we needed to understand when, why, and how they apply typology in evaluating their solutions. Kuhn argues that epistemology becomes important to learning when students become aware that knowledge is constructed and proceeds from people and not from facts. Students need opportunities to be supported in their thinking and judgment of opinions from fact and from rational versus irrational lines of thinking.
Some researchers have developed instruments and surveys which add quantitative measures of students' world views and epistemology [41]. Our methods then require us to push students' typical thinking to explain how they know and how they judge the appropriateness of their approach. Questions like, "How do you know this? Show me which you contributed, and why it is better. How do you compare who is right?" are all questions towards epistemological beliefs. Only by asking them the uncomfortable questions in the beginning before they are introduced to learning context where justification and metadiscourse is regularly scaffolded and demanded of students, can it be compared in an exit interview as a measure of shifts in epistemological commitments.
Researchers claim that more complex and more ill-structured problems require a higher level of critical thinking and an examination of one's own thinking-a practice seldom reinforced in typical lecture format instruction [41], [93]. Greeno [94] argues that the common belief held by students that mathematical problems are best solved by applying procedures carries with it another belief that as a learner I either know or do not know the correct procedure. This common belief is neither a good representation of the work of mathematicians nor engineers, yet is commonly shared because of the social construction of the learning context of both disciplines for delivering content. In contrast, Gourgey [95] identifies the patterns of good problem solvers: to identify elements of a problem, to clarify their goals, to choose strategies, to understand concepts, to monitor their understanding, and to evaluate actions that lead toward the overall goal of a reasoned solution.
Our experience with students open problems suggests that students approach problems with efficiency in mind. Very little thought is often given to how the problem will be evaluated, what knowledge students can rely upon or believe, the basis for their belief in best strategies, and what biases the students may be bringing to the problem at hand. It is a dangerous process that drives engineering students to just crunch the numbers. Our first approach in examining students' epistemological commitments in our analysis was to direct students in the instructions and on the stated problems with ways to think about HOW they would know if they were right. Bullets are provided to guide students in thinking about the appropriateness of their solutions and guides them toward specific characteristics that should be a part of their answers (Appendix 1, facets 4 and 5). In addition to the scaffolding of the stated problem our second approach for examining epistemology, was to debrief interviews directly following their recorded problem-solving sessions. We asked questions like "Tell me how you would evaluate if you were right. Tell me why you drew _____ this way" (Appendix B, Interview Protocol). Our third approach to supplement our problemsolving scenarios through pre and post interviews was to draw from the epistemology inventory research. Researchers like Faber and Benson [41] and Chinn et al. [93] have studied the influences on students' approaches to problem solving. Faber and Benson demonstrated through quantitative survey instruments that engineering epistemic beliefs and students' epistemic motivation affect the approaches students take when solving an open-ended homework problem [41]. Their work challenged the assumption that students holding a positivist or constructivist perspective can make better epistemic gains based upon their world view and that both may be subject to an inherent close-mindedness about classroom tasks and assigned problems. They wrote, "Our findings revealed the possibility that students' epistemic beliefs prompt, but do not guarantee, their adoption of epistemic aims depending on the situation." Rather, Faber and Benson argue that in order to properly support student learning, "we must fully understand the factors that influence the approaches and strategies students use, including epistemic metacognition." Jonassen would agree that the relationship between student epistemology and ill-structured problem solving needs to be carefully examined [35].

Attention to affective domains [lines 1-3, 8-10]
For students to effectively and authentically engage all their cognitive resources and reasoning capacities on problem solving, Jonassen argues that there must be attention to the affective domain of an ill-structured problem [35]. Having given this problem for students to solve in other instructional contexts we recalled evidence of how lack of caring about an open problem led to common errors, lack of internalization of the task, and recall of the problem solving process they chose. When examined through the lens of professional competencies, students exiting the program often could not explain their project, their role, or their reasons for choosing their problemsolving strategy when explaining their work to external potential employers. Engineering students in our program tend to treat tasks as transactional. Some approach their experiential learning projects as ill-structured real problems and bring a disposition of, "Right. That's it, it's a project. I need something on my resume. I should do this because I am getting a grade and it will get me a good job." None of these reasons taps the learners' affectual domain of the problem.
Our original problem referred generally in lines 1-3 to a global issue of water shortages. While it is a relevant, realistic, and contemporary problem, most of our students live a very short distance from two of the largest bodies of fresh water on the earth (Lakes Erie and Ontario). Few if any of our students come from backgrounds where water shortages or crises occurred. Further, our problem was situated in three cities that most students in our program had never visited. California is as foreign to most of our students as Canada or any other English speaking country.
To remedy this we chose cities which were in the local news regularly and we also situated it on a map which provided geographical context (See Appendix A, facet 2). We also provided a range of common local water quality issues for students to posit the larger problem of water quality within the realm of humanly preventable tragedies. In some of the example cases we reminded students of events which human error caused civic tragedies to evoke some common interest for the feelings of outrage or despair (See Appendix A, facet 6). Finally, we diversified the given data for male and female consumption to bring about an intentionally gendered view of the impact of the local water problem.
We should note that affect plays a larger role with some students than with others. There is much literature to support the notion that redesigning engineering curricula around societal improvement and attention to an emotional, caring approach to engineering would stem the exodus of diversity from the ranks of engineers [14], [96] [98]. The traditional problem solving, lecture and Socratic Method of questioning leaves female engineers in particular with less confidence making them less apt to engage in illstructured problems. Women and other under-represented students may compare themselves against their classmates, which often results in negatively impacting the selfconfidence and the development of a positive engineering identity for students of diverse backgrounds [99]. Consequently, it may be more important to include attention to affective domains of assigned problems for retaining those students exiting the pipeline than for assuring the success of those who already do well for a variety of other reasons.

Conclusion: Final Thoughts on Open-Ended Problems and Aspirations for Future Work
Solving open-ended problems in engineering is accepted as exemplary practice. It is also a practice which generates strong debate with regard to what matters in assessing students' proficiency. One such contention is that there is no agreed problem type that would best represent engineers' thinking. According to Jonassen, there are at least six different problem types that are representative of the discipline. We endeavored to explore just one within Jonassen's problem typology framework [35] -namely Case Analysis. Even within a specific type of analysis problem, there is little agreement among experts regarding what an ideal example problem might look like. A hundred different engineering educators may give you a hundred different example problems that they believe best represent what they want their students to know, to practice, and to believe. An even deeper problem of teaching students to solve open-ended problems is that there is no one agreed way to assess students' proficiency. While some evaluators may favor accurate calculations, another may favor innovation and creativity, and yet another may favor civic responsibility or efficiency.
How is one to measure student growth in answering open-ended problems? The field of engineering education research discuss engineering dispositions, professional competencies and practices, and expert thinking as recognized tensions [100]. There is no agreement of what experts look like when doing this. Nor is there any research evidence that would conclusively support any single inventory or protocol or rubric to measure students' growth along the way as they build expertise in solving open problems. Our approach was to take a well cited theoretical framework in engineering problem typology and apply what we know as educational researchers to this framework in order to create a protocol of pre-and post-assessments with a refined instrument to begin measuring students' metacognitive growth. At a minimum, we hope to capture students' use of problem typology in the language and the solutions they create. Aspirationally, we hope to map a set of attributes which can describe growth and perhaps even stages of growth in the appropriation of problem typology and its norms of discourse within the solutions and long term trajectory of our engineering students. Perhaps we may even create enough discussion surrounding engineering case analysis and problem typology that the field may eventually come to some agreement about what an expert problem and solution may look like for students.