Measuring Engineering Faculty Views about Benefits and Costs of Using Student-Centered Strategies

Dispositions of 286 engineering faculty members were assessed to determine views about three student-centered classroom strategies and how frequently faculty used those strategies. The student-centered classroom strategies examined were: using formative feedback to adjust instruction, integrating real-world applications, and promoting student-to-student discussions during formal class time. The Value, Expectancy, and Cost of Testing Educational Reforms Survey (VECTERS), based on expectancy theory, was designed, tested, and validated for this purpose. Results indicate using strategies, such as formative feedback, are significantly tied to perceived benefits and expectation of success. Using student-centered strategies is inversely related to the perceived cost of implementation – with more frequent users perceiving lower cost of time and materials. Keywords—Student-centered; expectancy theory; formative feedback; engineering education


Introduction
As part of an evaluation of a grant funded professional development for engineering faculty, a goal was to detect attitudes of faculty members toward specific practices and compare that to how often the practices were used in classrooms. This is different than evaluating faculty dispositions toward universal concepts such as inquiry learning or constructivism. A review of research led to several related instruments, but none provided the desired functionality. Ultimately, what was found lacking in the canon was an instrument detecting dispositions about specific strategies, particularly for engineering. The desire was to evaluate mindsets regarding specific teaching strat-egies and begin to understand what variables may be affecting use and nonuse. Because it had been informally observed that engineering professors may integrate one student-centered strategy but not another, it was preferred to evaluate dispositions per strategy. To achieve this goal, a new instrument, based on expectancy theory, was developed: this was the Value, Expectancy, and Cost of Testing Educational Reforms Survey (VECTERS).
In this paper we first provide a reporting of the VECTERS instrument construction and usability that includes its development, theoretical framework, results from testing with a large sample, and its revised form. The three classroom strategies integrated into VECTERS are (1) using formative feedback to adjust instruction (2) integrating real-world applications, and (3) facilitating student-to-student discussion during class time. Second, we address our primary research question: What is the relationship between the use of a particular student-centered strategy and an instructor's disposition regarding, (a) value of the strategy, (b) expectation of the strategy to succeed, and (c) considered cost of using the strategy? 2 Relevant Literature

Student-Centered Strategies
VECTERS solicits views about the implementation of three student-centered learning strategies. While there are certainly other strategies that fall under this banner, the three selected are pedagogical rallying points promoted within the professional development. In general, the professional development encouraged instructors to move their classrooms toward being environments where students have voice, instructors are responsive to varying student backgrounds, and relevancy between coursework and real-world applications become evident. The limits of this paper do not allow an exhaustive review regarding the three strategies, nevertheless an overview is provided for each.
Formative feedback. The iterative use of formal and informal assessments as a means to support a learner is valuable to the learning process [1]. The concept of learning from one's own attempts and integrating newly developed understandings to existing schema aligns to learners using formative feedback to construct and reconstruct their knowledge and skills [2]. Much of the literature on formative feedback focuses on learners being primary consumers of the feedback and using it to consciously improve their own understanding [3]. In higher education, the activity of formative feedback often ends with the transmission of the feedback from instructor to student, with the onus on students to make improvements [4].
The professional development from which VECTERS emerged focused on deliberate use of formative feedback that is two-way. This emphasizes instructors adjusting instruction based on what they learn about students' comprehension. This moves the dynamic of formative feedback beyond students just studying "what they missed," and places responsibility on faculty to adapt instruction. This may take the form of immediate adaptation of instruction based on responses from electronic response systems, aka "clickers" [5]. Likewise, formative feedback may manifest through formal collection of what students find to be the "muddiest points" from a lesson followed by altering instruction the next day [6]. This type of instructional responsiveness has shown to positively affect classroom dynamics as well as persistence and achievement in undergraduate engineering courses [7,8].
Real-world applications. Clearly many facets of an engineering course can be argued as relating to the real world. To provide focus, the definition of real-world applications is described as when an instructor deliberately demonstrates relevance through the integration of problems that are related to real-world problems and/or underscores connections to industry and design. Integration of pedagogy that emphasizes relevance and connections to the real-world have been shown to support student engagement, persistence, and comprehension [9]. Integration of real-world applications moves the responsibility to instructors to be explicit about the real-world application of what is being learned and to clarify how future careers integrate these skills. Realworld connection can manifest on a large scale such as having students form engineering design teams that address community problems [10] or more ordinarily occur as emphasizing real-world connections in the form of contextualized problems and workplace connections [11].
Student to student discussion. When instructors attempt to make their classes more dynamic, getting students to engage in discussion is one of the most popular strategies [12]. The term discussion in a college classroom context can have broad interpretations including integrating peer tutors or even interpreting the use of an electronic response system (clicker questions) as a type of discussion. Discussion can also be minimally interpreted as a lecturer asking a series of questions followed by a negligible amount of student responses before proceeding. Here discussion is defined as student-to-student discourse that is deliberate, occurring during class time, initiated by the instructor, and focused on furthering understanding of concepts.
Although research supports the efficacy of student-to-student discussion [13], actual use of the strategy in college lecture halls has been slow to progress [14]. Facilitation of discussion that seeds students with questions that flow within the context of a lesson can be helpful in promoting comprehension of new concepts. If students are left to their own devices and merely encouraged to discuss with one another after class, they may not possess necessary linguistic and interactional skills needed to develop shared meaning as they would during facilitated classroom discussion [15].

Expectancy Theory
Expectancy theory in educational research literature is typically described from student perspective. A student's expectation of how well they will perform on a task is viewed as influencing effort and consequently performance [16]. From an instructor's perspective, expectancy theory frames the effort that must be expended in order to modify instruction. In this context, our expectancy framework is based on an expectation of success, considered value, and an accounting of costs.
Related to the expectancy of success is the value that individuals place on attainment of an end goal. Attainment value therefore also predicts effort and determination [17,18]. Value is sometimes equated as a combination of the value of the input (i.e., costs) plus the value of the output. This combines the cost of achieving a goal with the attraction of achieving the goal. We chose to separate these elements of value. The input values are considered costs and the outputs are value.
Based on expectancy theory, implementation of an educational reform often meets limited success for one or more of three reasons: perception of low value, belief of likely to fail, and assessment of high cost. In many cases the reform is never even transferred from professional development to the classroom because an instructor believes the strategy will have little or no added value for students; or because instructors anticipate instituting the strategy will lead to a less effective learning environment; or simply because instructors consider the expenditure of time and materials too great of a price tag to pay.

Research Questions
1. What is the relationship between the value placed on a student-centered teaching strategy and use of the strategy? 2. What is the relationship between the expectation of success with a student-centered teaching strategy and use of the strategy? 3. What is the relationship between the cost associated with implementing a studentcentered teaching strategy and use of the strategy?

VECTERS Instrument
The three constructs of VECTERS are based on expectancy theory: value, expectation of success, and cost.
Value. The construct of value is tied to benefit. Implementing a classroom strategy may have benefit for students but it can also be supposed that the strategy can have detrimental effect (i.e., negative value). VECTERS contains eleven value items. Eight of the value items address perceived value (negative and positive) for students; and three of the value items focus on how implementing a strategy may have direct value for the instructor.
Expectancy. The expectancy construct hinges on the vision of the learning environment when the strategy is implemented. These visions, or expectations, of the learning environment outcome are categorized by internal and external attribution types. That is, expectation of success, or lack thereof, might be attributed to students' ability to "handle" the strategy, or might be attributed to the instructor's view of their ability to implement the strategy. Further, expectation of success can be externally attributed to the physical classroom environment -a lecture hall versus a small classroom, or hundreds of students versus a couple of dozen students. VECTERS contains ten expectancy items. Five of these items align with expectancy related to students, two items focus on expectation of success due to the instructor's capacity, and three items associate expectancy of success with the physical environment or actual content.
Cost. Cost items address the perceived expenditures of implementing a classroom strategy. VECTERS includes five cost items. Among these five items, three address time costs, one item addresses the cost of teaching assistants, and one addresses the cost of overall effort in implementing a strategy.
Overall Design. Twenty-six value, expectancy, and cost items appeared on VECTERS as a mix of both negative and positive statements to which respondents indicated their level of agreement on a Likert four-point scale, from strongly disagree to strongly agree. Engineering faculty responded to the 26 items for each of the three classroom strategies (formative feedback, real-world applications, and initiating student-to-student discussions), thus yielding 78 datum points. An example of the question format is provided in Table 1. This layout was influenced by the work of Abrami, Poulsen, and Chambers [19] who developed the cooperative learning implementation questionnaire (CLIQ).

Sample
To acquire an adequate sample for testing VECTERS, an invitation to complete the survey was sent to 19 of the 20 largest colleges of engineering in the United States. One of the 20 largest colleges is the authors' institution and was omitted since several of those faculty members would be requested to complete a subsequent version of VECTERS. Engineering faculty members were invited via email to complete VECTERS online. The invitation was for faculty who taught undergraduate engineering.
A total of 286 responses were received. Though this was suitable to conduct reliability and validity testing, it was not possible to determine the response rate because the request was sent to all available email addresses of engineering faculty members listed on college websites and many of those email addresses were associated with faculty not teaching undergraduate courses.

Coding
Data from negatively worded value and expectancy items were adjusted so that a 4 on the 1 to 4 scales indicated perception of high value or high expectancy of success. Likewise, data from cost items that implied high expenditure (e.g., implementing this strategy takes too much preparation time) were adjusted so that a response of 4 indicated the respondent viewed cost as being high.

Internal Consistency
Reliability of VECTERS was assessed by calculating Cronbach's alpha coefficients. The Cronbach's alpha coefficient calculations were first made across all 26items for all three strategies (i.e., 78 items collectively), and then applied to each of the three sub-instruments (formative feedback, real-world applications, and studentto-student discussion).

Construct validity
VECTERS construct validity was evaluated by examining relationships between respondents' self-reports of extent to which the three strategies are (1) currently being implemented, and (2) are planned to be implemented. For each strategy, a 2x3 matrix was produced; these indicated relationships between usage (current and planned) with VECTERS mean ratings (value, expectancy, cost). The supposition was that those scoring higher on VECTERS' value and expectancy items, and lower on cost items, would be more likely to currently be integrating a classroom strategy and would be more likely to plan on using the strategy in the future (either initiating or continuing to use). Construct validity was further examined by applying orthogonal (varimax) rotation factor analysis. Analysis was applied to VECTERS' three sub-tests. This supported item reduction analysis and a resulting final version of VECTERS (Appendix).

Relationship of Variables
The internal consistency reliability for VECTERS items was high (Cronbach's ! = .90). Cronbach's alpha values for the subsets of formative feedback, real-world appli-cations, and student-to-student discussion (0.83, 0.76, 0.82, respectively) were also at acceptable reliability levels.
Mean responses regarding the extent to which respondents used a particular strategy, now and in the future, were calculated. Faculty members indicated on a Likert scale if they were using or planned to use each of the three strategies from "not at all" (value = 1) to "entirely" (value = 4). Results indicated real-world applications were used most often by engineering faculty, with the other two strategies approximately equivalent in use ( Table 2). Table 3 provides correlations between mean scores for the constructs of value, expectancy and cost, per each classroom strategy, with the reported level of implementation of the strategy.  These relationships met predictions. Among all three classroom strategies, instructors' reported use of the strategy was positively correlated to their dispositions regarding the value of the strategy and their expectation of success. The first of these positive relationships implies that instructors who believe a strategy has value for their students and for themselves uses that strategy more often. Similarly, instructors who expect successful implementation of a strategy are more inclined to use that strategy. The negative relationships found between cost and reported usage met expectations of expectancy theory. These negative relationships imply that higher use correlates with diminished view of the cost of integration.
Because the constructs of value and expectancy were comprised of items that could be further categorized, we drilled down deeper and conducted an exploratory correlation analysis. Again bivariate analyses were examined between the sub-classifications with reported current implementation and with planned implementation. The subcate-gories and example items are provided in Table 4. Because there were only five cost items, and these were cohesive, no cost subcategories were isolated. The greatest predictor for current use (r = 0.6) and planned use (r = 0.61) of formative feedback was the subcategory of seeing value for students. Similarly, current use (r = .48) and planned use (r = .46) of real-world applications was best predicted by seeing it as valuable for students. This finding was consistent for facilitating studentto-student discussions which was also best predicted by seeing value for students (use now, r = .61; planned use, r = .62). Nearly all other subcategories were significantly correlated (p < .05) with both current implementation and planned implementation, with r-values ranging from approximately 0.2 to 0.6. The exception was there was no significant relationship between current use (r = .01) or planned use (r = .05) with the belief that using student-to-student discussion had value for the instructor.

Exploratory factor analysis
Factor analysis was applied to VECTERS' three sub-tests, per the classroom strategies of formative feedback, real-world applications, and student-to-student discussion (26-items per sub-test). Factor analysis was used because of interest in determining how variables grouped on the basis of strength and correlation, and to assess how well those groupings aligned to the designed constructs of value, expectancy, and cost. Based on traditional guidelines of retaining all factors with eigenvalues greater than 1 [20], initial analysis of eigenvalues and the scree plots suggested retaining five factors for formative feedback, accounting for 59.5% of the variance; eight factors for real-world applications, accounting for 65.5% of the variance; and five factors for student-to-student discussion, accounting for 61.6% of the variance. To streamline comparisons, only six factors were retained for real-world applications because the seventh and eighth factors did not contain at least two items loading at a level of 0.6 or above. The total variance accounted for by the six retained factors for real-world applications was 55.0%.
The strongest VECTERS items for each of the three sub-tests are provided in Tables 5, 6, and 7. Four items are shown for each factor unless the factor loading was less than 0.6 -in those cases fewer than four items are presented. To streamline presentation, the factors are presented across the three tests as A1, A2, A3 . . .B1, B2, . . C1 . . . etc. Items in the third columns are in order of descending relative strength.   Expectancy -disagrees that physical set-up of my classroom is an obstacle Expectancy -disagrees that there are too many students to implement successfully A crosswalk examination of the factor analyses presented in Tables 5, 6, and 7, led to connections and themes becoming evident. A dimension termed "functionality" is represented in factors A1, B1, and C2. These three factors share items that point toward a belief that the strategy simply will work with students and a dismissal of the notion that the strategy somehow interferes with learning.
Also cutting across all three strategies is a dimension referred to as "expense." The expense dimension (represented by factors A3, B2, and C3) is present among all three of the strategies and the exact same four cost items rank highest across all three strategies. The only cost item that did not load heavily in the expense dimension was the statement that "there is too little time available during class to implement this strategy effectively." The implication here is that the commodity of class time is viewed differently than the cost of out-of-class expenditures such as teaching assistants (TAs), materials, and preparation time. This may also speak to high variability in the amount of minutes allotted per week across the different colleges and courses.
A third important dimension is termed "student benefit." This is represented by factors A2, B3, and C1. This dimension corresponds to the sentiment that use of a strategy aids student comprehension and motivates students. A dimension of "personal ability" also cut across all three strategies and is represented by A4, B4, and C4. Finally, other dimensions cutting across at least two strategies but accounting for comparatively less of the variance than those listed above are the following. Factors A5 and B5 represent a dimension of "job expectation." Additionally, factors B6 and C5 represent a dimension of "physical environment."

Conclusion
In response to the primary research questions, this study indicated strong relationships exist between the use of a student-centered strategy and an engineering instructor's disposition about that strategy. This supports the theoretical framework of expectancy theory. More importantly, the findings underscore how attitudes and perceptions can act as gatekeepers. An implication here is that effective professional development must address not just the logistics and mechanics of integrating classroom lessons, but must tackle the difficulty of affecting attitude.
This study also yields an assessment tool with broad application potential. Item analysis pointed toward cohesiveness and strong interconnections. The examination of construct validity played out well for value, expectancy, and cost. The costdecreases-with-usage relationship indicates that using a strategy is negatively related to perception of high cost. This finding aligns to research indicating that when people perceive a reform to have first-order barriers (i.e., external cost) they are less likely to implement; however users of a reform tend to minimize first-order barriers and focus on the more important second-order barriers such as views about effectiveness and potential for success.
VECTERS is seen has having two useful future roles. First, as a diagnostic tool for engineering educators. However, this need not be limited to engineering faculty since the three classroom strategies are supported across multiple disciplines. Using VECTERS as one means of evaluating the dispositions of faculties over time can help to pinpoint mindsets and experiences that are hampering implementation of strategies. The revised VECTERS (Appendix) can be used in whole. Researchers adapting the instrument for their needs may choose to reduce and/or interchange the topics and then evaluate if the new instrument persists with sufficient reliability and validity strength. A second useful role for VECTERS may be as a tool that helps to facilitate discussion about teaching. In our professional development, items were selected from VECTERS as seeds of conversation among faculty members. Having meaningful discourse about the specifics of value, expectation, and cost, enriches dialogue. This type of deeper discussion aids instructors in developing introspection regarding their own beliefs and perceived obstacles of implementation. Real-world Applications (demonstrate relevance, integrate realworld problems, underscore connections to industry and design) Instructor initiated student-tostudent discussions during class (focused on furthering understanding)