Initial Validation of a Technical Writing Rubric for Engineering Design

Engineering design serves as the capstone experience of most undergraduate engineering programs. One of the key elements of the engineering design process is the compilation of results obtained into a technical report that can be shared and distributed to interested stakeholders including industry, faculty members and other relevant parties. In an effort to expand the tools available for assessment of engineering design technical reports, this study performed an initial validation of a previously developed Technical Writing rubric. The rubric was evaluated for its reliability to measure the intended construct, inter-rater reliability and external validity in comparison to an existing generalized written communication rubric. It was found that the rubric was reliable with Cronbach’s alpha for all dimensions between 0.817 and 0.976. The interrater reliability for the overall instrument was also found to be excellent at 0.85. Finally, it was observed that there were no statistically significant differences observed between the measurements obtained on the Technical Writing rubric in comparison to the more generalized Written Communication Value rubric. This demonstrates that although specific to engineering design environments the Technical Writing rubric was able to measure key constructs associated with written communication practice. This rubric can now serve as one additional tool for assessment of communication skills within engineering capstone design experiences. Keywords—engineering design, written communication, assessment, entrepreneurship


Introduction
Communication is a professional skill that is essential to engineering practice [1]. As we approach 2020, the year envisioned in the National Academies Report on Visions of Engineering in the New Century, we can recall their inclusion of strong communication skills as an important skillset to allow for engineers to be able to address the needs of multiple stakeholders, work on interdisciplinary teams and apply various means of communication [2]. More recently, industry and academics have remained committed towards the need to develop communication skills amongst engineering students with an emphasis placed on the development of these skills prior to students transitioning into the workforce [3].
Communication has been taught to engineering students through a variety of approaches such as full courses focused on communication [4], integration of communication skills throughout the curriculum [5,6] or the inclusion of communication based assignments within a given course [7,8,9]. Despite the variety of approaches taken, research has shown that three key strategies tend to allow for the best development of communication skills amongst engineering students. These strategies include providing a situated learning experience for students as they develop their communication skills, incorporating communication intensive courses within the curriculum, and using communication assignments to assist with content learning [1].
Engineering research and design experiences are an ideal opportunity to reinforce the need for strong communication as it provides a situated learning experience for students. Although engineering design can focus on technical based elements such as work process, more recently work in this area has expanded to include social process and professional development skills that are necessary for student success in practice such as teamwork, communication, and project management [10].
Communication skills have been found to be most effectively taught when related to assignments that mirror professional context and expectations [1]. In this case, a design report that students need to provide to instructors at the end of a design experience would mirror what may be expected from them within industry. The key for ensuring that students can translate their communication skills into practice requires that the assignment be structured around the intended audience for the work and to allow this to shape the students' writing experience [11].
Prior work on the development of communication skills within engineering capstone design courses has shown that there often exists a dichotomy in the expectations for the final communication product between academic requirements and what would align with professional practice [12,13]. In a study in Ref. 12, Dannels observed that there were conflicts in three areas that related to the structure of design presentationsaudience, identities, and ways of structuring the design presentation. Often times, students were instructed to include sufficient technical information to demonstrate that they had a technically sound design, but at the same time were told to ensure that the information was not detailed to the point that it wouldn't be understood by a multidisciplinary audience. They were also told to appear professional when making their presentations and to demonstrate that they had prepared well in advance for their presentation, but were then asked questions on whether they had followed the guidelines provided to them within the accompanying course text. Finally, students struggled with trying to demonstrate through their presentation that they had touched on each of the design process steps as outlined in their course materials even though this wouldn't be necessary in a professional work presentation [12]. Similarly, in Ref. 13 it was observed that students felt there was a mismatch between the work that was expected of them and what would be required in professional practice. For instance, students were required to prepare a notebook and progress reports as part of their capstone design experience. However, the requirements of the assignment given to them didn't align with how the students felt they could best capture their work on the project and what they would be capturing if they were working on this project within a professional work environment [13].
This work demonstrates the need for providing students with clear guidance on how communication pieces they will prepare for courses align with working environment conditions. To improve translation of communication skills it will be necessary to identify any discrepancies and discuss them with students early and often throughout their coursework. Although structuring the assignment in the correct manner to facilitate translation of these skills to a professional context is important, it is also necessary to determine methods to assess students' communication skill performance. Technical writing rubrics were developed and have been used as assessment tools in engineering design courses [14]. The assessment links the processes of rhetorical complexity in writing and technical complexity in the research and design process as similar. Thus, students need to be able to articulate their research and design ideas effectively in writing, by considering writing as one of many "languages of design," a phrase used by Dym et al. [15]. Along with text common to all written communication, graphical representations and mathematical models are also integral to design and research, particularly in science and engineering. Development of communication skills and assessment of these in the context of design and research go hand in hand as others have shown [16,17].
The proposed study seeks to complete an initial validation of a Technical Writing rubric that has been developed for use in engineering design courses by examining its external validity when compared to an existing generalized validated technical writing rubric and its reliability at measuring the intended construct as well as its ability to provide consistent scoring amongst graders. The proposed validation strategy was developed as a result of reviewing work on how to validate assessment instruments in Refs. 18 and 19 and a study performed on initial validation of a project based rubric in Ref. 20 [18,19,20].

Participants
The participants in this study were students (n = 33) working on eight (8) multidisciplinary teams enrolled in a capstone engineering course at a medium-sized, mid-Atlantic research university. Of these 33 students, 28 were male and 5 female. All students identified as Caucasian, with three also identifying as Latino/Hispanic. The teams worked on eight biomedical research and design projects that were initiated in the first year by faculty and in the second and third years by needs discovered by student Scholars who participated in a Clinical Bioengineering Summer Immersion Scholars program as part of a larger NIH funded project, see Refs. 21 and 22 [21,22].

Study Design
Eight reports written by student teams during a three year period of a capstone engineering course were included in the study. The topics of these eight reports described various biomedical research and design projects. Of the eight reports, five reports were written by student teams in the first year of the program when faculty initiated biomedical research and design projects. The other three reports were written by teams where the needs were discovered in a Bioengineering Summer Immersion Scholars program [21]. Briefly, during immersion, Scholars shadowed hospital staff on rounds on medical and surgical floors. They participated in discussions with clinicians, technicians, and other hospital staff. Scholars observed clinical processes, identified problems and formulated needs statements. The needs statements formed the basis for projects to be solved during the academic year in the engineering course and written about in the reports.
Teams of students were led by different faculty instructors based on interest and expertise on the projects. The students were not given a common set of instructions or guidelines for their reports, since different faculty were advising them, and the projects were conducted during different years. Most, if not all, of the students had taken previous engineering design courses in the sequence at the university where they would have learned how to write research and design reports.

Technical Writing Rubric
The Technical Writing rubric was generated by faculty members at the mediumsized, mid-Atlantic research university where the study was conducted. The Technical Writing rubric has five core objectives that integrate communication, competencies with design thinking and experimental research strategies. Specific outcomes or indicators are listed, which allow a reviewer to determine student ability to do the following: 1) provide an appropriate context for the project or study, 2) understand and apply appropriate methodologies, 3) present and evaluate results, 4) gather and present information from sources, 5) produce properly formatted and edited text. The descriptive scales help benchmark expectations and are on a 6-point scale (5/6 = Skillful; 3/4 = Competent; 1/2 = Unsatisfactory) [14].

Comparison Rubric
The written VALUE rubric was selected as a comparison rubric as it is a broad writing rubric that has been validated and shown applicability in multiple disciplines. This rubric had its development sponsored by the Association of American Colleges and Universities and was designed by a group of faculty experts [23]. Both this rubric and the Technical Writing rubric measure similar key constructs including context, content, genre, sources and syntax, which is why they were chosen to serve as comparators.
The written VALUE rubric was designed by reviewing existing rubrics as well as related documents for written communication and determining which performance indicators best demonstrated progressively higher levels of achievement [23]. The written VALUE rubric measures five key constructs including: 1) context and purpose of writing, 2) content development, 3) genre and disciplinary conventions, 4) sources and evidence, and 5) syntax. Each construct is measured on a scale of 1 to 4, where 4 represents mastery of material [23].

Technical Writing Rubric Evaluation
Initially two capstone design reports from previous years that were not included in this study were used for training of the raters. The raters are both faculty members within a college of engineering, who have had previous experience with assessment of student engineering design reports. Each rater separately read and annotated these two sample reports, and then completed the VALUE and Technical Writing rubrics on their respective scales. After working individually, the raters met to compare notes, discuss and confirm that indicators were being interpreted in the same fashion.
After completion of training, the two raters independently scored the eight design reports that were produced by the Bioengineering Scholar teams. Upon completion of scoring, the two raters met and discussed any discrepancies leading to a final decision on the scores for each report across all dimensions of the two rubrics.
Reliability of each dimension of the Technical Writing rubric was performed using Cronbach's alpha. Inter-rater reliability for each of the rubrics was measured using the intraclass correlation coefficient (ICC). The ICC for the Value rubric varied between 0.101 and 0.902 with an overall ICC of 0.699 which represents fair to good reliability [24]. The lowest ICC (0.101) was on the 3rd dimension of the rubric representing genre. It is believed that poor reliability was found on this one dimension since the two raters had different impressions of the expectations and disciplinary conventions that the students had to meet when preparing the design reports. The ICC values on the remaining dimensions were all between 0.715 and 0.902 representing fair to excellent reliability [24].
To perform comparisons between the scores on the two rubrics for external validation purposes, the values for each dimension were first normalized to a score out of 1 as each rubric was measured on a different total number scale (maximum = 4 for the Value rubric and maximum = 6 for the Technical Writing rubric). When comparing the rubrics to one another, the content category of the VALUE rubric was grouped together with the Methodologies section of the Technical Writing rubric due to the similar properties that were being measured in each of these dimensions. For instance, in the content dimension of the VALUE rubric, the grader is looking for a report that uses content that is appropriate for the designated audience and clearly conveys the intended message to the audience. This is similar to the methodologies section of the Technical Writing rubric where the grader is looking for content that showcases the design that was performed, how the parameters were identified and the rationale for any justifications that were made when setting up the report. Similarly, the genre and disciplinary conventions section of the VALUE rubric was compared to the results section of the Technical Writing rubric as they were also aligned in terms of content that the grader would be assessing.
An overall rubric score was determined by averaging the score across all dimensions of the rubric. Due to the small sample size, a Wilcoxon Rank Signed Test was used when comparing the average value for each dimension and the overall rubric between both the VALUE and Technical Writing rubrics.

Research Results and Discussion
The following section describes the initial validation measures that were performed on the Technical Writing rubric to determine its suitability for use as an assessment instrument in engineering research and design courses. The validation process examined the total scores measured on the rubric including the mean values obtained for each dimension and ranges of these values, the reliability of the rubric and the external validity of the instrument in comparison to an existing broad natured validated writing rubric.

Total Scores
The range of total scores across all eight projects for the Technical Writing rubric between the two raters was 11.5 to 26.5 out of a possible 30 points. A plot of the scores for each project from each rater can be seen in Figure 1. It can be observed that overall most of the projects were graded similarly between the two raters with project 1 having the largest discrepancy (5.5 points) whereas projects 4, 6 and 8 had the closest similarities in ratings (1 to 1.5 points). It can also be noted from the results that the Technical Writing rubric was able to capture a range of outcomes with projects that didn't have as much technical proficiency in writing scoring lower in the range on the Technical Writing rubric (final agreed upon score of 12.5 out of a possible 30 points; project 6) whereas those that demonstrated strong writing capabilities and clear communication scored higher in the range on the Technical Writing rubric (final agreed upon score of 25 out of a possible 30 points; project 3). The average overall score obtained across projects was 20.56 out of a possible 30 points as shown in Table 1.
The average score for each dimension of the project along with its possible range of scores is shown in Table 1. Overall most categories had an average close to two thirds of the range of possible points (4 on a scale from 1 to 6) whereas some categories showed higher values than this such as syntax. These results could indicate that the student projects showed reasonable understanding of the different elements that are involved in producing a technical design or research report but had better mastery of syntax than other elements that would have only been introduced to them more recently within their education such as how to frame their communication for a specified technical audience.

Reliability
The internal consistency of the different category elements within the rubric was measured using Cronbach's alpha. The Cronbach alpha scores across the five dimensions of the rubric ranged from 0.817 for Context to 0.976 for Sources (see Table 1). Within the context dimension most of the items were worthy of retention except for the prompt relevant to delineating goals and scope which if deleted would have increased the Cronbach's alpha to 0.845. Across all other dimensions there were no   [25]. ICC values for the Technical Writing rubric ranged between 0.567 and 0.922 with an overall ICC of 0.85 representing excellent reliability [24]. Table 2 shows the normalized results for the mean values of the eight reports analyzed for each dimension of the two rubrics under consideration as well as the statistical analysis that was performed. The results obtained demonstrate that there are no statistically significant differences between the scores obtained on any of the dimensions analyzed with these two rubrics indicating that both rubrics are capable of providing accurate measurement of students' written communication skills. It can also be noted that on each dimension the difference in measurement between the two instruments is quite small, in the range of 0.0 -0.08, except for syntax where the difference was 0.15. As mentioned earlier in the Technical Writing Rubric Evaluation section this dimension was one of the areas where the two raters had differences in expectations which may have led to the discrepancy in measurements that is observed.

External Validity
While the VALUE rubric was designed with the intent to measure fundamental constructs associated with written communication skills that could then be translated to a specific disciplinary context [23], the Technical Writing rubric was designed with the explicit intent to be able to measure the written communication skills associated with engineering technical writing in response to a research or design problem [14].
Although the two rubrics were developed in complete isolation from one another, it can be observed that they fundamentally agree on the constructs necessary for clear written communication. Specifically, each rubric aligned directly on three of its five dimensions (context, sources and syntax). This alignment demonstrates that regardless of the field of writing there are basic principles that are necessary for any clear piece of written communication which include an understanding of the purpose of the writing and who the target audience for the final product will be, the evidence or sources used to support the arguments that are being included in the report and the clarity of the communication demonstrated through proper spelling and grammatical practices.

Limitations
In this initial validation study there were some limitations. For instance, only two raters were involved in the validation process for the Technical Writing rubric. In the future, it would be beneficial to have 4 or more raters to ensure that robust inter-rater reliability can be obtained. Another limitation is that this study only examined eight design reports as part of the validation process due to the limited number of reports available from multidisciplinary teams participating in the Bioengineering Scholars project. As additional projects are completed in the future, it will be possible to collect more samples of student work being generated from this process and determine whether any differences exist from the initial sample investigated. Other future work will be to compare capstone reports that were not part of the Bioengineering Scholars program to those that are, to determine if components of the program may or may not have an effect on the students' abilities to write about their projects. Finally, this work was confined to reports obtained from a single institution and hence may not represent the broad nature of engineering design reports that can be produced from senior level students. The study should be expanded in the future to include engineering design projects taken from other institutions and projects to confirm the ability of this Technical Writing rubric to serve the needs of the greater engineering design community.

Conclusions
This study performed an initial validation of a previously developed Technical Writing rubric for use in evaluation of engineering design and research reports. The Technical Writing rubric includes five different dimensions that focus on key elements associated with engineering communication practice in design and research such as context, methodology, results, sources and syntax. This study determined that the rubric was reliable based on Cronbach alpha scores above 0.80 for all dimensions under investigation, was able to achieve fair to good reliability across all dimensions and excellent reliability overall. It was also observed to be able to measure technical communication skills with no significant differences when compared against an existing general written communication rubric. Overall, this study confirms that the Technical Writing rubric is a valid assessment tool for the review and grading of engineering design and research reports although additional work remains to be done to confirm that the results observed are generalizable to a broader engineering design audience.