Effective Assessment of Computer Science Capstone Projects and Student Outcomes

A capstone project is a culminating experience that entails creativity, critical thinking, and advanced problem-solving skills. To that end, capstone projects enable students to prove their abilities, demonstrate their attained skills, and carry out a significant project in their field. In Computer Science Bachelor programs, there is a strong mapping between learning outcomes of capstone projects and all student learning outcomes. This paper presents an assessment framework for capstone courses that allows for sound evaluations of the performance of students and project qualities; besides assessing the student outcomes of the program. The developed framework comprises criteria, indicators, extensive analytic rubrics, and an aggregate statistical formulation. The presented course and framework are supported by the results, analysis, and evaluation of a pilot study for a single institution to explore the effectiveness of


Introduction
Capstone projects are a rich resource for assessing the level of attainment of student outcomes and in most of the knowledge acquired by students during their study. This assessment is done towards the end of students' studies giving a true indication of the level of attainment of learning outcomes. However, capstone project assessment is very challenging. It is different from assessment in a regular course. Capstone projects do not feature traditional modes of assessment that include lab assignments, home assignments, period quizzes and term exams. In contrast also, students usually are undertaking learning tasks that are more research oriented rather than application oriented. As a result, evaluators would be employing different evaluation strategies inspired by the wide spectrum reflecting the versatility in their expertise. Another challenge is that a capstone project covers all student outcomes.
It is very important to develop a capstone project assessment tool that will address the above challenges. One should base this tool on performance criteria that adhere to the capstone project, particularly its learning outcomes. It should also be flexible to be used for assessing projects of different nature. Moreover, it should be easy enough to be used by evaluators having different expertise. Indeed, the tool should be clear so that students would be able to prepare themselves accordingly.
In this article, we develop a framework to assess the learning outcomes of capstone projects (CPs), and hence the mapping that binds them to Student Outcomes (SOs) at the program level. Our focal point of interest is in Computer Science and related Computing programs [1]. Our tool follows the ABET Accreditation criteria where we adopt a large part of its terminology. In line with the works of [2] and [3], our tool consists of a suite of criteria and their indicators, supported by thorough rubrics, and concluding with a summative statistical aggregation.
Our manuscript is organized so that in Section 2 we present related work. In Section 3, we develop the measurement scheme and address aspects of assessment criteria as well as indicators. Section 4 presents a discussion on the measurement rubrics, together along the statistical formulation for both evaluating student performance, as well as the attainment of SOs. In Section 5, we delve into the pilot study and present its analysis and evaluation. Section 5 presents the conclusion and offers insight for future work.

Related Work
Several investigations are reported in the literature to present effective Capstone Project (CP) course setups, structures, and assessment frameworks in computer science and engineering programs [1][2][3][4][5][6][7][8][9][10][11][12]. In [1], the author describes a course structure that includes a carefully designed prerequisite course on project management and scheduled milestones. Furthermore, assessment is done using a holistic rubric that enables the measurement of a set of Course Learning Outcomes (CLOs) that are derived from ABET Student Outcomes (SOs). Yousafzai et al. [2] and Damaj et al. [3] present a unified framework which allows for sound evaluations of student performance and CP qualities in addition to assessing SOs. Along the lines of those works, the authors develop a framework consisting of a suite of criteria and their indicators, supported by thorough analytic rubrics, and concluding with a summative statistical formulation. The authors conduct a pilot study examining several capstone projects. The results reveal that after raters' calibration, only a minor difference in average scores exist. In fact, the authors examine the weighted average assessment scores pre and post calibration and report it to be at 3.2%.
Olarte et al. [4] present a study that compares student and staff perceptions of key aspects of CPs, such as, characteristics of projects, competencies of students, in-volvement of advisor, and perceived learning of students. The study employs a holistic Likert scale of 1-4 to aid the evaluation. Three questionnaires were designed for the student, the advisor, and the committee. The questionnaires are divided into four blocks, one for each of the key elements. The blocks examine project characteristics, student competency, advisor involvement, and the level of learning perceived by students. primary finding of this study is that the students' expectations differ greatly from those of staff.
Furthermore, a supervision typology in Computer Science Engineering CPs is presented in [5]. The study develops and validates an instrument, and then utilizes it to determine different styles of supervision. A questionnaire is developed to survey capstone project advisors at a university during the past two years. A total of 109 surveys are successfully collected. A combination of multivariate statistical methods, such as factorial analysis, is employed. This study distinguished seven main supervision factors: technology, arrangements, keep alive, execution, meetings, management, and reports.
Assessment rubrics were presented for software-related undergraduate capstone projects in [6]. In addition, it was recommended that capstone projects should always undergo a continuous improvement process. A survey was carried out at different Pakistani universities. The survey results highlighted several challenges such as poor process visibility, difference in support and information documents, limited guidelines for project assessment, lack of adequate requirements on software process and documentation, and limited incentives for supervisors. The proposed rubrics specify the key assessment criteria to be assessed using quality levels.
Instead of having a capstone project course, a series of courses include capstone projects is presented by Bihari et al. [7]. Among those is a course in Software Engineering in which control was inverted so that the industrial sponsor has more control and management duties over the project than the faculty member supervising the students. The course was scaled up successfully through developing unique assessment and evaluation tools to monitor, measure and fairly assess a wide spectrum of projects. Students are evaluated based on several presentations done at various points in the quarter, in-class weekly reports, industrial sponsor feedback, project workbook, poster and an individual report. The rubrics are designed to be immune to variability in projects, variability in industry sponsors and variability in technology. Each deliverable is evaluated on a combination of the choice of the method as well as the execution of the method.
Moreover, the meetings-flow (MF) method is evaluated in, in [8], terms of its effect on teams undergoing their capstone projects. Previous studies have shown that MF is beneficial for monitoring student work and product quality. In this study, it was empirically proven that MF enhances a team's communication and coordination and creates a balance between the contributions of all team members. However, MF was observed to have small influence on team cohesion.
In addition, an instrument is developed, by Pérez et al [9], to determine the different styles of supervision for advisors of capstone projects. Six supervision styles were identified based on seven factors of supervision. The identified supervision styles and factors can be used to guide advisors on how to supervise students as well as where to assess competencies of students and how to give meaningful feedback to the students.

3
The Measurement Scheme

Background
According to [2,3], the CP's learning outcomes can be associated with or replaced completely by the SOs at the program level. According to ABET, student learning outcomes capture the knowledge that students are to attain or to perform by graduation time. There are six student learning outcomes associated with the Computer Science program and as set out by ABET [10]. The CP is the most critical juncture in a student's undergraduate journey where they get exposed to a significantly practical experience. The CP is the very first encounter in their educational timeline that propels students towards an area of their own interest. With contemporary worldwide societal challenges, it also becomes imperative to train students on using their knowledge for the service of public good. As such, the CP provides an optimal venue where students can aim to develop technologies and analytics for addressing societal challenges.
As discussed earlier, our assessment approach is dedicated for Computer Science programs and related Computing fields. In our approach we consider a CP course taking place in one semester only. The CP course aims to enhance students' skills with practical experience giving them the opportunity to integrate knowledge accumulated in different courses. The pre-requisites for the CP are three junior and senior level courses following a three-year undergraduate CS program under the liberal arts model:  An introductory software engineering course introducing the fundamentals  An introductory Operating Systems Course introducing the fundamentals of OS function, design, and implementation  An introductory database systems course with an overview of the nature and purposes of database systems and an introduction to data modeling.
At the end of this course, students must deliver a product with a major software component. The product must demonstrate aspects in the design, analysis, implementation, testing, and evaluation. In our approach, we distribute those four aspects over sixteen weeks with the following milestones:  Definition of the problem and its objectives  Management of the project and teamwork  Literature overview  Alternatives for design and software and or project design methodology  Specifications: software/project requirements  Modeling and analysis  Early-release prototyping  Mid-semester software/project documentation  Mid-semester checkpoint presentation  Implementation  Testing and verification  Critical appraise: the analytical process for the systematic evaluation and validation of the software product  Documentation  Final product demonstration: Final software/project delivery and oral presentation The above milestones are carefully scheduled by the project team and overseen by the supervisor over the course of the semester. Close follow-ups, by the team and monitoring by the supervisor, are necessary for the thorough completion of requirements. Indeed, enabling the evaluation of the project aspects, including the proposed milestones, is the aim of the target assessment criteria and the overall framework.

3.2
The assessment criteria Although CPs are of joint efforts of a team of students, any developed assessment framework must allow for sound evaluations of contributions per student. Assessment of student contribution carefully considers the contents and their quality, timeliness of achievements, and professionalism. A variety of assessment tools can be used for CPs and student evaluations. Moreover, assessment tools are deployed to measure indicators within specific criteria. For an increased reliability of measurements, tools are carefully selected to enable the following multiple sources of measurements:  Project proposal  Supervisor consultations  Mid-semester reports on project progress  An oral final exam per student  A report delivered upon project completion  Project presentation examination by a committee The project examination committee is mainly formed of professor from within the department that the students belong to. In some instances, external examiners are invited from outsides the department and the university; however, all examiners are university professors in areas related to the project topic. In this paper, we propose the following assessment CPs criteria; some names are inspired by those presented in [2]: a) Content 60%) b) Impact of the CP on the social good (5%) c) Integrity and ethical and legal implications (5%) d) Project management and teamwork skills (10%) e) Written communication (10%) f) Presentation and oral communication (10%) The developed six criteria, A through F, are carefully selected to cover all intended CP aspects within its indicators and map to all SOs at the program level. The most significant part of CP assessment is under Criterion A that measures the level of project contents achievement. The contents comprise reviewing the literature, the design and implementation techniques, use of technology, analysis and evaluation, and the identification of future work. Furthermore, a weight of 60% of the CP evaluation is allocated for Criterion A (See Table 1). Criterion B assesses the extent to which the proposed CP addresses a challenging problem for the social good. Criterion C ensures that students are clearly aware of the ethical and legal implications surrounding human subject data. Moreover, Criteria C through F captures a bouquet of CP requirements that comprises the understanding of legal implications and adherence to professional ethics, project management aspects, and documentation and presentation skills. Table 1 presents the list of criteria, indicators, and the allocated weights.

Bridging capstone projects and outcomes at the program level
The framework's set of carefully developed performance indicators enables a variety of measurements of CP outcomes at the course levels. To benefit from CP measurements in the program review process, the framework is built upon the adoption of ABET 2019 complete set of SOs as both the program and the course outcomes. The adoption of the complete set of SOs as CLOs guarantees the coverage of required outcomes within a CP course. With no doubt, such a unified arrangement of CLOs and SOs facilitates closing the continuous improvement of ABET's Criterion 4. In Table 2, we present the mapping among assessment criteria and ABET SOs. Indeed, the relationships between CP indicators and SOs are many-to-one, where several indicators' scores are aggregated to measure the attainment of a single SO. The following represent our CP CLOs that are the same as the newly developed ABET SOs for Computer Science Programs:  "Analyze a complex computing problem and to apply principles of computing and other relevant disciplines to identify creative and original solutions.  Design, implement, and evaluate a computing-based solution to meet a given set of computing requirements in the context of the program's discipline.  Communicate effectively in a variety of professional contexts: example presentations and software documentation  Recognize professional responsibilities and make informed judgments in computing practice based on legal and ethical principles.  Function effectively as a member or leader of a team engaged in activities appropriate to the program's discipline.  Apply computer science theory and software development fundamentals to produce computing-based solutions." Adopting ABET SOs as the intended CLOs of the capstone course enables multiple mutual benefits. First, the adoption of SOs as CLOs unifies their assessment without the need for mapping them to each other and accordingly the need to develop an additional statistical aggregation. Second, such an adoption guarantees the literal coverage of all SOs within the course intended outcomes. Accordingly, the framework maintains a 3-level hierarchy of evaluation metrics, namely, criteria, indicators, and their rubrics.

Hierarchy of evaluation metrics: criteria, indicators, and rubrics
The suggested framework is a three-level hierarchy. The top-most level is the set of criteria that covers all intended CP aspects. A rich set of indicators stems from the criteria to specify the intended measurements. The bottom-level of the framework includes extensive set of analytic rubrics for each indicator. The rubric descriptors specify the quality scale of achievements and the intended requirements. The developed rubrics and the statistical formulations are presented in Section4.

Measurement Rubrics
To further develop the proposed framework, analytic scoring rubrics are carefully created based on the specific requirements of the intended CP context. We base our analysis around twenty-Four indicators that map onto the set of revised ABET SOs (See Table 2). Furthermore, the adopted scale of rubrics consists of four attainment levels: a beginning level (B), a developing level (D), a competent level (C), and an accomplished level (A).
To verify the suitability of the created rubrics, we have consulted with four Professor of Computer Science, besides, comparing with rubrics from [2,3,14,15]. The aim of the developed rubrics includes adopting solid descriptors and a variety of performance levels. These four levels in turn correspond to a range of percentages given by [40-69, 70-79, 80-89, 90-100] respectively. The selected ranges of percentages are for a scale that considers 70% as the starting point of the D level. With this level of granularity, we can assess the deliverables of the CP at any of the criterion or indicator levels or the combination of the two. We adopt the following weighted average to aggregate all indicators: . where is the combined score, is the score percentage of the th indicator and is the weight of the th performance indicator such that , and , where is the number of performance indicators i.e. in our case, . The weights are described in Table 1.
Capstone project assessment criteria cover aspects related to global and regional social impacts; understanding of integrity and ethical issues, legal implications, and regulations; management; and effective communication skills. For some part of the created indicators, we are inspired by the rubrics developed in [2,14,15]. Although our framework is primarily created for assessing computer science CPs, minor modifications enable its use in similar disciplines.
In Tables 3 through 8 we delve into many details surrounding Criteria A through F. Each row in these tables represents a certain relevant mapping to an ABET outcome. The various columns in each table describe the attainment per scale point (Beginning, Developing, Competent, or Accomplished). We present an overview of the content of these tables as follows. Criterion A in Table 3 carries with it a significant percentage of the overall score, as it addresses several focal points related to the overall quality of the project. The stated rubrics cover the various stages of the project from beginning to end. The indicators dwell on evidence of a thorough literature survey, of a robust understanding of computing principles and techniques, and of a sound methodology that is interpretable and yet creative and original. The indicators investigate whether there has been enough exploration of alternative research solutions, and to what extent the team has conducted benchmarks against competitor solutions.
Criterion A is also concerned with the extent to which appropriate hardware and/or software tools have been exploited, and finally, with the level by which results are interpretable and future work and improvements are identified. Beginner levels are those mostly lacking in all these indicators whereas competent levels meet those criteria and beyond.
Criterion B in Table 4 probes into the social impact that the proposed CP aims to address. The indicators investigate the extent to which the project addresses issues of social impact, examples of which can include poverty, education, or crisis management, to name a few, and the extent to which the team exhibits awareness of the high risks and/or low resourced settings surrounding our society. We then carry on having more indicators of this criterion investigate the level by which the project can be adapted for use by people without the relevant technical background, and the extent to which the team evaluates computing solutions that consider global and regional societal factors. Competent students are expected to demonstrate exemplary awareness of societal challenges and to offer solutions of high impact that are of utility and can be adopted by lay people from all walks of life.
Criterion C in Table 5 assesses the understanding and application of integrity and ethical and legal implications, and regulations. To that end, competent students exhibit consideration and compliance with professionalism and integrity, especially with team members and advisor. Moreover, a team member is to abide by the regulations that govern the project, its data, and show proper adherence to handling guidelines.
Criterion D in Table 6 is aimed at assessing the management skills within the team work as well as the level to which the student has individually contributed to the project and taken responsibility for sub-tasks. Also, the indicators tackle the time management skills required for achieving major milestones in a timely fashion. The indicators stipulated under Criterion D require the assessment of the project supervisor exclusively, as external examiners have no way to monitor those aspects of the project. Competency at this criterion requires that a student demonstrate active participation in the project as well as a strong initiative leading up to monumental ideas in a timely fashion.
Written communication is addressed in Criterion E in Table 7. The rubrics begin by addressing the organization and the logical order and coherence of ideas. A competent student exhibits a solid logical rationale and a smooth transitioning among ideas as well as a highly relevant body of information. The rubrics then examine the writing style involving the choice of words as well as the grammatical proficiency and the readability of the written document. A competent student has a compelling writing style that captivates the reader throughout. The rubrics finish off with examining the use of references and the level to which the writer provides citations in the text, is accurate in referring to the citations, and chooses relevant and impactful literature references. Table 8 presents the last criterion assessing the oral and presentation communication skills, as evident from the student's own slides as well as their delivery of the presentation. Particularly, we pay attention to the mechanics manifested in the slides as well as the extent to which they are effectively written, the sequencing and pace of topics in the presentation. We also examine the actual delivery including voice and tone, as well as body language. The engagement with the audience and the level by which the response to questions is appropriate are also address. A student is competent at this criterion if they present extremely creative and well written slides in an engaging manner and show confident presence on stage and excellent engagement with the audience and can navigate through and adapt the presentation considering real time response from the audience.
A weighted aggregation of indicator scores produces the overall percentage grade per student. In addition, simple averaging using the indicator mapping presented in Table 2 enables the calculation of attainment scores per indicator, criteria, and student outcome. In Section 4, the benefits, challenges, results of deployment, and validation of the proposed framework are investigated. Renders that project completely grounded in computational principles and technologies; applies them to problem correctly and clearly establishes their relevance.
Interpretability, creativity, and originality of the adopted methodology and the developed solution.
[SO2, SO6] The interpretability of the methodology is poor. The creativity of solutions is poor. The novelty of the proposed work is completely lacking and there is no impact observed to the work proposed.
The interpretability of the methodology is satisfactory. The creativity of solutions is satisfactory. The proposed work has some novelty and there is some mild impact observed to the work proposed.
The interpretability of the methodology is good. The creativity of solutions is good. The proposed work has evident novelty and there is adequate impact observed to the work proposed.
The interpretability of the methodology is excellent. The creativity of solutions is excellent. The proposed work has substantial novelty and there is an extremely promising impact observed to the work proposed that can propel it into multiple directions. Testing of the design is somewhat fair; results are inconclusive but not usable for further investigation. Attempts to interpret results are made but not to a satisfactory level.
Testing is adequate; analysis and results are acceptable, complete, and sufficiently interpretable to an expert.
Testing is thorough; analysis and results are robust, usable and highly interpretable even to a nonexpert.
Identification of further improvements and future work. [SO2] No direction for further improvement is provided.
One or two ideas for future expansion are listed but may not be practical.
Several ideas, of which one or two are practical and adequate, for further improvements are explained.
Several novel directions for important expansions of the current ideas are thoroughly explained. Table 4. Criterion B (Impact of the CP on the Social Good); rubric is partly inspired by the tool presented in [2].

[Mapping to ABET outcomes] Beginning Developing Competent Accomplished
Addressing a problem that stems from a social need, and thus has a social impact.

[SO6]
No addressing of needs on issues of social impact, examples of which can include poverty, education, or crisis management, to name a few. No awareness of the high risks and/or low resourced settings surrounding our society.

Contribution to the team project/work [SO5]
Individual contributions to the team are not relevant or useful, and do not address the team's needs; Only when prompted, embarks on contributing information to the team. Tries to provide some ideas but suggestions are not sufficiently developed to meet the team's needs.
Can provide some basic and useful information to assist in the project and occasionally makes some useful suggestions to the team that meet its needs.
Is able to provide extremely relevant information to assist in the project. Systematically offers well developed and clearly expressed ideas that fall at the heart of what the project needs.

Taking responsibility [SO5]
Takes no responsibility whatsoever and shows no initb ?
'iative at all relyin /g on the other team members to do the work. By and large misses meetings and when present demonstrates marginal participation.
Can perform assigned tasks but regularly needs reminders and prompts. Delegates the challenging parts of the project to others. Does not have a constructive presence during meetings.
Can perform all assigned tasks. Attends all meetings and is generally engaged in the discussions that take place then.
Is able to perform all assigned tasks highly effectively. Takes initiative in setting up meetings and is the lead participant in the discussions that take place then. Table 7. Criterion E (Written Communication); rubric is partly inspired by the tool presented in [2].
[Mapping to ABET outcomes]

Organization and logic [SO3]
The information in the text has no logical order, lacking in many important details, and is difficult to understand.
The text exhibits a weak organization. The presentation of ideas lacks coherence and shows no smooth transition between them.
The text exhibits a reasonable organization. The information in the text has somewhat a logical backing and an attempt to provide a project rationale is made.
The written report exhibits strong clarity and a solid logical rationale. Transitioning among ideas is smooth. The information presented is at large very relevant and thorough, all resulting in a highly informative piece of text.
Writing style (word choice, grammar and sentence structure) Choices of words and expressions often misleading. Text suffers from numerous errors in grammar that compromise on the clarity of the document.
Choice of words and level of grammatical proficiency is generally adequate. Yet, the document still is difficult to read.
The writing style and the general flow of the text are satisfactory.
The writing style is compelling and is able to captivate the reader till the end.

Use of References [SO3]
Most references included are inaccurate and are not relevant. Almost inexistent attempt to provide citations in the text.
Most references provided are clearly indicated but have little impact in the literature. A conservative attempt to provide citations in the text is made.
Prior work is properly cited in most places where needed (e.g. when referring to theories, assumptions, and findings). Minor exceptions exist. References are accurate in referring to author names, journals or proceedings, volume numbers, page numbers, and year of publication. References have a modest impact in the literature.
Prior work is properly cited in most places where needed (e.g. when referring to theories, assumptions, and findings), with no exceptions whatsoever. References are accurate in referring to author names, journals or proceedings, volume numbers, page numbers, and year of publication. References have a great impact in the literature. Generally unprepared to answer questions. Misunderstands most of the questions and fails to provide appropriate response.
Partially prepared to answer questions. Understands most of the questions but demonstrates difficulty in providing correct or well-informed responses.
Demonstrates a clear understanding of the questions and is well prepared to answer them. Provides mostly correct and well-informed responses.
Fully prepared for questions to the extent that speaker can anticipate questions and respond with ample information beyond what's needed.
Demonstrates a deep understanding of the project's intricacies and controversial topics.

Analysis and Evaluation
This section presents the result and evaluation of the proposed framework. In addition, the benefits of the proposed assessment framework and its challenges are discussed. The framework uses an assessment structure, formulation and scoring like the one used in [16]. In addition, several CPs covering an extensive selection of computer science problems are used to evaluate the assessment tool. The tool is also evaluated with several evaluators who have different experience and level of expertise in different areas.

Case-study setup
The presented study includes seven projects with a total of 25 students and a typical project team size of 3-4 students. Each project has a single supervisor and is examined by a committee of four professors. The examination committee evaluates the project report, presentation, and the developed prototype during a demonstration. The supervisor examines all deliverables. The target passing score per student is 70% for the overall course grade; the same percentage is adopted as the target score of SOs.
A pilot study for a single program is applied to evaluate the proposed framework. The evaluated CP's are from an institution of higher education which adopts the American model of higher education. Data collection started with refining the documents and forms already in use at the institution. This led to balanced artifacts. The analytic version of the tool was then used to collect data. Training sessions were then conducted for the evaluators to ensure evaluators understood the rubrics. Then data collection was conducted after calibration.

Analysis
As shown in Tables 9 and 10, most of the indicators and SOs were not different before and after calibration. Even with the indicators which had different scores before and after calibration the 4-point rubric scale mapping did not change. This indicates that the Rubric is clear enough to be used by evaluators before and after calibration without the fear of changing the 4-point score of any of the indicators. Based on the rater calibration, several improvements are identified. The improvements include the following:  Add the identifier "ACM" for core areas in indictor A1 to specify the target core areas. ACM core areas are detailed in [17].  Add "during the design phase" to indicator A2 to identify the specific stage of development.  Remove "impact" from indicator A3 to avoid overlapping measurements with indicator B1.  Specify the code of conduct in C2 as the "Code of conduct of the Institution"  Modify "good" under F1 under Competent by "well-designed" to better match the intended meaning by the rubric designers.
The measurements made for the proposed indicators identified additional opportunities for improvement on the SO level (See Table 10), namely SO4 and SO6. The identified improvement is on the student abilities to recognize professional responsibilities and make informed judgments in computing practice based on legal and ethical principles. In addition, the needed improvement is related to the student effective application of computer science theory and software development fundamentals to produce computing-based solutions. At the program level, the attainment scores are usually combined with triangulated measurements from other courses to reach a final attainment score and improvement decisions.
As presented in Section 3, the proposed framework enables evaluating student performance as the weighted sum of indicator scores. The results of the evaluated 25 students ranged between 70% and 90%. Although all students met the projected passing grade of the course with their overall score, the tool allowed for the identification of improvements to their intended abilities at the indicator, criteria, and SO levels.
Upon incorporating the suggested modifications to the rubric, the proposed framework is ready for deployment within Computer Science Programs that are aligned with ABET requirements and ACM recommendations. However, tuning and customizing the framework is straightforward. Customizations can be applied to the criteria, indicators, rubrics, and the choice of the aggregating statistical formulation.

Evaluation
Several benefits exist for the framework proposed in this paper. The framework limits bias and uncertainty of evaluator's thereby promoting quality in assessment. This is due to the clear measurement structure resulted from the conceptual basis of the framework. At the program level, a main source of measurement are the integrated key CP indicators. In addition, conclusions at different levels of abstraction are possible using the framework. Measurements are made at the indicators, criterion as well as ABET SO levels of abstraction. The measurement structure and statistics in the framework are not limited to computer science and can be applied for any other discipline. Indicators and rubrics are comprehensive and rich yet simple enough to be understood by faculty members as well as students. A smooth transition in descriptors is used in the rubrics which makes it easier for evaluators to pick the appropriate descriptor.
However, the framework poses several challenges to its implementation. A prerequisite for implementing the framework is that a culture of assessment be present. The evaluators should be first trained on how to use the rubrics. They should also be committed to reviewing thoroughly all artifacts being measured such as presentations, reports and essays. In addition, the time constraints placed on the evaluation process are also considered as a great challenge. To that end, the allocation of four examiners per project can be reconsidered and replaced by a smaller evaluation committee.

Conclusion
Computer Science programs rely heavily on senior CPs to demonstrate student abilities accumulated throughout the program. CPs are rich in requirements and deliverables; this makes them of unique importance in evaluating student performance and assessing their attainment of SOs. In this investigation, we present a framework for systematically, accurately, effectively, and jointly evaluating student performance, assessing learning outcomes, and accordingly assessing SOs. The hierarchy of the developed framework is of three levels that comprise criteria, indicators, and an extensive set of analytic rubrics. A single-institution pilot study is executed to calibrate the proposed rubrics. The study includes several CPs from wide spectrum of computer science topics. The tool tuning attained a small variance in scores after the calibration of raters. A difference of 2% is found between the scores-before and after calibration. The proposed framework is easy to deploy and was found to effectively eliminate subjectivity in assessment and evaluation. Future work includes carrying out a study that involves multiple programs.