Program Evaluation Design Evaluation Approach

In reference to Dr. Jennifer Greene's "Approaches to Evaluation" (Greene, 2017), my PED best suits a process and outcome-based evaluation for learning and use approach.  A Utilization-Focused Evaluation design would best work as I would like to see the results of this study convince decision-makers of its merit in schools (Patton, 2014). The audience for this evaluation includes school administration, teachers, the school district, parents, and other interested users (including teachers from other schools). The purpose of this evaluation is to maximize the usefulness of Freshgrade's e-portfolios in an effort to contribute to the quality of our school's core value of deep student learning.  As a school, we want quality criteria with a strong program design that is sustainable and meaningful for students.  Success criteria for this evaluation includes having students take more ownership of their learning, having parental involvement in student learning, evidence of ongoing assessment and immediate feedback for students and parents, and documentation of reliable (and diverse) evidence of learning on student portfolios. 

​

The design of this evaluation is varied with both quantitative and qualitative (mixed) methods in order to truly reflect the program's merit.  While the number of posts for reflection, celebration, student selection of learning evidence, and parental and teacher input can be counted and quantified, other methods of data collection are more qualitative in nature.   These methods include interviews with participants, a survey, observation of student portfolios (finding evidence of student reflection and determining quality evidence of learning posted on e-portfolios), and data from academic research (secondary source).

Because we have been using Freshgrade for the past four years, we can evaluate the outcome of the program by asking the following key questions:

A) How well has Freshgrade worked?

1. Has it been frequently accessed by students, teachers, and parents?

2. What type of evidence of student learning (artifacts, videos, observations recorded, etc.) is included in portfolios?

3. How have teachers responded to dedicating extra time and effort to implementing Freshgrade?

4. Is there consistency across the learning community in the way portfolios are organized?

5. How is the availability and reliability of technology been addressed?

6. How do parents respond to the program?

7. Does is align with the assessment standards outlined in the curriculum by the Ministry of Education?

B)  What are the particular features of Freshgrade's e-portfolios that make a difference in student learning?

1. How are portfolios organized?

2. Are posts showing student learning growth over time?

3. Are curricular standards accessible from the program?

4. How are students reflecting about their learning?

5. Have students demonstrated higher order thinking skills?

While we can look back and evaluate it from an outcome approach, we are also still in the midst of implementing this program and making improvements each year.  If we want to continue using e-portfolios, how might we learn and make changes in order to make the program more usable?  Therefore, process is also an important factor.  In our evaluation, we ask questions such as:

C)  Is Freshgrade contributing to depth of student learning?

1. Are students demonstrating reflection?

2. How well are students able to articulate learning growth?

3. How are students able to independently select and analyze evidence of learning?

4. Are students demonstrating a better understanding of teacher standards and curriculum objectives?  Are students aware of criteria for success?

5. Are comments from parents and teachers helpful for student learning? 

6. Is there ongoing, specific teacher feedback?

D)  Does Freshgrade address student learning needs?

1. Are students engaged?

2. Are all students able to access and utilize the technology?

The evaluation for learning and use approach to evaluation assesses whether or not the program is appropriate as well.  In this evaluation, we want to know whether Freshgrade e-portfolios are aligned with the Ministry of Education and government standards, addressing learning needs of students.

Overall, this mixed method, process and outcome-based evaluation for learning and use approach best fits my program evaluation design.  The evaluation is formative in nature and can be used to help navigate the use of e-portfolios for better student learning in the future, and to ensure that the use of Freshgrade is in fact reaching its targeted goals.