Data Collection and Analysis

Methods for Data Collection

As this is an internal evaluation where I am both a primary intended user and evaluator, a variety of methods to collect data using mixed methods would help to authenticate the study. The design of this evaluation is varied with both quantitative and qualitative methods. While the number of posts for reflection, celebration, student selection of learning evidence, and parental and teacher input can be counted and quantified, other methods of data collection are more qualitative in nature. These methods include interviews with participants, use of published research, and determining quality evidence of learning posted on e-portfolios.

In order to address my four key questions, the following methods for data collection will be utilized: interviews, surveys, documented evidence through observation of student portfolios, and published research.

Interviews with both students and teachers (primary intended users) will be conducted from a sample of convenience (Stufflebeam, 2004). Four students per class will be randomly selected, as well as teachers who volunteer (at least 10) to be interviewed These will be conducted by the evaluator using some of the following questions:

Students

Do you enjoy using Freshgrade? (engagement)
Do you know what curriculum or "I Can" statements mean? (knowledge of curricular objectives)
How often do you choose what goes on Freshgrade?
Who is responsible for your learning?
How do you know if your work is meeting expectations?
Can you describe your next steps in learning?
Has Freshgrade helped you learn technology?
What do you like best about Freshgrade?
What is most challenging about using electronic portfolios on Freshgrade?
Any other thoughts?

Teachers

Do you think that using Freshgrade e-portfolios is a good use of time?
Have you noticed a difference in student learning?
Does it benefit all students?
Is there consensus when it comes to organizing portfolios among staff?
Has technology been reliable and accessible?
Are students engaged in the portfolio process?
What do you like best about Freshgrade?
What has been most challenging for you in facilitating e-portfolios on Freshgrade?
Any other thoughts?

A survey will be sent out to all parents in the school community via email, requesting feedback on the use of Freshgrade e-portfolios for students. The following are some examples of questions to be included. Keeping the number of questions to a minimum may help to provide a larger sample of respondents. A scale 1-5 can be used for the first 3 questions with an open-ended response box for the last two questions.

How often do you access Freshgrade?
Do you comment on your child's work?
Do you find it challenging to give your child feedback?
What do you like best about using Freshgrade at our school?
What do you find most challenging about using Freshgrade?

Observations of student portfolios will also provide data for this evaluation. 4 student portfolios from each class will be randomly selected to observe. This includes quantitative data:

Number of reflection entries
Number of students self-assessments
Number of parent posts
Number of teacher feedback responses
Number of student posts on growth
Number of celebration posts
Number of activities with curricular standards included as titles

Qualitative observations include:

Students selecting own evidence of learning
Varied evidence of learning (conversations, artifacts, and observations) in portfolios
Student engagement
Quality of responses and feedback from students, teachers, and parents

Finally, published research is a method of data collection that will be used. A variety of information on e-portfolios from the following academics has been found:

Abrami and Barrett (2005) – evidence for deeper reflection, higher-order thinking, self-awareness, and more feedback through electronic portfolios (EPs).
San Jose (2017) – evidence for EPs shifting responsibility of learning to the learner
Chan and Cheng (2013) – evidence for better assessment and self-regulated learning through EPs
Clarkston (2018) – evidence for parental involvement influencing academic success

An evaluation matrix, as described in the UNICEF video (UNICEF Innocenti, 2014), will be used to ensure that each key questions (with sub-questions) has been addressed in gathering data. Allowing 3 weeks for survey results to come in and interviewing students and teachers over the course of a month, it is expected that the data will be collected within 2 months.

Evaluation Matrix:

Data Analysis

After the data has been collected, we want to examine it to see whether our results match the predictions from our program theory. Have the short and long-term outputs in fact led to the outcomes? Have students experienced deeper, more authentic learning due to the use of electronic portfolios on Freshgrade?

It is important to ensure that the data quality was valid, reliable, complete, precise, and was collected with integrity and timeliness (UNICEF Innocenti, 2014). Since I am both evaluator and a primary user in this program, collaboration with an outside evaluator would help to reduce bias. Having an individual or team of professionals examine the results of the data would strengthen its integrity. Furthermore, ethical guidelines to ensure that data is collected with respect to dignity, diversity, and rights for both adults and children need to be taken into account. Childethics.com can be accessed to ensure compliance (UNICEF Innocenti, 2014).

When examining the qualitative data, trends, patterns, and themes must be examined to see whether the program theory has been confirmed (Stufflebeam, 2017). Do many students seem engaged in participating in their learning on Freshgrade? In the interviews, are teachers noticing more reflection and deep thinking from students when contributing to their portfolios?

A holistic evaluation summary report (1-2 pages) describing the data in the simplest way possible to primary intended users will be provided (Scriven, 2007). Quotes from students and teachers from interviews may be used in the report, as well as a bar graph showing the results of the parent survey. The qualitative data from interviews and academic research (such as student understanding of growth over time) can also be used to explain some of the quantitative findings such as the number of posts on reflection and growth on e-portfolios.

Data Collection and Analysis

Data Analysis

​

​

​