Summary of Client’s Challenge
After a number of years with the current mandatory training and licensure exams for child care workers throughout the State of Florida, the Child Care Office of the Florida Department of Children and Families realized they needed to reevaluate their mandatory training offerings and related exams. In order to deliver recommendations that could impact state and federal legislation, they had to determine, in detail, current and desired results of the training system, as well as gain recommended action plans.
Three-Stage Approach To Addressing Training Deficiencies
Change by Design was hired to research and develop the following three deliverables:
Deliverable 1: CURRENT MATERIALS EFFICACY
What is the current status of the training and exam system from a quality and best practice instructional design point of view and what are the recommendations for improvement?
Deliverable 2: LEARNER AND STAKEHOLDER PERCEPTIONS
What is the current status of the training and exam system from the mouths of child care workers, their employers, and trainers throughout the state of Florida, and what are the recommendations for improvement?
Deliverable 3: CHANGE RECOMMENDATIONS
What are the recommendations regarding policy, upgrades to training and training/exam system, and other changes that will ensure that the state of Florida delivers an effective mandatory training and exam system?
Efficacy: Building An Evaluation Rubric
To accomplish this task, Change by Design collaboratively developed an evaluation rubric with the client that enabled us to then thoroughly evaluate the ten instructor-led courses, ranging from one to eight hours in length. Each course was evaluated on percentage alignment with the each of the three categories that together indicated course viability:
Instructional Quality
Content Quality
Effectiveness
Each of these three categories had several statements our evaluator validated with the client as being acceptable measures.
Instructional Quality Example
An example statement for “Instructional Quality,” along with relevant citation justifying this question, was:
“Use of best practices in eLearning design in addition to instructional design and learning theory are apparent in the eLearning version of this course based on principles of eLearning design and other design principles (above mentioned).”
Citation: Best Practices of Instructional Design in eLearning. Colvin, R. and Mayer, R. (2011). e-Learning and the Science of Instruction: Proven Guidelines for Consumers and Designers of Multimedia Learning 3rd Edition.
CONTENT QUALITY EXAMPLE
An example statement for “Content Quality” was:
“Role as early childhood educator is woven throughout some or all of the modules in this course (all delivery formats); specific details are included about how this role relates to the course and module topics.”
Learning Transfer: Kirkpatrick’s 4 levels of training evaluation
EFFECTIVENESS EXAMPLE
An example statement for “Effectiveness”, and justification citations, was:
“Course simulates (or uses) actual on-the-job situations, tasks, and environment in order to aid retention and transfer to the job by using examples, case studies, woven stories, visualizations, images, videos, learners’ anecdotal experiences, and/or role plays as much as possible according to instructional design principles.”
Efficacy: How We Calculated The Final Composite Score
Percentage points for each course were then added and divided by three to achieve a final composite score. We also analyzed a large number of available statistics related to test taking and test passing, which added statistical depth to our findings for Deliverable 1.
Population Perceptions
We next sought out people in the child education field for input, including:
teachers with 1-3 years of experience,
teachers with more than three years of experience,
child care instructors, and
child care facility owners.
The individuals hired were divided into seven focus groups, each with six or seven people (at most), to serve as representative of greater populations across the state. After convening the groups and gathering input, we quantitatively and qualitatively analyzed the information. These analyses enabled Change by Design to confidently build and distribute a subsequent survey to all state of Florida child care parties in the same identified categories as those from the focus groups.
OBTAINING MEANINGFUL FEEDBACK FROM Each Segment Of The Target Learner Population
The final survey developed and disseminated by Change by Design included content written in the two most prevalent languages of the target learner population, English and Spanish. Change by Design also designed bi-lingual marketing materials to obtain meaningful engagement from the survey recipients. After it was distributed, our team analyzed the survey results to establish a comprehensive understanding of the training’s effectiveness for these populations. That lead to recommendations for how to best address what participants felt were the most important factors.
Our Recommendations For Policy Change
Our third deliverable was developed using reports 1 and 2 to inform actionable recommendations to DCF on improving their training and exam system. We examined rules, regulations, and legislation at the state and federal levels to provide a set of changes to be made at both levels. We further provided a set of action plans to be used by the DCF Child Care Office to implement each recommendation.
Contact Change by Design today if you need workforce training evaluation performed, recommendations for improving learner and stakeholder perceptions, and actionable next steps that your agency can take to align staff members to organizational goals.