Evaluation is an analytical accountability process that critically examines a program, its materials, its implementation, and its results. The two goals of evaluation in the instructional design realm are (1) ensuring the program under scrutiny is effective, and (2) holding those responsible for the program to account, based on the promised goals or scope. This simplified list of evaluations discusses the top 5 most important and essential types of instructional design evaluations for helping ensure high quality standards for curricula, trainings, and other learning resources your association, business, or non-profit organization produces.
The Basis For Employing 5 Types Of Evaluation To Instructional Design
Entering evaluation into your search engine will likely generate lists upon lists of evaluation types, full of jargon, that all sound the same. One researcher’s list may represent the different evaluation types as distinct from each other.. And another researcher’s list may configure those very same types of evaluations as subtypes, nested beneath others. The truth is that there is no full consensus on one definitive list of evaluation types that agrees with all others. Not only can the same evaluation types be defined differently (depending on the individual), but each evaluation type is configured a bit differently, depending on the subject, methodology, tools, or funder’s requirements. Therefore, any exhaustive discerning of distinctions between them actually leads to more questions, rather than fewer. The one thing we can all agree on is what evaluations are for:
Evaluation involves collecting and analyzing information about the program's activities, characteristics, and outcomes (Patton, 1987).
The five instructional design evaluation types we cover here are anchored to time, each falling along a continuum of maturity. The first type begins from the point of a program’s infancy, when we examine more narrow aspects of a program. And on the opposite end of the continuum, the last type occurs later in time, when we study broader aspects and results of the program.
First Evaluation Type: Formative Evaluation
Formative evaluation seeks to improve the instructional program and its materials before the program goes live, while it is still in a formative stage of development. Completing evaluation at this stage can help you avoid incorrect assumptions which will create costly problems for your organization in the long run.
To find out how effective the materials and overall design are before the program launches, you should prepare your specific inquiries and bring them directly to the learners. Traditionally, you sit one-on-one with representatives of the target population, then run the program with a small group, and/or conduct a pilot test with your intended audience. Through these activities, you can find out how the materials function for learners, how learners perceive them, how well they help learners learn, and how the logistics of the program run. Some of the questions a formative evaluation asks include:
Could the intended audience demonstrate the skills prior to the intervention of training, or was there a skill and knowledge deficit as expected?
Was the program designed and developed according to instructional design best practices?
Is there alignment between goals, objectives, content, activities, and assessment?
Does the training train all of the skills and related knowledge as planned?
Are there gaps in content, or is there content that is superfluous?
What was clear and what was confusing to learners?
Did the program function well when implemented?
Which activities ran smoothly and which were problematic?
Were there any technical glitches?
Did the audience meet the program’s goals and objectives in its trial runs?
Do you project there will be a change in performance on the job?
Did participants like the program overall?
What were the learners reactions to specific sections and activities?
Practical Application For An instructional design Formative Evaluation IN HEALTHCARE
Here is an example to illustrate formative evaluation in action. You have just developed an eLearning module for emergency room physicians and their care support teams about reversing cardiotoxicity. Many subject matter experts have reviewed it, and you have incorporated their input. The course has been quality assured several times by medical directors. Now it’s time to present it to a portion of your target learner population before its full launch. Because there are budget limitations, your formative evaluation will be less full blown than what is traditionally recommended as the industry standard.
You prepare for a couple of one-on-ones with different members of your target population who operate at the top of their license. One is a physician with years of experience, and the other is a med student in residency. Your preparation also includes developing a set of relevant questions and defining at which points in the program you will interview your learners with them. It is a highly interactive process.
After your one-on-ones, you document the findings and their implications in a report, share with leadership, and revise your materials accordingly. But due to budget constraints, you skip the small group evaluation and next move directly to a pilot of fifteen participants. They take the self-paced course over the span of a few days. There is no interaction with you during their training, but you have instructed them ahead of time to document very specific things. Items like what do they find especially helpful? And what are some challenges they encounter?
At the end of the course, they take a scenario-based assessment and complete a detailed survey about their learning experience. You follow up with a few participants for further questioning. Finally, you report pilot results to the client and make final revisions to the program accordingly.
Can you think of a risk you might be facing, after skipping the small group evaluation?
Second Evaluation Type: Performance Evaluation
Performance evaluation refers to judging the on-the-job behavior of an employee or other staff member. This type of evaluation seeks to find out if the skills trained in the “laboratory” setting (i.e., the training program) transfer to the “real-world” setting (i.e., the job). If everything transfers as intended, then it has achieved validation. However, if for some reason the evaluation shows that the performance will not transfer from one setting to the other, then it can also help you determine why. Maybe it’s because the training fell short. Or maybe it’s due to the environment, tools, communication, supervision, motivation, and so on.
Preparing for a performance evaluation requires you to create an observation document. This document is used to compare the skills trained to the employee’s actual performance. One additional variable is when the evaluation needs to be scheduled. For instance, the evaluation situation might require some scheduling to set up the evaluation situation for greatest validity, and have the least interruption to the work environment.
The performance evaluation will involve an evaluator, who may be a supervisor, manager, other leader with authority, or better yet an objective, independent third party. The evaluator will compare the job incumbent’s on-the-job performance to criteria, in a format like an observation checklist or a balanced scorecard. This approach helps identify the extent to which the evaluated employee is meeting the mark, and where they are falling short.
Practical application for an instructional design performance evaluation in a non-profit organization
Here is a real-world example that demonstrates how and why a performance evaluation is used in a non-profit organization. Imagine you are working for a non-profit organization who has recently merged with two smaller regional groups. Each group has numerous service locations within their regions. Despite the synergy gained through the merger, the individual teams are still holding onto legacy practices and conflicting cultural norms, which appears to be undermining results.
You are assigned to head up an intervention. Your team begins by polling a representative sample of all service locations and identifying the key practices, skills, knowledge, and attitudes which contribute to first-class customer service. From there, your team creates instruction to train these attributes and to set clear performance expectations.
Finally, you design a performance evaluation. The evaluation involves an evaluator observing and scoring employees’ demonstration of training skills, according to detailed rating criteria. The design is three-pronged:
direct observation of job performance in the actual job setting;
one-on-one role playing between evaluator and employee, recorded over Zoom and scored after each call; and
review of specific documentation produced by each employee.
The evaluations occur at the three-month, six month and one-year intervals after training. Your hope is to align performance across the newly merged organization and achieve first-class customer service.
What are some of the common criteria you think should be included on any employee’s balanced score card to improve performance?
Third Evaluation Type: Summative Evaluation
You’ve seen how formative evaluation seeks to revise the program before launch, and performance evaluation seeks to measure how the employee behavior changes as a result of a program. Now mid-way through the evaluation spectrum, is the summative type. The summative evaluation looks at many aspects of post-launch training, especially:
how things are going now everything is up and running, and
whether the program should be kept as-is, kept with revisions, retired, or replaced.
To answer the “how things are going” question, you examine the curricula, training materials, and learning support elements to once again make sure they have instructional integrity, according to best practice. You also look at:
how well does the program function, and are there any problems delivering the content, executing activities, or operating equipment?
how well does the program operate in the larger sense, and is it being administered properly?
if there are any issues with preparation, learner access, or availability?
are there any problems with obtaining needed resources?
Additionally, you look at post-test data to judge the extent to which learners are learning as planned. And you consider the answers to these questions alongside a determination of its continued viability. Because there are different factors your organization may weigh for continuance, depending on its unique interests and requirements, such as:
Problems - What were the problems that the training tried to remedy and to what extent have these problems been remedied due to the program? What problems has the program been creating?
Stakeholder interests - What were stakeholders’ needs and concerns that led to the training and have those been addressed as a result of the program? What are their needs and concerns now in relation to the program?
Goals - Beyond the program’s instructional goals, what were the other goals of the training? To what extent have those been met? Is the training still in line with the organization’s mission, goals, vision, and values?
Costs – How much has been invested in this program and how can we qualify and quantify its return on investment, either tangibly or intangibly?
Practical application for an instructional design summative evaluation in a government agency
Imagine that you work at a State Department of Children and Families, helping the organization effectively partner with other agencies in the area of early child care provision. Your supervisor would like you to evaluate the agency’s mandatory training for child care personnel and has provided you with their high-level areas of interest, such as:
Is the instruction meaningful to learners?
Does it position their role as educators sufficiently?
Does it prepare new educators for classroom operations and working with children in specific ways?
Where is there duplication?
And ultimately: How should we adjust the curriculum?
After the due diligence of stakeholder meetings and literature review, you devise an extended set of key quality and effectiveness indicators. You then chunk them into a few logical groups, providing descriptive criteria, and devising a 5-point measurement scale by which to measure each. You place all of this into an evaluation tool formatted into a spreadsheet with dropdowns for rating and an area for comments. After evaluating each course, you report on your methodology, findings, and recommendations.
In the next phase of this summative evaluation, you will go directly to those affected by the curriculum, including new child care providers, experienced child care providers, and their supervisors. You will conduct focus groups and distribute surveys to find out what these groups have to say about the curriculum.
Fourth Evaluation Type: Outcome Evaluation
Outcome evaluation goes beyond the scope of formative and summative evaluations, looking specificallys at the consequences for the organization, those in the organization, and the beneficiaries of the program. It essentially inquires about the variety of ripple effects which result from training participants’ new knowledge and skills.
Practical application for an instructional design outcome evaluation in a manufacturing company
Imagine you are training several sales teams on the use of a new AI-powered myoelectric prosthetic hand that enables finger movement by the wearer. The medical devices are equipped with “smart socket” sensors that interpret nerve signals from the patient's muscles, and translate the signals to movement by the prosthetic. When finished, the salespeople you are training will present their new product knowledge to target customers in the medical community, educating them, and demonstrating this fascinating technology to close the sale.
In this example, your outcome evaluation will go beyond the program- and learner-centered questions of formative, summative, and performance evaluations. It will explore the outcomes of learning as related to the organization and beneficiaries, helping determine what were the previously-defined target outcomes of the training, and what problems did it try to solve. Then it will identify how do the actual outcomes compare to what was intended, and even find out how well the training solved the stated problems.
On the customer and financial side, the outcome evaluation will examine the sales volume for this product after the training, then determine if the sales of this device have affected return business or sales in other areas. Next, the evaluation will find out if there are indicators of customer support for this product (i.e., related events to support customers either proactively or reactively, customer satisfaction, complaints, repairs, returns, etc.) that has affected internal resources.
The outcome evaluation will build on the internal question to look at the entire organization, asking:
How can you characterize or quantify the various effects this training has had on the participants it trained?
How can you characterize or quantify the various effects the training has had on other positions, other departments, the company as a whole, or even the industry as a whole?
How can you characterize or quantify the experiences of new recipients of these prosthetic devices?
What are the unintended outcomes results of the training (if any)?
Fifth Evaluation Type: Impact Evaluation
Whereas an outcome evaluation focuses closer to the organization, the impact evaluation assesses for broader, more longitudinal influence in the world—particularly for the beneficiaries of the program. It aims at finding how the program has moved the needle in the world, and how we have moved the needle for the people who were ultimately affected by our program. So the main purpose of impact evaluation is to determine whether the program is making a positive difference, especially in the lives and well-being of its beneficiaries over time. The impact evaluation might ask:
How have beneficiaries’ lives improved as either a direct or indirect result of the program?
How can we quantify the differences between treatment groups and control groups (i.e., beneficiaries who were either participants in the program or were helped by those who took the program versus individuals with same needs who were not).
What other impacts in the world have resulted, either directly or indirectly, as a result of the program?
Practical application for an instructional design outcome evaluation in a manufacturing company
In this scenario, you are the executive director of a workforce support non-profit who is tasked with defending a program within your oversight. Budget constraints have funding authorities wondering if the program was worth the resources it requires, so you have gathered your data to build a report.. The program in question trains job counselors to coach and help low-resourced and unemployed individuals in their job search. Your team will work with an evaluator to set up the impact evaluation. The study will focus on the beneficiaries of the program (i.e., those job-seekers the trainees work with to help to find employment).
The evaluation finds that 78% of beneficiaries who worked with the trained job counselors in the last three years were employed in either a full- or part-time capacity within one year of counseling. This is slightly better result than relevant employment data reported by the U.S. Bureau of Labor Statistics. As a result, the funders of the program believe it is delivering enough of an impact to justify continuation of the program.
Ready to validate the quality and effectiveness of your learning and development program(s)?
Are there types of evaluation we did not mention that you believe, in your experience, are also critical and commonly used? We’d love to hear your input so please send us your comments. And you may already know where you want to start, but what system will you use to chronicle your findings? Not to worry - we have created a free worksheet that your team can use to start the accountability planning process.
Also, consider subscribing to our email newsletter today to automatically receive the next issue in your inbox.
Related Articles For Further Reading
Effective Organizational Change Requires You To Correctly Understand Context
Transforming Customer Service Through An Evidence-Based Customer Satisfaction System
6 Easy Steps To Deliver Business Process Improvement Results For Your Organization