Evaluation is concerned with the quality and efficacy of outputs produced by the design process. With evaluation, these outputs or learning products are tested and measured against certain criteria, and then revised for optimal quality.
"Problem analysis involves determining the nature and parameters of the problem by using information-gathering and decision-making strategies."
—Seels & Richey, 1994, p. 56
CMR Institute - 2021 - present
Once every 6 months or so, a core group at CMR meets to analyze the usage of our library and make decisions accordingly. We pull reports of our 360+ module titles with the number of launches over the past two years as well as which clients and programs have each module. We systematically go through a series of guiding questions to help us decide whether to keep, archive, or change a module. For the low-use modules that we determine still have intrinsic value to our organizations and our target audience, we strive to understand what factors could be contributing to the low use — a comparatively weak design, a not-terribly-relevant topic, a title that missed the mark, etc.
In early 2024, for example, we identified one title "Engaging in a Productive Dialogue about Drug Prices," which had a low number of launches. We agreed it contained relevant and important information for our target learners — the challenge was likely in getting clients to see its value. We are currently in the process of redesigning and redeveloping the module as a scenario-based training resource with the subtitle "Or, How to Defend Yourself at a Dinner Party." The attention-grabbing title combined with practice opportunities that may resemble real-life conversations ensures a higher probability of customer appeal as well as learning transfer.
"Criterion-referenced measurement involves techniques for determining learner mastery of pre-specified content."
—Seels & Richey, 1994, p. 56
VT TLOS / EDIT 5904 Project & Report - Spring 2021
On the left below is the instructional analysis (featuring performance objectives and corresponding criterion-referenced items) and on the right below are the pretest and posttest survey results. These tests provided insights into the strength of the instruction, and which areas needed improvement. Objective 4, "Prepare the Digital Environment" was identified as a notable weakness, perhaps because less time was spent on this content chunk.
"Formative evaluation involves gathering information on adequacy and using this information as a basis for further development. Summative evaluation involves gathering information on adequacy and using this information to make decisions about utilization."
—Seels & Richey, 1994, p. 57
TLOS / EDIT 5904 Project & Report - Spring 2021
On the left is the evaluation report, which includes the evaluation methods used, an analysis of the pretest and posttest results, the workshop feedback survey, and the webpage feedback survey. As noted previously, Objective 4, "Prepare the Digital Environment," was the weakest area in the posttest analysis.
"Long-range planning that focuses on the organization as a whole is strategic planning....Long-range is usually defined as a future period of about three to five years or longer. During strategic planning, managers are trying to decide in the present what must be done to ensure organizational success in the future."
· —Certo, et al, 1990, p. 168
CMR Institute - 2022 - present
At the start of each fiscal year, as Learning Solutions Manager, I have played a key role in writing department-level Strategic Initiatives to align with our organizational goals. This is often an iterative process with input from many members of the team. Once the Strategic Initiatives are finalized and approved by leadership, I break these into measurable objectives and tactics (key results) that are tied to specific quarters over the course of the year. From there, I break these objectives out to individual employee's performance goals for the year, ensuring each employee has personal goals they will strive to meet each quarter that support the department, and ultimately support the organization.
Many of our initiatives are often based around innovations, expanding our products and services, and streamlining processes. I am pleased to say that since taking on the role of Learning Solutions Manager, my team has met 100% of our goals to date, often exceeding set expectations.
VT TLOS / EDIT 5904 Project & Report - Spring 2021
On the right is my reflection of the Zoom CORE project. Because the Zoom CORE is part of a larger effort at TLOS to encourage best practices, this project and the Canvas CORE both have the potential for long-term use. In both projects, the central message is provided in a variety of formats to reach as many target learners as possible. For example, on every Canvas course home page, TLOS has included a Canvas CORE button so instructors may refer to the 5 tenets while they're actively designing their course. With these workshops, we also leverage instructor guides so that any TLOS employee may easily teach a session and preserve quality and consistency.