Training ROI Calculator

June 19, 2012 By Comments

 

Assessing Training Effectiveness

Assessing training effectiveness often entails using the four-level model developed by Donald Kirkpatrick (1994). According to this model, training evaluation should always begin with reaction to the training (level one), and then, as time and budget allows, should move sequentially through learning, transfer, and results – levels two, three, and four. Information from each prior level serves as a base for the next level’s evaluation. Thus, each successive level represents a more precise measure of the effectiveness of the training program, but at the same time requires a more rigorous and time-consuming analysis.

The Fifth Level of Evaluation

Kirkpatrick’s four-level model, however, does not require a specific cost savings to be ascertained. To obtain a true return on investment (ROI) valuation, the cost savings of the training program are compared to the cost of implementing the training program. According to Phillips (2005), this process moves Kirkpatrick’s four-level model to the next level. During the fifth level of evaluation, Level 4 data is collected, converted to monetary values, and compared to the cost of the program to denote the return on the training program’s investment..

ROI Calculation

The ROI formula, according to Phillips (2005), is the annual net program benefits divided by program costs, where the net benefits are the monetary value of the benefits minus the costs of the program. Phillips (2005) ROI formula is as follows:

ROI (%) = Benefits – Costs x 100 ÷ Costs

As an example, let’s assume that the benefits of a project management training program, isolated from other factors occurring within the organization, are $350,000.
Let’s also assume that the total cost of the program is $125,000.
The net program benefits are $350,000 – $125,000 = $225,000.
Following the formula above:
ROI = $225,000/$125,000 (1.8) x 100
ROI = 180%; thus for every $1.00 invested, $1.80 is returned after the costs of the project management training program are recovered.

ROI Calculator

The ROI Calculator is designed to help you calculate the ROI of a training program.

Summary

Most organizations view the process of measuring ROI as requiring too many resources. It is perceived as difficult, time-consuming, labor intensive, and expensive. Nevertheless, if the worth of the chief learning officer (CLO) and the training function are to be demonstrated, some process must be utilized to establish the value of learning at the corporate level. Evaluation determines the worth, value, or meaning of a training program, and whether or not and to what degree evaluation is performed determines the worth, value, or meaning of the CLO. CLOs would be jeopardizing their value to an organization if they are not equipped to provide the heads of an organization with evidence of a training program’s success.

By Shirley J. Caruso, M.A., Human Resource Development

Resources

Kirkpatrick, D. L. (1994). Evaluating Training Programs. San Francisco: Berrett-Koehler Publishers, Inc.

Phillips, P., & Phillips, J. (2005). Return on Investment (ROI) Basics. Alexandria: ASTD Press.

Filed Under: Adult Learning, Employee Performance, Evaluation, Instructor Resources, Resources & Tools Tagged With: , , ,

Donald Kirkpatrick’s Four Levels of Evaluation

March 13, 2012 By Comments

Donald Kirkpatrick’s four levels of evaluation are designed as a sequence of ways to evaluate training programs:

Level 1 – Reaction

The participants’ immediate reactions to a course can be measured in this level.  This level of evaluation obtains the opinions, feelings, and perspectives of the learner regarding a training program.

Level 2 – Learning

This level tests participants to determine what they have learned.  This level of evaluation demonstrates which principles, facts, skills, and attitudes were obtained from training.   The test questions must be linked to the training objectives.  This approach generally involves a pre-and-post test.

Level 3 – Behavior

Examines behavior changes back on the job as a result of training.  This is usually conducted at intervals after training and requires an established data baseline.

Level 4 – Business Results or Impact

Determines the effects of learning on the organization.  Over 60% of HRD departments fail to conduct this type of evaluation.  However, given the current economic climate, Level 4 evaluation is becoming more and more important as HRD departments are required to demonstrate accountability and how they are adding value to the organization.

As a way of helping determine the feasibility of a result or impact evaluation, program designers would want to answer the following questions:

Do learners value the knowledge, skills, and attitudes being taught?  Do they feel confident to use the knowledge and/or skills?

Do managers of learners coach, reinforce, and model the skills being taught?  Does the organization reward the use of the knowledge, skills, or attitudes? Do they eliminate task interferences?  Is organizational feedback provided?

By Shirley J. Caruso, M.A., Human Resource Development

Filed Under: Adult Learning, Evaluation Tagged With: , , ,

The ADDIE Instructional Design Model: Discussion of the Design, Develop, Implement, and Evaluate Phases

March 12, 2012 By Comments

This video discusses the events that take place during the Design, Develop, Implement, and Evaluate phases of the ADDIE Instructional Design Model.  For more information on the ADDIE Model, please click the following link:

http://www.eadulteducation.org/adult-learning/the-addie-instructional-design-process/

Filed Under: Adult Learning, Evaluation, Instructional Design, Instructional Strategies Tagged With: , ,

Workplace Training and Development

November 8, 2011 By Comments

The workplace is progressively becoming a place where organizations can foster, enhance, and encourage the potential and capabilities of employees.  One way in which organizations foster these talents is by implementing training and development programs with the goal of preparing employees to better meet the challenges of today’s competitive workplace.  However, unless training programs are carefully designed and developed, such programs are unlikely to meet the demands of the competitive workforce as well as the needs of the employees and the needs of the organization.

Designing a Training Program

When designing a training program, it is important to keep in mind that the goal of training is for the employees to acquire new knowledge, skills, and/or attitudes (KSA).   A good training program must be detailed enough to meet the goals of the organization with comprehensible specifications of the purpose and goals of the training program.  Any effective training program must have specific, measurable, achievable, realistic and time bound (SMART) goals in which a clear definition of the goals is communicated to the employees (learners).  The employees’ needs are determined in the context of the nature of work.  It is also very important for the management to define the goals of the training program to make sure that the training program is aligned to the organizational values.  The content of the training program as well as the learner activity, which is central to communicating the course content, is very necessary and should be determined prior to the start of the course.   

Training Evaluation

The success of a training program is not entirely dependent on organization and the content of training, but rather, on the execution and follow up after the training.  This is where evaluation of the success of the training program comes in.  It is imperative that management obtains  feedback for the training from the participants (employees).

By Shirley J. Caruso, M.A., Human Resource Development

Filed Under: Adult Learning, Employee Performance, Evaluation, Instructional Design Tagged With:

The ADDIE Instructional Design Process

April 7, 2011 By Comments


 

Instructional design refers to the process of instructional program development from beginning to end. There are many instructional design models for use by different levels of instructional designers and for different instructional purposes. The instructional design process can be condensed into five phases, commonly known as ADDIE (Analyze, Design, Develop, Implement, Evaluate). Although these phases can sometimes be interrelated, they provide a guideline for developing effective instruction. The steps in the ADDIE instructional design process can provide an organized design approach for developing anddelivering face-to-face and online instruction.

The ADDIE Instructional Design Process

Analyze

The Analyze phase is the basis for all other phases of instructional design. This phase can be broken down into Performance Analysis, Instructional Analysis, Audience Analysis, and Delineate Objectives. The purpose of this phase is to define if training is the solution, define traiing content, goals, and objectives, identify learning outcomes, and determine instructional methods, materials, and media that will appeal to all learning styles.

The objective of this phase is to assess if a discrepancy exists between “actual performance” and ideal performance”, to break down the instructional goal into its compenent parts, to describe the target population’s specific characteristics and entry level skills, knowledge, and attitudes, and to design specific goals and objectives.

Design

The Design phase entails using the outputs from the Analyze phase to plan a strategy for developing the instruction. The purpose of thes phase is to assess ideal performance and describe the content, methods, media, and practices of the training program for the trainer.

The objective of this phase is to use objectives to design written, verbal, simulation, and/or performance tests (pretest, embedded tests, post tests).

Develop

The Develop phase feeds on both the Analyze and Design phases. The purpose of this phase of is to produce the lesson plans and lesson materials. The objective of this phase is to collect learning materials for the training.

Implement

The Implementation phase refers to the actual delivery of the instruction, whether it’s classroom-based or computer-based. The purpose of this phase is the effective and efficient delivery of instruction. The objective of this phase is to promote the students’ understanding of material, support the students’ mastery of objectives, and ensure the students’ transfer of knowledge from the instructional setting to the job.

Evaluate

The Evaluate phase measures the effectiveness and efficiency of the instruction and should occur throughout the entire instructional design process.

Formative Evaluation

Formative Evaluation is ongoing during and between phases. The purpose is to improve the instruction before the final version is implemented. The objective of formative evaluation is to improve training by removing all the imperfections. Formative Evaluation should be triangulated and include one-on-one evaluation by a content expert, evaluation by members of the target audience or another instructional designer, and evaluation after a field of study.

Summative Evaluation

Summative Evaluation occurs after the final version of instruction is implemented. It objective is to improve the program for continued use once it has been implement.

Levels of Evaluation

Levels of Evaluation according to Kirkpatrick’s Model include Reaction (learners’ feelings, opinions and perspectives), Learning (measures achievement of learning objectives), Behavior (behavior change measured at intervals), and Impact (determines effects of learning on the organization).

Summary

The ADDIE model is a systematic instructional design model consisting of five phases: (1) Analyze, (2) Design, (3) Develop, (4) Implement, and (5) Evaluate.

Resources

http://www.eadulteducation.org/adult-learning/the-importance-of-instructional-evaluation-in-hrd/

http://www.about-elearning.com/image-files/addie-instructional-design-model.gif

Filed Under: Adult Learning, Evaluation, Instructional Design Tagged With: , , , , , , , ,

Valuing Intangibles and Phillips ROI Methodology

February 18, 2011 By Comments

By Brad Minor, M.Ed. in Human Resource Development, Peabody College of Vanderbilt University

Phillips (2002) says, “Intangible assets are key to competitive advantage in the knowledge era and are invisible, difficult to quantify, and not tracked through traditional accounting practices” (p. 3).  It is fairly common to divide intangible assets into three categories: research and development; intellectual assets; and knowledge, a category which is then subdivided into tacit and codified knowledge (Phillips, 2002, p. 4). 

One popular method that requires the evaluator to place dollar values on intangibles is the Phillips ROI Methodology (Phillips, 2005), which is especially popular for calculating the ROI of training and development initiatives, though it can also be applied to many other types of programs.  “ROI” stands for “return on investment.”  Phillips views ROI as a fifth level that should be added to Kirkpatrick’s Four Levels of Evaluation (Phillips, 2005), though Jim Kirkpatrick, the originator of the Four Levels of Evaluation (reactions, learning, transfer, and results) disagrees; Kirkpatrick’s company is now pushing the “Return On Expectations” method, or “ROE,” (not to be confused with the unrelated “ROE” of the finance field, which stands for “Return On Equity”) as a direct competitor with the Phillips method.  (Side note: Both of these companies are, of course, in the business of selling certification courses, videos, books, job aids, evaluation instruments, and speaking engagements.)  The ROE method does not place dollar values on intangibles, so we will focus this discussion only on the ROI method.

Phillips (2005) uses the following formula to calculate ROI: ROI (expressed as a percentage) = (Net Program Benefits / Net Program Costs) x 100 (Phillips, 2005, p. 2).  This seems fairly simple and straightforward until one examines the way that program benefits and costs are calculated.  This method goes beyond calculating only the items that are already expressed in dollars; it requires a conversion of intangibles to monetary values.  So how do we do this?  In sum (this is a bit of a purposeful oversimplification) we do the following (adapted from concepts in Phillips, 2005):

  1. Ask stakeholders how much they think something is worth in dollars
  2. Ask the same individuals to express, as a percentage, how certain they are of their estimates
  3. Convert the percentage to a decimal
  4. Multiply the dollar figure (perceived value) by the decimal (level of certainty)
  5. Arrive at a conservative conversion estimate (in dollars)
  6. Qualitatively report any unconverted intangibles along with the ROI figure   

This method is clearly not completely accurate, but it is a common method for reporting the ROI of programs that have intangible benefits.  With this understanding, though, it is easy to understand why accountants, who are concerned with the accuracy and reliability of reported numbers, are often concerned with the use of such methods; this might, in turn, help us understand why such conversion methods have not been adopted into the generally accepted accounting principles (hereinafter referred to as “GAAP”) that govern financial reporting in the United States.

So how does GAAP treat intangibles?  Basically, the only intangibles that are addressed are ones that deal with “intangibles acquired in a business combination” (Foster, Fletcher, & Stout, 2006), with the most common being the monetary value of intellectual property, particularly in companies whose money is derived from the sale or exploitation of it; in the knowledge era, this is becoming increasingly common.  Goodwill, which might be explained as brand power and reputation, can also be reported under GAAP guidelines.  The monetary value of various aspects of human capital, such as specialized knowledge that is not patented, synergy, training processes and capabilities, effective leadership, and corporate culture, cannot currently be reported on standard financial statements.  The United Kingdom’s former Secretary of State, Patricia Hewitt, created the Accounting for People Task Force to address these issues in that country (Weiss & Finn, 2005); perhaps the United States will eventually follow suit, if we haven’t already instigated such an initiative.    

Another way that intangibles are handled is by conducting what Ulrich (2005, p. 52) refers to as “intangibles audits,” which are composed of qualitative measures of intangible performance factors and ultimately end with action plans for the future improvement of those measures. The burning question, though, is this: Are qualitative measures really enough?  Might it make sense to figure out a standard, GAAP-approved method for the valuation of intangibles, in addition to these qualitative reports?

References

Foster, B., Fletcher, R., & Stout, W. (2003, October 3). Valuing intangible assets. Retrieved 12 12, 2010, from:

http://www.nysscpa.org/cpajournal/2003/1003/features/f105003.htm

Phillips, P., & Phillips, J. (2002). In Action: Measuring Intellectual Capital. Alexandria: ASTD Press.

Phillips, P., & Phillips, J. (2005). Return on Investment (ROI) Basics. Alexandria: ASTD Press.

Ulrich, D., & Brockbank, W. (2005). The HR Value Proposition. Boston: Harvard Business School Press.

Weiss, D., & Finn, R. (2005). Hr metrics that count: aligning human capital management to business results. Human Resource Planning, 28(1), 33-38.

Filed Under: Adult Learning, Employee Performance, Evaluation Tagged With: ,

Assessment of Performance

February 6, 2011 By Comments

Instructional Evaluation in Human Resource Development (HRD) is necessary in order to adequately assess the learning and development of a student.  Assessment tools must be developed that reflect your objectives and how the presented instructional activities relate to those objectives. If the objectives are not reflected in the assessment, the assessment will not be valid, and you will likely arrive at inaccurate conclusions regarding the assessment tools and the students themselves.

In instructional design, evaluation serves to assess individual students’ performance and to provide information regarding the types of revisions needed in the instructional materials.  It is important to receive feedback from learners to determine whether or not the learners learned what was intended for them to learn.  In other words, evaluation helps determine if the learning objectives were achieved.

Resources

http://www.eadulteducation.org/adult-learning/the-importance-of-instructional-evaluation-in-hrd/

http://t0.gstatic.com/images?q=tbn:ANd9GcRj8GGDbQD9AwQmVZMqNuGpWrG5LFjwuaeJzSMdSNtkb_hrguZp

Filed Under: Adult Learning, Evaluation, Instructional Design Tagged With:

The Importance of Instructional Evaluation in HRD

September 6, 2010 By Comments

Instructional Evaluation in Human Resource Development (HRD) is necessary in order to adequately assess the learning and development of a student.  Assessment tools must be developed that reflect your objectives and how the presented instructional activities relate to those objectives. If the objectives are not reflected in the assessment, the assessment will not be valid, and you will likely arrive at inaccurate conclusions regarding the assessment tools and the students themselves.

Kirkpatrick’s Four Levels of Evaluation

Assessing training effectiveness often entails using the four-level model developed by Donald Kirkpatrick (1994).  According to this model, training evaluation should always begin with level one, and then, as time and budget allows, should move sequentially through levels two, three, and four. Information from each prior level serves as a base for the next level’s evaluation. Thus, each successive level represents a more precise measure of the effectiveness of the training program, but at the same time requires a more rigorous and time-consuming analysis.

Level 1 Evaluation – Reactions

Evaluation at this level measures how participants in a training program react to it. It attempts to answer questions regarding the participants’ perceptions such as how well they liked the training and if the training design presented was relevant to their work. According to Kirkpatrick, every program should at least be evaluated at this level to provide for the improvement of a training program. In addition, the participants’ reactions have important consequences for learning (level two). Although a positive reaction does not guarantee learning, a negative reaction almost certainly reduces its possibility.

Level 2 Evaluation – Learning

Level two evaluations often use tests conducted before training (pretest) and after training (post test) to assess the amount of learning that has occurred due to a training program.

Assessing at this level moves the evaluation beyond learner satisfaction and attempts to assess the extent students have advanced in skills, knowledge, or attitude. Measurement at this level is more difficult and laborious than level one. Methods range from formal to informal testing to team assessment and self-assessment. If possible, participants take the test or assessment before the training (pretest) and after training (post test) to determine the amount of learning that has occurred.

Level 3 Evaluation – Transfer

This level measures the transfer that has occurred in learners’ behavior due to the training program and attempts to determine if the newly acquired skills, knowledge, or attitude is being used in the everyday environment of the learner.  For many trainers this level represents the truest assessment of a program’s effectiveness. However, measuring at this level is difficult as it is often impossible to predict when the change in behavior will occur, and thus requires important decisions in terms of when to evaluate, how often to evaluate, and how to evaluate.

Level 4 Evaluation- Results

Level four evaluation attempts to assess training in terms of business results. This level measures the success of the program in terms that managers and executives can understand, such as increased production, improved quality, decreased costs, reduced frequency of accidents, increased sales, and even higher profits or return on investment (ROI). From a business and organizational perspective, this is the overall reason for a training program, yet level four results are not typically addressed. Determining results in financial terms is difficult to measure, and is hard to link directly with training.

Summary

In instructional design, evaluation serves to assess individual students’ performance and to provide information regarding the types of revisions needed in the instructional materials.  It is important to receive feedback from learners to determine whether or not the learners learned what was intended for them to learn.  In other words, evaluation helps determine if the learning objectives were achieved.

By Shirley J. Caruso, M.A. Human Resource Development

 

Filed Under: Adult Learning, Evaluation, Resources & Tools Tagged With: , , , ,

Training Evaluation, ROI, and IOL

August 31, 2010 By Comments

This article places emphasis on measuring the value of training, or training evaluation, in terms of its contribution to business goals.  It introduces Impact of Learning (IOL) as a simple, three-step process that organizations can follow to evaluate employee development programs.   IOL equips chief learning officers (CLOs) with a mechanism for providing evidence of a training program’s success by demonstrating the value of learning rather than measuring its return on investment (ROI), or the financial benefits of learning.

Training Evaluation is a Systematic Process

According to Phillips (1997), “Evaluation is a systematic process to determine the worth, value, or meaning of an activity or process.” (p. 36).  Crucial to the evaluation strategy is the determination of what to evaluate.  Exactly what should be evaluated depends on the type of training program, the organization, and the purpose of the evaluation.

One thing is certain; adult learners are a credible source from which evaluation of training stems.  Their reactions to training are solicited, they are tested, and they are even observed to determine the merit of a given training program.  IOL is a three-step process that organizations can follow to evaluate employee development programs.  Insight, Individual, and Impact comprise the three basic steps to the IOL process. Naughton (2008) suggests an insight map be created to visualize what needs to happen and how and when success will be measured.  The second part of the process is to listen to statements of the learners as they provide feedback on the alignment of the training with the goals of the organization.  Thirdly, an impact statement is made as soon as learning outcomes can be seen in the organization.  The impact statement should include at least two cases that demonstrate the impact of learning and refer back to the original insight map.

IOL promises connection to business objectives while producing quicker and more persuasive evaluation results that lie within the credibility of the learners themselves.  This connection is important to determine whether or not the learning intervention is strategically aligned with organizational goals before training is implemented.   

Kirkpatrick and Phillips Models of Evaluation

When taking a comprehensive look at the Kirkpatrick model and the Phillips model for purposes of comparison to the IOL model, the Kirkpatrick and Phillips models mimic a check and balance system.  If the learners did not learn (Level 2), reactions to the training compiled at Level 1 (Reaction) reveal the barriers responsible for the lack of learning.  If the learners did not apply the knowledge and skills (Level 3), the knowledge and skills may not have transferred to begin with (Level 2).  The difficulty and related cost of conducting an evaluation based on the models of Kirkpatrick and Phillips increases as evaluation at each level increases.

The Difficulty of Measuring ROI

Phillips (1997) and Naughton (2008) are in agreement that most organizations view the process of measuring ROI as requiring too many resources.  It is perceived as difficult, time-consuming, labor intensive, and expensive.  Nevertheless, if the worth of the CLO and the training function are to be demonstrated, some process must be utilized to establish the value of learning at the corporate level. Evaluation determines the worth, value, or meaning of a training program, and whether or not and to what degree evaluation is performed determines the worth, value, or meaning of the CLO.  CLOs would be jeopardizing their value to an organization if they are not equipped to provide the heads of an organization with evidence of a training program’s success. The IOL model factors out time and cost as barriers to measuring ROI , and the CLO is left with a means of justifying his/her existence while simultaneously determining the merit of the training function.

Summary

The IOL model offers CLOs a win-win situation. Organizations are provided with the data they need, and CLOs spend less time and resources providing the data, all while the CLO has justified his/her existence.  Additionally, placing value on the learners creates a climate of self-esteem, and the more the learners are involved in the training program, the more likely its implementation is a success.

References

Naughton, J. (2008).  IOL:  Determining the impact of learning.  Retrieved July 29, 2010 from http://www.clomedia.com/includes/printcontent.php?aid=2343

Phillips, J. J. (1997). Handbook of training evaluation and measurement methods (3rd ed.).  Houston:  Gulf Publishing Company.

By Shirley J. Caruso, M. A. Human Resource Development

Filed Under: Employee Performance, Evaluation, Instructor Resources, Resources & Tools Tagged With: , ,
 
In case you missed it, Beginning Spanish for the Native English Speaking Adult Learner, Lesson 2: Pronunciation http://t.co/hydJ9COdy42 months ago