Lesson 5 Basics of an Exercise Program

New vocabulary for this lesson:

  • Discussion-based exercise
  • Operations-based exercise
  • After Action Report (AAR)
  • Corrective Action Plan (CAP)
  • Foundation
  • Exercise Planning Team
  • Design and Development
  • Exercise Objectives
  • Exercise Scenario
  • Briefing
  • Hot-Wash
  • Cold-Wash
  • Exercise Evaluation
  • Improvement Plan

Remember our definition of an Exercise?

An exercise is a simulated emergency, in which members of various agencies perform the tasks that would be expected of them in a real emergency.

You can also think of an exercise as a simulated emergency condition.   This involves planning, preparation, and execution.  An exercise is carried out to test, evaluate, plan, develop, train, and/or demonstrate emergency management systems and individual components and capabilities.  It’s also used to identify areas of strength and weakness for improvement of an emergency plan.

In Exercise Design, there are two basic types of exercises:

  • Discussion-based
  • Operations-based

Discussions-based Exercises familiarize participants with current plans, policies, agreements and procedures, or may be used to develop new plans, policies, agreements, and procedures.  Let’s briefly review these.

Types of Discussion-based Exercises include:

  • Seminar. A seminar is an informal discussion, designed to orient participants to new or updated plans, policies, or procedures (e.g., to review a new Evacuation Standard Operating Procedure).
  • Workshop. A workshop resembles a seminar, but is used to build specific products, such as a draft plan or policy (e.g., to develop a Multi-year Training and Exercise Plan).
  • Tabletop Exercise (TTX). A tabletop exercise involves key personnel discussing simulated scenarios in an informal setting. TTXs can be used to assess plans, policies, and procedures.
  • Games. A game is a simulation of operations that often involves two or more competitive teams, using rules, data, and procedures designed to represent an actual or simulated real-life situation. 

Operations-based Exercises are used to validate plans, policies, agreements and procedures, clarify roles and responsibilities, and identify resource gaps in an operational environment. Let’s briefly review these. Types of Operations-based Exercises include:

  • Drill. A drill is a coordinated, supervised activity usually used to test a single, specific operation or function (e.g., a fire department conducts a decontamination drill).
  • Functional Exercise (FE). A functional exercise examines and/or validates the coordination, command, and control between various multi-agency coordination centres (e.g., an emergency operation centre). A functional exercise does not involve any "boots on the ground" (i.e., first responders or emergency officials responding to an incident in real time).

Full-Scale Exercises (FSE). A full-scale exercise is a multi-agency, multi-jurisdictional, multi-disciplinary exercise involving functional (e.g., emergency operation centres) and "boots on the ground" response (e.g., firefighters decontaminating mock victims).

Exercise Design has its own unique terminology.  We’ve introduced the types of exercises, shown you some examples of exercises, and given you a definition of an exercise.  Here are a few more terms that you’ll be seeing as you make your way through the world of exercises:

  • After Action Report (AAR)
  • Corrective Action Plan (CAP) 

What is a Corrective Action Plan (CAP)?  It’s a process that follows an exercise to identify program shortfalls and necessary corrective actions to address those shortfalls. The Plan provides the techniques to manage the capabilities improvement process.

What is an After Action Report (AAR)?  It’sthe formal written documentation analyzing the performance of assigned personnel after an exercise or an actual event. It is the final product of an exercise and captures observations and recommendations based on the exercise objectives as associated with the capabilities and tasks.

You’ll learn more about these reports as we go through the guide.  It’s introduced now so that you can start to become familiar with this new language.

What is Included in an Exercise Program?  

Remember our definition of an Exercise Program?

An exercise program is risk-based and includes a cycle, mix, and range of exercise activities of varying degrees of complexity and interaction.  

  1. An Exercise Program is risk-based and reviewed annually to see if the risks and hazards of the community/organization have changed.
  2. An Exercise Program is part of the Preparedness function of your Emergency Management Program.  It supports the prevention of, mitigation of, response to, and recovery from, an emergency.
  3. An Exercise Program is a multi-year exercise plan, such as a five year exercise plan.  Your plan should note the requirements of your exercise program and include an exercise schedule that is updated annually.
  4. An Exercise Program’s multi-year exercise plan is a cycle of activity with increasing levels of complexity using discussion based and operations based exercises.
  5. In an Exercise Program, all tabletop exercises, drills, functional exercises, and full-scale exercises are evaluated so you can see if they have achieved your identified goals and to measure performance.
  6. In an Exercise Program, an After Action Report (AAR) is prepared following every exercise – no matter how big or small.
  7. In an Exercise Program, a Corrective Action Plan (CAP) is developed, and implemented, to address the findings and recommendations that you identified in the After Action Report (AAR).  Every one wants to see measures of success.  For that reason, the most immediate improvements seen to be needed should be emphasized. 

Any Exercise Program will take into account jurisdiction wide exercises.  Perhaps your program will be part of these exercises, or your program will structure its exercises to test the same elements 

Seven Key Principles of Developing a Program

Now that we have a basic idea of what an Exercise Program is, let’s talk about the seven key principles to use in developing your own Multi-Year Exercise Program.

  1. Coordinate your exercise schedule with other jurisdictions.  For example, perhaps you want to test cross-border response.
  2. Link a full scale operations-based exercise with other jurisdictions, as appropriate.  For example, if you test evacuation procedures, you want to work with the jurisdiction          you will evacuate to so that they, in turn, can test their response to receiving evacuees from another area.
  3. Coordinate major exercise activities through a committee structure.
  4. Conduct an annual review of the exercise program to make sure that the objectives are being met. Revise the exercise program as required after the review.
  5. Include both discussion-based and operations-based exercises in your exercise program. 
  6. Prepare an After Action Report (AAR) after every exercise. 
  7. Develop and implement a Corrective Action Plan (CAP) to deal with the findings and recommendations identified in the After Action Report (AAR).

Five Phases of an Exercise

OK, we’ve talked about some of the types of exercises, identified some terminology, and discussed the components that make up an Exercise Program.  Now let’s take a look at an exercise itself.  Did you know that there are five phases to an exercise?  Based on what we’ve already discussed, can you figure out what they would be?  To get you thinking in an exercise design frame of mind, here are the phases you go through in designing any exercise:

Phase 1      Prepare the Foundation

Phase 2      Design and Develop the  Exercise

Phase 3      Conduct the Exercise

Phase 4      Evaluate and Report on the Exercise

Phase 5      Improvement Planning

What is involved in each of these phases?  Let's discuss each phase in more detail.

Phase 1 Prepare the Foundation

Phases of An Exercise

Phase 1 Prepare the foundation

Phase 2 Design and Develop the Exercise

Phase 3 Conduct The Exercise

Phase 4 Evaluate And Report On The Exercise

Phase 5 Improvement Planning

What is Foundation?This is the first stage in the exercise stage, focusing on developing a project management timeline, establishing milestones, identifying an exercise planning team, and scheduling planning conferences.

A good exercise program starts with a foundation.  From the foundation the building blocks are added, renewed, and revised on a recurring basis. Preparing the foundation for a successful exercise requires project management skills and includes the following steps:

  • Assess capability to conduct an exercise.
  • Define the exercise scope.
  • Develop an exercise planning timeline with milestones.
  • Select participants for a planning team.
  • Schedule planning conferences.
  • Develop an exercise work plan.

Exercise Planning Team 

The individuals who prepare the foundation are part of an Exercise Planning Team.

What is an Exercise Planning Team? Thisis a group of individuals with the overall responsibility for all phases of an exercise.

An exercise planning team is needed for each exercise. It’s important that each participating agency nominate members onto this group. All agencies with a role to play in the exercise should be invited to take part.

The exercise planning team is responsible for designing, developing, conducting, and evaluating all aspects of an exercise. The planning team determines exercise design objectives, tailors the scenario to local needs, and develops the documentation used in exercise evaluation, control, and simulation. Planning team members may also assist with developing and distributing pre-exercise materials, conducting exercise briefings and training sessions. 

The exercise planning team should be chaired, where possible, by an Exercise Coordinator who is nominated from the pre-determined lead organization for the exercise. The Exercise Coordinator works with the assistance of the planning team, and has overall charge of planning, exercising and debriefing.  It falls to the Exercise Coordinator to control the exercise tempo, and ensure continuity from one phase to the next, including early termination for safety or other reasons. 

Exercise Planning Timelines

The exercise planning team sets a timeline for the planning process. This timeline identifies key planning meetings, critical responsibilities and activities. Be aware that timelines vary, depending on the exercise scope and complexity. 

TIP:  The planning process may be easier if an exercise work plan is developed first.  

Planning Conferences

During the process of developing your exercise, you’ll find that the planning conferences/meetings that your team will schedule and attend fall into four categories:

  • Concept and Objectives Meeting
  • Initial Planning Conference
  • Mid Term Planning Conference
  • Final Planning Conference

Concept and Objectives Meeting

What is a Concept and Objectives Meeting (C&O Meeting)?It’s the formal beginning of the exercise planning process, held to agree upon already-identified type, scope, capabilities, objectives, and purpose of the exercise.

This meeting is the formal beginning of the planning process.  We say “formal” as likely there have been informal discussions prior to calling this meeting.  The goal of this meeting is to identify:

  • The type of exercise needed
  • Scope of the exercise
  • Objectives of the exercise
  • Purpose of the exercise

Who would attend this meeting?  Usually this meeting is attended by representatives of the sponsoring organization, the Lead Exercise Planner, and selected senior officials.

Decisions made in this meeting need to be written down in a briefing or concept paper.  This paper is then used as a point of reference for future planning meetings and the exercise itself.

If your proposed exercise is not complex, and/or you have limited resources, this meeting may be conducted at the same time as the Initial Planning Conference.

Initial Planning Conference

What is an Initial Planning Conference (IPC)? It brings together the stakeholders and plan the upcoming year(s) of exercises.  The IPC is typically the first step in the planning process and lays the foundation for the exercise (unless a C&O Meeting is held).

The foundation for the exercise development process is laid in the Initial Planning Conference.  The goal of the Initial Planning Conference is to:

  • Gain agreement and support from the exercise planning team on scope, design requirements, and conditions.
  • Determine objectives, levels of participation, and scenario variables from each participating organization.

Before holding this meeting, some groundwork, such as a concept paper, would have been laid, based on the Concept and Objectives meeting.  Additional preparation work prior to the Initial Planning Conference includes a briefing for the exercise planning team that gives an overview of the exercise and briefly explains: the purpose, goals, objective, and a narrative of the scenario contemplated.

If the initial work in the Concept and Objectives Meeting and the preparations for the Initial Planning Conference are done well, then you can expect the following to be accomplished:

  • Clearly defined, obtainable, and measurable objectives.
  • Exercise narrative.
  • Identification of major events.
  • Identification of scenario variables (such as the threat scenario, any victims, venue).
  • Participation by appropriate organizations.
  • Identification and recruitment of subject matter experts (SMEs) and facilitators.
  • Assignment of responsibility for exercise document development and presentations/briefings.
  • Where to get all source documents (including policies, plans, and procedures) needed to draft exercise documents and presentations.
  • Identification and assignment of logistic responsibilities (such as invitations, badges, registration).
  • Determining dates of completion for action items and tasks.
  • A planning schedule.
  • Identification of critical tasks for the next planning conference.
  • Decision on the date, time, location of the next planning conference and the actual exercise.

That’s a lot of work, isn't it?  As you can imagine, there is some follow up work involved after the Initial Planning Conference.   So that no one forgets what was discussed and the responsibilities that were assigned, meeting minutes should be prepared and sent out within a week of the meeting.  You’ll also want to encourage all members of the exercise planning team to stay in contact with each other.  

Mid-Term Planning or Master Scenario Events List Conference

What is a Mid-Term Planning Conference (MPC)?  This is an exercise planning conference, used to discuss exercise organization and staffing concepts; scenario and timeline development; and scheduling, logistics, and administrative requirements. It is also a session to review draft documentation. 

What is a Master Scenario Events List Conference (MSEL Conference)?This conference may be held in preparation for more complex exercises, specifically to review the scenario timeline and focus on MSEL development.

Depending on the level of complexity of your exercises, all mid-term planning may be accomplished in one Mid-Term Planning Conference.  However, for a more complex exercise, a Master Scenario Events List Conference may be needed to focus on the scenario and all of the events that will drive that scenario.  A Mid-Term Planning Conference and/or a Master Scenario Events List Conference applies to both discussion-based and operations-based exercises and gives planners a chance to develop a chronological listing of events and injects that will drive the exercise play. 

Final Planning Conference

What is a Final Planning Conference?  It is the final forum for reviewing exercise processes and procedures before the exercise begins.

Prior to this conference, all members of the planning team should receive:

  • An agenda.
  • Minutes of the Initial Planning Conference.
  • Final drafts of all exercise materials.  At the Final Planning Conference, no major changes should be made to the design or scope of the exercise, nor of the supporting documentation.

As a general guideline, you may find that the Final Planning Conference is ½ day for a discussion-based exercise, and a full day for an operations based exercise.  Given the purpose of the conference, it’s a good idea to have the meeting close enough to the exercise site so that your team members can have a final site walk through. 

The purpose of the Final Planning Conference is to:

  • Work out any remaining issues related to exercise planning.
  • Identify last minute concerns that may arise.
  • Review all exercise logistical tasks (such as schedule, registration, attire, special needs, refreshments, room configuration and set up, audio visual equipment).
  • Conduct a comprehensive final review of and approve all exercise documents and presentation materials.

Follow up work after the Final Planning Conference means you'll have to:

  • Prepare and send out minutes to the exercise planning team members within a week of the conference conclusion
  • Discuss  any outstanding issues with the exercise planning team members, especially issues related to the logistics for conducting the exercise
  • Check that the planning team finalizes all publications, prepares all supporting materials, rehearses presentations and briefings, and prepares to conduct the exercise
  • Prior to the exercise, give information and documentation to personnel such as presenters, facilitators, controllers, evaluators, simulators

Phase 2 Design and Develop the Exercise

Phases of An Exercise

Phase 1 Prepare the foundation

Phase 2 Design and Develop the Exercise

Phase 3 Conduct The Exercise

Phase 4 Evaluate And Report On The Exercise

Phase 5 Improvement Planning

What is Design and Development? Building on the exercise foundation, the design and development process consists of identifying capabilities, tasks, and objectives, designing the scenario, creating documentation, coordinating logistics, planning exercise conduct, and selecting an evaluation and improvement methodology.

Once a foundation is established, you can begin to design and develop your exercise.  Planning a successful exercise requires coordination skills to help you work with participating agencies and officials.  This phase includes the following steps:

  • Managing the project.
  • Convening a planning team.
  • Conducting effective planning conferences.
  • Identifying exercise design objectives.
  • Developing the scenario and documentation, including major and minor events.
  • Assigning logistical tasks.
  • Coordinating the involvement of participating organizations and officials.
  • Identifying the evaluation methodology.

The extent of work and time required for this phase depends on the complexity of the exercises planned.

Exercise Objectives

The first step of an exercise is to decide upon the goals.  This helps to set clear objectives and outcomes to meet those goals. Having everyone agree on the goals, objectives, and outcomes will help to make sure that the appropriate type of exercise is selected.  This in turn helps to decide how the exercise will be evaluated.  For the first few exercises you run, keep the objectives simple, clear and limited in scope.

Exercises can occur within individual organizations or on an inter-organizational basis. When joint exercises are planned, the senior management of all participating organizations need to agree on the overall goal of the exercise.  Specific objectives for each organization can be set individually but need to be consolidated so that that they don’t conflict with those of another participating organization. Broad participation from all stakeholders is important for training and exercises if a wide range of preparedness needs will be met.  

These are established for every exercise. Well-defined objectives provide a framework for scenario development, guide individual organizations’ objective development, and inform exercise evaluation criteria. 

Exercise Scenario

What is an Exercise Scenario? It provides the backdrop and storyline that drive an exercise. For discussion-based exercises, a scenario provides the backdrop that drives participant discussion. For operations-based exercises, the scenario provides background information on the incident catalyst of the exercise.

A scenario is developed using the agreed goals, objectives and outcomes as guidelines.  Scenarios enliven and focus an exercise.  However, a scenario should not take over the exercise as it is just a means to an end. Any scenario selected should complement the main goal of the exercise. Avoid unlikely or unusual hypothetical incidents. Scenarios which fit with local geography, and which could reasonably happen, add realism which, in turn, will add to the interest in, and credibility of, the exercise.

A scenario provides the backdrop and storyline that drives an exercise. The first step in designing the scenario is to determine which threat or hazard to use. Each type of hazard presents its own strengths and weaknesses to be used for evaluating different aspects of prevention, response, and recovery.

The next step is to determine the facility or site that the scenario will affect during the exercise.  A balance needs to be struck between exercising in the area that the scenario problem is likely to affect, and letting the day-to-day activities carry on as normal. Exercises held outside of normal working hours have a number of advantages.

Exercise Timing

One key decision you’ll have to make early on in your planning is whether the exercise is to run in real time, or whether the scenario will unfold in a series of vignettes corresponding to stages along a timeline. You’ll also have to decide whether you’ll call a stop at any point during the exercise to allow for review, or to consider alternative actions that might be taken due to variables such as weather, time of day or year. 

An exercise tale……

Sorry…..we only work 9-5!” To keep costs at a minimum by not entailing overtime costs, it was agreed that the exercise would be played during normal working hours at all locations and only on work days (ie not on weekends).  This created a number of problems particularly with the maintenance of exercise momentum and reality with respect to the flow of information and resources.  It also caused a restricted time overlap of only six hours between the participants in Ottawa and those of the Western Provinces.”

Exercise Location

Whatever type of exercise is chosen, it’s important to note that the planning team should visit the location – at a similar time and day as the exercise – to ensure that it is appropriate. Written permission to use a location may be needed, and any potential users will need to be notified that the location may not be available on a certain date. 

An exercise tale……

 “Did you know a bridge has two ends?” Only the east side of the bridge was initially secured by police.  Inner perimeter barricades were not set up with a control entry point, which caused problems for police officers as they were unable to control pedestrians. The region had barricades available for the other side of the bridge, but they were not requested for use.”  

Phase 3 Conduct the Exercise 

Phases of An Exercise

Phase 1 Prepare the foundation

Phase 2 Design and Develop the Exercise

Phase 3 Conduct The Exercise

Phase 4 Evaluate And Report On The Exercise

Phase 5 Improvement Planning

After you’ve designed and developed the exercise, and arranged the logistical details, you’re ready to begin your exercise!  The day of the exercise is the result of all your team’s planning.

Conducting a successful exercise requires facilitation and project management skills to ensure the exercise takes place and includes the following steps:

  • Setup
  • Briefings
  • Facilitation/control/evaluation
  • Wrap up activities

Exercise Control Centre

Is an exercise control centre necessary?  This is an issue to be considered by the exercise planning team. Many times a control centre is needed only for live exercises. An exercise control centre should be in a suitable building close to the exercise site.  It can be used as an assembly point, for briefings, and where victims, if used, can be prepared. Don’t forget to consider car-parking facilities! If the exercise control centre is not within walking distance from the exercise location, then you’ll need to figure out how to provide transport.  

An exercise tale……

Is restricted parking a reality?”  It was mentioned by one of the media persons, on the scene, that in the event of a real disaster, no attention would be paid to the ‘parking only’ spots.  Media would park their cars as near as possible to the site.  What confusion would this cause?

Health & Safety

The safety of personnel during an exercise is important. All participants – including controllers, evaluators, volunteers and observers- need to be made aware of any hazards within the area and reminded of safety issues. You’ll need to remember that exercise participants may not be familiar with the location, and control may be needed to make sure that they stay within the exercise area. 

An exercise tale……

Watch out or the newscaster could be the news item!”  Location of the real Media Press Conference so close to the scene was inappropriate.  Press conferences should be conducted away from the scene in a controlled location such as a boardroom, or outside the secure perimeter, so as not to interfere with operations. 

It’s a good idea to appoint a Safety Officer and carry out a risk assessment for every live exercise to test that structures are safe and no unseen dangers are present on the site. All participants need to be reminded to comply with safety requirements and not place themselves, or others, in danger. At complex exercises, or where conditions are particularly hazardous, each participating agency may need its own safety officer. An exercise can’t be seen as a reason not to comply with health and safety requirements. The Safety Officer must be easily identifiable and have the authority to intervene, as necessary, to ensure the health and safety of personnel. 

An exercise tale……

Keep in mind the lay of the land!”  A triage and casualty collection area was established at 40 minutes into the exercise, but it was not used.  It was the opinion of the evaluators that the location chosen was a poor one.  The area was located on a small hill some distance from the crash scene.  Because of the distance and elevation it possibly proved to be too much of a drain on the physical resources of the personnel carrying the stretchers.

First aid / Ambulance coverage is provided to deal with any health problems or injuries sustained during an exercise. For safety reasons, have an agreed procedure for interrupting the exercise, including stopping the exercise if necessary. The exercise planning team needs a codeword for this purpose and a way of relaying it to all participants.

Take care of your people

Don’t forget to take care of your people during exercises. Their needs may vary depending on the type, timing and duration of the exercise. Personal care support such as providing refreshments, changing areas, washing and toilet facilities are important. 

An exercise tale……

But I really have to go…” The lack of a rest station, feeding facilities and limited washroom facilities would be a serious shortcoming in an extended 24/7 real life emergency.

The use of victims adds realism to exercises but their needs also have to be considered. For example, exercise victims should not be placed in or near unsuitable conditions, e.g. cold, wet or hard surfaces, without appropriate care. Often the length of time planned for an activity turns out to be much longer. It’s therefore important to have an area which is warm and dry. 

An exercise tale……

Don’t make the victims ill.” It’s a good idea to separate the walking wounded from the red tagged victims.  These green and yellow casualties were sitting around the base of the deck & loading ramp.  One side was sunny and the casualties were comfortable and warm.  However, if you stuck your head around the corner, directly under the ramp, it was shady.  The shade and cold wind lead to convection cooling, which left the casualties quite cold and agitated.  This could lead then to either hypothermia or shock.  Why create more work?  First Aid is essentially common sense under pressure.

Identification

Exercise participants need identification that is similar to what would be used in a real emergency. To avoid confusion, all those who are not actively participating in the exercise scenario e.g. the lead exercise planner, exercise staff, evaluators, observers, should be easily distinguishable.  It is important in all types of exercise to be able to identify which organization each person represents. 

An exercise tale……

Who are you?” Identification of major players was not sufficient, as was the case for Observers, Media, Casualties, etc.  Issued tags were very small and hard to read from any distance.  One suggestion is to use coloured arm bands in addition to identification.

Public Information

The exercise planning team needs to decide whether there should be any prior publicity. It may be a good idea to give prior public information to members of the public in the surrounding area of the exercise to prevent any undue alarm, particularly for exercises at hazardous sites. All reasonable steps need to be taken to make sure that the public does not think that any live exercise is a real event. 

What is a Briefing? It’s a meeting held, before the exercise begins, to inform participants on the ground rules of conduct and their roles and responsibilities.  

Briefing

A full briefing needs to be given to all exercise participants.  Each organization should take responsibility for the briefing of their staff. The extent of the briefing varies with the type of exercise. 

Exercises are given a code word (also sometimes called a code name).  It’s a good idea to give instructions that the code word be mandatory as a prefix to all messages – verbal or written – during the exercise. The use of code words helps make sure that everyone involved is aware that they are part of the exercise and not a real incident. Control rooms and operation centres of all participating organizations need to know about the codeword - before the exercise. A code word, which can be used to identify that a real incident has occurred, e.g. “No Duff” and is not part of the exercise, should also be decided, and given to all participants prior to the exercise.

Media 

Dealing with the media is a significant part of any major incident, which means that you should take every opportunity to practice your media plan during an exercise.  Exercise press conferences can test media skills and information management. 

When planning any large exercise, expect the ‘unplanned’ arrival of the media to cover your exercise. 

An exercise tale……

Why not use real media?” Students playing the role of media created confusion for Public Affairs Officers who also had to deal with the legitimate media covering the exercise as a news event, not content to remain merely observers.  Student media are not required.  Let the real media play their normal role.

Debriefing, or “Hot-Wash”

Debriefing is a crucial stage of any exercise.  If well done, it’s an opportunity to evaluate efficiency, learn from the experience gained and determine how well the emergency management process went. 

A ‘Hot-Wash’ that’s held immediately after the exercise is a good way of capturing participants’ instant reactions.  A ‘Cold-Wash’ is held some time after the event and gives participants time to reflect on their experiences.  

What is a Hot-Wash? It’s an immediate debriefing session between players and members of the exercise planning team to discuss their preliminary observations. A hot-wash is done while events are fresh in everyone’s minds. What went right; as well as what went wrong, is identified. Ideas about how to improve in the future are freely shared. The Exercise Controller must carefully avoid two dangers here: first, self-congratulatory accounts that mask important deficiencies and, second, the creation of an impression that someone or something is to blame. This information will inform the After Action Report.

What is a Cold-Wash? It’s a post-exercise meeting that is held after a period of time, not immediately after the exercise.  Preliminary observations and evaluations are discussed and participants have an opportunity to provide feedback that might have been missed in the hot-wash.  

Debriefings need to occur both at an individual service level and at the inter-organizational level.  The exercise coordinator and exercise team need to make sure that the necessary debriefing sessions take place. All participant organizations, including the voluntary sector and any private sector involvement, should be represented at an inter-organizational debriefing.

Phase 4 Evaluate and Report on the Exercise 

Phases of An Exercise

Phase 1 Prepare the foundation

Phase 2 Design and Develop the Exercise

Phase 3 Conduct The Exercise

Phase 4 Evaluate And Report On The Exercise

Phase 5 Improvement Planning

What is Exercise Evaluation?  It’s the act of observing and recording exercise activity or conduct, by comparing the behavior or actions against the exercise objectives, while noting strengths and weaknesses.

OK.  You’ve designed, developed, and conducted an exercise.  Your job isn’t finished.  Now you have to evaluate the results to see if your exercise objectives were met.  Evaluation is the cornerstone of exercises. Exercises can be expensive and disruptive so you’ll want to make sure that you get the maximum benefit for the effort involved. The quality of evaluation and identification of learning points is crucial.

Exercises that result in no improvements are, at best, limited to a good experience for the participants. Exercises are costly in time and resources and so the best possible use should be made of them. Lessons learned at one exercise should produce benefits for all stakeholders.

As an exercise is underway it’s a great idea to put in place a process to observe the exercise and follow this up with a constructive critique of the events as they occurred. Your participants need a chance to comment on the exercise from their point of view. The results should be collated in a final report and communicated to all concerned. 

Exercise evaluation isthe process of

  • Observing and recording exercise activities.
  • Comparing the performance of the participants against the exercise objectives, and
  • Identifying strengths and weaknesses as this information will be the backbone of the final report.

A good evaluation of an exercise can identify:

  • Whether an exercise has achieved its objectives.
  • Needed improvements in standard emergency procedures or guidelines.
  • Needed improvements in the emergency management system.
  • Training and staffing deficiencies.
  • Needed operations equipment.
  • Need for continued exercising of the plan and the emergency management functions.

Successful evaluation of and reporting on an exercise requires analytical and diplomatic skills if an exercise is to be of value.  Evaluations need to record strengths, as well as opportunities for improvement, in an emergency management program.  This phase includes the following steps:

8 Steps in the Evaluation Process

Step 1: Plan and organize the evaluation

 Step 2: Observe the exercise and collect data

 Step 3: Analyze data

 Step 4: Develop the draft AAR

 Step 5: Conduct an exercise debrief

 Step 6: Identify improvements and corrective actions that need to be implemented

 Step 7: Finalize and issue the AAR

 Step 8: Track implementation

All the steps from the evaluation phase lead to the improvement planning phase.

Phase 5 Improvement Planning

Phases of An Exercise

Phase 1 Prepare the foundation

Phase 2 Design and Develop the Exercise

Phase 3 Conduct The Exercise

Phase 4 Evaluate And Report On The Exercise

Phase 5 Improvement Planning

Successful follow-up to an exercise requires analytical and practical skills in order to turn the lessons learned from an exercise into concrete, measurable steps that result in improved capabilities.  Successful follow-up activities include the following steps:

  • Corrective Action Plan (CAP)
  • Improvement Plan (IP)

Remember our definition of a Corrective Action Plan? 

It’s a process that follows an exercise to identify program shortfalls and necessary corrective actions to address those shortfalls. The Plan provides the techniques to manage the capabilities improvement process.

Improvement Planning Process

The means for converting the recommendations from the After Action Report (AAR) into measurable steps that, when implemented, lead to improved response capabilities.

The Improvement Plan identifies:

  • Actions to address each AAR recommendation
  • Who will be responsible for taking each action
  • A timeline for completion of those actions

Once recommendations and action items have been identified, organizations should ensure that each item is tracked to completion and improvements are implemented.

Evaluation and improvement planning are linked together. Improvement planning is a process where concrete actions that address the issues observed during an exercise are developed, assigned, implemented and tracked. 

It’s important for the Exercise Planning Team to make sure that all learning points and action items are agreed to by each of the participating organizations. The action items identified are documented in the final report, and a time frame agreed for implementation of the action items. This process is monitored by each of the organizations and the results validated during subsequent exercises.

It is through this cycle of continuous improvement that exercises can prepare organizations for all hazards. Without effective evaluation and improvementplanning, this won’t happen.

Here are some suggested questions to help you develop action items:

  • What changes need to be made to plans and procedures to improve performance?
  • What changes need to be made to each organizational structure to improve performance?
  • What changes need to be made to leadership, coordination and management processes to improve performance?
  • What training is needed to improve performance?
  • What changes to equipment is needed to improve performance? Is additional equipment needed?
  • What lessons can be learned that will help us to approach a similar problem in the future?

OK, we’ve talked about the different phases of an exercise, and some of the key tasks.  Let's recap:

Any exercise program functions in a cyclical way, in that it starts with:

  • a foundation and a plan
    • moves into the design and development stage
      • moves on to exercise execution and then
        • completes a full cycle with an evaluation
          • then goes into the corrective action and improvement planning stages. 

In the next lesson, let’s take a look at the key roles played by the exercise participants and how they would fit into the phases.



Previous Table of Contents Next