Parks Canada
Symbol of the Government of Canada

Common menu bar links

Internal Audit and Evaluation Documents

Parks Canada 2010-2011 Evaluation Plan

Parks Canada 2010-2011 Evaluation Plan

Recommended for Approval by Parks Canada Evaluation Committee: March 24, 2010
Approved by CEO: March 30, 2010

 

Office of Internal Audit And Evaluation

 

I submit to the Treasury Board of Canada Secretariat, the evaluation plan that I approved for the Parks Canada Agency for fiscal years 2010-2011, as required by the Policy on Evaluation.

As per section 6.1.7 and 6.1.8 of the policy, I confirm that this five-year departmental evaluation plan:

  1. aligns with and supports the departmental Management, Resources and Results Structure;
  2. has been designed to help support the requirements of the Expenditure Management System, including strategic reviews; and,
  3. includes all ongoing programs of grants and contributions administered by the department, as required by section 42.1 of the Financial Administration Act.

I will ensure that this plan is updated annually and will provide information about implementation of the Agency’s evaluation plan to the Treasury Board of Canada Secretariat, as required.

 

___________________________
Alan Latourelle
Chief Executive Officer
Parks Canada Agency
_____________
Date

 

Table of Contents

Executive Summary

1. Introduction

2. Parks Canada Agency

3. The Evaluation Function

Applicable Policies and Professional Standards
Mission and Services Offered
Governance
Office of Internal Audit and Evaluation (OIAE)

4. Evaluation Universe and Planning Considerations

a) Agency Renewal
b) Agency’s 2010-2011 Corporate Risk Profile
c) Status of Projects in 2009-2010 Plan
d) Past Evaluation Coverage and Work of Other Assurance Providers
e) TBS Directed Work
f) Planned Work of External Assurance Providers
g) Considerations Arising from TB Submissions, RMAFs and Budget 2009
h) Evaluation Priority Assessment and Management Priorities

5. Planned Projects

Extent and Limitations of Coverage

6. Appendices

1. Assumptions in the Calculation of Evaluation Capacity
2. Corporate Risk Profile 2010-2011
3. Agency RMAF Evaluation Commitments
4. Evaluation Priority Assessment of Elements of the PAA Universe

List of Tables

Table 1: Program Activities
Table 2: Planned Evaluation Spending for 2010-2011
Table 3: Past Coverage of Program Activity Elements
Table 4: 2010-2011 Evaluation 5-year Plan
Table 5: Evaluation Project Scheduling for 2010-2011

Executive Summary

The 2010-2011 Parks Canada Evaluation Plan outlines the mandate, organizational structure and resources for evaluation at Parks Canada, the considerations in developing the Plan and details of individual evaluation projects for the FY 2010-2011, together with the associated resource allocation. The Evaluation Committees of the Agency, chaired by the Chief Executive Officer (CEO) discussed and recommended approval of the plan. The CEO approved the plan on March 29, 2010.

The Office of Internal Audit and Evaluation (OIAE) at Parks Canada adheres to the government’s policies and standards for evaluation. Currently, the evaluation function in the Office consists of an executive level director and five staffed evaluator positions. Given a fully staffed function, the Office will have the capacity to conduct approximately two to three risk-based evaluations (i.e., an average or typical project is 2,200 hours), as well as addressing other non-risk based work (i.e., provision of advice, coordination, follow-up and participation in TBS directed evaluation work).

Planning is based on the Agency’s Program Activity Architecture (PAA). Work for the 2010-2011 fiscal-year will include the following projects:

  • Concluding evaluations of the Through Highways Management sub-activity and the Visitor Service Offer;
  • Commencing evaluations of the National Park Establishment and Expansion and Through Waterways Management sub-activities (both to be completed in 2011-12);
  • Completing evaluations of the General Class Contributions and Species at Risk Programs (the latter is an Environment Canada-lead interdepartmental evaluation); and
  • Commencing an evaluation framework for National Historic Sites Conservation.

The five-year evaluation coverage proposed in the plan will cover about 92% of the Agency’s direct program expenditures based on the average expenditures over 2006-2007 and 2007-2008, with approximately 65% of the expenditures covered in the next two years.

1. Introduction

The 2010-2011 Parks Canada Evaluation Plan outlines the mandate, organizational structure and resources for evaluation at Parks Canada, the strategy and process employed in developing the Plan, a project schedule for the five-year period from April 2010 to March 2015, and details of individual evaluation projects for the FY 2010-2011, together with the associated resource allocation.

The plan was designed in conformity with requirements of the TB Evaluation Policy (2009), and the implications of the Government’s Federal Accountability Act, as well as Parks Canada’s Evaluation Charter (http://www.pc.gc.ca/docs/pc/rpts/rve-par/47/index_e.asp).

The ultimate goal of evaluation planning is to provide comprehensive evaluation coverage of material program spending over a five-year period in support of the Strategic Review process of the TB Expenditure Management System, consistent with TB Evaluation Policy.

2. Parks Canada Agency

Parks Canada was established as a separate departmental corporation in 1998. The Agency's mandate is to:

“Protect and present nationally significant examples of Canada's natural and cultural heritage, and foster public understanding, appreciation and enjoyment in ways that ensure the ecological and commemorative integrity of these places for present and future generations.”

Parks Canada delivers on its mandate through five major program activities shown in Table 1.

Table 1: Program Activities

Heritage Places Establishment Including the establishment to date of a system of 42 national parks, 3 national marine conservation areas, and the designation of 925 persons, places and events of national historic significance to Canada (e.g., of which 167 national historic sites are administered by the Agency) as well as supporting designation of other heritage places or structures (e.g., federal heritage buildings, heritage rivers, heritage railway, gravesites of prime ministers).
Heritage Resources Conservation Including the maintenance and restoration of ecological integrity in national parks, ensuring the commemorative integrity of national historic sites managed or influenced by Parks Canada; the protection and management of cultural resources under the administration of Parks Canada; and, the ecologically sustainable use of national marine conservation areas. Conservation also includes fulfilling legal responsibilities assigned to Parks Canada by the Species at Risk Act and the Canadian Environmental Assessment Act.
Public Appreciation and Understanding Including outreach to Canadians and selected audiences designed to increase understanding, appreciation, support and engagement in conserving natural and cultural heritage.
Visitor Experience Including the provision of pre- and on-site trip planning information, reception and orientation, interpretation, campgrounds, infrastructure, hiking trails, opportunities for recreational activities, visitor safety and the ongoing post-visit relationship.
Town-Site and Throughway Infrastructure Managing, operating and providing municipal services to five townsite communities within Canada’s national parks (i.e., provision of safe drinking water, snow removal, garbage pick-up and disposal, sewage treatment, road and street maintenance, and fire services to support residents and visitors). It also involves the operation of provincial and inter-provincial highways and waterways that connect communities and pass through national parks and national historic sites.

Responsibility for the Parks Canada Agency rests with the Minister of the Environment. The Parks Canada Chief Executive Officer (CEO) reports directly to the Minister.

National parks and national historic sites are organized into thirty-two geographically based field-units. About 80% of Parks Canada’s work force is based in the field where most of its program expenditures take place. The work of field units is supported by Service Centres located in Halifax, Quebec City, Cornwall/Ottawa and Winnipeg (with small branch offices in Calgary and Vancouver). The Service Centres comprise about 10% of the work force and provide technical and professional services to field units (e.g., science, research, design services). National Office, with less than 10% of the employee base, consists of five directorates (National Parks, National Historic Sites, Strategy and Plans, Human Resources and External Relations and Visitor Experience) who provide legislative, operational policy, planning, program direction, financial management, and human resources functions and services.

3. The Evaluation Function

Applicable Policies and Professional Standards

The evaluation function at Parks Canada adheres to the Evaluation Policies, directives, standards and guidelines of the Government of Canada. In 2007-2008 a new charter for the evaluation function was approved consistent with the then draft Evaluation Policy, directives and standards.

Mission and Services Offered

For evaluation the mission is:

To contribute to the achievement of Parks Canada's mandate by providing the Chief Executive Officer with evidence-based, credible, neutral and timely information on the ongoing relevance, results, and value of policies and programs, alternative ways of achieving expected results, and program design improvements.

Services include:

  • Evaluation frameworks related to programs, activities or initiatives;
  • Evaluations of programs, policies and functions; and
  • Special projects and the provision of advice, as required, on performance measures, targets and information systems.

Evaluations are preceded by the development of evaluation frameworks that describe the elements of a program or sub-activity, its logic and systems for monitoring relevance and performance. These are listed as separate projects in the Agency’s evaluation plan (see below).

Follow-up on Management Responses

The evaluation cycle includes a systematic follow-up on the management responses six months after the final approval of the reports by the Evaluation Committee. Managers are requested, by e-mail from the CEO, to complete a template that provides a status report on action plans in response to evaluation recommendations in their area of responsibility. The template is returned directly to the Chief Audit and Evaluation Executive, who reports regularly to the Committee. The process continues for five-years or until all actions plans are reported to be complete.

Governance

The Agency’s Evaluation Committee is chaired by the Chief Executive Officer (CEO) and composed of six senior managers (i.e., Directors General, Chief Administrative Officer) who are appointed for two-year terms by the CEO. The committee approved a revised terms of reference in December 2009.

The Evaluation Committee is responsible for reviewing and providing advice or recommendations to the CEO on:

  • Evaluation Function and Products: the Agency’s Evaluation Charter, the rolling 5-year Evaluation Plan, the adequacy and neutrality of resources allocated to the evaluation function, the performance of the evaluation function, key elements of an evaluation product lifecycle such as terms of reference, scoping documents, evaluation reports, management responses and action plans including following-up to ensure action plans are implemented.
  • Performance Management Framework: and the adequacy of resources allocated to performance measurement in support of evaluation activities, and recommend to the CEO changes or improvements to the framework and an adequate level of resources for these activities.

Office of Internal Audit and Evaluation (OIAE)

The organizational chart for the evaluation function is shown below along with planned spending for 2010-2011.

organizational chart for the evaluation function

Table 2: Planned Evaluation Spending for 2010-2011

  CAEE Office Evaluation Total
FTE 1 5 6
Salary 24,000 393,000 417,000
Benefits     83,401
Accommodations     54,210
Sub Total     554,611
O&M For Staff Support 5,000 70,000 75,000
O&M for Professional Services and Travel   140,000 140,000
Sub Total O&M 5,000 210,000 215,000
Grand Total 29,000 603,000 769,611
CAEE Office FTEs includes .5 of the FTE for the Chief Audit and Evaluation Executive and for the Executive Assistant. It only includes the salary, and associated benefit/accommodations for the Executive Assistant position, as the salary for the CAEE is administered centrally for all executive level employees in the Agency.

Given a completely staffed function, it is estimated that the OIAE would have the capacity to conduct approximately two to three risk-based evaluations (i.e., an average or typical project is 2,200 hours). The actual number of projects undertaken in any one year may differ from this estimate to the extent that actual project hours differ from average project hours. Details of the calculations involved in the estimate are shown in Appendix 1.

4. Evaluation Universe and Planning Considerations

The key driver of evaluation planning is the requirement in the TB Evaluation Policy to provide comprehensive coverage of direct program spending (i.e., the program activities, sub activities and sub sub-activities in the PAA) over a five-year period in support of the Strategic Review process. The following information was considered in evaluation planning:

a) Agency Renewal

In December 2007 the Agency began a project directed toward renewal of its programs in response to a number of external drivers for change (e.g., changing demographics, changing technology, changes in leisure patterns, increased urbanization and increased national and international competition for tourist visits).

The foundation for the Agency Renewal was the development and communication, starting in January 2009, of a case for change and a new Vision Statement. The Agency is also renewing the Parks Canada brand designed to communicate the vision outside the Agency and increase the public’s appreciation of the Agency and its programs. It includes definition of brand essence, positioning and a brand identity statement and associated implementation guidelines (i.e., links to PA 3).

External Relations and Visitor Experience program realignment has been undertaken to equip field units with the support and capacity required to create enhanced relevance and connection to place among Canadians. This work included creation of separate visitor experience and external relations organizational structures and additional specialized capacity particularly in urban outreach, internet web content, and stakeholder and partner relations. Much of this work is targeted for completion by April 2010 (i.e., links to PA 3 and 4)

National Historic Sites program renewal has focused on developing a strategy that will make the Agency’s national historic sites more known to, appreciated by and responsive to the needs of, and connected with Canadians. An initial situation analysis has been completed focusing on five key areas and an integrated renewal strategy has been developed (i.e., links to PA 2)

In May, 2008 the Government of Canada announced improvements to law enforcement capabilities in Canada’s National Parks and authorized PC to arm up to 100 law enforcement. The new program was developed and partially staffed to begin operating in 2009-2010. There is an approved RMAF with evaluation commitments for this program (i.e., links to PA 2).

The Agency has also development of a more coherent approach to prevention activities (i.e., preventing incidents before they occur and resolve incidents when they do occur safely and effectively). The approach is now embedded in the new Parks Canada Service Program (service standards and prevention guidelines). Prevention activities cut across the PAA structure of the Agency. This program was developed concurrently and in coordination with the new law program and has involved training over 100 trainers who subsequently trained over 3,500 front line employees prior to the 2009-2010 operating season.

Changes in law enforcement also led to undertaking renewal of the resource conservation function (i.e., accountability framework,, roles, responsibilities, accountabilities organizational models, work descriptions related to all aspect of natural resource conservation at the national office, service centre and field level.) (i.e., links to PA-2).

In 2009-2010, the Agency also undertook a review and renewal of its top governance structures (i.e., the roles, responsibilities, and operating procedures of its executive committee and related sub-committees and support structures). As a result a new governance regime will be introduced starting in April 2010.

In some cases, these change priorities are well advanced (e.g., vision, brand, law enforcement) while in others the change effort is only beginning (e.g., renewal of the NHS program). The OIAE has consulted with management on these initiatives and taken them into account in planning.

b) Agency’s 2010-11 Corporate Risk Profile

In 2009-2010 the Agency updated its risk profile (see Appendix 2 for details). The four major risk areas were:

  • Competitive Position: Parks Canada’s service and experience offer may be less attractive or of less interest to Canadians in comparison to other parks and cultural attractions and/or leisure activities.
  • Environmental Forces: Parks Canada may not be able to adapt effectively or quickly enough to environmental forces, such as climate change, biodiversity loss, and exotic/invasive species, which may hinder the Agency’s ability to maintain or improve overall EI in national parks.
  • Delivery/Management of Infrastructure Projects: Failure to deliver and/or effectively manage infrastructure projects may weaken the credibility of Parks Canada within government and with the Canadian public.
  • Information Management: Failure to identify, capture, manage and report pertinent data and information may hinder the ability to effectively manage all program areas and meet legal requirements.

Some of these risk areas cut across many of the program activities (i.e., information management) while others relate to specific programs particularly the Conservation and Visitor Experience program activities.

c) Status of Projects in 2009-2010 Plan

Evaluation Frameworks

Frameworks document the relevant goals and objectives of the programs or sub-activities in the PAA as well as providing details on resources, activities, outputs and reach. Based on this specific evaluation questions, methods, timing and a work plan for one or more evaluations are developed.

Carried Over From 2008-2009 Plan

  • Framework for Visitor Experience Program Activity approved in July 2009. 
  • Framework for Through Highways Management approved in July 2009.

Not In 2009-2010 Plan

  • Framework for Trans-Canada Highway Twinning Project, was not part of the 2009-2010 evaluation plan but prepared as a condition of receiving new funding (i.e., $130M over five years) to complete the Twinning of the Trans Canada Highway (TCH) through Banff National Park. The Agency will undertake a summative evaluation of the whole project starting in 2014 for completion by March 2015. This replaces a number of evaluation requirements that had accumulated as the Agency received funds to undertake segments of the project.

Frameworks to be completed in 2009-2010

  • Framework for National Park Establishment and Expansion: will be tabled at evaluation committee May 2010. 
  • Evaluation Framework for National Parks Conservation: will be tabled at evaluation committee May 2010.

Evaluations

  • Evaluation of Parks Canada’s Asset Management Program: This project, examining the relevance and performance of the Agency’s Asset Management Program, was launched in May 2006 and included in the last three evaluation plans. The evaluation was approved in August 2009 and is available on the Parks Canada web site (www.pc.gc.ca/docs/pc/rpts/rve-par/58/index_e.asp).
  • Evaluation of Through Highways Management: Through highways are national or provincial numbered highways that pass through a national park or national historic site administered by Parks Canada. The evaluation assessed the sub-activity’s relevance, performance, design, and delivery. The report will be tabled in May 2010. 
  • Evaluation of the Visitor Service Offer: This evaluation covers three sub-activities and a portion of a forth in the PPA. The visitor service offer includes the provision of on-site reception and orientation, campgrounds, accommodations, other infrastructure, hiking trails, opportunities for recreational activities, and maintaining ongoing post-visit relationships at National Parks, National Marine Conservation Areas and National Historic Sites. The evaluation began in September 2009 and is scheduled to be completed by December 2010.

d) Past Evaluation Coverage and Work of Other Assurance Providers

Table 3 shows past evaluation work within the Agency and relevant work of the Office of the Auditor General and the Commissionaire of the Environment and Sustainable Development against elements of the evaluation universe.

e) TBS Directed Work

The TB Evaluation Policy calls for the creation of a Government of Canada Evaluation Plan to be supported by evaluation activities within departments. It is not known at the time of the planning whether TB would require specific evaluation work.

f) Planned Work of External Assurance Providers

The Office of the Auditor General conducts review level assurance work each year on the fairness and reliability of the performance information in the Agency’s Annual Performance Report. The OIAE has taken this assurance work into account in planning its priorities.

Table 3: Past Coverage of Program Activity Elements

Program Activities and Sub-Activities 2008-2009 Expenditures Parks Canada Evaluations January 2005 to March 2010 Work of External Assurance Providers
  Heritage Places Establishment 33,593    
1 National Park Establishment and Expansion 24,178    
2 National Historic Sites Designation (Persons, Places, Events) 4,154    
3 National Marine Conservation Area Establishment 3,680    
4 Other Heritage Places Designations 1,581    
  Heritage Resources Conservation 216,823    
5 National Parks Conservation 125,850   CESD Chapter 2—
Ecological Integrity in Canada's National Parks - September 2005
6 Species at Risk   Formative Evaluation of Federal Species at Risk Programs-July 2006
Contributing to a Environment Canada lead Evaluation of the Habitat Stewardship Component of the Species at Risk Program (2008-2009)
CESD Chapter 5—
Ecosystems—Protection of Species at Risk – March 2008
7 National Historic Sites Conservation 59,786   AG Chapter 2—
The Conservation of Federal Built Heritage
Feburary 2007
8 National Marine Conservation Areas Sustainability 2,193 Evaluation of Parks Canada's Phase One of Oceans Action Plan -June 2007  
9 Other Heritage Places Conservation 28,859    
10 National Historic Sites Cost-Sharing 134 Evaluation of Issues Related to the National Historic Sites of Canada Cost-Sharing Program -August 2008  
11 Commercial Heritage Properties Incentive Fund   Formative Evaluation of the Commercial Heritage Properties Incentive Fund (CHPIF) -January 2007  
12 Historic Places Initiative   Formative Evaluation of the Historic Places Initiative -March 2005  
  Public Appreciation and Understanding 40,126    
13 Public Outreach and External Communication 26,825 National Performance and Evaluation Framework for Engaging Canadians: External Communications at Parks Canada– February 2005
Formative Evaluation of Engaging Canadians External Communications Strategy -September 2006
CESD Chapter 2—Ecological Integrity in Canada's National Parks - September 2005
14 Stakeholder and Partner Engagment 13,301    
  Visitor Experience 283,105    
15 Marketing and Promotion 19,952    
16 National Parks Interpretation 18,046    
17 National Parks Visitor Activities and Services 129,506    
18 Public Safety 11,224 Evaluation of Parks Canada’s Public Safety Program-February 2005  
19 National Historic Sites Interpretation 40,796    
20 National Historic Sites Visitor Activities and Services 60,864    
21 National Marine Conservation Areas Interpretation 1,470    
22 National Marine Conservation Areas Visitor Activities and Services 1,247    
  Town-Site and Throughway Infrastructure 110,148    
23 Townsite Management 16,604    
24 Through Highways Management 87,618    
25 Through Waterways Management 5,927    
    683,795    
In 2009-2010 the Evaluation of the Parks Canada Asset Management Program was approved. Assets are a key element in delivery of the Heritage Resources Conservation, Visitor Experience and Town-site and Through Infrastructure Programs and as such this evaluation contributes to coverage of these program activities.

g) Considerations Arising from TB Submissions, RMAFs and Budget 2009

Currently Parks Canada has evaluation commitments resulting from TB submissions and/or RMAFs related to several multi-department programs or initiatives and three of its own programs. The various commitments are listed in Appendix 3 and reflected in the five-year evaluation plan shown below.

Individually the initiatives and funds covered by these evaluation activities represent from less than 1% to about 5% of the Agency’s spending in any one year. Therefore, while important to meet commitments to evaluate transfer payment programs as specified in the Federal Accountability Act, or to demonstrate results related to particular initiatives, they do not provide coverage of most of the Agency’s core programming activities.

Budget 2009 provided the Agency with $75M for infrastructure projects related to the Visitor Experience Program and an additional $75M for addressing concerns related to built cultural heritage (NHS Conservation) including $8M to new funding for the NHS Cost-Share Program. Both of these amounts are provided over two years. The impacts of these new resources will be considered in the planned evaluations of these programs. As noted, budget 2009 also provided additional funds for completing the twinning of the TransCanada Highway in Banff National Park (i.e., $130M), requiring the completion of the Evaluation Framework for this project.

h) Evaluation Priority Assessment and Management Priorities

Many of the preceding factors are combined in conducting a formal assessment of evaluation priorities. The basis of evaluation priorities setting is the PAA sub-activities and sub sub-activities with the addition of a few elements and activities (e.g., law enforcement, the General Class Contribution Program) for which there are specific evaluation commitments. Prioritization of sub-activities in the universe involves assessing each element on eight dimensions, each with a three point scale (i.e., high, medium and low). The dimensions are modified from those suggested in the TB Guide to Evaluation Planning. The total score of an element is based on the overall evaluation priority of the sub-activity with higher scores reflecting a higher priority.

The dimensions on which elements are assessed are:

  1. Existing Commitments to Conduct an Evaluation
  2. Materiality (i.e., the amount of spending on a sub-activity)
  3. Links to Corporate Risk Profile
  4. Completeness of Performance Framework
  5. Intended Direct Reach
  6. Degree of Direct Control Over Program Outcomes
  7. Importance of Health and Safety Considerations in Program Delivery
  8. Public Interest and Sensitivity

Scores were determined by the evaluation function and then discussed and validated with a committee of senior management (i.e., 12 ADM and DG equivalents in the Agency) in February and March 2010.

In some cases, sub-activities in the universe have been combined in the prioritization exercise (e.g., the sub-activities related to visitor services and activities or to interpretation across the three systems NPs, NMCAs and NHSs). Actual evaluation frameworks and subsequent evaluation studies may cover a portion of a sub-activity, a whole sub-activity or more than one sub-activity depending on logical and practical considerations in conducting studies (e.g., the NMCA system is small so it makes practical sense to build a single framework for both the establishment/expansion and conservation sub-activities and conduct a single evaluation).

More details on the rated dimensions and the priority ratings for elements of the universe are shown in Appendix 4. Based on the assessment, the highest priorities for evaluation are sub-activities related to the visitor service offer and through highways management (for which evaluations are already underway). The sub-activities related to Stakeholder and Partner Engagement and Outreach Education and External Communications are also high priorities. Intermediate level priorities include sub-activities related to on-site heritage interpretation, national parks conservation (including SARA and law enforcement), through waterways management, and public safety. Lower priorities activities include most of the establishment activities, conservation related to other heritage programs and the NMCA program, townsite management, and the Agency’s grant and contribution programs.

5. Planned Projects

Based on the factors reviewed, the five-year evaluation plan shown in Table 4 is proposed with the development of evaluation frameworks for program activities/sub-activities/sub sub-activities shown in yellow and evaluations shown in green. Table 5 shows more detail on the projects to be completed in 2010-2011, planned timing, and associated resource requirements.

Extent and Limitations of Coverage

Evaluations of the lower priority sub-activities identified in the priority assessment exercise noted above are not scheduled except where required by policy or TB commitments (e.g. evaluations of grants and contribution programs). Elements not covered in the five-year plan amount to just over 8% of the Agency’s average spending between April 2006 and March 2008. Particular evaluation projects may not cover all components of a sub-activity or may combine and split sub-activities for evaluation projects depending on logical and practical considerations (e.g., visitor services and activities elements in three sub-activities are evaluated together for reasons of economy and efficiency; the complex and heterogeneous nature of sub-activities such as outreach education and external communications may warrant more than one evaluation).

The evaluation plan for the 5-year cycle is determined by first scheduling projects with TB commitments; higher priority projects are then scheduled early in the 5-year schedule. In allocating projects across the 5-year planning span, the following considerations were also taken into account: balancing planned work with the evaluation unit’s annual capacity for project work;[1] ensuring that a framework, if required, is developed prior to the commencement of the corresponding evaluation project; and, if possible, minimizing the time lapse between the framework development and the subsequent evaluation study. The proposed sequencing of evaluation projects is also influenced by discussion at evaluation committee.

Table 4: 2010-2011 Evaluation 5-year Plan

Activities (In Black Rows)
Sub-Sub-Activities (indented)
% Of Expenditures Planned Evaluation Work
2006-
2007
2007-
2008
2008-
2009
2010-
2011
2011-
2012
2012-
2013
2013-
2014
2014-
2015
Heritage Places Establish-
ment
3.4   3.1   4.9                              
National Park Establish-
ment and Expansion
2.4   1.6   3.5     Evalua-
tion**
             
National Marine Conser-
vation Area Establish-
ment
0.0   0.5   0.6               Frame-
work
Evalua-
tion
National Historic Sites Desig-
nation (Persons, Places, Events)
0.8   0.9   0.5                      
Other Heritage Places Desig-
nations
0.2   0.2   0.2                      
Heritage Resources Conser-
vation
32.1   31.8   31.7                      
National Parks Conser-
vation
21   20.4   18.4       Evalua-
tion
           
Species at Risk             Evalua-
tion
               
Law Enforce-
ment
                        Evalua-
tion
   
National Marine Conser-
vations Areas Sustain-
ability
0.2   0.3   8.7               Frame-
work
Evalua-
tion
National Historic Sites Conser-
vation
8.2   8.6   30.0     Frame-
work
          Evalua-
tion
Other Heritage Places Conser-
vation
2.6   2.4   4.2                      
National Historic Sites Cost-
Sharing
    0.1   0.0       Evalua-
tion
           
Public Appre-
ciation and Under-
standing
6.6   5.3   5.9                      
Outreach Education And External Communi-
cations
        3.9       Frame-
work
    Evalua-
tion
   
Stake-
holder And Partner Engage-
ment
        1.9              
Visitor Expe-
rience
39.6   45.2   41.4                      
Marketing and Promotion     2.4   2.9                      
National Parks Inter-
pretation
4.1 11.6 3.1 9.0 2.6 8.8         Evalua-
tion*
       
National Historic Sites Inter-
pretation
7.3 5.8 6.0                
National Marine Conser-
vation Areas Inter-
pretation
0.2 0.1 0.2                
National Parks Visitor Activities and Services 19.3 28.0 20.5 33.7 18.9 28.0 Eva-
lua-
tion*
                 
National Historic Sites Visitor Activities and Services 8.5 12.9 8.9                  
National Marine Conser-
vation Areas Visitor Activities and Services
0.2 0.3 0.2                  
Public Safety     1.5   1.6           Evalua-
tion
       
Town-Site and Through-
way Infra-
structure
16   14.7   16.1                      
Townsite Manage-
ment
2.1   2.6   2.4                      
Through Highways Manage-
ment
12.4   10.9   12.8               Evalua-
tion TCH
   
Through Waterways Manage-
ment
1.5   1.2   0.9   Evalua-
tion
               
Other                                
General Class Contri-
butions Program
            Evalua-
tion
               
Prevention                             Evalua-
tion
Total Agency Expen-
ditures (in millions)
$603 $634 $684     100
%
             
Percentage of Direct Expenditures Covered
by Evaluations Completed in Year
43% 22% 12% 6% 9%
* Will include elements of Marketing and Promotion, providing evaluation coverage for this sub-activity
**Will address the cases relevant to the interdepartmental Evaluation of the Establishing Federal Protected Areas in the NWT

Table 5: Evaluation Project Scheduling for 2010-2011

Topic Type Description Planned or Actual Dates Resources Required
2010-2011
Planned
2009-10
Start
date
Comple-
tion of the field-
work
Comple-
tion of
report
Date of approval by Commit-
tee
Approx hours O&M
Carried Over From 2009-2010                
National Park Establish-
ment and Expansion
Framework Completion of a description of the program sub-activity and objectives, evaluation issues and methods for a subsequent evaluation of relevance and perfor-
mance.
Y April 2009 Feb 2010 April 2010 May 2010 0 0K
National Parks Conser-
vation
Framework Completion of a description of the program sub-activity and objectives, evaluation issues and methods for a subsequent evaluation of relevance and perfor-
mance.
Y May 2009 March 2010 April 2010 May 2010 100 0K
Visitor Service Offer Evaluation A summative evaluation of relevance and perfor-
mance of the visitor services component of the Visitor Experience Program Activity.
Y July 2009 Aug 2010 Oct 2010 Dec 2010 1000 92K
Through Highways Manage-
ment
Evaluation Completion of an evaluation of the relevance and perfor-
mance of this sub-activity.
Y July 2009 Jan 2010 April 2010 May 2010 200 0K
New in 2010-11                
National Historic Sites Conser-
vation
Framework A description of the program activity and objectives, evaluation issues and methods for a subsequent evaluation of relevance and perfor-
mance.
Y Sept 2010 June 2011 July 2011 Sept 2011 500 0K
Species at Risk Horizontal Evaluation Inter-
depart-
mental evaluation of the programs and activities in support of the Species at Risk Act.
Y June 2010 Oct 2010 Jan 2011 March 2011 200 20K
National Park Establish-
ment and Expansion
Evaluation An evaluation of the relevance and perfor-
mance of this sub-activity.
N Sept 2010 June 2011 July 2011 Sept 2011 1000 15K
Through Waterways Manage-
ment
Evaluation An evaluation of the relevance and perfor-
mance of this sub-activity. A brief scoping document will precede the evaluation, describing the sub-activity and the evaluation’s issues and scope.
N April 2010 February 2011 April 2011 May 2011 1700 50K
General Class Contri-
bution Program
Evaluation An evaluation of the relevance and perfor-
mance of the program.
Y April 2010 Aug 2010 Nov 2010 Dec 2010 1000 18K
Total 5700 195K
* Approximate hours reflect required effort. Projects that extend beyond a fiscal year will require more effort than shown in the table. Staff or consultants may perform the work. O&M required includes costs of professional services and/or costs of staff travel.

6. Appendices

  1. Assumptions in the Calculation of Evaluation Capacity
  2. Corporate Risk Profile 2010-2011
  3. Agency RMAF Evaluation Commitments
  4. Evaluation Priority Assessment of Elements of the PAA Universe

1. Assumptions in the Calculation of Evaluation Capacity

a.
Hours Available for Work Per FTE Hours Available
52 week/year * 5day week *7.5 hour/days 1950.0
Average days holiday (20 days) -150.0
Average days sick (5 days) - 37.5
Average days training (5 days) - 37.5
Total Work Time Available 1725.0*
* Equivalent to 230 working days per evaluator
b.
Hours Available for Evaluation Work: On average it is assumed that one hour per working day is allocated to breaks (i.e., 230 hours per evaluator per year), leaving 1,495 hours of actual working time. It is further assumed that approximately 20% of the actual working time per evaluator (i.e., 299 hours per year) is allocated to unit administration, including planning, meetings, supporting internal systems, support to the evaluation committee, human resources issues, follow-up on the status of previous evaluation recommendations, etc. The remaining time is allocated to evaluation work in point C, and to risk based evaluation work.
c.

Types of Evaluation Work: Available work hours are allocated to the following tasks:

  • TBS Directed Evaluation Requirements

    • TB Evaluation Policy envisions a Government-Wide Evaluation Plan that could dictate the scope and timing of Agency evaluation work.
    • This category does not include evaluation work required as part of TB submissions or RMAF/RBAF commitments.

  • Special Requests

    • Periodic special requests by management outside the approved evaluation plan are typical. Pervious examples include the extensive work carried out on quantitative risk assessment associated with warden law enforcement duties.

  • Consulting/Advice/Coordination

    • Consulting and providing advice related to evaluation or performance measurement is a normal part of the activities of the OIAE.
d.
O&M Dollars Available for Contracting: Additional capacity can be purchased from contracted professionals based on available O&M budgets. An average per diem of $1,200 per day or $160 per hour is used for this calculation. For 2010-2011, the O&M budget for professional services is $100K.
e.
The resource consumption of the average evaluation project: The average number of hours consumed by an evaluation project is affected by many factors, including its scope and complexity as well as the skills of the personnel performing the work. Based on a review of prior experience with contracted projects and estimates of internal staff time devoted to projects as well as the expectation that future evaluation projects will need to be more complex given the TB policy directions, typical values of 2,200 hours per evaluation project were identified.

2010-2011 Capacity For Evaluation

A) Resources CAEE Office Evalu-
ation
  Total  
FTEs 1 5 6
Hours Available for Work (Less vacation, sick and training days, and 1 hour per working day for breaks leaving 1,495 hours per FTE) 1,495 7,475 8,970
Professional Service Hours Purchased (O&M for professional Services/$160 per hour/$100K budget)   625 625
Total Hours Available 1,495 8,100 9,595
B) Demands      
Administration i.e., (80% of CAEE, 100% of Admin. Assistant and 20% of other staff hours are allocated for activities such as planning, meetings, work on internal systems, support to the evaluation committee, human resources, quality control, follow-up on previous recommendations, etc) -1,345 -1,495 -2,840
TBS Directed Evaluations (none in 2009-2010)     0
Special Requests (.25% of one FTE) -10 -374 -384
Consulting/Advice/Coordination (40% of one FTE) -140 -598 -738
Residual Hours for Risk Based Projects 0 5,633 5,633
C) Project Capacity      
Number of Typical Projects (2,200 hours per project)     2.6
2. Corporate Risk Profile 2010-2011
Risk Category and Label Description Relevant PAs Mitigation OIAE Work
Competitive Position Parks Canada’s service and experience offer may be less attractive or of less interest to Canadians in comparison to other parks and cultural attractions and/or leisure activities. 4    
Environmental Forces Parks Canada may not be able to adapt effectively or quickly enough to environmental forces, such as climate change, biodiversity loss, and exotic/invasive species, which may hinder the Agency’s ability to maintain or improve overall EI in national parks. 1, 2 and 4    
Delivery/Management Of Infrastructure Projects Failure to deliver and/or effectively manage infrastructure projects may weaken the credibility of Parks Canada within government and with the Canadian public. 2, 4    
Information Management Failures to identify, capture, manage and report pertinent data and information may hinder the ability to effectively manage all program areas and meet legal requirements. 1-5 and ISD Records Disposition Authority; Enterprise Information Committee; Information Management and ATIP and Privacy Awareness training, Phase 1. Audit of Information Management
Recruitment And Retention Failure to recruit and retain qualified employees may lead to challenges in delivery of all programs and support functions. 1-5 and ISD    
Public Support Support from local communities, stakeholders, NGOs, and the Canadian public may not exist or be insufficient to advance Parks Canada’s programs 1-5    
Aboriginal Support Support from Aboriginal peoples may diminish and become insufficient to advance Parks Canada’s programs 1-5    
Adoption Of Technology Use of information technology in Parks Canada’s service offer may not meet employee, visitor and outreach expectations. 2-4    
Inter-Governmental Collaboration Collaboration with other federal departments, provinces, territories, and municipalities may be insufficient to advance Parks Canada’s programs. 1-5    
Development Pressures Development pressures may limit opportunities for NP/NMCA establishment, NHS commemoration, maintenance of EI and CI, and development of connection to place. 1, 2, 4    
Partner Capacity Parks Canada’s partners may not have capacity to assist in protection, presentation and promotion of Parks Canada administered places.      
Disasters Disasters may impair or destroy critical infrastructure and/or assets of national historic significance, or lead to serious injury or loss of life. 1, 2, 4 and 5   Audit Business Continuity Planning
PA-1: Heritage Places Establishment, PA-2: Heritage Resources Conservation, PA-3: Public Appreciation and Understanding
PA-4: Visitor Experience, PA-5: Town-Site and Throughway Infrastructure, ISD: Internal Service Delivery
3. Agency RMAF Evaluation Commitments
  2010-2011 2011-2012 2012-2013 2013-2014 2014-2015
Horizontal Evalu-
ations[2]
Evaluation of the Species at Risk Program: Environment Canada, Fisheries and Oceans and the Agency are the lead federal organizations for the Species at Risk Program. Parks Canada received a total of $10.6M between 2007/08 and 2011/12 from this initiative and expects to receive $6.8M per year thereafter, representing about 10% of the overall available funding. The RMAF for this initiative commits to a summative evaluation in 2010-2011. Evaluation of the Establishing Federal Protected Areas in the NWT: Indian and Northern Affairs Canada, Environment Canada and Parks Canada are partners working together to advance the Northwest Territories Protected Areas Strategy. Parks Canada will receive $8.05M between April 2008 and March 2013 (i.e., roughly a third of investment) to assist in establishing a national park in the East Arm of Great Slave Lake, and to help develop and make operational of the Sahoyue and Ehdacho National Historic Site of Canada. A summative evaluation of the initiative will be lead by DIAND in early 2012. Evaluation of the Health of the Oceans Initiatives: The Agency has or will receive $6.25M between April 2007 and March 2012 to help develop and implement a federal marine areas strategy in concert with Fisheries and Oceans and Environment Canada and to conduct a feasibility study for establishing a national marine conservation area in Lancaster Sound.    
Parks Canada Evaluation of the Parks Canada Agency’s General Class Contributions Program: The GCCP is not a program within itself but rather a funding authority that is available to Agency managers to support activities or projects related to National Parks, Marine Conservation Areas, Historic Sites and Canals. Approximately $2.2M in funding is allocated per year. The program was renewed in April 2006 and continues until March 2011. A proposed Results Reporting Capacity Check for 2007-2008 was not completed. A summative evaluation is scheduled for 2010-2011 prior to the renewal of the terms and conditions of the program.   Evaluation of the National Historic Sites of Canada Cost-Sharing Program: The program provides matching contribution funds to eligible national historic sites undertaking conservation and presentation work. It was renewed in 2008-2009 with an annual budget of between $2.3 and 3.3M per year over the next five years. The budget has been supplemented in 2009 through 2011 with EAP funding. A summative evaluation is scheduled for 2012-2013. Evaluation of the Law Enforcement Program: The program, involving up to 100-armed law enforcement officers responsible for enforcement of national parks, national marine conservation area and national historic site laws and regulations, but not for criminal code enforcement, was funded and developed in 2008-2009 with on the ground activities commencing in 2009-2010. The program will have start-up costs of $8.5M in 2008-2009 and ongoing costs of $2.3M per year thereafter (i.e., less than one percent of the Agency’s annual spending). A summative evaluation is planned for in 2013-2014. 

Evaluation of the Twinning of the Trans-Canada Highway in Banff National Parks: The recent initiative (i.e., 2004-05 on) involves twining 32 kilometres of the TCH at a total cost of $317M over 10 years. Funding was received through four different TB submissions including portions from the Asia Pacific Gateway and Corridor Initiative Fund and the Gateway and Boarder Crossing Fund.[3] The TB approval of the last $130M in funding (March 2009) provided an exception from the planned horizontal evaluations of the two funds and instead allowed for a summative evaluation of the whole project in 2013-2014.
 
4. Evaluation Priority Assessment of Elements of the PAA Universe
Priority Rating Criteria and Definitions
  4 2 0
1. TB commitments Required in the next 12 to 18 months Required but not in the next 18 months None required
2. Materiality Greater than 10% (approximately 60+ million) 5% to 10% (approximately 31 to 60 million) 5% (approximately 30 million)
3. Links to Corporate Risk Profile Links primarily to high priority corporate risks Links to primarily lower priority corporate risks No links to corporate risks
Activities linked to the four highest corporate risks: competitive position, environmental forces, Delivery/Management Of Infrastructure Projects and Information Management are rated four (see section 2 above). Activities related to other risks in section 2 are rated a moderate and activities not related to the risk profile are rated one.
4. Completeness Of Performance Framework None or few elements of the framework in place Partially complete Complete
A complete framework consists of defined measurable goals and objectives, baseline measures of performance, quantifiable targets with clear time frames for accomplishment of goals and systems to measure and report on progress and goal attainment, and evidence of monitoring and reporting.
5. Extensiveness of Program Reach Extensive reach to communities, stakeholders, NGOs, Aboriginal peoples, and the public. Moderate and/or regional-level reach to communities, stakeholders, NGOs, Aboriginal peoples, and the public. Limited and/or localized reach to communities, stakeholders, NGOs, Aboriginal peoples, and the public.
High intended direct reach is typified by activities related to building awareness and understanding the Agency and its mandate and promotion and marketing Parks Canada sites as well as the visitor experience program which are intended to reach millions of Canadians and international visitors. Low reach is typified by sub-programs in the Other Heritage Places Establishment Sub-Activity such as Grave Sites of Prime Ministers which is effectively targeted at a few families of former prime ministers whose grave sites are not yet formally commemorated. When the target reach of a program are organizations, or provinces, as in park establishment for example, we count reach as the number of groups targeted and not the size of the constituencies represented by these groups. Most program activities have ultimate beneficiaries i.e., Canadians as a whole, who are not counted as the program or sub-activity reach.
6. Degree of Direct Control Over Outcomes Low Direct Control Moderate Direct Control High Direct Control
Low control over outcomes is exemplified by the national park and national marine conservation areas establishment and expansion sub-activities, which require extensive consultation and negotiations over many years with dozens of different stakeholders, who differ in their capacities and interests, and have the capability to block a particular establishment process. More control is available over a contribution program where the Agency, with TB agreement, has set the terms and conditions for receiving funding and evaluates and recommends who will be funded. An intermediate example might be conservation in national parks and NMCAs where the Agency may have a relatively high degree of control over what occurs within the boundaries of the park but is also interested in influencing regional land use practices than impact on the park’s ecological integrity.
7. Importance of
Health and Safety Considerations in Program Delivery
High Level of Consideration of health and safety issues in delivery of a sub-activity. Moderate Level of Consideration of health and safety issues in delivery of a sub-activity. Low Level of Consideration of health and safety issues in delivery of a sub-activity.
Many activities involving visitors require consideration of health and safety issues as a fundamental part of the program delivery. Examples include the potential for human wildlife conflicts in national parks, possibilities of contamination when providing potable water, the potential of accidents on highways managed by the Agency, and the potential for accident or injury when conducting law enforcement or search and rescue activities. We do not assess the nature or quality of management measures to mitigate health and safety issues involved in sub-activity delivery only whether and the extent to which these considerations have been inherent in delivery of the activity.
8. Public Interest and Sensitivity High Moderate Low
Activities which have received recent public or political attention are rated higher (i.e., the lead up to the decision to arm park wardens and the new law enforcement program had extensive media coverage but this has largely abated since the new program began operating). Introduction of new legislation such as the Heritage Lighthouse Act, creates temporary political interest in a particular activity (the Act would protect heritage lighthouses in Canada and is considered part of the Other Heritage Places Designation sub-activity). Some consideration is also given to potential for public or political interest. Many of the health and safety concerns reviewed above have high potential interest should they occur (e.g., the failure of a dam or a potable water system resulting in a significant number of injuries or deaths).

Total Scores Across Eight Rated Dimensions

PAA Elements Priority Rating
(range 0 to 32)
Estimated
O&M required for contracting or travel Hours of evaluator time Estimated Completion Date based on current plan[4]
High Outreach Education And External Communications 16 30k 1000 March 2014
National Parks Visitor Activities and Services 16 175k 2200 Dec 2010
National Historic Sites Visitor Activities and Services
National Marine Conservation Areas Visitor Activities and Services
Through Highways Management[5] 16 20k 2200 March 2014
Species at Risk 14 20k 200 March 2011
Stakeholder And Partner Engagement 14 30k 1000 March 2014
Medium National Parks Conservation 12 100k 2200 March 2012
Law Enforcement 12 60k 2000 March 2014
National Marine Conservation Areas Sustainability 12 10k 1000 March 2015
Marketing and Promotion[6] 12 - - -
National Historic Sites Conservation 12 80k 2200 March 2015
National Parks Interpretation 12 60k 2000 March 2013
National Historic Sites Interpretation
National Marine Conservation Areas Interpretation
Public Safety 12 45k 2000 March 2013
Through Waterways Management 12 45k 2000 March 2013
Low National Park Establishment and Expansion 10 30k 2000 Sept 2011
National Marine Conservation Area Establishment 10 10k 1000 March 2015
National Historic Sites Designation (Persons, Places, Events) 6 - - -
Other Heritage Places Designations 6 - - -
Other Heritage Places Conservation 6 - - -
National Historic Sites
Cost-Sharing
8 60k 1800 March 2012
Town-site Management 8 - - -
General Class Contributions Program 8 18k 1000 Dec 2010

[1] The evaluation unit has an annual capacity of 5633 hours of project work and an anticipated $140,000 in project O&M.

[2] Parks Canada has or will receive $1.3M between April 2007 and March 2012 to provide services and support to the International Polar Year Initiative. The Agency is not considered one of the six lead departments with respect to this initiative. DIAND, is to conduct a formative evaluation for 2008-2009. The Agency does not plan any evaluation work related to this initiative.

[3] Parks Canada is a stakeholder in the Asia Pacific Gateway and Corridor Initiative which provides $591M over eight years to several departments to invest in transportation infrastructure and other projects. Parks Canada received $37M or about 6% of the funds for twinning of a portion of the TCH. A summative evaluation led by Transport Canada is scheduled for 2010-2011. The Gateway and Boarder Crossing Fund provides $2.01B over seven years between April 2007 and March 2014 for both infrastructure and non-infrastructure projects. Parks Canada received $100M from the fund for twinning of the TCH or about 5% of the funds. A summative evaluation is anticipated in 2013-2014.

[4] Expected completion date is the anticipated month of approval by the Parks Canada Evaluation Committee.

[5] Includes 200 hours to complete the Evaluation of Through Highways Management sub-activity in 2010-2011 and 2000 hours to complete the Evaluation of the Twinning of the Trans-Canada Highway in Banff National Park in 2013-2013.

[6] Elements of Marketing and Promotion will be covered during evaluations of other elements of PA4 Visitor Experience.