Parks Canada
Symbol of the Government of Canada

Common menu bar links

Internal Audit and Evaluation Documents

Evaluation of Parks Canada’s General Class Contributions Program

Final Report
November 17, 2010

Report Submitted to the Parks Canada Evaluation Committee: December 16, 2010
Approved by the Agency CEO: January 7, 2011

Office of Internal Audit and Evaluation
Parks Canada

Table of Contents

Executive Summary

1. Introduction

2. Program Description

2.1 Outcomes/Goals
2.2 Eligible Recipients and Project Types
2.3 Funding Criteria
2.4 General Component Application, Approval and Monitoring Processes
2.5 GCCP Cooperating Associations Component
2.6 Expenditures
2.7 GCCP Logic Model

3. Evaluation Design

3.1 Approach and Questions

3.1.1. Methods
3.1.2. Limitations

4. Evaluation Findings

4.1 Relevance
4.2 Effectiveness
4.3 Cost-Effectiveness and Efficiency
4.4 Program Design and Delivery

5. Conclusions and Recommendations

Appendix A: Strategic Outcome and Program Activity Architecture

Appendix B: GCCP Funding Criteria

Appendix C: Cooperative Association Funding Criteria

Appendix D: Evaluation Questions, Expectations, Indicators and Data Sources

Appendix E: Key Documents Reviewed

Appendix F: File Review Template

Appendix G: Projects Selected for Detailed Review

Appendix H: Case Studies of Project Goals, Results, and Reporting Mechanisms

Appendix I: Grant to the International Peace Garden

Printable Version


Note: To read the PDF version you need Adobe Acrobat Reader on your system.

If the Adobe download site is not accessible to you, you can download Acrobat Reader from an accessible page.

If you choose not to use Acrobat Reader you can have the PDF file converted to HTML or ASCII text by using one of the conversion services offered by Adobe.



List of Tables

Table 1: Eligible GCCP Project Types

Table 2: Contribution Expenditures by Year

Table 3: Logic Model for Parks Canada General Class Contributions Program

Table 4: Evaluation Questions

Table 5: Number of Agreements and Planned Spending by Year

Table 6: Number of Agreements and Value of Planned Funding by Recipient Types

Table 7: Number of Agreements and Value of Planned Funding by Agreement Type

Table 8: Relationship Between Recipient Type and Purposes of Agreement

Table 9: Percentage of GCCP expenditures by Program Activity (April 2006 to October 2010)

Table 10: Sources of Other Funding

Table 11: Percentage of Parks Canada Funding for Projects

Executive Summary

The General Class Contributions Program (GCCP) is one of two contribution programs in the Agency. Yearly contribution payments have averaged approximately $4.3M over the last five years. The program represents less than 1% of the Agency’s direct yearly program spending. The evaluation is intended to support the renewal of the current Terms and Conditions (Ts&Cs) which expire at the end of March 2011. If the program is not adequately managed, there are risks that funds will be wasted on projects that are not aligned with and supportive of the Agency’s mandate and overall government priorities and that results will not be achieved, or not achieved efficiently and economically.

The GCCP is not a “program” in the traditional sense but rather functions as a general funding authority available to Agency managers to support projects and infrastructure costs across the full range of the Agency’s program activities rather than within a specific program activity. With the exception of a sub-component of the GCCP dedicated to funding co-operative associations, there is no central source of funding for GCCP agreements. Instead, they are funded from the applicable program or Field Unit’s budget through the reallocation of O&M funding.

Evaluation Issues

Consistent with the requirements of the Treasury Board (TB) Directive on the Evaluation Function, the evaluation addressed:

  1. Relevance: Does the GCCP serve an important need or function and is it aligned with Agency and government roles and priorities?
  2. Effectiveness: Have GCCP funded projects been implemented and produced the outputs and outcomes intended?
  3. Cost-Effectiveness (efficiency and economy): Is the process timely and effective and are administrative costs of the program efficient relative to outputs and outcomes produced?
  4. Design and Delivery: Are roles, responsibilities, program structures and processes relevant, clear, and proportional to risk and is the program accessible, fair and understandable?

Methodology

Data from multiple lines of evidence was collected for the evaluation. These included document review, a review of all valid project approval files in National Office (n=276), an in-depth review of performance information for a sample of agreements (n=43), key informant interviews with those involved in central program administration (n=8), as well as GCCP project administrators within the Agency (n=24) and GCCP funding recipients (n=21), and short case studies of selected GCCP projects and their results (n=7). Although projects selected for in-depth review and key informant interviews are not necessarily statistically representative of the entire population of GCCP projects, the use of multiple methods and compensating analysis provides a reasonable basis for drawing conclusions about the relevance and performance of the GCCP.

Findings

The evaluation found strong evidence that the GCCP continues to be a relevant mechanism for delivery of the Agency’s mandate and priorities and is aligned with overall government objectives. The projects funded can be linked to various Agency priorities, objectives and programs. It is seen to be useful and offer advantages over other funding mechanisms by both Agency staff and recipients. The mechanism is widely used in the Agency with an average of 54 agreements per year over the last five years and about $21M in expenditures as of October 2010. There is little sense that it duplicates other programs. There are no other G&C programs that cover the range of eligible recipients and types of agreements.

There is sufficient evidence to conclude that GCCP funded projects are largely implemented as intended and produce, or are likely to produce, the intended outputs defined in contribution agreements. In most cases, reporting on the project goes beyond simply demonstrating that outputs are produced and includes other information on the reach or quality of the outputs or other relevant performance information (i.e., volunteer hours contributed, sales and revenue generated).

GCCP projects are generally considered to be economical and provide good value for the money invested, particularly given the amount of funds provided from other sources (i.e., an estimated $46.5M based on an incomplete record of other sources of funds for the period under review), allowing the Agency to obtain outputs and results that it could not achieve on its own. The issues or challenges identified in the evaluation pertain to the program design and delivery including:

  • A lack of good data regarding the total administrative costs of the program which prevents a rigorous analysis of the efficiency and economy of the GCCP
  • Lack of clarity in roles and responsibilities for overall program monitoring and performance reporting
  • Inconsistent file and data management practices in National Office for the overall project files, though some improvements in data capture have recently occurred, and a lack of consistent and consolidated reporting of project outputs and outcomes
  • Concerns about the timeliness of project approval and a lack of service standards that would clarify expected timeliness for various steps of the process
  • A number of specific suggestions to expand or change the Ts&Cs, provide training or templates, or clarify administrative issues
  • Questions about the continued relevance of the central funding and application and evaluation processes for the cooperative association component of the program.

Our recommendations follow from these issues and challenges.

Recommendations

Recommendation 1

The CFO and DG ERVE should jointly review and provide a reasonable basis for determining the administrative costs of the program in order to be able to assess program efficiency (i.e., administrative costs per dollar of contribution provided).

Estimates of administrative costs do not require exact time tracking to identify all salary costs allocated to the program. Modeling and estimation based on a sample of average or prototypical agreements of different types that included costs outside of National Office is reasonable.

Management Response
Disagree:
Given the small amount of funding spent on the GCCP, the variety in the amounts and scope of agreements, and the fact that the Agency operates using a decentralized model for its administration, the costs of gathering and reporting on the administrative costs of the GCCP would outweigh the benefits that could be derived from the information.

Recommendation 2

The National Office DGs should clarify the purpose and role of National Office in collecting information on GCCP funded agreements particularly with respect to overall program monitoring and reporting. Once purposes and roles are determined, they should ensure that the systems, personnel, and resources are in place so that the required information to support the National Office function is captured effectively and efficiently and the requirements are clearly understood and communicated.

Central office data capture need not simply duplicate information in files project managers are expected to maintain. For example, rather than collect copies of interim or final performance reports, it may be more efficient to create short templates and have project managers provide summaries of results at the close of a project or on an annual basis.

Management Response
Agree:
National Office DGs will clarify the purpose and role of National Office and project managers in collecting information on GCCP funded agreements and ensure that the required information to support the National Office function is captured effectively and efficiently and that the requirements are clearly understood and communicated. Action to be completed by March 31, 2011.

Recommendation 3

The National Office DGs should review, clarify and communicate expectations on the form and content of final project reporting for GCCP funded projects (i.e., including project managers’ responsibilities to document performance beyond what is provided by recipients) and provide appropriate templates where required consistent with managements’ responsibilities to maintain sufficient performance information to support evaluations of programs.

Management Response
Agree:
This will be addressed as part of the response to Recommendation 2.

Recommendation 4

CFO and DG ERVE should review the Agency’s approach to project approval and at a minimum propose internal service standards for approval by Executive Management Committee that would clearly communicate expectations with respect to the timeliness of the process. In reviewing the project approval approach they should assess the Agency’s level of risk tolerance in this regard and clearly communicate the rationale for the proposed approach.

Management Response
Disagree:
Parks Canada will continue its current policy that all contributions are to be approved by the Chief Executive Officer, as per the Agency's Financial Delegation of Authorities authorized by the Minister. Timely processing will be addressed by producing an annual plan which ensures that approval in principle for an agreement is provided before the beginning of the fiscal year. The Agency will undertake to have all final approvals of contribution agreements secured within one month of the sign-off by originating Director or FU Superintendent.

Recommendation 5

DG ERVE, in consultation with the CFO as necessary, should review the status of the cooperative association component of the GCCP and either provide a rationale for its continued status as a centrally funded and administered component of the program or merge it into the general GCCP.

The recommendation is not intended to imply that the cooperative association component of the GCCP has poorer performance than the general GCCP component. Evidence suggests this is not the case. Rather the focus is on the need for and the cost efficiency of the different administrative approach to this aspect of the program.

Management Response
Agree:
DG ERVE and CFO will review the status of the cooperative association component of the GCCP, including the process by which a contribution to a cooperative association is approved or denied. Action to be completed by March 31, 2011.

1. Introduction

Parks Canada’s mandate is to:

“Protect and present nationally significant examples of Canada's natural and cultural heritage, and foster public understanding, appreciation and enjoyment in ways that ensure the ecological and commemorative integrity of these places for present and future generations.”

The Agency is responsible for three major heritage systems:

  • 42 National Parks of Canada
  • 167 National Historic Sites of Canada (administered by the Agency)
  • 4 National Marine Conservation Areas of Canada

Parks Canada carries out its mandate through five program activities (PA) and twenty sub-activities. The major program activities are heritage places establishment, heritage resources conservation, public appreciation and understanding, visitor experience and the townsite and throughway infrastructure program (see Appendix A).

The focus of the evaluation is the Agency’s General Class Contributions Program (GCCP), one of two contribution programs in the Agency. An evaluation of the GCCP was identified as a commitment in the Agency’s 2010-2011 Evaluation Plan[1] in keeping with Treasury Board (TB), and Agency Evaluation Policies and section 42.1 of the Financial Administration Act, to evaluate all grants and contributions programs over a five-year period. The evaluation is intended to support the renewal of the current Terms and Conditions (Ts&Cs), which expire at the end of March 2011. In evaluation planning the GCCP was rated as a low priority for evaluation work based on factors such as materiality, reach, complexity, and health and safety risks. The program accounts for less than 1% of the Agency’s annual direct program spending.

The Agency’s second contribution program, the National Historic Sites Cost-Sharing Program, is scheduled for evaluation in 2011-2012.[2]

2. Program Description

The GCCP is not a “program” in the traditional sense and is not represented as a specific activity or sub-activity in the Agency’s Program Activity Architecture (PAA), unlike the National Historic Sites Cost-Sharing Program which is a sub-sub activity of the Heritage Resources Conservation Program. The GCCP is defined as “a funding authority that is available to Agency managers when programming needs periodically arise that may be most appropriately met by entering into contribution agreements with appropriate recipients.”

GCCP goals and objectives, eligible recipients and project types and funding criteria are summarized below. The source of funds as well as the application and approval process for cooperative associations (i.e., one class of eligible recipients) is distinct from the processes used for other recipient classes, though cooperative associations may also be funded under the general component. The vast majority of contribution agreements and expenditures are through this other “general” component of the program.

The approved Treasury Board submission for the GCCP Ts&Cs also authorized a $22,700 annual unconditional grant for 2006-07 through 2010-11 to help defray the cost of operating the International Peace Garden (i.e., a garden to commemorate the peaceful co-existence of Canada and the United States that straddles the Manitoba/North Dakota Border and is run by a non-profit organization). This grant has been made by the federal government since the 1930s. The grant is not scoped into the evaluation. Descriptive information and observations regarding the grant are presented in Appendix I.

2.1 Outcomes/Goals

Under the current framework, the GCCP is intended to assist recipients in conducting activities and delivering projects that will support the Parks Canada Agency in fulfilling its mandate to protect and present nationally significant examples of Canada's natural and cultural heritage and foster public understanding, appreciation and enjoyment in ways that ensure their ecological and commemorative integrity for present and future generations. Specific results and outcomes contained in the Ts&Cs include:

  • Canadians recognize, appreciate and are engaged in the values of natural and cultural conservation
  • Stakeholders are engaged in terms of interest and involvement of common objectives towards ecological or cultural integrity
  • Parks Canada managers and stakeholders have access to a better knowledge base for informed decision-making and dialogue on commercial, ecological or Aboriginal issues of mutual interest
  • Visitors are provided opportunities to purchase souvenirs or experience traditional meals;
  • Heritage assets are protected, secured and researched
  • Targeted audiences are educated in such areas as ecology, safety and other issues.

2.2 Eligible Recipients and Project Types

Under the Ts&Cs, eligible recipients include voluntary organizations, management boards, non-profit organizations (including environmental groups and Aboriginal communities), provincial/territorial/municipal/regional governments, and research institutions. International organizations are eligible for contributions for one-time events (e.g., conferences) and knowledge building in fields of mutual interest. The GCCP is not currently intended to provide contributions to for-profit organizations or to provide international organizations with core funding.

The ten types of projects eligible for GCCP funding are presented in Table 1.

Table 1: Eligible GCCP Project Types
Agreement Type Purpose
1) Cooperative Organizations Originally, the provision of seed or interim funding to move the organization to financial sustainability, currently focused on project funding to provide enhanced (value-added) services for visitors (services outside the Agency’s direct delivery mandate)
2) Research Project specific funding for research on and in support of commemorative and ecological integrity issues
3) Education, outreach and/or tourism Project specific funding to promote education, outreach and tourism
4) National Historic Sites Provision of operational support for national historic sites - for sites not owned or administered by the Agency
5) Aboriginal Groups (Land Negotiations, Park/Site Management Planning and Relationship Building) Project specific funding to Aboriginal groups to obtain expertise related to land negotiations, park/site management planning and relationship building
6) Aboriginal Groups (Economic Opportunities and Presentation of Culture) Project specific funding to Aboriginal groups for capacity development for economic opportunities and presentation of culture
7) Management Boards Operational support to management boards
8) Community Relationships/Support Community relationships/project specific support to municipalities within national parks to meet environmental or other standards set by the Agency (e.g., sewage, water treatment plants)
9) Land Assembly/Easements Bordering National Parks Project specific funding for ecological protection to nature conservation groups re land assembly and easements bordering national parks. The lands will not be owned by Parks Canada but are adjacent to national parks and enhance their ecological integrity.
10) Areas of national historic or ecological significance establishment Operational support to not-for-profit organizations to help establish areas of national historic or ecological significance

Source: General Class Contributions Program Terms and Conditions

The size of the eligible population for the GCCP is not known except in the case of specific sub-groups of eligible recipients, such as the number of cooperating associations, management boards or provincial/territorial governments. In general, it can be assumed that there are a large although not unlimited number of potential recipients in other groups of eligible recipients (e.g., researchers, non-profit organizations). In addition, except in limited circumstances, there is no information on how many potential recipients approach the Agency seeking funding (i.e., the number of applications for assistance that are refused).[3]

2.3 Funding Criteria

Section 6 of the GCCP’s Ts&Cs sets out five criteria that should be considered in deciding which projects to fund, though “the relative weighting of criteria and sub-criteria will differ.” General criteria include:

  • Program Delivery Capacity of Applicant
  • Objectives and Intended Results
  • Management Capacity of the Applicant
  • Proposed Budget
  • Project Design.

Additional details on the criteria are shown in Appendix B.

2.4 General Component Application, Approval and Monitoring Processes

The Agency does not advertise the general component of the GCCP to potential recipients. Potential projects may be identified from unsolicited proposals from outside the Agency (e.g., a proposal to do research on some aspect of ecological or commemorative integrity where the proponent is likely not aware of the GCCP as a funding mechanism), identified within the Agency by a manager who then seeks to engage an eligible recipient to develop an agreement, or flow from other agreements (e.g., arising from consultations to establish a national park where there is a commitment to fund activities of specific groups and where a manager can use the GCCP to fund these obligations).

There is no application form for a contribution under this component of the GCCP. Instead, a manager interested in using the mechanism confirms with the Manager, Financial Strategies and Resource Analysis in National Office that the GCCP is the appropriate financial instrument (i.e., as opposed to a contract or some other instrument).

If the proposal is appropriate for GCCP funding, it is then sent, through the Finance Manager in National Office, to the CEO for pre-approval. This step was introduced in April 2010 to deal with government-wide financial restraints, and includes direction that new contribution agreements be limited to making payments in only one fiscal year and that new multiyear agreements will only be considered in exceptional circumstances (such as land claim negotiations) and requires the CEO’s pre-authorization before multi-year funding is discussed with the potential recipient. These measures are intended to limit the Agency’s liabilities and provide for future budget flexibility. Prior to April 2010, the CEO only reviewed agreements for final approval.

Following pre-approval, the manager, most often working closely with the recipient, prepares a draft contribution agreement and a draft letter from the CEO to the recipient, and related briefing notes, transmittal slips and other information. The detailed information is again submitted to the Manager, Financial Strategies and Resource Analysis for review (i.e., of the completeness of the documentation, not the merit of the project) and to a local or regional communications advisor for review of the associated communication plan. Once approved at this level, the documents are sent to the relevant Director General for approval, following which they are again provided to the national lead in finance who coordinates review by functional leads in National Office[4] and the final approval by the CEO (i.e., the CEO has the authority to approve and amend contribution agreements of up to $1 million. Treasury Board approval is required for contributions exceeding this amount).

A signed letter from the CEO to the recipient constitutes the formal approval for an agreement. The actual contribution agreement is signed by the manager with delegated authority on behalf of the Agency once formal approval is issued. Managers are expected to provide a copy of the signed and dated agreement to the Manager, Financial Strategies and Resource Analysis in National Office to ensure there is a centralized file for at least the administrative aspects of each agreement (i.e., approval processes, final agreements and amendments).

Monitoring: Managers are expected to monitor projects following a risk-based plan developed during the negotiation phase of, and included in, each agreement. They are to maintain files with copies of reports, evidence of results achieved, and details of project expenses and payments. There is no requirement to provide evidence of project performance to National Office.

2.5 GCCP Cooperating Associations Component

Cooperating associations are registered non-governmental, non-profit corporations that assist Parks Canada by providing services to the public and/or generating revenues to direct towards approved activities that support Parks Canada’s mandate.[5] They are unique in that they exist solely or largely for the purposes of supporting the Agency and one or more national parks or national historic sites, in contrast to other classes of recipients.

While we identified a few cases where cooperative associations accessed funding under the general component of the GCCP, in the vast majority of cases, the associations are funded through a small central fund (i.e., $189, 200 per year) specially allocated for this class of recipients. Central funding of cooperative associations has existed as far back as the 1980’s. Up until 2005, this component of the GCCP was administered by the National Parks Directorate in National Office, at which point administration was transferred to the External Relations and Visitor Experience Directorate (ERVE).

The original funding objective was to support cooperating associations moving toward financial self sustainability. Funding to a particular association was limited to five years and a maximum of $40K per year. Under these criteria it was reported that as of 2009-2010 nearly 60% of the existing cooperative associations (n=53) were “no longer eligible to receive any contribution funding”. Accordingly, in 2009-2010 the objective of funding cooperative associations shifted from a focus on financial sustainability to assisting associations with projects that contribute to “…park or site objectives for visitor experience” (see Appendix C for more background).

Cooperative associations have always completed, in one form or another, applications for funding which were collectively reviewed in National Office. Available documentation was insufficient to provide a clear picture of how applications were evaluated and recommended for funding prior to 2009-2010, although assessment of total contribution amounts received against the funding limits was a key factor.

Under the current process, a call for applications letter is sent from National Office to Field Unit Superintendents (FUS) and copied to cooperating association liaison officers within the Agency. FUS/liaison officers ensure that the local association president/chairperson receives a copy of the application form. If the association proposes a project, FUSs and local visitor experience managers are expected to confirm the capacity of the association to undertake the proposed project. Applications are received centrally in February for consideration of funding for the upcoming fiscal year. The proposals are rated by a team of representatives from Stakeholder and Partner Relations Branch and the DGs Eastern and Western/Northern Canada. The highest rated proposals are recommended for funding and sent to the CEO for approval.

Formal approval is again provided through a letter from the CEO to the recipient confirming funding. Prior to April 2010, the final agreements were prepared in National Office. As of this fiscal year, agreements are prepared by field unit managers based on consultations with National Office as needed.

Monitoring and Reporting: The nature and extent of monitoring and reporting by cooperative associations prior to April 2009 is unclear. Comments from management suggest that reporting generally involved recipients confirming by a checkbox that they had used the funding for the purpose for which it had been intended. Current requirements are identical to the general component of the GCCP where monitoring and maintaining evidence of project performance is the responsibility of the project manager in the field. Unlike the general component of the GCCP, National Office makes efforts to collect project performance reports.

2.6 Expenditures

Yearly contributions payments under the GCCP over the last five years are shown in Table 2.

Table 2: Contribution Expenditures by Year
Pre April 2006 Evaluation Focus Post March 2011 Grand Total
2006-07 2007-08 2008-09 2009-10 2010-11
NA 3,615,322 3,991,055 4,765,316 4,626,540 4,469,772* NA 21,468,005

Source of Actual Spending: Financial system
Note: 2010-11 spending includes both actual and committed expenditures in the financial system as of Oct 12, 2010

Expenditures averaged $4.3M per year between April 2006 and October 2010 and represent about 0.7% of the Agency’s direct program spending for the four years where complete data is available.

2.7 GCCP Logic Model

The logic model showing the relationship between inputs, activities/outputs, reach, and intermediate and long-term outcomes is shown in Table 3. The model is a visual summary of the program description.

Table 3: Logic Model for Parks Canada General Class Contributions Program
Activities
  • Receiving and reviewing funding requests
  • Providing contributions
  • Project support and monitoring
Reach Direct (through provision of funding):
  • Voluntary associations
  • Management boards
  • Non-profit organizations (e.g., environmental groups, Aboriginal communities)
  • Provinces/ Territories, Municipal/ Regional governments
  • Canadian research community
  • International organizations (for one-time, specific events and knowledge building)
Beneficiaries:
  • Parks Canada
  • Canadian public
  • Visitors/ tourists
  • Stakeholders with shared mandates
Outputs
  • Products for visitors to purchase (souvenirs/ activities)
  • Ecological/ comme-
    morative integrity research
  • Education/ outreach/ tourism activities
  • Support for national historic sites
  • Expertise from Aboriginal groups regarding land negotiations, park/site management planning and relationship building
  • Developed capacity among Aboriginal groups for economic opportunities and presentation of culture
  • Functional management boards
  • Municipalities within Parks that meet environmental standards
  • Support to enhance the ecological integrity of lands bordering national parks
  • Support to establish areas of national historic or ecological significance
Immediate Results
  • Enhanced visitor experience
  • Better knowledge base for decision making and dialogue on commercial, ecological or Aboriginal issues of mutual interest to Parks Canada and other stakeholders
  • Education of targeted audiences in ecology, safety and other issues
  • Enhanced public recognition, appreciation and engagement in the values of natural and cultural conservation
  • Stakeholder engagement and involvement in ecological or cultural integrity objectives
  • Enhanced understanding and protection of heritage assets
Ultimate Results Contribution to:
  • Preservation and protection of nationally significant examples of Canada’s natural and cultural heritage
  • Public understanding, appreciation and enjoyment in ways that ensure ecological and commemorative integrity for present and future generations

3. Evaluation Design

The evaluation examined the relevance, performance (e.g., effectiveness, efficiency and economy) and design and delivery of the program consistent with the requirements of TB Evaluation Policy (2009). The scope included all GCCP contribution agreements involving payments between April 2006 and March 2011. Parks Canada Agency evaluation staff and contractors conducted the evaluation between April and October 2010, using multiple methods of data collection to address the evaluation questions.

3.1 Approach and Questions

The evaluation is based on a review of documents and files, key informant interviews and case studies. The specific methodologies and questions were originally identified in the Integrated Results-based Management and Accountability Framework (RMAF) / Risk-Based Audit Framework (RBAF) for the GCCP prepared in 2005 at the time of last renewal of the program’s Ts&Cs. Some questions and methods have been updated to conform to current TB Evaluation Policy requirements. In keeping with evaluation policy, the evaluation approach was calibrated to the low materiality and the low level of program risk identified in overall evaluation planning. Insofar as the GCCP has documented Ts&Cs, including high-level expected outcomes, the evaluation was goals-based and designed to assess the extent to which the program is meeting objectives.

The evaluation addressed 10 key questions and 14 associated expectations. The key questions are shown in Table 4. A more detailed matrix of evaluation questions, expectations, indicators and relevant data sources is shown in Appendix D.

Table 4: Evaluation Questions
Relevance
1. Is there a demonstrated demand/need for the GCCP?
2. Is the GCCP aligned to government strategic priorities and Parks Canada’s mandate and objectives?
3. Does the GCCP duplicate other funding sources? Should some of the funded projects be part of other programs?
Performance
4. Are mechanisms in place for monitoring project progress and are final results documented?
5. Are the projects achieving expected results?
6. Are GCCP projects cost-effective?
7. Is the administration of the GCCP cost-effective and efficient?
Design and delivery
8. Are roles and responsibilities with respect to overall program management and individual agreements clear and understood, and effective?
9. Are the GCCP’s Ts&Cs and the clauses in contribution agreements appropriate?
10. To what extent is the GCCP fair, accessible, and understandable to its intended users?

3.1.1. Methods

Document Review: Documents reviewed included TB policies, directives and guidance related to transfer payment programs, the approved Ts&Cs for the program, various Agency plans and reports, other government and Agency documents, audits and evaluation reports, and program management records and files (e.g., an Excel logbook maintained by National Office Finance Branch, e-mails, briefing notes). A list of key documents reviewed during the course of the evaluation is included in Appendix E. Document review allowed us to build an understanding of overall government direction with respect to G&C programs, how the GCCP is intended to work in theory, how it works in practice and the nature and kinds of issues and risks involved in administering G&C programs.

File Review: We obtained all the GCCP agreement files (n=339) from Finance Branch for the period under review for coding by consultants using a standard template. File review was intended to capture basic information about the nature of the contributions and the recipients, including timing/duration, project type and associated program activity areas, project value, other sources of project funds, proportion of total funding provided by Parks Canada, and the nature and extent of agreement amendments, as well as other pertinent information. The file review template is shown in Appendix F.

The 339 agreement files contained proposals that did not lead to a contribution agreement, cases of agreements in progress but not yet finalized, and cases where payments did not occur during the period under evaluation. Eliminating these cases lead to a final population of 276 projects funded through the general GCCP component. The population included agreements where some of the payments were made prior to April 2006 and others where the payments will continue after March 2011.

We were unable to obtain a complete set of project files of the cooperative association component of the GCCP from the External Relations and Visitor Experience Directorate (ERVE) for the entire evaluation period. Instead, we obtained information on the number of requests for funds, the number of approved projects, and the value of the projects for the period under evaluation. This information was mainly in the form of briefing notes and other documents. After April 2008, the availability of funding requests and/or agreements improved although performance reporting, for example, was only available for half the projects in 2009-2010. The number of agreements identified in this process (i.e., 68 for the period) and total funding was validated by a program manager in ERVE.

Collection of Performance Information: As Finance Branch does not collect information on project performance, we selected a sub-sample of projects for this component of the GCCP for a more detailed review. Working with Finance Branch we identified a list of 260 agreements in place as of

The 260 agreements were stratified by contribution funding (within the evaluation period) and then by whether they were completed as of the end of March 2010 or ongoing, to ensure coverage based on materiality and project status. We selected projects randomly within each strata for review with the exception of high value projects (i.e., 250K or more) where we selected all projects given that risks related to non-performance are presumed to increase with higher value investments.[6]

Project managers responsible for the 40 selected agreements were contacted via e-mail and asked to provide a variety of information including project proposals, agreements, and monitoring and reporting documents, in electronic or hard copy form. Two reminders were sent to maximize response and we worked with contacts to confirm what types of information and documents were to be provided. In all, 36 of 40 (90%) requested files were provided within the evaluation’s timeframe. The projects represent 13% of the universe of 276 GCCP general component agreements and 39% of the value of all contributions (i.e., $11.M out of $29.1M). Appendix G provides some information on the characteristics of the selected agreements.

We also reviewed performance reports available in National Office for seven of the 14 contributions in 2009-2010 with cooperating associations representing $75K of the $149.2K in awarded funds for that year. Reports for other years were not readily available.

Key Informant Interviews: We conducted eight semi-structured interviews with staff in National Office (e.g., Finance, executive advisors to DGs and the CEO and those involved in the administration of agreements with cooperating associations) to gain an overall understanding of the program, approval processes and issues and challenges.

Key informant interviews with project managers were generally targeted at those involved in the sub-sample of agreements selected for review of performance information. We interviewed 20 project managers within the Agency responsible for administering individual or multiple contribution agreements, selected to maximize geographic diversity and to ensure a mix of contacts with experience administering agreements varying in materiality and types of recipients (e.g., universities, Aboriginal groups, towns/municipalities, charitable organizations). Four additional contacts who had not been contacted for the in-depth file review were also interviewed. Two were selected for their experience with the Cooperating Associations Program; one was added to enhance representation from a particular region, and one was interviewed based on the recommendation of another key informant. The interviews covered questions about the need for and relevance of the GCCP, its design and delivery (e.g., processes, Ts&Cs, governance structure) and the performance of the GCCP as a mechanism as well as its funded projects.

We interviewed the recipients for 21 of the projects for which we also interviewed program managers. In many cases, the project managers made first contact with potential interviewees. A few of the selected recipients were not available for an interview during the evaluation period. Recipient interviews addressed issues such as the importance of the GCCP to the projects, program design (e.g., the effectiveness of the funding/approval process, clarity of roles and responsibilities, and the appropriateness of agreement clauses), and performance (the success and value of the project and its contribution to Parks Canada’s mandate and the effectiveness of the GCCP as a funding mechanism). Efforts were made, based on project documents and interviews with Parks Canada project managers, to tailor the performance questions to the nature of the project(s) being discussed.

Case Studies of Project Performance: During key informant interviews, Agency personnel were asked to identify projects appropriate for case studies demonstrating the results achieved by GCCP projects. Some key informants indicated the projects selected for in-depth review would be appropriate and others identified other projects. We used the information in project documentation and Agency and recipient interviews to develop seven short case studies of results representing a range of different types of projects and validated this information with the relevant project managers.

3.1.2. Limitations

The quality and completeness of information available in the project files for the general component of the GCCP (n=276) was highly variable in part due to turnover in the administration of the program and in part due to changing documentation requirements for the period of the evaluation. We adopted a variety of strategies to mitigate these limitations including obtaining additional electronic documents not included in the paper files, cross-referencing file information with Finance Branch logbook data and with other information in the files, and in a few cases where there were ambiguities in the funding allocation of high materiality projects, contacting project managers to confirm amounts across fiscal years. In consequence, our file review provided a reasonably complete and accurate description of the universe of projects funded under this component of the program.

As noted, for the cooperative association component of the program, we did not have access to the universe of all contribution agreement files for the period under evaluation and therefore had to rely on management records and documents (e.g., letters, briefing notes) to construct basic information about the number and value of agreements. In most cases, this was sufficient for the purposes of the evaluation.

The GCCP general component projects selected for detailed analysis of results achievement, while not statistically representative of the universe of all projects, provide a sufficiently diverse sample (i.e., in terms of types of recipients and project purposes and the levels of funding provided) to support conclusions about the state of performance reporting and results achievement in this component of the program as a whole. Over sampling high value projects provides assurance that conclusions regarding results achievement will be applicable to the agreements with higher risks in the event of non-performance.

Given limited resources and time, we did not attempt to obtain a representative sample of performance reporting from the cooperative association component of the program but used instead the readily available reports in National Office.

The majority of our key informants within the Agency were associated with the projects selected for detailed analysis of results achievement and all but one recipient key informant was associated with these projects or from cooperative associations. The views and opinions of these managers and recipients are not necessarily representative of all project managers or recipients. However, given the range of projects and key informants selected, the diversity of views expressed, and the consistency of the information with other reviews and analysis of grants and contribution programs and with documentation available in the Agency, it is likely that the interviews were effective in identifying key issues or concerns with the GCCP.

Project-level performance information is widely dispersed across the country with project managers and not readily available for evaluation purposes. The problems are compounded given staff turnover, loss of corporate memory, and changes in organizational structures or processes, which lead to problems in locating some documents, particularly from early in the evaluation period. However, use of multiple follow-ups and collection of a variety of types of information allowed us to capture sufficient information for evaluation purposes.

Data on the value of contribution agreements and expenditures overall and by recipient types and project purposes was collected for the evaluation. Actual expenditure data for the GCCP as a whole is available by year and can be disaggregated by program activity and fund or cost center. However, the expenditure data is not reliably linked to individual agreements. Finance Branch has recently begun reconciling planned versus actual expenditures for GCCP projects on a monthly basis however the data was not available for the evaluation. As a result, some of our analysis depends on planned expenditures (i.e., the planned payments identified in the contribution agreement rather than actual payments made to recipients). Planned expenditures identified in the files are greater than actual expenditures. In consequence the absolute value of the expenditures is likely overstated in some parts of our analysis; however the risk that conclusions resulting from the analysis will be materially affected by this overestimation is low.

Another limitation is the lack of reliable data on the administrative costs of the program (i.e., time or salary costs of many Agency employees across the country working with potential recipients to develop proposals, draft agreements, review, approve and monitor the agreements). We explore this issue in more detail in the section on program design and delivery. In the absence of good quantitative information on the costs of program delivery, conclusions regarding economy and efficiency are largely based on qualitative data from key informant interviews.

Although the resulting data and analysis has limitations, the use of multiple methods and compensating analysis provides a reasonable basis for drawing conclusions about the relevance and performance of the GCCP. The level of effort and investment in the evaluation is consistent with the low materiality and low level of program risk.

4. Evaluation Findings

4.1 Relevance

Question 1

Is there a demonstrated demand/need for the GCCP?

Indicators

  • Program manager and recipient (i.e., user) support for mechanism (i.e., advantages cited)
  • Number and value of contribution agreements by year and cumulatively over time
  • GCCP contribution funding as a percentage of overall project value
  • Reported consequences if the GCCP were not available

Expectation: The GCCP is seen to be useful and add value in achieving the Agency’s mandate.

Grants and contribution programs are widely recognized as important instruments for achieving federal government results and delivering on responsibilities to Canadians (see, for example, the December 2006 Report From Red Tape to Clear Results: The Report of the Independent Blue Ribbon Panel on Grant and Contribution Programs and the October 2008 TB Policy on Transfer Payments).

Agency managers and staff interviewed for the evaluation noted several advantages to the GCCP including:

  • Allowing the Agency to collaborate and develop close working relationships with outside organizations
  • Helping meet the mandate when expertise and time are not available internally (e.g., access significant research expertise, graduate students at a reasonable cost)
  • Providing greater influence, involvement and accountability than other means of acquiring services (e.g., contracts)
  • Achieving greater value (as projects often have other sources of funding)
  • Enhancing Agency visibility and profile
  • Fostering on-going multi-year relationships in some cases
  • Developing an expanded service offer beyond the scope of the mandate (e.g., supporting cooperative associations in the delivery of products or activities to enhance visitor experience).

The program is also identified as a mechanism to help Parks Canada deliver on commitments arising from previous Ministerial Roundtables[7] to build new relationships with nature-oriented and cultural organizations in the non-profit sector, other levels of government, Aboriginal communities and organizations to advance the cause of protecting Canada's heritage.

Some of the advantages identified by managers are supported by observations from the file review. For example, 79% of recipients in the general GCCP component reported other sources of funding for their projects, such that Parks Canada’s investment supported just 38% of the estimated $76.6M in total project value. Similarly, approximately 38% of the agreements were for a period of two or more years, serving to foster multi-year relationships with recipients.

Expectation: Managers make use of the GCCP.

The number of agreements and the value of planned expenditures by year are shown in Table 5.

Table 5: Number of Agreements and Planned Spending by Year
  Pre April 2006 Evaluation Focus Post March 2011 Total
2006-07 2007-08 2008-09 2009-10 2010-11
General Component
# New 7 46 67 53 63 40   276
Planned Spending ($) 404,350 3,570,079 3,922,726 5,245,371 5,013,746 7,153,291 3,372,387 28,681,950
Cooperative Component
# New   16 11 13 14 14   68
Planned Spending ($)   148,223 81,223 118,833 149,200 178,049   683,278
Total
# New   62 78 66 77 54   344
Planned Spending ($)   3,718,302 4,003,949 5,364,204 5,162,946 7,331,340 3,372,387 29,365,228

Note: The pre-April 2006 count of agreements and dollar values only include those that continued into the evaluation period.

There were a total of 344 agreements in effect under the GCCP with total planned expenditures of $29.4M (i.e., $24.9M was to be expended between April 2006 and March 2011).[8] On average there are 54 new agreements per year, a figure that will increase as more agreements are signed in 2010-2011. This level of program use is substantially higher than anticipated in the 2005 Integrated RMAF/RBAF for the GCCP, where an average of 25 to 40 agreements per year was reported with $2M per year in anticipated disbursements to recipients.

There is widespread management uptake of the program based on an analysis of actual and committed program expenditures by fund center for the period April 2006 to October 2010. Thirty-one of the Agency’s thirty-two field units have used the GCCP during the period, as have four directorates in National Office, the office of the DG Western/Northern Canada and the office of the Executive Director Mountain Parks, and four of the Agency’s five service centers (i.e., a total of 42 users). The extent of use as measured by total expenditures varies considerably. The National Parks and External Relations and Visitor Experience Directorates in National Office together account for 27% of the expenditures. Four field units account for an additional 22%, while fully half of the units account for only 10% of the total expenditures.

Recipient key informants stressed the importance of GCCP funding for their projects and indicated that without the funding projects would not have been undertaken, would have been scaled back or delayed or the final project would have been of lesser quality. Some indicated their project requires a number of sources of funding and having Parks Canada as a source of funding can help leverage funds from other sources. Some noted that by providing funding, Parks Canada is able to influence project focus, particularly in the case of research.

Question 2

Is the GCCP aligned with government strategic priorities and Parks Canada’s mandate, program activities and objectives?

Indicators

  • Presence of legal authority for program
  • Logical links between GCCP goals and Agency mandate and overall government priorities
  • Perceived and documented alignment with Agency objectives and program activities
  • Profile of recipients and agreement purposes

Expectation: The goals and objectives of the GCCP are logically related to the Agency’s mandate and to overall government priorities.

The legal authority for the GCCP derives from section 19(1) of the Parks Canada Agency Act. The program’s specific objectives as set out in the Ts&Cs (see section 2.1) focus on supporting natural and cultural conservation and ecological or cultural integrity, creating a better knowledge base for decision making and dialogue, enhancing visitor opportunities, protecting heritage assets, and educating targeted audiences. As such, they are logically related to the Agency’s mandate and strategic objectives.

Achieving these objectives in turn advances several objectives set out as whole of government strategic outcomes (http://www.tbs-sct.gc.ca/ppg-cpr/frame-cadre-eng.aspx) including a clean and healthy environment, a vibrant Canadian culture and heritage, and a safe and secure Canada (i.e., the latter applies to contributions that support operations for improvements to townsite or throughway infrastructure).

Managers and staff interviewed for the evaluation consistently report that GCCP supported projects are aligned with the Agency’s priorities and objectives. Managers who use the GCCP mechanism and those who approve the projects are responsible for ensuring that projects are aligned with Parks Canada’s mandate and that the connections are identified throughout the approval process. Since July 2009, the expectation has been that managers will explain how the contribution directly supports one or more of the Agency’s specific performance expectations in the Agency’s corporate plan within the briefing note accompanying the request for approval. It was noted that since projects approved under the general component of the program are financed by managers’ existing O&M budgets, managers have a built in incentive to ensure the project is meaningful and appropriate to the mandate.

Expectation: Agreements support a wide range of recipient types and agreement purposes as specified in the Terms and Conditions.

The GCCP is specifically intended to support a wide range of recipients and project types. Table 6 shows the profile of recipients along with planned spending. The table reflects the number of agreements rather than the unique number of recipients under each category receiving funding (i.e., some recipients have been party to more than one agreement).

Table 6: Number of Agreements and Value of Planned Funding by Recipient Types
Category of Recipients Agreements Planned Funding
Number % Total % Average Per Agreement
University 64 19% 4,116,945 14% 64,327
Aboriginal community/group 51 15% 8,279,629 28% 162,346
Provincial/municipal/regional government 13 4% 654,532 2% 50,349
Cooperative association* 76 22% 865,528 3% 11,389
Management board** 6 2% 883,744 3% 147,291
Non-profit/charitable organization 129 38% 13,927,074 47% 107,962
International organization*** 4 1% 617,776 2% 154,444
Other 1 0% 20,000 0% 20,000
  344   29,365,228    

Notes:

*
The number of cooperative association recipients reflects the 68 agreements supported by central funding and eight additional cooperative association agreements identified in the 276 agreements held by Finance Branch.
**
Management boards result from various land claim and park establishment agreements. While they vary in scope and role, they often encompass the provision of advice on park or species management issues in keeping with an inclusionary consensus-based approach to resource use and development in national parks. There are only three unique recipients in this category in our sample.
***
International organizations are those based outside of Canada such as the International Union for Conservation of Nature (IUCN) and the International Council on Monuments and Sites (ICOMOS). Some recipients based outside of Canada were coded to other categories (e.g., four agreements with American universities were assigned to the university category and three agreements with recipients based in the United States were assigned to the non-profit/charitable association category).

All the recipient types identified in the Ts&Cs have received some agreements and funding. Not-for-profit/charitable organizations represent the single biggest recipient category, followed by cooperative associations, universities and Aboriginal groups/communities. Although cooperative associations receive a large number of agreements, the size of these agreements tends to be small (i.e., less than $12K on average and they account for only 3% of the planned spending). In contrast, the other common recipient groups account for 89% of the total planned spending. Aboriginal communities/groups and non-profit/charitable organizations also tend to have bigger average contribution agreements, although management boards (i.e., a small defined sub-group) have the biggest average funding per agreement.

Table 7 shows the profile of agreements and planned spending by the project purposes set out in the Ts&Cs. Project types are not explicitly stated in project files and were coded based on project titles and descriptions. Consistent with what we heard in interviews with project managers, we found that assigning each agreement to a single project type was not necessarily straightforward since agreements may relate to a number of different project types or may not be fully aligned with any category. In coding project files, we tried to select the best fit.

Table 7: Number of Agreements and Value of Planned Funding by Agreement Type
Purpose of Agreement Agreements Value Average Total
Number % Total %
Seed or interim funding for cooperative associations or support for park or site objectives for visitor experience 75 22% 855,528 3% 11,407
Research on commemorative and ecological integrity 125 36% 9,235,429 31% 73,883
Education/outreach/tourism 73 21% 6,105,283 21% 83,634
Aboriginal groups (Land Negotiations, Park/Site Management Planning and Relationship Building) 32 9% 5,780,228 20% 180,632
Aboriginal groups (Economic Opportunities and Presentation of Culture) 17 5% 3,520,282 12% 207,075
Operational support to management boards 6 2% 883,744 3% 147,291
Community relationships/municipalities within parks 6 2% 658,634 2% 109,772
Ecological protection (land assembly and easements bordering national parks) 6 2% 1,528,100 5% 254,683
Support to non-profits for establishing areas of national historic/ecological significance 4 1% 798,000 3% 199,500
  344   29,365,228   85,364

Notes: The purpose of funding one cooperative association was coded to support to non-profits for establishing areas of national historic/ecological significance rather than seed or interim funding for cooperative associations.

Again, with the exception of “operational support for national historic sites not owned or administered by the Agency”, which was not identified in coding as the primary purpose of any of the agreements, all the intended agreement purposes identified in the Ts&Cs have received some funding. The majority of projects and planned spending is focussed on research related to ecological and commemorative integrity, education/outreach/tourism projects and on Aboriginal groups for land negotiations, management planning, economic opportunities and presentation of culture.

There are very clear and consistent links between the classes of recipients and the purposes of agreements, as shown in Table 8.

Table 8: Relationship Between Recipient Type and Purposes of Agreement
Category of Recipients Predominant Purpose of Agreement
University 97% of funding is for research on commemorative and ecological integrity
Aboriginal community/group 91% of funding is for land negotiations, park/site management planning and relationship building, and economic opportunities and presentation of culture
Provincial, municipal, regional government 96% is for research related to commemorative and ecological integrity and education/outreach/tourism projects
Cooperative association With one exception all the funding was for seed or interim funding to lead to financial sustainability or more recently for projects supporting visitor services objectives at particular parks and sites.
Management board 100% is operational funding for management boards including supporting costs of meeting or office space, administrative support and translation, payroll/ honoraria, and funding for a special event, or joint management of an ecosystem.
Non-profit/charitable organization 71% is for research on commemorative and ecological integrity or for education/outreach/tourism. 20% is directed at Aboriginal groups for land negotiations, park/site management planning and relationship building, to other non-profits for ecological protection (land assembly and easements bordering national parks)
International organization 98% of funding is directed to education/outreach/tourism including support for conferences, developing country participation at Agency led meetings, or knowledge building in fields of mutual interest.

Table 9 shows the percentage of actual GCCP spending for each program activity based on data from the financial system.

Table 9: Percentage of GCCP expenditures by Program Activity (April 2006 to October 2010)
Program Activity %
Heritage Places Establishment 20%
Heritage Resources Conservation 40%
Public Appreciation and Understanding 25%
Visitor Experience 14%
Townsite & Throughway Infrastructure 1%
Total 100%

There is at least some GCCP expenditures in support of each of the Agency’s program activities although the majority of spending (i.e., 85%) supports the first three programs. This allocation does not simply mirror the portion of total Agency spending by PA (i.e., roughly 40% of the Agency’s annual spending is on PAs 1 to 3) and highlights how the GCCP supports primarily the achievement of specific aspects of the mandate rather than others.

Question 3

Does the GCCP duplicate funding sources outside of the Agency? Should some of the funded projects be part of other programs?

Indicators

  • Extent and nature of other funding sources
  • Existence of other appropriate funding instruments within the Agency for projects funded under the GCCP

Expectation: The GCCP as designed does not substantively duplicate other G&C programs.

As noted previously, 79% of the agreements in the general component of the GCCP reported other sources of funds for their projects. Table 10 shows the number and percentage of agreements by sources of other funding.

Table 10: Sources of Other Funding
Source Agreements With Funding Source
# %
Federal 75 27%
Provincial 85 31%
Municipal 35 13%
Recipient 161 58%
Other 110 40%

Note: Counts total to more than the number of agreements since many agreements have multiple sources of funds.

Other federal sources of funds include the Natural Sciences and Engineering Research Council, Environment Canada, Department of Fisheries and Oceans, Atlantic Canada Opportunities Agency, and Indian and Northern Affairs Canada. The category “other” includes a variety of sources; the most commonly cited were non-profit organizations and corporate donations.

Agency and recipient key informants generally did not perceive or were unaware of overlap or duplication between the GCCP and other sources of funding. Other funding was generally perceived as complementary rather than duplicative.

Expectation: The GCCP is not used in place of other more appropriate funding instruments available within the Agency.

A few key informants within the Agency suggested that some projects funded under the GCCP might also have qualified for funding under the Agency’s other contribution program, the National Historic Sites Cost- Sharing Program (NHSCSP). The NHSCSP is specifically targeted at incorporated not-for-profit organizations, other levels of government, and not-for-profit Aboriginal organizations that own and operate national historic sites. Similarly, a few respondents in the Agency suggested that GCCP project objectives might be accomplished through other instruments rather than contribution agreements including contracts, contractual arrangements (i.e., with Crown corporations, provincial governments, municipalities), exchange of letters (e.g., for travel reimbursements for non-public servants) and administrative arrangements (e.g., for transfers within the federal government).

Examples of cases where other instruments might be used could include projects for which there are multiple federal sources of funds (i.e., in such cases one option may be for one or more departments to transfer funds to another department to minimize the number of agreements in place). Some key informants interviewed suggested that the GCCP may not be the best instrument for funding management boards or municipal services, though clear alternatives were not offered. The new pre-approval process for GCCP agreements is intended in part to identify the most appropriate mechanism for a project.

Overall Finding: Relevance

The GCCP continues to be a relevant mechanism for addressing Agency needs as evidenced by the number and value of agreements entered into each year and the extent of management uptake of the mechanism. As designed, the program objectives are aligned with the Agency’s mandate and priorities and with overall government outcomes. It is seen to be useful and provides important benefits for both managers and recipients. Until recently, projects have not consistently been linked to intermediate and long-term objectives and priorities of the Agency prior to entering into the agreements, although it is possible to do so in most cases after the fact. Consistent with program intent, the GCCP supports a wide variety of recipients and project types and primarily three of the Agency’s program activities. There is little sense that it duplicates other sources of funding. In the absence of the GCCP, managers would in most cases lack an alternative mechanism within the Agency to provide project funding, a critical gap for example in cases where the mechanism is used to fund obligations arising from agreements with Aboriginal peoples.

4.2 Effectiveness

Our focus in this section is on results reporting and achievement at the level of individual agreements rather than at the level of the GCCP as a whole. The latter issue is discussed in relation to clarity of roles and responsibilities under program design and delivery. The analysis is based on the detailed review of performance information for 43 projects, two of which were only recently completed and 14 of which were not complete and for which no final performance report was available, as well as interviews with 20 projects managers and 20 recipients associated with these agreements.

Question 4

Are mechanisms in place for monitoring project progress and are interim and/or final results documented?

Indicators

  • Reports of monitoring activities
  • Existence and quality of project performance information
  • Correspondence between expected and actual types of performance information

Expectation: Projects are monitored and performance information is collected consistent with expectations.

The 2005 Integrated RMAF/RBAF for the GCCP identifies a number of expected project monitoring mechanisms. Agency and recipient key informants described using virtually all of these including e-mail and telephone contact, participation in meetings and conference calls, and interaction with recipients working in the Park. In some cases, a Parks Canada representative actively participates in the project via steering committee or board meetings or through co-supervision or co-authoring of research.

It is expected that all agreements will include provisions for interim and final reports, financial monitoring and reporting, and submission of draft and final deliverables (e.g., research papers, education/outreach products). Our review of selected project files (n=43) confirmed that all of the agreements included reporting requirements (i.e., all projects require final financial and activity reporting, and multi-year projects typically require reporting at least annually, if not biennially or quarterly). A few projects in our sample also required that the recipient provide a detailed work plan with costs prior to starting the project.

Under the 2005 RMAF/RBAF, final project reporting should include a narrative report with the project objective, a description of the project activities and partners (and their contribution to the project), a discussion of key barriers and challenges, and a description of the results of the project and how the project contributes to one or more of the immediate and ultimate results in the GCCP logic model. The extent and nature of results reporting was expected to vary depending on the nature and materiality of the project. Larger value projects (i.e., over $100K) should include a brief evaluation with data on project reach and/or visitor satisfaction where relevant. Evidence for lesser value projects could include a variety of documents demonstrating performance (i.e., pictures, media clippings/excerpts, reports, publications and any other evidence of a tangible output).

Though relevant reports and evidence were sometimes difficult to locate due to turnover in the administration of projects within the Agency, with the exception of one recent agreement signed earlier in 2010, we obtained performance information for all agreements included in the in-depth review (42 of 43). In two cases (one complete and one ongoing project that had been effect for many years), we were only able to obtain weak evidence of project progress or results (i.e., financial statements with no narrative to provide context, an e-mail summarizing results after we requested clarification).

Among completed projects for which final reports were available, there was no standard format and a lack of alignment with the requirements in the RMAF (e.g., many included some description of activities and results, but generally no section devoted to barriers and challenges). Only one project file included an evaluation report despite the fact that we over sampled high value projects where evaluations of some kinds were expected under the RMAF. In a small minority of cases, Parks Canada managers responsible for the projects had added additional material to the file on project results beyond the reports provided by recipients (e.g., consolidated a list of relevant publications and presentations, list of parks where a tool was being used).

Question 5

Are the projects achieving expected results and contributing to intermediate and long term outcomes?

Indicators

  • Extent agreements are carried out as intended
  • Managers’ and recipients’ reports of success
  • Extent outputs are produced, reach and immediate impacts targets are achieved

Expectation: Projects achieve the results set out in contribution agreements or have a reasonable likelihood of achieving the results.

For the 42 projects with sufficient information to assess progress and/or results, we found evidence that they were all substantially implemented, or are in the process of being implemented as intended. Eight of the 43 projects reviewed had an amendment to the original agreement to increase project funding, project duration, reporting or payment frequency, or re-profile the funding across fiscal years without changing the substantive nature of the project. One project was expanded in scope although the total amount of funding did not change. For six projects under the general component of the program and one under the cooperative association component, delays in the project schedule (i.e., either the implementation of some tasks or the reporting of results) were also noted although these did not result in amendments. For a few projects, there was documentation to suggest the need to expand the scope or to adjust performance measures due to difficulties in developing appropriate indicators. In short, although changes occur in the scheduling, funding, duration or payment schedules of some projects, the substantive nature of most projects does not change and they are implemented as intended.

Not surprisingly, Parks Canada project administrators and key informant recipients reported that the projects they were involved in were successful in producing the expected outputs and objectives. Only a few recipients indicated that projects had fallen short of some expectations. A few managers indicated they were aware of other projects not included in the in-depth review that did not meet all objectives.

For the 42 agreements with sufficient performance information, the evidence indicated that the anticipated tangible outputs were produced or were likely to be produced (i.e., products for visitor purchase or consumption, research or information products such as reports and articles, databases, protocols, software, and guidelines; information, communications or media products; negotiations and agreements with Aboriginal groups; and meeting or events held).

Under the 2005 RMAF/RBAF, performance reporting is expected, where possible, to go beyond documenting simple outputs and include as well information related to the reach of the project (e.g., attendance at events, distribution of promotional materials, citations, journal articles, presentations at conferences, media coverage, website visits and downloads) and indicators of the quality of products (e.g., awards) or other relevant statistics (sales or revenue figures, volunteer hours engaged in the case of cooperative associations). The majority of the files we reviewed included this kind of results information although frequently it is not summarized or quantified and must be inferred from other data in the file (e.g., a list of journal articles and conference presentations is available, but no summary is provided; meeting minutes with lists of attendees are available, but no roll-up of total meetings and attendance figures is presented).

Appendix H provides additional examples of the nature and kinds of performance reporting and results achieved for the seven projects used as case studies.

A minority of the project files included additional impact data (e.g., some form of trend analysis of relevant indicators, assessment of satisfaction) although this information is not always relevant to demonstrating results. As there is typically a series of intervening events and activities over a long time frame and various extenuating factors that influence achievement of ultimate outcomes, it is not expected that project proponents will measure and report on longer term outcomes (e.g., park establishment, increased ecological integrity, enhanced conservation, or increased visitor safety). However, the logical relations of the project to intermediate and long term outcomes are documented in proposals and agreements.

Some key informant managers or recipients provided anecdotal evidence of additional project results or benefits not documented in the files. Often these represent consequences of projects that occur after project closure, or were unintended positive impacts of the agreements. These included strengthened relationships and collaboration; a couple of recipients from Aboriginal groups in particular noted the importance of having enhanced trust and mutual understanding. Some noted ongoing collaborative work or additional collaborative arrangements had followed from the agreements. A couple of key informants noted that projects can sometimes result in training for future Parks Canada employees. One said the relationship with Parks Canada had opened their organization to new opportunities and ways of doing things. One recipient noted that the agreement had helped their region shine.

According to key informants, results achievement is facilitated when the terms of reference, expectations, and roles and responsibilities are clear at the inception and project managers remain actively involved in the project and keep good records. Other factors contributing to success include:

  • ensuring that the Agency’s level of influence is clear in a multi-funder project; steering committees can be useful in such cases
  • ensuring enough time for agreement approval and for the project itself and having multi-year agreements provides flexibility to mitigate unforeseen delays, since it allows for the reallocation of funds
  • assisting recipients, particularly those facing capacity issues, with reporting; suggestions included providing training and/or templates
  • including controls within the agreements (e.g., payment hold-backs pending the provision of deliverables).

Overall Finding: Effectiveness

The intended results of the agreements funded through the GCCP have for the most part been achieved or have a reasonable expectation of being achieved.

All projects in our sample required interim and final reporting on project performance. For 98% of the files reviewed, there was sufficient information to conclude that the projects were being, or had been, implemented as intended and that the expected outputs will likely be or had been produced. In a majority of cases, project reporting went beyond simply documenting activities and outputs and provided evidence of project reach, quality or other relevant statistics. Additional results or benefits of agreements occur but these are rarely documented in project files. Factors that promote the achievement of results include clear expectations and roles and responsibilities, setting reasonable time frames, active project management, assisting recipients, and maintaining controls over project payments.

Although it was possible to assemble evidence demonstrating project performance, the information is widely dispersed, not always readily available, and not in the standard narrative format set out in the RMAF.

4.3 Cost-Effectiveness and Efficiency

Question 6

Are GCCP projects cost-effective?

Indicators

  • Project manager and recipient reports of cost-effectiveness
  • Extent of Agency funding compared to total project value

Expectation: GCCP projects provide value for the dollars spent.

Parks Canada GCCP project administrators generally indicated that, once the approval process is complete, the GCCP projects themselves provide good value for money, though some said it was too early to assess this for their project. The fact that many projects leveraged funds from other contributors is seen to add value and maximize overall cost-effectiveness for the Agency. As noted previously, the majority of projects funded under the general GCCP component have other sources of funding. Table 11 shows a breakdown of the number of agreements by percentage of the Agency’s funding of total project costs (i.e., Parks Canada contributed between 1% to 24% of project costs for 68 projects or 25% of the 276 projects reviewed).

Table 11: Percentage of Parks Canada Funding for Projects
% of Agency Contribution to Project Agreements
# %
1% to 24% 68 25%
25% to 49% 73 26%
50% to 74% 58 21%
75% to 99% 19 7%
100% 58 21%
Total 276 100%

Parks Canada’s planned contributions of $29.3M for these projects support $76.2M of work.

Some Parks Canada officials indicated they have a good sense of the costs involved in the project and therefore can challenge unreasonable expenses during negotiation of an agreement. It was also noted that, since funding for GCCP projects comes out of managers’ operating budgets, there is an incentive to ensure agreements provide good value.

Views on project cost-effectiveness varied depending on the nature of the projects. For example, some of those responsible for administering research-related agreements noted the cost-effectiveness of contribution agreements, including funding of student research projects, over acquiring the expertise and/or conducting the research in-house. Others noted that in some cases (e.g., agreements with management boards or certain Aboriginal groups), it would not be relevant to consider alternate recipients or undertaking the work in-house in assessing cost-effectiveness since funding those organizations may constitute a legal obligation, but that proposed expenses can nevertheless be scrutinized to ensure they are reasonable.

Recipients also generally indicated that their projects provided Parks Canada with good value. Like Agency personnel, they noted that value had been added by leveraging other funds and promoting collaboration. Some indicated it was unlikely the project could have been more cost-effectively undertaken by the private sector or internally. A number of recipients indicated that Parks Canada’s level of contribution was appropriate/fair, but others indicated that it was too low to meet demands or that more funding might have enhanced the speed or quality of the project.

Question 7

Is the administration of the GCCP cost-effective and efficient?

Indicators

  • Evidence and perception that the application, approval and project management components of the program are timely and efficiently administered (including risk-based)
  • Ratio of administrative costs to total funds dispersed

Expectation: Administrative processes are seen to be timely and appropriate.

Agency key informants were split on whether the assessment and approval process was timely and effective or slow and cumbersome due to the level of approval/number of sign-offs and reported large number of documents that are required to support approval. A similar split was evident among recipients interviewed for the evaluation (i.e., a few had to start work before the agreement was finalized or risk losing the support of other funding sources). It was widely suggested by these managers and recipients that approval or documentation requirements be scaled to the size of the agreement and track record or capability of the recipient to deliver on the expected results.

Suggestions for improving the efficiency of the process focus on training managers or recipients, providing templates or tools, obtaining multiple sign offs simultaneously rather than sequentially, and devoting full time resources (i.e., one FTE) to evaluate and assess all proposed agreements from a policy perspective to ensure they are aligned with Agency priorities. Recipient suggestions were similar and included ensuring the GCCP was the appropriate mechanism before the project was far advanced, use of standard forms, in-person rather than telephone meetings, and minimizing the number of funding sources and ensuring they shared the same vision.

We noted that under the TB Policy on Transfer Payments (2008) deputy heads are required to establish reasonable and practical departmental service standards for transfer payment programs (section 6.5.9). This requirement did not exist in 2005 when the GCCP Ts&Cs were last renewed. Service standards state a level of performance that citizens, recipients or users can expect of a program and typically focus on the timeliness of delivery. For example, the Community Development Program at Agriculture and Agri-Food Canada (AAFC) has set standards in days for length of time to respond to inquiries, acknowledge receipt of applications, provide notification of acceptance or rejection of proposals, provide a final contribution agreement, and for issuing payments following recipient claims.[9] Establishing service standards can help to address managers’ or recipients’ concerns about timeliness by clearly communicating reasonable expectations in advance and throughout the approval process. For the general component of the GCCP, where the existence of the mechanism is not advertised to citizens or potential recipients and proposals are not solicited, service standards would essentially be directed at Agency managers (i.e., the users of the instrument) who would in turn be able to inform actual recipients about expectations throughout the process of establishing an agreement. In the case of the cooperative association component, service standards could be publicly communicated to the recipient group.

The 2008 TB Transfer Payment Policy differs from the 2000 policy in placing an emphasis on a risk-based approach to designing and managing G&C programs. Deputy heads are specifically responsible for ensuring that the administrative requirements on recipients (i.e., monitoring and reporting requirements) are proportionate to the level of program risk, the value of funding in relation to administrative costs, and the risk profile of the recipient (section 6.5.7). The Agency’s approach with respect to monitoring, reporting and auditing is clearly consistent with the policy direction as evidenced by the discussion of these issues in the 2005 RMAF/RBAF, the current direction with respect to project monitoring (i.e., for the general component) and our review of agreements and reporting practices for a sample of agreements. The Internal Audit function has not identified the GCCP as a high priority for audit; given the known evaluation requirements, it has not scheduled an audit in its three year plan.

The Transfer Payment Policy does not specifically mention a risk-based approach to project approval, although lack of this was suggested by several management and recipient key informants as the reason why approvals were not timely (i.e., timeliness would be improved if documentation requirements and level of approval were scaled to the materiality of the agreement and the risk profile of the recipient). Under the current Agency model, the approval process, although differing somewhat for the general and cooperative association components of the GCCP, is not risk-based as all agreements regardless of their materiality must be approved by the CEO. Starting in 2010-2011, the requirement is to obtain pre approval by the CEO prior to entering into discussions on the substance of an agreement, as well as final approval, representing a reduction in the risk tolerance of the Agency, particularly with respect to entering into long term agreements (see section 2.4). Whether and to what extent this will affect the timeliness and cost-effectiveness of approval is not clear at this point. A few management key informants thought adoption of the process was a positive innovation, but the rationale for having CEO approval for all agreements regardless of materiality and recipient risk is not evident for a minority of the key informants interviewed for the evaluation.

Expectation: Administrative costs of the GCCP are reasonable compared to the value added of the agreements.

The administrative costs of the program have previously been reported as $300K per year to provide $2M in contribution payments (i.e., in the 2005 Integrated RMAF/RBAF and section 20 of the approved Ts&Cs). We obtained a copy of a spreadsheet from Finance Branch showing a breakdown of the expenditures into $125K for the Branch, $75K in yearly “audit and evaluation” and $100K in total for all other administrative costs including for the cooperative association component and for costs in National Parks Branch for agreements related to ecological integrity and park establishment, and a residual category for other program managers. Finance Branch was unable to provide farther detail on the basis for the estimates (e.g., whether audit costs were intended to include costs of recipient audits or only those in the Office of Internal Audit and Evaluation).

It seems possible that these estimates only captured the National Office costs for coordination and administration of both components of the GCCP and do not include the considerable resources devoted to developing and managing agreements by local managers. If this is the case, the estimate may be reasonable, although incomplete. Given the lack of specification on how costs were calculated, we were unable to draw a reliable conclusion.

The administrative costs were based on an assumed $2M per year in contribution expenditures. Actual contribution expenditures have averaged almost $4.3M per year over the last five years. It is therefore likely that National Office costs to coordinate, administer and monitor this level of contribution funding will have increased although the relationship between administrative costs and the value of contribution funding need not be linear.

Assuming the reported administrative costs are reasonable orders of magnitude estimates of National Office expenditures, it follows that the cooperative association component, with a reported administrative cost of $25K per year, is much less efficient than the general component of the program (i.e., as measured by administrative costs per dollars of funding awarded). For the cooperative association component, one dollar in administrative costs provided on average $5.40 of funding over the five year evaluation period compared to the general component where the ratio is one dollar in administrative costs to $15.12 in funding provided.

Overall Finding: Cost-Effectiveness and Efficiency

The program is generally considered to be economical and provide good value for the money invested, particularly given the amount of funds provided from other sources (i.e., an estimated $46.5M for these projects), which allow the Agency to obtain outputs and results that it could not achieve on its own.

With respect to administrative efficiency, a substantial minority of Agency and recipient key informants suggested that the project approval process was not timely due to documentation requirements and/or approval steps. This issue could be addressed, in part at least, by establishing service standards as required under the 2008 TB Transfer Payment Policy, to communicate a reasonable expectation of timeliness to managers and recipients.

We found that estimates of the actual administrative costs of the program are old, not well documented, and likely limited in scope so that a reliable quantitative assessment of program efficiency (e.g., costs per dollar of contribution funding provided) could not be determined. If the estimated costs of administering the cooperative association component are accurate, it is much less efficient than the general component.

As required under the 2008 TB Transfer Payment Policy, there is a clear risk based approach to monitoring and reporting for individual projects. The program has not been identified as a high risk and has not been subject to internal audits since 2004. Although not specifically addressed in the TB Policy, a number of managers and recipients suggested scaling project documentation and approval requirements to materiality and risks of the particular agreements to enhance timeliness.

4.4 Program Design and Delivery

Question 8

Are roles and responsibilities and the governance structure with respect to overall program management and individual agreements clear and appropriate?

Indicators

  • Reported clarity of roles
  • Consistency between prescribed roles and practices
  • Extent processes function as intended

Expectation: Roles and responsibilities are clear and understood, and carried out effectively.

According to the 2005 Integrated RMAF/RBAF for the program, no one manager is responsible for the overall governance of the GCCP. Responsibility for planning, managing and administering the program is shared between a variety of senior managers (i.e., operational and functional Directors General and equivalents) who in turn delegate responsibility to the operational level to enter into agreements and for monitoring progress and performance in relation to individual agreements. Generally, Agency key informants indicated that roles and responsibilities are clear, though some expressed confusion about the roles and accountabilities of Director Generals, ERVE, and local finance personnel.

The question of the appropriate role of National Office in the GCCP was raised by half a dozen key informants, none of whom were project administrators. Essentially, a larger role for National Office was suggested including a review of all potential agreements against corporate priorities at the beginning of the fiscal year (i.e., involving relevant program areas in National Office), which could make the program more strategic, identify areas for collaboration, and minimize duplication. In a similar vein, some key informants suggested there should be a centralized roll-up of results achieved with the GCCP agreements, but they noted there is a need to balance national reporting for accountability purposes with considerations of cost and efficiency.

Some ambiguities and misalignments between roles and responsibilities for program monitoring and reporting as outlined in the RMAF and current practices were identified. For example, it is implied that senior managers and others as “monitoring authorities” (the RMAF is not always clear or consistent on who is responsible for the monitoring function) would undertake some annual analysis of the number and nature of agreements and their degree of progress or success within their functional or operational area of responsibilities and identify issues and successes. We found no evidence this has been done, though such analysis is more feasible for the cooperative association component given its more centralized administration.

The RMAF also states that an overall annual report “should” be established for reporting purposes with a description of all projects, project recipients, project types, coordinates of recipients, and expended funds, and report on all performance indicators set out in the RMAF, and identify exceptionally successful projects or accomplishments. Responsibility for producing the report was not clearly identified and it has never been prepared.

Although management and monitoring of the program is shared, a coordinating role for the Finance Branch for aspects of the general component of the GCCP was established in 2005 following an internal audit (2004) of the management and controls for grants and contributions in general in the Agency. In the subsequent 2005 Integrated RMAF/RBAF for the GCCP it was stated that the Centre of Expertise, as it was then called, would among other things:

  • develop and implement standardized tools, templates, and checklists for use by program officers/managers
  • review and approve contribution agreements (prior to their signature by the recipient), principally to ensure the agreement contained all the required information and clauses
  • oversee the review by program officers/managers of the risk-based monitoring plans for each agreement
  • monitor and ensure compliance with the Centre’s approved administrative and management standards and processes
  • track results achieved by the agreements, in consultation with the relevant program officers/managers
  • identify the need to conduct a recipient audit and be responsible for scoping and planning the audit and retaining external auditors to conduct the audit.

In practice, the Centre/Finance Branch has and continues to do some of the activities and tasks set out in the RMAF but not others. Most Agency key informants indicated the tools and templates available to them are useful and have become more useable over time. Finance Branch has a clear role in coordinating the approval of GCCP agreements for the general component and we heard few criticisms of their performance in fulfilling this role. They have maintained files and logbooks on the agreements sent for approval since 2005 and the current GCCP approval process indicates managers are to forward Finance copies of the signed contribution agreements.

However, as noted in our file review, many of the agreement files are incomplete (e.g., only 52% of the 276 agreement files contained signed and dated contribution agreements)[10] and proposed but abandoned agreements versus actual agreements are not always clearly indicated. The Centre has and is making efforts to improve its information base over time. For example, over the period covered by the evaluation, recording of project objectives, relevant program activities, locations and type of recipients, and the fund/cost center responsible for payments and documenting approved versus paid amounts by fiscal year, has improved. As well, the Centre is in the process of identifying and linking expenditures in the financial system with specific agreements. The purpose of collecting this information and how it will be used is not fully articulated and Finance has never tracked results or carried out summary analysis of agreements to inform management decision-making about the program during the years covered by the evaluation. Finance has also not assumed responsibility for identifying the need for recipient audits and has not arranged for any during the period under review.

Unlike the general component of the GCCP, the cooperative association component has by its design always had a degree of central coordination and record keeping. Record keeping practices were identified as strong in the 2004 Internal Audit of Grants and Contributions. However, for the first three of the five years covered by the evaluation, little information is available, in part due to turnover in the administration of this component. Since April 2008, more complete documentation on individual agreements is in place and efforts are currently made to collect performance information centrally (though it was only available in about half of the cases). Management of this component have used the information collected nationally to produce some analysis (i.e., past funding patterns) which supported revising the objective of the program from financial sustainability to funding projects.

Neither the RMAF nor other program documents provide an explicit rationale for why the cooperative association component of the GCCP is managed differently than the general component of the program (i.e., why it has a central dedicated source of funds and a central application and evaluation process). It is clear that cooperative associations represent a unique class of recipients and the importance of supporting this group is discussed in several documents. Historically, the basis of funding the group (i.e., financial sustainability) was different than other groups, but with the recent changes to this component of the program, this is generally no longer the case. According to ERVE, less than half of the cooperative associations funded in 2009-2010 and only two of those funded in 2010-2011 would have been eligible under the previous criteria (i.e., the remaining recipients would not have been funded).

Recipients generally indicated that their roles and responsibilities are sufficiently clear (i.e., as set out in contribution agreements and in other documents such as MOUs between Parks Canada and their organizations). We also heard that once potential recipients are identified by managers, there is often a concerted effort to work with the project proponent (e.g., including drafting agreements with proponents) to make the process as accessible and understandable as possible.

Question 9

Are the GCCP’s Ts&Cs and the clauses in the contribution agreements appropriate?

Indicators

  • Reports that the Ts&Cs are sufficient and appropriate for the Agency’s needs
  • Contribution agreement clauses are seen as reasonable, clear and comprehensive

Expectation: The Ts&Cs are seen as appropriate to the intended purposes of the GCCP and Agency needs and the agreement clauses are understood and appropriate.

Most Agency key informants indicated that the GCCP’s Ts&Cs meet the Agency’s needs. The most common suggestion, offered by approximately half a dozen key informants, was to expand the list of eligible recipients to include partnerships with private/for-profit organizations. A few key informants indicated that, although some such agreements are in place, the eligibility of international research institutions or universities outside of Canada for contributions should be clarified in the Ts&Cs.

Additional suggestions for changes, each made by a few respondents, included:

  • Clarifying the differences between a contract and a contribution, the purpose of the GCCP, and what can be funded
  • Standardizing recipients’ eligible overhead expenditures and the treatment of in-kind contributions and allowing for the provision of operational support when appropriate.
  • Clarifying at the outset the implications of profit making for cooperative associations when entering into an agreement. One recipient key informant from a cooperative association indicated that there had been confusion and a need to re-conceptualise a project because it would generate profit for the recipient.
  • Dropping the central funding approach for cooperative association projects and funding these projects through the reallocation of managers’ operating funds in the same manner as other GCCP projects, or conversely increasing the funding envelope for cooperative associations and clarifying what constitutes “services outside of Parks Canada’s direct delivery mandate”.

At the level of individual contribution agreements, recipients generally indicated that the clauses were straightforward and appropriate. A few indicated that the agreements allowed sufficient flexibility to reallocate funds across fiscal years or among priorities in order to accomplish the projects. Some noted the agreement was comprehensive or long, but this was generally an observation more than a complaint. Specific suggestions by a few respondents related to improving the clarity of the agreement with respect to issues involving foreign currency or an extended agreement duration or project end date that would correspond with the end of the recipient’s fiscal year.

Question 10

To what extent is the GCCP fair, accessible, and understandable for recipients and users?

Indicators

  • Existence of processes to ensure the GCCP is accessible, fair and understandable
  • Reports that it is fair, accessible and understandable

Under the 2008 TB Transfer Payment Policy, G&C programs are expected to be “…accessible, understandable and useable by applicants and recipients (section 5.2.1). For the cooperative association component of the program, the processes in place for informing all eligible recipients on an annual basis of the availability of funding and the application of standard criteria and practices to rate proposals and recommend projects for funding (see section 2.5) support achieving these objectives. We did hear a few comments from cooperating association recipients that accessibility, fairness or understanding could be improved by changing the criteria to rate projects (i.e., allowing for profit making), improving the information provided about application deadlines and Agency priorities, announcing earlier in the year that a project was approved (i.e., mid-May rather than late June or early July), and providing an opportunity for unsuccessful applicants to have a debrief on why they were not successful.

The situation is more complicated in the case of the general component of the GCCP since the existence of the mechanism is not publicized and is therefore not easily accessible or necessarily understood by the population of eligible recipients. Only one recipient key informant under this component of the program remarked on this and suggested raising the profile of the GCCP outside the Agency. When asked, managers within the Agency expressed some concern that publicizing the availability of the instrument would lead to many proposals that they could not fund. Managers did consistently report that once a potential recipient is identified there is often a concerted effort to work with the project proponent (e.g., including drafting agreements with proponents) to make the process as accessible and understandable as possible.

In principle, the general component of the GCCP is made more understandable by having clear criteria for assessing the merit of the proposals and more fair by having several review steps leading to final approval that exist, in part, to ensure that proposals are consistent with the criteria and meritorious projects are funded. However, the Agency does not track unsuccessful proposals under the general component of the program (i.e., as was envisioned would be the case in the RMAF) and it is not clear how many of these occurred, why proposals are rejected, how a lack of success is communicated to proponents, or whether the proponents viewed the process as fair and understandable.

Considerations of access, understanding and use of the general component of the GCCP also apply to managers within the Agency as key “clients” of the instrument. As noted in the discussion of demand for the GCCP, there is evidence of widespread awareness and uptake of the GCCP across management units within the Agency. In our interviews with managers, we were not told of any problems in accessing, understanding or using the instrument (barring timeliness), although it should be noted that except for the National Office managers, our sample consisted of managers who had been successful at least once in establishing an agreement.

Overall Finding: Program Design

The GCCP as designed lacks a centralized role for managing and monitoring its overall implementation and reporting on its success. Monitoring responsibilities and reporting instruments beyond those required for individual projects were identified in the 2005 Integrated RMAF/RBAF but these have largely not been implemented, in part due to a lack of clarity in who was responsible for this function and perhaps in part due to a lack of demand for the information on the part of management.

A central coordination point has existed for the cooperative association component of the program for many years, but its effectiveness in capturing and retaining information at a national level appears to have declined since a 2004 internal audit and is only starting to recover in the last few years. The Centre for Excellence in Procurement and Transfer Payments was created in 2005 following the audit to fulfill a central coordination role in the approval process for agreements funded under the general component of the GCCP. Many agreements have been signed since the Centre was established and the process appears to work effectively. However, there have been problems with records management and information capture, although again it is improving over time.

The Ts&Cs of the GCCP are generally seen as appropriate although we noted a few suggestions for improvements focused either on clarifying or adding to the categories of eligible recipients. Additional suggestions related to improving the program design included adjusting administrative practices (e.g., standardizing the treatment of eligible administrative expenses and in-kind contributions) or providing training or tools related to the program (e.g., clarification of contribution vs. contracting, development of tools for recipient reporting of project results).

Finally, we concluded that the program has processes in place to ensure it is accessible, fair and understandable to applicants and recipients in the case of the cooperative association component and for management users for the general component.

5. Conclusions and Recommendations

The GCCP is one of two contribution programs in the Agency. Yearly contribution payments under the program have averaged approximately $4.3M over the last five years. The program represents less than 1% of the Agency’s direct yearly program spending. If the program is not adequately managed, there are risks that funded projects will not be aligned with and supportive of the Agency’s mandate, projects will not be implemented as intended, and/or that results will not be achieved efficiently and economically.

The GCCP is not a “program” in the traditional sense but rather functions as a general funding authority available to Agency managers to support projects and infrastructure costs across the full range of the Agency’s program activities rather than within a specific PA. With the exception of a sub-component of the GCCP dedicated to funding cooperative associations, there is no central source of funding for GCCP agreements. Instead, they are funded from the applicable program or Field Unit’s budget through the reallocation of O&M funding.

The evaluation focused on the continuing relevance of the GCCP, the effectiveness, efficiency and economy of its operations and questions related to program design and delivery. Our broad conclusions are:

  • The GCCP continues to be a relevant mechanism for delivery of the Agency’s mandate and priorities and is aligned with overall government objectives. Funded projects over the period April 2006 to March 2011 can be linked to the Agency program activities and the types of eligible projects set out in program Ts&Cs (i.e., particularly the heritage places establishment, heritage resources conservation and public appreciation and understanding programs in the Agency’s PAA). The GCCP is seen to be useful and offer advantages over other funding mechanisms by both Agency staff and recipients. The use of the program is fairly consistent over the last five years, averaging about 54 agreements per year and about $21M in expenditures as of October 2010. There is little sense that it duplicates other programs. There are no other G&C programs that cover the range of eligible recipients and types of agreements.
  • All the contribution agreements contain provision for interim and final project reporting. GCCP funded projects are largely implemented as intended and produce, or are likely to produce, the intended outputs defined in contribution agreements. In most cases, project reporting goes beyond simply demonstrating outputs are produced and includes other information on the reach or quality of the outputs or other relevant performance information (i.e., volunteer hours contributed, sales and revenue generated).
  • GCCP projects are generally considered to be economical and provide good value for the money invested, particularly given the amount of funds provided from other sources (i.e., an estimated $46.5M based on an incomplete record of other sources of funds for the period under review), allowing the Agency to obtain outputs and results that it could not achieve on its own.

The issues or challenges identified include:

  • A lack of good data regarding the total administrative costs of the program which prevents a rigorous analysis of the efficiency and economy of the GCCP
  • Lack of clarity in roles and responsibilities for overall program monitoring and performance reporting
  • Inconsistent file and data management practices in National Office for the overall project files, though some improvements in data capture have recently occurred, and a lack of consistent and consolidated reporting of project outputs and outcomes
  • Concerns about the timeliness of project approval and a lack of service standards that would clarify expected timeliness for various steps of the process
  • A number of specific suggestions to expand or change the Ts&Cs, provide training or templates, or clarify administrative issues
  • Questions about the continued relevance of the central funding and application and evaluation processes for the cooperative association component of the program.

Our recommendations follow from these issues.

Recommendation 1

The CFO and DG ERVE should jointly review and provide a reasonable basis for determining the administrative costs of the program in order to be able to assess program efficiency (i.e., administrative costs per dollar of contribution provided).

Estimates of administrative costs do not require exact time tracking to identify all salary costs allocated to the program. Modeling and estimation based on a sample of average or prototypical agreements of different types that included costs outside of National Office is reasonable.

Management Response
Disagree:
Given the small amount of funding spent on the GCCP, the variety in the amounts and scope of agreements, and the fact that the Agency operates using a decentralized model for its administration, the costs of gathering and reporting on the administrative costs of the GCCP would outweigh the benefits that could be derived from the information.

Recommendation 2

The National Office DGs should clarify the purpose and role of National Office in collecting information on GCCP funded agreements particularly with respect to overall program monitoring and reporting. Once purposes and roles are determined, they should ensure that the systems, personnel, and resources are in place so that the required information to support the National Office function is captured effectively and efficiently and the requirements are clearly understood and communicated.

Central office data capture need not simply duplicate information in files project managers are expected to maintain. For example, rather than collect copies of interim or final performance reports, it may be more efficient to create short templates and have project managers provide summaries of results at the close of a project or on an annual basis.

Management Response
Agree:
National Office DGs will clarify the purpose and role of National Office and project managers in collecting information on GCCP funded agreements and ensure that the required information to support the National Office function is captured effectively and efficiently and that the requirements are clearly understood and communicated. Action to be completed by March 31, 2011

Recommendation 3

The National Office DGs should review, clarify and communicate expectations on the form and content of final project reporting for GCCP funded projects (i.e., including project managers’ responsibilities to document performance beyond what is provided by recipients) and provide appropriate templates where required consistent with managements’ responsibilities to maintain sufficient performance information to support evaluations of programs.

Management Response
Agree:
This will be addressed as part of the response to Recommendation 2.

Recommendation 4

CFO and DG ERVE should review the Agency’s approach to project approval and at a minimum propose internal service standards for approval by EMC that would clearly communicate expectations with respect to timeliness of the process. In reviewing the project approval approach they should assess the Agency’s level of risk tolerance in this regard and clearly communicate the rationale for the proposed approach.

Management Response
Disagree:
Parks Canada will continue its current policy that all contributions are to be approved by the Chief Executive Officer, as per the Agency's Financial Delegation of Authorities authorized by the Minister. Timely processing will be addressed by producing an annual plan which ensures that approval in principle for an agreement is provided before the beginning of the fiscal year. The Agency will undertake to have all final approvals of contribution agreements secured within one month of the sign-off by originating Director or FU Superintendent.

Recommendation 5

DG ERVE, in consultation with the CFO as necessary, should review the status of the cooperative association component of the GCCP and either provide a rationale for its continued status as a centrally funded and administered component of the program or merge it into the general GCCP.

The recommendation is not intended to imply that the cooperative association component of the GCCP has poorer performance than the general GCCP component. Evidence suggests this is not the case. Rather the focus is on the need for and the cost efficiency of the different administrative approach to this aspect of the program.

Management Response
Agree
DG ERVE and CFO will review the status of the cooperative association component of the GCCP, including the process by which a contribution to a cooperative association is approved or denied. Action to be completed by March 31, 2011.

Appendix A: Strategic Outcome and Program Activity Architecture

Parks Canada Agency Strategic Outcome and Program Activity Architecture 2009/2010

Parks Canada Agency Strategic Outcome and Program Activity Architecture 2009/2010

Click to enlarge

Appendix B: GCCP Funding Criteria

Under the current Ts&Cs for the GCCP the detailed funding criteria to be assessed in establishing an agreement are:

  1. Program Delivery Capacity of Applicant
    • Capacity of applicant organization/team to carry out the proposed activities or project (in terms of technical expertise/education, skill sets, and prior experience).
  2. Objectives and Intended Results
    • Clarity of stated objectives and expected outcomes
    • Consistency of stated objectives with relevant Parks Canada objective and programming requirement
    • As applicable, reasonability of proposed approach to measuring, evaluating and reporting on progress/results
  3. Management Capacity of the Applicant
    • Strength of the organization’s governance structure
    • Financial stability of the organization
    • Financial management capacity of the organization to administer and manage funding to be provided
    • As applicable, extent to which the organization appears capable of being/becoming a sustainable organization
  4. Proposed Budget
    • Reasonability of the proposed budget(s)
    • Extent to which the budget reflects due regard for economical and prudent use of funding
    • Availability of other funding sources for the proposed project/activity
  5. Project Design
    • Reasonability/robustness of proposed plan/approach
    • Consistency of proposed plan/approach with Parks Canada principles (e.g., collaboration, community support, leveraging of resources)
    • Extent to which the proposal, applicant and or proposed activities/projects demonstrate community support (e.g., size of membership, letters of support, degree of participation by community members, number of local chapters, partnerships, etc.)
    • Reasonability/appropriateness of proposed dissemination plan

Appendix C: Cooperative Association Funding Criteria

Until recently, cooperative associations were funded under direction set out in February 1988 (Environment Canada – Parks Management Bulletin # 2.7.5). The bulletin commits to “… providing contributions for the establishment of strong and financially independent cooperating associations which can enhance the programs and services available to the public.” It is recognized that one means of supporting associations “…. is through direct financial contributions to help with their establishment and the initiation of revenue-generating services” with the intention “… that cooperating associations not become dependent on contributions to remain solvent, but establish sound revenue-generating operations and seek other sources of operational support as quickly as is reasonably possible.”

Eligible organizations (Section 1) are those “… which have signed a formal cooperating association agreement by the contribution funding submission date”. Funding limitations (Section 4) included:

  • that the contribution funding period of five fiscal years per cooperating association, for each property, is not exceeded;
  • that the maximum yearly total contribution to any one cooperating association does not exceed $40,000;
  • that the total of all contributions to any one cooperating association with zero or one sales outlet does not exceed $80,000 for the first property;
  • that the maximum additional contribution to each association for each additional property with one or no sales outlets does not exceed $50,000; and
  • that the maximum total additional contribution to each cooperating association for each additional sales outlet does not exceed $10,000.

Provision was included in the bulletin (Section 5) for “an extension of the contribution funding period and contributions greater than the established maximums in exceptional cases”.

Funds were to be allocated (Sections 16 and 17) based on the following priorities in order:

  • completion of contribution funding plans of existing associations;
  • establishment of new cooperating associations or expansion to additional sites by existing cooperating associations;
  • expansion to additional sales outlets by existing cooperating associations
  • theme-related interpretive products or the presentation of theme-related visitor services which will provide visitor opportunities not otherwise available.

The criteria used in assessing proposals for 2010-11 are shown below.

Description and objectives of the work or activities to be funded Reasonability/robustness of the proposed approach
  • Logical and convincing
  • Objectives clearly stated
  • Overview of activities
Consistency of the project with the park/site objectives for visitor experience
  • Potential to add value to the visitor experience
  • Accepted and approved by the Visitor Experience Manager and the Field Unit Superintendent (screening criteria*)
Extent to which the proposal demonstrates stakeholder/community support
  • Has the potential to engage local communities and volunteers
  • Has the potential to involve or work collaboratively with other stakeholders or partners
Creativity/innovative approach
  • Offers new and/or innovative services or activities
Proposed approach to measuring, evaluating and reporting on progress/results
  • Monitoring and evaluation are addressed
  • Willingness to share best practices identified (Screening criteria*)
Timelines & key milestones Project timelines provided and are realistic and activities are clearly identified with multi-year projects
Budget/financial considerations Direct project expenditures are itemized
Budget reasonable in relation to project objectives
Revenue generating/leveraging of other funding sources

Appendix D: Evaluation Questions, Expectations, Indicators and Data Sources

Evaluation Questions What Should be Observed Indicators Data source
Relevance
1. Is there a demonstrated demand/need for the GCCP? 1. The GCCP is seen to be useful and add value in achieving the Agency’s mandate.

2. Managers use the GCCP.
  • Program manager and recipient (i.e., user) support for mechanism (i.e., advantages cited)
  • Number and value of contribution agreements by year and over time
  • GCCP contribution funding as a percentage of overall project value
  • Reported consequences if the GCCP were not available
  • Expenditure data/ review of project approval files/ document review
  • Key informant interviews
2. Is the GCCP aligned to government strategic priorities and Parks Canada’s mandate and objectives? 3. The goals and objectives of the GCCP are logically related to the Agency’s mandate and to overall government priorities.

4. Agreements support a wide range of recipient types and agreement purposes as specified in the Terms and Conditions.
  • Presence of legal authority for program
  • Logical links between GCCP goals and Agency mandate and overall government priorities
  • Perceived alignment with Agency objectives
  • Profile of recipients and agreement purposes
  • Document review
  • Review of project approval files
  • Key informant interviews
3. Does the GCCP duplicate funding sources outside of the Agency? Should some of the funded projects be part of other programs? 5. The GCCP as designed does not substantively duplicate other G&C programs.

6. The GCCP is not used in place of other more appropriate funding instruments available within the Agency.
  • Extent and nature of other funding sources
  • Existence of other appropriate funding instruments within the Agency for projects funded under the GCCP
  • Key informant interviews
  • Review of project approval files
Performance
4. Are mechanisms in place for monitoring project progress and are interim and/or final results documented? 7. Projects are monitored and performance information is collected consistent with expectations.
  • Reports of monitoring activities
  • Existence and quality of performance information
  • Correspondence between expected and actual types of performance information
  • Key informant interviews
  • Results data for a sample of agreements
  • Case studies
5. Are the projects achieving expected results and contributing to intermediate and long term outcomes? 8. Projects achieve the results set out in contribution agreements or have a reasonable likelihood of achieving the results.
  • Manager and recipients’ reports of success
  • Extent agreements are carried out as intended
  • Extent outputs are produced; reach and immediate targets are achieved
  • Results data for a sample of agreements
  • Case studies
  • Key informant interviews
6. Are GCCP projects cost-effective? 9. GCCP projects provide value for the dollars spent.
  • Project managers and recipient reports of cost-effectiveness
  • Extent of Agency funding compared to total project value
  • Key informant interviews
  • Document Review
  • Expenditure Data
7. Is the administration of the GCCP cost-effective and efficient? 10. Administrative processes are seen to be timely and appropriate.

11. Administrative costs of the GCCP are reasonable compared to value added of agreements.
  • Evidence and perception that the application, approval and project management components of the program are timely and efficiently administered (including risk-based)
  • Ratio of administrative costs to total funds dispersed
  • Document review / expenditure data
  • Key informant interviews
Design and Delivery
8. Are roles and responsibilities and the governance structure with respect to overall program management clear and appropriate? 12. Roles and responsibilities are clear and understood, and carried out effectively.
  • Reported clarity of roles
  • Consistency between prescribed roles and practices
  • Extent processes function as intended
  • Document review
  • Key informant interviews
9. Are the GCCP’s Terms and Conditions and the clauses in contribution agreements appropriate? 13. The Ts&Cs are seen as appropriate to the intended purposes of the GCCP and Agency needs and the agreement clauses are understood and appropriate.
  • Reports that the Terms and Conditions are sufficient and appropriate for the Agency’s needs
  • Contribution agreement clauses are seen as reasonable, clear and comprehensive
  • Key informant interviews
10. To what extent is the GCCP fair, accessible, and understandable to its intended users? 14. The GCCP is seen to be fair, accessible, and understandable by intended users.
  • Existence of processes to ensure the GCCP is fair, accessible, and understandable
  • Reports that it is fair, accessible, and understandable
  • Key informant interviews
  • Document review

Appendix E: Key Documents Reviewed

Legislation

Parks Canada Documents

  • Parks Canada Corporate Plans
  • Parks Canada Departmental Performance Reports (DPRs) and Reports on Plans and Priorities (RPP)
  • Parks Canada (2004) Grants and Contributions Audit Final Report

GCCP Program-Related Documents

  • Parks Canada (2006) Contribution Agreements – General Class Contributions Program (PowerPoint presentation)
  • Parks Canada (2005) Integrated RMAF/RBAF for Parks Canada General Class Contributions Program.
  • Parks Canada (2005) Treasury Board Submission: Parks Canada Agency – Renewal of the General Class Contributions Program Authority and Grant to the International Peace Garden.

Intranet documents:

  • “Contribution Agreement Approval Process under the General Class Contributions Program (GCCP)”
  • “General Class Contributions Program Terms and Conditions”
  • “General information about contributions”
  • “Summary of Funding Instruments”

Cooperating Associations Program Documents and Background Materials

  • (1988) “Environment Canada - Parks Management Bulletin #2.7.5 Contribution Funding for Cooperating Associations” File C4010-1.2
  • (1981) “Parks Canada Management Guidelines 2.7.3 Cooperating Associations” File: 4010-101.

Treasury Board Secretariat Documents

Appendix F: File Review Template

1. Internal Reference: GC - ______________________

Review Date:__________________________________

Reviewer:____________________________________

All Documents in File?

Signed CA included ☐

Transmittal slip included ☐

Project Cancelled? Yes, project was cancelled ☐

Reasons for Cancellation:

Issues: Follow-up required (see notes and/or inconsistencies) ☐
Notes (e.g., missing documents/information, things to be checked or clarified, etc.):

Inconsistency in existing database (Excel data) and file:

____________________________________________

National Park or Historic Site Impacted:

Recipient Information

2. Project name/description: _________________________

3. Recipient organization: _________________________

4. Province/region:

Canadian Province: _________________

Other Country:

Country Name _____________________

Country state/province _____________________

5. Recipient type

1. University ☐
2. Aboriginal community/group ☐
3. Provincial/municipal/regional government ☐
4. Co-operative association ☐
5. Management board ☐
6. Non-profit/charitable organization ☐
7. International organization ☐
Other (Specify) ____________________

Contribution Agreement Information

6. Date signed (dd/mm/yyyy)
 
 / 
 
 / 
 


7. Amendments

1 Yes ☐

2 No ☐

Date Signed (dd/mm/yyyy)
 
 / 
 
 / 
 


Change made (Amendment details)

1 Financial (e.g., change in funding amount)

_________________________________________

2 Administrative (e.g., in duration of agreement)

_________________________________________

Project Resources

8. Parks Canada Commitment by fiscal year ($):

Contribution Agreement (initial) Amended Contribution Agreement Amended Contribution Agreement
2003-04 $  2003-04 $  2003-04 $ 
2004-05 $  2004-05 $  2004-05 $ 
2005-06 $  2005-06 $  2005-06 $ 
2006-07 $  2006-07 $  2006-07 $ 
2007-08 $  2007-08 $  2007-08 $ 
2008-09 $  2008-09 $  2008-09 $ 
2009-10 $  2009-10 $  2009-10 $ 
2010-11 $  2010-11 $  2010-11 $ 
2011-12 $  2011-12 $  2011-12 $ 
2012-13 $  2012-13 $  2012-13 $ 
2013-14 $  2013-14 $  2013-14 $ 
9. Total (all years): $  Total (all years): $  Total (all years): $ 

Funds in other Currency ☐ (drop down menu of most likely currencies: USD, Euros)

10. Parks Canada in-kind: $ _________________________

11. Other sources of funding

Federal: $ ________________________
Details: __________________________

Provincial: $ ______________________
Details: __________________________

Municipal: $ ______________________
Details: __________________________

Recipient $ ______________________

Other: $ ________________________
Details: _________________________

12. Total project cost $ ___________________

13. Parks Canada contribution (incl. in-kind) as % of total project cost _______ %

14. Advance payment

1 Yes ☐

2 No ☐

Amount (for one time payments): $ _________________________

Amount in % of project total (for multiple advance payments): ______________

Agreement Type/Objectives

15. Agreement type (Terms and Conditions)

  1. Co-operative associations ☐
  2. Research/commemorative and ecological integrity ☐
  3. Education/outreach/tourism ☐
  4. National historic sites ☐ →
    • ☐ not Agency owned
    • ☐ Agency owned, but managed by 3rd party
  5. Aboriginal groups (Land Negotiations, Park/Site Management Planning and Relationship Building) ☐
  6. Aboriginal groups (Economic Opportunities and Presentation of Culture) ☐
  7. Management boards ☐
  8. Community relationships/municipalities within parks ☐
  9. Ecological protection (land assembly and easements bordering national parks) ☐
  10. Support to non-profits for establishing areas of national historic/ecological significance ☐

16. Associated Parks Canada Activity Area(s):

Explicit ☐    Subjective ☐

PA1: Heritage Places Establishment ☐
PA2: Heritage Resources Conservation ☐
PA3: Public Appreciation and Understanding ☐
PA4: Visitor Experience ☐
PA5: Townsite and Throughway Infrastructure ☐

Administrative/Financial Information

17. Program activity code: ________________________

18. GL Account/Funds Commitment:

25113 Contributions ☐

25110 Contributions to Co-Operating Associations ☐

Cost Centre ________________

Fund Centre (commitment) ________________

Internal Order ________________

Vendor Account ________________

19. # of signatures (on transmittal slip prior to Centre of Expertise/Finance): ________

Appendix G: Projects Selected for Detailed Review

Pro-
vince
Recipient type Agreement type Year First Pay-
ment
Number of Years of Agree-
ment
Parks Canada in-kind
Contri-
bution
Total Contri-
bution
Total
Project Value
% PC
AB University Research/ commemorative and ecological integrity 2007-08 4   210,000 959,850 22%
AB Non-profit/ charitable organization Research/ commemorative and ecological integrity 2008-09 3   490,000 3,471,740 14%
AB Non-profit/ charitable organization Research/ commemorative and ecological integrity 2009-10 5   25,000 279,000 9%
AB Non-profit/ charitable organization Research/ commemorative and ecological integrity 2009-10 5   1,000,000 1,742,400 57%
Int’l University Research/ commemorative and ecological integrity 2007-08 2   93,000 250,490 37%
Int’l Non-profit/ charitable organization Research/ commemorative and ecological integrity 2009-10 1   50,000 50,000 100%
MB Non-profit/ charitable organization Research/ commemorative and ecological integrity 2006-07 5   300,000 1,091,000 27%
NB University Research/ commemorative and ecological integrity 2008-09 2 Yes 20,010 35,510 56%
NL University Research/ commemorative and ecological integrity 2008-09 1   7,000 23,000 30%
ON University Research/ commemorative and ecological integrity 2006-07 2   160,000 330,000 48%
ON University Research/ commemorative and ecological integrity 2008-09 1   18,240 43,340 42%
ON Non-profit/ charitable organization Research/ commemorative and ecological integrity 2009-10 2   165,000 390,000 42%
QC University Research/ commemorative and ecological integrity 2007-08 3   136,800 187,800 73%
BC Non-profit/ charitable organization Education/ outreach/ tourism 2007-08 4   400,000 2,400,000 17%
BC Non-profit/ charitable organization Education/ outreach/ tourism 2007-08 4   8,000 60,000 13%
Int’l International organization Education/ outreach/ tourism 2008-09 1 Yes 192,776 458,388 42%
ON Non-profit/ charitable organization Education/ outreach/ tourism 2007-08 2   500,000 5,342,485 9%
ON Non-profit/ charitable organization Education/ outreach/ tourism 2008-09 3   39,000 91,500 43%
ON Non-profit/ charitable organization Education/ outreach/ tourism 2009-10 2   300,338 311,338 96%
ON International organization Education/ outreach/ tourism 2008-09 1   400,000 1,169,052 34%
QC Non-profit/ charitable organization Education/ outreach/ tourism 2007-08 1   33,468 374,635 9%
QC Non-profit/ charitable organization Education/ outreach/ tourism 2008-09 1   28,770 462,770 6%
BC Aboriginal Community/ Group Aboriginal groups (Land Negotiations, Park/Site Management Planning and Relationship Building) 2007-08 3   75,150 96,210 78%
MB Aboriginal Community/ Group Aboriginal groups (Land Negotiations, Park/Site Management Planning and Relationship Building) 2007-08 4   270,700 270,700 100%
NT Aboriginal Community/ Group Aboriginal groups (Land Negotiations, Park/Site Management Planning and Relationship Building) 2007-08 3   947,760 947,760 100%
NT Aboriginal Community/ Group Aboriginal groups (Land Negotiations, Park/Site Management Planning and Relationship Building) 2007-08 2   769,232 769,232 100%
NT Aboriginal Community/ Group Aboriginal groups (Land Negotiations, Park/Site Management Planning and Relationship Building) 2008-09 2   102,080 146,980 69%
NU Aboriginal Community/ Group Aboriginal groups (Land Negotiations, Park/Site Management Planning and Relationship Building) 2009-10 2   626,240 626,240 100%
BC Non-profit/ charitable organization Aboriginal groups (Economic Opportunities and Presentation of Culture) 2005 or before 3   1,549,999 3,216,009 48%
YT Management board Management boards 2007-08 4   287,437 287,437 100%
AB Non-profit/ charitable organization Community relationships/ municipalities within parks 2008-09 1   60,000 101,221 59%
BC Non-profit/ charitable organization Community relationships/ municipalities within parks 2007-08 4   100,000 100,000 100%
ON Non-profit/ charitable organization Community relationships/ municipalities within parks 2009-10 1   34,916 81,812 43%
YT Aboriginal Community/ Group Community relationships/ municipalities within parks 2006-07 2   419,718 524,718 80%
BC Non-profit/ charitable organization Ecological protection (land assembly and easements bordering national parks) 2010-11 4   852,500 1,430,500 60%
NB Non-profit/ charitable organization Support to non-profits for establishing areas of national historic/ ecological significance 2007-08 4   600,000 1,161,000 52%
  11,273,134 29,284,117 38%

Appendix H: Case Studies of Project Goals, Results, and Reporting Mechanisms

Description Project goal(s)/ objective(s) Results Monitoring/ Reporting Mechanisms
Cooperative Association

Funded 2009/10,
$9K from Parks Canada Agency (PCA)
To develop and sell new product offerings and associated signage around 4 themes to help visitors feel an authentic connection to Jasper National Park while generating revenue for future projects. Outputs/outcomes achieved and expected:
  • Creation of various products and signage around 4 themes, including bilingual and environmentally sustainable products (e.g., water bottles, bandanas, t-shirts, reusable bags, pins)
  • Net earnings of $6,437.76 from products (per report provided in November 2009), with projected earnings of $14,975.91 by July 2010
  • 48 people contributed 216 volunteer hours to the Jasper Trails Project
  • 76 fact sheets about Jasper’s historic Raven Totem pole, which was taken down in April 2009, were distributed
  • Updates provided at Board meetings attended by Park liaison
  • Final report
Research

Funded 2 years April 2009 to March 2011

$165K from PCA

$225K from other sources
Study to determine the methods and cost of mitigating shoreline erosion of the Great Lakes shore ecosystem, including the eastern shoreline of Point Pelee National Park in order to help inform decisions, investments, future management strategies, and alternatives. Outputs/outcomes already achieved:
  • Provision of report/deliverables following field data collection with erosion mitigation alternatives and their costs/ benefits/ impacts and conclusions and recommendations
  • Ongoing meetings/ phone calls with recipient and contractor and participation in project steering committee
  • Preliminary findings/ draft deliverables for review
  • Final study/ technical report
  • Final project narrative and financial report for PCA
Research

Funded 2 years April 2007 to March 2009

$93K from PCA

$157K from other sources
(including $24,500 in-kind)
Project to acquire baseline data on sage grouse in northeast Montana and Grasslands National Park to document population parameters, understand habitats, and assess movement.
Results will inform the Greater Sage Grouse Recovery Strategy and assist in the identification of critical habitat.
Outputs/outcomes already achieved:
  • Provision of final report with findings and recommended management practices
  • Completion of student thesis
  • PCA co-authored article in International Journal of Conservation
  • 10 or more invited presentations related to the project
  • Ongoing interaction/ communication
  • Interim/ progress reports
  • Final project report to PCA
  • Provision of Master thesis
Research

Funded 2008/09

$7K from PCA

$16K from other sources
To refine a previously developed index of ecosystem connectivity and develop software to generate data and inform decision making in parks around aquatic barriers, their impacts, and the prioritization of restoration efforts.

Secondary goals include renewal of research and recognition of PCA’s contribution to the field.
Outputs/outcomes achieved:
  • Production of connectivity monitoring protocol and software tools
  • Use of tools in 6 national parks to assess aquatic ecosystems and prioritize restoration initiatives (e.g., culverts)
  • Sharing of software internationally
  • Publications in scientific literature(2)
  • Conference and other presentations (8)
  • Media coverage of project
Future/ongoing expected outputs/outcomes:
  • Final Masters thesis
  • Implementation of tool in additional parks
  • Ongoing collaboration and implementation of a larger project
  • Ongoing collaboration (via co-supervision of student, co-authoring of papers)
  • Interim progress report
  • Final report with financial allocations and study findings
Education/ outreach/ tourism

Funded 4 years
April 2007 to March 2011

$400K from Park Canada

$350K (x4) from other sources
To further develop avalanche safety programming in areas adjacent to Mountain National Parks and improve the effectiveness of PCA’s avalanche safety program by increasing visitors’ public avalanche awareness, thereby reducing injuries and fatalities. Outputs/outcomes already achieved:
  • Increased avalanche awareness through media outreach, signage, website, use of public avalanche warnings (e.g., 800,000 hits in 2008-09), attendance at avalanche safety workshops and forums (training programs growing 20% annually)
  • Application of safe practices in backcountry
  • Increased awareness of PCA and mountain parks
  • Engaged stakeholders in avalanche safety (i.e. all key players at the table through CAC)
  • PCA has access to more knowledge through the CAC
  • Development of world class outputs (e.g., revised public avalanche scale)
  • Increased back country visitation.
Future/ongoing expected outputs/outcomes:
  • Continued avalanche safety outreach/educational activities and resultant awareness and safe practice in back country
  • Expected decrease in avalanche fatalities in national parks/Canada
  • Engagement of key stakeholders
  • Roundtable (on which PCA liaison sits)
  • Weekly communication
  • Provision of recipient’s annual financial statements; last annual instalment of funding withheld pending financials
  • Provision of recipient organization’s annual report
  • Monitoring of attendance at workshops, media interest, feedback from communities, use of safety bulletins
  • Monitoring over the long term of the 10 year moving trend in avalanche deaths
Education/ outreach/ tourism

Funded 2 years
April 2009 to March 2011
To offer an entry pass to all grade 8 students in Canada and provide all grade 8 classes an entry fee exemption allowing free entry for one year to all PCA administered national parks and national historic sites. The long-term goal is to encourage youth to experience nature and history so they will become visitors and supporters now and in the future. Outputs/outcomes already achieved:
  • Developed program tools and met 100% coverage for distribution of “My Parks Pass” kits to teachers and to all grade 8 students in Canada (390,000 in 17,500 schools).
  • For 27 parks and 68 historic sites that charge fees, provided free access to and discount incentives to visit with their families.
  • Developed Agency operational guidelines (including tracking sheets) and distributed to all participating locations.
  • Early results for 2010: 100 students used pass, 565 upgraded to family group pass, 1,200 arrived as part of 36 school groups.
  • 700 contest entries have been made on micro site (by end of July); 150 call centre requests have been made
Future/ongoing expected outputs/outcomes:
  • Promotion of grade 8 curriculum-based learning programs to teachers in fall 2010.
  • Grow the project with further interactivity with pass-holders, and development of regional learning tools
  • progress updates (including for example count of passes distributed, learning materials downloaded) at scheduled meetings
  • Annual report and financials
  • Final project report
  • PCA tracking use of passes, and incentive upgrades in order to set future benchmarks
  • PCA tracking associated visitation increases
  • News media coverage being tracked
Aboriginal expertise/ relationship building/ capacity development

3 agreements from
April 2008 to March 2011

$169K from PCA
To deliver a teaching and healing camp in Saoyu and Edacho National Historic Site (which is co-operatively managed) to support the Aboriginal community’s involvement and commitment to preserving their culture by learning traditions and language from their elders and permit the community to learn about Canada`s system of historic sites and parks, the principle of commemorative integrity, and the Agency’s mandate.
  • Delivery of camps involving youth and elders that educate audiences and help preserve culture (with approximately 80 attendees)
  • Stronger relationship between the Agency and the recipients, which contributes to better knowledge and information sharing over time and ultimately, more effective management of the Saoyu and Edacho National Historic Site
  • Confirmation of camp logistics, attendees, facilitators, and coordinators (to accompany initial request for advance)
  • Ongoing communication and collaboration
  • Final accounting summary
  • Healing and Teaching Camp activity reports (to accompany request for release of remaining funds)

Appendix I: Grant to the International Peace Garden

The International Peace Garden (IPG) is a non-profit organization whose purpose is to create and maintain a garden as a memorial to the peace between Canada and the United States.[11] The idea for the garden came from Ontarian Dr. Henry Moore, who envisioned it as a place “where the people of the two countries could share the glories found in a lovely garden and the pleasures found in warm friendships.” On September 17, 1930 the International Peace Garden, Inc. was established for the creation and maintenance of a garden approximately half of which would be situated in the United States and the other half in Canada, as a memorial to the peace that has existed between the two nations. The Province of Manitoba provided 1451.3 acres of land, and the State of North Dakota 888, giving the Garden an area of 2339.3 acres. The dedication of the IPG took place on July 14, 1932.[12] The IPG is operated by a sixteen-person Board of Directors comprising half Americans and half Canadians. The Governor of North Dakota and the Premier of Manitoba each appoint one director.

The State of North Dakota and the Province of Manitoba provide equal funding to the garden, with some additional assistance provided by the American and Canadian federal governments, primarily for capital projects.

The Treasury Board submission under which the $22,700 annual grant for the IPG was approved for 2006-2007 through 2010-2011 indicates that the federal government has made an annual, unconditional grant to the IPG since the 1930s. Documents suggest that the grant was $15,000 up until 1979 when it was increased to $30,000 and then decreased by 10% to $27,000 in 1993-1994 and then annually decreased by 5.3%, 5.6%, and 5.93% in the three following years to $22,700 in 1996-1997 and subsequent years.

Funds provided through Parks Canada are for general operations and management of the IPG.

Following an internal audit of the grant (2001), a decision was made to require a formal request (i.e., an “application”) each year from the IPG prior to issuing payment of the grant (i.e., prior to that, payment was made automatically without any formal request by the recipient). The most recent documented approval for the grant from the Agency CEO is in the form of an internal letter to the Manitoba FUS authorizing payment of the grant in fiscal year 2001-2002 “and every fiscal year thereafter, subject to the availability of funding”. Each year, a letter from the Manitoba Field Unit is prepared for the recipient to accompany a cheque for the funding. Since at least June 2007, the Agency has requested that the IPG provide annual financial statements when it receives funding each year. No other reporting is required.

According to the IPG’s statement of financial position (December 31, 2008), total assets were approximately $1 million Canadian and:

  • revenues for the year were approximately $2.4M (i.e., 17% facility revenue such as admission and camping fees, rentals, retail sales and catering; 57% state/provincial/federal grants; and the remainder from donations currency conversation income and miscellaneous income) and,
  • expenditures were approximately $4.3M (i.e., 72% of which is for “capital outlay’)

The amount of grant revenue by source of funds is not identified in the financial statements. Parks Canada’s grant of $22.7K represents about 1.6% of the total grant revenue of the organization in 2008 and less than 1% of the total revenue of the organization.

Documents made available during the evaluation do not contain any information on the relevance of the grant for advancing the Agency’s or the federal government’s mandate or priorities, although it is possible to infer links to Parks Canada’s program supporting “other heritage places conservation” in the PAA. Given the size of the grant in relation to the IPG’s overall revenue, it appears it is not essential for ensuring on-going operations and has a symbolic rather than instrumental importance. None of the documentation provided indicated whether the IPG receives other grants or contributions from the federal government.


[1] http://www.pc.gc.ca/docs/pc/rpts/rve-par/ep-2010-2011_e.asp

[2] The Agency is involved in a new contribution program, the Aboriginal Economic Development Strategic Partnerships Initiative (AEDSPI) but does not operate this program directly.

[3] In the case of cooperative associations it is reported that in 2009-2010 44 (i.e., about 83% of the associations) requested and that 14 (32%) actually received funding (Source: E-mail correspondence from External Relations Branch). In 2010-2011 an estimated 35 proposals for funding were received of which 14 were recommended for approval for funding totalling $178,049 not including in-kind Agency contributions (i.e., 40% of proposals funded).

[4] These include the National Director Corporate Communication and the Chief Financial Officer.

[5] Source: Parks Canada Management Guidelines 2.7.3 Cooperating Associations: February 1981.

[6] Given the lack of congruence between this sample frame and our final identification of the universe of relevant agreements within the evaluation period, we do not treat the resulting sample selected for in-depth study as statistically representative of the population and did not estimate population parameters based on the sample data.

[7] The Minister responsible for Parks Canada is required by the Parks Canada Agency Act to convene a round table of persons interested in matters for which the Agency is responsible once every two years. This round table is a primary mechanism for the Minister Responsible for Parks Canada to receive input from Canadians on the recent performance of the Agency and its future direction. The most recent Minister’s Round Table (MRT) on Parks Canada was held on November 20, 2009 in Toronto, Ontario.

[8] These figures do not include in-kind support provided by the Agency. We identify a total of $655K in in-kind support for 24 agreements under the general component of the program. Management reported however that this likely underestimated the true value of in-kind support since this kind of support is not always clearly indicated in files or agreements.

[9] See http://www.rural.gc.ca/RURAL/display-afficher.do?id=1282663307449&lang=eng, for specific targets. AAFC has committed to establishing service standards. Almost all the published standards concern timeliness of delivery. In theory, service standards could focus on other aspects of the service process (e.g., permissible error rates, quality of product) which may or may not be applicable to contribution programs.

[10] The percentage of files with a signed and dated agreement varies over fiscal years from 29% to 76% having an agreement. There is no evidence that this is getting better over time.

[11] Source: Notes to Financial Statements December 31, 2008

[12] http://www.peacegarden.com/allpdf/Peace%20Garden%20History-%20Web[1].pdf