Internal Audit and Evaluation Documents

• Table of Contents
• 1 Background
• 2 Summary of Findings, Conclusions and Recommendations
• 3 Introduction
• 4 Evaluation Methodology and Constraints
  • 4.1 Methodology
  • 4.2 Lines of Evidence
    • 4.2.1 Program Logic Model and Underlying Rationale
• 5 Evaluation Findings
• 6 Recommendations
• 7 Management Response to the Recommendations
• 8 Appendices

4 Evaluation Methodology and Constraints

4.1 Methodology

The project team set out to evaluate the CHPIF program based on the following four criteria:

1. Relevance and Rationale
   
   
   • Likelihood of CHPIF meeting its objectives
     
     
2. Program Design
   
   
   • Design and Delivery,
     
     
   • Management Adequacy of Resources
     
     
   • Governance and Accountability
     
     
   • Performance Measurement and Reporting
     
     
3. Program Success and Impacts
   
   
   • Results measured against targets
     
     
   • Impact on the commercial heritage sector
     
     
   • Efficient use of resources
     
     
   • Client satisfaction
     
     
4. Cost Effectiveness and Alternatives
   
   
   • Pertinence of US style tax credit program

4.2 Lines of Evidence

Lines of evidence were used to address the study objectives and reach the desired results:

• Logic model analysis of the program
  
  
• Interviews
  
  
• Program documents and file review
  
  
• Literature review including a review of US tax credit program.

Each line of evidence has been integrated into the overall analysis and presentation of results in the section 'Evaluation Findings'. In the following section, we provide a summary description of the lines of evidence, sourcing and limitations encountered.

4.2.1 Program Logic Model and Underlying Rationale

CHPIF was developed as a key component program of and integral to the logic of the Historic Places Initiatives (HPI). The logic model (presented in section 5) is based on a review of the HPI, CHPIF program literature, the Terms and Conditions of the CHPIF program and Treasury Board submission. It also reflects comments and suggestions based on discussions and interviews with program officials. Thus, CHPIF goals and desired outcomes reflect the department's policy framework for heritage buildings as well as the interface with the HPI program.

4.2.1.1 Interviews

An extensive interview process was undertaken with key stakeholders in the program. In total, 40 interviews were completed, divided into the following categories:

• CHPIF staff, management and certification service providers (10)
  
  
• Applicants and owners (9)
  
  
• Private banks and financing institutions (1)
  
  
• Federal finance, tax and investment officials (5)
  
  
• Heritage developers (non applicants) (4)
  
  
• Knowledgeable individuals (2)
  
  
• Information requestors (non applicants) (4)
  
  
• HPI partners (provinces and territories) (5)

The interview process was subject to a number of caveats:

• Some individuals who were identified to be interviewed could not be reached¹⁹
  
  
• Not all questions were answered by all respondents.

As indicated above, the interviews were part of a qualitative research process and consequently, the results are not meant to be statistically representative of the categories.

4.2.1.2 Program Documents and File Review

The project team reviewed program performance data from program start-up in 2003-4 to the end of July 2006.

From internal CHPIF program data (the 'CHPIF program activity and financial commitments' report), the following key program activity parameters were examined:

• Application round
  
  
• Number of applications per round
  
  
• Average processing time from reception to pre-certification decision
  
  
• Average response time before contribution agreement
  
  
• Number of contribution agreements signed
  
  
• Estimated time from reception to project certification.

From internal CHPIF financial data, the following breakout of program costs was examined:

• 2003 expenditure review
  
  
• Payouts to certification service providers
  
  
• PCA Department costs
  
  
• PCH and PWGSC levies and other non-project costs
  
  
• Total CHPIF program costs

4.2.1.3 Literature Review

The project team reviewed the following literature, studies and reviews on the CHPIF program:

• Program literature and records
  
  
• Contribution agreements
  
  
• Previous evaluations and consultant reports
  
  
• Treasury Board submissions
  
  
• Documentation on alternative program configurations
  
  
• Documentation and key parameters of the US tax credit program

19 This was a particular issue with the federal partner agencies CRA and Finance. In light of the importance of these agencies to the future of CHPIF, a number of substitutions were made within the target groups.