SPR Studies in a Balanced Scorecard Framework

SPR has conducted many public sector studies within a Balanced Scorecard (BSC) Framework, several of which have had major program, organizational development, or public policy impacts.  Generally, these studies have focused on measuring systemic changes and making strategic changes for future improvements, based on the linkages between organizational results and capacity (resources) and internal processes.  Such studies usually move away from a classic "measurement perspective," typical of evaluations, and towards a "management perspective" which emphasizes performance improvement.

These studies also require a causal model of key processes, or (in the BSC framework) strategy maps, so that findings can show how management decisions can improve performance.  Such studies often focus on "the bottom-line" (profits) in the private sector, but more broadly on the achievement of goals or "results" in the public sector.

BSC studies differ from other studies in that they generally have been developed within organizations which are engaged in major change and improvement initiatives.  Thus, such studies have generally required strong usership and leadership from the client organizations. BSC studies have also required the development of broader organizational or multi-constituency buy-in to a change process (with dissemination of results and, ideally, impact throughout the organization).  Many of these studies have also been multi-year efforts resulting in major organizational or public program development.

Balanced scorecard-type studies which SPR has implemented include:

  • Evaluation of the Victim/Witness Assistance Program (V/WAP):  Utilizing a multi-method approach, this balanced scorecard-type evaluation assessed the quality and effectiveness of the program which provides assistance in court to 300,000 Ontarians annually, mainly women who are victims of family violence. The evaluation identified success factors such as effective staffing and also assessed future priorities. The project involved a web survey of 200 V/WAP staff, Crown Attorneys and 500+ agencies (community services, police, etc.) and identified important issues in the program, including worker burnout (for the Ontario Ministry of the Attorney General, 2010-2011);

  • RCMP Performance Measurement:  This project provided baseline data for the RCMP's performance measurement within a balanced scorecard framework. The project included a number of multi-method surveys (on-line and mail). A Canada-wide mail-out survey of 30,000 households asked about awareness, experience, and satisfaction regarding services received from the RCMP. Particular attention was given to Western and Atlantic Canada, and the Territories, where the RCMP provides a wider range of services. Over 9,000 RCMP employees were surveyed on-line, as well as policing agencies, clients of the RCMP's contract policing (selected provinces, territories, municipalities and First Nations), and international policing partners (for the RCMP, 2003-2004);

  • Strategic Review of Exchange Programs.  This review examined a number of youth exchange programs, particularly interprovincial reciprocal visits between high school youth.  The review reflected a balanced scorecard-type framework by examining processes in a through-time context.  Research examined resources, reach, impact and future potential and involved interviews, focus groups, and surveys with exchange providers and youth in every province and territory.  Study results showed that youth exchanges achieve important impacts on national identity, use of official languages, and social and work skills of youth.  This study aided the development of Exchanges Canada, a new program providing over 100,000 youth exchanges per year (for Canadian Heritage, 1997-2000);

  • Evaluation of the Federal Labour Code:  Phase I.  This project, implemented in a 'balanced scorecard' type framework, examined labour standards as a tool for the achievement of broader labour and economic goals of the Government of Canada.  Phase I of the evaluation (1996-98) examined how human and technological resources affected bottom-line outcomes in complaints and costs under the Canada Labour Code.  The evaluation focused on the need for improved information processes to aid the reduction of employer infractions. Key data sources included employer, union, worker and staff satisfaction surveys.  A strategy to allocate new resources to Labour Code communications and information was approved as a result of this project (for Human Resources Development Canada, 1996-1998);

  • Evaluation of the Special Measures Initiatives Program.  This balanced-scorecard-type assessment focused on success in achieving employment equity goals in the Federal Public Service. This was achieved by examining program finances, departmental access and buy-in to the program, client satisfaction, and the impact of some 161 innovative projects on specific Federal Departments and agencies. Data collection included administrative information, financial data, focus groups and stakeholder consultations, which were conducted across Canada. The evaluation resulted in a highly-modified business relationship between Treasury Board and the Public Service Commission (for Treasury Board of Canada Secretariat, 1997-98); and

  • Review of Occupational Health and Safety Training in Ontario.  This study, implemented in a balanced scorecard framework, examined program dynamics and future directions for Ontario's twelve health and safety organizations -- a program with a budget of over $55 million in 1991. The review examined resources, efficiency, staff capabilities and goal achievement under the Occupational Health and Safety Act. Sub-studies examined staff satisfaction, resources, and organizational capacity. Related research continued over a five-year period. The review aided the development of Canada's largest training program in Occupational Health and Safety (with over 300,000 workers and managers certified since 1994) (for the Ontario Workplace Health and Safety Agency, 1994-96).