Advising on the commercial impact of technology and
    market changes in telecoms, software and IT services

Will Barrar, Consulting Director

Ovum has a highly experienced team of analysts and consultants who have worked on government projects around the world.

Technical and general evaluation of the local electoral pilot schemes in England 2003

In December 2002, a number of local government authorities in England applied to the Office of the Deputy Prime Minister (ODPM) to run pilot schemes in the local government elections, on 1 May 2003. These pilot schemes included all-postal voting; voting using the Internet, digital television, telephone and text messaging; and electronic counting.

The Electoral Commission required an evaluation report on each local electoral pilot scheme, to be run by the successful local authority applicants, with an electronic voting or counting component. The evaluation of e-based pilot schemes had to cover both technical and non-technical issues. Ovum provided evaluation services to the Electoral Commission for six pilot schemes.

Aims of the evaluation

The evaluation reports of the e-based pilot schemes assisted the Electoral Commission in its aims to:

• Determine, independently of Government and the local authorities, the extent to which each pilot scheme had met its objectives and the extent to which the overall pilot programme was meeting these objectives

• ensure that lessons were learned from evaluation and that results are used to inform and improve the design and implementation of future pilots

• determine the extent to which the pilot schemes represented value for money

• establish whether key stakeholders, especially those intended to benefit, have done so and whether they themselves perceive the new electoral arrangements to have been beneficial.

In accordance with the requirements of the Representation of the People Act (RPA) 2000, The Electoral Commission was required to publish individual evaluation reports on each pilot scheme within three months of the election. The reports were required by law to include a description of the scheme and an assessment against five set criteria. These covered:

• the scheme’s success or otherwise in facilitating voting or the counting of votes, or in encouraging voting or enabling voters to make informed choices at the elections

• whether the turnout of voters was higher than it would have been had the scheme not taken place

• whether voters found the procedures provided for their assistance by the scheme easy to use

• whether the procedures provided for by the scheme led to any increase in impersonation or other electoral offences, or in any other malpractice in connection with elections, and

• whether those procedures led to any increase in expenditure, or to any savings, by the authority.

In addition to these statutory criteria, the Electoral Commission also required the evaluation to specifically address:

• the extent to which the pilot facilitated or otherwise encouraged participation among particular communities, including young people, ethnic minority groups and people with disabilities;

• overall levels of user awareness and comprehension of the voting method being tested, including an assessment of the effectiveness of any literature or other materials used in the promotion of the pilot;

• the attitudes and opinions of key stakeholders, including voters, with a view to determining overall levels of confidence in the voting method being tested;

• whether the pilot resulted in measurable improvements, or had any adverse impact, with respect to the provision of more efficient and effective service delivery to voters;

• whether the pilot resulted in measurable improvements to, or had any adverse impact on, the existing system of electoral administration;

• whether the pilot represented good ‘value for money’;

• lessons for future pilot schemes.

Technology Assessment

The ODPM contracted with a number of technology suppliers for the delivery of their solutions as part of a framework agreement. In addition, pilot authorities were able to contract with non-framework suppliers at their own expense.

For e-voting and e-counting pilot schemes, evaluation therefore also had to address:

• the degree to which the suppliers met the requirements of their contract

• the effectiveness of the operation of the entire technology solution over the election period, particularly with respect to accuracy, reliability, robustness and security

• the effectiveness of the security and other control measures in place

• the total project management of the pilot, covering both supplier and local authority components, including risk management, configuration management etc.

• the adequacy of the Quality Assurance (QA) process, including QA performed by suppliers, local authorities and ODPM, and

• the actions taken by suppliers as a result of the QA process.

How Ovum’s experts helped

As a company with strong consulting expertise, particularly within the public sector, Ovum understood the critical importance for consultation with all key stakeholders in this project. This is even more crucial where Government is sponsoring new technology and development that will have significant impact on the lives of citizens.

Ovum therefore worked very closely with the local authorities to understand how each of their pilots were progressing prior to and on election day itself. We also liaised with technology suppliers and Quality Assurance contractors in order to track critical developments, and to assess technological performance including security, usability and accessibility.

Our findings were provided to the Electoral Commission in the form of individual authority reports, and a combined strategic assessment of the electoral pilot schemes covered by Ovum.

Contact Will Barrar directly on WXB@Ovum.com or by phone on +44 (0) 20 7551 9084.