Evaluating European Commission NMP and Horizon 2020 SME and FTI Proposals (2008 - present)
Richard Ball acts as an independent expert for the evaluation of SME Instrument, Fast Track to Innovation (FTI) and, previously, FP7 NMP proposals on behalf of the European Commission (EC).
Working individually, the task for each expert is to assess proposals against specific criteria, with explanatory comments encapsulated within an ‘individual evaluation report (IER)’.
After carrying out an individual evaluation, an expert will join other experts who have evaluated the same proposal in a consensus group, to agree on a common position, including comments and scores.
After the consensus phase a panel of experts:
- Reach an agreement on the scores and comments for all proposals within a call, checking consistency across the evaluations.
- If necessary, propose a new set of marks or revise comments, and resolve cases where evaluators were unable to agree.
- Rank the proposals having a qualifying score and give a priority order for proposals with the same score
Before notifying coordinators of the final evaluation results, the EC reviews the results of the experts’ evaluation and puts together the final ranking list.The EC then:
- Produces a ranked list of proposals, including all proposals with scores above the qualifying score.
- Draws up a list of proposals for possible funding from that list. The number of proposals in the list depends on the available budget.
After the finalisation of the evaluation, applicants receive the Evaluation Result Letter (ESR).
Observer - Reviewing Past Evaluation Processes and Providing Recommendations on Improvement for Future European Commission FTI Programmes (2018 - present)
Richard Ball acts as an observer for the H2020 European Innovation Council (EIC) Fast Track to Innovation (FTI) programme. In this role he:
- Observes the process of the evaluations (Individual Expert Reports (IER) and Evaluation Summary Reports (ESR).
- Selects proposals from each call deadline and from across the 15 keyword panels.
- Reviews and make suggestions on the evaluation grid and how it could be improved.
- Reviews and comments on the approach in general with a view to the future.
The observations are carried out in two stages. First, a review of the briefing material to ensure familiarity with the call process and coverage and to assess the quality of the briefing material. Secondly, selected evaluations were accessed to look at the comments and markings that assessors had provided and assess the ways that individual evaluators had interpreted the guidance notes. Comments and scores are also reviewed for consistency, with particular attention paid to large scoring variations, especially where one evaluator’s scoring or comments are vastly different from the others. A number of conclusions and recommendations are then drawn from across the following subcategories:
- Evaluator Markings and Evaluator Comments.
- Scale and Complexity and Transparency.
- Throughput Time, Efficiency, Reliability and Usability.
- Impartiality, Fairness and Confidentiality.
- Conformity with Rules.
- Quality of Process and Overall Quality of Evaluation.
- Quality of Briefing Materials and Quality of Scoring Sheet.
- Quality of Monitor’s Report Template.
Independent Review of the Technology Developer Accelerator Programme (2015-2019)
The Advanced Propulsion Centre (APC) has developed and implemented the Technology Developer Accelerator Programme (TDAP), a programme aimed at small and medium sized technology developers (TDs) with low carbon propulsion technologies that can be applied to the automotive sector. Working through Beta Technology Ltd, Keston Technologies has undertaken several independent reviews of the TDAP, in successive waves, including an assessment of economic impact and an opinion on its design and implementation.
Project Technical Advisor (PTA) for the European Commission on collaborative R&D projects (2012 – 2017)
On behalf of the European Commission, and through Beta Technology Ltd, Keston Technologies provided project monitoring services for six European-funded projects.
These were large, multi-partner, multi-national projects lasting between three and five years and with budgets ranging from 5m to 15m Euros each. Most of the activity was undertaken remotely but did entail meetings with the project consortia on a six monthly basis.
Monitoring and Evaluation of the UK’s Micro and Nano Technologies Facilities Network (2007-2012)
The UK Government established several facilities offering world class MNT technology. The aim of these facilities was to understand and stimulate their markets to generate growth as well as developing technology. Richard Ball was assigned as a monitor to several of these projects for a six year period, with the monitoring having both a strategic and business focus. The monitoring focused on impact & additionality, business planning, management & governance and business development.
Development of the PTA monitoring framework for the European Commission (2011)
Working with Beta Technology Ltd, Keston Technologies assisted in the development of the framework through which a large number of European funded projects were monitored and evaluated. The requirement was to provide a framework to assist the EC directorate to monitor the progress on projects funded in Theme 4 – Nanosciences, Nanotechnologies, Materials and new Production Technologies within the 7th Framework Programme. The framework ensured that the participants in the projects performed correctly their activities to the best possible program requirements.
The monitoring framework that was developed operated to:
- Provide advice to the EC Project Officers;
- Monitor the progress of the projects;
- Assess periodic and final reports of the projects;
- Propose workable solutions to any contractual and/or managerial matters;
- Develop links between projects of a similar nature.
Development of a pilot environmental technology evaluation scheme as part of the European Environmental Technologies Action Plan (ETAP) (2006-2009)
Through Beta Technology Ltd, Keston Technologies worked on the development of an evaluation scheme for environmental technologies on behalf of the European Commission.
A total of 15 technologies were put through the pilot scheme in three areas – waste water, soil remediation and energy technologies (renewables and energy efficiency).
The project results have fed into the process of developing a proposed mechanism and recommendations for a European-wide evaluation scheme.
The project involved collaboration at international level with other evaluation schemes in the USA and Canada.
Recommendations from the pilot were used to draft a policy proposal.
Other Monitoring and Evaluator Roles
Keston Technologies has also been responsible for evaluating proposals for various other funding schemes and monitoring projects throughout Europe and Australia. Most notably, these include:
- International expert for the evaluation of the project “Centre for Research and Utilization of Renewable Energy - CVVOZE” on behalf of the Czech Ministry of Education, Czech Republic (2013).
- Expert Evaluator of projects supported under the Priority Axes 1 and 2 OP RDI.
- Expert evaluator for Czech-Norwegian Research Programme (CZ09).
- Eurostars (Eureka Programme) evaluations (2011 to present).
- The Flanders ‘Strategic Initiative Materials’, Belgium.
- The Czech Ministry of Education, Czech Republic, (evaluator for a call for proposals for regional R&D centres in the Czech Republic – and member of Panel of International Experts, allocating €250m of funding to technology R&D centres in a variety of disciplines, including biotechnology, ICT, engineering and materials science (2009 to present)).
- The Great Southern Development Commission, Australia, Independent Assessor for the evaluation of grant submissions under the Regional Grant Scheme (2005 to 2016).
Keston Technologies is based in Oxford, UK and Albany & Sydney, Australia
