Mel'nikov R.M.Russian Presidential Academy of National Economy and Public Administration, Moscow, Russian Federation rmmel@mail.ru
Importance The article compares international and Russian methodological approaches to evaluation of return on public investments in research and development programs, and underpins proposals for their enhancement. Objectives The study aims to reveal reserves to improve the Russian practice of evaluation of public investment efficiency in research and development programs. Methods The study draws upon the methodology for designing the logic models to transform the inputs of research and development programs into their outcomes, and a mathematical technique of linear programming. Results I developed a unique technique to assess the comparative efficiency of R&D projects and subprograms based on the synthesis of the U.S. Advanced Technology Program's logic model and the data envelopment analysis model by A. Charnes, W. Cooper and E. Rhodes. The advantage of the technique is a balance between program results evaluation areas (outputs, outcomes and impacts). Conclusions To enhance the efficiency of public investments in R&D programs, it is required to shift the emphasis to preliminary assessment and subsequent monitoring of economic results of innovators using knowledge created and disseminated within the program execution. I advocate a linear programming model as a means of detecting the most efficient projects and subprograms at the formation and execution phases of the R&D program. The model enables to reveal how far the project (or subprogram) is from the efficient border of inputs transformation into outputs, outcomes and impacts.
Keywords: research and development program, public investment, logical model, envelopment analysis
References:
Fahrenkrog G. et al. RTD Evaluation Toolbox: Assessing the Socio-economic Impact of RTD Policies. Seville, European Commission Joint Research Centre, 2002.
Ruegg R., Feller I. A Toolkit for Evaluating Public R&D Investment. Models, Methods, and Findings from ATP's First Decade. Gaithersburg, National Institute of Standards and Technology, 2003.
Ruegg R., Jordan G. Overview of Evaluation Methods for R&D program. A Directory of Evaluation Methods Relevant to Technology Development Programs. U.S. Department of Energy, Office of Energy Efficiency and Renewable Energy, 2007.
Reed J., Jordan G., Vine E. Impact Evaluation Framework for Technology Deployment Programs. US Department of Energy, Office of Energy Efficiency and Renewable Energy, 2007.
Marjanovic S., Hanney S., Wooding S. A Historical Reflection on Research Evaluation Studies, Their Recurrent Themes and Challenges. Santa Monica, RAND Corporation, 2009.
Guthrie S., Wamae W., Diepeveen S., Wooding S., Grant J. Measuring Research. A guide to Research Evaluation Frameworks and Tools. Santa Monica, RAND Corporation, 2013.
Evaluating Outcomes of Publicly-Funded Research, Technology and Development Programs: Recommendations for Improving Current Practice. American Evaluation Association, 2014.
Kapeller J. Citation Metrics: Serious Drawbacks, Perverse Incentives, and Strategic Options for Heterodox Economics. American Journal of Economics and Sociology, 2010, vol. 69, no. 5, pp. 1376–1408.
Jaffe A. Economic Analysis of Research Spillovers: Implications for the Advanced Technology Program. Gaithersburg, National Institute of Standards and Technology, 1996.
Mansfield E. Social and Private Rates of Return from Industrial Innovations. Quarterly Journal of Economics, 1977, vol. 91, no. 2, p. 221–240.
Methodology for Technology Tracking and Assessment of Benefits. In: Industrial Technologies Program: Summary of Program Results for CY 2009. US Department of Energy, 2009.
Gilmour J., Lewis D. Assessing Performance Assessment for Budgeting: The Influence of Politics, Performance, and Program Size in FY 2005. Journal of Public Administration Research and Theory, 2006, vol. 16, no. 2, pp.169–186.
Gilmour J. Implementing OMB's Program Assessment Rating Tool (PART): Meeting the Challenges of Integrating Budget and Performance. OECD Journal on Budgeting, 2007, vol. 7, no. 1, pp. 1–40.
Guide to the Program Assessment Rating Tool (PART). Washington, Office of Management and Budget, 2007.
Lane J. Lets's Make Science Metrics More Scientific. Nature, 2010, vol. 464, pp. 488–489.
Lane J., Bertuzzi S. Research Funding. Measuring the Results of Science Investments. Science, 2011, vol. 331, pp. 678–680.
Largent M., Lane J. STAR METRICS and the Science of Science Policy. Review of Policy Research, 2012, vol. 29, no. 3, pp. 431–438.
REF 2014: Assessment Framework and Guidance on Submissions. Bristol, HEFCE, 2011. Available at: Link.
REF 2014: The Results. Bristol, HEFCE, 2014. Available at: Link.
Charnes A., Cooper W., Rhodes E. Measuring the Efficiency of Decision Making Units. European Journal of Operational Research, 1978, vol. 2, iss. 6, pp. 429–444.
Charnes A., Cooper W., Lewin A., Seiford L. Data Envelopment Analysis: Theory, Methodology and Applications. Boston, Kluwer, 1994.
