IGF::OT::IGF : An Innovative Approach for Peer-Review. Period of Performance: September 15, 2017 - March 14, 2018. N43DA-17-1214. BASE AWARD.

Pevals, Llc, Tallahassee FL

Investigators

Abstract

Under this Small Business Innovation Research (SBIR) Phase I project, Research Topic 162, the contractor will investigate the effectiveness of new peer-review methods, Concordance Index based Global Ranking (CIGR) and Modified Borda Count (MCB), to enhance professional evaluation and decision making in life sciences management and administration. The intended result is a novel set of metrics that can be used by NGOs/disease foundations, advocacy groups, research funders, policy makers, and by academic institutional bodies. This SBIR phase I project has the following objectives: 1. Simulation studies to investigate the effectiveness of the method used in the NSF pilot study and the contractor's new method, CIGR. The contractor will perform simulation studies to investigate the effectiveness of the new review method in terms of how it can recover the true ranking of proposals or papers. The contractor will also study which parameter settings are optimal for the new review method. 2. Testing the new approach in real evaluation settings The contractor has implemented the method used in the NSF?s pilot study as an online platform at http://pevals.com. The contractor will implement our new ranking method, CIGR, in the phase I project and test the new peer-review approach in graduate student course project/essay evaluation at Florida State University. In these tests, the true rankings and scores of the project reports or essays will be obtained from the course instructor and used as the true rankings. After collecting the ranking from individuals, we will use both CIGR and the modified Borda Count (MBC) score, proposed originally for the new peer-review method and also used in the NSF?s pilot study, to obtain the final rankings and compare the accuracy of the two different ranking methods based on their agreement with the true rankings. This test is unique in the sense that it has the true rankings obtained from the course instructors, which will help us to evaluate the effectiveness of different ranking algorithms. This test of using graduate student course projects will establish solid basis for our future tests on scientific papers and proposals. The contractor will also test our method for a conference paper evaluation for a workshop in IEEE BIBM 2017 conference.