Responsible use of Publication Forum classification

It is becoming increasingly common to use quantitative indicators describing the quantity, quality and impact of publication activities to evaluate research. At the same time, there is a growing need for guidelines on the responsible use of research metrics.

The steering group of the Publication Forum (known by its Finnish acronym, JUFO) has updated the instructions for using the JUFO classification, which is intended to provide an evaluation of the average quality of the large numbers of publications produced by universities but has, in recent years, been used for evaluations and comparisons at unit and researcher level. 

Unit and researcher level evaluations should be based primarily on expert evaluation

The new guidelines include recommendations on using the classification at unit and researcher level, as well as information on the limitations related to the use of the classification for evaluating research. In addition, the classification is described, along with the background assumptions and the role of the classification in the university funding model. The guidelines have made use of international statements on responsible metrics: the Declaration on Research Assessment (DORA), the Leiden Manifesto for Research Metrics and the Metric Tide report. The instructions supplement the national recommendations on the Responsible Evaluation of Researchers, as well as the guidelines on Responsible Research Metrics.

No absolute JUFO criteria for assessments

The user instructions emphasise the fact that unit and researcher level evaluations should be based primarily on expert evaluation. If a JUFO classification is used to support an evaluation , the limitations on using JUFO classifications should be considered, other indicators should also be used in accordance with the field of study in question, and the competences of experts in the metric should be utilised. Transparent information should be provided to personnel on which contexts and how JUFO classifications is used, and researchers’ viewpoints on the suitability of JUFO classifications for evaluation should be heard.

The instructions provide examples of unit and researcher level evaluation for which JUFO classifications are suitable – and ones for which it is not recommended. For example, it is not recommended to set absolute criteria and targets based on JUFO classifications.

The JUFO classification is a responsible research metric

The use of JUFO classifications in the Ministry of Education and Culture’s university funding model is based on an evaluation of the average quality of large numbers of publications. The publication indicator included in the financing model and the JUFO classification fulfil the principles for responsible research metrics: stability, transparency, diversity and reflectivity.

The publication indicator included in the financing model and the JUFO classification fulfil the principles for responsible research metrics: stability, transparency, diversity and reflectivity.

Stability is enabled by comprehensive and reliable publication data , which includes all peer-reviewed publications produced by universities. The publication data and the JUFO classification are both transparently and openly available. 

Diversity is supported by taking different publication types into consideration in the funding model, irrespective of the country and language of publication, and by balancing the JUFO classifications between different fields of science. In addition, the funding model uses several other indicators of education and research.

Monitoring changes in publication activities, research and investigation work, and public debate all serve to fulfil the requirement for the reflectivity of research metrics. The suitability of the indicators used in the financing model is evaluated regularly, and the JUFO classification is also updated and continuously developed.

The JUFO classification is a classification of the standard of scientific publication channels intended to support the assessment of research quality. The classification was created by the Finnish scientific community. The classification divides peer-reviewed publication channels into three classes based on the average quality and influence of publications, with class 3 being the highest. The JUFO classification is used as a tool in the university funding model. 

Eeva Savolainen is a Senior Planning Officer at the Publication Forum. Janne Pölönen is Head of Planning at the Publication Forum.

_{Further information:}

_{The Publication Forum’s user instructions: http://www.julkaisufoorumi.fi/fi/kaytto
Kosten, J. (2016). A classification of the use of research indicators, Scientometrics 108:457–464 https://doi.org/10.1007/s11192-016-1904-7
Leiden Manifesto for Research Metrics https://doi.org/10.1038/520429a
Metric Tide report: https://doi.org/10.13140/RG.2.1.4929.1363
San Francisco Declaration of Research Assessment: https://sfdora.org
Sugimoto, Cassidy R. & Vincent Larivière: Measuring research: what everyone needs to know. Oxford University Press 2018.
Wahlfors, L. & Pölönen, J. (2018). The use of the Publication Forum classification at universities, Finnish Association for Administrative Studies 37(1):7–21.}