Authors: Esther Plomp, Maria Cruz, Anke Versteeg
On the 14th of March 2019 the fourth VU Library Live talk show and podcast took place at the Vrije Universiteit Amsterdam (VU). By choosing topics that appeal to researchers and that are at the forefront of scholarly communications and research policy, this podcast series aims to bring researchers back to the library in the current age of digitalisation, where university libraries are becoming increasingly invisible to researchers. The topic of this show was the academic award system of the future.
We need to stop being lazy with just counting numbers of papers and citations and actually start reading stuff – Vinod Subramaniam
Vinod Subramaniam, Rector Magnificus of the VU,opened the show and claimed that we have lost sight of “the core business of the university, which is education”. Vinod stated that the academic reward system should not be based solely on research activities, let alone the traditional impact factor. As a researcher you can also have impact by communicating your results through newspapers and by providing guidelines that are used in society (such as medical guidelines or political policies). “We need to stop being lazy with just counting numbers of papers and citations and actually start reading stuff.”Vinod added that universities are at a turning point: “it is a perfect storm that is converging now, where I think a lot of things are happening where we as universities, but also grant agencies and other stakeholders have to start thinking about how do we reshape the reward system in academics”. Vinod also highlighted the timely occurrence of the meeting, right before the demonstration for education in the Hague on the 15th of March (WOinActie) against the increasing funding reductions in higher education and the very high workload of university staff.
it is a perfect storm that is converging now, where I think a lot of things are happening where we as universities, but also grant agencies and other stakeholders have to start thinking about how do we reshape the reward system in academics – Vinod Subramaniam
Maria Cruz opened the podcast with the phrase ‘publish or perish’, a strategy that is increasingly affecting the academic system, leading to high levels of workload and skewing research priorities. The current focus on publications as the golden standards in the evaluation process at Universities and research institutes decreases the value that education and valorisation activities have in scientific careers. The academic performance evaluation system furthermore barely takes any other academic outputs into account, such as software, data and other forms of communicating scientific research. Maria wonders if there is a way out of this system and asked: “Can we change this system to facilitate research that is open and transparent and contributes to solving key societal issues?”
Can we change this system to facilitate research that is open and transparent and contributes to solving key societal issues? – Maria Cruz
To address this question, Maria had four guest speakers around the table with her: Barbara Braams, Stan Gielen, Jutka Halberstadt, and Frank Miedema.
Barbara Braams, Assistant Professor at the VU, who recently wrote an opinion piece for theVolkskrant on the topic, agreed that we should move to a more transparent scientific system, but said that currently the amount of publications on your CV is more important when you write a grant, particularly the number of first author publications and the journal in which they are published. Barbara wants to change how researchers are evaluated and place the focus on not just sharing the articles but also scripts and data sets. “But to do so and to make sure that your data is actually usable by someone else, it takes a lot of time and effort…”, said Barbara. “ I think when you’re moving towards an open science system, we should also think about how can we reward these type of efforts, because it means that if I put a lot of effort in making my data understandable for someone else … it takes time away from my publications.” She argued that it should be clear for Early Career Researchers (ECR) in particular what it is that is expected of them and for what practices they are rewarded, as they have to deal with small time frames in which they have to write grant proposals. This will be important in the coming year, as NWO aims to sign DORA in 2019 but will implement the new indicators from 2020 onwards.
we should also think about how can we reward these type of efforts, because it means that if I put a lot of effort in making my data understandable for someone else … it takes time away from my publications. – Barbara Braams
Stan Gielen, President of The Netherlands Organisation for Scientific Research (NWO), agrees with Barbara that the evaluation of researchers needs to be revisited: “I don’t care where you publish, I want to know what was your contribution to science.” Stan claims that NWO wants to change the system, but that scientists are working in the international scene. This means that, according to Stan, if we want to change the system, this will have to be in cooperation with other funding agencies, at least in Europe. As a member of the Science Europe Governing BoardStan will seek agreement between EU funding agencies and will come up with general guidelines on how to evaluate scientists: “I expect that we will have a document available by the end of this year, which will be open for public consultation…. I hope that in let’s say, first half 2020, we will have some level of agreement among all the funding agencies in Europe about the basic criteria that should be used for evaluating the scientists,” he added. NWO also agreed to sign the San Francisco Declaration on Research Assessment (DORA, a set of recommendations to move away from assessing researchers using the impact factor) in September 2018, but has not yet signed the declaration. This will happen before the 23rd of May, when a meeting will take place to discuss the evaluation of researchers: “We decided to sign DORA and also to make statements what we are going to do before that meeting,” said Stan. “…We will also indicate at that time what [NWO] will do to implement DORA.”
On the question of how ECRs can be more actively involved in the process of change and the evaluation of researchers, Stan answers that The Young Academy is invited to participate actively in the consultation processes in the Netherlands and Europe. Stan added, “we are in close contact with VSNU, the Society of Dutch universities, because they’re talking now with the other communities about academic evaluation system. [NWO] should make sure that our criteria are overlap or are the same as those that are used by universities to evaluate the research component.”
Jutka Halberstadt, Assistant Professor at the VU, is of the opinion that the impact factor should not matter but that we should focus on what benefits society. Her work centres around valorisation. Jutka describes valorisation as “using our knowledge from teaching and research to help build a better society, to have societal impact.” She thinks that research should be available to society, and said that “we should make it understandable, usable to society… we should be in close contact with society to see what the societal needs are so we can translate those back to relevant research questions.” She adds that “in an ideal world [we should] also do the research in close collaboration with partners in society because I think that will make the research better. And for for me having societal relevance is a vital aspect of being an employee of the VU University.”
Jutka developed a national standard for obesity in collaboration with healthcare professionals and patient organisations in a project called ‘Care for Obesity’, funded by the Dutch Ministry of Health. No one knows how to measure the impact of these standards, even though they have enormous societal impact. Instead of prioritising publications in academic journals, the project produces other outputs such as blogs, questionnaires, guidelines and workshops for health care professionals: tools that are scientifically based and practical for people to use. Yet these research outputs are not valued at the same level as scientific publications. She thinks that universities should focus more on collaborative efforts and use altmetrics, e.g. how many times a name is mentioned in the news. “It won’t be a perfect system,” said Jutka, “but it’s something we can develop and work with.” She added that researchers should not be obliged to excel in education, research and valorisation at the same time, as this is “really a lot to ask” from them.
Barbara also highlights the current focus of the reward systems on the excellence of the individual. “I think one of the great things that we can also use from this transition to open science is make more space for other type of scientists and work as a team. But how are we going to do that, for instance, in a grant system?” Stan answers that NWO is not going to implement separate grants for different types of expertise, but recognises the worth of team-effort. For example, the Spinoza price (the highest award in Dutch science) is awarded to an individual, but as Stan puts it: “every person who gets the Spinoza price spends quite a lot of time explaining and thanking everyone who helped [them]”. Stan mentioned that the Spinoza price should therefore become a team award but did not comment on whether this change would actually be implemented. Instead, he thinks that “it’s up to the HRM policy of universities to make sure that they have excellent teams.” This may be difficult for universities to implement when the funders decide where the money flows.
Frank Miedema, Vice Rector Research and chair of the Open Science Programme Utrecht University and UMC Utrecht, thinks we are on the way to recognising the many forms of excellence. “[As a scientist] you want to produce real significant knowledge”, said Frank. According to him there are frustrations because this work is now not rewarded. “…a couple of years ago, this was still considered as taboo, especially at NWO,” he said, “But now we have it on the table”. When Maria asked if we should break free of the bibliometric mind-set, he said that this dependence on the impact factor has to stop, but that there are “many people who are addicted, especially of course, the people who do well in the system.” Frank is also involved in the evaluation of the current research evaluation protocol and thinks that more excellences need to be rewarded. He warned that the road is long and difficult and it may scare some people off, as scientists are very insecure when it comes to changes in the academic system. Frank raised the question whether the review committees of NWO were trustworthy.
It will take I think, 10, maybe 20 years until the mindsets of the reviewers have really changed – Stan Gielen
Stan indicates that it will take multiple years, perhaps even decades, before the new evaluation system is fully into place because the review panels have to incorporate these new instructions. Stan indicated that instructing reviewers has a positive effect, but said that “it will take I think, 10, maybe 20 years until the mindsets of the reviewers have really changed.” Stan indicates that the transitional phase will not restrict the rewards for more traditional scientists, because it would be too soon. Instead they will be implemented as the primary criteria in the next two years. Stan thinks that researchers will be triggered by these questions in the grant proposals: “we ask you to come up with a short narrative explaining first of all, why this is an important problem, what the impact will be scientifically, but also societal impact and why you or your team is qualified to pursue this project.” This can be done by listing your open science track record and shared data sets, next to the publications. Stan said that it is up to the researcher “to explain what you have done for open science and how you will pursue these activities when your grant application will be funded…” Barbara shedded some more light on the complexities of the transitional phase. She needs to ask for informed consent to share the data of her research, and after this it will still take 4-5 years before the data is available. Barbara explained that “there are so many differences in different fields. Some disciplines need more time to open up their data sets and panellist should be made aware of these differences,” she added.
We ask you to come up with a short narrative explaining first of all, why this is an important problem, what the impact will be scientifically, but also societal impact and why you or your team is qualified to pursue this project. – Stan Gielen
When Barbara asks how universities are going to support their staff in the transition to open science, Stan answered that “we need data stewards, because we should not …. put the burden on the scientist.”
We need data stewards, because we should not put the burden on the scientist – Stan Gielen
The answer to Maria’s final question on what they would advise young scientists to do with traditional supervisors is to take matters into their own hands. Young scientist should follow their hearts, as Jutka puts it. Barbara agreed and says that the young generation will move this forward. Stan thinks supervisors should let young people bring in their own expertise: “if you stick to your own principles, you will be lost in four years.” Frank however, has a different view. “I think to put the burden on the young academics … [is] not really fair, because they have the least power in the system,” he said. Frank thinks that that deans and rectors need to set steps in this “power game” and decide on the right incentives for the academic leaders. The move towards a more transparent, open and societal relevant way of practising science thus requires the effort of all stakeholders: researchers at all academic career levels as well as support staff, funding agencies, universities, libraries and the involvement of the general public.
Links to the VU Library Live podcasts: