Development and piloting of a self-assessment measure of core research competencies for palliative care clinicians: the RESPACC project

Oana Predoiu1 , Aliki J. Tserkezoglou2 , Sheila Payne3 , Daniela Moșoiu4,1, María Arantzamendi5 , Marina Martínez García5 , Julie Ling6 , Mary Dionisi2 , Stephen Mason7 , Carlos Centeno Cortes5 1Fundația HOSPICE Casa Speranței, Brașov, România 2 “Galilee” Palliative Care Unit. Athens, Greece 3 International Observatory on End-of-Life Care, Faculty of Health and Medicine, Lancaster University, Lancaster, United Kingdom 4Universitatea Transilvania Brașov, România 5 Institute for Culture and Society, ATLANTES Research Group, University of Navarra, Pamplona (Navarra), Spain 6European Association for Palliative, Luchthavenlaan, Belgium 7 Institute of Life Course and Medical Sciences, University of Liverpool, United Kingdom Received: 10.03.2023 • Accepted for publication: 5.04.2023



Abstract RESPACC was an Erasmus + funded project to develop a framework of core research competencies for palliative care (PC) clinicians and supporting education materials ( The supporting materials included a self-assessment quiz to guide PC multidisciplinary team members on areas of research understanding that needed further development. The objective was to pilot the self-assessment quiz developed based on the research competency framework for PC clinical teams and test it for face validity and clarity. Methods: A pilot observational study was conducted to test the self-assessment quiz for face validity and clarity. The self-assessment quiz was developed between May-November 2021. It was structured to include questions about each of the competencies in the seven domains of the framework, using the examples. Items within the quiz were responded on a 5- points ordinal scale, with participants identifying their perceived competency as: Novice, Advanced Beginner, Competent, Proficient, Expert (each clearly defined). The quiz was developed in English and then translated into the national languages of partners (Greek, Romanian, Spanish). The quiz was piloted with Romanian PC professionals attending the National Palliative Care Conference. Questions about clarity of the quiz, suggestions for improvements were asked together with some demographic data about participants. Results: Multidisciplinary PC clinicians participated in the pilot testing (18 physicians, 4 nurses, 2 psychologists and 1 physiotherapist). Most (n=15) had less than one year of experience in PC. Participants identified themselves as: Novice (n=5), Advanced beginner (n=9), Competent (n=5), Proficient (n=3), Expert (n=3). All respondents answered all question without difficulties and considered the quiz to be clearly written, well structured and adequate. Conclusions: The self-assessment quiz is easy to use, and may be useful in helping members of PC teams assess their competencies and identify their research education needs. Keywords: pilot test, competency, research, palliative care Rezumat RESPACC a fost un proiect finanțat de Erasmus + pentru a dezvolta o matrice de competențe de cercetare de bază pentru profesioniștii din domeniul îngrijirii paliative și materiale educaționale ( Materialele educaționale au inclus un chestionar de autoevaluare pentru a ghida membrii echipei multidisciplinare de îngrijire paliativă cu privire la domeniile de înțelegere a cercetării care necesită o dezvoltare ulterioară. Obiectivul a fost de a pilota chestionarul de autoevaluare elaborat pe baza matricei de competențe în cercetare pentru echipele de îngrijire paliativă și de a testa validitatea facială (face validity) și claritatea acestuia. Metode: A fost efectuat un studiu observațional pilot pentru a testa chestionarul de autoevaluare în ceea ce privește validitatea facială și claritatea PALIAŢIA Vol. 16 Nr. 2 Aprilie 2023 Predoiu O. et al ISSN 1844 – 7058 6 Fundația HOSPICE Casa Speranței Chestionarul de autoevaluare a fost dezvoltat în perioada mai-noiembrie 2021. Acesta a fost structurat pentru a include întrebări despre fiecare dintre competențele din cele șapte domenii ale matricei, folosind exemplele de la fiecare competență. La itemii din cadrul chestionarului s-a răspuns pe o scară ordinală în 5 puncte, participanții identificând competența percepută ca fiind: Novice, Începător avansat, Competent, Experimentat, Expert (fiecare fiind clar definită). Chestionarul a fost elaborat în limba engleză și apoi tradus în limbile naționale ale partenerilor (greacă, română, spaniolă). Chestionarul a fost testat cu profesioniști români din domeniul îngrijirii paliative care au participat la Conferința Națională de Îngrijiri Paliative. Au fost adresate întrebări privind claritatea chestionarului și sugestii de îmbunătățire, împreună cu unele date demografice despre participanți. Rezultate: La testarea pilot au participat profesioniști din echipa multidisciplinară de îngrijirii paliative (18 medici, 4 asistente medicale, 2 psihologi și 1 fizioterapeut). Majoritatea (n=15) aveau mai puțin de un an experiență în domeniul îngrijirii paliative. Participanții s-au identificat ca fiind: Novice (n=5), Începător avansat (n=9), Competent (n=5), Experimentat (n=3), Expert (n=3). Toți respondenții au răspuns la toate întrebările fără dificultăți și au considerat chestionarul ca fiind scris clar, bine structurat și adecvat. Concluzii: Chestionarul de autoevaluare este ușor de utilizat și poate fi util pentru a-i ajuta pe membrii echipelor de îngrijire paliativă să își evalueze competențele și să își identifice nevoile de formare în domeniul cercetării. Cuvinte cheie: pilotare, competență, cercetare, îngrijire paliativă ______________________________________________________________________________________________ Introduction RESPACC was an Erasmus + funded project designed to develop a framework of basic research competencies for palliative care clinicians and supportive education materials ( The notion of competence corresponds to the ability to apply knowledge and skills to successfully perform an activity at work (1). The RESPACC Framework (2) was developed together with a self-assessment quiz to guide palliative care clinicians on areas of research competence that need further development or improvement (3). As one of the core competencies in palliative care, research is important as there is a need: – to identify and address relevant problems for patients in palliative care; – to establish a baseline of what is good practice Thus, improved research competence in clinicians working in palliative care may enable them to engage in using and designing research to continue improving clinical practice. The framework and competency model may also be adapted in other clinical specialties. The focus of the framework was on enhancing core research competences for all clinicians working in multidisciplinary palliative care teams. Hence, some competencies could be considered essential within the team, but not essential for every team member. In the development of the palliative care competency research framework, there were three phases. First, after a literature review, eight competency research frameworks were identified and analysed (1), (4), (5), (6), (7), (8), (9), (10). Based on the Global Competency Framework for Clinical Research (TDR) (1) from the World Health Organisation, a seven-domain framework for palliative care multidisciplinary team was developed. In the 2nd phase, nominal group technique was utilised in three countries to establish the views and priorities of palliative care clinicians regarding the development of clinical research competencies. The nominal group technique is a consensus method that involves mixed methods approaches, following eight sequential steps normally conducted in a single face-to-face session (11). In the present study, due to pandemic context, the nominal group technique needed to be adapted for online format (12). The 3rd phase consisted of consultation with international palliative care professionals, with respondents from 17 European countries. Results of the review, nominal groups and experts’ consultation were integrated into the final framework format ( The findings are presented in English, Spanish, Greek and Romanian, with the domains and competencies presented in Figure 1. Vol 16 / Nr. 2 / Aprilie 2023 ISSN 1844 – 7058 7 Fundația HOSPICE Casa Speranței Figure 1 – Domains and competencies of RESPACC Framework Professionals in palliative care may consider themselves competent in skills on care management, awareness, and professional leadership, but less competent in the use of research (13). A quiz could help them to appraise their level of comprehending research and the areas where they need improvements, hence, the self-assessment quiz was developed to measure the levels of research literacy and competencies for clinicians, assess educational needs and guide competency-based training requirements (3). The quiz was designed to be suitable for all palliative care professionals who are part of a multidisciplinary team. The objective was to pilot test the self-assessment quiz developed based on the research competency framework for palliative care clinical teams. Material and Methods Study design A pilot observational study was conducted to test the selfassessment quiz for face validity and clarity. Quiz development The quiz items were developed in May-November 2021. Questions were drafted for each competency from the seven domains of the framework, using examples provided for each competency. All questions were reviewed by the authors during four meetings, over 6 months. In an iterative process, each person provided feedback, and the quiz was updated. At piloting, the quiz had 39 items grouped in the 7 domains of the framework, in electronic format. A 5-points scale with the following categories Novice, Advanced beginner, Competent, Proficient and Expert was used by palliative care clinicians to self asses their competency in clinical research in palliative care. The five categories were each clearly defined as: Novice: no previous experience of research but is aware of its implications. Advanced Beginner: some knowledge of research and can perform some activities with assistance and supervision. Competent: basic knowledge of research and can undertake research tasks without support. Proficient: good knowledge of research, has an understanding of the activities and tasks involved and can undertake them. Expert: comprehensive knowledge of research and can mobilize and coordinate resources for its implementation. Respondents were advised to check Novice for questions that have terminology they do not fully understand. Upon completion of the quiz, feedback was provided according to the summative score. The average time for completion was approximately 20 minutes. Respondents were asked to express their opinion about: – The clarity of instructions to fill in the quiz – The clarity of the titles of domains – The clarity of the quiz items PALIAŢIA Vol. 16 Nr. 2 Aprilie 2023 Predoiu O. et al ISSN 1844 – 7058 8 Fundația HOSPICE Casa Speranței Sample Palliative care clinicians attending the RESPACC workshop at the Romanian National Palliative Care Conference, 22-23 Oct 2021, were invited to participate. Data collection Participants at the RESPACC workshop received access to the electronic format of the quiz, for completion. The pilot version included questions about clarity of the quiz and suggestions for improvements and demographic data. Participation in the pilot was optional and, by completing the quiz, respondents gave consent to use their personal information in data analyses. Those that completed the quiz were provided with feedback about their summative scores. Analysis Descriptive analysis of the responses was performed in an Excel file and statistical analysis was done with software GraphPad InStat 3.0. As this activity related to the development of the core competency framework for palliative care clinicians which was part of the RESPACC project, ethical approval (#: 2020-1-RO01-KA202-080128-28th Dec 2020) was obtained by the lead organisation, for the entire intellectual output 1, Core research competencies framework for the multidisciplinary palliative care team. Results The RESPACC workshop during the Romanian National Palliative Care Conference was attended by 94 participants, all palliative care professionals. Among them, 25 (19 women, 6 men) participated in the pilot testing (see Figure 2) and returned the completed quiz, including the feedback questions. Figure 2 – Distribution of respondents by profession Most respondents (60%) had less than one year of experience in palliative (Figure 3), 4 (16%) had 2-3 years of experience in palliative care, 3 (12%) between 8-10 years and 3 (12%) of them had over 10 years experience in palliative care. Figure 3 – Distribution of respondents by experience in palliative care Respondents were asked to assess their level in general and for each domain. According to the total score (Figure 4), 9 respondents (36%) obtained a score for Advanced Beginner level, 5 respondents (20%) each for Novice and Competent level, and 3 respondents (12%) each for Proficient and for Expert level. Respondents’ scores for each domain are shown in Figure 5. Figure 4 – Distribution of responses per total Figure 5 – Distribution of responses per domain Correlating clinicians’ experience in palliative care with the total score generated a coefficient r=0.6269 with a significant p

<0.001. Participants (n=19) with experience in palliative care less than 3 years had total scores equivalent to Novice (n=5), Advance Beginner (n=9) and Competent (n=5). Participants PALIAŢIA Vol. 16 Nr. 2 Aprilie 2023 Predoiu O. et al ISSN 1844 – 7058 9 Fundația HOSPICE Casa Speranței with palliative care experience over 8 years in domain, obtained scores for Proficient (n=3) and Expert (n=3) level. Respondents answered all questions, without having difficulties of understanding the content. All participants considered the instructions explicit and clear; the definition of each domain of the framework was clear. Quiz items were reported as “well structured, complex, comprehensive, clear and adequate”. Participants enjoyed receiving feedback, especially as it was framed in positive language. Example: “41-60% you have basic knowledge of research and can undertake research tasks in this domain without support but would be good to consult with your peers” Discussion This study demonstrated that a self-assessment quiz designed to assessing competency regarding research had face validity and clarity when used by multidisciplinary teams. Feedback showed that the quiz was clear, well structured based on the framework domains, with clear instructions for completion. Participants appreciated receiving immediate feedback, phrased in an accurate supportive language. Results show a positive correlation between years of experience in palliative care and levels of self-rated competence, which parallels other international studies (14), (15) which suggests professional experience is a factor that affects the development of competencies (13). Professional staff in Romania with palliative care training covers only 0,1% of those working as palliative care professionals (17), hence our sample has a majority with less than one year of experience in the field., This fact is reflected in the results of the piloting study, where most respondents (n=19) obtained scores in the levels: Novice, Advance Beginner and Competent. A further study to explore more this aspect should be organized. Conducting a pilot study was a mandatory step before enabling the wider dissemination of the quiz to a larger group. Although a small sample, literature (16) suggests a pilot study sample should be 10% of the potential population; in our study, 25 persons (26%) from 94 attending the workshop, responded to the invitation to fill in the quiz and give feedback related to its content. After the final meeting, the consortium decided to remove two items that seemed redundant, and one was split into two items, for more clarity. The final format of the quiz (38 items) was presented in Excel format, enabling intermediate and final scores to be given in real-time following completion. The quiz was also designed in SurveyMonkey Platform, where the respondent receives immediate feedback only for each domain. The quiz is available in English, Greek, Spanish and Romanian. A further study could include a test for knowledge related to all competencies included in the RESPACC framework. This would provide a perspective about what a respondent knows and understands related to research, and can be correlated with the results from the self-assessment quiz. Integrating the results of the two methods of evaluation may help to identify educational needs and appropriate training for individuals (13), but also for the team willing to initiate research studies in their settings. Among the limitations of the study is the lack of internal validation of the quiz items and the relatively inexperienced and small sample. Further, pilot testing in other countries would determine the applicability and utility of the quiz. Conclusions The self-assessment quiz is easy to complete and is a useful guide for palliative care clinicians to assess their competencies and identify their educational needs concerning research. The quiz may also be a useful tool for organizations to identify selfperceived strengths and weaknesses regarding the research competencies of its employees and develop proper training to enhance staff’ abilities in research in palliative care (3). Acknowledgement This project has been funded with support from the European Commission by ERASMUS+ agreement #: 2020-1-RO01- KA202-080128. This publication reflects the views only of the authors, and the Commission cannot be held responsible for any use which may be made of the information. Special recognition needs to be addressed to palliative care professionals who voluntary took part in the study. Conflict of interest: None declared. References: 1. The Special Programme for Research and Training in Tropical Diseases. TDR. (2016). Using the TDR Global Competency Framework for Clinical Research: A set of tools to help develop Clinical researchers. Accessed 15th Feb 2023. 1-eng.pdf 2. RESPACC Framework of Core Research Competencies for Palliative Care Clinicians,, Accessed 15th Feb 2023 3. Hornung CA et al. Competency indices to assess the knowledge, skills and abilities of clinical research professionals. Int J Clin Trials. 2018 Feb;5(1):46-53 4. Gowie M, Wilson D, Gregory S, Clark LL. Development of a core competency framework for clinical research staff. Journal of Interprofessional Education & Practice. 2020 Mar;18:100301. Epub 2019 Dec 9 5. Core Competency Framework for the Clinical Research Professional Version 3.1 February 2020. Accessed 15th Feb 2023 6. NIHR CRN Integrated Workforce Framework, IWF (2017), IWF, An introduction © v1.0.pdf – Google Drive, Accessed 15th Feb 2023 7. Competency Framework for Clinical Research Nurses (2011; 2013 revised), Microsoft Word – Competancy Framework Version 2.docx ( Accessed 15th Feb 2023 8. Research competencies framework, Faculty of General Dental Practice, The Royal College of Surgeons (UK), research competencies dental practice.pdf – Google Drive, Accessed 15th Feb 2023 9. 2016 Oncology Clinical Trials Nurse Competencies, Oncology_Clinical_Trials_Nurse_Competencies.PDF ( Accessed 15th Feb 2023 10. Prof. Sheila Payne, Dr Nancy Preston, Dr Mary Turner, Dr Liz Rolls, Research in palliative care: can hospices afford not to be involved? A report for the Commission into the Future of Hospice Care (2013), (PDF) Research in palliative care:can hospices afford not to be involved? | Liz Rolls –, Accessed 15th Feb 2023 11. Junger S, Payne S: The crossover artist: Consensus methods in health research. In: Walshe C, Brearley S (eds): Handbook of Theory and Methods in Applied Health Research. Cheltenham: Elgar, 2020 12. Stephen Mason, Julie Ling, Daniela Mosoiu, María Arantzamendi, Aliki J. Tserkezoglou, Oana Predoiu, Sheila Payne, Undertaking Research Using Online Nominal Group Technique: Lessons from an International Study (RESPACC). Journal of Palliative Medicine.Dec 2021.1867-1871 PALIAŢIA Vol. 16 Nr. 2 Aprilie 2023 Predoiu O. et al ISSN 1844 – 7058 10 Fundația HOSPICE Casa Speranței 13. Magro G., Dellai M., Notarnicola I. Self-assessment the competencies of surgical and critical area nurses. A crosssectional study. Acta Biomed 2022; Dec 16;93(6):e2022348 14. Faraji A, Karimi M, Azizi SM, Janatolmakan M, Khatony A. Evaluation of clinical competence and its related factors among ICU nurses in Kermanshah-Iran: A cross-sectional study. Int J Nurs Sci. Oct 10 2019;6(4):421-425 15. Connelly LM. Pilot studies. Medsurg Nurs. 2008; 17(6):411–412. 16. Rizany I, Hariyati RTS, Handayani H. Factors that affect the development of nurses’ competencies: a systematic review. Enferm Clin. 2018;28:154-157 17. Analiza situației furnizorilor de servicii de îngrijiri paliative din România în anul 2019-report Accessed 15th Feb 2023>