Background & Aims
Core outcome sets (COS) are important for research to compare data across studies and maximize resources to improve health outcomes. COS are often developed through a Delphi method, which collects stakeholders’ opinions in a stepwise approach for consensus. INTEGRATE-Pain, a US-EU collaboration to advance pain research and clinical practice, undertook a large-scale, 3-round, international Delphi to reach consensus on COS across the pain continuum. The Delphi was unique because: 1) persons with lived experience of pain (PWLE) advised the research team from its inception to completion, represented 50% of respondents, and had their views weighted equally to those of researchers and clinicians; 2) participants represented diverse geographic and professional backgrounds; and 3) the COS are intended for use in research and clinical settings for all pain-related conditions. Given the scale and uniqueness of this Delphi, we aim to illuminate the lessons learned to inform future COS efforts.
Methods
We describe the decision-making for each step of the INTEGRATE-Pain Delphi, outline the strengths and challenges we experienced at these steps, and provide recommendations for future COS researchers to consider. Specifically, we discuss strengths, challenges, and recommendations regarding: 1) engaging stakeholders to establish the need for a COS, defining the scope of the COS, and registering the project; 2) identifying key personnel; 3) identifying initial domains and subdomains; 4) implementing the Delphi; and 5) finalizing the COS. We also outline recommendations related to COS uptake and guidance for developing COS through a Delphi. Findings presented were compiled based on feedback from Delphi participants (via the open-ended solicitation of comments/feedback during voting), from meetings with advisory committees (AC) and steering committees, and from reflections from the co-chairs that led the study.
Results
To engage stakeholders, we recommend a kick-off meeting to gather input, and inclusion of people with diverse backgrounds/opinions and PWLE at all stages of the process. When identifying key personnel, ensure diversity on each AC. Before commencing Delphi voting, train those inexperienced in COS/Delphi methods and be mindful of PWLEs’ needs. To identify initial domains for voting, we suggest holding focus groups and qualitative interviews and balancing domains from published literature with input from Delphi stakeholders. Ensure all domains are defined and combine similar domains or subdomains. Ensure surveys and communication with voters are clear and accessible. Consider using a Delphi management tool for survey administration/participation. Use mixed methods to analyze surveys and feedback/suggested domains submitted by voters. When finalizing the results, consider a final consensus meeting and an appropriate dissemination plan, and provide avenues for feedback.
Conclusions
We led an international Delphi to develop COS for acute, recurrent, and chronic pain, and acute to chronic pain translation. The Delphi was unique in that over 350 individuals from around the world participated, 50% of whom were PWLE. This abstract highlights our methodological decisions and the lessons we learned conducting an international Delphi. We hope to provide recommendations to future COS researchers related to engaging stakeholders and building buy-in, recruiting Delphi participants, using literature and qualitative feedback to generate initial lists of domains for voting, implementing the Delphi, and finalizing the COS. It is important to consider robust dissemination plans for the COS because COS have had variable uptake among researchers/clinicians and international enforcement of COS is difficult. Lastly, the COS field could benefit from further research to empirically test various Delphi methodologies and establish evidence-based guidelines for COS development.
References
1.Bellucci C. et al. Trials. 2021 Dec 19;22(1):937. doi: 10.1186/s13063-021-05891-5. PMID: 34924001; PMCID: PMC8684586.
2.Bova et al. Developing Consensus on Core Outcome Sets of Domains for Acute, the Transition from Acute to Chronic, Recurrent/Episodic, and Chronic Pain: Results of the INTEGRATE-Pain Delphi Process. eClinicalMedicine. 2023 Dec 1;66. https://doi.org/10.1016/j.eclinm.2023.102340.
3.Brookes ST, et al. Impact of question order on prioritisation of outcomes in the development of a core outcome set: a randomised controlled trial. Trials. 2018 Jan 25;19(1):66. doi: 10.1186/s13063-017-2405-6. PMID: 29370827; PMCID: PMC5784591.
4.Chevance A, et al. Controversy and Debate Series on Core Outcome Sets. Paper 1: Improving the generalizability and credibility of core outcome sets (COS) by a large and international participation of diverse stakeholders. J Clin Epidemiol. 2020 Sep;125:206-212.e1. doi: 10.1016/j.jclinepi.2020.01.004. Epub 2020 Jan 13. PMID: 31945480.
5.COSMIN. I’m developing a Core Outcome Set (COS). https://www.cosmin.nl/finding-right-tool/developing-core-outcome-set/.
6.European Pain Federation, What is the biopsychosocial model of pain?, https://europeanpainfederation.eu/what-is-the-bio-psycho-social-model-of-pain/.
7.Jensen MP. Psychosocial approaches to pain management: an organizational framework. Pain. 2011 Apr;152(4):717-725. doi: 10.1016/j.pain.2010.09.002. Epub 2010 Dec 18. PMID: 21168972.
8.Keeney S, et al. Consulting the oracle: ten lessons from using the Delphi technique in nursing research. J Adv Nurs. 2006 Jan;53(2):205-12. doi: 10.1111/j.1365-2648.2006.03716.x. PMID: 16422719.
9.Kearney A et al. A systematic review of studies reporting the development of core outcome sets for use in routine care. J Clin Epidemiol. 2023 Jun;158:34-43. doi: 10.1016/j.jclinepi.2023.03.011. Epub 2023 Mar 21. PMID: 36948407.
10.Khan K; Chief Editors of Journals participating in The CROWN Initiative (Appendix 1). The CROWN Initiative: journal editors invite researchers to develop core outcomes in women’s health. BJOG. 2016 Sep;123 Suppl 3:103-4. doi: 10.1111/1471-0528.14363. PMID: 27627608.
11.Nasa P, et al. Delphi methodology in healthcare research: How to decide its appropriateness. World J Methodol. 2021 Jul 20;11(4):116-129. doi: 10.5662/wjm.v11.i4.116. PMID: 34322364; PMCID: PMC8299905.
12.Wandner et al. INTEGRATE-Pain: a transatlantic consortium to advance development of effective pain management, Pain Medicine, Volume 24, Issue 6, June 2023, Pages 730–733, https://doi.org/10.1093/pm/pnad033.
13.The OMERACT Handbook, https://omeract.org/wp-content/uploads/2021/12/OMERACT-Handbook-Chapter-Final-June_2_2021.pdf
14.Schmalz U, et al. Lessons Learned from a Two-Round Delphi-based Scenario Study. MethodsX. 2020 Dec 13;8:101179. doi: 10.1016/j.mex.2020.101179. PMID: 33365260; PMCID: PMC7749424.
15.Williamson PR et al. The COMET Handbook: version 1.0. Trials. 2017 Jun 20;18(Suppl 3):280. doi: 10.1186/s13063-017-1978-4. PMID: 28681707; PMCID: PMC5499094.
16.Williamson PR et al. Review finds core outcome set uptake in new studies and systematic reviews needs improvement. J Clin Epidemiol. 2022 Oct;150:154-164. doi: 10.1016/j.jclinepi.2022.06.016. Epub 2022 Jun 30. PMID: 35779824.
17.Williamson PR, et al. Controversy and Debate Series on Core Outcome Sets. Paper 4: Debate on Paper 1 from the perspective of COMET [Core Outcome Measures in Effectiveness Trials]. J Clin Epidemiol. 2020 Sep;125:222-224. doi: 10.1016/j.jclinepi.2020.05.014. Epub 2020 May 13. PMID: 32413391.
Presenting Author
Giulia Bova
Poster Authors
Giulia Bova
National Institute of Health
Lead Author
Laura Wandner
PhD
National Institute of Health
Lead Author
Janelle Letzen
PhD
National Institute of Health
Lead Author
Kate Nicholson
National Pain Advocacy Center
Lead Author
Judy Birch
B.Ed
Pelvic Pain Support Network
Lead Author
Daniela C. Rosenberger
MD
Department of Anaesthesiology, Intensive Care and Pain Medicine, University Hospital Muenster
Lead Author
Adam Anicich
MBA
NIH-DoD-VA Pain Management Collaboratory
Lead Author
Ulrike Kaiser
PhD, MD
University Hospital Schleswig-Holstein, Clinic for Anesthesiology / Intensive Care Medicine, Lübeck
Lead Author
Anthony Domenichiello
PhD
Lead Author
Esther Pogatzki-Zahn
PhD
Lead Author
Topics
- Assessment and Diagnosis