What "Good" RWE Looks Like

Research Transparency to Strengthen Trust in RWE

Transparency throughout the research process is essential to build decision-makers’ (i.e., regulatory authorities and HTA organisations) confidence in real-world evidence (RWE) studies and their results. Implementing and documenting concrete actions in key phases of the study will enable decision-makers to assess the reliability, reproducibility and validity of the findings.

This page lists all subrecommendations linked to the overarching recommendation 1, "Research Transparency to Strengthen Trust in RWE". You can use the tiles below to jump directly to a specific subrecommendation.

1.1: Define the Research Question and Study Design
1.2: Sharing Pre-Specified Study Protocols
1.3: Data Selection

1.4: Statistical Analysis Plan (SAP) Development
1.5: Analysis and Interpretation of Evidence
1.6: Reporting and Dissemination of Results

Subrecommendation 1.1: Define the Research Question and Study Design

Ensure RWE studies begin by setting clear and specific research questions, objectives and hypotheses, supported by relevant background literature.

Rationale

Setting a clear research question helps to establish a foundation for why and how the research study is taking place. It also encourages consideration of the type and nature of data required to answer a question, allowing due consideration as to whether RWE is the appropriate approach for answering the question, and whether suitable data are available. A structured research question also allows for greater study reproducibility.

Details

The study development should include, as a minimum:

Essential A clear and specific research question should be formulated at the start of a study, before study design or data sources are selected. Primary and, where relevant, secondary study objectives should be set. Consideration should be given as to whether RWE is appropriate to answer the chosen research question and achieve the objectives. The research question should directly align with the intended use of the RWE i.e. where the results of the study are to inform healthcare decision-making e.g. an HTA or regulatory situation, it should be framed to answer the question relevant to that context Consider if one research question is required, or whether several will be needed to get all the relevant information needed to inform the healthcare decision. Structure research questions using a relevant framework such as PICO (Population, Intervention, Comparator, Outcomes) framework or PICOT (including Timeframe). Research questions should be included in study documentation as a descriptive study title including the study design, population and a version identifier if appropriate. Justify why the study is taking place and how the research question and study objectives will address the well-defined problem or issue. State what hypotheses are being tested. For studies aiming to test hypotheses around comparative effectiveness, having an explicit a priori hypotheses is recommended. Consider the availability, quality and limitations of RWD sources. Ensure that local guidance is consulted to determine which sources of RWD are considered acceptable. If this is unclear from local guidance, contact the stakeholders for whom the expected RWE is being generated to discuss. Do not retrofit research questions to data sources. Choose a study design that is consistent with the research question.

Important

Conduct a critical review of the literature to assess what has previously been done to assess the research question being posed. Summarise applicable information from other studies and describe the extent to which these studies address the research question.
Engage early with relevant stakeholders to ensure the relevance of the research question and the applicability of its intended results. For example, engagement may include consultation with:
1. Regulators & HTA bodies to understand if the questions being explored are answering relevant questions and whether there are any additional considerations.

Optional

Adapt the PICO/PICOT framework as required to include the relevant elements, for example PECO to include exposure or PIRO for Diagnostic evidence (Population, Index Test; Reference Standard; Outcome)
Use an estimand framework when describing the rationale and appropriateness of outcome measures to be used.
Engage Patients/public to confirm relevancy of the question to their needs and experiences

Subrecommendation 1.2: Sharing Pre-Specified Study Protocols

Pre-specification and registration/sharing of RWE protocols is critical to establish the transparency of the research and create shared accountability.

Rationale

Pre-specification of study protocols is a critical step to enhancing the transparency, credibility and overall scientific rigor of the research. Policies that support opportunities to voluntarily and confidentially share study protocols with regulatory authorities and/or HTA bodies enhance compliant engagement and transparency. An additional step of pre-registering key protocol details could also support the credibility of the study results. As always, compliance with relevant regulatory requirements is required. The timing of disclosure of relevant study and result details should not prejudice the rights of the sponsor and study partners. Where premature disclosure of such details could harm the study and/or the rights of the sponsor an acceptable method of deferral and redaction of commercially confidential information would sustain the scientific rigor of the research and protect the rights of the sponsor and study partners.

Protocol pre-specification advances quality and illustrates the rigor of RWE research.
Sharing details of the protocol (as outlined above) enhances transparency and builds trustworthiness:

Declaration of Intent: Registering study protocol details on a public platform before analysis begins publicly declares the study's intent and provides basic study information.
Allows stakeholders and decision-makers to understand the scientific thought process and agree on the study's approach.
Building Confidence: Registration of protocol details helps to build confidence among decision-makers.

Increase Methodological Credibility:

Prevents Data Manipulation: Finalising and publicly posting the study protocol details) before reviewing outcome data or performing analyses safeguards against selective analysis or elevating secondary questions to primary ones based on preliminary findings.
Prevents Publication Bias: Study registration helps reduce publication bias, as it allows identification of studies that might otherwise not be published (e.g. those with inconclusive or null results, or those that were only partially completed).
Promotes Accountability: Public registration encourages careful deliberation, planning and accountability from those conducting the study.

Details

The pre-specified protocol should include, as a minimum:

Essential Research Question(s) and Objectives: Clearly define the specific research question(s) and objectives, including whether the study is exploratory or hypothesis-evaluating (HETE). The research question should cover elements of the PICO (Population, Intervention, Comparator, Outcome) scheme. Please see the recommendation on “Define the Research Question and Study Design” for additional details. Justification and Background: Present the theoretical and scientific rationale for conducting the RWE study, including knowledge gaps it aims to address and its positioning (e.g. critical supporting evidence, auxiliary evidence, or exploratory objective). Study Design & Methodology Study Design Type: Clearly state the specific study design, such as observational cohort studies, pragmatic clinical trials (pRCTs), case-control, nested case-control, or single-arm trials with external controls. A flow chart illustrating the implementation process is recommended. Target Population & Patient Selection: Pre-specify the definition of the study population, including clear inclusion and exclusion criteria. Describe the representativeness of the target population and any potential heterogeneity between the study population and the intended target population. Detail the source of patient recruitment (e.g. clinical centres, registries) and the methods for identifying, selecting, including and retaining participants to minimise selection bias. For external controls, justify the selection or exclusion of relevant data sources and ensure the choice aligns with the research question. Observation/Follow-up Period: Reasonably define the study's observation period, follow-up period, start and end times, time intervals, and specific time points for observation or follow-up. Endpoints: Clearly state the selection of effectiveness, efficacy and safety endpoints (primary, secondary, procedural, device, etc.), their definitions and measurement methods. If objective hard endpoints are difficult to apply, specific blinding measures might be needed. Control Group: Describe the setting of the control group, including whether it's concurrent or historical. If matching-based methods are used, predetermined matching criteria should be included. Sample Size Estimation: Include the determination of the sample size, ensuring it is adequate to answer the research question with sufficient statistical power. Data Management & Quality Data Sources: Describe the real-world data sources used (e.g. electronic medical records, claims data, registries, patient-generated data, device-generated data). Data Curation/Management Plan: Specify the data curation and data management plan, including processes for collection, cleaning, processing, governance, storage system, record form and data security. It should ensure traceability of source data for key variables. Address considerations for data quality, accuracy, completeness, provenance, reliability and relevance. Detail auditing rules, methods and mitigation strategies to reduce errors. Statistical Analysis Plan (SAP): The main analysis plan should be synchronously determined with the study protocol to avoid result-driven bias and guarantee transparency. It should clarify specific statistical methods, parameter settings and the rationale for their choice. Bias Minimisation and Control: Describe methods for minimising and controlling potential biases (e.g. selection bias, information bias, confounding bias). This includes justifying the handling of unknown or unmeasurable confounders and using sensitivity and quantitative bias analyses. Data Protection & Privacy: Detail measures to protect patient privacy and ensure compliance with applicable data protection rules (e.g. GDPR). Informed Consent: Describe how/if informed consent was needed and if so, how it was obtained. Provide the consent details (e.g. study purpose, duration, intended use of data, and access by third parties (monitors, auditors, regulatory authorities)). Protocol Amendment: Any substantial changes to the protocol, data curation plan, or main analysis plan must be revised, dated, time-stamped, justified and communicated.

Important

Data Governance: Describe preliminary applicability evaluations of source or governed data (e.g. data governance, access, and use, including compliance with relevant laws and regulations).
Ethics Committee Approval: The acquisition and use of RWD for RWE must be reviewed by an ethics committee.

Optional

Necessity and Feasibility: Highlight why the research question is being addressed through RWE and the suitability of RWE to address the question.
Past Use of RWD source(s): Provide documentation of any previous fit-for-purpose assessments of the data source.
Transparency of Evidence-Generating Process: Address how the entire process of data collection and governance is transparent, clear and traceable. Measures include timely communication with evaluation agencies and disclosure of key protocol information.
Early Communication: Regulators encourage active communication on study plans and protocols before study implementation to ensure that consensus is reached on the use of RWE and the conduct of real-world studies. Compliance with any regulatory requirements for pre-submission of protocols and/or SAPs is required.

Subrecommendation 1.3: Data Selection

Ensure that real world data sources can be accurately identified and established that they are suitable to address the study question.

Rationale

Identification and description of data sources is critical to elaborate on the quality of the real-world data and conduct a preliminary applicability evaluation based on study objectives and design. This is the first step in establishing the data “reliability” and assessing the feasibility of the study. Please see the recommendation on Establishing the Fundamentals for Data Integrity and Governance for additional information.

Details

The considerations for initial selection of data source(s) are, as a minimum:

Essential The data must represent the target population of the of the product, considering the context of use, geographical and temporal coverage. Use metadata for accurate identification and qualification of information, as it provides context about the data's purpose, generation, location, ownership, key variables and format. Metadata should be treated as data if changes to it would necessitate a revision of the generated evidence Conduct a preliminary applicability evaluation of the data based on study objectives and design. Key information to be specified includes the data collection start and end times, data storage system and record form. To address prioritisation of data sources, consider their suitability, which is assessed by relevance and reliability. To address relevance, consider whether the data contains sufficient detail to capture needed information for the study question, its timeliness and generalisability. This includes assessing coverage of critical variables like outcome, exposure/intervention, demographics, and important covariates. It also involves understanding the care settings, geographical and temporal coverage and the representativeness of the data to the target population Determine how closely the data, encompassing accuracy, completeness, provenance and traceability. This will address reliability. Consider factors that influence reliability, such as, consistent and methodical data collection and processing, established data quality assurance and quality control policies, and documentation of data management practices, including transformation to Common Data Models (CDMs) and data cleaning. Conduct data quality checks to identify errors, missing data, implausible values, and logical inconsistencies, with verification and remedial measures implemented. The extent of missing data must be assessed and its potential impact examined, with predetermined thresholds for unacceptable levels of missingness To ensure traceability of all variables back to the original source data, all aspects of data extraction, aggregation, curation, storage and availability for research should be planned and documented.

Important

If a single source of data is insufficient, data linkage, with supplemented sources may be necessary to obtain missing information and increase breadth and depth. The methodology for data linkage must be predefined, scientifically valid and protect privacy.

Optional

Registration of protocols may be dictated by regional/local laws or by the research institutions. For additional details, please see the recommendation on Sharing Pre-Specified Study Protocol.

Subrecommendation 1.4: Statistical Analysis Plan (SAP) Development

Develop a pre-specified SAP and documenting any changes to it, is essential for transparency and accountability.

Rationale

The study protocol outlines the overall research plan, including objectives, methodology and the data source(s). The statistical analysis plan (SAP) provides detailed specifications for how the data will be analysed. SAPs are often separate from the study protocol, though they can be included within it. Documenting the original SAP shows the study intentions and proposed methods, as well as prevents “data dredging.” Standard practice is to:

SAP should be finalised in advance and before the analysis
A sufficient description is expected to enable replication of analysis

Details (1)

The analysis plan should include, as a minimum, a pre-specified analysis plan:

Essential Use of standardised framework such as PICOS (Population, Interventions, Comparators, Outcomes, Study design) Where applicable, use an estimand framework to clearly define the treatment effect of interest, aligning the research question, population, endpoint and handling of intercurrent events Inclusion of timelines and key milestones on expected deliverable Inclusion of robust data governance frameworks (e.g. audit trails) Pre-specification of all study design elements (e.g. index date), analysis (e.g. interim or final), conduct and reporting Documentation of IRB/ethics committee approval or exemption Documentation of human subject protections, including informed consent or waivers Description of a clear data privacy plan (e.g. confidentiality terms, personal information safeguards, and data protection guarantees; compliance with legal and data governance frameworks) Description of a change control process (e.g. planned deviations or amendments) Inclusion of justification of analytical methods Pre-specification of the primary, secondary and subgroup analyses, defining the objective for each Pre-specification of potential sources of confounding and bias as well as methods that we will used to avoid them Detailed data management and quality control procedures (e.g. strategies for handling missing data)

Important

Ethical and Methodological Considerations for Data Linking
Engagement of Stakeholders in the analysis plan design

Optional

Pre-registration (e.g. journal requirements for publication)
Use of standardised templates

Details (2)

The analysis plan should include, as a minimum, a final analysis plan (outcome blinded):

Essential

The SAP should be finalised prior to conducting prespecified analyses and before reviewing outcome data.
Clear Reporting of Analytical Procedures (e.g. decision tree)
Procedures to mitigate potential sources of bias
Application of FAIR principles in data analysis
Description of data access and handling (e.g. who had access to the data, who conducted the analysis, and under what controls)
Inclusion of results from all planned and conducted analysis, with a clear statement

Optional

Documentation of Analytical Methodologies
Engagement in Collaborative Analyses Across Data Sources

Subrecommendation 1.5: Analysis and Interpretation of Evidence

Clear criteria for assessing the study results to inform the interpretation creates consistency between and across research findings.

Rationale

The increasing use of RWE in regulatory and HTA decision-making has highlighted the need for a structured and consistent approach to the interpretation of study findings. There is a lack of clear standards for real-world evidence research to assess whether results are coherent, robust methodological assumptions, or aligned with existing clinical and real-world evidence. This variability can undermine the reliability and credibility of RWE, especially its use in decision-making.

Details

The analysis of the findings should include, as a minimum:

Essential The consistency of results within the study should be evaluated, particularly across safety and effectiveness endpoints. Divergences between safety and effectiveness results should be explained, considering factors such as endpoint timing, outcome subjectivity, or potential biases. Sensitivity analyses should be conducted to assess the robustness of the results to different model assumptions, analytical methods, and definitions. Areas to examine for variations include: Operational definitions of exposures, outcomes, and covariates Eligibility criteria and follow-up periods Statistical model specifications Approaches to handling missing data and outliers Treatment switching and non-adherence Quantitative bias analysis is also encouraged, particularly when assessing the potential influence of residual confounding, selection bias, and measurement bias. All sensitivity results must be transparently reported regardless of whether they support the primary analysis. Subgroup analysis should be undertaken for key characteristics likely to influence treatment effect heterogeneity (e.g. age, sex, disease severity, site). Interaction testing should be used to evaluate differences across subgroups. These factors should be pre-specified where possible. Interpretation should be aligned with the primary research objectives and hypotheses, as defined in the protocol. The validity of conclusions should be assessed in the context of the statistical hypotheses and whether findings are consistent with what was anticipated. Deviations from hypotheses or analysis plans should be reported and explained. Results should include information on the study’s strengths and limitations. This includes discussion of study design, follow-up period, sample size and statistical power, outcome definitions, data source limitations, and risk of bias or residual confounding. The potential impact of each limitation on the interpretation of the findings should be clearly stated.

Important

The consistency of study findings with existing literature and evidence base should be assessed. This may include comparing findings with existing Investigator initiated studies, other RWE studies and RCTs. If consistent, the study may strengthen confidence in real-world generalisability. If inconsistent, researchers should explore and seek to explain potential causes.
Study results should be interpreted in relation to the intervention’s mechanism of action or device technical features.
1. For medicines, this may include expected onset, duration of effect, and pharmacodynamics.
2. For devices, the analysis may include mechanical design, procedural elements, clinical care setting, clinician characteristics and physiological compatibility.
Interpretation should be informed by the type and quality of data used in the study.
1. This includes the relevance of the data to the research question, the reliability of variable measurement (e.g. outcome, exposure, confounders), and the completeness, accuracy, and traceability of the data.
2. Data quality issues such as inconsistent coding, implausible values, or missingness should be assessed for their impact on study conclusions.

Optional

More than one analytical approach may be used to assess the consistency and robustness of study results, where feasible. This may include comparing adjusted regression models with propensity score methods, or alternative causal inference frameworks. Divergences in findings should be explored, and convergence between methods strengthens validity.
The generalisability of study findings to the broader real-world target population and care setting should be discussed. This includes:
1. assessing the representativeness of the study population,
2. the relevance of the care context, and
3. the compatibility of outcomes with routine clinical practice.
Stakeholder perspectives, including clinicians, patients, or decision-makers, may inform the interpretation of results, especially where the findings are intended to support regulatory or HTA decisions.

Subrecommendation 1.6: Reporting and Dissemination of Results

Clear practices for the dissemination and submission of RWE studies intended to support regulatory and/or HTA decision-making will build confidence in the findings.

Rationale

Beyond structured reporting, how and where RWE findings are disseminated is important. Limited publication of results and variable submission formats reduces the transparency, comparability, and impact of RWE. Consistent dissemination expectations and practices build trust and improve utility across diverse audiences.

Details

The framework should include, as a minimum:

Important

Encourage publication of results in peer-reviewed journals, preprints, or summary reports including lay summaries of peer-reviewed publications, where appropriate.
Where relevant, report how study results were shared with or communicated to patients, (but not targeting those whose data may have been used in the research) clinicians, and other non-regulatory stakeholders.
Align reporting with HTA and regulatory submission templates (e.g. tables, appendices, narrative summaries) to improve consistency and ease of review. Where submission is mandatory templates should be those provided by regulatory/HTA bodies.

Optional

Where possible and permissible, share metadata, and code lists as supplementary material in a public repository to improve transparency and reproducibility.
Provide plain language summaries of peer-reviewed publications of key findings for patients (but not targeting those whose data may have been used in the research) where appropriate.

This page belongs to a series of pages about the IDERHA report "Recommendations on policies to support the acceptance of heterogeneous health data research in regulatory and HTA decision-making", published in November 2025. The full report is available as a PDF, or you can visit the page with an executive summary.