SDTM automation: A practical guide to clinical trial data mapping

A practical guide to SDTM mapping, automation methods, and clinical data standardization.

The automation of clinical trials has been a goal of the pharmaceutical industry for many years. While Electronic Data Capture (EDC) systems and other technologies have streamlined the collection and management of clinical trial data, transforming raw clinical data into the required SDTM format remains a manually intensive and error-prone process.

The Study Data Tabulation Model (SDTM), developed by CDISC, provides a standardized structure for organizing clinical trial data and preparing it for regulatory submission. By automating SDTM mapping, pharmaceutical companies can reduce manual effort, improve data integrity, and ensure that datasets meet regulatory requirements.

In this article, we explore the SDTM implementation process, including mapping approaches, automation methods, and technologies that support the generation of compliant SDTM datasets.

What is SDTM?

SDTM (Study Data Tabulation Model) is a data standard developed by the Clinical Data Interchange Standards Consortium (CDISC) to organize and standardize clinical trial data. It provides a common framework for managing medical research data and supports consistent data submission for regulatory review.

SDTM provides a common structure for the organization and tabulation of clinical trial data, establishing unified data standards for pharmaceutical companies through the CDISC SDTM framework. As a proven format for the dissemination of clinical trial data, SDTM provides regulatory reviewers from the US Food and Drug Administration (FDA) a clear description of the structure, attributes, and contents of each dataset in the model. This common structure allows for the exchange of data between different clinical trial software applications and databases. Once you’ve obtained all the relevant data for your research, it must be transformed into the specific table format suitable for FDA review.

Explore how Avenga developed a set of mature telehealth applications that were integrated into a single innovative service delivery system. Read more

What are the benefits of SDTM automation in clinical trials?

SDTM automation helps pharmaceutical companies streamline clinical data management by reducing manual mapping efforts, improving data consistency, and supporting more efficient regulatory submissions. By automating repetitive tasks in the SDTM mapping process, organizations can improve data quality and reduce the risk of errors throughout clinical trials. Here are some of the key benefits of SDTM automation:

• Increased efficiency – cut down on the overall time and cost of clinical trials by streamlining data management.
• Improved data quality – raise the quality of clinical trial data by ensuring its integrity and traceability.
• Reduced risk of human error – lower the risk of human error in data collection and management, and avoid errors in the analysis of trial data.
• Higher transparency – increase transparency in clinical trials by providing a clear audit trail of data collection and management processes.
• Enhanced patient safety – safeguard patients by ensuring that critical trial data is managed accurately.

Last but not least, SDTM automation in clinical trials can facilitate communication between different sponsors, sites, and Clinical Research Organizations (CROs), leading to better coordination and overall management of clinical trials.

SDTM automation: A hybrid mapping method

A manual method, a hybrid process comprising application and manual parts, or a complete application technique might be used to map raw data sources into SDTM variables based on SDTM specifications and mapping requirements.

In the past, requirements were written into an Excel or Word file. Then, a programmer manually transferred the relevant specifications into a Statistical Analysis System (SAS) or Structured Query Language (SQL) code as part of an all-manual procedure. The hybrid method reads specifications from an Excel file into a dataset, which is then used to dynamically build code in a program in order to map the source variables. When using a complete application method, such as SAS Clinical Data Integration, the application stores all of the requirements, reads them, and develops all of the SDTM mapping and derivations

The manual process of clinical data mapping increases the time of the data standardization, hence enlarging resources spent on the project. In this way, automation seeks to decrease the additional time necessary to create high-quality CDISC-compliant data for FDA submission. That’s why it is crucial to work on tools that would save resources spent on the SDTM creation.

What are the main difficulties of automating A SQL script creation?

Automating SQL script creation for SDTM mapping is challenging because clinical trial data specifications often contain complex rules, dependencies, and transformation requirements. These specifications are typically stored in standardized source mapping Excel workbooks, with separate worksheets for each SDTM domain that must be accurately translated into executable code.

Creating a standardized Excel file may be the most difficult component of the data flow because numerous scenarios must be properly mapped by the user. Today, we will consider some of the possibilities of creating SDTM mapping in a way that it is possible to transform it into an executable script.

The ability to create SQL scripts with plain language statements has the potential to appeal to users who are unfamiliar with query languages such as SQL. Text to SQL mapping is a Semantic Parsing issue, which is defined as converting natural language input into a machine-interpretable representation. Semantic Parsing is a well-studied subject in Natural Language Processing (NLP) that has a lengthy history.

As a result, Semantic Parsing attracts the interest of people who want to make the process of SDTM creation less time-consuming and more effective. All of these approaches might be eventually integrated to make a broader task of translating natural language to a fully functional application. To address the Semantic Parsing problem, different approaches have been developed. Meanwhile, the difficulty of creating SQL is more complex than the typical Semantic Parsing problem. A brief natural language inquiry may necessitate the combining of numerous tables or the use of multiple filtering criteria. That’s why more context-based techniques are required.

NLP in SQL query generation

Annotated complex questions and SQL queries comprise the datasets that are meant for semantic processing of natural language phrases to SQL queries. The sentences are inquiries for a certain area, while the answers are drawn from existing databases. As a result, the specific inquiry is linked with a SQL query. The SQL query executes and retrieves the response from the existing data.

There are various Semantic Parsing datasets available now for SQL query mapping. They differ in numerous ways. For example, WikiSQL and Spider, the most recent datasets, are cross-domain and context-independent. In addition, they have many questions and extensive inquiries. As the dataset size is critical for successful model assessment, unexpectedly complicated questions in the test sets can be used to assess the model’s generalization capacity.

These datasets vary in complexity, with some supporting basic SQL queries and others involving more advanced operations, such as table joins and nested queries. This complexity reflects the challenges of using NLP for automated SQL generation in clinical data workflows, where accurate context understanding is required.

As a result, Semantic Parsing attracts the interest of people who want to make the process of SDTM creation less time-consuming and more effective. All of these approaches might be eventually integrated to make a broader task of translating natural language to a fully functional application. To address the Semantic Parsing problem, different approaches have been developed. Meanwhile, the difficulty of creating SQL is more complex than the typical Semantic Parsing problem. A brief natural language inquiry may necessitate the combining of numerous tables or the use of multiple filtering criteria. That’s why more context-based techniques are required.

Automating SDTM mapping with rule-based code generation

Automating SDTM mapping with rule-based code generation involves transforming raw source variables into SDTM variables using predefined rules, functions, and metadata-driven logic. This approach typically relies on the following steps:

• Reading SDTM mappings
• Merging relationships in Excel worksheets
• Converting the Excel worksheets to datasets
• Deriving functions and standard expressions to read the datasets
• Creating the SDTM mappings
• Merging the data
• Appending the data

The advantage of functions is that it directly maps the Excel requirements, which should result in fewer conflicts and faster development. With this approach, it would be possible to generate from 50 up to 70 % of the variables for different SDTM domains. To collect all of the derivations for the other output variables, post-processing of the SDTM dataset would be required. Provided metadata should contain all of the important information about the source variable, format, and new variable, and it might be used for validation of the mappings.

Closing remarks

The process of getting the final SDTM mappings can be automated. There can be different approaches used for that purpose, and they can possibly give a different percentage of automation based on the initial conditions of mapping and its complexity. Yet, automating data mapping in clinical trials with SDTM can speed up the process of data analysis and help ensure a higher accuracy of data management.

Interested in learning more about our digital solutions for pharma & life sciences? Contact our teamto discuss your requirements.

Rate this article!

Average 0.0 out of 5

Pavlo Chaikivskyi

Senior Engineer