Quick Recap

The Submissions Working Group meeting focused primarily on technical challenges with R version compatibility and package installations, particularly regarding Pilot 5 submissions and FDA review processes. The group discussed issues with curl library compilation errors and the challenges of maintaining reproducibility across different R versions, with Ben and Hye sharing their experiences about testing Pilot 5 in different R environments. The discussion also covered updates on Pilot 6’s progress with Kadji AI for data quality checks and Pilot 7’s collaboration with KGAI for developing realistic trial datasets, including potential future collaboration with Absalom’s MetaForce AI suite. The conversation ended with a brief discussion about AI tools in regulatory submissions, where Paul clarified that while FDA uses AI for research and administrative purposes, it is not currently used for direct drug review.

Summary

Next Steps

• Hye Soo: Test the resubmitted Pilot 5 in R 4.4.3 and report if the same error occurs or not.
• Gabriel: Install R 4.4.3 and RTools 4.4 on a workstation and notify Hye which machine it is after the meeting.
• Gabriel: Consider tracking all R versions on each workstation in a more structured way to help team members find required versions in the future.
• Ning: Start an email thread with representatives from Roche, GSK, Lilly, and Boehringer to discuss and potentially sync up R version update cycles, and share information with FDA.
• Vital: Check internally at Boehringer how often R versions are updated and what the schedule is, then share this information with the group.
• Ben: Push internally at GSK to sync R version update cycles with the group.
• Eric: Boot up a VM with R 4.4.3 and 4.5.2 to attempt to replicate the curl/SSL error and report findings.
• Ben: Offer to meet one-on-one with Hye to discuss technical troubleshooting when Hye gets access to R 4.4.3.
• Jared: Do some digging on the curl/SSL error and share findings with the group.

Quick Recap

The February Working Group Meeting focused on updates and discussions around ongoing pilot projects, website improvements, and technical challenges. Eric led the meeting, addressing the status of Pilot 4’s summary report, which is near completion with feedback incorporated, and Pilot 5’s resubmission, awaiting review. Hye provided updates on Pilot 4’s draft report and clarified file name discrepancies due to legacy tools. The group discussed technical challenges with Docker containers, particularly on Windows, and explored alternative technologies like Nix for future submissions. Ning introduced Pilot 7, which aims to simulate clinical trial data using AI tools, and encouraged collaboration. Vital, a new member, was welcomed and encouraged to participate in ongoing pilots. The conversation ended with a brief overview of Pilot 6 and discussions on improving the website and documentation.

R Consortium Blog Post Review

The meeting began with Eric greeting participants and discussing technical difficulties with his microphone 😅. The group then reviewed the agenda, which included a discussion about a new blog post for the R Consortium blog. Eric inquired about the status of the blog post, which was supposed to recap new pilots and existing pilots, but he was unsure if it had been published yet. Ben shared that an open pull request to the R Consortium web site has the contents of the blog post, and the goal is to publish the blog post by Monday (February 9).

Meeting Notes Review and Automation

Ben and Eric discussed the review and publication of meeting notes. They discussed the potential to streamline the process of linking agenda and minutes across the main submissions working group site and the GitHub Discussions board where the meeting agendas are posted. Eric mentioned the possibility of using an LLM to automate this. Ben also raised the need for help with visualizing pilot timelines on the website, but no new volunteers came forward.

Website Improvements and Pilot Updates

The group discussed website improvements, with Ben suggesting adding hex stickers and Eric proposing a timeline schematic for pilot access. Vital, a new member from Boringheim, introduced himself and expressed interest in contributing to the group’s R submissions work. Phanikumar offered to help with website development and suggested creating a template for the corto site. The group agreed that Vital should join the Slack channel to participate in ongoing pilot discussions. Eric provided an update on Pilot 4, mentioning that the summary report was being finalized with additional research being conducted to provide more information.

Pilot Updates and Technical Clarifications

Hye Soo provided updates on Pilot 5, which has been resubmitted but not yet reviewed, and discussed Pilot 4, for which the draft summary report has been completed and is being revised based on Eric’s feedback. They addressed a file name discrepancy issue, which was clarified by Beverly as being caused by an old ECTD publishing tool at Genentech that automatically converts underscores to hyphens. Eric expressed relief at having a clear explanation for the file name discrepancy, which had been perplexing the team. The meeting also touched on the logistics of running Docker containers, with Eric explaining the technical details behind networking and local application running.

Docker Implementation for FDA Applications

Eric explained the technical details of running Shiny applications in Docker containers, particularly how network addresses and port mapping work. He noted that while Docker works well on Linux, there are challenges with Windows due to the WSL virtual machine layer. Hye clarified that while the FDA previously had restrictions on Docker access, these issues have been resolved, though new users still need IT approval to install Docker on FDA laptops. Ben acknowledged the significant technical hurdles faced in developing the container-based submission system for Pilot 4, and Eric warned that Docker’s practical use at FDA review divisions may be limited due to evolving IT policies.

Container and Package Manager Discussion

The team discussed various container technologies and package managers for pharmaceutical submissions, with Eric explaining that Docker and Podman build on a common Linux framework for managing segregated processes. Eric introduced Nix as a potential alternative to container technology, describing it as a package manager that creates reproducible environments through symbolic links and sandboxing, though its adoption in R is still limited and it requires WSL on Windows systems. Phanikumar noted that Nix is gaining traction as a better alternative to RENV lock files in the DevOps community, while Eric acknowledged that while Nix offers advantages over Docker/Podman, the current infrastructure and WSL requirements make it less suitable for their immediate needs.

Cloud Solutions for FDA Computing

Ning suggested exploring the possibility of FDA developing a cloud-based solution to address computing environment issues, referencing the Precision FDA platform built on AWS for research. Gabriel clarified that while cloud-based solutions might be feasible for certain FDA divisions, the Office of Biostatistics and Cedar likely won’t adopt such environments soon due to restrictive computing environments, particularly concerning the WSL dependency. Gabriel emphasized the need to align with the most restrictive environments to ensure convenience for reviewers, explaining the complexity of running Docker containers in these settings.

Challenges in FDA Submission Approaches

The team discussed challenges with using WebAssembly and Docker containers for FDA regulatory submissions, noting that while these approaches are promising, they may not be the most efficient or robust options in the near term. Eric provided detailed explanations about networking and WebAssembly, which Hye will consider for inclusion in the pilot summary report. The group agreed to highlight this gap in their communications, including in a blog post and at conferences, to potentially enable further discussions with the right FDA personnel. They also acknowledged that the PIO4 report will demonstrate the viability of container-based approaches, which could influence industry trends.

Pilot Projects Update and Collaboration

The meeting focused on updates and discussions about various pilot projects, including Pilot 5, 6, and 7. Ben and Sam are working on Pilot 5, which involves exploring dataset JSON and will be presented at a conference in March. Pilot 6 aims to create more datasets and tables for future use, while Pilot 7, led by Ning, involves simulating clinical trial data using an AI tool developed by Jimin’s team. The team discussed the need for newer, more realistic data for testing tools and evaluating submissions. Participants expressed interest in collaborating and contributing to these projects, with Ning inviting others to join the weekly Pilot 7 stand-up meetings.

Next Steps

• Eric: Review the blog post draft summarizing Pilots 4, 5, 6, and 7 and provide feedback, with aim to publish on Monday.
• Ben: Share the blog post PR link in the chat for others to review and comment.
• Hye Soo: Finalize and begin the clearance process for the Pilot 4 summary report this month, incorporating relevant technical findings and Eric’s feedback.
• Phanikumar: Create a draft of a new website (Quarto) for the group, share with the team for feedback within a couple of weeks, and coordinate further discussion on Slack.
• Vital: Join the Slack channel to participate in ongoing pilot discussions and connect with Phanikumar for onboarding and collaboration.
• Ning: Continue leading Pilot 7, including weekly stand-up meetings; welcome new contributors and coordinate collaboration (e.g., with Yogesh/Pfizer).
• Ben and Sam: Present on Pilot 5 at the Fuse conference (March 22/23), focusing on dataset JSON and Pilot 5 exploration.
• Eric: Investigate automation or LLM-based solutions to link meeting agendas with published minutes and consider modernizing the documentation pipeline.
• Hye Soo: Discuss with Paul the possibility of a meeting with the FDA IT team regarding cloud-based solutions for containerized submissions, and report back if there is any update.
• Hye Soo: Consider highlighting the gap/need for centralized cloud-based solutions in future letters, blog posts, and conference discussions, as suggested by Ning.
• Eric: Add entries for Pilots 6 and 7 to the public website as they become more involved.
• Yogesh and Ning: Connect to discuss potential collaboration on Pilot 7, especially regarding SDTM/ADaM implementation and sharing of internal packages.

Agenda

• Update on Pilot 4 (container / WebAssembly) review report status
• Pilot 5 (DatasetJSON) re-submission update
• Pilot 6 (AI-generated programming) status and communication update on early progress (i.e. blog post)
• Pilot 7 (Realistic Submission Data) - Review synthetic data repositories and kick off planning for sub-team

Pilot 4

• FDA’s formal review report is still in progress
• The report will be shared via email before the next working group meeting in February
• Discussion about an XML file issue that appeared during the Pilot 4 submission process - Eric and Ben plan to follow up with Beverly to investigate why this occurred

Pilot 5

• Ben is preparing to resubmit Pilot 5 after removing the RDS data file objects
• The PR is ready and awaiting final steps from Beverly at Roche for submission
• Expected to be submitted by Monday (January 12, 2026)
• The submission script may need updates to remove references to the old RDS files
• Ning Leng has left Roche but will continue involvement in the working group in her new role

Pilot 6

• Officially focusing on AI-generated programming to build out all datasets from the original CDISC pilot
• This is NOT a formal submission to FDA - it’s an exploratory effort to create a more comprehensive submission package
• Team is leveraging various AI platforms (GitHub Copilot, KG AI, Conviva) to assist with programming
• Meetings moved to Zoom platform and occur every Friday at 10 AM Eastern Time
• Open to anyone interested in participating - join via the R Consortium calendar link
• Discussion about creating a blog post to publicize early progress
• Current submission package is small (5 datasets, 3 tables, 1 figure) - Pilot 6 aims to expand this

Pilot 7

• New pilot focusing on realistic submission data for Phase 3 trials
• Yilong Zhang connected the group with OpenClinica for synthetic data
• Initial synthetic dataset (8MB XML file) now available in GitHub repository
• New Slack channel created: “Pilot 7 Benchmark Data”
• Eric will organize a sub-team with dedicated meeting times (weekly or bi-weekly)
• First task is to evaluate the OpenClinica synthetic data and determine if additional data sources or simulation approaches are needed
• Team will explore using AI to convert XML data into tidy datasets

Website Updates

• Ben encouraged contributions to improve the R Consortium Submissions Working Group website
• Suggested additions include a timeline of the group’s history and integration of hex stickers created by J&J colleagues
• The repository is public and open to contributions via pull requests

Pilot 4 Review Status

• Hye Soo shared key updates on the Pilot 4 (Docker Container and WebAssembly Shiny applications) review progress:
  • Approximately 15 different computing environments were tested amongst the reviewers for both the Docker and WebAssembly portions of Pilot 4.
  • The WebAssembly application performed well overall across the environments
  • A handful of random issues occurred in the Docker container version which were able to be resolved, and do not seem to be caused by any of the Pilot 4 container instructions / code. The Pilot 4 Container Learning Guide was a great resource to help solve those issues.
  • A draft of the review summary is in progress in which these issues will be shared.
• A significant revelation during the discussion: Due to recent security policy updates, the Windows Subsystem for Linux (WSL) utility is no longer permitted for installation on FDA reviewer computers. The WSL utility is a key dependency of running Docker on a Windows host, due to the Docker runtime requiring a Linux-compatible environment.
• Recent security issues associated with malicious packages entering popular open-source language package systems (such as the soopsocks package entering PyPi) and the node package manager (npm) with NodeJS have caused concern over the use of open-source repositories like GitHub for accessing software in the FDA internal networks.
• In light of these factors, the use of containers in future submissions will prove to be difficult as container runtimes such as Docker will be much more difficult to set up on a typical FDA reviewer’s working machine.
• Any future blog posts which summarize Pilot 4 should not just include the results of this submission, but also include the practical challenges such as the aforementioned container environment logistics as many in life sciences are inquiring about the viability of a container-based submission. All learnings from these Pilots (not just the “success” of a submission) are valuable pieces of information for the future.

Pilot 5 Review Status

• Hye Soo shared that their team have performed preliminary testing of bootstrapping the Pilot 5 execution instructions from the ADRG and begun to evaluate the Pilot 5 R programs. The following questions were raised:
• R version 4.4.3 was specified in the ADRG as the version to install. Was there a particular reasons for using this version? Ben shared that R 4.4.3 was the latest version available when development of the Pilot 5 codebase.
• FDA considers 4.4.3 as a “newer” version that is not generally available on the FDA reviewer computers.
• As another workaround, Hye Soo demonstrated the use of a separate workstation environment running R 4.5.0. There was an error in the bootstrapping of the {renv} package library when R tried to compile a package called {mgcv} from source. While the ADRG provided instructions to install Rtools, that version of Rtools was only appropriate for R versions in the 4.4.x series. A separate version of Rtools for the R 4.5.x series (installer available from CRAN at this link) should be used instead.
• Clarification on the intent of the Pilot 5 code: Why were R scripts created to convert the DatasetJSON versions of the source data to the rds format? Ben shared that the original goal of Pilot 5 was to demonstrate that the JSON format could successfully be used in place of the xpt format to transfer data files to the FDA through the eCTD portal.
• Hye Soo noted that with the current Pilot 5 code base using this intermediate rds format, it seemed that they would be tasked with verifying outputs created from the rds versions of the data would be the same as when using the JSON data versions as the source instead.
• An interesting dichotomy was identified: Most workflows within companies involve the creation of an intermediate format such as sas7bdat or more recently parquet after the raw XPT data files to streamline processing workflows. However, when FDA reviewers examine submissions they will treat the xpt data sets as the “source of truth” in any custom programming they create to evaluate a sponsor’s submission, without creating an intermediate format.
• After this enlightening discussion, the FDA reviewers and the Pilot 5 team aligned on creating a re-submission of the Pilot 5 code that removes the creation of rds files and simply uses the JSON data files as the source for any outputs created.
• As part of this re-submission, the ADRG instructions will be updated to include instructions for locating the appropriate version of the Rtools utility based on the version of R being used for the analysis. With that said, the version of R will still be stated as 4.4.3.
• FDA reviewers inquired about how the goals of Pilot 5 compare to the Phuse/CDISC pilot in partnership with FDA conducted in 2024. While there is a bit of overlap in which both Pilots wanted to verify the utility of using Dataset-JSON as the transport file format with no loss of data (complete details can be found in the Phuse/CDISC Pilot report), Pilot 5 follows the similar paradigm of our previous submission pilots by evaluating the practical operations of how R can be used to produce submission deliverables and test an actual transfer of the submission bundle to FDA.
• Others on the call mentioned the emerging use of Parquet as an intermediate format as the source for industry workflows. Certain issues concerning metadata exist with Parquet. A detailed writeup on the benefits / tradeoffs between Dataset-JSON and Parquet can be found in Same Hume’s post: Top 10 Reasons for Using Dataset-JSON over Parquet for Data Exchange
• With the holiday season rapidly approaching, the goal will be to assemble the updated submission package for Pilot 5 in January for a re-submission via the eCTD portal to FDA.

Pilot 6 (AI-generated programming) Progress

• The Pilot 6 team is looking for ways to make it easier for the development of the key programs in light of this Pilot not being considered a submission package.
• Thus far, five volunteers are ready to develop programming using the available AI tools. Access to the platforms is still in progress.
• The team would be interested in a year-end blog post to highlight the goals of Pilot 6 as well as recap the journey of Pilot 4.
• Terry expressed support of publishing the blog post when ready, ensuring that the messaging is consistent with the previous Pilots’ blog post emphasizing the open-source mindset and not being driven by commercial interests.

Pilot 7 (Benchmark Submission Data)

• Yilong shared that he has worked closely with Open Clinica to obtain a synthetic set of EDC data whcih could be a great launching point for the Pilot 7 data sources. These data have no identifying information. Yilong would like to have a new repository created in the RConsortium GitHub organization to hold these data, with likely separate directories corresponding to the different types of data.

Pilot 4 Review Status

• Hye Soo shared that their group is continuing to test the Docker container and WebAssembly versions in multiple systems. Their team will share test results once available.
• If there are minor issues identified that do not require immediate resolution, they will be documented in the review summary.
• Assuming everything goes smoothly, the team hopes to have the Pilot 4 evaluation completed by end of the year!

Pilot 5 Review

• FDA reviewers successfully received the eCTD transfer of the Pilot 5 submission bundle earlier in November.
• The team will provide an updated on their progress at the next Working Group meeting in December.

Future Work in 2026

Action Items

• Investigate the root cause of the extra XML file that was produced in the Pilot 4 eCTD transfer. Eric will follow up with Ning and other Roche colleagues to gather further information.
• Invite Doug Kelkhoff to a future Submissions WG meeting to discuss potential new work on validating the R package suite used in Pilot 5.
• Brandon will assess the possibility of creating a custom organization within their TrialMind platform and how many members could be added for prototyping solutions.

Pilot 4 Status

• FDA reviewers Looking at end of October for the next round of review
• Re-submission folder created an index.xml file in the root of the document folder.
• Why did that file appear when it is not part of the GitHub repository? The group hypothesized that it was a configuration issue on the sponsor side for how the eCTD transfer tool was configured.
• At this time we are not clear on the specific validation procedures that were executed during the transfer itself.
• FDA noted that it will be considered a “high” error that will be reported in the submission feedback.

Pilot 5 Status

• The team is ready for submission of the Pilot 5 eCTD bundle to FDA
• The Pilot 5 ADRG has been updated with the conformance report on the XPT files to be sent through Pinnacle 21 community. No issues were identified
• Another report was created using the DatasetJSON 1.0 standard, where issues were identified that may persist with 1.1 standard
• The underlying data remains the same, and all outputs remain the same.
• Label and formatting issues appeared.
• The 1.0 standard was akin to the POC, but the 1.1 standard is the stable version that is meant for future development
• Another appendix has been created with the issue summaries in the conformance checks, with citation links
• Certain differences in class types for variables changed between pilot 3 and pilot 5. Further discussions with the datasetJSON package developers on the reason for these issues are pending.
• Additional warning messages appear on the console regarding NA values being introduced by coercion
• Ben and/or Sam will likely be giving a presentation accepted at Phuse 2026 as their paper on Pilot 5 was accepted.
• Minor formatting issues have been identified in the PDF version of the ADRG such as table rows extending past the margins of the page and certain code blocks extending beyond the margins.

ARS/ARD Future Pilot

• A smaller working group will discuss what the specific asks and key goals will be if it goes to a future Pilot in the Submissions WG
• There have been open questions on the use of these formats in a regulatory perspective
• The ARS/ARD specs can be helpful for internal validation and QC process
• There is broad consensus amongst industry that ARS/ARD is a worthwhile endeavor
• There is a little hesitation around some companies already being meta-data driven in their processes, but will there be potential push-back from regulatory
• From the CDISC group: ARS is not anywhere close to being mandated by regulatory
• Typically regulatory will not comment on specific tooling, so in the context of TLF we have a handful of packages that can be used to create those deliverables
• Naturally there will likely be more than one solution to create and manage ARS in the R community
• In Pilot 1, multiple frameworks and formats were tested to generate TLFs. A future pilot for ARS/ARD could follow a similar pattern
• CDISC representatives will be engaging with regulatory colleagues and then there will be a future discussion with industry representatives to align on a future pilot scope.
• For those who wish to participate in the future calls, the meetings are open and asynchronous communication will be done within Slack

Other Future Pilots

• 2026 could have a few future pilots:
  • Python used for machine learning and medical devices
  • Large-scale data generation that can serve as a new reference set to be used in future method benchmarking and future pilots

Upcoming Events

• R/Pharma 2025 virtual conference
• R & AI conference

Welcoming New Attendees!

• Stephen Clieply
• Orla Doyle
• Conor Moloney
• Jared Woolfolk

R/Pharma Summit and Gatherings

• For anyone attending the 2025 R/Pharma Summit and/or posit::conf(2025), Phil Bowsher shared that there are plans for social gatherings possibly for the evenings of September 14 and September 15. Be sure to check the R/Pharma summit repository for updates!
• The R/Pharma summit board is open to new members and input on the rountable session topics. Visit the summit discussion board for details.

Pilot 4 Updates

• Earlier in August, Eric led two leanring sessions on the fundamentals of containers built with Docker with FDA reviewers and interested members of the working group.
• The recordings for these session can be found on the revampled Pilot 4 web site
• The next learning session will cover the fundamentals of WebAssembly technology and how the Shiny application included in the Pilot is converted to WebAssembly. Eric will coordinate with the FDA reviewers on timing of this next learning session.

Pilot 5 Updates

• Ben shared the current state of Pilot 5 efforts.
• Current status: Debugging issues with using Pinnacle 21 Community edition
  • At this time, the community edition incoporates the datasetJSON 1.0 standard
  • However, pilot 5 is using the more recent datasetJSON 1.1 standard
• Ben asked if others in the group have experience with updating the community edition to the latest standard
  • Paul mentioned that the community edition is used at the FDA, and recommended reaching out to the Pinnacle 21 team.
  • Is this a showstopper for the submission?
  • If the XPT files are converted within Pinnacle 21 to convert to JSON, then those converted files seem to pass the conversion test.
• The R package {datasetJSON} uses the 1.1 standard
• Additional effort: Create the integrated report of the TLFs that use the datasetJSON files
• Additional polishing is ongoing with the ADRG. Ben and Eric will finish up the remaining minor issues in the next week.
• Ben, Eric, Eli Miller, and Ning Leng recently led an exciting webinar on how Pilot 5 utilizes the power of {renv}, CI/CD with GitHub Actions, and novel uses of LLMs in partnership with the R Consortium. You can view the recording of the webinar on the R Consortium YouTube Channel.

eSUB Portal

• Ning informed the group that certain text files such as README and DESCRIPTION files do not require a suffix of .txt in order to pass the submission gateway portal.
• This is a very important development, as certain files required by R packages such as DESCRIPTION cannot have their name modified or they will not be considered a valid R package in the R session.

New Pilot Idea: ARS

• Orla and Lovemore shared a set of slides about analysis results standards (ARS)
• Novo Nordisk and Novartis are actively developing tools using the CDISC ARS (Analysis Results Standards) to help structure this format as well as the required metadata
• Exploring the possibilities of a submission (such as what files would be part of a submission, and how it would be submitted)
• How would this effort compare with existing tools such as cards package? The cards package will give youa piece of that result, but this effort helps to augment these pieces with the necessary metadata and ensure that the results could be produced by the metadata
• With the data being machine readable, we have a lot of possibilities for how they are consumed by other frameworks such as AI based tools
• The cards package does not cover all analyses, so these tools would add additional functionality
• Up to this point, there has not been a discussion with regulatory on the adoption of the CDISC ARS specifications. But a new pilot could help provide valuable feedback
• Ning mentioned that these efforts are in an early stage
• Paul is not aware of a process underway with FDA on the ARS effort. However the team that would have handled that responsibility was impacted by recent events
• At this moment, Paul’s team is in the division of statstical programming issues, where this pilot idea requires buy-in from other groups at the FDA that might require a more formal process from other divisions
• Orlo expressed concern on the tools being developed in the ARS effort might get too far ahead and we find out later that the tools may not be acceptable for submission
• October WG meeting could be an opportunity to continue the discussion and potentially invite lead of the CDISC group looking at this space

Submissions WG and R Validation Hub Collaboration

• Doug Kelkhoff led the discussion on the potential of a new collaboration with the Submissions Working Group on using new package repositories in a test submission.
• Visit the R Validation Hub site for additonal background on the group and contributions.
• Current effort is to kick off a repository of pre-assessed packages
• Potential to bring a pilot to this group for using the new repository
• Since there isn’t a pre-defined new set of analysis, this pilot could integrate nicely with a previous pilot that performed an analysis, but with the packages from this repository
• Doug reviewed key sections of the group’s white paper:
  • Components of a Risk Assessment Framework
  • Some are facts, others are how to intrepret these facts
  • They are trying to de-couple these two pieces
  • Create a database of these “facts” and let user decide if it is appropriate for their work
• New packages as part of a re-imagining of the risk assessment framework:
  • {val.criterion}: Decision-making tools for {val.meter} package metrics
  • {val.repo}: A curated repository of R packages that is structured similarly to CRAN but with additional metadata: It hosts validation reports on its associated website, published via GitHub Pages.
  • {riskscore}: A data package for cataloging riskmetric results across public repositories.
• The new framworks allow the user to link their R session up to a repository, alongside a metrics repository, and supply their own filters
• What are the possibilities of using this in a future pilot?
  • Paul mentioned that there isn’t a formal software validation team at FDA
  • Separate teams handle the e-submission process
  • Validation of the software is up to the end-user groups such as the office of biostatistics, they can help determine the tools appropriate for their workflows
• Doug mentioned that typically the only feedback a sponsor receives on this type of effort is through an inspection
• Short term idea would be to target the “end-user” group like we’ve been doing up to this point with previous pilots
• Paul asked for a crafted proposal (1-2 pages) that he could share with colleagues in different divisions such as the office of compliance, hence their division can facilite while technical details would be outside of their responsibilities
• Doug is supportive of creating that initial proposal, they can edit an existing writeup to target a compliance division as the target audience.

Pilot 5 Update

• Team is working on the ADRG for Pilot 5
• Aiming for a transfer to FDA reviewers near the end of August
• Goal is to have ADRG wrapped up by middle of August
• The team is going to adopt an approach from J&J for setting paths to relevant data directories and R files at the start of execution
• Meeting minutes https://github.com/RConsortium/submissions-wg/discussions/150

ECTD File type feedback

• Link to issue https://github.com/RConsortium/submissions-wg/discussions/150
• Unique to R, many programmers write R scripts with the .R extension
• eCTD transfer portal automatically renames any files with a capital letter are converted to lower-case
• Package structures tend to have many file types, it may not be sustainable to try and list all of the different possible files
• FDA does hope to have the zip archive’s file types be listed in the acceptable file list
• In light of resources available, it could be advantageous to prune the list to the essential types and moving the “nice to have” file types could be a subsequent update
• Certain file types such as DESCRIPTION, NAMESPACE LICENCE are actually plain-text files, but they need to be named as they are in order for an R package to be compiled correctly as a ZIP installer
• Next steps: FDA will have internal discussion to propose the essential file types
• Next step: Ensure that the proposed file types also include recommended use cases that could inform the updated ECTD guidance document

Pilot 4 Update

• Hye Soo shared their group’s comprehensive initial feedback
• Pilot 4 WebAssembly version: No major issues identified and the team was able to run the application.
• Minor issues identified
  • In certain file paths that contained underscores _, something in the transfer process automatically converted the underscore characters to hypens -. The root cause is unknown at this time. As a result, portions of the file names referenced in the ADRG instructions had to be manually updated to ensure the application was able to run.
  • Hye Soo began investating a custom index.xml file that is produced at some point during the transfer, which is referenced in the eCTD Backbone Files Specifications for Modules 2-5-3.2.2
  • Ning offered to discuss the issue with her colleagues with expertise in the transfer process to learn more about this issue.
  • Joel mentioned that during the Pilot 3 preparations, file names with underscore had to be converted to hyphen
  • Additional guidance on file conventions can be found in the Specifications for File Format Types Using eCTD Specifications document.
  • HyeSoo noted that certain elements of the WebAssembly application appeared differently than the Pilot 2 application. Eric noted that certain packages used for the data filtering involving Teal modules were not supported out of the box in the WebR package repositories at the time the application was created. At that time, Appsilon developers assisting with the Pilot 4 project re-implemented the filtering modules using native Shiny functions. Had the Pilot 4 WebAssembly project been launched in the present, these issues would not have been present as the Teal packages are now supported in WebR.
• Pilot 4 Container version: Unfortunately major issues were identified that prevented successful execution of the application:
  • On the computers used by the reviewers, running Docker Desktop was not stable, and often would lead to unexplained crashes. They had to consult with their IT group for an attempt to solve these issues, one of the solutions was to update the version of the Windows Subsystem for Linux (WSL) on their machines. While that helped somewhat, they still experienced stability issues.
  • Multiple failures occurred during the process of building the Docker container image that prevented the image from being built successfully. Examples of the issues include packages not installing correctly (such as {MASS}), package repositories not able to be accessed (for instance the version of certain Teal packages that were used at the time of the Pilot 2 application development).
  • Eric shared that in his extensive testing involving Windows virtual machines, none of these issues were present and he was able to build and execute the application successfully. He acknowledged that even these virtual machines may not be fully representative of the various technical configurations added to the reviewers’ workstations due to IT policies and other administration settings.
• In light of these difficulties, Hye Soo noted that they would like to learn more about the processes and workflows used to create the Pilot 4 container submission bundle, as well as a general tutorial on the fundamentals in container technology. Unlike previous pilots focused solely on R installations, trying to debug issues preventing the Docker process from running as well as the Docker container images being built successfully is a difficult situation without additional knowledge on this technology.
• Eric offered to set up a dedicated session with FDA reviewers and others from the working group to explore the fundamentals of the container process and the workflows used to develop Pilot 4. Additional details will be shared in the Slack channel.

Pilot 5 Update

• On track for getting a first submission ready by the end of July
• View the group’s latest meeting minutes on GitHub at this link.
• In progress: Pull requests to add the hex stickers to the repositories!
• They are trying to get ahead of issues that could be identified by the reviewers, such as creating the KM plots using updated packages that might alter appearances
• Novel tooling being used in the QC pipelines. You can view Eli Miller’s QC GitHub action workflow at this link

Upcoming Events

• R adoption series webinar Open Source Software Adoption in Japan’s Pharma Industry: Key Findings from the 2024 JPMA R Usage Survey. Registration available at this link. The webinar will be offered on July 23rd 5 PM PDT (July 24th 9 AM JST).
• Another webinar schedule for August 28 focusing on Collaboration tools used in submission pilots (such as GitHub actions). Registration details forthcoming.

Proposal: CDx and open source in submissions

• Hong Zheng shared an overview of their proposal for collaboration
• With the rise of Clinical Diagnostics (CDx) co-development in targeted therapy approvals, there is an urgent need for the use of novel technology and open-source solutions to meet this challenge
• They are proposing an effort to develop open-source tools based in R to modernize and streamline CDx submissions, aligning with regulatory (in particular the CDRH division at FDA to start)
• The submissions WG is certainly eager to help share the learnings from our previous pilots and to be a resource for this new effort
• An important next step is for Hong’s group to share their prototype package mentioned in their proposal
• Also it will be important to identify a representative to contact in CDRH
• Ning recommends to draft a proposal for a first pilot in this collaboration, ensuring the scope is clearly defined before engaging with CDRH colleagues.

Other Topics

• Andre asked if ARDs will become part of future submissions?
• Ning mentioned it is still early stage for the CDISC standard for ARD, but it will be worthwhile to investigate in a future pilot.
• Andre mentioned they are creating internal tooling for the creation of ARDs using key inputs such as metadata.
• The working group has discussed a few potential ideas for future pilots, and it’s becoming more important to capture these centrally. Eric will set up a GitHub discussion board with topics to capture these future pilot ideas.

Pilot 4

• Hye Soo provided an update on high level feedback of current pilot 4 submission
• some discrepancy were found between the ADRG and the actual github folder structure, etc
• outdated links were noticed in ADRG (e.g. https://cran.r-project.org/bin/windows/base/old/4.4.1/, moved from base/ to base/old/)
• Hye Soo will work with Eric offline on the re-submission
• Some issues Hye Soo encountered may be due to windows system versioning. If needed, we will engage more WG members to test the submission bundle using different windows versions.

Pilot 5

• Part 1 is near completion (reproduce pilot 3 contents but in json format)
• Pilot 5 team will submit part 1 to FDA in late July. Will confirm the submission date in the July WG meeting.
• The Pilot 5 team will initiate a discussion on the scope of pilot 5 part 2. In part 2, we will cover more complicated TLFs, and more ADaM data sets to make the submission closer to a real submission.
• Eli implemented two QC bots for pilot 5 using github actions. One uses diffdf to check consistencies between pilot3 data and pilot 5 data; the second bot uses llm to check consistencies between pilot 3 graphs and pilot 5 graphs.
• Ning used LLM to generate three tables of the ADRG PR. The LLM generated, human reviewed tables will be included in pilot 5 ADRG
• When generating new codes for pilot 5 part 2, the team may also consider using code assistants to write the first line code. (Roche is exploring building customized bots for admiral and NEST) ## Pilot 6
• Keaven suggested exploring submission of html in pilot 6. The proposed scope is having an interactive report to assist review - especially for large tables such as AE tables, be able to drill down to specific terms and display listing of patients involved
• Hye Soo and Paul commented that they are happy to explore the review of html as a supplementary material to the primary pdf file
• will also explore submitting html version of ADRG
• The pilot 5 team will align their part 2 scope to pilot 6 proposal, to make sure we maximize the reusability of the codes
• Keaven will reach out to Merck individuals to seek interest of participation

Pilot 5 lessons learned webinar

• We will host a R consortium webinar with the pilot 5 team. The focus will be on 1) how to use git efficiently among a relatively large team of collaborators, 2) the usage of github actions to build automated QC bots
• Terry, Ning and Ben will organize the event

Hex stickers

• Ben will finalize the design of the Hex stickers
• Terry will help printing out the stickers and send to key WG members to distribute at conferences

SCE blog #2

• Jared asked whether there are any discussions on producing a whitepaper on container management (example topic: for a long running trial, what are the suggestions on when to upverison the container)
• As we are at a very early stage of exploring of the usage of container in regulatory submission, we will start with a few interviews with big pharma and produce a blog post (similar to this one)
• Jared agreed to lead this effort

Pilot 4

• Eric to set up a informal review meeting with FDA colleagues in Mid May (EDIT: The revised transfer will take place on June 13th, 2025)
• Full speed ahead to use {pkglite} for assembling the web assembly and container submission bundles.

Pilot 5

• Part 1 is moving well (conversion from xpt is complete)
• Team will wait until pilot 4 review is complete before transferring pilot 5
• Paul: Not sure where the evaluation of datasetjson currently stands within FDA. The team that was sponsoring it has been affected by recent events. They think the gateway will accept the format, but needs verification.
• Previous transfers in the CDISC effort last year may or may not have been through the ectd gateway
• ESUB team is still having discussion on testing the json format
• Heather Crandle from a previous R Pharma conference is part of that team (2021)
• In current eCTD instructions, JSON is allowed for the PK category
  
• Ben will follow up with team involved in the CDISC pilot to see how they transferred the JSON files.
• The team has embedded a datasetJSON viewer with shiny and webassebly in the pilot 5 website available at https://rconsortium.github.io/submissions-pilot5-website/viewer.html
• Ning had started investing using LLMs to generate portions of the ADRG. There may be an update at the next WG meeting.

ADRG updates

• The group’s approach is to help communicate the version of different packages and how to best display that information in the ADRG template.
• Feedback has been received from FDA reviewers. The group is looking into ways to update templates such that multiple software languages can be supported
• Keeping in close sync with Christine at Pfizer on ensuring the existing standards are being met.
• Also have plans to contact reps from Pinnacle21 on how they populate the ADRG, while ensuring minimal overlap occurs.
• Goal of the group is to reduce manual effort of populating the various tables in the ADRG such as package versions, number of programs, functions used in those programs, which is all a manual effort at the moment

Other Topics

• Yilong would like to know if there is interest in a Python-based pilot for submitting tables and listings produced in Python.
• Thus far not as much interest in the office of Biostatistics according to FDA reviewers
• CDRH handles things differently than CDER. CDRH does not require CDISC standards to be followed. A greater range of issues can occur in those cases.
• Meta has utilized standards inspired by CDISC in their initial work
• FDA is open to a more focused discussion, but they will need to pull in internal experts that have more expertise
• Eric asked about the FDA having any preference on the method of rounding as there are rare cases where the default rounding method in R can be different than SAS default method
• FDA does not have a “preferred method” to share at this time, and there may not be a single one answer that’s correct in this area
• Pilot 1 had a small issue with rounding dealing with the approximation method for confidence intervals
• We are looking for help on the website! Please get in touch with the working group and browse the GitHub issue list if you would like to contribute.

Pilot 4 (Web-assembly and Docker Container Shiny app)

• Eric has investigated the possibility of utilizing {pkglite} to assemble the submission files necessary for both the container and web-assembly versions of Pilot 4.
  • Container version: No issues, as the necessary Dockerfile can easily be renamed to Dockerfile.txt with the command to build the image updated accordingly. The existing Pilot 2 R files can be assembled alongside the aforementioned Dockerfile.
  • Web-assembly: As long as the pre-compiled version of the application is not included in the transfer, the modified application files and helper R scripts used to compile the application would still be using the permitted file types (R scripts, text files).
• The FDA reviewers would like a re-transfer of the web-assembly application using the new structure. The reviewers are in agreement to perform separate transfers for each of the Pilot 4 types, as trying to assemble each in to a single transfer package could introduce more complications as they require much different steps for running their respective applications.
• Paul mentioned that any discussions on the future use of Zip archive files in ECTD transfers is being put on hold due to circumstances happening at the agency outside of their control.

Pilot 5 (Dataset-JSON format)

• Ben shared that terrific process is being made:
  • The use of XPT files has been removed in favor of native .rds files for data storage.
  • Additional code cleanup is on track.
• Next steps are to merge the candidate Dataset-JSON files into the main repostory.
• Based on discussin in the March working group meeting, Ning and Nabil created a prototype workflow that leverages the {ellmer} pacage to populate portions of the ADRG using LLMs.
• Thanks to Sam, we have a new page on the Pilot 5 web site that embeds the DatasetJSON Viewer Shiny application.
• Paul shared that the aforementioned “circumstances” have impacted the FDA’s internal resourcing on the Dataset-JSON effort. How this impacts the future of a Pilot 5 transfer is to be determined. As of now, there is still an embargo on written communications.

Phuse Metadata / ADRG Working Group

• Lovemore shared that preparation is underway for a interactive session at the Phuse CSS conference in Europe.
• Team is working to build functions to gen metadata for the different sections of the ADRG, to be eventually folded into a package.
• Christine from Pfizer shared the current template for the ADRG which will be distrbuted to the other team members. The group will determine how it aligns with comments previously provided by the FDA reviewers.

Other Topics

• The working group welcomes a new member: Camilla Calmasini!
• Ben reminded the group of an opportunity to add a timeline to the Submissions Working Group site, as a way to present visitors a quick visual of the many accomplishments coming from these pilots. A great example can be found on the NEST website. Another approach could be to leverage the {timevis} HTML widget authored by Dean Attali, as well as clever use of Observable to create an interactive Gantt chart.
• Lovemore shared that a new Phuse Working group called QC Workflow Optimisation has launched to investigate the use of double-programming in the QC process of validating clinical programming. There could be opportunities for them to share an update with our group as they make progress, and to determine how future pilots may benefit. Ben mentioned this was a topic discussed in the recent PHUSE Connect, especially with how AI could help in this process.
• Keaven asked the group how reproducibility is being addressed currently for Shiny applications?
  • The Teal framework leverages a [custom-build solution called qenv]*https://insightsengineering.github.io/teal.code/latest-tag/articles/teal-code.html() as documented on the {teal.code} package website
  • Phil reminded the group that the Shiny team created a package years ago called {shinymeta} to help application authors capture logic based on user interactions and create a reprodcuible R script to replicate those interactions.
  • Matthew Kumar & Srinivas Verragoni from Bayer presented their use of {shinymeta} in a presentation titled Reproducibility of Interactive Analyses at the 2023 Phuse US Connect conference.
• Bonus trivia: Why was the SAS JMP software called JMP? JMP was named as “John’s Mackintosh Program” as it was an early attempt within the company to make SAS compatible with the Apple Macintosh operating systems (h/t Paul).

Pilot 4: File Structures

• Discussions internally are taking place at FDA on whether the Zip file can include the file types that may not be on the approved software list
• In light of the evolving discussions, Paul recommends to explore the use of {pkglite} for bunlding the applicaiton files in a similar fashion as Submissions Pilot 2 (traditional shiny application)
• Eric will examine both the web assembly and Docker container versions of Pilot 4 and provide FDA reviewers a list of the essenstial file types necessary to review each submission bundle
• Nan Xiao recommends we examine his recent releaes of pkglite for Python in case that could help streamline the process, especially for the web-assembly version

Pilot 5: DatasetJSON

• The pilot 5 team is aiming to have dataset programming completed by the end of March, and TLF programming completed by end of April.
• Questions remain on whether the JSON format can be included as a submission format, Paul mentioned that they are waiting to confirm with the E-sub group internally
• The R-Consortium Blog recently featured a fantastic post announcing the launch of Pilot 5:
• Minutes from the Pilot 5 working group meetings are available on the Pilot 5 datasetJSON GitHub repository: https://github.com/RConsortium/submissions-pilot5-datasetjson/discussions/3
• One potential benefit for education and awareness: Embed an example Shiny application inside the Pilot 5 Quarto website for users to explore contents of dataset JSON files.
  • After Robert shared a list of apps created in a recent DatasetJSON Hackathon, the team saw an apportunity with the Dataset-JSON Viewer Shiny application
  • Sam Parmer quickly created a novel way to compile the application as a web-assembly Shiny app and dynamically deploy the app on Posit’s shinylive-R hosting platform to embed insde the Pilot 5 Quarto website as an iframe! You can view the page at https://rconsortium.github.io/submissions-pilot5-website/viewer.html
  • In parallel, Eric also experimented with compiling the same Shiny application into web-assembly, and has a prototype automatically deployed on GitHub Pages via GitHub Actions: https://rpodcast.github.io/Dataset-JSON-hackathon/
  • Eric will reach out to the developers of the Dataset-JSON viewer to see if they are receptive to Eric sending a pull request to have the application hosted from their GitHub repository (thus their GitHub Pages account). In addition, Eric will ask the maintainers if they would support having a slightly streamlined version of the application that pre-loads JSON data files created for this pilot.
• Eric and other team members are currently updating GitHub actions for repository maintenance and automation

Phuse Metadata / ADRG Working Group

• Joel Laxana shared the latest updates: They group is trying to hash out the information to be added in the ADRG for future open source submissions
• During recent discussions, the team is going forward with creating a separate supplement document to go along with the traditional ADRG submission document in the case of an R-based submission
• The group is reviewing feedback provided by FDA reviewers from the past couple of pilots submitted
• The open source metadata working group will continue to leverage metadata from a particular project to populate sections of the document, using R functions that can help fill in manual steps
• The group ponders if generative AI help to populate these templates?
• While AI is not in the scope of the Phuse project, the team is open for exploration
• Indeed there could be an opportunity to spin off a separate pilot on exploring generative AI in a clinical submission documentation preparation such as the ADRG.
• A new channel will be created in the R Consortium Submissions Working Group Slack channel. For details on obtaining access to group, please visit the How to Join page.

Other Topics

• Members of the wroking group (Ben, Sam, and Eric) will join Appsilon for the April 29th edition of Shiny Gatherings discussing the insights, lessons learned, and what’s next for 2025.
• The group assured colleagues from FDA that there would not be any direct quotes shared from FDA colleagues inside of this webinar, and any issues from the regulatory side would be discussed in general terms.

ADRG metadata group updates

• Lovemore and Sam shared the group’s recent updates progress. They recently invited Christine Rossin (from the Phuse ADRG working group) to their meetings for discussion on the proposed template.
• Christine’s recommendations include moving the instructions for executing the R code and setting up the execution environment to a supporting document and not part of the ADRG itself, ensuring the ADRG remains software-agnostic, and separate guidelines be developed for specific software.
• The metadata group will be taking the lead on authoring guidelines for the R exection and environment language
• The group will also establish dialogs with other sub-teams addressing additional sections of ADRG such as such as estimands team to ensure everyone is on the same page for updating the ADRG sections
• Next steps include authoring R scripts to help automate population of certain sections in the ADRG template.

Pilot 4 Container Submission updates

• Eric shared recent progress on assembling a proposed submission structure for the container portion of Pilot 4.
• Substantial updates have been made to the development repository available at https://github.com/RConsortium/submissions-pilot4-container as well as the proposed submission structure at https://github.com/RConsortium/submissions-pilot4-container-to-fda.
• Eric will share the proposed structure with our FDA colleagues to ensure this is aligned with the eCTD guidelines.
• Robert inquired about the specific version of the Windows Subsystem for Linux (WSL) used in the testing environment. Eric believes it is WSL2 but that will be confirmed and the appropriate sections of the ADRG will be updated accordingly.
• Vedha shared that running the command wsl --version in the testing environment should output the key metadata on the WSL installation.

Pilot 5 DataSet-JSON updates

• Sam and Ben shared the recent progress from the Pilot 5 working group
• The team has created GitHub repositories for key portions of the pilot:
  • Development repository: https://github.com/RConsortium/submissions-pilot5-datasetjson
  • R package repository: https://github.com/RConsortium/submissions-pilot5-utilities
  • Website: https://rconsortium.github.io/submissions-pilot5-website/
  • Submission package structure: https://github.com/RConsortium/submissions-pilot5-datasetjson-to-fda
• The data sets and R code used in Pilot 3 will be ported over to the Pilot 5 repositories
• The team is very interested in utilizing GitHub actions to perform quality checks of the data set
• {datasetjson} was recently updated to version 1.1 and the new update is available on CRAN. You can view the changelog at https://atorus-research.github.io/datasetjson/news/index.html.
• As development begins for the quality checks and other testing, they will use the Pilot 3 code as a starting point.
• Robert shared that CDISC will hold a general meeting on February 18, 2025. Members of the working group are encouraged to attend. Additional details and registration are available at the pharma-oversight Slack channel in the February 14, 2025 post.

ADRG Working Group Updates

• Lovemore provided an update on recent ADRG working group discussions, with minutes available at https://github.com/phuse-org/OSDocuMeta/pull/7/files.
• Part of the recent discussions involved sharing the FDA’s proposed draft of ADRG content with Christine Rossin at Pfizer (a member of the PHUSE ADRG working group). She had many comments pertaining to keeping the ADRG template language-agnostic. The team will set up a

Pilot 5 (Dataset-JSON) Updates

• Ben shared a two-stage approach to the pilot that mimics how Pilot 4 was split to two components
• The pilot 5 team will hold weekly 30-minute calls on Fridays (10 AM EST) for discussions and updates.
• At a high level the pilot will have two key stages of development:
  • Convert the data sets created in Pilot 3 to JSON.
  • Adding additional JSON data sets and possibly new TLGs for additional testing and to pinpoint edge cases.
• Pilot 5 now has the following GitHub repositories created:
  • Development repository: https://github.com/RConsortium/submissions-pilot5-datasetjson
  • Submission (eCTD-struture) repository: https://github.com/RConsortium/submissions-pilot5-datasetjson-to-fda
  • R package repository: https://github.com/RConsortium/submissions-pilot5-utilities
  • Pilot web site repository: https://github.com/RConsortium/submissions-pilot5-website

Pilot 6 (Realistic Submission Data)

• Ateendees expressed excitement about the ways this new pilot’s data package could help many aspects of future submissions involving open-source software, ranging from the ability to handle larger data sets, to having custom versions of the data tailored to benchmarking statistical methodologies.
• Paul reminded the group that while they have access to a vast set of data from submissions, they are not allowed to share sponsor data externally.
• Ben had a suggestion that part of the pilot 6 deliverables include a mock study protocol and statistical analysis plan.
• Eric will begin to create GitHub repositories for the pilot and set up working group meetings.

Additional items

• The working group expressed interest in sharing publicly the new Pilot 5 and 6 objectives and goals in a future R Consortium Blog Post.
• Nan Xiao has created a Python version of {pkglite} which is now part of the Pharmaverse! Details on the package can be found on GitHub at <https://github.com/pharmaverse/py-pkglite?
• The group discussed the possibilities of Python being part of future submission using multiple languages? Time will tell, but there could be a future Pilot that addresses the possibilitiy.
• Paul confirmed that Python is acceptable in submissions per the eCTD Specifications document.
• Terry informed the group about important efforts for open-source software repositories involving R to comply with the pending Cyber Resiliance Act (CRA).
• The R Consortium along with representatives from the R Foundation, Bioconductor, ROpenSci, and the new Multiverse project are in close collaboration to ensure their repositories meet the requirements of the act.
• As of this writing, open-source software repositories must comply with the CRA mandates by December 11, 2027.

Joining the Submissions Working Group

A friendly reminder that you can find complete details on joining the Submissions Working Group at https://rconsortium.github.io/submissions-wg/join.html.

ADRG Formats

• HyeSoo shared additional feedback from FDA on the file formats used for sending the Analysis Data Reviewers Guide (ADRG) in clinical submissions:
  • For an FDA submission, a final ADRG document should be submitted for review. It is not recommended for reviewers to take additional steps such as rendering the ADRG using Quarto or another technical publishing system/package.
  • The PDF format is required for submitting an ADRG. However there is a possibility of conducting a pilot test with submitting the ADRG in alternative formats such as the source Quarto file .qmd as well as the HTML output format .html.
• The rational for submitting an ADRG in HTML format was discussed. Eric and Joel shared that having the HTML format brings the possibility of interactive displays (such as tables) which could be beneficial, especially for a submission including a large amount of programs and TLG outputs.
• The group agreed that sending the ADRG source file in .qmd format would not be as beneficial in the current stage, as the rendered document itself is the main focus.
• Future pilots such as Pilot 5 could be the first project to test submitting the ADRG in HTML format
• While Pilot 4 (WebAssembly Shiny app) formally submitted the ADRG in PDF format, Eric created a separate version of the ADRG in HTML format as a technical preview, which can be viewed at this link.

eCTD and Zip Archives

• HyeSoo shared clarifications on the current eCTD specifications for including a ZIP archive file in a submission package. The accepted location is M5.3, with the permissible uses of grouping large sets of aECG XML files.
• The method of including both the data sets and programs in the ZIP archive (utilized by the Pilot 4 WebAssembly submission) is not permissible.
• Eric will set up a targeted discussion with Paul and HyeSoo in January 2025 on how to best utilized the ZIP format in the future Pilot 4 Docker Container submission.
• Questions were raised on the possible inclusion of additional archive formats such as compressed Tar archives .tar.gz in the eCTD specifications. Currently that format is not supported, but it could be explored for discussion with FDA next year.

Dataset-JSON Overview

• Sam Hume from CDISC joined the meeting to share CDISC’s efforts in creating JSON datasets for clinical submissions, and feedback from previous pilots with FDA. You can view slides from Sam’s previous presentation on Dataset-JSON from the 2024 CDISC & TMF Europe Interchange conference at this link, as well as the slides from a Public Review webinar held by CDISC at this link.
• Many of the standards in data exchanges are now using the JSON format as default.
• There is increasing interest in additional structures besides the traditional rectangular/tabular approach, such as in the Electronic Study Protocol Template M11 module.
• CDISC ran a pilot within CDER at FDA to verify the Dataset-JSON format could be successfull transferred and viewed by reviewers. The testing was deemed successful.
• A mix of tools were utilized in the pilot. One of the over-arching goals is to ensure that the Dataset-JSON format is agonostic to the tools/languages used to ingest the format. For example, multiple languages like SAS, R, and Python should be able to import and analyze the data in this format.
• Recently the Dataset-JSON version 1.1 was released, addressing feedback generated from the earlier pilot. You can view the updated user guide for v1.1 at this link.
• Mike Stackhouse shared that the `{datasetjson} package, tasked with reading and writing CDISC Dataset JSON files, is in active development to accomodate v1.1 of the Dataset-JSON format.
• Nick and Mike would welcome collaboration from others on the package updates.
• Another important fix ongoing for the package is switching to a more performant JSON processing backend, as detailed in issue 32 of the package’s GitHub repository.
• In light of learning about the successful pilot, questions were asked on whether our submissions group should still initiate a new Pilot that leverages JSON format instead of XPT. Sam and others in the call concurred that the pilot would still be worthwhile to conduct, as it could be the first test of the new Dataset-JSON v1.1 specification in a realistic setting.
• Ben shared that this could be a great opportunity for others in the working group to take the lead on this new Pilot, now being called Pilot 5. Ben will take on the task of creating a new channel in Slack dedicated to Pilot 5 within the R Consortium Slack Channel.

Admin & Housekeeping

• The group agreed to keep the current cadence of having the Submissions WG meetings on the first Friday of every month, with adjustments as needed.
• A few members on the call were not clear on how to get invitations properly on their calendar. Theresa shared the ways anyone can add the working group meetings on their calendar. However the group agreed that having a dedicated page on the Working Group site for how to join the group and the various communication platforms would be a beneficial.
• Eric will create a new page on the site to hold this information. In fact that page is now live! Visit the How to Join page for complete details.

Pilot 4 Updates

• HyeSoo shared concerns regarding the appropriateness of zip file formats for program delivery under FDA specifications. Suggestions to explore new architectures or formats for future submissions were raised, particularly for web-based applications.
• Testing for the Docker container component of Pilot 4 is ongoing, thus far no issues but Eric will share more updates in the next meeting.

Working Group Coordination

• Theresa Christiani introduced an inter-organization coordination committee to streamline overlapping efforts across working groups (R Consortium, PhUSE, Pharmaverse). This committee will promote efficient use of resources and avoid redundant work. Robert Devine volunteered to present at the next coordination meeting.
• Robert will present at the R Consortium’s inter-organization coordination committee to share the group’s current projects and align efforts across R Consortium, PhUSE, and Pharmaverse.
• Those interested in attending that coordinating meeting can contact Theresa for getting on the invitation to those meetings

Analysis Data Reviewer’s Guide (ADRG) Updates

• The PhUSE working group on open-source metadata documentation is working to update the ADRG template to better support open-source submissions, with feedback from FDA and industry representatives.
• Lovemore gave a recap of a presentation about the ADRG workstream at the PHUSE FDA quarterly meeting
• The team discussed the potential use of Quarto (QMD) and HTML formats for more accessible, navigable submissions, as well as potential supplementary documents for open-source submissions. Paul confirmed that R Markdown (.rmd) format has already been approved as a file type.
• Joel and Lovemore will set up a future meeting to gather additional industry feedback as a first step to developing a new prototype of the ADRG using enhanced metadata and the Quarto format.
• While Quarto is an approved software package at the FDA, and Quarto file format (.qmd) is part of the approved file types within the Modeling and Simulation section of the eCTD Format Specifications document.
• Paul and HyeSoo will touch base with Helena Spiglin (part of the FDA committee currently assessing the .json formats as an alternative to SAS Transport .xpt) for confirming the Quarto and HTML formats can be added as supported formats for ADRG submissions.

Data Set JSON Format and Alternative Data Formats

• The team discussed exploring JSON and alternative data formats like Parquet, given the large data sizes in modern studies.
• Although the industry largely uses JSON, there are concerns about its efficiency for high-volume data. Parquet, used internally by several team members, offers a promising alternative, particularly for big data applications.
• For a comprehensive overview of the benefits and tradeoffs between JSON, Parquet, and other formats, see the recent R/Pharma 2024 presentation An arrow towards the future: A critical look at data formats for clinical reporting by Craig Gower-Page (recording link).
• The FDA has primarily used JSON but is open to considering alternatives like Parquet with further testing and demonstrations.
• Paul mentioned he is not aware of previous efforts to convert the publically available CDISC data sets utilized in the previous pilot submissions to JSON format and then perform statistical analyses.
• Opportunity for Pilot 5: Using the structure of Pilot 3 as a template, leverage JSON as the format to assess performance. Ben Straub will draft a proposal for an initial review at the next Submissions Working Group meeting in December.
• Nicholas Masel shared that the {datasetjson} R package will need updates to align with the latest schema (1.1) to ensure compatability with the future Pilot 5.
• In addition, Nick and Ben will gather input from industry colleagues and the FDA to determine if Parquet is a viable alternative to JSON for high-volume data in future pilots, and discuss any potential technical or bureaucratic limitations with FDA.

Future Pilot Projects

The team plans to prioritize two future pilot projects:

• Pilot 5: A repeat of Pilot 3 but using JSON format to assess package performance and ADRG requirements.
• Comprehensive Benchmark Pilot: A longer-term pilot for H2 2025 using a large, realistic simulated data set to evaluate JSON vs. Parquet for performance, usability, and compatibility.

Administrative Updates

• The working group welcomed Terry Christiani, the new executive director for the R Consortium. We are greatly appreciate the important leadership provided by Joseph Rickert, the previous director. More information can be found on this R Consortium Blog post.

Submissions Pilot 4

• The submissins pilot 4 (web-assembly version) was transferred on September 20, 2024 successfully and received by the FDA reviewers through the eCTD gateway transfer. The news was also shared in this announcement post on the R Consortium blog.
• Hye Soo informed that their internal reviewer computers required an update to R version 4.4.1 to match the version used in the pilot submission. She also verified that there were no issues in rendering the Shiny application with the web-assembly method of execution.
• A small issue with column labels being switched on Primary Table was identified (a legacy issue that unfortunately was carried forward from an earlier version of the Pilot 1 output). While this is not a critical issue for the scope of pilot 4, Ben Straub shared how leveraging continuous integration / continuous deployment (CI/CD) for verifying the integrity of the data and outputs could be helpful.
• The full review of the submission is ongoing.
• Eric Nantz shared that the Docker container version of Pilot 4 is undergoing additional testing with the working group, and the goal is to have a transfer to the FDA reviewers in November (exact timing TBD).
• Communication with the Nature journal on a potential manuscript is ongoing. Michael Kane noted that additional journals such as the Journal of Statistical Software are another possibility for sharing the concepts of web-assembly in clinical submissions.

Analysis Data Reviewers Guide (ADRG)

• Joel Laxamana shared the ADRG Phuse working group established quarterly meetings starting on October 16.
• Initial steps involve identifying all of the pieces required for an ADRG, and how can templates be enhanced using the learnings from previous open-source efforts such as the Pilot submissions.
• Additional members from the submmissions working group will be invited to future discussions after the initial brainstorming of high-level ideas.

Repositories Working Group Notes

• The Repositories Working Group is continuing their efforts to deliver a reproducible environment structure in which each organization can adopt to fit their needs.
• It was recommended to invite Doug Kelkoff to a future submissions WG meeting to stay updated with their efforts

Future Pilot Ideas

JSON Data Format

• The group is interested in learning more on the current progress of the Dataset-JSON effort.
• Mike Stakhouse will share more information on the {datasetjson} package in the next submissions working group meeting.

Analysis Data Reviewers Guide (ADRG)

• Joel Laxamana is co-leading a new PHUSE working group on updating a template ADRG to account for open-source software such as R. A preliminary site for that working group is available at https://phuse-org.github.io/OSDocuMeta/. Note that it is in the early stages of development, but it will be updated as progress is made.
• Attendees from the meeting interested in learning more and joining that working group are encouraged to contact Joel.
• With the success of our previous pilots, attendees expressed interested in using a similar workflow to provide support for the PHUSE working group. For instance, Sam Parmar has already created a template ADRG in Quarto that could serve as inspiration: https://github.com/parmsam/quarto-adrg

Pilot 4 (Web-Assembly)

• Eric shared that the repository is just about ready for transfer. The tentative date for the transfer of the Pilot 4 bundle through the eCTD gateway is set for September 20, 2024.
• Paul and HyeSoo asked if there is a desire for a small test before the transfer. Eric mentioned that from his perspective a possible issue could be access to the WebR binary package repository, which is a different repository than the CRAN repository and its mirrors around the world. Eric mentioned that there will be two different ways the web-assembly bundle can be executed:
  • Method 1: Utilize the pre-compiled bundle that contains the R packages as well as the application itself. In this way, the WebR binary package repository CDN should not need to be contacted from the reviewer’s machine
  • Method 2: Compile the web-assembly bundle upon first launch of the application, which will need to access the WebR binary package repository CDN to install the packages.
• Eric will arrange a dedicated testing meeting for the Docker container setup and testing to see if a Docker container can be built on the reviewers’ machines.

Potential Manuscript on Web-Assembly in Life Sciences

• Paul requested a change in the proposed title of the paper to remove any references to “timelines.” Having any hint of that terminology could introduce additional policy issues during the disclosure process on their side.
• The plan is to propose the manuscript for publication in Nature. It was noted that the fee for open-access manuscripts is $11,000, and we would need to explore sponsorship from the Consortium or other avenues to address that cost.
• Eric will follow up with the editor he collaborated with on the recent Nature article No installation required: how WebAssembly is changing scientific computing to inquiry on interest from the editors at Nature to pursue this manuscript
• Michael Kane mentioned that he is an editor of the Journal of Statistical Software and there could be an opportunity to publish a similar manuscript on this topic in that journal.

Upcoming Conferences

• Eric asked the Pilot 3 team if anyone from that group would like to join a presentation at the upcoming R/Pharma virtual conference focusing on clinical submissions. Both Joel and Ben expressed interest in joining that session. Eric will provide more details on timing and expectations once the R/Pharma program is completed.
• Michael noted that the R-Medicine conference next year would also be a potential venue for an additional presentation.

Future Pilot Ideas for 2025

A key focus of the meeting was to brainstorm on potential topics for future pilots in the next year:

• Focused pilot on the ADRG template (more background in the ADRG section above)
• Working with the repositories working group on a robust container image for a reproducible computing environment
• A more comprehensive submission that closely resembles a “real” submission (e.g. a large number of deliverables and outputs)
  • A consistent piece of feedback received at previous conference talks is that the current pilots only have sparse content (only 4 tables and a figure).
  • At the time the previous pilots launched, the publically-available CDISC data was the best avenue for data
• Explore creating a large-scale mock clinical trial data via simulation methods.
  • On top of providing a rich source to generate many submission deliverables, this set could also be used as a robust source to test how statistical methods perform, enabling these assessments to be done truly in the open instead of behind company doors.

Additional Topics

• Ben asked the FDA reviewers if the approaches used in the Pilot submissions could inform a future guidance document by FDA on using open-source software for submissions. Paul mentioned that they officially will not likely make any suggestions on the use of open-source software, as current guidances are meant to be high-level and will stay software-agnostic.
• It is possible FDA could share that sponsors should follow best practices, whatever they happen to be.
• Paul recommended our group work closely with PHUSE who have provided best practices in the past. One potential avenue to explore is the use of the JSON format instead of XPT. Atorus Research has open-sourced a package called {datasetjson} to address the viewing/importing of this format.

Pilot 3

Hye Soo reported that the Pilot 3 review has not yet been completed by he FDA authorities. There are no known problems. The delay is most likely due to a summer slowdown.

Before the meeting Joel Laxamana shared a list of recent and upcoming Pilot 3 related talks.

1. [Pilot 3 Demo : R/Medicine 2024](https://rconsortium.github.io/RMedicine_2024/Program.html)
2. Pilot 3 Demo : useR! 2024 On-line talk
3. Roche-Genentech Coursera training on ‘Hands on Clinical Reporting using R’
4. Joel will also be a part of a PHUSE WG project coming up. Yet to be started, but will share more details soon.

Acton: Ning will begin a draft blog post for Pilot 3 to be posted after the FDA review is complete. The post should provide context for the achievement that is helpful for making newcomers to the use of R in FDA submissions comfortable. For example, multiple R packages were deliberately used in the Pilots in order to signal that there is no one correct way to do things.

Pilot 4

Expanding the Reach of the R Working Group

• Ning Leng proposed that it is time to expand the reach of the R Submissions working group to include clinicians and non data scientists.
• She proposed blog posts and perhaps writing an academic paper noting the nature papers on
  • Array programming with NumPy and
  • WebAssembly.

Action: The group will pursue writing a paper.

R Licenses

There was a robust discussion about the impediment that the GPL licenses pose to the use of R that begins about 40 minutes into the video.

• JBR noted that company lawyers invariably seem to want to impose worst case measures when first encountering first encountering GPL licenses, and that if they do form an opinion about the limits of the license they are reticent to share their arguments publicly.
• It was suggested that companies avoid using GPL packages
• It was also noted that R ecosystem is mostly GPL based. An effort to track down the use of GPL licenses in packages and their dependencies included 72% of he CRAN ecosystem.
• One apparent solution is to avoid bundling their code with GPL licensed packages, but to advise users that they should load the required GPL packages themselves.
• The conversation settled on the notion of what is being shared, distinguishing between the code that produces tables and statistical results and code that modifies R in some way to profit from it.
• Several members described how licensing seems to work within their companies.
• An opinion that was offered that GPL 3 IS very “viral” but unless code was linking directly GPL2 is permissive.
• This Tibco Blog post from 2015 mentions how Microsoft was able to stay on the right side of GPL with its integration of Revolution Analytics code.
• Also see this InfoWorld Article by Sedar Yegulap from 2016 which states:

The easy way to do so and still respect the terms of the GPL is via an “arm’s length” provision. Instead of integrating the R binaries directly into SQL Server, Microsoft could invoke the R software from its own executable as needed, then return results to the host application through a file or a data stream.