Improving Inter-Rater Reliability

 

Challenge

 

When multiple raters will be used to assess the condition of a subject, it is important to improve inter-rater reliability, particularly if the raters are transglobal. The complexity of language barriers, nationality custom bias, and global locations requires that inter-rater reliability be monitored during the data collection period of the study. Identifying these deviations and retraining staff as quickly as possible would improve the quality and consistency of data collection.

 

In addition to addressing the challenges of raters being located globally, the data review monitors may also be located globally, and the study project manager and supervising monitor may also be in another global location.  The client was seeking to identify an Electronic Data Capture (EDC) system that could not only manage the data collection, but could monitor multiple raters for inter-rater reliability. One way to do this would be to compare the score assigned to the observation to photos of the observed area, which would require the ability to upload and download photos, maintain metadata which ties the photos to a specific subject and visit date, allow feedback/queries and their responses, and provide a full audit trail of all data in the system. Ideally, the system would also notify the monitors that rating and photo data is entered and is ready for follow up if any discrepancies or potential inter-rater reliability issues are identified. Similarly, the system would also notify the rater when a feedback/query was entered and needed the rater’s response. Finally, the monitor would be notified when feedback/queries were resolved.

 

Proposal

 

Prelude Dynamics proposed the client use VISION™, an electronic data capture system, which already provides the ability to customize the electronic Case Report Forms to collect the data necessary for the study. VISION™ could be customized to collect basic subject information, physical exams, patient medical history, visit history, study scales, and capture photos of the observation areas.  VISION™’s notification system could be programmed to notify monitors of observation and photo entries as well as resolved queries and notify the rater of queries needing to be resolved.

 

Additionally, VISION™’s Randomization Module could provide just-in-time randomization globally across the study to ensure it would be difficult to predict the subject’s treatment group or it could randomize at the site level depending on the study design. The randomization module could also allow the stratification of subjects based on specific criteria, depending on the study’s needs.

 

The Inventory Management Module could be used to track medications and supplies throughout the study. The inventory module could even automatically place orders to ensure the minimum level of supplies and medications would be on hand. Ad-hoc reports and summaries would assist the monitors in keeping track of the study progress as well as monitor inter-rater reliability.

 

Background

 

When multiple clinical sites are used to collect data, it is necessary to train multiple raters, which requires taking the extra step to ensure inter-rater reliability. Assuming that the scale is sensitive enough and appropriate to use in a multiple-rater study, then any identified inter-rater reliability issues would require retraining of the rater.

 

It is often difficult to identify inter-rater reliability issues, particularly when a rater needs to be retrained. When the rater and the monitor are located remotely from each other, prompt monitoring is difficult because the monitor may only be able to review the data when they make a site visit, or the monitor may have access to the data, but not the related photos that are being used to monitor the observation. The necessity to retrain staff can incur costs to the study, and there are often multiple deviations of data as scales are rescored or removed from the study altogether, which is also costly.

 

Solution

 

Prelude Dynamics trained the staff to utilize VISION™ to capture all study data, upload photos, and take advantage of the system’s custom notification capability .  Raters use tablets to record rating observations directly into the system. As the raters work through various data collection and observation screens, diagrams or illustrations will remind them of areas to be observed and the details of the rating scale, and a rating section will be provided. When scales are completed, photos can be uploaded into VISION™ to represent the area observed. In order to maintain study and data integrity, once submitted, the rater cannot change the scales. In fact, the scales will no longer be accessible via a tablet, and will only be visible to the rater via a read-only summary table which has captured the scale input. The Monitor can view the summary table and uploaded photos, and can compare them to the rater’s input and query the information as necessary. The solution also included:

 

  • Standard notifications to notify the Monitor and Project Manager of the following events:
  • New patient enrollment (i.e., patient successfully randomized)
  • Patient terminated from study
  • Changes to form state from in-work to in-review, in-review to reviewed, reviewed to finalized
  • Query resolved
  • Serious adverse events occurred
  • Observation scale and/or photos uploaded
  • Custom notifications would be created to notify of the following events
  • Query entered by Monitor (notify the site personnel)

 

VISION™’s Inventory Management Module will be used to keep track of both medication and supplies for this study. The module will also be set up to automatically order medication and supplies when they meet the minimum threshold to ensure the study progress is not compromised.

 

VISION™ additionally provides mid-study monitoring capacity through pre-set summaries, search/filter, and ad-hoc reporting. One of the standardized summary reports contains a listing of all queries both resolved and open. This will assist the monitor in keeping track of the progress of the study. For additional mid-study monitoring requiring a statistical package, export is available 24/7/365 for those roles with the appropriate permissions.

 

Results & Benefits

 

As a result of using VISION™ to capture study data, the Monitors are able to better track the study progress and better able to ensure inter-rater reliability. The monitors receive emails notifying them that a new scale and photos have been uploaded. They are able to manage their time better and provide prompt feedback. Designated site personnel receive an automatic email notifying them if a Monitor enters a query that needs resolution. Queries are promptly resolved when information is fresh in the researcher’s mind. As a result, the quality and reliability of the study data is improved.

 

The overall or primary Monitor is able to communicate directly and without delay with globally located site/country Monitors about any concerns over the scale ratings. The primary monitor has access to all scales from all sites, including photos and scale results, and can communicate to resolve any concerns or discrepancies identified. This helps ensure overall inter-rater reliability.

 

The Monitors and site report that data and query notifications through the system saved time, ensured quick responses, and kept them connected to the data and personnel through the study. This is especially important because all to the participating sites are active busy clinics that have their own patients and not just study participants.

 

Additionally, the site found it very valuable to have VISION™ keep track of the inventory, citing the efficiency of a system which automatically updates as patients are provided with medication and supplies, rather than keeping separate track of inventory. The automatic reordering feature reduced the stress associated with ensuring they always had enough supplies on hand to continue study progress. The Project Manager also found this feature reduced stress and logistics that otherwise would have posed a headache to handle. Imagine ordering supplies for different sites in different countries and keeping track of that! It would have been a full-time job just to do that.

 

The Project Manager and primary monitor tracked the study using the summaries and ad-hoc reports, staying up-to-date with the enrollment and progress at each site. They were also able to review the mix of subjects to ensure there were going to be sufficient subjects of each age and gender, and specifically noted how easy it was to copy and paste a report from VISION into a spreadsheet so they could provide monthly updates to the Sponsor.

 

Overall, everyone participating in the study found that VISION™ made the study run more efficiently and reduced their stress over participation in the study. Edit checks ensured the data was collected appropriately, and immediate feedback for missing or out-of-range data could be immediately collected  or verified as correct. They also found that it greatly improved inter-rater reliability which gave them more confidence in the data they collected throughout the study.

 

About Prelude Dynamics & VISION™

 

Prelude Dynamics is a global provider of customized web-based software systems for data collection, analysis and management of clinical trials, studies and registries. We streamline eClinical operations through our unique and innovative VISION™ software system, which allows us to rapidly configure data collection solutions for pharmaceutical, CRO, medical device, animal health, and university organizations.

Vaccines – Tracking Multiple Doses

Challenge

Vaccine dosing studies usually require information to be entered with high accuracy in a short amount of time. Blood draws often occur every 10 to 15 minutes during an initial time period and then hourly or every several hours for a second time period. When collecting information on paper for several cases at the same time, this can become challenging since the data requires the phlebotomist’s signature, and the date and time the information was recorded. Additional observations, such as the case’s physical reactions (injection site reactions, mood, level of energy or lethargy, etc.) are often required in addition to lab data.  Depending on study objectives, the extent of data collection can vary and collecting large amounts of data in a short time on paper can be stressful and prone to errors. Further, when labs results are returned and analyzed, the results must be integrated with existing patient data. Each data entry process on paper introduces the change for human entry errors. The challenge was to develop an electronic system that would streamline the data collection process, reduce errors, and stress.

Proposal

Prelude Dynamics proposed the study use VISION™, an electronic data collection system, to allow researchers to capture information in real-time, follow the natural work flow of the study process, and ease data collection. The proposed system would automatically integrate lab results into patient files, compare them to established lab normals and flag outliers, and allow the researcher to determine if the flagged outliers were clinically significant.

Background

Vaccine trials usually consist of understanding real-time antibody development occurring post-introduction of the vaccine. Generally, different dose amounts are compared to determine the optimal dose to produce an effective vaccine. This requires the researcher to collect multiple blood samples after initial dosing. The intervals of the blood collection are typically every 10 to 15 minutes for an initial time period and then hourly or every several hours for secondary time period.

During each blood collection event, additional data on various physical responses to the vaccine is also collected (blood pressure, heart rate, etc.). Depending on the method of vaccine administration, other observations are recorded, such as injection site redness, heat, and pain if administered via injection, or coughing rate and breaths per minute if administered via breathing. These observations demand close attention from the researcher and increase the stress level the researcher experiences during data collection.

Historically, vaccine studies are costly because the researchers-to-subjects ratio of at each data collection event is low to allow enough time to collect the data. Development of an electronic data capture system that streamlines the process can increase the accuracy of data collection, and allow for more subjects to undergo the vaccine therapy simultaneously, ultimately lowering costs, decreasing data collection time, and reducing time to market.

Solution

Prelude Dynamics proposed to utilize VISION™ to develop an electronic data system for vaccine trials that:

  • Includes an integrated time management system that prompts the researcher to administer the doses or collect specific data within designated timeframes.
  • Expedites data entry of dosing or blood draw by collecting researcher information, data and time stamp for each data collection event with one button click.
  • Standardizes data collection by allowing researchers to select designated options rather than writing out observations.
  • Allows researchers to augment the clickable answers with additional details when pertinent.
  • Alerts researchers immediately of missing required data to ensure the data is collected prior to the data collection event.
  • Integrates lab analysis into patient files, including notifying researchers of the receipt of lab results and flagging outlying results based on designated lab normal values.
  • Allows monitors and sponsors to view data collection in real-time even when they are located remotely so that if data needs to be queried, it can occur quickly.
  • Allows sponsors and researchers to examine demographic results of the data collected to date to allow mid-study monitoring.

VISION™ contains administrative panels that allow the sponsor to determine the frequency of vaccine dosing and challenge dosing and blood draws. Acceptable timeframe deviation ranges can be entered into this panel. When site personnel or researchers initiate the time clock for the case by clicking the “now” time button, VISION™ automatically starts the countdown to the next event and displays any data that must be collected in the interim. When the “now” button is clicked, the user’s ID, date, and time stamp are recorded in the system, eliminating time to write the dame and time, reducing future data entry costs, and increasing the accuracy of data collection. Once the researcher clicks “save,” the data entered is hidden. When it is time for the next event (blood draw, dosing, etc.), the system alerts the researcher on the screen and displays the data to be collected for that event. Because VISION™ is an online data collection system, field researchers are able to access the study and enter data on any mobile device.

These features allow for dosing and blood draws to occur for multiple cases in a tiered fashion, which maximizes the researchers’ time without compromising data integrity. When multiple cases are being entered simultaneously, switching between patients can be done either through the list page or by entering the case ID and hitting the search button. The system is constantly communicating with the researcher what needs to occur next and what data needs to be collected.

The additional data to be collected between doses or draws is standardized in order to allow entry through selection from a drop down menu of choices, radio buttons or checkboxes. When additional information needs to be added, comments or notes can be added directly to the pertinent field. Additionally, if needed, free text fields can be provided for entry of comments or in response to certain questions. Edit checks that fire when the form is saved alert the researcher to missing or out-of-range information. VISION™ collects a full audit trail of all data recorded in the system, and each data point is stamped with who entered the data when (date/time).

While lab analyses do take time, VISION™ allows the labs to enter data directly into the system, and the researcher receives an email notification of the lab data arrival. The lab results are automatically compared with the designated lab normal ranges and out-of-range data is flagged to allow the researcher to determine if it is clinically significant.

Remotely located sponsors or other researchers can access data in real-time, which allows queries to appear in the system at the site during data collection. Site or field personnel respond within the system to the queries, and can collect additional data immediately.

For vaccines using injections as the administration route, digital photos can be captured and uploaded into the system. An additional module allows measurements to be collected.

VISION™ also contains pre-existing summaries which assist in monitoring the data collected at the patient level, at each site or for the study as a whole. Using VISION™’s search and filter capacity allows these summaries to display data for only cases fitting the search criteria. Additionally, designated users can create ad-hoc reports to monitor additional criteria and answer important research questions. The data within the system is available for export 24/7/365 with the appropriate user permission. Export features the ability to export in multiple formats, including SAS-ready format.

Results & Benefits

During the initial testing of the system, researchers collected data both on paper as well as within VISION™. Within minutes, it was obvious that the VISION™ solution was far superior, reduced time and data entry errors, and was well liked by the researchers. In fact, the users were so impressed with VISION™ that they called upper level management to come observe the difference at the last minute. Upper management was also very impressed at the capacity of VISION™ to streamline the process, and agreed there were cost savings and efficiency benefits to their studies.

Real-time data is entered into VISION™ and has eliminated the need for paper CRF collection. Researchers spend less time entering data and the data entered is standardized as they select from designated options. In cases where text entry fields are available, as items are entered, they become available for future selection in a drop down menu adding real-time standardization to data entry.

Additionally, data is accessible in real time for review, analysis, export, or additional data entry. Users have access to all digital photographs, and labs are compared to designated lab normals and outliers flagged for analysis. Users report the streamlining of the process has helped reduce errors and improved efficiency.

They are currently working on standardizing additional data collection forms so that they can be used as templates for future studies further standardizing their studies. Meanwhile, Prelude is working on developing a way to provide them with the ability to pick the forms they want to use from their existing library of forms via an administrative panel so they can design future studies faster and at a lower cost. Additionally, within each form, they will have the ability to select what modules of information they desire to use on the form for that particular study.

About Prelude Dynamics & VISION™

Prelude Dynamics is a global provider of customized web-based software systems for data collection, analysis and management of clinical trials, studies and registries. We streamline eClinical operations through our unique and innovative VISION™ software system, which allows us to rapidly configure data collection solutions for pharmaceutical, CRO, medical device, animal health, and university organizations.

Capturing Subject Scales

Challenge

Collecting data entered on a Visual Analog Scale (VAS) using paper is always subject to data measurement and entry errors. The challenge was to replicate the paper process electronically in order to eliminate post-data collection measurement and data entry errors. This would eliminate the need for multiple measures and duplicate data entry reducing errors and cost. This was especially important for this study as each data collection period consisted of 26 VAS and was repeated 5 times per subject throughout the study.

Adding to the challenge’s complexity was the necessary capability of the VAS line to be standardized to exactly 10 millimeters or expanded to fit across the screen of the data entry device. This complexity required logistics development to standardize the analysis and/or data capture of the information when different line sizes are used by different individuals and devices. In this case, the data collection location required the use of mobile device to collect the data. Further, if a subject received a certain combination of subscale scores during the pre-screening or follow up visits, the researcher was to receive an alert that the subject needed to be withdrawn from the study.

And finally, the Sponsor required the ability to monitor progress mid-study, including a table of results and a graph across time on a per-subject basis, as well as aggregated across sites.

Proposal

Prelude Dynamics proposed to use VISION™, an electronic data collection system, to accurately capture real-time study data and allow study participants to respond to questions using a VAS on a mobile device.

An administrative panel was proposed so that site-based researchers could provide access to appropriate set of scales.  The scales designation would be set up such that the VAS scale could be fixed or varied, depending on the researcher’s preferences or the visual ability of the study participant.

The proposal included tables and graphs designed to aggregate the data for mid-study review, including on a per-subject basis or across sites or study visits. Users could further refine mid-study reviews by selecting between complete scales or a single subscale. Utilizing VISION™’s standard features available to all clients, the Sponsor could customize ad-hoc reports as well as search/filter the data to be analyzed in the tables, graphs and ad-hoc reports.

Background

Using paper-based VAS creates a cumbersome post-data collection process because the responses have to be accurately measured and recorded. This generally requires a minimum two measurements, the comparison between them to ensure they fall within an acceptable error range, and the collection of an additional measure when they do not. Each measurement is an opportunity for error, and the process is time-consuming and costly. Nevertheless, VAS scales are popular because they provide an accurate scale and allow more variability to be measured between subject results. Instead of being confined to 10 data points, a maximum generally used in Likert scales, VAS allows for 100 data points to be measured when the standard 10 cm line is used and measures are taken every millimeter. Likert scales are also limited by the bias introduced through the labeling of each choice. VAS provides a greater degree of accuracy and are known to better detect clinically significant changes.

Solution

A solution to the issue is to allow subjects to enter information on an electronic VAS directly onto a mobile device using VISION™, an electronic data capture system. Prelude provided the researchers with an administrative panel where they could enter basic subject information, assign a temporary password, and activate each survey. The researchers then instructed the subjects on how to change their password and use the system. As subjects entered the data directly on the mobile device, it became available for review on the administrative panel as well as for analysis via summaries (tables and graphs) and for use in ad-hoc reports.

VISION™’s standard summaries, which assist in monitoring the data being collected on a per-subject, per-site or whole-study basis, were customized so that researchers could view tables and graphs across time on a per-subject basis, or a scale or subscale basis. Using VISION™’s search and filter capacity allows these summaries to display data of only the cases fitting the search criteria. Additionally, certain users have permissions to create and export ad-hoc reports, which can be used to monitor additional criteria or to answer research questions. The data within the system is available for export 24/7/365, including the ability to export in multiple formats (e.g., SAS-ready format).

Results & Benefits

As subjects select a data point on the VAS, real-time data is entered into VISION™. This eliminates the need for post-data collection measurement and data entry, reduces the opportunity for errors, and lowers cost to measure and duplicate data entry. VISION™ also prevents a subject from changing an entered answer, which ensures they answer in real-time and are not able to enter the information retrospectively or prospectively. Results are automatically calculated for the subscales. If a subject’s scores fall outside of the required threshold, site personnel receive immediate feedback that the subject does not qualify for the study.

Another VISION™ advanced feature is the Randomization module. When a subject is removed from the study, VISION™’s Randomization module works behind the scenes to reassign the removed subject’s treatment code to a new subject. This ensures that sufficient cases were assigned for each treatment group to maintain the study’s viability. All collected data is accessible in real time for query, review, analysis and export.

About Prelude Dynamics & VISION™

Prelude Dynamics is a global provider of customized web-based software systems for data collection, analysis and management of clinical trials, studies and registries. We streamline eClinical operations through our unique and innovative VISION™ software system, which allows us to rapidly configure data collection solutions for pharmaceutical, CRO, medical device, animal health, and university organizations.

VISION Filters Have Been Enhanced to Allow Definition of Study-Wide Patient and Site Filters

AUSTIN, TX – August 17, 2015 – Prelude Dynamics announced today that the study filters have been enhanced. This enhancement allows study management staff to develop both patient and site filters that become available automatically to all users.

This enhancement eliminates the need to share a filter or search and then have the user with whom it is shared to pull in the filter or search before it can be used. Considering the number of users who generally pull shared filters and searches, this enhancement represents a lot of time savings.

The new enhanced study-wide patient and site filters also allow the project manager to define the default filters for the study.

Prelude Dynamics is an industry leader at providing innovative software-as-a-service technology solutions for clinical trials across the globe. The company’s flagship product, VISION, offers secure and configurable data capture and information awareness tools for the advancement of human and animal health, compliant with FDA/CVM and EMEA regulations. VISION facilitates real-time data collection, querying, cleaning, reporting, and analysis all based on the needs of the researchers. The company was established in 2003 with a goal to speed the study process, improve data quality, and facilitate easy data entry through automation.

Contact

Jim Pedzinski – VP of Business Development
jpedzinski[at]preludedynamics.com – 512-476-5100 ext. 210

Prelude Announces New Initiative to Build a “Composer” Build-it-Yourself Study Definition Feature Within VISION

AUSTIN, TX – August 06, 2015 – Prelude is constantly listening to its client’s needs and desired because we are dedicated to improving the clinical trial workflow, experience, and accuracy. Our clients love VISION™ and want to be able to use it for smaller trials and earlier trial phases. After considering the logistics, the Prelude Change Control Board announced a new initiative – a build-it-yourself product called “Composer.”

“We are excited to announce the development of Composer, a build-it-yourself tool that will allow our clients to leverage the power of EDC for pre-clinical and phase I studies that have a smaller number of subjects. We expect it to be available next summer.” stated Prelude Dynamics’ CEO, Richard Tieken

Composer will have a user-friendly intuitive interface that will allow current VISION™ users to learn to quickly build studies within a few days. Composer will leverage Prelude’s flagship product, VISION, as its behind the scenes core. Prelude is certain Composer will soon be the premier build-it-yourself solution in the market place.

Prelude Dynamics is an industry leader at providing innovative software-as-a-service technology solutions for clinical trials across the globe. The company’s flagship product, VISION, offers secure and configurable data capture and information awareness tools for the advancement of human and animal health, compliant with FDA/CVM and EMEA regulations. VISION™ facilitates real-time data collection, querying, cleaning, reporting, and analysis all based on the needs of the researchers. The company was established in 2003 with a goal to speed the study process, improve data quality, and facilitate easy data entry through automation.

Contact

Jim Pedzinski – VP of Business Development
jpedzinski[at]preludedynamics.com – 512-476-5100 ext. 210

Prelude Dynamics Enhances Automated Lab Interface to Allow Direct File Upload and Inter-server Communication

AUSTIN, TX – May 11, 2015 – Prelude Dynamics, a global provider of web-based software systems for data collection, analysis and management, announced today the release of our upgraded automated lab result interface integrated into VISION, our industry-leading EDC and CTMS system.

Integrating labs into EDC can be somewhat of a challenge, but one that must be addressed in order to ensure accurate data is collected. Prelude Dynamics has addressed this challenge by developing a server-to-server interface that automatically post the lab results to the appropriate patient record without the need for the human middleman.

As soon as the labs are available and reviewed by the lab company, the technician saves the results to the lab’s computers as part of their regular workflow. Behind the scenes, our integrated lab interface automatically sends the results to Prelude Dynamics’ servers. This seamless server-to-server interaction posts the data to the appropriate site and subject and sends a notification email to the relevant investigator and other designated site personnel to let them know the lab results are posted and ready for review.

“Prelude is a technology company that is constantly seeking ways to streamline workflows to improve clinical data trials. It is natural to develop server-to-server interfaces eliminating steps that can inadvertently introduce error.” stated Prelude Dynamics’ CTO, Alicia Browner

Prelude Dynamics is an industry leader at providing innovative software-as-a-service technology solutions for clinical trials across the globe. The company’s flagship product, VISION, offers secure and configurable data capture and information awareness tools for the advancement of human and animal health, compliant with FDA/CVM and EMEA regulations. VISION™ facilitates real-time data collection, querying, cleaning, reporting, and analysis all based on the needs of the researchers. The company was established in 2003 with a goal to speed the study process, improve data quality, and facilitate easy data entry through automation.

Contact

Jim Pedzinski – VP of Business Development
jpedzinski[at]preludedynamics.com – 512-476-5100 ext. 210