All right, we are going to go ahead and get started with the next session. So this is FDA Hot Topics. We'll be providing a quick overview of basic FDA topics and then open the floor for questions and discussions. My name is Jennifer Kozen. I'm the Assistant Director for the External Heart Rhythm and Rate Team. I'll pass it down the line. Hi, I'm Luke Ralston. I'm a Scientific Reviewer in Jen's team and I've been working with the agency now for about 18 years, emphasis on AIML devices. I'm Stephen Browning. I am the Assistant Director for the Other Diagnostics Team, which we call Heart Failure and Hemodynamics. And I've been at the FDA for about 12 years. Rob? Hi, I'm Rob Kazmerski and I'm Stephen's team lead and I've been at FDA for about 10 years. Great, thanks. All right, we're going to open the session with a few short slides and we'll have plenty of time for discussion afterwards. Luke? I guess that's me. Okay, thanks for everybody making it out here late on the first day. So we just wanted to start with a little background. Right now we have it called Harmonization Efforts and I think it helps to kind of start at the beginning why harmonization was needed. Way back when I started, we just kind of had one overarching software guidance document. We kind of had an ancillary one on verification and validation. But I think the thought at the time was that one document is going to cover everything that you need to know about software or software-related issues. And we have since learned that that's not the case. So starting back in 2013, our recently former center director, Dr. Jeffrey Shuren, helped to stand up the, let me make sure I get this right, International Medical Device Regulatory Forum. And the purpose of the IMDRF is really to just accelerate international device regulatory harmonization and convergence. I know that that sounds a little bit like a mouthful and maybe some buzzwords, but I think that it's really important because FDA doesn't do what it does in a vacuum. We obviously want a lot of input from the stakeholders in the U.S., you know, innovators, developers, manufacturers, marketers, and that type. But we also saw that across the world there was a bigger need for the IMDRF and to help us discuss these issues with our counterparts since I'm sure a lot of you are aware that FDA is sometimes the only stamp of approval that a company needs in order to market medical devices and that once it says FDA cleared or approved, other companies are perfectly willing to say, we're willing to pay for this, we're willing to put this in our health system. And so it's been our effort to reach out and really bring in as much outside expertise as possible, outside experience as possible. I think right now my last count, I'm looking at my managers to help correct me, we have ten full countries that are part of IMDRF with a number of affiliate countries. So that got off the ground in 2013. In 2014, what we tried to do is introduce a bit of a modified risk framework. I think for those of you who are familiar with software and probably our legacy software guidance document, there were three basic areas, low, moderate, and high risk. And we felt like that that probably wasn't covering everything that we cared about in software. So what the framework really tries to do, and if you probably, I'm assuming you can't see it on those slides, but along one access you have the criticality of the conditions. So is this software giving you critical information? Is it giving you, you know, moderate information, nominal information in terms of when do you need to act on it? Along the other access, you're going to see more or less what intends to be done with this. Is this information coming out intended to treat or diagnose a disease? Is it intended to drive clinical management? Or really is it intended to just, I don't want to say for information purposes, but is it intended to just inform the doctor as to possible problems in the future? So that was the first thing that IMDRF tried to do was just make those things concrete. The second thing that we as FDA tried to do in addition to working with IMDRF, we've really drilled down since 2015 on separating out that single software guidance that I talked about first of all into some discrete categories that really were not getting very much, they weren't getting very much love, they weren't getting very much attention, but it was assumed that somehow they fell within the previous guidance, which they didn't. You'll see in 2015 the quality management systems came out. I think the big one for us has been 2020, which is our cybersecurity principles and practices. I, for one, am not a cybersecurity expert. We have Priya Aswani over here who has been nice enough to join us. I highly encourage you to stop by our table and talk with her about cybersecurity practices because I'm sure I'm preaching to the choir here to say how critical it is. And then in 2021, maybe a year or two late, two or three years late, we really started to recognize the importance of machine learning and AI. So we published these good machine learning practices for device development. And I think that those are not in draft form anymore. I think that those are good to go. And then starting in 2023, the predetermined change control plans or PCCPs for AI enabled devices, and then you see a little bit of a gap on the right side of that line. The last week of April, so just about two weeks ago, we published and expanded a predetermined change control plan for all medical devices because they're now a statutory requirement that was dictated by Congress in late 2022. So please, if you haven't had a chance, go ahead and take a look at both of those guidance documents. And if I'm not mistaken, I know the second one, I'm pretty sure the first one is still open for comments. So if you have any additions or improvements or anything like that, we'd love to get outreach from you guys, from the user community about things that are clear or unclear, areas that we missed, areas that we can strengthen. So yeah, that's it for harmonization efforts. So for AI device authorizations, I think I touched on this briefly, but as we become aware about how important AI is, and really this entire conference is kind of about that, is we wanted to make some type of an external facing webpage or external facing area where we can consolidate this information. And our idea is that at the bottom line, AI regulation is really about transparency. We don't, and let me really draw a line under this, we don't know everything. I think everybody that you hear talking today is going to say that this is an evolving science. It's very specific to the intended use of the device. And so while we might clear a device with a certain data set and certain metrics and certain thresholds, we as FDA don't know everything, but we do believe that if we make the data available to everybody and analyzable to everybody so that they can at least make apples to apples comparisons, that that's going to be a net good. And so that's kind of, you see a screenshot on the left side here of, I think, you know, the first 20, 15 or so of these, what panel they fall under. You'll notice that a lot of them historically have been radiology, but that more and more of them are becoming cardiovascular. And the intention is really twofold. It's a public resource so that all those out there either developing it or using these devices can see the work that the FDA has done in this area. And that it's also intended to show how AIML is being used across medical disciplines. So the caveat there is it's not an exhaustive list because sometimes the review process itself can be not as targeted as we'd like, but we do have a flag that we put up on these and they should make it to the list sooner or later. For those of you who are product developers, if you believe your device is AIML enabled and it doesn't make it on the list, shoot us an email. It's not a whole lot of work to put it up there. Next issues. So we got six of these. Some of them are cross-cutting. I just want to focus on a couple of the ones that we've been seeing most often. So performance drift. I think a lot of you will maybe have heard this term, depending on who you talk to, it's got different terms. If you talk to computer scientists, they'll say performance drift or data set drift. If you talk to statisticians, they tend to say things like performance bias. What does that mean? It really means research by the developers, by the companies about the clinical use of these in real life. What does that mean? We all have data sets that we collect, we adjudicate, we train the models on, and then we deploy them. And the training is all well and good, but how does it function when you deploy it? Does the intended user population change in such a way that the metrics start to deteriorate? That's a real problem. That's one that we've seen. And so we really think that doing a good job of adjudicating your data sets before training and testing is one critical key. Another one that I'll plug in, and I'll say right up front, we don't have any teeth to enforce this, but I think that it's a good thing to think about, is post-market monitoring of how your models perform. Second one I'll talk about is data generalization. So I love Andrej Karpathy. I don't know those of you who are aware of him. He's one of the wonderkins of ML in the last 10 years. I think he's best known for being the leader of Tesla's self-driving program for many years until recently. He liked to say that data sets need to be large, clean, and representative. So what does large mean for us? Large for Facebook is different, right? Facebook probably has 100 million data points that they can train their stuff on. For medical devices, large is probably something different. Large is probably a couple thousand data sets, right, that we want to train on, maybe 10,000 data sets if we're lucky. I think it bleeds into the retrospective data sets if we really want to go there. Right now, retrospective is kind of the best we have in a lot of areas, and it's not perfect. There's a lot of missing data. There's a lot of unrepresentative data. But if you put in the work, I think that we can get to those data sets that are large enough for training. And then for testing, explore how much do you need. How much do you need to prove your endpoint? How much do you need to prove your performance goal? The clean data sets is relatively self-explanatory, but I'll put in one small caveat. I'll take an example, the MNIST data set, which some of you are familiar with. It's just a data set of zeros through nines. You can call it up in almost any library now and train a model that will identify one, two, three, four, five, six, seven, eight, nine, zero. That's really easy to do. Why? Because all of us write numbers every single day. And so any one of us could probably go through this data set and say, that's a two, that's a four, that's an eight. When we're talking about cardiovascular medicine, that gets a bit more difficult to be able to say that's atrial fibrillation or that's atrial flutter. That's pulseless ventricular tachycardia or that's just high rate tachycardia. So when we talk about clean data sets, we're really talking about the adjudication process that's being used by clinicians to determine ground truth, because that's what we're training for is ground truth. And then the last thing is representative. This is a, I won't go on for another 10 minutes about this, but maybe if you ask a question, I think we need to expand our idea of what representative means. I think all of us have an idea over the last couple of years that the demographics of data sets are critical. We can't just all have 60-year-old men, 60-year-old white men in every atrial fibrillation trial and say, this is going to generalize to the entire population. So demographics are important. I think what gets overlooked sometimes is, you know, what's the hardware used to acquire those rhythms? What are the hospital systems that are being used to acquire these? Every hospital system has slightly different, you know, workflows, not only within the hospital and the hospital system, sometimes different floors of the same hospital. What are we doing to look at those workflows to make sure that they're truly representative of the intended patient population, the intended use of the device? And then I'll touch on generative AI. Why for us as FDA and the devices we see, there's not been a lot of generative AI. Maybe it's just there's a lag in the flash to bang and everybody else is talking about how great generative AI is and we're just not seeing it. I will refer you to a keynote address by Professor Gary Marcus at the most recent AGI 2024 conference. I think that's last year, or I'm sorry, last month. He pours a little bit of cold water about the applicability of generative AI. I'm certainly not qualified to agree or disagree with any of his points, but I think that they bear keeping in mind and some of the recurrent problems that a lot of generative AI models still have in terms of being able to give reproducible output, not being sexist. That's been a big one for a long time that they still haven't solved. So again, Dr. Marcus speaks in depth for this for quite a lot of time and some of the remaining hurdles that we have to go over. Next slide. Sorry, I'm talking for kind of a long time. I'm wrapping it up. This is my last one. So first thing to emphasize on this is that you'll see both of the PCCP guidance documents on the right are in draft guidance. So draft guidances are not, they're not binding. In fact, I think we don't even mention them. If you come in with a PCCP and we talk to you about it, we're not going to mention them, but we do want to put them up here to let you know that they're available. The one on the left, as you can, you might be able to read, I don't know, the writing's kind of small. That was published last year for AIML specific devices. The one on the right is the one I referenced. It was published in August of this year, and it has to do with all devices. And both of them have the same goal, which is to describe a science-based approach to put safe and effective advancements in the hands of healthcare providers and users faster. The initial idea was to have the ability, companies to have the ability to gather additional training data. I'm talking mostly with the AIML devices, and retrain their models and then deploy them so long as they'd come to a sufficient agreement with us beforehand about this is the data we're gathering, this is how we're adjudicating it, this is what the labeling is going to look like afterwards. It's designed to help increase the pace of medical device innovation and facilitate more rapid improvements for AI-enabled developed devices. The last part, I think really the most critical part, is placing that emphasis on clearly communicating information. How do you update the labeling? That's a big question, right? Do people read the labeling? If they don't read the labeling, do they look at the user interface? How does the user interface change so that the users know how the performance is changing? Presumably, it's getting better, right? But does it have caveats to that better performance? Is it only better in a certain patient population? So these are the kind of questions that we really think the PCCPs are designed to answer. Be happy to take questions about that afterwards. I realize I've been rambling for a while, but I'll turn it over to Stephen now. I forgot mine was next. Can you go one more? Oh, yeah. So, you know, one sort of illustrative example that we talk about at these conferences that sort of helps describe the complexity that we're dealing with in these devices and challenges and validation is cuffless blood pressure. So when we say cuffless, that basically means any blood pressure measurement type of device that doesn't use the cuff on an upper arm that you're used to at the doctor's. You know, as we move into this new types of devices, we're challenged because those previous to cuff devices have very standardized testing. You meet a very sort of prescriptive standard and you are sort of demonstrated to be accurate. But because of how many different types of these technologies, as you can see on the left of this slide, there is no perfect standard for performance evaluation of these. And there's so many different like particularities of each device that affect that performance evaluation. So we have not written recognized completely recognized a standard for all of them. And they sort of each require a bespoke validation, which has been, I think, challenging for industry. So you know, to give a sense, I think we've talked with the number that we throw around as we've talked to 150 or 200 companies developing a device of this type. And we're sort of now starting to see a few of them become successful, but it's been challenging. So one of the things that I think that you can do to greatly improve the speed and the quality of the interaction with FDA and likelihood that you're going to get on the market is sort of a super clear device description about exactly how the algorithm works. We know companies don't always want to reveal that, but you know, at FDA, we are all sort of by nature, a little bit worst case thinkers. So if you leave some blank space, we are going to fill it in with the worst case. So the more that you can describe to us about it and exactly how it works, the better that we're going to be able to sort of design that bespoke clinical validation. Things to sort of expect as we get there are sort of larger sample sizes for demographics and confounding factors that aren't applicable to those previous generation of blood pressure monitors. Anything that's calibrated to a CUFS device is going to require change and stability tests, which are significantly harder to run than the typical blood pressure accuracy studies. And then, you know, the acceptance criteria that we've been moving forward with, especially from a substantial equivalence perspective, but in de novo as well, is a five plus or minus eight, which is the one that's set out by the old standard, but with the knowledge that you're going to need to different methods to get there. All right. So I think a lot of these items were previously mentioned by my colleagues, so I'll just go through these really quickly. But at least on my team, we see a lot of the ECG devices. So some of the best practices that we've identified is providing representative data. So data that is of your intended use for your device. So your environment of use, if your device is intended to be used in motion conditions, being able to validate your device for that. Technological characteristics, wet versus dry electrodes. See number of leads and lead locations, 12 lead versus three leads. Standard lead locations versus patch type devices. And also demographics, just taking into consideration the presence of certain arrhythmias and BMI, ethnicity, age, that sort of thing. Combatable input signals. So the software and algorithms that your device is intended to be used with. AIML statements. So that's specifically just being really clear about the existence of AIML in your submission. So specifically the functionality that uses AIML and if there are any functionalities that do not, be really clear about that. And also a statement that your AIML is locked and not intended to be changed. Also confounding factors. So I think Luke can mention this, but just taking into consideration in the validation your sample sizes, so race, ethnicity, BMI, comorbidities, any type of subgroup analysis that you might need to consider as well, and diverse sites. So ideally at least three geologically separate sites so that they're independent for testing and validation. And hopefully that can, you know, alleviate a lot of the problems that we see. Next slide. So we just wanted to quickly share some of the digital health resources that we have available. So first is the Digital Health Policy Navigator. This is an external facing website. It's a tool that can help you determine if your product software functions are potentially the focus of FDA oversight. It walks you through various questions to get to that answer and also has links and other resources to guidance documents. We also have a Frequently Asked Questions, or FAQs, that could maybe potentially answer a lot of the questions that you have already. And I will also put a quick plug in for the FDA lounge. It's over in that back corner and we'll be there for the next couple days. Happy to chat with everyone further on the technology that you're developing and any questions that you might have. Next slide. All right, so thank you. And this is the email for digital health staff if you want to contact anyone from there after the talk. All right, so I think we will go ahead and start off with some questions for our panelists. Feel free to input any questions that you have into the app and we'll get the discussion started. So Luke, I will turn the first question over to you. With the draft PCCP guidances that are now in effect, what are some common issues that you see with PCCPs and how can sponsors, what can they do to keep that in mind to be able to have a smooth submission? Yeah, I think the first thing to say is that they are draft, so they're not complete and they're not binding, but we've seen a lot of interest from industry about them. And there are a couple of issues and maybe I'll go, maybe I'll take a step back and I'll try and do my best to explain. At a high level each of the guidances is broken down into three sections. The first is the description of the change that you'd like to implement, the second is the actual modification protocol, and then the third is the impact assessment. The description modification gives you three examples that you can, like very generic examples of types of changes that are generally appropriate for PCCPs. The vast majority that we see though are ones where you have a model, you've trained it, you've deployed it, now you have either an ongoing clinical study or now that you have a device deployed in the field you can collect more data. You'd like to collect more and then retrain it to improve the algorithm a little bit. So that's what we see the most often. Where companies get tripped up is the second part, the modification protocol. I'm not even sure if it's called a protocol. I got to read the guidance a little bit more detailed I suppose. But I'm gonna call it a protocol because I think that that's the best most germane term for it. What we see are a recapitulation of the types of changes that would like to be made and then not a lot of details. And in fact to the extent that there are details it'll generally say for adjudication processes refer to engineering document ABC XYZ. For appropriate metrics and performance criteria refer to attachment one two three four five. The purpose of a PCCP is really to be its own standalone document. And to that extent I think we need to have as much detail as possible. And so to outline these are our inclusion exclusion criteria for gathering additional data. And this is our data management practices and this is how the database is going to be split. And then after that you know the training process these are our performance criteria that we're gonna meet. These are the metrics that we're going to use to measure whether or not we pass or fail. And then the impact assessment is kind of looking back and saying at each process along this way as we've you know superimposed our risk management processes on top of it we think that it will have a positive benefit-risk profile and this is why. So that's kind of I think a high level but the thing I focus on like I was saying is the protocol itself. And the more details that we can get in that the better. The analogy that I like to use I'm not sure how many of you have gone through the FDA process there's the most common one is called a 510 K. And if you're submitting a 510 K for I don't know any given device a screw of some kind that's used in surgery that's going to come in some packaging and you're gonna have to give some some data to show that the packaging is stable over time and that the shelf life is stable over time. And there's a provision within the 510 K area that says if you provide FDA six months of data and you provide a promissory note that says nothing is going to change in our test process for two years you know nothing about the packaging is going to change nothing about the heat the humidity anything's going to change for two years then FDA can say okay assuming everything passed the pass-fail criteria is not going to change either. Assuming you pass in two years you can update your labeling. I think that that might be a good mental model for how to think of PCCPs. You lay out the details you lay out this is how we're collecting the the data this is how we're adjudicating it this is how we're housing it this is how we're splitting it this is how we're feeding it into our data management pipeline and these are our pass-fail criteria at the end of the day. I think that's a good way to think of a PCCP and the modification protocol and the amount of frankly work that goes into it. I say work because it's gonna feel like you're rehashing a little bit of what you've already told FDA and some of the other documents but my other pitch is in the long run it's probably gonna save you work because then the option you have let's say you get from version 1 to version 3 and you think you know we're not making as much progress in the sensitivity as we were hoping for but the specificity is getting really good like we're really honing in on the use the intended use population that we want and we think that that progress is getting very good we'd like to change our performance goals a little bit. You can submit a special 510k at that point to change the PCCP and then if the PCCP is one self-contained document a redline copy to FDA saying these are a new performance criteria and then it's almost I'm not I shouldn't say rubber stamp but hopefully it's a very simple straightforward review process that's easier for you and it's easier for us and you're able to get a better more effective safer product onto the market more quickly. Yes thanks all right I'm gonna move to audience questions since we have a number of them coming in. Could you please elaborate how the AIML algorithm locked or not will change the review of device clearance application by FDA? I can start yeah I mean I think the the presence of AI just sort of adds some considerations to the validation in comparison to a device where you say have a physical model you know there we are concerned a little bit more about over training to the to the training set having a segregated validation set whereas you know when we have a physical model and it's trained based on sort of physical rules we can have a more limited validation set that we are testing set I guess is what we say now testing set because we sort of understand how it's working and the underlying model works. I think that the short answer is that it's it's a lot more testing you know the fixed model is a is a known quantity so to speak because we know that you took that out there you tested it on patients you had this performance so if you have a model that you are planning on having unlocked then I think you need to be able to demonstrate that this unlocked model is performing as expected in clinical conditions that it would be expected and that type of a trial is much more complex to run than just basing your your locked model on a retrospective data set so there's a lot more testing to go into it. I think the other thing I'd add is that in the past probably ten years or so maybe we've had an increased emphasis on having a pre-market post-market balance and so we accept a little bit more risk for questions that can't be answered entirely pre-market we still would accept we still expect you to do a full risk analysis and be able to elaborate on what those risks are what type of data would be needed to support it but in my opinion having an unlocked algorithm and I'm looking at the panel who might disagree with me on this having an unlocked algorithm we haven't seen it yet but I think that that would be a good candidate for a post-market balance in the regulatory ease type of language we speak we talk about 522 studies where you design a prospective study Stephen shaking his head with me. I don't think 522 works. But some type of post-market data to ensure that continued monitoring of it is not hurting the algorithm in any way I think a PCCP would also be a good adjunctive to that but it's not a question we've answered yet and if you're thinking about it in the near future we're certainly interested in talking to you about it. All right so the next question is about cuffless blood pressure devices. You mentioned some companies getting results do you believe this technology is feasible for measuring blood pressure hurt blood pressure or perhaps what would be the best use case? As several devices are on the market so the feasibility is there I think it's extremely challenging area best use case I think I think I'm gonna sort of turn that one back and sort of give one of my stock lines that I say about all these digital health products is that you it's best if you guys pitch the use case to us so what we see a lot for a lot of different products is you know we've done some fancy math we have this great stuff and we have this one medical thing and then you give it to us and worse we're and again sort of talking in worst case thinkers we're sort of thinking about the worst case and exactly how it's gonna be used the better way to do it to approach us is you know hey we have this cool thing here's like three clinical vignettes of like the patient interactions or you know patient stories where we think this device is really well positioned to be used and to sort of move the needle for for X type of patient and if you do that thinking then we can sort of help you with exactly what validation is required to get you there all right the next question how important is it to do validation of ECG or PPG algorithms in an inpatient or outpatient setting for the I think it depends and I know that's kind of taking the easy way out but you know if you're if you're looking at the potential for a patient to have try not to use too many examples that I shouldn't so for instance if you're you're trying to diagnose heart failure patient for BMP values looking at a patient inpatient versus outpatient obviously you're going to have a different range of values it may be important to look at both but again it would depend on whether or not you're indicating your device for the full range of measurements are only one versus the other also if you're looking at heart failure worsening I do a lot of heart failure files so sorry we're going to sit in that area for a little bit it's not applicable to have data from inpatient recordings to be testing your algorithm because by definition they're in a hospital and they're most likely in a state of decompensation so it's not predicting anything at that point it's just verifying they're in the state so again it really depends on what you're intending to do and whether or not those readings are applicable to that intended use so I think if it's unclear please come by and talk to us about your specifics but at the highest level I think it it really just depends what differences characterize three geologically diverse sites and would you accept three diverse sites where the diversity is not just geological but in ways you're looking for in representative data sets for example three different hospitals in New York could provide a more diverse data set than three sites in different states can you describe how you evaluate this yeah so the three different sites is not written in stone anywhere it's something that if you've dealt with us you've probably heard it is written in stone some special control the 2380 special control oh okay I stand corrected government's a big place you can't keep track of everything going on at the same time however three is a good number and there are a couple of reasons for that this goes back to a little bit of the discussion of diversity data diversity that I had a little while ago and expanding our scope of what we consider to be geographically diverse or diverse in general and we think that geographic diversity is one of the key ways of collecting actual diversity and so again demographic diversity a hundred percent important and typical breakdowns of the demographic diversity of a clinical trial is still going to be expected in devices and device trials going forward I think one area that we're starting to become a little bit more educated on is the differences in things like workflow within in between units in the same hospital the differences in EMR techniques you know do you use epic do you Cerner do you something else is it a kluge of a couple of different systems differences in expertise this is a big one differences in expertise across hospitals or across centers of a hospital you would not expect and attending in an ED to have the same level of expertise as in an ICU as in a cardiac ICU as in a neonatal ICU and so all of these matter and reasons that we think that just simply aggregating the information is not probably the best way to go I think one last plug I'll make is that coding practices can differ a lot I don't know if we have any clinicians in the audience coding practices can different a lot so ICD 10 codes or ICD 9 codes and CPT 10 code CPT codes are not the most reliable thing for actually determining ground truth however they're always present why because it comes down to billing right almost all billing is tied to those and so while it's acceptable getting three geographic diverse sites gives you a little bit more insight into more of a representative population that would be using this entire AI system because not just about the patient all the time it's about the input data and a lot of times that's coming from the doctor that's coming from the nurse that's coming from the clinician and so that's a little bit more of the thinking behind the geographic diversity and let me just provide a little bit of transparency into how we would assess that internally typically we have a multidisciplinary review team consisting of medical officer who's familiar with the disease phenotype statistician and then a machine learning expert along with any other people we want to pull into the review and they would have an open discussion based on the information provided by the sponsor saying okay they provided this breakdown of patient population specialties whatever about each site and we think it meets the definition or it doesn't meet the definition for X reasons and just you know to add some color to how those conversations go internally and to speak to the specific example and the question New York City three hospitals in Manhattan might be an excellent place to get an extremely racially diverse data set it also might over index for people who walk to work and sort of live a lifestyle that is quite different than you know Denver Colorado or somewhere elsewhere public transportation and walking is much less common do you have a list of expected bias verification or is it up to the manufacturer to provide all analysis that they believe to be relevant so bias is tricky the short answer is no but we do think that a lot of those basic questions are addressed by the diversity in your data set and really digging into that and providing justifications for why your data sets diverse and then another plug that I will give I don't know if we have any statisticians in the audience today I don't know how much they come to this conference but if you know statisticians make friends with them because this is they are exquisitely educated in the language of bias and I think that the more you can get statisticians involved early they may tell you things you don't want to hear about your data set but this is their bread and butter and I think that that's probably one of the best overlooked resources that you can tap into how do you approach a RX only device change to be both RX and OTC is this captured under PCCP that's not often what we were thinking of in a PCCP I think PCCP is generally we're thinking of changes that you want to make five six seven eight times in a very protocolized way RX to OTC is often only one one time you're making that decision you know if this one's going to be a FDA's favorite answer where it depends when you're when you're switching that indication on what you're measuring I think some of the heart failure stuff that Rob deals with on a day-to-day basis is going to be significantly different than a blood pressure monitor which is sort of a well long history of being available in the OTC space and sort of literature where we expect patients to be able to understand it so it's very much dependent yeah and I'll just say two things that really pop out would be human factors usability you know that the actual user can still understand how the device works how to use it properly and in some cases performance may need to be different it may need to be higher it may need to be lower it depends on the circumstances but those are two things that we typically would assess if we're switching from RX to OTC all right with less than a minute left we have one last question how does FDA judge if a software system is a healthcare information system without the need of applying for clearance or software as a medical device I mean we go to the digital health policy navigator no I mean it's it's it's a very wide open question and there's there's a lot of things that go into it I think I am pointing to the navigator in a non-facetious way because I think we have about a dozen guidance documents that may or may not be relevant to that question but in general if it's starting to prioritize patient information or if it's starting to suggest diagnoses or be doing doing things beyond collating and and collecting information then it becomes possible that there may be device functions there and without more information I don't think I could give a more specific answer but I would highly recommend the digital health policy navigator or talking to us for more information all right well thank you everyone for joining us today if you have any other remaining questions will be available at the FDA lounge for the next couple days

FDA

medical devices

AI/ML

cybersecurity

post-market monitoring

cuffless blood pressure devices

ECG/PPG algorithms

Premarket Change Control Plans