A Novel Ontology and Machine Learning Driven Hybrid Clinical Decision Support Framework for Cardiovascular Preventative Care

Contents

ABSTRACT

CONTENTS

LIST OF FIGURES

LIST OF TABLES

DECLARATION

ACKNOWLEDGEMENTS

GLOSSARY OF ABBREVIATIONS

1 INTRODUCTION
1.1 Organisation of Thesis
1.2 Motivation and aims
1.3 Original Contributions
1.4 Publications

2 LITERATURE REVIEW
2.1 Clinical Decision Support Systems
2.1.1 Ontology Driven Clinical Decision Support Frameworks
2.1.2 Clinical Decision Support Systems in Cardiovascular Care
2.1.3 Cardiovascular Risk Estimation Systems for Disease Pre- vention
2.1.4 Machine Learning Driven Cardiovascular Decision Support Systems
2.1.5 Role of Feature Selection in Clinical Decision Support Sys- tems
2.2 Conclusion and Discussion

3 A Novel Ontology and Machine Learning Driven Hybrid Clini- cal Decision Support Framework for Cardiovascular Preventative Care
3.1 Proposed Framework
3.2 ODCRARS for Cardiovascular Preventative Care
3.2.1 Ontology driven intelligent context aware information col- lection component
3.2.2 Patient Medical Records
3.2.3 Ontology Driven Decision Support
3.3 Machine Learning Driven Prognostic Modelling for Cardiovascular Preventative Care
3.4 Machine Learning Driven Prognostic Model
3.4.1 Data Acquisition
3.4.2 Data Pre-Processing
3.4.3 Feature Selection
3.4.4 Prognostic Model Development
3.4.5 Prognostic Model Validation and Evaluation
3.4.6 Online Clinical Prognostic Model
3.5 Conclusion and Discussion

4 Ontology Driven Clinical Risk Assessment and Recommendation System (ODCRARS) for Cardiovascular Preventative Care
4.1 Implementation of the Ontology Driven Clinical Risk Assessment and Recommendation System (ODCRARS)
4.2 Ontology driven intelligent context aware information collection: Design and Implementation
4.2.1 Ontology Driven Intelligent Context Aware Ontology Model
4.2.2 Adaptive Clinical Questionnaire: Design and Implementation
4.2.3 Proposed Novel Decision Tree based Approach
4.2.4 Dynamic Adaptation
4.3 Patient Medical Records
4.4 Patient Semantic Profile : Design and Implementation
4.4.1 Ontology Development
4.5 Ontology Driven Clinical Decision Support: Design and Implemen- tation
4.5.1 Recommendation Ontology
4.6 Clinical Rules Engine: Design and Implementation
4.6.1 Clinical Rules Data - Patient Fact Representation
4.6.2 Jess: Java based Rules Engine
4.6.3 Partitioning the Rules
4.6.4 Cardiovascular Risk Assessment
4.7 System Implementation: Integration of ODCRARS and MLDPS .
4.7.1 Patient Module
4.8 Doctor’s Module
4.8.1 Integration of the ODCRARS with the machine learning driven cardiac chest pain and heart disease prognostic models
4.9 Conclusion and Discussion

5 Machine Learning Driven Prognostic System (MLDPS) for Car- diovascular Preventative Care
5.1 Case Study 1: Rapid Access Chest Pain Clinic
5.1.1 Background
5.1.2 Aims
5.2 RACPC Clinical Dataset 1
5.2.1 Data Acquisition
5.2.2 Data Preparation
5.2.3 Missing Data Handling
5.2.4 Feature Selection
5.2.5 Prognostic Model Development: Experimental Setups and Results
5.2.6 Final Diagnosis
5.2.7 Evaluation of RACPC Results
5.2.8 Results of Comparative Machine Learning Classification
5.2.9 Analysis of Variance (ANOVA) Test for Performance Eval- uation
5.3 RACPC Clinical Dataset 2: Demonstrating Effects of missing Data on Verification Results
5.3.1 Background
5.3.2 Pre-processing of Missing Data using Probability Estimation
5.3.3 Expectation Maximisation (EM) Approach
5.3.4 Experiments
5.3.5 Classification for the Incomplete Clinical Data
5.3.6 Filling the Incomplete Data
5.4 RACPC Clinical Case Study: RACPC Clinical Dataset 3
5.4.1 Study Group 1: Clinical Risk Factors
5.4.2 Evaluation
5.4.3 Performance evaluation of experimental setups
5.4.4 Study Group 2: Test Results
5.4.5 Evaluation
5.4.6 Performance evaluation of experimental setups
5.4.7 Implementation of online Clinical Prognostic Models
5.4.8 Machine Learning Driven Cardiac chest pain prognostic model’s integration with the recommendation system
5.5 Case Study 2: Heart Disease
5.5.1 Background
5.5.2 Aims
5.5.3 Data Preparation
5.5.4 Feature Selection
5.5.5 Prognostic Model Development
5.5.6 Prognostic Model Validation and Evaluation
5.5.7 Performance evaluation of experimental setups
5.5.8 Implementation of online Clinical Prognostic Models
5.6 Case Study 3: Breast Cancer Prognostic Modelling
5.6.1 Background
5.6.2 Aims
5.6.3 Candidate Clinical Variable Selection
5.6.4 Prognostic Model Development
5.6.5 Prognostic Model Validation and Evaluation
5.6.6 Performance Evaluation of Experimental Setups
5.6.7 Online Clinical Prognostic Model
5.7 Verification and Validation of the Clinical Prototypes
5.7.1 Validation of the Machine Learning Driven System (MLDPS) and Ontology Driven Clinical Risk Assessment and Recom- mendation System (ODCRARS)
5.8 Summary and Conclusion

6 CONCLUSIONS AND FUTURE WORK
6.1 Conclusions
6.2 Discussion and Summary of Contributions
6.3 Future Work
6.3.1 Utilisation of Fuzzy Cognitive Maps for Collaborative Care
6.3.2 Active Manifold Learning Strategy in Machine Learning Driven Prognsotic Modelling based on Big Data
6.4 Limitations

Appendices
A Clinical Experts Validation Feedback
B RACPC Clinical Case Study: Clinical dataset 3 detailed analysis
C Breast Cancer Clinical Case Study: Comparative Machine Learn- ing Analysis
C.1 Kernel Models Implementation with Logistic Regression
C.1.1 Performance Vector
C.2 Random Forest Classification Results

Bibliography

Kamran Farooq : A Novel Ontology and Machine Learning Driven Hy- brid Clinical Decision Support Framework for Cardiovascular Preventative Care

Doctor of Philosophy, March

ABSTRACT

Clinical risk assessment of chronic illnesses is a challenging and complex task which requires the utilisation of standardised clinical practice guidelines and doc- umentation procedures in order to ensure consistent and efficient patient care. Conventional cardiovascular decision support systems have significant limitations, which include the inflexibility to deal with complex clinical processes, hard-wired rigid architectures based on branching logic and the inability to deal with legacy patient data without significant software engineering work. In light of these chal- lenges, we are proposing a novel ontology and machine learning-driven hybrid clinical decision support framework for cardiovascular preventative care

An ontology-inspired approach provides a foundation for information collec- tion, knowledge acquisition and decision support capabilities and aims to develop context sensitive decision support solutions based on ontology engineering prin- ciples. The proposed framework incorporates an ontology-driven clinical risk assessment and recommendation system (ODCRARS) and a Machine Learning Driven Prognostic System (MLDPS), integrated as a complete system to pro- vide a cardiovascular preventative care solution. The proposed clinical decision support framework has been developed under the close supervision of clinical do- main experts from both UK and US hospitals and is capable of handling multiple cardiovascular diseases

The proposed framework comprises of two novel key components: (1) ODCRARS (2) MLDPS

The ODCRARS is developed under the close supervision of consultant cardiologists Professor Calum MacRae from Harvard Medical School and Professor Stephen Leslie from Raigmore Hospital in Inverness, UK. The ODCRARS comprises of various components, which include:

(a) Ontology-driven intelligent context-aware information collection for con- ducting patient interviews which are driven through a novel clinical questionnaire ontology.
(b) A patient semantic profile, is generated using patient medical records which are collated during patient interviews (conducted through an ontology- driven context aware adaptive information collection component). The semantic transformation of patients medical data is carried out through a novel patient semantic profile ontology in order to give patient data an intrinsic meaning and alleviate interoperability issues with third party healthcare systems.
(c) Ontology driven clinical decision support comprises of a recommendation ontology and a NICE/Expert driven clinical rules engine. The recommendation ontology is developed using clinical rules provided by the consultant cardiologist from the US hospital. The recommendation ontology utilises the patient semantic profile for lab tests and medication recommendation

A clinical rules engine is developed to implement a cardiac risk assessment mechanism for various cardiovascular conditions. The clinical rules engine is also utilised to control the patient flow within the integrated cardiovascular preventative care solution

The machine learning-driven prognostic system is developed in an iterative manner using state of the art feature selection and machine learning techniques. A prognostic model development process is exploited for the development of MLDPS based on clinical case studies in the cardiovascular domain. An additional clinical case study in the breast cancer domain is also carried out for the development and validation purposes. The prognostic model development process is general enough to handle a variety of healthcare datasets which will enable researchers to develop cost effective and evidence based clinical decision support systems. The proposed clinical decision support framework also provides a learning mechanism based on machine learning techniques. Learning mechanism is provided through exchange of patient data amongst the MLDPS and the ODCRARS. The machine learning- driven prognostic system is validated using Raigmore Hospital’s RACPC, heart disease and breast cancer clinical case studies

List of Figures

2.1 Example of a Rule encoded in MYCIN.¹

2.2 Hybrid architecture of the rule-engine / clinical knowledge-base preoperative risk assessment system²

2.3 Ontology Driven Breast Cancer Decision Support System³

2.4 Hybrid Clinical Decision Support System⁴

2.5 Hybrid Decision Support Model for Optimal Ventricular Assist De- vice Weaning⁵

2.6 Feature selection process based on wrappers and filtering methods

2.7 Block Diagram of SFFS Algorithm as described by Hicham et al

3.1 A Novel Ontology and Machine learning-driven hybrid Clinical De- cision Support Framework for Cardiovascular Preventative Care

3.2 Chest Pain risk assessment questionnaire encoded in MUMPS, de- veloped by Professor Warner Slack from Harvard Medical School⁶

3.5 A sample ROC curve. The dotted line on the 45 degree diagonal is the expected curve to show that the classifier is making random predictions

4.1 The Ontology Driven Clinical Risk Assessment and Recommenda- tion System’s Generic Clinical Questionnaire Ontology

4.2 Context Sensitive Questionnaire Tree Structure

4.3 Tree Structure detail

4.4 Stack implementation of the context-sensitive questionnaire

4.5 The Architecture of the Ontology Driven Intelligent Context Aware Questionnaire

4.6 The Architecture of the Ontology Driven Intelligent Context Aware Questionnaire

4.7 Answers collated during Patient’s System Review

4.8 Patient Semantic Profile classes and visualisation in OWLVIZ In- terface

4.9 Object Properties list in Protg-

4.10 Data Properties in Patient Semantic Profile ontology

4.11 Patient Semantic Profile developed in Protege OWL

4.12 OWLVIZ classes view of the Recommendation Ontology

4.13 Clinical Rules for Lab Tests Recommendation

4.14 List of Suggested Lab Tests

4.15 Clinical Rules for Medication Prescription

4.16 Clinical Rules Execution Life Cycle

4.17 Patient’s basic details representation as a fact using the patient fact template

4.18 Patient symptoms and signs representation as facts

4.19 Flow Chart Diagram for Review of the System Procedure

4.20 Flow chart diagram represents patient flow within the Recommen- dation system

4.21 Rules for the first two steps to control Patient Flow

4.22 Screenshot showing the first two steps in patient flow

4.23 Screenshot showing a visit-doctor halt

4.24 Cardiac Risk Assessment Mechanism provided by the Clinical De- cision Support Framework

4.25 Use Case for the Patient and Clinicians

4.26 Patients’ Interface

4.27 Doctor’s interface

4.28 Integration of ODCRARS and MLDPS

5.1 Data Acquisition Stages - Raigmore Hospital’s RACPC Databases

5.2 Graphical output of weighted classification accuracies using differ- ent setups

5.3 Confusion Matrix for a binary classification problem

5.4 ROC curves for different Experimental Setups

5.5 ROCs using different experimental setups, SFFS feature selection is also compared

5.6 ROCs using different experimental setups, mRMR feature selection is added

5.7 Upper figure: Multi-colour graph represents 5 randomly selected datasets in which 4 datasets were used for training and 1 for testing (for each M). Lower figure: Experimental results showing average accuracies of different number of mixture density models

5.8 Upper figure: Multi-colour graph represents accuracies obtained using 5 randomly selected datasets in which 4 datasets were used for training and 1 for testing for each different type of kernel func- tion. Lower figure: Experimental results showing average accura- cies of different types of kernel functions including: 1- Linear, 2- Polynomial, 3- Radial Basis Function and 4- Sigmoid Function

5.9 ROC curves of various experimental setups utilised in the study group 1 for comparison purpose

5.10 ROCs for various experimental setups utilised in Test Results (study group 2) for comparison purpose

5.11 Cardiac Chest Pain Prognostic Model’s front end

5.12 Output example of the Cardiac Chest Pain Prognostic Model.

5.13 Output example of the Cardiac Chest Pain Prognostic Model.

5.14 Output example of the Cardiac Chest Pain Prognostic Model.

5.15 ROC curves of the best classification setups for comparison purpose

5.16 Machine Learning Driven Heart Disease Prognostic Model’s front end, is available at http://www.cs.stir.ac.uk/ kfa/HDP/hd3/hd3.html

5.17 Output example of the Machine Learning driven Heart Disease Prognostic Model

5.18 Machine Learning Driven Heart Disease Prognostic Model’s front end, is available at http://www.cs.stir.ac.uk/ kfa/HD1/hd1/hd1.html

5.19 Output example of the Cardiac Chest Pain Prognostic Model, is available at http://www.cs.stir.ac.uk/ kfa/HDP/hd2/hd2.html

5.20 ROC curves of the best classification setups for comparison with the expert driven LR experimental setup

5.21 The machine learning driven Breast Cancer Prognsotic Model’s front end, is available at http://www.cs.stir.ac.uk/ kfa/bc/bc1.html

5.22 Clinical use case for the validation of ontology driven clinical risk assessment and recommendation system

5.23 Clinical use case for the validation of Ontology Driven Clinical Risk Assessment and Recommendation System

5.24 Clinical validation of the Ontology Driven Clinical Risk Assess- ment and Recommendation system (ODCRARS)

5.25 Cardiac Chest Pain Risk Score Calculation as part of the Inte- grated ODCRARS

6.1 The Architecture of Sentic Avatar proposed by Cambria et al.

6.2 Representation of an FCM Model as in⁷

A.1 Consultant Cardiologist, Professor Stephen Leslie’s Feedback on RACPC Clinical Prototypes

A.2 Clinical validation report issued by General Medical Practitioner from a GP practice in Edinburgh, Scotland

A.3 Clinical validation report issued by a cardiac thoracic surgeon from Kings College Hospital in London

A.4 Clinical assessment by clinical informatics expert, Professor Warner Slack from Harvard Medical School, US

A.5 Clinical validation report issued by the oncologist from The Beat- son, Cancer Centre, West of Scotland, UK

C.1 Comparison ROCs

C.2 Comparative ROCs after applying various classification techniques

C.3 Comparative ROCs Decision Trees

List of Tables

2.1 The clinical impact of a combination of risk factors on CVD test

3.1 Different types of Coding Schemes for Categorical Variables, adapted from ”Multiple Regression (MR) Using Categorical Variables in MR” tutorial

3.2 Confusion matrix for two-class classification problem

4.1 Questionnaire Types for the Review of the System

4.2 Prediction Equation Coefficients

4.3 Global Risk Score Calculation

5.1 Clinical Variables Selected for the RACPC Clinical Case Study

5.2 Weighted classification Accuracies with common clinical variables (highlighted in bold) in each iteration

5.3 Classification results in terms of several evaluations

5.4 Confusion Matrix of Logistic Regression (LR) based Experimental Setups

5.5 Confusion Matrix of Decision Tree (DT) based Experimental Setups

5.6 Confusion Matrix of Support Vector Machine (SVM) based Exper- imental Setups

5.7 P-values of the candidate clinical variables

5.8 Experimental Setups based on machine learning classifiers and fea- ture selection techniques

5.9 Anova Summary Table - RACPC Classifiers Performance Measure- ment

5.10 Anova Test Results shows F static value, P-value and F critical value

5.11 RACPC Features List after further Pre-Processing of Smoking free text Description

5.12 Final Diagnoses

5.13 Clinical Risk Factors and Test Results in two study groups

5.14 Study group 1 (Risk Factors)- Feature Selection

5.15 The confusion matrix of LR and feature selection based classifica- tion setups, study group

5.16 Experiment results in terms of different evaluation measurements

5.17 Confusion Matrix of DT and feature selection based classification setups, study group

5.18 Confusion Matrix of SVM and feature selection based classification setups, study group

5.19 One-way ANOVA Test for the performance evaluation of LR, DT and SVM based classification setups

5.20 P-values of the clinical variables (study group 2)

5.21 Feature Selection results, Study group 2 (Test Results)

5.22 Experiment results in terms of different evaluation measurements

5.23 Confusion matrix obtained using LR based classification setups

5.24 Confusion matrix obtained using DT based classification setups

5.25 Confusion matrix obtained using SVM based classification setups

5.26 One-way ANOVA Test for the performance evaluation of LR, DT and SVM based classification setups (Study group 2- Test Results)

5.27 Classification setups considered for the development of machine learning driven cardiac chest pain prognostic model

5.28 Clinical Variables extracted from the UCI heart disease dataset

5.29 Final list of clinical variables after the effects coding scheme

5.30 P-values of the clinical variables selected in the heart disease clin- ical case study

5.31 Experimental setups based on the machine learning classification and feature selection methods

5.32 The confusion matrix of LR based classification setups

5.33 The confusion matrix of DT based classification setups

5.34 The confusion matrix of SVM based classification setups

5.35 Experiment results in terms of different evaluation measurements

5.36 Performance Analysis of different classification techniques

5.37 ANOVA Test Results

5.38 P-values of the clinical variables used in the breast cancer clinical case study

5.39 Experimental Setups including feature selection results

5.40 The confusion matrix of different experimental setups based on Logistic Regression and Feature Selection Methods

5.41 The confusion matrix of different experimental setups based on Decision Tree and Feature Selection Methods

5.42 The confusion matrix of different experimental setups based on Support Vector Machine and Feature Selection Methods

5.43 Experiment results in terms of different evaluation measurements

5.44 Performance Analysis of different classification techniques using One-Way ANOVA

5.45 ANOVA Test Results

B.1 Risk Factors and two Classes (Weighted)

B.2 Test Results and Two Classes (Weighted)

C.1 Logistic Regression - Performance Vector

C.2 Performance Vector kNN

C.3 Random Forests Decision Trees

C.4 Performance Vector Random Forest

DECLARATION

I understand the nature of plagiarism, and I am aware of the University’s policy on this. I certify that this dissertation reports original work by me during my University project. I confirm that this thesis has not been previously submitted for the award of a degree by this or any other university

illustration not visible in this excerpt

ACKNOWLEDGEMENTS

This thesis would not have been possible without the help and support of a large number of individuals. First and foremost, I would like to thank my family members, especially my beloved parents, my wife and my lovely daughter, Safa and who have endured my absence during my research and helped me tremendously in all ways possible. Without their continued help, support and guidance, this would never have been possible. Thank you.

My heartfelt thanks to my principal supervisor, Professor Amir Hussain, for his generous offer of the PhD position so that I can fulfil my dream of doctoral study. Thank you for your support and guidance throughout this research: for constantly guiding me toward exploration in the right directions; for questioning me about unclear key thoughts; and for shaping my ambiguous concepts by inter- preting the research from different perspectives. I am thankful to the Engineering and Physical Sciences Research Council (EPSRC Grant Ref. no. EP/H501584/1) and Sitekit Solutions for funding my PhD.I would further like to thank Dr David Cairns and Professor Evan Magill for their support and encouragement.

I would like to thank my industrial supervisor Chris Eckl and Campbell Grant, CEO of Sitekit Solutions for providing me this excellent research opportunity to xxiii work closely with researchers at the Sitekit Lab and for trusting my abilities to move the project forward and for his invaluable insights. I am deeply indebted to Professor Stephen Leslie, consultant cardiologist from Raigmore Hospital in Scotland for providing me the required domain expertise as well as facilitating me to utilise the RACPC patient’s data for this thesis. I would like to thank Professor Calum MacRae from Brigham and Women’s hospital for his continuing support and guidance and for acting as my domain clinical expert. My heartfelt thanks to Professor Warner Slack, Hollis Kowaloff, Charles Safran and Henry Feldman from Beth Israel Deaconesses Medical Centre, Harvard Medical School for providing guidance and encouragement to me every step of the way.

I am also thankful to Professor Cheng Lin Liu and Professor Chengqing Zhong from the Chinese Academy of Sciences in Beijing; Professor Bin Luo and Professor Jin Tang from the Anhui University in China for trusting me with visiting research fellow opportunities to carry out work on UK-China joint research projects. I am also thankful to John Moore from MIT New Media Medicine lab for his invaluable input and feedback on clinical questionnaires and clinical prototypes that I have developed during this PhD. I am very grateful to Hicham Atassi and Jan Karasek for offering me a visiting research fellow opportunity to exchange technical expertise with researchers at the Brno University of Technology in Czech Republic. I am very thankful to RACPC clinicians in Raigmore hospital for their kind support and timely advice.

Lastly, I would like to thank my colleagues and friends from the COSIPRA Lab with whom I had the opportunity to discuss areas of mutual interests : Muaz Niazi, Wajeeha Aneel, David Vidal, Peipei, Aihua, Amjad Ullah, Zeeshan Malik, Thomas Mazzocco, Erik Cambria, Erfu Yang and Zhengzheng Tu. I would like to thank Alexander Saunders from University of Aberdeen. I also would like to thank Grace McArthur, Lynn Reilly, Linda Bradley and Gemma Gardiner for providing support throughout my PhD.

GLOSSARY OF ABBREVIATIONS

illustration not visible in this excerpt

Chapter 1 INTRODUCTION

Clinical data is the foundation of health learning, with the aim of creating effective clinical solutions for healthcare providers all over the world⁸. Issues motivat- ing discussion include the potential for clinical data as a resource for continuous learning. A key component of an efficient healthcare system revolves around the key area of data transformation through interoperable data resources and creates awareness among clinical domain and informatics experts regarding these issues. Healthcare organisations have been collecting and storing large amounts of data for decades. Most of this invaluable legacy patient data resides in distributed hos- pital repositories, which are often ignored or badly utilised for learning purposes that aim to improve clinical pathways, and are difficult to access and pre-process (data interoperability, disparate coding standards like SNOMED CT, HL7 and missing data issues) for a meaningful purpose by healthcare solution providers.

With the advent of “Big Data”, predictive clinical analytics is now one of the most researched areas of academic and commercial partners globally and has an aim to develop cost effective healthcare solutions to promote evidence-based/data driven preventative care. Clinical predictive analytics has the potential to trans- form the way healthcare solution providers develop clinical decision support tech- nologies using synthetic data. Healthcare solution providers can develop more cost effective and efficient prospective and preventative care solutions by way of learning from the legacy data stored in clinical data repositories. Thus, they can make more informed decisions and improve data-driven/evidence-based patient care⁹. The onus is on healthcare organisations at a national level to enable domain experts, clinicians, researchers and healthcare trusts to unlock the true potential of the legacy data stored within their proprietary healthcare systems.

Big data is transforming the discussion of what is appropriate for a patient and for the healthcare ecosystem. The release of big data has helped authorities to develop patient-centric healthcare models by considering a holistic view of care. New care models have been proposed, which are built on 5 key pathways, as presented by Groves et al¹⁰ ; details of these key pathways are as follows:

1. Right Living: Patients can be made custodians of their well-being by getting them involved in the decision-making process, the prescription of treatment plans and decision prevention schemes. The right living pathway focuses on encouraging patients to make lifestyle choices such as lowering their Body Mass Index (BMI), dieting and engaging in exercise.
2. Right Care: This pathway entails ensuring that patients get the most timely, appropriate care when needed. It also specifies a need for a coordinated
approach to be followed across different healthcare providers and aims to share the same clinical data amongst its stakeholders to avoid duplication while fostering effort and promoting suboptimal strategies.
3. Right Provider: This pathway proposes that patients should always be treated by professionals who are best suited to the task and can deliver the best outcome. This clinical pathway also reiterates that healthcare providers be selected as per their track record[10].
4. Right Value: This pathway involves multiple measures that can be intro- duced to ensure the cost effectiveness of care by eliminating redundant clinical workflows in healthcare systems.
5. Right Innovation: This pathway involves promoting research and devel- opment activities in the healthcare sector so legacy clinical data could be utilised to learn from existing clinical systems and improve clinical trials and treatment protocols[10].

Big data predictive clinical analytics paves the way for the development of next generation healthcare learning systems and promote personalised care for patients. Healthcare learning systems are built on the core principle of learning from existing clinical practices through legacy clinical data, as well as utilising existing clinical practice guidelines to facilitate efficient clinical decision-making operations. A learning activity in these intelligent healthcare systems can be de- scribed as an activity which focuses on the delivery of the healthcare operations or uses personalised health information (derived from legacy clinical data repos- itories) and has a targeted objective of learning from existing clinical work flows to improve clinical practice guidelines. This with a view to improving the quality, efficiency of the systems, institutions and modalities through which healthcare services are provided by healthcare providers. All of the aforementioned activities are deemed as learning activities which are enshrined in the next generation of healthcare learning systems. These systems can benefit from conventional clinical research, comparative effectiveness research, quality improvement research, qual- ity improvement and patient safety practices, healthcare operations, quality as- surance or evidence-based personalised care. All of these operations/components are the building blocks for the next generation of healthcare systems based on learning activities¹¹.

Legacy clinical data combined with clinical practice guidelines is a data sci- ence methodology that can identify patterns in home monitoring physiologic data. Coupled with interaction with the patient and their caregivers, we can give the care team early warning of a worsening of the patient’s clinical status. In the UK, NICE (National Institute of Clinical Excellence) states that all clinical domains can be used as a means of evidence. These guidelines are defined as systematically developed rules to assist clinicians in clinical decision-making about appropriate health care for specific clinical circumstances. These guidelines are based on the most rigorous research available, and are often referred to as best practice guidelines. Applied at the individual patient level, these guidelines provide a set of corrective actions based on conditional logic for solving problems or accom- plishing tasks. Appropriately applied, the guidelines can reduce the uncertainties associated with clinical decisions, diminish the variation around usual practices, demystify unfamiliar terminology and decrease the need to search for journals and articles¹². It is therefore vital to make use of these guidelines combined with clinical data if we are to build efficient and personalised care models. Predictive Clinical Analytics based on learning retrospective clinical data focuses on patients with complex chronic diseases and aims to improve health, reduce avoidable hospitalisations and acute care events and, as a result of the decreased need for expense acute care, also reduce costs. Predictive Analytics has the po- tential to help physicians make better decisions across the board and help to deliver evidence-based personalised care and treatments as part of a preventative care solution; hence increasing efficiency, thereby reducing the burden on primary and secondary care.

The aim of this interdisciplinary research project is to develop a hybrid clinical decision support framework for cardiovascular preventative care. Our proposed ontology and machine learning-driven hybrid clinical decision support framework builds on Bouamrane et al.’s clinical decision support framework[2] by provid- ing an advanced ontology driven clinical decision support and machine learning driven prognostic modelling capabilities. The proposed ontology and machine learning driven hybrid clinical decision support framework comprises of Ontology Driven Clinical Risk Assessment and Recommendation system (ODCRARS) and the Machine Learning Driven Prognostic System (MLDPS) to provide a cardio- vascular preventative solution.

The ODCRARS provides intelligent context aware information collection for gathering a patient’s medical history. This is then transformed into a semantic profile (to alleviate interoperability issues) by using answers provided in patient interviews. The patient semantic profile combined with a recommendation on- tology is utilised for the recommendation of lab tests and medications for car- diovascular patients. A clinical rules engine is developed to provide cardiac risk assessment tools to carry out cardiac risk scores calculation for various cardio- vascular diseases.

The proposed clinical decision support framework also incorporates a ma- chine learning-driven prognostic system. The machine learning-driven prognostic system is validated in the cardiovascular and breast cancer domains and online prognostic models have also been developed and deployed online for further clin- ical trials and validation. The proposed ontology and machine learning-driven hybrid clinical decision support framework provides a learning mechanism built using machine learning techniques. The learning facility is provided through the exchange of patient data amongst the MLDPS and ODCRARS.

The MLDPS and ODCRARS are integrated in order to provide a cardiovascu- lar preventative care solution for patients and clinicians in primary and secondary care using dedicated interfaces. The machine learning driven cardiac chest pain and heart disease risk scores calculation is provided in the integrated system along with other cardiac risk scores to facilitate clinicians in the clinical decision making process.

1.1 Organisation of Thesis

This thesis is organised as follows. Chapter 2 provides a literature review of the existing clinical decision support systems.

Chapter 3 presents the proposed Novel Ontology and Machine Learning Driven Hybrid Clinical Decision Support Framework for cardiovascular preventative care and its two key components: (1) ODCRARS and (2) MLDPS. Chapter 4 explains the development of the ODCRARS for preventative cardiovascular care. Details of design, the development and validation of ontology-driven intelligent context aware information collection, patient semantic profiles, a clinical rules engine (for lab tests and medication prescriptions), the cardiac risk assessment tools and the integration of a machine learning cardiac chest pain prognostic model including cardiac chest pain risk score calculation are explained. Chapter 5 introduces a MLDPS for cardiovascular preventative care. It describes key development stages (keeping in line with the prognostic model development process as described in chapter 3), while a clinical case study for RACPC patients is discussed in de- tail along with the development and clinical validation of a cardiac chest pain prognostic model. Utilisation of additional two datasets in the heart disease and breast cancer domains, for validation purposes, along with development of breast cancer and heart disease prognostic models are discussed at the end. Chapter 6 presents an analysis of the work and discusses the future directions of research.

1.2 Motivation and aims

Conventional healthcare information management systems suffer from a general lack of intelligence. They are successful in offering basic patient management capabilities to their end users but they do not offer substantial decision support functionalities or automation to lend a helping hand to clinicians. These sys- tems have been designed using branching logic-based rigid architectures, which are hard to maintain and upgrade without considerable labour intensive effort. Retrospective clinical data is often discarded by the machine learning experts while efficient feedback loops are not built into the decision support mechanism and do not support continuous learning and refining processes.

Clinical decision support systems in particular have been built with a signifi- cant amount of design weaknesses, which is why very few decision support oper- ations have been built into the core fabric of the clinical infrastructure governed by national and regional healthcare service authorities. Healthcare systems have a substantial amount of limitations, such as rigidity and nonconformity to com- plex clinical protocols like electronic healthcare records and effective utilisation of clinical practice guidelines, which can help to promote clinical standardisation.

Information collection systems provide episodic historic data to clinical de- cision support systems for inference purposes. Clinical patient assessment is currently being performed using clinical questionnaires (non-standard question- naires), which vary from one practice to another within the same healthcare region. In order for CDSSs to be fully successful in a problem domain like cardio- vascular disease, efforts are required to develop adaptive clinical questionnaires using standardised expert knowledge in order to promote better exploitation of these clinical systems. The success of these clinical decision support systems re- lies on its generated outcome, which is normally referred to as Electronic Patient Records or Electronic Healthcare Records. A clinical decision support system relies on each patient’s factual data along with clinical risk assessment guidelines as it aims to construe a clinical conclusion as part of the decision-making process.

This multidisciplinary industrial research project set out to develop a hybrid clinical decision support mechanism for cardiovascular preventative care, which could be utilised as a triage mechanism for patients undertaking primary and secondary care. The primary aim of this thesis is to provide a clinical decision support mechanism for cardiovascular patients by combining evidence, extrapo- lated through legacy patient data (based on AI-inspired techniques like ontology and machine learning-driven techniques) in order to facilitate cardiovascular pre- ventative care. As part of our research, clinical case studies in the RACPC, heart disease and breast cancer domains have been considered for the development and clinical validation of the machine learning prognostic system.

The proposed ontology and machine learning driven integrated system could be used as a triage system in the cardiovascular preventative care, which could help clinicians to prioritise patient appointments after reviewing snapshot of their medical history. This would be collected through ontology-driven intelligent con- text aware information collection using standardised clinical questionnaires. The results contain patient demographics information, cardiac risk scores, cardiac chest pain score, medication and recommended lab test details. We also aim to validate the proposed novel ontology and machine learning-driven hybrid clinical decision support framework in other application areas.

1.3 Original Contributions

1. Developed a novel ontology and machine learning driven hybrid clinical

decision support framework for cardiovascular preventative care under the close supervision of UK (Professor Stephen Leslie from Marmoreal Hospital) and US (Professor Calum MacRae and Professor Warner Slack from Har- vard Medical School) clinicians. The developed framework provides cardiac risk score calculation, lab tests and medication recommendation through the ontology driven clinical risk assessment and recommendation system (ODCRARS).

2. The MLDPS is validated using Raigmore Hospital’s RACPC. Two addi-

tional clinical case studies in the heart disease and breast cancer domains in collaboration with primary (General Medical Practitioner in the heart disease clinical case study) and secondary care (breast cancer oncologist in the breast cancer clinical case study) clinicians were undertaken for the development and clinical validation of the MLDPS. We highlight the prob- lem of learning from incomplete real patient from statistical perspective the likelihood-based approach to deal with imbalanced and missing data issues. There are multiple benefits of our approach: to complement existing SVM techniques to deal with missing data within a statistical framework, and to illustrate a set of challenging statistical machine learning algorithms, derived from the likelihood-based framework that handles clustering, clas- sification, and function approximation from missing/incomplete data in an intelligent and resourceful manner. New benchmark prognostic models have been developed using RACPC, Heart Disease and Breast Cancer datasets which have been validated through clinical domain experts in the UK and US.

3. A novel ODCRARS provides an ontology driven intelligent context aware

information collection built on a standardised questionnaire ontology for generating patient medical records.

4. The patient medical records are transformed semantically through patient

semantic profile ontology to give patient data an intrinsic meaning and also to alleviate interoperability issues.

5. A novel decision tree based adaptive questionnaire is proposed and utilised for the system development purposes.

6. Developed a generic ontology based on clinical questionnaires at the system level and demonstrated its utilisation in the cardiovascular preventative care solution. This ontology is developed based on generic classes which could be utilised in a variety of different clinical domains and it is particularly useful for providing metadata and structure of questionnaires elements at the database level.

1.4 Publications

The following papers have been published or accepted for publication during the course of this research and included additional work to the material presented in this thesis.

Refereed International Conference Proceedings

1. Kamran Farooq, Amir Hussain, Warner Slack and Bin Luo: An Ontol- ogy and Machine Learning Driven Hybrid Cardiovascular Decision Support Framework. IEEE SSCI, Cape Town, December 2015, In Preparation.
2. Kamran Farooq, Jan Karasek, Hicham Atassi, Amir Hussain, Peipei Yang, Calum MacRae, Chris Eckl, Warner Slack and Bin Luo: A Novel Cardiovas- cular Decision Support Framework for Effective Clinical Risk Assessment. IEEE SSCI, Orlando 2014: 14925.
3. Kamran Farooq, Peipei Yang, Amir Hussain, Kaizhu Huang, Chris Eckl, Calum MacRae, Warner Slack: Efficient Clinical Decision Making by learn- ing from missing Clinical Data. IEEE SSCI, Singapore 2013: p1024. (Nom- inated for the best paper award).
4. Kamran Farooq, Amir Hussain, Stephen Leslie, Chris Eckl, Warner Slack: Ontology-driven cardiovascular decision support system. Pervasive Health 2011: 283-286.

Peer Reviewed Book Chapters

1. Kamran Farooq, Amir Hussain, Hicham Atassi, Stephen Leslie, Chris Eckl, Calum MacRae, Warner Slack- A Novel Clinical Expert System for Chest Pain Risk Assessment. BICS, Beijing, June 2013.
2. Kamran Farooq, Amir Hussain, Stephen Leslie, Chris Eckl, Calum MacRae, Warner Slack: An Ontology Driven and Bayesian Network Based Cardio- vascular Decision Support Framework. BICS 2012: 31-41
3. Kamran Farooq, Amir Hussain, Stephen Leslie, Chris Eckl, Calum MacRae, Warner Slack: Semantically Inspired Electronic Healthcare Records. BICS 2012: 42-51.

Peer Reviewed Journal Papers

1. Kamran Farooq, Amir Hussain, Warner Slack A Novel Ontology and Ma- chine Learning Driven Hybrid Clinical Decision Support Framework for Cardiovascular Preventative Care, BioMed Medical Informatics and Deci- sion Making journal, impact factor 1.5, Conditionally Accepted April 2015.
2. Kamran Farooq, Amir Hussain, Warner Slack, A Machine Learning Driven Prognostic System for Holistic Clinical Prognosis for Cardiovascular Pa- tients: Elsevier Expert Systems with Applications, Under Review 2015.
3. Kamran Farooq, Amir Hussain, Warner Slack, Efficient Cardiovascular Prognosis by Learning from Missing Clinical Data : Elsevier Artificial In-
telligence in Medicine, Under Review 2015.
4. Kamran Farooq, Amir Hussain, Warner Slack, A Novel Machine Learning Driven Prognostic System for Breast Cancer Preventative Care: Elsevier Computers in Biology and Medicine, Under Review 2015.
5. Kamran Farooq, Amir Hussain, Warner Slack, A Novel Ontology and Ma- chine Learning Driven Hybrid Clinical Decision Support Framework for Cardiovascular Preventative Care: Elsevier Computer Methods and Pro- grams in Biomedicine, Under Review 2015.
6. Kamran Farooq, Muaz Niazi, Stephen Leslie, Amir Hussain, Warner Slack, A Scientometric Review of Clinical Decision Support Systems, Springer Scientometrics Journal, In Preparation.

Posters and Clinical Prototypes Demonstration

1. Demonstration of RACPC and Heart Disease Risk Assessment prototypes

(developed as part of my PhD) at the 3rd SICSA Workshop on Technology for Health and Well Being (THAW), 20 June 2014, held at the University of Strathclyde, Scotland, UK.
2. Poster Presentation along with the demonstration of Cardiovascular Risk Predictors (developed as part of my PhD) at the 2nd SICSA Workshop on Technology for Health and Well Being (THAW), 20 March 2014, held at the Glasgow Caledonian University, Scotland, UK.
3. Poster presentation at the SICSA Cognitive Computation Summer School, University of Stirling, 25-30 Aug 2013- Presented poster title: A Novel Cardiovascular Decision Support Framework for Effective Clinical Risk As- sessment.
4. Poster presentation at the 5th China-Scotland SIPRA Workshop on the Next Generation Intelligent Signal Image Processing Technologies and Applications, 15-19 April, 2013, poster title: A Novel Expert System for Chest Pain Risk Assessment.
5. Sitekit Labs Future of e-health symposium, Napier University, Edinburgh 17-18 May 2012, poster title: An Ontology Driven Cardiovascular Decision Support Framework. Also demonstrated UPrevent’s context sensitive Elec- tronic Healthcare Records (EHR) building mechanism, which was developed as part of my PhD (cardiovascular preventative care prototype).
6. Poster presentation at the Judge Business School, University of Cambridge 2011, title: Cardiovascular Decision Support Framework - A Preventative Care Enterprise Solution.
7. Poster presentation at the Sitekit Labs, Highland Games Conference, Isle of Skye, Inverness, Sept 2010, poster title: Clinical Expert System for Car- diovascular Risk Assessment.

Invited Talks

1. The Care Technologies Group at the Department of Computing Science and Mathematics, University of Stirling, March 2011, title: Effective Cardiovas- cular Risk Assessment using Ontology driven Decision Support Framework.
2. Sitekit Labs Future of e-health symposium, Napier University, Edinburgh, 17-18 May 2012, title: Next generation Clinical Decision Support Frame- work for Cardiovascular Patients.
3. The Computational Intelligence Group, University of Stirling, seminar talk, March 2012, title: How to build Effective Prospective Clinical Decision Support Systems.
4. The Cognitive Signal Image and Control Processing Research (COSIPRA) Lab Seminar, University of Stirling, Oct 2011, title: Learning from missing clinical data to build effective clinical decision support systems.
5. BICS conference 2012, Beijing, title: Semantically Inspired Electronic Health- care Records.
6. BICS 2012 conference, Beijing, Title: Ontology Driven and Bayesian Net- work Inspired Cardiovascular Decision Support Framework.
7. Chinese Academy of Sciences, Beijing, Feb 2012, seminar talk, tile: An AI Inspired Clinical Decision Support Framework.
8. 2013 IEEE Symposium on Computational Intelligence in healthcare and e- health (IEEE CICARE 2013), title: Efficient Clinical Decision Making by Learning from Missing Clinical Data. (Nominated for the best publication).
9. Anhui University, Hefei, China, June 2013, seminar talk, title: Towards Learning from Retrospective Legacy Data for making Effective Prospective Clinical Decision Support Systems.
10. COSIPRA Lab, University of Stirling, seminar talk, July 2013, title: Fea- ture Selection and Machine Learning based Classification Techniques for Chest Pain Patients.
11. FP7 funded Signal Processing workshop organized at the Brno University of Technology, October 2013, on Efficient clinical risk assessment of cardiovas- cular patients using an Ontology driven and Machine Learning Approach.
12. IEEE SSCI 2014, Orlando, USA, Session Chair CICARE 2014, title, ”A Novel Cardiovascular Decision Support Framework for Effective Clinical Risk Assessment Kamran Farooq, Jan Karasek, Hicham Atassi, Amir Hus- sain, Peipei Yang, Calum MacRae, Chris Eckl, Warner Slack, Bin Luo and Mufti Mahmud.

Chapter 2 LITERATURE REVIEW

This chapter covers general background material for the thesis and provides com- prehensive reviews of related topics that are investigated in the thesis. In the beginning, an overview of clinical decision support systems and their benefits, followed by utilisation of different techniques in the cardiovascular clinical deci- sion support solutions based on different techniques. In the latter part, a concise review of relevant clinical decision support systems used in this thesis is explained.

2.1 Clinical Decision Support Systems

Since the advent of computers, healthcare professionals have anticipated the time when machines would assist them in clinical decision making and other restorative procedures. The very first articles dealing with this provision appeared in the late 1950s¹³ and experimental prototypes were made available within a few years[13]. Three advisory systems from the 1970s provide a useful overview of the origin of work on clinical decision-support systems: deDombals system for diagnosis of abdominal pain [¹⁴, 15], Shortliffes MYCIN system for selection of antibiotic therapy¹⁵ and the HELP system for delivery of inpatient medical alerts [¹⁶, ¹⁷].

The adoption of clinical decision support systems (CDSSs) in the diagno- sis and administration of major chronic diseases e.g. Dementia¹⁸, cancer ¹⁹,diabetes²⁰, hypertension²¹ and heart disease²² have made significant contributions in improving the clinical outcomes at primary and secondary care healthcare organisations all over the world. CDSS have also made it possible for system developers and knowledge engineers to collate and construct domain expert knowledge for the purpose of clinical risk assessment and screening by clinicians [²³, ²⁴].

Many reviews have identified the benefits of CDSS, in particular CPOE (com- puterised physician order entry) systems²⁵ [26, 27].CDSS as part of CPOE have been found to alleviate medication errors and adverse drug events [28, 29, 30]. Clinical decision support systems also have demonstrated to improve clinician performance, by way of promoting electronic prescription of drugs, adherence to guidelines and to an extent efficient use of time [30, 29]. CDSSs perform a key role in providing preventative measures at outpatient clinics and primary care, for example by alerting care givers of the need for routine blood pressure checking, to offer influenza vaccination and to recommend cervical screening²⁶ and[31].

The key benefits of CDSS reported in the studies conducted in [24, 32, 33, 34] and[1] are as follows:

1. Higher Standards of Patient Safety

Clinical decision support systems have helped healthcare organizations all over the world acquiring higher standards of patient safety.They adhere to standardized clinical procedures governed by the clinical workflows thus reducing diagnostic, prescribing errors and drug doubling issues.

2. Improving quality of direct patient care

Furthermore, authors concluded that with the advent of CDSS, quality of care has improved considerably levels with this extra support provided to clinicians (who are already struggling to cope with current healthcare demands). This has made it possible for clinical experts to allocate more time to providing direct patient care.

3. Standardization and Conformance of Care using Clinical Practice Guide- lines

The standardisation of clinical pathways and procedures set precedents and evaluation benchmarks for healthcare trusts to achieve higher patient sat- isfaction levels set out by different healthcare organizations in different re- gions. CDSSs also promote the utilisation of clinical practice guidelines (CPGs) for the development of knowledge-aware systems capable of per- forming effective clinical decision making to promote standardised care.

4. Collaborative Decision Making

CDSSs have helped healthcare stakeholders that include clinicians, health- care trusts and policy makers to develop safe and efficient care models using collaborative decision making approach to benefit both patient and a clinician. CDSS have also helped healthcare trusts to Improve effectiveness in prescribing facility through cost effective drugs order dispensation[24]. CDSS are also playing an important role in the integration of EHRs (Electronic healthcare records) which will help healthcare authorities to streamline information collection and clinical diagnosis operations in order to promote efficient data gathering[34]. Audit trail is another important aspect of modern healthcare systems which is achieved through the intelligent exploitation of clinical decision support capabilities.

Clinical decision support systems are being extensively deployed in health- care settings all over the world. Modern clinical decision support systems are increasingly dissimilar to each other, despite following the same generic architec- ture which defines a typical CDSS[35]. These clinical decision support systems incorporate a variety of innovative techniques to perform various key operations which include clinical knowledge dissemination and collecting patient’s medical history for effective clinical decision making. These systems aim to provide clinical decision support and automatic personalised clinical advice through inference ca- pabilities[36].They also help to streamline clinical workflows through integration with electronic healthcare records for patient clinical history collection, diagnosis, inference and training.

Clinical decision support operations are an integral part of modern healthcare management systems. They assist clinicians, patients and healthcare stakehold- ers by providing expert clinical knowledge and patient-centric information[37].

The information provided by these intelligent clinical systems is used for clinical decision making in order to improve the effectiveness and quality of healthcare. Automated cardiovascular decision support systems are now being deployed in hospitals and primary care organizations in order to meet the ever growing clinical needs of prognosis in the areas of cardiovascular disease and coronary heart dis- ease. Computerized decision support strategies have already been implemented successfully in several areas of cardiovascular care[38]. These applications are being used as part of the extension of clinical informatics infrastructure in the UK and US. These systems are also being used in both primary and secondary care settings for providing efficient healthcare delivery to its patients. In order to capitalise on the benefits provided by cardiovascular decision support systems, a strong foundation in evidence-based medicine and well-established clinical prac- tice guidelines (CPGs) have to be considered to ensure clinical governance in the next generation clinical systems. An alternate approach to computer-assisted decision support was provided in the MYCIN development program, a clinical consultation system that de-emphasized diagnosis to concentrate on appropriate management of patients who have infections[39]. Knowledge of infectious dis- eases in MYCIN was represented as production rules, each containing a packet of knowledge derived from discussions with collaborating experts (2.1). The MYCIN program determined which rules to use and how to chain them together to make decisions about a specific case.

In MYCIN, rules are conditional statements that indicate what course of action to be taken if a specified condition is set to True. A team of clinical

illustration not visible in this excerpt

Figure 2.1: Example of a Rule encoded in MYCIN.[1]

experts evaluated MYCINs performance on therapy selection for patients with blood-borne bacterial infections[40] and for those with meningitis[40]. MYCIN, however, is best known as a system based on early exploration of methods for capturing and applying ill-structured expert knowledge to solve important medical problems. Although the program was never used clinically, it paved the way for a great deal of research and development in the 1980s[41].

[...]

---

¹ M. A. Musen, B. Middleton, and R. A. Greenes, “Clinical decision-support systems,” in Biomedical informatics, pp. 643–674, Springer, 2014.

² M.-M. Bouamrane, A. Rector, and M. Hurrell, “A hybrid architecture for a preoperative decision support system using a rule engine and a reasoner on a clinical ontology,” in Web Reasoning and Rule Systems, pp. 242–253, Springer, 2009.

³ S. R. Abidi, S. Hussain, M. Shepherd, and S. S. R. Abidi, “Ontology-based modeling of clinical practice guidelines: a clinical decision support system for breast cancer follow-up interventions at primary care settings,” in Medinfo 2007: Proceedings of the 12th World Congress on Health (Medical) Informatics; Building Sustainable Health Systems, p. 845, IOS Press, 2007.

⁴ A. Khan, J. Doucette, and R. Cohen, “A framework for practical medical decision support systems using structured knowledge & machine learning,” 2012.

⁵ L. C. Santelices, Y. Wang, D. Severyn, M. J. Druzdzel, R. L. Kormos, and J. F. Antaki, “Development of a hybrid decision support model for optimal ventricular assist device weaning,” The Annals of thoracic surgery, vol. 90, no. 3, pp. 713–720, 2010.

⁶ J. Bowie and G. O. Barnett, “Mumpsan economical and efficient timesharing system for information management,” Computer programs in biomedicine, vol. 6, no. 1, pp. 11–22, 1976.

⁷ N. Douali, E. I. Papageorgiou, J. De Roo, H. Cools, and M.-C. Jaulent, “Clinical decision support system based on fuzzy cognitive maps,” Journal of Computer Science & Systems Biology, vol. 8, no. 2, p. 112, 2015.

⁸ M. McGinnis, L. Olsen, A. W. Goodby, et al., Clinical Data as the Basic Staple of Health Learning:: Creating and Protecting a Public Good: Workshop Summary. National Academies Press, 2010.

⁹ W. Raghupathi and V. Raghupathi, “Big data analytics in healthcare: promise and potential,” Health Information Science and Systems, vol. 2, no. 1, p. 3, 2014.

¹⁰ P. Groves, B. Kayyali, D. Knott, and S. Van Kuiken, “The big datarevolution in healthcare,” McKinsey Quarterly, 2013.

¹¹ R. R. Faden, N. E. Kass, S. N. Goodman, P. Pronovost, S. Tunis, and T. L. Beauchamp, “An ethics framework for a learning health care system: a departure from traditional research ethics and clinical ethics,” Hastings Center Report, vol. 43, no. s1, pp. S16–S27, 2013.

¹² G. LoBiondo-Wood and J. Haber, Nursing research: Methods and critical appraisal for evidence-based practice. Elsevier Health Sciences, 2013.

¹³ R. Ledley and L. B. Lusted, “The use of electronic computers to aid in medical diagnosis,” Proceedings of the IRE, vol. 47, no. 11, pp. 1970–1977, 1959.

¹⁴ C. A. Nugent, H. R. Warner, J. T. Dunn, and F. H. Tyler, “Probability theory in the diagnosis of cushing’s syndrome,” The Journal of Clinical Endocrinology & Metabolism, vol. 24, no. 7, pp. 621–627, 1964.

¹⁵ W. J. Clancey, E. H. Shortliffe, and B. G. Buchanan, “Intelligent computeraided instruction for medical diagnosis,” in Proceedings of the Annual Symposium on Computer Application in Medical Care, p. 175, American Medical Informatics Association, 1979.

¹⁶ G. J. Kuperman, R. M. Gardner, and T. A. Pryor, “The pharmacy application of the help system,” in HELP: A Dynamic Hospital Information System, pp. 168–172, Springer, 1991.

¹⁷ H. R. Warner, Computer–Assisted Medical Decision-Making. Academic Press, Inc., 1979.

¹⁸ H. Lindgren, “Integrating clinical decision support system development into a development process of clinical practice–experiences from dementia care,” in Artificial Intelligence in Medicine, pp. 129–138, Springer, 2011.

¹⁹ S. B. Clauser, E. H. Wagner, E. J. Aiello Bowles, L. Tuzzio, and S. M. Greene, “Improving modern cancer care through information technology,” American journal of preventive medicine, vol. 40, no. 5, pp. S198–S207, 2011.

²⁰ P. J. OConnor, J. M. Sperl-Hillen, W. A. Rush, P. E. Johnson, G. H. Amundson, S. E. Asche, H. L. Ekstrom, and T. P. Gilmer, “Impact of electronic health record clinical decision support on diabetes care: a randomized trial,” The Annals of Family Medicine, vol. 9, no. 1, pp. 12–21, 2011.

²¹ S. H. Luitjes, M. G. Wouters, A. Franx, H. C. Scheepers, V. M. Coup´e, H. Wollersheim, E. A. Steegers, M. P. Heringa, R. P. Hermens, and M. W. van Tulder, “Study protocol open access,” 2010.

²² R. F. DeBusk, N. Houston-Miller, and L. Raby, “Clinical validation of a decision support system for acute coronary syndromes,” Journal of the American College of Cardiology, vol. 55, no. 10, pp. A132–E1240, 2010.

²³ P. Khong and R. Ren, “Healthcare information system: building a cyber database for educated decision making,” International Journal of Modelling, Identification and Control, vol. 12, no. 1, pp. 133–140, 2011.

²⁴ A. Wright, D. F. Sittig, J. S. Ash, D. W. Bates, J. Feblowitz, G. Fraser, S. M. Maviglia, C. McMullen, W. P. Nichol, J. E. Pang, et al., “Governance for clinical decision support: case studies and recommended practices from leading institutions,” Journal of the American Medical Informatics Association, pp. jamia–2009, 2011.

²⁵ S. Eslami, N. F. d. Keizer, and A. Abu-Hanna, “The impact of computerized physician medication order entry in hospitalized patientsa systematic review,” International journal of medical informatics, vol. 77, no. 6, pp. 365–376, 2008.

²⁶ D. L. Hunt, R. B. Haynes, S. E. Hanna, and K. Smith, “Effects of computerbased clinical decision support systems on physician performance and patient outcomes: a systematic review,” Jama, vol. 280, no. 15, pp. 1339–1346, 1998.