Explosive amounts of biological and physiological data, including medical images, electroencephalograms, genomic information, and protein sequences, have been made available to us thanks to advances in biological and medical technologies. Understanding human health and disease is made easier by using this data for learning. Deep learning-based algorithms, which were developed from artificial neural networks, have significant potential for identifying patterns and extracting features from large amounts of complex data. However, these recent advancements involve blackbox models: algorithms that do not provide human-understandable explanations in support of their decisions. This limitation hampers the fairness, accountability and transparency of these models; the field of XAI tries to solve this problem providing human-understandable explanations for black-box models. This paper focuses on the requirement for XAI to be able to explain in detail the decisions made by an AI in a biomedical setting to the expert in the domain, e.g., the physician in the case of AI-based clinical decisions related to diagnosis, treatment, or prognosis of a disease. In this paper, we made use of the Indian Patient Liver Dataset (IPLD) collected from Andhra Pradesh region. The deep learning model with a 0.81 accuracy score (0.82 for the hyperparameter- tuned model) is built on Keras-Tensorflow and due to the imbalance in the target values, we integrated GANs as a means of oversampling the dataset. This study integrated the XAI concept of Shapley Values to shed light on the predictive results obtained by the liver disease detection model. © 2024 Slovene Society Informatika. All rights reserved.