Machine Learning and Knowledge Extraction

Large language models (LLMs) have recently made significant advances, excelling in tasks like question answering, summarization, and machine translation. However, their enormous size and hardware requirements make them less accessible to many in the machine learning community. To address this, techniques such as quantization, prefix tuning, weak supervision, low-rank adaptation, and prompting have been developed to customize these models for specific applications. While these methods have mainly improved text generation, their implications for the text classification task are not thoroughly studied. Our research intends to bridge this gap by investigating how variations like model size, pre-training objectives, quantization, low-rank adaptation, prompting, and various hyperparameters influence text classification tasks. Our overall conclusions show the following: 1—even with synthetic labels, fine-tuning works better than prompting techniques, and increasing model size does not always improve classification performance; 2—discriminatively trained models generally perform better than generatively pre-trained models; and 3—fine-tuning models at 16-bit precision works much better than using 8-bit or 4-bit models, but the performance drop from 8-bit to 4-bit is smaller than from 16-bit to 8-bit. In another scale of our study, we conducted experiments with different settings for low-rank adaptation (LoRA) and quantization, finding that increasing LoRA dropout negatively affects classification performance. We did not find a clear link between the LoRA attention dimension (rank) and performance, observing only small differences between standard LoRA and its variants like rank-stabilized LoRA and weight-decomposed LoRA. Additional observations to support model setup for classification tasks are presented in our analyses. Full article

Retailers depend on accurate sales forecasts to effectively plan operations and manage supply chains. These forecasts are needed across various levels of aggregation, making hierarchical forecasting methods essential for the retail industry. As competition intensifies, the use of promotions has become a widespread strategy, significantly impacting consumer purchasing behavior. This study seeks to improve forecast accuracy by incorporating promotional data into hierarchical forecasting models. Using a sales dataset from a major Portuguese retailer, base forecasts are generated for different hierarchical levels using ARIMA models and Multi-Layer Perceptron (MLP) neural networks. Reconciliation methods including bottom-up, top-down, and optimal reconciliation with OLS and WLS (struct) estimators are employed. The results show that MLPs outperform ARIMA models for forecast horizons longer than one day. While the addition of regressors enhances ARIMA’s accuracy, it does not yield similar improvements for MLP. MLPs present a compelling balance of simplicity and efficiency, outperforming ARIMA in flexibility while offering faster training times and lower computational demands compared to more complex deep learning models, making them highly suitable for practical retail forecasting applications. Full article

This study aims to explore the efficacy of Bayesian Neural Networks (BNNs) in enhancing predictive modeling for healthcare applications. Advancements in artificial intelligence have significantly improved predictive modeling capabilities, with BNNs offering a probabilistic framework that addresses the inherent uncertainty and variability in healthcare data. This study demonstrates the real-world applicability of BNNs through three key case studies: personalized diabetes treatment, early Alzheimer’s disease detection, and predictive modeling for HbA1c levels. By leveraging the Bayesian approach, these models provide not only enhanced predictive accuracy but also uncertainty quantification, a critical factor in clinical decision making. While the findings are promising, future research should focus on optimizing scalability and integration for real-world applications. This work lays a foundation for future studies, including the development of rating scales based on BNN predictions to improve clinical outcomes. Full article

Recent studies have shown that, due to redundancy, some heads of the Transformer model can be pruned without diminishing the efficiency of the model. In this paper, we propose a constrained optimization algorithm based on Hebbian learning, which trains specific layers in the Transformer architecture in order to enforce diversification between the different heads in the multi-head attention module. The diversification of the heads is achieved through a single-layer feed-forward neural network that is added to the Transformer architecture and is trained with the proposed algorithm. We utilize the algorithm in three different architectural variations of the baseline Transformer model. In addition to the diversification of the heads, the proposed methodology can be used to prune the heads that capture redundant information. Experiments on diverse NLP tasks, including machine translation, text summarization, question answering and large language modeling, show that our proposed approach consistently improves the performance of baseline Transformer models. Full article

In the context of hierarchical system modeling, ensuring constraints between different hierarchy levels are met, so, for instance, ensuring the aggregation constraints are satisfied, is essential. However, modelling and forecasting each element of the hierarchy independently introduce errors. To mitigate this balance error, it is recommended to employ an optimal data reconciliation technique with an emphasis on measurement and modeling errors. In this study, three different machine learning methods for development were investigated. The first method involves no data reconciliation, relying solely on machine learning models built independently at each hierarchical level. The second approach incorporates measurement errors by adjusting the measured data to satisfy each constraint, and the machine learning model is developed based on this dataset. The third method is based on directly fine-tuning the machine learning predictions based on the prediction errors of each model. The three methods were compared using three case studies with different complexities, namely mineral composition estimation with 9 elements, forecasting of retail sales with 14 elements, and waste deposition forecasting with more than 3000 elements. From the results of this study, the conclusion can be drawn that the third method performs the best, and reliable machine learning models can be developed. Full article

AI Alignment is a term used to summarize the aim of making artificial intelligence (AI) systems behave in line with human intentions and values. There has been little consideration in previous AI Alignment studies of the need for AI Alignment to be adaptive in order to contribute to the survival of human organizations in changing environments. This research gap is addressed here by defining human intentions and values in terms of survival biophysics: entropy, complexity, and adaptive behavior. Furthermore, although technology alignment has been a focus of studies for more than thirty years, there has been little consideration in AI Alignment studies of established resources for aligning technologies. Unlike the current focus of AI Alignment on addressing potential AI risks, technology alignment is generally focused on aligning with opportunities. Established resources include the critical realist philosophy of science, scientific theories, total quality management practices, technology alignment methods, engineering techniques, and technology standards. Here, these established resources are related to the alignment of different types of machine learning with different levels of human organizations. In addition, established resources are related to a well-known hypothetical extreme example of AI Misalignment, and to major constructs in the AI Alignment literature. Overall, it is argued that AI Alignment needs to be adaptive in order for human organizations to be able to survive in changing environments, and that established resources can facilitate Adaptive AI Alignment which addresses risks while focusing on opportunities. Full article

Previous reviews have investigated machine learning (ML) models used to predict the risk of developing preeclampsia. However, they have not addressed the intended deployment of these models throughout pregnancy, nor have they detailed feature performance. This study aims to provide an overview of existing ML models and their intended deployment patterns and performance, along with identified features of high importance. This review followed the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) 2020 guidelines. The search was performed in January and February 2024. It included all papers published before March 2024 obtained from the scientific databases: PubMed, Engineering Village, the Association for Computing Machinery, Scopus, and Web of Science. Of a total of 198 identified studies, 18 met the inclusion criteria. Among these, 11 showed the intent to use the ML model as a single-use tool, two intended a dual-use, and two intended multiple-use. Ten studies listed the features of the highest importance, with systolic and diastolic blood pressure, mean arterial pressure, and hypertension frequently mentioned as critical predictors. Notably, three of the four studies proposing dual or multiple-use models were conducted in 2023 and 2024, while the remaining study is from 2009. No single ML model emerged as superior across the subgroups of PE. Incorporating body mass index alongside hypertension and either mean arterial pressure, diastolic blood pressure, or systolic blood pressure as features may enhance performance. The deployment patterns mainly focused on single use during gestational weeks 11+0 to 14+1. Full article

Assessing the sustainable development of green hydrogen and assessing its potential environmental impacts using the Life Cycle Assessment is crucial. Challenges in LCA, like missing environmental data, are often addressed using machine learning, such as artificial neural networks. However, to find an ML solution, researchers need to read extensive literature or consult experts. This research demonstrates how customised LLMs, trained with domain-specific papers, can help researchers overcome these challenges. By starting small by consolidating papers focused on the LCA of proton exchange membrane water electrolysis, which produces green hydrogen, and ML applications in LCA. These papers are uploaded to OpenAI to create the LlamaIndex, enabling future queries. Using the LangChain framework, researchers query the customised model (GPT-3.5-turbo), receiving tailored responses. The results demonstrate that customised LLMs can assist researchers in providing suitable ML solutions to address data inaccuracies and gaps. The ability to quickly query an LLM and receive an integrated response across relevant sources presents an improvement over manually retrieving and reading individual papers. This shows that leveraging fine-tuned LLMs can empower researchers to conduct LCAs more efficiently and effectively. Full article

Interaction networks are a method of displaying the significant characters in a narrative text and their interactions. We automatically construct interaction networks from dialogues in German fairy tales by the Brothers Grimm and subsequently visualize these networks. This requires the combination of algorithms for several tasks: coreference resolution for the identification of characters and their appearances, as well as speaker/addressee detection and the detection of dialogue boundaries for the identification of interactions. After an evaluation of the individual algorithms, the predicted networks are evaluated against benchmarks established by networks based on manually annotated coreference and speaker/addressee information. The evaluation focuses on specific components of the predicted networks, such as the nodes, as well as the overall network, employing a newly devised score. This is followed by an analysis of various types of errors that the algorithms can make, like a coreference resolution algorithm not realizing that the frog has transformed into a prince, and their impact on the created networks. We find that the quality of many predicted networks is satisfactory for use cases in which the reliability of edges and character types are not of critical importance. However, there is considerable room for improvement. Full article

Error correction is a vital element in modern automatic speech recognition (ASR) systems. A significant portion of ASR error correction work is closely integrated within specific ASR systems, which creates challenges for adapting these solutions to different ASR frameworks. This research introduces Lexical Error Guard (LEG), which leverages the extensive pre-trained knowledge of large language models (LLMs) and employs instructional learning to create an adaptable error correction system compatible with various ASR platforms. Additionally, a parameter-efficient fine-tuning method is utilized using quantized low-rank adaptation (QLoRA) to facilitate fast training of the system. Tested on the LibriSpeech data corpus, the results indicate that LEG improves ASR results when used with various Whisper model sizes. Improvements in WER are made, with a decrease from 2.27% to 2.21% on the “Test Clean” dataset for Whisper Large with beam search. Improvements on the “Test Other” dataset for Whisper Large with beam search are also made, from 4.93% to 4.72%. Full article

In this paper, we present a novel model that enhances performance by extending the dual-modality TEVAD model—originally leveraging visual and textual information—into a multi-modal framework that integrates visual, audio, and textual data. Additionally, we refine the multi-scale temporal network (MTN) to improve feature extraction across multiple temporal scales between video snippets. Using the XD-Violence dataset, which includes audio data for violence detection, we conduct experiments to evaluate various feature fusion methods. The proposed model achieves an average precision (AP) of 83.9%, surpassing the performance of single-modality approaches (visual: 73.9%, audio: 67.1%, textual: 29.9%) and dual-modality approaches (visual + audio: 78.8%, visual + textual: 78.5%). These findings demonstrate that the proposed model outperforms models based on the original MTN and reaffirm the efficacy of multi-modal approaches in enhancing violence detection compared to single- or dual-modality methods. Full article

The exceptional performance of ImageNet competition winners in image classification has led AI researchers to repurpose these models for a whole range of tasks using transfer learning (TL). TL has been hailed for boosting performance, shortening learning time and reducing computational effort. Despite these benefits, issues such as data sparsity and the misrepresentation of classes can diminish these gains, occasionally leading to misleading TL accuracy scores. This research explores the innovative concept of endowing ImageNet models with a self-awareness that enables them to recognize their own accumulated knowledge and experience. Such self-awareness is expected to improve their adaptability in various domains. We conduct a case study using two different datasets, PBC and BCCD, which focus on blood cell classification. The PBC dataset provides high-resolution images with abundant data, while the BCCD dataset is hindered by limited data and inferior image quality. To compensate for these discrepancies, we use data augmentation for BCCD and undersampling for both datasets to achieve balance. Subsequent pre-processing generates datasets of different size and quality, all geared towards blood cell classification. We extend conventional evaluation tools with novel metrics—“accuracy difference” and “loss difference”—to detect overfitting or underfitting and evaluate their utility as potential indicators for learning behavior and promoting the self-confidence of ImageNet models. Our results show that these metrics effectively track learning progress and improve the reliability and overall performance of ImageNet models in new applications. This study highlights the transformative potential of turning ImageNet models into self-aware entities that significantly improve their robustness and efficiency in various AI tasks. This groundbreaking approach opens new perspectives for increasing the effectiveness of transfer learning in real-world AI implementations. Full article

Recent studies have shown correlations between the microbiota’s composition and various health conditions. Machine learning (ML) techniques are essential for analyzing complex biological data, particularly in microbiome research. ML methods help analyze large datasets to uncover microbiota patterns and understand how these patterns affect human health. This study introduces a novel approach combining statistical physics with the Monte Carlo (MC) methods to characterize bacterial species in the human microbiota. We assess the significance of bacterial species in different age groups by using notions of statistical distances to evaluate species prevalence and abundance across age groups and employing MC simulations based on statistical mechanics principles. Our findings show that the microbiota composition experiences a significant transition from early childhood to adulthood. Species such as Bifidobacterium breve and Veillonella parvula decrease with age, while others like Agathobaculum butyriciproducens and Eubacterium rectale increase. Additionally, low-prevalence species may hold significant importance in characterizing age groups. Finally, we propose an overall species ranking by integrating the methods proposed here in a multicriteria classification strategy. Our research provides a comprehensive tool for microbiota analysis using statistical notions, ML techniques, and MC simulations. Full article

Artificial Intelligence (AI) has the potential to revolutionise the medical and healthcare sectors. AI and related technologies could significantly address some supply-and-demand challenges in the healthcare system, such as medical AI assistants, chatbots and robots. This paper focuses on tailoring LLMs to medical data utilising a Retrieval-Augmented Generation (RAG) database to evaluate their performance in a computationally resource-constrained environment. Existing studies primarily focus on fine-tuning LLMs on medical data, but this paper combines RAG and fine-tuned models and compares them against base models using RAG or only fine-tuning. Open-source LLMs (Flan-T5-Large, LLaMA-2-7B, and Mistral-7B) are fine-tuned using the medical datasets Meadow-MedQA and MedMCQA. Experiments are reported for response generation and multiple-choice question answering. The latter uses two distinct methodologies: Type A, as standard question answering via direct choice selection; and Type B, as language generation and probability confidence score generation of choices available. Results in the medical domain revealed that Fine-tuning and RAG are crucial for improved performance, and that methodology Type A outperforms Type B. Full article

This study explores the integration of directional microphones with convolutional neural networks (CNNs) for long-range bird species identification. By employing directional microphones, we aimed to capture high-resolution audio from specific directions, potentially improving the clarity of bird calls over extended distances. Our approach involved processing these recordings with CNNs trained on a diverse dataset of bird calls. The results demonstrated that the system is capable of systematically identifying bird species up to 150 m, reaching 280 m for species vocalizing at frequencies greater than 1000 Hz and clearly distinct from background noise. The furthest successful detection was obtained at 510 m. While the method showed promise in enhancing the identification process compared to traditional techniques, there were notable limitations in the clarity of the audio recordings. These findings suggest that while the integration of directional microphones and CNNs for long-range bird species identification is promising, further refinement is needed to fully realize the benefits of this approach. Future efforts should focus on improving the audio-capture technology to reduce ambient noise and enhance the system’s overall performance in long-range bird species identification. Full article

Traditional methods of agricultural disease detection rely primarily on manual observation, which is not only time-consuming and labor-intensive, but also prone to human error. The advent of deep learning has revolutionized plant disease detection by providing more accurate and efficient solutions. The management of potato diseases is critical to the agricultural industry, as these diseases can lead to substantial losses in crop production. The prompt identification and classification of potato leaf diseases are essential to mitigating such losses. In this paper, we present a novel approach that integrates a lightweight convolutional neural network architecture, RegNetY-400MF, with transfer learning techniques to accurately identify seven different types of potato leaf diseases. The proposed method not only enhances the precision of potato leaf disease detection but also reduces the computational and storage demands, with a mere 0.40 GFLOPs and a model size of 16.8 MB. This makes it well-suited for use on edge devices with limited resources, enabling real-time disease detection in agricultural environments. The experimental results demonstrated that the accuracy of the proposed method in identifying seven potato leaf diseases was 90.68%, providing a comprehensive solution for potato crop management. Full article

Accurately predicting financial entity performance remains a challenge due to the dynamic nature of financial markets and vast unstructured textual data. Financial knowledge graphs (FKGs) offer a structured representation for tackling this problem by representing complex financial relationships and concepts. However, constructing a comprehensive and accurate financial knowledge graph that captures the temporal dynamics of financial entities is non-trivial. We introduce FintechKG, a comprehensive financial knowledge graph developed through a three-dimensional information extraction process that incorporates commercial entities and temporal dimensions and uses a financial concept taxonomy that ensures financial domain entity and relationship extraction. We propose a temporal and relational graph convolutional network (RGCN)-based representation for FintechKG data across multiple timesteps, which captures temporal dependencies. This representation is then combined with FinBERT embeddings through a projection layer, enabling a richer feature space. To demonstrate the efficacy of FintechKG, we evaluate its performance using the example task of financial performance prediction. A logistic regression model uses these combined features and social media embeddings for performance prediction. We classify whether the revenue will increase or decrease. This approach demonstrates the effectiveness of FintechKG combined with textual information for accurate financial forecasting. Our work contributes a systematic FKG construction method and a framework that utilizes both relational and textual embeddings for improved financial performance prediction. Full article

Implementing management systems in organisations of all types and sizes often raises the following question: “What benefits will this bring?” Initial resistance and criticism are common as potential challenges are identified during the implementation process. To address this, it is essential to highlight the advantages of these systems and engage stakeholders in supporting management efforts. While the planning, implementation, use, maintenance, auditing, and improvement of management systems are generally voluntary, certification is frequently driven by external factors, particularly customer demands. Employees also stand to gain significantly, with knowledge and information serving as valuable resources, especially for leveraging artificial intelligence. This article explores the management’s readiness to adopt and fully utilise two management systems based on international standards: the ISO 30401 Knowledge management system (KMS) and the ISO/IEC 42001 Artificial intelligence management system (AIMS). Through interviews, we assess the challenges and solutions associated with implementing these systems, whether planned or partially adopted. The findings illustrate the synergistic benefits of integrating the KMS and AIMS, demonstrating how their combined use can enhance Integrated Management Systems (IMSs). Such integration supports comprehensive planning, operation, and performance evaluation of processes and services while also promoting continuous improvement. Full article

Brain tumors are among the most lethal diseases, and early detection is crucial for improving patient outcomes. Currently, magnetic resonance imaging (MRI) is the most effective method for early brain tumor detection due to its superior imaging quality for soft tissues. However, manual analysis of brain MRI scans is prone to errors, largely influenced by the radiologists’ experience and fatigue. To address these challenges, computer-aided diagnosis (CAD) systems are more significant. These advanced computer vision techniques such as deep learning provide accurate predictions based on medical images, enhancing diagnostic precision and reliability. This paper presents a novel CAD framework for multi-class brain tumor classification. The framework employs six pre-trained deep learning models as the base and incorporates comprehensive data preprocessing and augmentation strategies to enhance computational efficiency. To address issues related to transparency and interpretability in deep learning models, Gradient-weighted Class Activation Mapping (Grad-CAM) is utilized to visualize the decision-making processes involved in tumor classification from MRI scans. Additionally, a user-friendly Brain Tumor Detection System has been developed using Streamlit, demonstrating its practical applicability in real-world settings and providing a valuable tool for clinicians. All simulation results are derived from a public benchmark dataset, showing that the proposed framework achieves state-of-the-art performance, with accuracy approaching 99% in ResNet-50, Xception, and InceptionV3 models. Full article