Texture analysis of the developing human brain using customization of a knowledge-based system

Research background & goals

The aim of this research was to find a customizable software framework (KBS) to mathematically determine an optimal combination of texture analysis methods to differentiate anatomical structures in the developing fetal brain (i.e. regions of interest (ROI)). Why did this experiment focus on normal development of central nervous system (CNS) rather than common conditions affecting fetal organogenesis? When considering fetal intervention to correct an anomaly in utero, the first ethical priority is actually mother safety. Prenatal well-being is clinically second, after evaluation of benefit-risk ratio^22–26. In the 1960s, fetal surgery was conceived and introduced in clinical practice²⁷. In spite of the improvement in surgical technology, the number of successfully-treated cases of congenital defects and life expectancy of survivors are still limited^28–31. The theoretical procedures are troublesome and have not been investigated enough³². Hence, they are considered as experimental treatments³². Whether invasive or minimally invasive, fetal intervention is not the ultimate solution of this societal conundrum. It is usually reserved for cases of severe fetal anomalies^28–32. In recent years, prenatal therapy is gaining popularity in religiously conservative territories particularly where abortion is prohibited^33,34. Fetal intervention is recommended for fetuses with mild and non-lethal defects, in order to discourage abortion and continue childbearing. In the country where this research was carried out (i.e. Poland), abortion is, once again, nationally banned ― except in cases of rape, life-threatening pregnancy and childbirth, and grave malformations^33–36. Pressures are due to growing pro-life supporters demanding total anti-abortion, child-bearing at all costs, and capital punishment for illegal termination of pregnancy^33–36. Nevertheless, Poland neither sanctions nor executes the outlaws of illegal abortion. Despite the absence of penalty, abortion is respectfully performed as permitted by local authority. The aforementioned dilemma justified the usage of fetal MRI (fMRI) in this research. In theory, it was previously hypothesized and documented that the high electromagnetic fields (EMF) used for MR procedures can disrupt the early stage of organogenesis. To this date, the embryotoxic, fetoxic, and teratogenic effects of MRI are not well known. Normally, fMRI is not recommended during the first and second trimester, unless it is absolutely necessary to confirm and/or supplement the diagnosis of fetal anomalies. Legal decision to prescribe abortion to a patient sometimes requires advanced clinical investigation, accompanied by psychological counseling before and after the operations^37–41. To this date, ultrasound (US) devices are preferred for obstetrical examinations. With 2D, 3D, and 4D US scans, physicians can effectively and efficiently diagnose the majority of life-threatening conditions affecting mother and fetus^42–46. Therefore, ultrasonography (USG) is sufficient for diagnosis of severe malformations affecting abdominal organs. Why was magnetic resonance imaging (MRI) prescribed? Fetal brain is where US devices struggle to produce desirable results. MRI was subsequently performed to rule out severe abnormalities in brain development, which are not visible on sonogram. Magnetic resonance (MR) samples came from fetuses with suspected heart and kidney defects. Disruption of organogenesis in the latter, depending on severity, might affect normal development of the brain. MR samples used in this experiment were visually investigated by specialists, structure-by-structure. No severe malformations were observed. In these cases, MRI studies did not add any further indication to legally fulfill the criteria to terminate pregnancy. The human visual apparatus has its limit. Unfortunately, missed diagnoses do occur. Malformations may not be apparent prior to birth. If fetal defects are suspected, bureaucracy may also restrict access to more advanced testing and healthcare. Consequently, pregnant women are deliberately forced to bear and deliver malformed babies. At the end of the day, physicians may still have to deal with the legal liability for failure to terminate pregnancy⁴¹. Unwanted fetuses may become neglected, and foundling is also a growing problem in society^47–49 . Congenital brain defects and its impacts on physical and cognitive development may not be detectable until after birth. Fetal outcome and mental retardation can be difficult for a physician to predict. The process requires access to better medical examination, development of more advanced tools and further investigation. That is why the ultimate goal of this feasibility study was to gather knowledge for the practicality of a proposed project seeking to improve diagnostic accuracy and precision, by extending HCI usage in medicine. There are many computer-aided diagnostic tools on the commercial shelves ( — e.g. Radiomics,^50–52. Definiens Tissue Phenomics®,⁵³ CAD4TB Diagnostic Software^54,55). Sadly, they are primarily designed for pre-loaded applications but not much else.

Ethics and informed consent

This feasibility study was approved by the Bioethics Research Board which regulates experiments carried out at the Medical University of Lodz and affiliated research hospitals in Poland (permit number: RNN/213/13/KE). Due to administrative and logistic delay as well as finding volunteers and expensive cost and availability of MRI^56,57, it took us nearly five years to collect the magnetic resonance (MR) samples. The nature of the study was explained to patients in Polish by attending physicians, and individual written informed consent was obtained for this research and its publication. Agreement number: 3/2011 - concluded on December 6, 2011, between the experimenter (Hugues Gentillon), research supervisor (Ludomir Stefańczyk) and rector of Medical University of Lodz (Radzisław Kordek), Faculty of Biomedical Science - Post-graduate research in Diagnostic Imaging and Radiotherapy. Consent for publication of these data was obtained. Furthermore all data from human participants were anonymized as per consent agreement. Informed Bioethics Committee Approval Number: RNN/213/13/KE, JULY 16, 2013. If you have any questions regarding the decision please include the above number and date in your letter. Send correspondence to: THE BIOETHICS COMMITTEE OF THE MEDICAL UNIVERSITY OF LODZ Al. Kościuszki 4, 90-419 Łódź, tel. 0 785 911 596, 42 272-59-05, fax 42 272-59-07.

Clinical trial registration

Though this research shares similarities with a clinical trial, it is “virtual” ― i.e. it is non-interventional. Such medical study does not meet criteria for clinical trial registration^58–62. Furthermore the investigational tools were merely used for technical exploration and to measure their feasibility in medical practice ― by using simulation settings. Lastly, the results were not used to alter patients’ therapeutic care and outcome^58–62.

Advance notice to readers. Readers should not expect us to teach the entire science of artificial neural network (ANN) (feedforward neural networks, recurrent neural network, etc.) in just a manuscript. It is not possible. A full introduction is not even possible. Unfamiliar readers are expected to make an effort on their own to read and learn the basic principles of artificial neural network and know the basic terminology. Like a human brain, an ANN can store memories. ANN can also judge based on stored memories and logical rules. To run a naïve trial run, it was ideal to have the ANN in a condition like ‘permanent global amnesia’ – i.e. a phenomenon where a brain is in a state of total blackout and thus cannot judge based on prior memories. The KBS used in this experiement was lacking an automatic memory cleaner and optimizer. Hence, stored memories were manually deleted in the ANN for every trial run.

Computer vision.Under a reciprocal cooperation-agreement between MUL and TUL, we obtained permissionto test a KBS consisting of a custom version of MaZda software package.63,64 It contains algorithms for data classification and visualization. How did it come to us? In 1998, a group of medical scientists, engineers, physicists, mathematicians and others initiated the B11 project at the Cooperation in Science and Technology (COST). The purpose was to develop software frameworks for quantitative analysis of MR scans, to improve medical diagnosis. MaZda, a Delphi/C++ computer program, was originally built in 1996 for applications in mammography. It was later used by COST to complement its MRI software modules (e.g. B11, B21). MaZda 4.6 package was the last official release (B11 version 3.3 included). In this research, we collaborated with modular programmers to further upgrade, separate and amalgamate the functionality and compatibility of MaZda (version 5 RC HG) with other modules. In the MaZda 5 version used in this research, the new features names were introduced to maintain compatibility with WEKA software (www.cs.waikato.ac.nz/ml/weka/, a popular package for data mining, classification and analysis). This way data generated by Mazda may be used as the input to the WEKA. Also, 3D deformable model that are used for 3D volume-of-interest (VOI) generating was introduced (— for details, see the publication by: Piotr M. Szczypiński, Ram D. Sriram, Parupudi V.J. Sriram, D. Nageshwar Reddy, A model of deformable rings for interpretation of wireless capsule endoscopic videos, Medical Image Analysis, Volume 13, Issue 2, April 2009, Pages 312–324). Moreover, to speed up and automate, some commands regarding 3D analysis were introduced. This way one can build a script (that contains e.g. commands for image loading, analysis option loading, performing analyses and storing results) for automatic analysis of a number of 3D data. Finally, this MaZda 5 version was compiled with Embarcadero software environment, which was not an ideal framework. Thus, the inventors decided to return to Builder C++. Currently, the latest version of Mazda being built is qMaZda (described and available at www.eletel.p.lodz.pl/pms/SoftwareQmazda.html).

Figure 1 shows the main steps of texture analysis with MaZda and B11. A key change in this custom version is the integration of MaZda with a variety of computational geometric algorithms from Qhull (e.g. VSCH_1, VSCH_2, VSCH_3, VSCH_4, VSCH_5). For details about 1D, 2D, 3D, and 4D features, see Dataset 7. VSCH network algorithms are located in the ‘Convex Border’ menu.VSCH_1, for example, can be used to identify the strongest discriminant parameter in a large dataset.

Figure 1. Main steps of texture analysis with MaZda and B11.

Our trial-and-error experiments and texture-analysis software development spanned over five years of research. All the findings shared common denominators. Quantitative brain-tissue segmentation was affected by several factors, such as characteristics of fetuses (e.g. gestational age, shape, normal/abnormal development) and the quality of fMR images (e.g. 1.5/3T, resolution, slice thickness, rotational planes, artifacts, etc). Automatic segmentation of newborn brain MRI has been documented in the literature⁶⁵. The algorithmic contributions reported so far have achieved limited success, unfortunately. Automatic segmentation of prenatal brain is even more challenging and time-consuming.

Unsupervised segmentation.Once an image is acquired in a readable format (bitmap format (BMP)), the first step is texture segmentation – i.e. partitioning an image into ROIs. B11 can perform unsupervised segmentation and cluster analysis. In some instances, B11 achieved accuracy closed to that of clinicians. However, unsupervised segmentation was not reliable enough for therapeutic use. Fetal brain segmentation with B11 still required extensive expert interaction. In our observations, the key problems were maternal factors, environmental effects, growth variability, randomness of fetal movements and its detrimental effects on image quality. Therefore, automatic segmentation was used for new insight into the possibility of improving supervised segmentation. The steps of unsupervised segmentation are relatively simple: image acquisition and run analysis. ROIs and segment numbers can be manually adjusted. The drawback with B11 segmentation is limitation to 8-bit grayscale BMP. 16-bit DICOM was converted to visually lossless BMP, by dropping least significant bits. Note that B11 identified textures not anatomical structures. The information collected from the unsupervised trials was later used as guidance to manually estimate boundaries of anatomical structures (ROIs) for the supervised trials. The preliminary trials were single-blinded ― i.e. the user knew the characteristics of the ROIs, and the KBS received no hints (no ROI selection). Further information was gathered with a semi-automatic (unblended) segmentation by defining 4 classes (ROIs): thalamus, ventricles, grey matter and white matter. In unsupervised mode, the KBS performs quite well when brain images are from MR examination of the same subjects and same sequences ― but performs poorly when they came from different subjects. The findings were likewise for same sequences of the same patient taken at a different time and MR scanner settings. The challenge was: how do we match macroscopic characteristics with electronic recognition, regardless of MR image shadings? MaZda and B11 are not yet designed to allow user to well define semantic rules and/or import plugins for fully electronic recognition of anatomical structures. Object-based image analysis tools such as eCognition work consistently well for geo-spatial applications (e.g. identification of a river in an image)^66,67. In fetal radiology, it is still a challenge to achieve consistent results with automated-pattern recognition of prenatal anatomy. Programming a reliably effective system for such highly sensitive application is feasible but also time-consuming. Such a task would require taking into account all known variations due to pregnancy chronology and fetal developmental, to minimize segmentation errors. MaZda and B11 were originally built for HCI rather than fully-automated applications. Therefore, the best practical methodology, in this research, was for the operator to at least have prior knowledge of human embryogenesis, in order to manually and correctly identify and select fetal ROIs. Often, macroscopic appearance of many brain structures are not well differentiated in the first trimester. Hence, the selected samples were at least 20 weeks of maternal age.

Supervised segmentation. 3-tesla (3T) and 1.5-tesla (1.5T) magnetic resonance (MR) sequences of fetal brain were manually segmented into 3456 ROI. The categories were predefined as followed: ventricles (class 1), thalamus (class 2), white matter (class 3) and grey matter (class 4). The selected samples did not have any brain malformations. As previously mentioned, the anomalies were in the cardiovascular and/or renal systems. The focus of this research was on normal anatomy of fetal brain. Forward processing method ― also known as “supervised segmentation”⁶⁸ ― was performed as delineated: (1) image acquisition from MR scanner, (2) selection of ROIs with MaZda, (3) image normalization with MaZda (4) feature extraction with MaZda (5) data preprocessing with B11 (6) texture classification with B11^63,64,68. The first four steps were done with MaZda and last two steps with B11 (Figure 1). After the preliminary trials, we became interested to learn what needs to be adjusted in order to reduce misclassification of MR images. The unsupervised segmentation revealed that the KBS was very sensitive to greyscale shading, artifacts, and image thickness as well as resolution quality. Thus, we trained the KBS accordingly. 1.5T images were originally encoded as 12-bit lossless JPEG (Joint Photographic Experts Group) format and wrapped in Digital Imaging and Communications in Medicine (DICOM). 1.5T images were transcoded from lossless JPEG to uncompressed DICOM. 3T images were natively uncompressed. 360 parameters were extracted with MaZda (histogram: 9; co-occurrence matrix: 220; run-length matrix: 20; gradient matrix: 5; auto-regression: 5; Haar wavelet: 28; geometry: 73). Parameters’ names are provided in the appendix at the end of this manuscript. We assessed three automated techniques of parameter selection ― i.e. Fisher selection, POE + ACC (classification error probability and average correlation coefficients), MI+PA+F (combination of mutual information, pair analysis and fisher selection). Further details about the mechanics and functionality of these techniques are provided in the manufacturer’s user manual and tutorial guides⁶⁹. For each technique, we also used the VSCH (vector supported convex hull) module in MaZda to enhance computation and visualization of geometric structures: 1) pre-reduction (before selection) to rule out insignificant parameters, 2) post-reduction (after selection) to identify strongest relevant parameters. All aforementioned parts of the process were computed automatically by the KBS without any manual interference. Data imported [SEL (Schweitzer Engineering Laboratories) to CSV (comma-separated values)] in B11 were preprocessed with PCA, LDA and NDA and classified by means of 1-nearest neighbor (1-NN) and ANN. In machine learning and cognitive science, 1-NN is also known as k-NN, where n=1. It is a “lazy-learning” algorithm, in which new ROIs are locally classified by getting interweaved into the closest cluster in the training set. The rest of the computation is delayed until the end of the classification process. K-NN is implemented in B11 classifiers, as well as in the preprocessing procedures in MaZda⁶⁹.On the other hand, ANN is a self-organizing algorithm with hidden layers and adjustable number of neurons^69–72. It can be used for both supervised and unsupervised classification. Neural classification algorithms are implemented in MaZda/B11. ANN training, for example, is standardized for NDA analysis^69,70.― i.e. a type of feedforward-artificial neural network, based on multilayer Perceptron (MLP)⁶⁹. Nonlinear procedures and classifiers in B11 are MPL algorithms^69,70. For optimal performance, two sets of samples are needed: one for training and another one for validation⁷⁰. The pitfall with this algorithm is its sensitivity to overtraining (too strong memorization)^69,70. ANN training time is shorter with standardization. For continuation, training without standardization was carried out, in spite of long processing time. ANN (one-class/n-class) and 1-NN training runs were conducted with different sequences of MR images: T2-weighted (T1), T1 weighted, and proton-density (PD) sequences. N-class training was discontinued due to repeated problems with overtraining and lack of reproducibility in F values and miss-classification errors.

Customization.

Despite the usage of multi-level, automated selection/reduction techniques, some extracted values still did not match the controlled ROI values. Differentiating thalamus from other thalamic nuclei and grey matter was the key problem. That was when we manually intervened to customize and improve the extracted data. First ROI surface areas were manually increased, in order to limit the number of parameters reporting zero and infinity values. Parameters which couldn’t be correctly computed were manually omitted in the report file. Some pre-processing procedures in both MaZda and B11 couldn’t be performed when the report file contained erroneous values. We accessed MaZda generated report files by changing the extension format from SEL to CSV and then imported them into Excel 2013 for adjustment. Parameters measured with other CAD tools can also be entered in the report files by simply using Microsoft Excel. The edited file can then be imported in B11 to perform texture classification.

Regions of interest.

Additional tests were carried out with same ROIs (i.e. thalamus, ventricle, grey matter and white matter) to dramatically improve accuracy and precision of the KBS: it was done with a customized dataset derived from MaZda algorithms, using semi-manual reduction and nearest-neighbor feature selection (see Data availability: Dataset 1–Dataset 2). The training data were used to orient the KBS to recognize what ROIs had the same tissue characteristics, in spite of being originated from different patients or different sequences of the same patients. The training was conducted with combination of two built-in classification tools (i.e. nearest neighbor (NN) and artificial neural network) and four data processing techniques (i.e. RAW: read as written; PCA: principal component analysis, LDA: linear discriminant analysis; NDA: nonlinear discriminant analysis). To measure the KBS sensitivity and specificity, we defined “normal” as "ROIs with identical tissue" characteristics and “abnormal” those with different tissue characteristics (Figure 2–Figure 5). Apart from noise and artifacts, we found out that the preliminary results were also affected by the planes (axial, coronal, sagittal) ― which refer to the rotational planes of the spinning MR scanner in relation to the mother, not the fetus. There flows the reason for the classification by rotational planes. In learning mode, we observed a consistent scoring for all the ROIs. Thus the logical and semantic information provided to the KBS was effective. Statistical binary tests (also known as classification function tests) were computed in STATISTICA version 10 to assess the performance of each procedure (combination of preprocessing techniques and classifiers). In medicine, binary scores (TP, FP, TN, FN etc.) are used to determine not just normal and abnormal characteristics but also classification property of an examination.

Figure 2. White matter ROI selection.

a) specificity scoring: ROI 1 = white matter control, ROI 2 = white matter | ― b) sensitivity scoring: ROI 1: white matter control, ROI 2: thalamic nucleus other than thalamus.

Figure 3. Thalamus ROI selection.

a) specificity scoring: ROI 1 = thalamus control, ROI 2 = thalamus | ― b) sensitivity scoring: ROI 1: thalamus control, ROI 2: thalamic nucleus other than thalamus.

Figure 4. Ventricle ROI selection.

a) specificity scoring: ROI 1 = ventricle control, ROI 2 = ventricle | ― b) sensitivity scoring: ROI 1: ventricle control, ROI 2: white matter.

Figure 5. Grey matter ROI selection.

a) specificity scoring: ROI 1 = grey matter control; ROI 2 = grey matter | ― b) sensitivity scoring: ROI 1: grey matter control, ROI 2: white matter.

Selecting KBS tools

The majority of the KBS we came across were designed for technical use and not easily customizable. Such programs required paying for marketing company maintenance and for in- house-developed customization service, on top of the annual license fee. Thus this option was not feasible for application in real-world settings, where resources are often sparse ( — e.g. eCognition,^66,67 Media Cybernetics,^77–78 Radiomics,^50–52 Definiens Tissue Phenomics®,⁵³ CAD4TB Diagnostic Software^54,55, etc.).

Logic and reasoning behind the research design

Previous medical studies done with MaZda include inflammation, brain cancer detection, multiple sclerosis, electrophoresis, etc^79–82. Herein, we defined some test samples as “abnormal” ROIs. However, they were, in reality, normal tissue. Not testing directly for a common anomaly doesn’t necessarily mean that there is no real medical application. Though the tests were simulated, the research design was conceived for real-world medical applications^83–85. For example, this research design could be used to detect ectopic tissue migration, neurogenesis and neuronal migration (brain function migration as a result of natural process or after injury), metaplasia and interference with brain development. Last but not least, this simulation research followed standards used in clinical trials⁸⁶.

Statistical test and interpretation

The choice of binary classification (sensitivity/specificity) was favored over frequentist inference (p-value) because it provides more information in terms of statistical relevance to medical diagnosis, prognosis and disease prevalence^87–89. One key difference between Fisher, POE+ACC, and MI+PA+F is the number of parameters. To perform MI+PA+F, the dataset must contain at least 30 parameters strongly matching its selection-reduction criteria⁶⁹. Otherwise, the KBS reported an error. In our study, it was a common occurrence when the surface area of a ROI was insufficient to extract 30 parameters meeting the MI+PA+F semantics. RAW and PCA were not so affected by the training process and thus remained very sensitive to minute difference in greyscale shading.

Recommendations

A solution to high misclassification (M) was to exclude some parameters which were very sensitive to post-editing sharpness. In this research, the images were, however, processed without post-editing sharpness because high M was not regarded as a problem. Instead, we used RAW and PCA as reference tests (results before the training of the KBS). On the other hand, LDA and NDA responded well to the training, and M was consistently zero.

KBS memory clearing

During the study, we had to obviously clear the KBS memory several times for every trial run. We hope that the software developers will soon implement a more effective and efficient way (e.g. one-click) to clear specific random-access memory (RAM) without closing module(s) or without manually dumping the entire RAM or restarting the computer.

Constraints, limitations, and assumptions

Patients gave consent to perform MRI examination and use of images in research and for the manuscript publication. Nevertheless, this authorization was not enough, as ownership and copyright of medical records are not always exclusively attributed to patients ― and such rights may not be assignable^90–92. For the sake of prudence, we had to also seek institutional/research- hospitals’ clearance and approval ― which in turn were then subject to different administrative and logistic factors and regulations. Consequently, it took nearly five years to collect sufficient MRI samples to make this research possible.