View source for Inductive logic programming

[[File:ILP family2.png|thumb|A photo of Family sample for Inductive Logic Programming article]]
'''Inductive logic programming''' ('''ILP''') is a subfield of [[symbolic artificial intelligence]]  which uses [[logic programming]] as a uniform representation for examples, background knowledge and hypotheses.  The term "''inductive''" here refers to [[Inductive reasoning|philosophical]] (i.e. suggesting a theory to explain observed facts) rather than [[mathematical induction|mathematical]] (i.e. proving a property for all members of a well-ordered set) induction. Given an encoding of the known background knowledge and a set of examples represented as a logical [[database]] of facts, an ILP system will derive a hypothesised logic program which [[Entailment|entails]] all the positive and none of the negative examples.

* Schema: ''positive examples'' + ''negative examples'' + ''background knowledge'' ⇒ ''hypothesis''.

Inductive logic programming is particularly useful in [[bioinformatics]] and [[natural language processing]].

== History ==
Building on earlier work on [[Inductive inference]], [[Gordon Plotkin]] was the first to formalise induction in a [[Horn clause|clausal]] setting around 1970, adopting an approach of generalising from examples.<ref name=":02">{{Cite book |last1=Nienhuys-Cheng |first1=Shan-hwei |title=Foundations of inductive logic programming |last2=Wolf |first2=Ronald de |date=1997 |publisher=Springer |isbn=978-3-540-62927-6 |series=Lecture notes in computer science Lecture notes in artificial intelligence |location=Berlin Heidelberg |pages=174–177}}</ref><ref>{{cite thesis |first=G.D. |last=Plotkin |title=Automatic Methods of Inductive Inference |date=1970 |type=PhD |publisher=University of Edinburgh |url=https://www.era.lib.ed.ac.uk/bitstream/handle/1842/6656/Plotkin1972.pdf |hdl=1842/6656}}</ref> In 1981, [[Ehud Shapiro]] introduced several ideas that would shape the field in his new approach of model inference, an algorithm employing refinement and backtracing to search for a complete axiomatisation of given examples.<ref name=":02"/><ref>{{cite tech report|first=Ehud Y.|last=Shapiro|title=Inductive inference of theories from facts|id=192|date=1981|publisher=Department of Computer Science, Yale University|url=http://ftp.cs.yale.edu/publications/techreports/tr192.pdf}} Reprinted in {{cite book |title=Computational logic : essays in honor of Alan Robinson |publisher=MIT Press |year=1991 |isbn=978-0-262-12156-9 |editor1-last=Lassez |editor1-first=J.-L. |pages=199–254 |editor2-last=Plotkin |editor2-first=G.}}</ref> His first implementation was the [[Model Inference System]] in 1981:<ref>{{cite book |last=Shapiro |first=Ehud Y. |url= |title=Proceedings of the 7th international joint conference on Artificial intelligence |date=1981 |publisher=Morgan Kaufmann |isbn= |volume=2 |location= |pages=1064 |chapter=The model inference system |chapter-url=https://www.ijcai.org/Proceedings/81-2/Papers/100.pdf}}</ref><ref>{{cite book |last=Shapiro |first=Ehud Y. |url= |title=Algorithmic program debugging |date=1983 |publisher=MIT Press |isbn=0-262-19218-7 |location= |pages=}}</ref> a [[Prolog]] program that inductively inferred [[Horn clause]] logic programs from positive and negative examples.<ref name=":02"/> The term ''Inductive Logic Programming'' was first introduced in a paper by [[Stephen Muggleton]] in 1990, defined as the intersection of machine learning and logic programming.<ref name=":02"/> Muggleton and Wray Buntine introduced predicate invention and [[inverse resolution]] in 1988.<ref name=":02"/><ref name="Proceedings of the 5th Internationa">{{cite book |last1=Muggleton |first1=S.H. |url= |title=Proceedings of the 5th International Conference on Machine Learning |last2=Buntine |first2=W. |date=1988 |isbn=978-0-934613-64-4 |pages=339–352 |chapter=Machine invention of first-order predicate by inverting resolution |doi=10.1016/B978-0-934613-64-4.50040-2}}</ref>

Several inductive logic programming systems that proved influential appeared in the early 1990s. [[First-order inductive learner|FOIL]], introduced by [[Ross Quinlan]] in 1990<ref>{{Cite journal |last=Quinlan |first=J. R. |date=August 1990 |title=Learning logical definitions from relations |journal=Machine Learning |volume=5 |issue=3 |pages=239–266 |doi=10.1007/bf00117105 |issn=0885-6125|doi-access=free }}</ref> was based on upgrading [[Propositional calculus|propositional]] learning algorithms [[AQ (machine learning)|AQ]] and [[ID3 algorithm|ID3]].<ref name=":12">{{Cite book |last1=Nienhuys-Cheng |first1=Shan-hwei |title=Foundations of inductive logic programming |last2=Wolf |first2=Ronald de |date=1997 |publisher=Springer |isbn=978-3-540-62927-6 |series=Lecture notes in computer science Lecture notes in artificial intelligence |location=Berlin Heidelberg |pages=354–358}}</ref> [[Golem (ILP)|Golem]], introduced by Muggleton and Feng in 1990, went back to a restricted form of Plotkin's least generalisation algorithm.<ref name=":12"/><ref name="Springer/Ohmsha">{{Cite journal |last1=Muggleton |first1=Stephen H. |last2=Feng |first2=Cao |date=1990 |editor-last=Arikawa |editor-first=Setsuo |editor2-last=Goto |editor2-first=Shigeki |editor3-last=Ohsuga |editor3-first=Setsuo |editor4-last=Yokomori |editor4-first=Takashi |title=Efficient Induction of Logic Programs |url=https://dblp.org/rec/conf/alt/MuggletonF90.bib |journal=Algorithmic Learning Theory, First International Workshop, ALT '90, Tokyo, Japan, October 8–10, 1990, Proceedings |publisher=Springer/Ohmsha |pages=368–381}}</ref> The [[Progol]] system, introduced by Muggleton in 1995, first implemented inverse entailment, and inspired many later systems.<ref name=":12"/><ref name=":3">{{Cite journal |last1=Cropper |first1=Andrew |last2=Dumančić |first2=Sebastijan |date=2022-06-15 |title=Inductive Logic Programming At 30: A New Introduction |journal=Journal of Artificial Intelligence Research |volume=74 |page=808 |doi=10.1613/jair.1.13507 |issn=1076-9757|doi-access=free |arxiv=2008.07912 }}</ref><ref name=":2">{{cite journal |last1=Muggleton |first1=S.H. |year=1995 |title=Inverting entailment and Progol |journal=New Generation Computing |volume=13 |issue=3–4 |pages=245–286 |citeseerx=10.1.1.31.1630 |doi=10.1007/bf03037227 |s2cid=12643399}}</ref> [[Aleph (ILP)|Aleph]], a descendant of Progol introduced by Ashwin Srinivasan in 2001, is still one of the most widely used systems {{As of|2022|lc=y|bare=}}.<ref name=":3" />

At around the same time, the first practical applications emerged, particularly in [[bioinformatics]], where by 2000 inductive logic programming had been successfully applied to drug design, carcinogenicity and mutagenicity prediction, and elucidation of the structure and function of proteins.<ref>{{Citation |last=Džeroski |first=Sašo |title=Relational Data Mining Applications: An Overview |date=2001 |url=http://link.springer.com/10.1007/978-3-662-04599-2_14 |work=Relational Data Mining |pages=339–364 |editor-last=Džeroski |editor-first=Sašo |access-date=2023-11-27 |place=Berlin, Heidelberg |publisher=Springer Berlin Heidelberg |language=en |doi=10.1007/978-3-662-04599-2_14 |isbn=978-3-642-07604-6 |editor2-last=Lavrač |editor2-first=Nada|editor2-link=Nada Lavrač}}</ref> Unlike the focus on [[automatic programming]] inherent in the early work, these fields used inductive logic programming techniques from a viewpoint of [[relational data mining]]. The success of those initial applications and the lack of progress in recovering larger traditional logic programs shaped the focus of the field.<ref>{{Citation |last=De Raedt |first=Luc |editor-first1=<!-- Deny Citation Bot--> |editor-last1=<!-- Deny Citation Bot--> |url=http://dx.doi.org/10.1007/978-3-540-68856-3 |title=Logical and Relational Learning |series=Cognitive Technologies |page=14|place=Berlin, Heidelberg |publisher=Springer |year=2008 |doi=10.1007/978-3-540-68856-3 |bibcode=2008lrl..book.....D |isbn=978-3-540-20040-6}}</ref>

Recently, classical tasks from automated programming have moved back into focus, as the introduction of meta-interpretative learning makes predicate invention and learning recursive programs more feasible. This technique was pioneered with the [[Metagol]] system introduced by Muggleton, Dianhuan Lin, Niels Pahlavi and Alireza Tamaddoni-Nezhad in 2014.<ref>{{Cite journal |last1=Muggleton |first1=Stephen H. |last2=Lin |first2=Dianhuan |last3=Pahlavi |first3=Niels |last4=Tamaddoni-Nezhad |first4=Alireza |date=2013-05-01 |title=Meta-interpretive learning: application to grammatical inference |url=http://dx.doi.org/10.1007/s10994-013-5358-3 |journal=Machine Learning |volume=94 |issue=1 |pages=25–49 |doi=10.1007/s10994-013-5358-3 |s2cid=254738603 |issn=0885-6125}}</ref> This allows ILP systems to work with fewer examples, and brought successes in learning string transformation programs, answer set grammars and general algorithms.<ref>{{Cite journal |last1=Cropper |first1=Andrew |last2=Dumančić |first2=Sebastijan |last3=Evans |first3=Richard |last4=Muggleton |first4=Stephen |date=2022 |title=Inductive logic programming at 30 |journal=Machine Learning |language=en |volume=111 |issue=1 |pages=147–172 |doi=10.1007/s10994-021-06089-1 |issn=0885-6125|doi-access=free }}</ref>

== Setting ==
Inductive logic programming has adopted several different learning settings, the most common of which are learning from [[entailment]] and learning from interpretations.<ref name="setting">{{Cite journal |last1=Cropper |first1=Andrew |last2=Dumančić |first2=Sebastijan |date=2022-06-15 |title=Inductive Logic Programming At 30: A New Introduction |journal=Journal of Artificial Intelligence Research |volume=74 |pages=779{{endash}}782 |doi=10.1613/jair.1.13507 |issn=1076-9757 |doi-access=free|arxiv=2008.07912 }}</ref> In both cases, the input is provided in the form of ''background knowledge {{mvar|B}}'', a logical theory (commonly in the form of [[Clause (logic)|clauses]] used in [[logic programming]]), as well as positive and negative examples, denoted <math display="inline">E^+</math> and <math display="inline">E^{-}</math> respectively. The output is given as a ''hypothesis'' ''{{mvar|H}}'', itself a logical theory that typically consists of one or more clauses.

The two settings differ in the format of examples presented.

=== Learning from entailment ===
{{As of|2022}}, learning from entailment is by far the most popular setting for inductive logic programming.<ref name="setting" /> In this setting, the ''positive'' and ''negative'' examples are given as finite sets <math display="inline">E^+</math> and <math display="inline">E^{-}</math> of positive and negated [[Ground expression|ground]] [[Literal (mathematical logic)|literals]], respectively. A ''correct hypothesis'' ''{{mvar|H}}'' is a set of clauses satisfying the following requirements, where the turnstile symbol <math>\models</math> stands for [[logical entailment]]:<ref name="setting" /><ref>{{cite book |last1=Džeroski |first1=Sašo |title=Advances in Knowledge Discovery and Data Mining |publisher=MIT Press |year=1996 |editor1-last=Fayyad |editor1-first=U.M. |pages=117–152 See §5.2.4 |chapter=Inductive Logic Programming and Knowledge Discovery in Databases |access-date=2021-09-27 |editor2-last=Piatetsky-Shapiro |editor2-first=G. |editor3-last=Smith |editor3-first=P. |editor4-last=Uthurusamy |editor4-first=R. |chapter-url=http://kt.ijs.si/SasoDzeroski/pdfs/1996/Chapters/1996_InductiveLogicProgramming.pdf |archive-url=https://web.archive.org/web/20210927141157/http://kt.ijs.si/SasoDzeroski/pdfs/1996/Chapters/1996_InductiveLogicProgramming.pdf |archive-date=2021-09-27 |url-status=dead}}</ref><ref>{{Cite journal |last=De Raedt |first=Luc |date=1997 |title=Logical settings for concept-learning |url=https://linkinghub.elsevier.com/retrieve/pii/S0004370297000416 |journal=Artificial Intelligence |language=en |volume=95 |issue=1 |pages=187–201 |doi=10.1016/S0004-3702(97)00041-6}}</ref>
<math display="block">\begin{array}{llll}
\text{Completeness:}
& B \cup H
& \models
& E^+
\\
\text{Consistency: }
& B \cup H \cup E^-
& \not\models
& \textit{false}
\end{array}</math>
Completeness requires any generated hypothesis ''{{mvar|h}}'' to explain all positive examples <math display="inline">E^+</math>, and consistency forbids generation of any hypothesis ''{{mvar|h}}'' that is inconsistent with the negative examples <math display="inline">E^{-}</math>, both given the background knowledge ''{{mvar|B}}''.

In Muggleton's setting of concept learning,<ref name="setting2">{{cite journal |last1=Muggleton |first1=Stephen |year=1999 |title=Inductive Logic Programming: Issues, Results and the Challenge of Learning Language in Logic |journal=Artificial Intelligence |volume=114 |issue=1–2 |pages=283–296 |doi=10.1016/s0004-3702(99)00067-3 |doi-access=}}; here: Sect.2.1</ref> "completeness" is referred to as "sufficiency", and "consistency" as "strong consistency". Two further conditions are added:  "''Necessity''", which postulates that ''{{mvar|B}}'' does not entail <math display="inline">E^+</math>,  does not impose a restriction on ''{{mvar|h}}'', but forbids any generation of a hypothesis as long as the positive facts are explainable without it. . "Weak consistency", which states that no contradiction can be derived from <math display="inline">B\land H</math>, forbids generation of any hypothesis ''{{mvar|h}}'' that contradicts the background knowledge ''{{mvar|B}}''. Weak consistency is implied by strong consistency; if no negative examples are given, both requirements coincide. Weak consistency is particularly important in the case of noisy data, where completeness and strong consistency cannot be guaranteed.<ref name="setting2" />

=== Learning from interpretations ===
In learning from interpretations, the ''positive'' and ''negative'' examples are given as a set of complete or partial [[Herbrand structure]]s, each of which are themselves a finite set of ground literals.   Such a structure ''{{mvar|e}}'' is said to be a model of the set of clauses <math display="inline">B \cup H</math> if for any [[Substitution (logic)|substitution]] <math display="inline">\theta</math> and any clause <math display="inline">\mathrm{head} \leftarrow \mathrm{body}</math> in <math display="inline">B \cup H</math> such that <math display="inline">\mathrm{body}\theta \subseteq e</math>, <math>\mathrm{head}\theta \subseteq e</math> also holds. The goal is then to output a hypothesis that is ''complete,'' meaning every positive example is a model of  <math display="inline">B \cup H</math>, and ''consistent,'' meaning that no negative example is a model of <math display="inline">B \cup H</math>.<ref name="setting" />

== Approaches to ILP ==
An inductive logic programming system is a program that takes as an input logic theories <math>B, E^+, E^-</math> and outputs a correct hypothesis {{mvar|H}} with respect to theories <math>B, E^+, E^-</math>. A system is complete if and only if for any input logic theories <math>B, E^+, E^-</math> any correct hypothesis {{mvar|H}} with respect to these input theories can be found with its hypothesis search procedure.  Inductive logic programming systems can be roughly divided into two classes, search-based and meta-interpretative systems.

Search-based systems exploit that the space of possible clauses forms a [[complete lattice]] under the [[Theta-subsumption|subsumption]] relation, where one clause <math display="inline">C_1</math>  subsumes another clause <math display="inline">C_2</math> if there is a [[Substitution (logic)|substitution]] <math display="inline">\theta</math> such that <math display="inline">C_1\theta</math>, the result of applying <math display="inline">\theta</math> to <math display="inline">C_1</math>, is a subset of <math display="inline">C_2</math>. This lattice can be traversed either bottom-up or top-down.

=== Bottom-up search ===
Bottom-up methods to search the subsumption lattice have been investigated since Plotkin's first work on formalising induction in clausal logic in 1970.<ref name=":02"/><ref>{{cite thesis |first=G.D. |last=Plotkin |title=Automatic Methods of Inductive Inference |date=1970 |type=PhD |publisher=University of Edinburgh |url=https://www.era.lib.ed.ac.uk/bitstream/handle/1842/6656/Plotkin1972.pdf |hdl=1842/6656}}</ref> Techniques used include least general generalisation, based on [[Anti-unification (computer science)|anti-unification]], and inverse resolution, based on inverting the [[Resolution (logic)|resolution]] inference rule.

==== Least general generalisation ====
A least general generalisation algorithm takes as input two clauses <math display="inline">C_1</math> and <math display="inline">C_2</math> and outputs the least general generalisation of  <math display="inline">C_1</math> and <math display="inline">C_2</math>, that is, a clause  <math display="inline">C</math> that subsumes  <math display="inline">C_1</math> and <math display="inline">C_2</math>, and that is subsumed by every other clause that subsumes  <math display="inline">C_1</math> and <math display="inline">C_2</math>. The least general generalisation can be computed by first computing all ''selections'' from <math display="inline">C</math> and <math display="inline">D</math>, which are pairs of literals  <math>(L,M) \in (C_1, C_2)</math> sharing the same predicate symbol and negated/unnegated status.  Then, the least general generalisation is obtained as the disjunction of the least general generalisations of the individual selections, which can be obtained by [[first-order syntactical anti-unification]].<ref>{{Cite book |last1=Nienhuys-Cheng |first1=Shan-hwei |title=Foundations of inductive logic programming |last2=Wolf |first2=Ronald de |date=1997 |publisher=Springer |isbn=978-3-540-62927-6 |series=Lecture notes in computer science Lecture notes in artificial intelligence |location=Berlin Heidelberg |page=255}}</ref>

To account for background knowledge, inductive logic programming systems employ ''relative least general generalisations'', which are defined in terms of subsumption relative to a background theory. In general, such relative least general generalisations are not guaranteed to exist; however, if the background theory ''{{mvar|B}}'' is a finite set of [[Ground expression|ground]] [[Literal (mathematical logic)|literals]], then the negation of ''{{mvar|B}}'' is itself a clause. In this case, a relative least general generalisation can be computed by disjoining the negation of ''{{mvar|B}}'' with both <math display="inline">C_1</math> and <math display="inline">C_2</math> and then computing their least general generalisation as before.<ref>{{Cite book |last1=Nienhuys-Cheng |first1=Shan-hwei |title=Foundations of inductive logic programming |last2=Wolf |first2=Ronald de |date=1997 |publisher=Springer |isbn=978-3-540-62927-6 |series=Lecture notes in computer science Lecture notes in artificial intelligence |location=Berlin Heidelberg |page=286}}</ref>

Relative least general generalisations are the foundation of the bottom-up system [[Golem (ILP)|Golem]].<ref name=":12"/><ref name="Springer/Ohmsha"/>

==== Inverse resolution ====
Inverse resolution is an [[inductive reasoning]] technique that involves [[wiktionary:invert|inverting]] the [[Resolution (logic)|resolution operator]].

Inverse resolution takes information about the [[Resolvent (logic)|resolvent]] of a resolution step to compute possible resolving clauses. Two types of inverse resolution operator are in use in inductive logic programming: V-operators and W-operators. A V-operator takes clauses <math display="inline">R</math> and <math display="inline">C_1</math>as input and returns a clause <math display="inline">C_2</math> such that <math display="inline">R</math> is the resolvent of <math display="inline">C_1</math> and  <math display="inline">C_2</math>. A W-operator takes two clauses <math display="inline">R_1</math> and <math display="inline">R_2</math> and returns thre clauses <math display="inline">C_1</math>, <math display="inline">C_2</math> and <math display="inline">C_3</math> such that <math display="inline">R_1</math> is the resolvent of <math display="inline">C_1</math> and <math display="inline">C_2</math> and <math display="inline">R_2</math> is the resolvent of  <math display="inline">C_2</math> and <math display="inline">C_3</math>.<ref name="invres">{{Cite book |last1=Nienhuys-Cheng |first1=Shan-hwei |title=Foundations of inductive logic programming |last2=Wolf |first2=Ronald de |date=1997 |publisher=Springer |isbn=978-3-540-62927-6 |series=Lecture notes in computer science Lecture notes in artificial intelligence |location=Berlin Heidelberg |page=197}}</ref>

Inverse resolution was first introduced by [[Stephen Muggleton]] and Wray Buntine in 1988 for use in the inductive logic programming system Cigol.<ref name="Proceedings of the 5th Internationa"/> By 1993, this spawned a surge of research into inverse resolution operators and their properties.<ref name="invres" />

=== Top-down search ===
The ILP systems Progol,<ref name=":2" /> Hail <ref>{{cite book |last1=Ray |first1=O. |url= |title=Proceedings of the 13th international conference on inductive logic programming |last2=Broda |first2=K. |last3=Russo |first3=A.M. |date=2003 |publisher=Springer |isbn=978-3-540-39917-9 |series=LNCS |volume=2835 |pages=311–328 |chapter=Hybrid abductive inductive learning |citeseerx=10.1.1.212.6602 |doi=10.1007/978-3-540-39917-9_21 |chapter-url=https://link.springer.com/chapter/10.1007/978-3-540-39917-9_21}}</ref> and Imparo <ref>{{cite book |last1=Kimber |first1=T. |title=Proceedings of the 10th international conference on logic programing and nonmonotonic reasoning |last2=Broda |first2=K. |last3=Russo |first3=A. |date=2009 |publisher=Springer |isbn=978-3-642-04238-6 |series=LNCS |volume=575 |pages=169–181 |chapter=Induction on failure: learning connected Horn theories |doi=10.1007/978-3-642-04238-6_16 |chapter-url=https://link.springer.com/chapter/10.1007/978-3-642-04238-6_16}}</ref> find a hypothesis {{mvar|H}} using the principle of the '''inverse entailment'''<ref name=":2" /> for theories {{mvar|B}}, {{mvar|E}}, {{mvar|H}}: <math>B \land H \models E \iff B \land \neg E \models \neg H</math>. First they construct an intermediate theory {{mvar|F}} called a bridge theory satisfying the conditions <math>B \land \neg E \models F</math> and <math>F \models \neg H</math>. Then as <math>H \models \neg F</math>, they generalize the negation of the bridge theory {{mvar|F}} with anti-entailment.<ref>{{cite journal |last1=Yamamoto |first1=Yoshitaka |last2=Inoue |first2=Katsumi |last3=Iwanuma |first3=Koji |year=2012 |title=Inverse subsumption for complete explanatory induction |url=https://link.springer.com/content/pdf/10.1007/s10994-011-5250-y.pdf |journal=Machine Learning |volume=86 |pages=115–139 |doi=10.1007/s10994-011-5250-y |s2cid=11347607}}</ref> However, the operation of anti-entailment is computationally more expensive since it is highly nondeterministic. Therefore, an alternative hypothesis search can be conducted using the inverse subsumption (anti-subsumption) operation instead, which is less non-deterministic than anti-entailment.

Questions of completeness of a hypothesis search procedure of specific inductive logic programming system arise. For example, the Progol hypothesis search procedure based on the inverse entailment inference rule is not complete by ''Yamamoto's example''.<ref>{{cite book |last=Yamamoto |first=Akihiro |url= |title=International Conference on Inductive Logic Programming |date=1997 |publisher=Springer |isbn=978-3-540-69587-5 |series=Lecture Notes in Computer Science |volume=1297 |location= |pages=296–308 |chapter=Which hypotheses can be found with inverse entailment? |citeseerx=10.1.1.54.2975 |doi=10.1007/3540635149_58 |chapter-url=https://link.springer.com/chapter/10.1007/3540635149_58}}</ref> On the other hand, Imparo is complete by both anti-entailment procedure <ref name="kimber2009induction">{{cite thesis |first=Timothy |last=Kimber |title=Learning definite and normal logic programs by induction on failure |date=2012 |type=PhD |publisher=Imperial College London |url=https://ethos.bl.uk/OrderDetails.do?uin=uk.bl.ethos.560694 |id=ethos 560694 |access-date=2022-10-21 |archive-date=2022-10-21 |archive-url=https://web.archive.org/web/20221021035457/https://ethos.bl.uk/OrderDetails.do?uin=uk.bl.ethos.560694 |url-status=dead}}</ref> and its extended inverse subsumption <ref>{{cite arXiv |eprint=1407.3836 |class=cs.AI |first=David |last=Toth |title=Imparo is complete by inverse subsumption |date=2014}}</ref> procedure.

=== Metainterpretive learning ===
Rather than explicitly searching the hypothesis graph, metainterpretive or ''meta-level'' systems encode the inductive logic programming program as a meta-level logic program which is then solved to obtain an optimal hypothesis. Formalisms used to express the problem specification include [[Prolog]] and [[answer set programming]], with existing Prolog systems and answer set solvers used for solving the constraints.<ref name=":0">{{Cite journal |last1=Cropper |first1=Andrew |last2=Dumančić |first2=Sebastijan |date=2022-06-15 |title=Inductive Logic Programming At 30: A New Introduction |journal=Journal of Artificial Intelligence Research |volume=74 |page=795 |doi=10.1613/jair.1.13507 |issn=1076-9757|doi-access=free |arxiv=2008.07912 }}</ref>

And example of a Prolog-based system is [[Metagol]], which is based on a [[Meta-interpreters in Prolog|meta-interpreter in Prolog]], while ASPAL and ILASP are based on an encoding of the inductive logic programming problem in answer set programming.<ref name=":0" />

=== Evolutionary learning ===
[[Evolutionary algorithm]]s in ILP use a population-based approach to evolve hypotheses, refining them through selection, crossover, and mutation. Methods like [[EvoLearner]] have been shown to outperform traditional approaches on structured machine learning benchmarks. <ref>{{cite conference |last=Heindorf |first=Stefan |last2=Blübaum |first2=Lukas |last3=Düsterhus |first3= Nick |last4=Werner |first4=Till |last5=Golani |first5=Varun Nandkumar |last6=Demir |first6=Caglar |last7=Ngonga Ngomo |first7=Axel-Cyrille |title=EvoLearner: Learning Description Logics with Evolutionary Algorithms |url=https://arxiv.org/abs/2111.04879 |conference=WWW |date=2022}}</ref>

== List of implementations ==
* [http://www.cs.bris.ac.uk/Research/MachineLearning/1BC/ 1BC and 1BC2: first-order naive Bayesian classifiers:]
* [http://dtai.cs.kuleuven.be/ACE/ ACE (A Combined Engine)]
* [http://web.comlab.ox.ac.uk/oucl/research/areas/machlearn/Aleph/ Aleph]
* [http://www.ahlgren.info/research/atom/ Atom] {{Webarchive|url=https://web.archive.org/web/20140326152728/http://www.ahlgren.info/research/atom |date=2014-03-26 }}
* [http://dtai.cs.kuleuven.be/claudien/ Claudien]{{Dead link|date=October 2022 |bot=InternetArchiveBot |fix-attempted=yes }}
* [http://dl-learner.org DL-Learner] {{Webarchive|url=https://web.archive.org/web/20190815184411/http://dl-learner.org/ |date=2019-08-15 }}
* [http://dtai.cs.kuleuven.be/dmax/ DMax]
* [https://spike-imperial.github.io/FastLAS/ FastLAS (Fast Learning from Answer Sets)]
* [[First Order Inductive Learner|FOIL (First Order Inductive Learner)]]
* [[Golem (ILP)|Golem]]
* [http://www.ilasp.com/ ILASP (Inductive Learning of Answer Set Programs)]
* Imparo<ref name="kimber2009induction" />
* [http://lacam.di.uniba.it:8000/systems/inthelex/ Inthelex (INcremental THEory Learner from EXamples)] {{Webarchive|url=https://web.archive.org/web/20111128110512/http://lacam.di.uniba.it:8000/systems/inthelex/ |date=2011-11-28 }}
* [https://web.archive.org/web/20020516195248/http://cs.anu.edu.au/people/Eric.McCreath/lime.html Lime]
* [http://github.com/metagol/metagol Metagol]
* [https://archive.today/20130219215544/http://libra.msra.cn/Publication/3392493/mio-user-s-manual Mio]
* MIS (Model Inference System) by Ehud Shapiro
* [https://github.com/dice-group/Ontolearn Ontolearn]
* [https://github.com/logic-and-learning-lab/Popper Popper]
* [[PROGOL]]
* [http://ida.felk.cvut.cz/zelezny/rsd/index.htm RSD]
* Warmr (now included in ACE)
* [http://ilp.doc.ic.ac.uk/ProGolem/ ProGolem] <ref>{{cite book |last1=Muggleton |first1=Stephen |last2=Santos |first2=Jose |last3=Tamaddoni-Nezhad |first3=Alireza |chapter=ProGolem: a system based on relative minimal generalization |citeseerx=10.1.1.297.7992 |chapter-url=https://link.springer.com/chapter/10.1007/978-3-642-13840-9_13 |title=International Conference on Inductive Logic Programming  |publisher=Springer |date=2009 |isbn=978-3-642-13840-9 |pages=131–148 |doi=10.1007/978-3-642-13840-9_13 |url=}}</ref><ref>{{cite journal|last1=Santos|first1=Jose|last2=Nassif|first2=Houssam|last3=Page|first3=David|last4=Muggleton|first4=Stephen|last5=Sternberg|first5=Mike|title=Automated identification of features of protein-ligand interactions using Inductive Logic Programming: a hexose binding case study|journal=BMC Bioinformatics|date=2012|volume=13|page=162|doi=10.1186/1471-2105-13-162|pmid=22783946|pmc=3458898 |doi-access=free }}</ref>

== Probabilistic inductive logic programming ==
Probabilistic inductive logic programming adapts the setting of inductive logic programming to learning [[Probabilistic logic programming|probabilistic logic programs]]. It can be considered as a form of [[statistical relational learning]] within the formalism of probabilistic logic programming.<ref>{{Citation |last1=De Raedt |first1=Luc |title=Probabilistic Inductive Logic Programming |date=2008 |url=http://dx.doi.org/10.1007/978-3-540-78652-8_1 |pages=1–27 |access-date=2023-12-09 |place=Berlin, Heidelberg |publisher=Springer Berlin Heidelberg |isbn=978-3-540-78651-1 |last2=Kersting |first2=Kristian|doi=10.1007/978-3-540-78652-8_1 }}</ref><ref name="pilp">{{Cite journal |last1=Riguzzi |first1=Fabrizio |last2=Bellodi |first2=Elena |last3=Zese |first3=Riccardo |date=2014-09-18 |title=A History of Probabilistic Inductive Logic Programming |journal=Frontiers in Robotics and AI |volume=1 |doi=10.3389/frobt.2014.00006 |issn=2296-9144 |doi-access=free }}</ref>

Given

# background knowledge as a probabilistic logic program {{Mvar|B}}, and
# a set of positive and negative examples <math display="inline">E^{+}</math> and <math display="inline">E^{-}</math>

the goal of probabilistic inductive logic programming is to find a probabilistic logic program <math display="inline">H</math> such that the probability of positive examples according to <math display="inline">{H \cup B}</math> is maximized and the probability of negative examples is minimized.<ref name="pilp" />

This problem has two variants: parameter learning and structure learning. In the former, one is given the structure (the clauses) of {{Mvar|H}} and the goal is to infer the probabilities annotations of the given clauses, while in the latter the goal is to infer both the structure and the probability parameters of {{Mvar|H}}. Just as in classical inductive logic programming, the examples can be given as examples or as (partial) interpretations.<ref name="pilp" />

=== Parameter Learning ===
Parameter learning for languages following the distribution semantics has been performed by using an [[expectation-maximisation algorithm]] or by [[gradient descent]].
An expectation-maximisation algorithm consists of a cycle in which the steps of expectation and maximization are repeatedly performed. In the expectation step, the distribution of the hidden variables is computed according to the current values of the probability parameters, while in the maximisation step, the new values of the parameters are computed. 
Gradient descent methods compute the gradient of the target function and iteratively modify the parameters moving in the direction of the gradient.<ref name="pilp" />

=== Structure Learning ===
Structure learning was pioneered by  [[Daphne Koller]] and Avi Pfeffer in 1997,<ref>{{Cite conference |last1=Koller |first1=Daphne |last2=Pfeffer |first2=Avi |date=August 1997 |title=Learning probabilities for noisy first-order rules |url=http://www.robotics.stanford.edu/~koller/Papers/Koller+Pfeffer:IJCAI97.pdf |conference=[[IJCAI]]}}</ref> where the authors learn the structure of [[First-order logic|first-order]] rules with associated probabilistic uncertainty parameters. Their approach involves generating the underlying [[graphical model]] in a preliminary step and then applying expectation-maximisation.<ref name="pilp" />

In 2008, [[Luc De Raedt|De Raedt]] et al. presented an algorithm for performing [[theory compression]] on [[ProbLog]] programs, where theory compression refers to a process of removing as many clauses as possible from the theory in order to maximize the probability of a given set of positive and negative examples. No new clause can be added to the theory.<ref name="pilp" /><ref>{{Cite journal |last1=De Raedt |first1=L. |last2=Kersting |first2=K. |last3=Kimmig |first3=A. |last4=Revoredo |first4=K. |last5=Toivonen |first5=H. |date=March 2008 |title=Compressing probabilistic Prolog programs |url=http://link.springer.com/10.1007/s10994-007-5030-x |journal=Machine Learning |language=en |volume=70 |issue=2–3 |pages=151–168 |doi=10.1007/s10994-007-5030-x |issn=0885-6125}}</ref>

In the same year, Meert, W. et al. introduced a method for learning parameters and structure of [[Ground term|ground]] probabilistic logic programs by considering the [[Bayesian network]]s equivalent to them and applying techniques for learning Bayesian networks.<ref>{{Citation |last1=Blockeel |first1=Hendrik |title=Towards Learning Non-recursive LPADs by Transforming Them into Bayesian Networks |url=http://dx.doi.org/10.1007/978-3-540-73847-3_16 |work=Inductive Logic Programming |pages=94–108 |access-date=2023-12-09 |place=Berlin, Heidelberg |publisher=Springer Berlin Heidelberg |isbn=978-3-540-73846-6 |last2=Meert |first2=Wannes|series=Lecture Notes in Computer Science |date=2007 |volume=4455 |doi=10.1007/978-3-540-73847-3_16 }}</ref><ref name="pilp" />

ProbFOIL, introduced by De Raedt and Ingo Thon in 2010, combined the inductive logic programming system [[First-order inductive learner|FOIL]]  with [[ProbLog]]. Logical rules are learned from probabilistic data in the sense that both the examples themselves and their classifications can be probabilistic. The set of rules has to allow one to predict the probability of the examples from their description. In this setting, the parameters (the probability values) are fixed and the structure has to be learned.<ref>{{Citation |last1=De Raedt |first1=Luc |title=Probabilistic Rule Learning |date=2011 |url=http://link.springer.com/10.1007/978-3-642-21295-6_9 |work=Inductive Logic Programming |volume=6489 |pages=47–58 |editor-last=Frasconi |editor-first=Paolo |access-date=2023-12-09 |place=Berlin, Heidelberg |publisher=Springer Berlin Heidelberg |doi=10.1007/978-3-642-21295-6_9 |isbn=978-3-642-21294-9 |last2=Thon |first2=Ingo |s2cid=11727522 |editor2-last=Lisi |editor2-first=Francesca A.}}</ref><ref name="pilp" />

In 2011, Elena Bellodi and Fabrizio Riguzzi introduced SLIPCASE, which performs a beam search among probabilistic logic programs by iteratively refining probabilistic theories and optimizing the parameters of each theory using expectation-maximisation.<ref>{{Citation |last1=Bellodi |first1=Elena |title=Learning the Structure of Probabilistic Logic Programs |date=2012 |url=http://dx.doi.org/10.1007/978-3-642-31951-8_10 |work=Inductive Logic Programming |pages=61–75 |access-date=2023-12-09 |place=Berlin, Heidelberg |publisher=Springer Berlin Heidelberg |isbn=978-3-642-31950-1 |last2=Riguzzi |first2=Fabrizio|doi=10.1007/978-3-642-31951-8_10 }}</ref> 
Its extension SLIPCOVER, proposed in 2014, uses bottom clauses generated as in [[Progol]] to guide the refinement process, thus reducing the number of revisions and exploring the search space more effectively. Moreover, SLIPCOVER separates the search for promising clauses from that of the theory: the space of clauses is explored with a [[beam search]], while the space of theories is searched [[Greedy search|greedily]].<ref>{{Cite journal |last1=Bellodi |first1=Elena |last2=Riguzzi |first2=Fabrizio |date=2014-01-15 |title=Structure learning of probabilistic logic programs by searching the clause space |url=http://dx.doi.org/10.1017/s1471068413000689 |journal=Theory and Practice of Logic Programming |volume=15 |issue=2 |pages=169–212 |doi=10.1017/s1471068413000689 |arxiv=1309.2080 |s2cid=17669522 |issn=1471-0684}}</ref><ref name="pilp" />

==See also==
* [[Commonsense reasoning]]
* [[Formal concept analysis]]
* [[Inductive reasoning]]
* [[Inductive programming]]
* [[Inductive probability]]
* [[Statistical relational learning]]
* [[Version space learning]]

== References ==
{{reflist}}
{{Free-content attribution|author=Fabrizio Riguzzi, Elena Bellodi and Riccardo Zese |date=2014-09-18 |title=A History of Probabilistic Inductive Logic Programming |documentURL=http://journal.frontiersin.org/article/10.3389/frobt.2014.00006/ |publisher=[[Frontiers Media]]|license=CC-BY 4.0|license statement URL = http://creativecommons.org/licenses/by/4.0/}}

==Further reading==
{{Refbegin}}
* {{Cite journal | last1 = Muggleton | first1 = S. | last2 = De Raedt | doi = 10.1016/0743-1066(94)90035-3 | first2 = L. | title = Inductive Logic Programming: Theory and methods | journal = The Journal of Logic Programming | volume = 19–20 | pages = 629–679 | year = 1994 | doi-access = free }}
* {{cite book | first1 = N. | last1 = Lavrac | first2 = S. | last2 = Dzeroski | title = Inductive Logic Programming: Techniques and Applications | publisher = Ellis Horwood | location = New York | year = 1994 | isbn = 978-0-13-457870-5 | url = http://www-ai.ijs.si/SasoDzeroski/ILPBook/ | access-date = 2004-09-22 | archive-url = https://web.archive.org/web/20040906084947/http://www-ai.ijs.si/SasoDzeroski/ILPBook/ | archive-date = 2004-09-06 | url-status = dead }}
* Visual example of inducing the grandparenthood relation by the [[#Implementations|Atom system]]. http://john-ahlgren.blogspot.com/2014/03/inductive-reasoning-visualized.html {{Webarchive|url=https://web.archive.org/web/20140326101408/http://john-ahlgren.blogspot.com/2014/03/inductive-reasoning-visualized.html |date=2014-03-26 }}
{{Refend}}

{{Programming paradigms navbox}}

[[Category:Inductive logic programming| ]]