Simon Brooke, Common Knowledge Ltd.
Chris Jackson, CJ Software Engineering.
This paper describes the practice of Elicitation by Exception; briefly outlines its historical and philosophical roots; describes the variants in current practice, with attention to The KnAcq; outlines the potential for further development; and discusses Common Knowledge's current work in developing the technique.
Common Knowledge Ltd
St Leonard's House
St Leonardgate
Lancaster
LA 11 NN
(tel) 0524 381334
(fax) 0524 845998
The term ' Elicitation by Exception' was introduced in [Mott & Brooke 87 ]1 to describe a technique, then novel, of eliciting knowledge from an expert by recursively seeking expectations about situations. The technique uses the structure of an ' exception graph' to both drive and record an knowledge elicitation interview. The technique was sketched briefly in that paper; as no fuller description is currently widely available, we will give a somewhat expanded exposition below.
Elicition by Exception, as a technique, forms the basis of The KnAcqTM knowledge acquisition methodology developed by Common Knowledge Ltd, and is supported by the KnAcqToolsTM CASE tool. However, the power of the technique has led to its adoption by other teams[2], and this is a development which we encourage.
The following description of the elicitation process assumes the use of the dTree syntax[3]. In principle the practice of elicitation by exception can adapted to many different exception graph syntax's, and the differences between these, together with their respective merits, are discussed in a later section of the present paper.
As briefly described in previous papers, and more fully in [Brooke, Duke & Mott 91]4, the process of eliciting knowledge using elicitation by exception is firstly to look at the normal case, and then to elicit a structure of exception to this case. Thus the first task of the knowledge engineer is to agree with the expert a ' key feature' - a predicate which expresses,as a truth function, the decision which the expert is seeking to make.
Having established the key feature, the next task is to establish the default value - whether it should be assured true or false in the absence of further information. The expert should be asked this, and it is our view that the expert's response should be simply recorded.
Once the details of the key feature are established, a dTree diagram started. This initially shows a single node, labelled with the name of the key feature, with a plus or minus sign representing the 'advice' of the node. The 'advice' of a root node is always the same as the default value of its feature.
So now the elicitor must ask the expert:
Given that the (key feature) is normally (default value), what is the first thing you would make you think it was (-default value)?
An example might be:
Given that most domains are not suitable for the application of Expert Systems, what is the first thing you would look for to make you believe that a particular domain was suitable?
The expert will reply by naming a predicate (or else the elicitor must persist until the expert does so). The following might be an example:
There is some practitioner who regularly makes good decision in the domain.
The elicitor now asks, given this new predicate is true, would this lead the expert to beleive, normally, that the key feature was true. In this case the expert might respond 'yes'. This answer provides the ' advice' for a new node in the diagram: the new feature must be added, and to do this its default value must be learned (would you normally expert that ...?).
The elicitor must also find out whether this is something which can be obtained other than by inference : e.g.
In practice we go through the list of possible at the beginning of the session (when we say ' can you ask this' we also want you to think about...); during the interview we only ask:
is that something the user of the system could reasonably be asked?
and trust to the expert to remember that the information could come from other sources than the user. This allows us to record the new predicate as a 'feature'.
Obviously, we are phrasing the questions so as to produce an alternation of the advises on nodes as we descend any path. This reflects the Popperian [5] paradigm. Occasionally this neat pattern won't occur: for example, in the case discussed, when asked whether the existence of a practitioner would suggest that a domain was suitable, the expert might have replied 'no; not unless the practitioner was willing to co-operate':
{EMBED MSDraw \* mergeformat|}
Such a diagram is perfectly legitimate. Again, at the new node, the elicitor must ask:
Given that all of (features of nodes on path, not including the root) are true, you would expect (feature of root node) to be (advice of last node). What is the first thing you would look for which would make you think it was (not advice of last node)?
Or in the present case
Given that there is a good practitioner, who is willing to co-operate, you would think that the domain probably was suitable for Expert Systems. What is the first thing you would look for to make you believe that it was not suitable for Expert Systems?
To which the expert might reply:
Well, if the practitioner relied on special manual skills, then that would not be a suitable domain.
This process of seeking exceptions, and recording them, continue until the expert declares that are no further exceptions down a particular path:
No , If the practitioner used special manual skills, there's nothing that could make the domain suitable.
This allows a path to be closed. We now move one node back up the tree. and ask again:
Given that all of (features of nodes on path.) are true and none of (features of children of the current node) are true, you would expect (feature of root node) to be (advice of last node). What is the first thing you would look for which would make you think it was (not advice of last node)?
Thus the process continues. This procedure, if carried out exhaustively, and if the expert co - operates, must extract from the expert all the knowledge which the expert would use in answering the key question. It is the exhaustiveness of this questioning that gives rise to the claim that the knowledge elicited will be complete.
Having concluded the elicitation of the first dTree, by pursuing all paths until they close, we look to see whether all the features which we have recorded either can be 'asked' (evaluated without further inference), or already have a dTree elicited. If this is the case, elicitation is complete.
Otherwise, the first feature which cannot be asked is taken as the key question for a new dTree, and elicitation continues.
It has been widely claimed that knowledge elicitation forms the bottleneck in Expert System development. This is, in our opinion, very largely because techniques used in knowledge elicitation have been drawn primarily from psychological technique to explore conceptualisation; these techniques, while more or less effective in drawing out informal, descriptive material, do not readily yield material which is easy to translate into executable code.
By contrast, the overwhelming strength of Elicitation by Exception is the tremendous rapidity with which knowledge is drawn out of experts in the form of logical structures with a clear formal semantics, allowing straightforward generation of declarative code.
However, a further important consequence of the use of a formal technique supported by a clear logical structure is that we can make some significant claims about the quality of the knowledge elicited: namely, that given certain specifiable assumptions concerning the correctness of the answers given by the expert in interview, knowledge elicited using Elicitation by Exception within the context of The KnAcq methodology will be a correct representation of the expert's decision making knowledge within the specified domain at the time of the interview; that it will be a complete representation of that knowledge; and -- whether or not the expert responded honestly -- that it will be a consistent representation of the knowledge.
To summarise : where it is applicable, Elicitation by Exception, within the context of a suitable methodology, provides an extremely rapid means of eliciting high quality knowledge from experts. Where it is applicable, the knowledge elicitation bottleneck ceases to exist.
Elicitation by Exception is applicable in a very wide range of the sorts of domains in which Expert Systems are currently applied, namely:
Unfortunately, at least at its present state of development, the technique becomes progressively less elegant as the number of possible outcomes rises, and is not a general solution in, for example, scheduling systems, where the number of possible outcomes is very large or cannot be enumerated. Furthermore, the technique essentially required someone who can act as an expert: it proves in practice extremely difficult to apply the technique to oneself, and the interactive nature of the technique precludes its use in acquisition of knowledge directly from texts - although it is often satisfactory for one member of a team to study the text in depth, and then for another member to elicit from the first. Finally, in domains where no expert yet exists, it is clearly impracticable.
A final limitation of the technique is that it elicits only declarative , and not procedural knowledge. This limitation is not as damning as it seems, however. In the first place, declarative knowledge about what constitutes the completion of a procedure is generally equivalent to procedural knowledge. In the second case, experts can very frequently give a procedural statement with little cueing, whereas declarative knowledge is notoriously difficult to elicit in unstructured situations.
The logical structure used to record the process of Elicitation by Exception is, necessarily, an exception graph. However, there are a wide variety of potential semantics for exception graphs, and the semantics for the variant describes in [Mott & Brooke 87], the dTree, are constrained to the point of being spartan. This allowed us to give a very clear exposition of the underlying ideas.
Essentially an exception graph is a directed graph whose edges are always read 'unless' and whose nodes are labelled with two elements: a guard, and a colour - alternate the guard can be thought of as labelling the edge, and the colour as labelling the node. The node at the root may be unlabelled (no - one to our knowledge has done this yet, but it is the cleanest and most orthogonal solution ), labelled with colour only but no guard or, as in the dTree graph with the same general appearance as other nodes but a very ( and importantly ) different interpretation : the left hand side names the function represented by the graph, and the right hand side represents the default value of the function within the domain.
Note that we now generally prefer to use the term 'advice' for ' , 'colour', as being clearer; however , our interest here is in the syntax of the exception graph, rather than its semantics, and we will continue to use 'colour' as a more neutral term.
The guards used in [Mott & Brooke 87 ] are essentially propositions; others have used relations [6], both are subsumed by the more general notion of a predicate [7]. It is , of course, inherent in the nature of a guard that it be two valued: it must either permit, or deny, further search. Colour, in all currently published material, is also two valued: (Mott & Brooke 87 ) always uses {true , false} for colour, in keeping with the propositional nuture of the underlying logic used. Where our approach has been adopted elsewhere [ A1 Attar 91] they use {accept, reject } for colour; generally, they are using a multi - valued logic, but it is clear that they fail to appreciate the general point that colour may take any value which it is legal within the underlying logic for the knowledge item described by the graph to take 8.
Further, they use the term ' Exception Tree' to describe their graph, rather than the more general ' exception graph'; and in the published example they show a tree. However, as there is no attempt to recover explicatory material from the nodes (as, for example, in KnqTools ), there is in principle no reason why the more general, and more compact form should not have been adopted.
As stated above, the dTree as described in our earlier work is an exception graph with having the following properties:
This highly constrained syntax was initially adopted for clarity, we have continued to use it because has proved extremely effective in a wide range of knowledge elicitation settings for capturing knowledge in areas as disparate as interpeting Europe Community directives and monitoring process plant. It's very simplicity has proved beneficial, in that it reduces the technicality of knowledge elicitation to a minimum , allowing it to be taught readily and carried out by less skilled staff; and even in the absence of a CASE tool such as KnAcqTools to automate the process, it greatly eases the generation of rules from exception graphs.
Finally, and not insignificantly, the simplicity of the dTree syntax allows the left - right ordering of nodes (and consequently the questioning order) to be largely independent of the declarative reading; consequently , question ordering can be turned without re-interviewing the expert.
However, the real world contains many things which are at best awkwardly represented in a language which allows only propositions. An example is shown in figure 2 of (Mott & Brooke 87)- a proposition "Husbands contributions OK." It's clear that this could be more elegantly rendered in a language which permitted functions as ( ContributionsOK ( HusbandOf ( subject))).
Equally, many things in the real world are not easily represented in a two valued logic. for example for some purposes it is convenient to characterise temperature as one (cold , cool , mild,warm, hot, scorching ): to represent this as a set of propositions:
......................
is clumsy and awkward, but not impracticable. However, for other purposes it is convenient to represent temperature as a real quantity in the range -273.15 <= x <= infinity; to represent this as a set of propositions would require infinitely many propositions , and it will be inconvenient to represent these in a machine with only finite store.
Finally , there is the matter of classification , and reasoning about types. While it is true (and, indeed, trivial ) that anything which can be represented in a frame basic logic - as used by object systems - can equally be represented in a set of rules , it is often inconvenient, and may be unclear , or computationally inefficient , to do so.
Clearly the set of circumstances which will lead us to believe that a bird can fly are different to those which lead us to believe that a person can fly, and while we can incorporate the predicates 'x is a person' and 'x is a bird' into an exception graph representing 'x can fly' it seems more natural to attach separate exception graph for 'x can fly' to classes representing birds and people respectively.
All the extensions sketched above are of course feasible, and indeed have already been demonstrated by Comman Knowledge Ltd in propotype software. All , however, have implications for the knowledge elicitation task, and may increase the burden on the elicitor, once again increasing the skill level required. Our objective as methodology designers is to achieve an optimum compromise between simplicity and representational power.The compromise represented by The KnAcq in its present incarnation is biased towards simplicity, while allowing rapid development of many of the types of knowledge based system in current use.
Our objective, where possible, will be to extend the methodology by increasing the power of the supporting toolkit rather than by increasing the complexity of the interview technique; for example we are experimenting with two separate technique for semi - automatic induction of class inheritance lattices from the interview material, and hope to be able to present the outcome of this work in 1992.
However , it is our current intention to relax the limitations we impose within the KnAcq methodology only slowly, recommending only those extensions to the dTree formalism which we find confer benefits in actual practice. The benifits of the current, very sparse technique are that it is very easy to teach, very easy to learn, very easy to carry out, and very reliable in use.
Furthermore, the simplicity of the underlying allows for a very simple rule generation algorithm; by contrast, rule generating algorithms (e.g.) multi- valued exception graphs which preserve the guarantee of consistency required to be substantially more sophisticated.
Elicitation by Exception is now a mature and well proven technique for the elicitation of knowledge, substantially faster and more effective than earlier, psychologically based techniques. Within the context of an appropriate, formally based knowledge elicitation methodology such as The nAcq, it offers guarantees of quality that no other technique can match. However, there is still potential for further development of the technique, and Common Knowledge are pushing ahead with this development.
1 Mott, P & Brooke, S: A Graphical Inference Mechanism : in Expert Systems iv, 2, may 87
2 see for example al Attar, A: Rule Induction from Myth to Methodology: in Research & Development in Expert Systems VIII, Cambridge University Press, 1991
3 Mott & Brooke op cit
4 Brooke, S, Duke, P & Mott, P: The KnAcq Training Manual, Common Knowledge Ltd 1991
5 Popper, K : Conjectures and refutations: Routledge & Kegan Paul, 1963
6 e.g. al Attar op cit.
7 Exception Graphs with predicates as guards were first seen in Mott, P: dTree with Objects: technical seminar paper for Alvey DHSS Demonstrator Project 1986: unpublished.
8 p 93, top of third, para: "An exception tree is a knowledge representation for capturing a decision making task involving two outcomes." our emphasis.
give me feedback on this page // show previous feedback on this page