From joe@sunia.u-strasbg.fr Sat Jun 1 08:27:04 1996 Received: from lucy.cs.wisc.edu (lucy.cs.wisc.edu [128.105.2.11]) by sea.cs.wisc.edu (8.6.12/8.6.12) with ESMTP id IAA04141 for ; Sat, 1 Jun 1996 08:26:56 -0500 Received: from TELNET-1.SRV.CS.CMU.EDU (TELNET-1.SRV.CS.CMU.EDU [128.2.254.108]) by lucy.cs.wisc.edu (8.6.12/8.6.12) with SMTP id IAA22689 for ; Sat, 1 Jun 1996 08:26:54 -0500 Received: from TELNET-1.SRV.CS.CMU.EDU by telnet-1.srv.cs.CMU.EDU id aa13147; 30 May 96 23:51:32 EDT Received: from DST.BOLTZ.CS.CMU.EDU by TELNET-1.SRV.CS.CMU.EDU id aa13145; 30 May 96 23:41:01 EDT Received: from DST.BOLTZ.CS.CMU.EDU by DST.BOLTZ.CS.CMU.EDU id aa14660; 30 May 96 23:40:15 EDT Received: from EDRC.CMU.EDU by B.GP.CS.CMU.EDU id aa29956; 30 May 96 7:46:37 EDT Received: from isis.u-strasbg.fr by EDRC.CMU.EDU id aa25892; 30 May 96 7:46:07 EDT Received: from sunia.diane (sunia.u-strasbg.fr [130.79.6.23]) by isis.u-strasbg.fr (8.6.11/8.6.9) with ESMTP id NAA26796 for ; Thu, 30 May 1996 13:44:48 +0200 Received: by sunia.diane (SMI-8.6/SMI-SVR4) id NAA19639; Thu, 30 May 1996 13:44:45 +0200 Date: Thu, 30 May 1996 13:44:45 +0200 From: Prof invite Message-Id: <199605301144.NAA19639@sunia.diane> To: connectionists@cs.cmu.edu Subject: Papers on Rule-Extraction from trained ANN The following papers are available via anonymous ftp: An Evaluation And Comparison Of Techniques For Extracting And Refining Rules From Artificial Neural Networks Robert Andrews* ** Russell Cable* Joachim Diederich* Shlomo Geva* Mostefa Golea* Ross Hayward* Chris Ho-Stewart* Alan B. Tickle* ** Neurocomputing Research Centre* School of Information Systems** Queensland University of Technology Brisbane Q 4001 Australia QUTNRC-96-01-04.ps.Z Abstract It is becoming increasingly apparent that without some form of explanation capability, the full potential of trained Artificial Neural Networks (ANNs) may not be realised. The primary purpose of this report is to survey techniques which have been developed to redress this situation. Specifically the survey focuses on mechanisms, procedures, and algorithms designed to insert knowledge into ANNs (knowledge initialisation), extract rules from trained ANNs (rule extraction), and utilise ANNs to refine existing rule bases (rule refinement). The survey also introduces a new taxanomy for classifying the various techniques, discusses their modus operandi, and delineates criteria for evaluating their efficacy. An additional facet of the report is a comparative evaluation of the performance of a set of techniques developed at the Neurocomputing Research Centre at QUT to extract knowledge from trained ANNs as a set of symbolic rules. Note: This is an extended version of: Andrews, R.; Diederich, J.; Tickle, A.B. A Survey and Critique of Techniques for Extracting Rules from Trained Artificial Neural Networks. KNOWLEDGE-BASED SYSTEMS 8 (1995) 6, 373-389. This version includes first empirical results and is distributed with permission of the editor and publisher. ******************************************************************************* DEDEC: Decision Detection by Rule Extraction from Neural Networks Alan B. Tickle* ** Marian Orlowski* ** Joachim Diederich* Neurocomputing Research Centre* School of Information Systems** Queensland University of Technology Brisbane Q 4001 Australia QUTNRC-95-01-03.ps.Z Abstract A clearly recognised impediment to the realisation of the full potential of Artificial Neural Networks is an inherent inability to explain in a comprehensible form (e.g. as a set of symbolic rules), the process by which an ANN arrived at a particular conclusion/decision/result. While a variety of techniques have already appeared to address this limitation, a substantial number of the more successful approaches are dependent on specialised ANN architectures. The DEDEC technique is a generic approach to rule extraction from trained ANNs which is designed to be applicable across a broad range of ANN architectures. The DEDEC technique is a generic approach to rule extraction from trained ANNs which is designed to be applicable across a broad range of ANN architectures. The basic motif adopted is to utilise the generalisation capability of a trained ANN to generate a set of examples from the problem domain which may include examples beyond the initial training set. These examples are then presented to a symbolic induction algorithm and the requisite rule set extracted. However an important innovation over other rule-extraction techniques of this ('pedagogical') type is that the DEDEC technique utilises information extracted from an analysis of the weight vectors of the trained ANN to rank the input variables (rule antecedents) in terms of their relative importance. This additional information is used to focus the search of the solution space on those examples from the problem domain which are deemed to be of most significance. The paper gives a detailed description of one possible implementation of the DEDEC technique and discusses results obtained on both a set of structured sample problems and 'real world' problems. ******************************************************************************* DEDEC: A Methodology For Extracting Rules From Trained Artificial Neural Networks Alan B. Tickle* ** Marian Orlowski Joachim Diederich* Neurocomputing Research Centre* School of Information Systems** Queensland University of Technology Brisbane Q 4001 Australia QUTNRC-96-01-05.ps.Z Abstract A recognised impediment to the more widespread utilisation of Artificial Neural Networks (ANNs) is the absence of a capability to explain, in a human comprehensible form, either the process by which a specific decision/result has been reached or, in general, the totality of knowledge embedded within the ANN. Currently, one of the most promising approaches to redressing this situation is to extract the knowledge embedded in the trained ANN as a set of symbolic rules. In this paper we describe the DEDEC methodology for rule-extraction which is applicable to a broad class of multilayer, feedforward ANNs trained by the 'back-propogation' method. Central to the DEDEC approach is the identification of the functional dependencies between the ANN inputs (i.e. the attribute values of the data) and the ANN outputs (e.g. the classification decision). However the key motif of the DEDEC methodology is the utilisation of information extracted from analysing the weight vectors in the trained ANN to focus the process of determining these functional dependencies. In addition, if required, DEDEC exploits the capability of a trained ANN to generalise beyond the data used in the ANN training phase. The paper illustrates one of a number of possible implementations of the DEDEC methodology, discusses results obtained on both a set of structured sample problems and a "real world" problem, and provides a comparison with other rule extraction techniques. ***************************************************************************** Artificial Intelligence Meets Artificial Insemination The Importance and Application of Symbolic Rule Extraction From Trained Artificial Neural Networks Robert Andrews* ** Joachim Diederich* Emanoil Pop* Alan B Tickle* ** Neurocomputing Research Centre* School of Information Systems** Queensland University of Technology Brisbane Q 4001 Australia QUTNRC-96-01-01.ps.Z Abstract In a recent article Andrews et al.[1995] describe a schema for classifying neural network rule extraction techniques as either decompositional, eclectic, or pedagogical. Decompositional techniques require knowledge of the neural network architecture and weights. Each hidden and output unit is interpreted as a Boolean rule with the antecedents being a set of incoming links whose summed weights guarantee to exceed the unit's bias regardless of the activations of the other incoming links. Pedagogical techniques on the other hand treat the underlyling neural network as a 'black box' using it to both classify examples and to generate examples which a symbolic algorithm then converts to rules. Eclectic techniques combine elements of the two basic categories. In this paper we describe some reasons why rule extraction is an important area of research. We then briefly describe three rule extraction algorithms, RULEX, DEDEC & RULENEG, these being representative of each of the abovementioned groups. We test these algorithms using two classification problems; the first being a laboratory benchmarking problem while the second is drawn from real life. For each problem, each of the rule extraction techniques previously described is applied to a trained neural network and the resulting rules presented. ******************************************************************************** Rule Extraction From CASCADE-2 Networks Ross Hayward Emanoil Pop Joachim Diederich Neurocomputing Research Centre Queensland University of Technology Brisbane Q 4001 Australia QUTNRC-96-01-02.ps.Z Abstract Rule extraction from feed forward neural networks is a topic that is gaining increasing interest. Any symbolic representation of how a network arrives at a particular decision is important not only for user acceptance, but also for rule refinement and network learning. This paper describes a new method of extracting rules that predict the firing of single units within a feed forward neural network. The extraction technique is applied to networks constructed by the Cascade 2 algorithm each of which solve a different benchmark problem. The hidden and output units within each of the networks are shown to represent distinct rules which govern the classification of patterns. Since a discrete rule set can be obtained for each of the units within the network, a logical mapping between input and output values can be achieved. ******************************************************************************** Feasibility of Incremental Learning in Biologically Plausible Networks James M. Hogan Joachim Diederich Neurocomputing Research Centre Queensland University of Technology Brisbane Q 4001 Australia QUTNRC-96-01-03.ps.Z Abstract The feasibility of incremental learning within a feed-forward network is examined under the constraint of biologically plausible connectivity. A randomly connected network (of physiologically plausible global connection probability) is considered under the assumption of a local connection probability which decays with distance between nodes. The representation of the function XOR is chosen as a test problem, and the likelihood of its recruitment is discussed with reference to the probability of occurrence of a subnetwork suitable for implementation of this function, assuming a uniform initial weight distribution. ****************************************************************************** These papers are available from ftp.fit.qut.edu.au cd to /pub/NRC/tr/ps From jan@uran.informatik.uni-bonn.de Sat Jun 1 08:27:06 1996 Received: from lucy.cs.wisc.edu (lucy.cs.wisc.edu [128.105.2.11]) by sea.cs.wisc.edu (8.6.12/8.6.12) with ESMTP id IAA04144 for ; Sat, 1 Jun 1996 08:27:01 -0500 Received: from TELNET-1.SRV.CS.CMU.EDU (TELNET-1.SRV.CS.CMU.EDU [128.2.254.108]) by lucy.cs.wisc.edu (8.6.12/8.6.12) with SMTP id IAA22691 for ; Sat, 1 Jun 1996 08:26:59 -0500 Received: from TELNET-1.SRV.CS.CMU.EDU by telnet-1.srv.cs.CMU.EDU id aa13585; 31 May 96 4:16:19 EDT Received: from DST.BOLTZ.CS.CMU.EDU by TELNET-1.SRV.CS.CMU.EDU id aa13583; 31 May 96 4:04:20 EDT Received: from DST.BOLTZ.CS.CMU.EDU by DST.BOLTZ.CS.CMU.EDU id aa15088; 31 May 96 4:04:06 EDT Received: from CS.CMU.EDU by B.GP.CS.CMU.EDU id aa16420; 31 May 96 4:03:06 EDT Received: from uran.informatik.uni-bonn.de by CS.CMU.EDU id aa16304; 31 May 96 4:02:18 EDT Received: from thalia.informatik.uni-bonn.de (jan@thalia.informatik.uni-bonn.de [131.220.10.27]) by uran.informatik.uni-bonn.de (8.7.4-ws4/8.7.1-ws3) with ESMTP id KAA29774 for ; Fri, 31 May 1996 10:02:13 +0200 (MET DST) From: Jan Puzicha Received: (jan@localhost) by thalia.informatik.uni-bonn.de (8.7.4-ws4/8.6.9) id KAA24140 for Connectionists@CS.CMU.EDU; Fri, 31 May 1996 10:02:08 +0200 (MET DST) Date: Fri, 31 May 1996 10:02:08 +0200 (MET DST) Message-Id: <199605310802.KAA24140@thalia.informatik.uni-bonn.de> To: Connectionists@cs.cmu.edu Subject: Re: Preprints and Abstracts available online X-Sun-Charset: US-ASCII This message has been posted to several lists. Sorry, if you receive multiple copies. The following six PREPRINTS are now available as abstracts and compressed postscript online via the WWW-Home-Page http://www-dbv.cs.uni-bonn.de/ of the |---------------------------------------------| |Computer Vision and Pattern Recognition Group| | of the University of Bonn, | | Prof. J. Buhmann, Germany. | |---------------------------------------------| This page also contains information about peoble and scientific projects (statistical mechanics for optimization, segmentation, stereo, compression, data clustering, vector quantization, multidimensional scaling, autonomous robotics, associative memories) Feel free to have a look at the new links to related sites, conferences and jounals. 1.) Thomas Hofmann and Joachim M. Buhmann, Inferring Hierarchical Clustering Structures by Deterministic Annealing. In: Proceedings of the International Conference on Knowledge Discovery and Data Mining (KDD), Portland, 1996. (to appear) 2.) Thomas Hofmann, Jan Puzicha and Joachim M. Buhmann, Unsupervised Segmentation of Textured Images by Pairwise Data Clustering. In: Proceedings of the International Conference on Image Processing (ICIP), Lausanne, 1996. (to appear) 3.) Jan Puzicha, Thomas Hofmann and Joachim M. Buhmann. Unsupervised Texture Segmentation on the Basis of Scale Space Features. Technical Report, University of Copenhagen, 1996. (to appear) 4.) Thorsten Frhlinghaus and J. Buhmann. Regularizing Phase-Based Stereo. in: International Conference on Pattern Recognition (ICPR), Vienna, 1996. (to appear) 5.) Thomas Hofmann and Joachim M. Buhmann. An Annealed Neural Gas Network for Robust Vector Quantization. International Conference on Artificial Neural Networks (ICANN), Bochum, 1996. (to appear) 6.) Jan Puzicha, Nils Goerke and Rolf Eckmiller. Optimal Trajectory Generation for an Industrial Robot by Markovian Networks. in: International Conference on Neural Information Processing (ICONIP), Hong Kong, 1996. (to appear) Greetings Jan Puzicha -------------------------------------------------------------------- Jan Puzicha | email: jan@uran.cs.uni-bonn.de Institute f. Informatics III | jan@cs.uni-bonn.de University of Bonn | WWW : http://www.cs.uni-bonn.de/~jan | Roemerstrasse 164 | Tel. : +49 228 550-383 D-53117 Bonn | Fax : +49 228 550-382 -------------------------------------------------------------------- ----- End Included Message ----- From svc@lacasa.com Sat Jun 1 18:51:11 1996 Received: from lucy.cs.wisc.edu (lucy.cs.wisc.edu [128.105.2.11]) by sea.cs.wisc.edu (8.6.12/8.6.12) with ESMTP id SAA09077 for ; Sat, 1 Jun 1996 18:51:06 -0500 Received: from TELNET-1.SRV.CS.CMU.EDU (TELNET-1.SRV.CS.CMU.EDU [128.2.254.108]) by lucy.cs.wisc.edu (8.6.12/8.6.12) with SMTP id SAA25644 for ; Sat, 1 Jun 1996 18:51:04 -0500 Received: from TELNET-1.SRV.CS.CMU.EDU by telnet-1.srv.cs.CMU.EDU id aa16442; 1 Jun 96 15:53:32 EDT Received: from DST.BOLTZ.CS.CMU.EDU by TELNET-1.SRV.CS.CMU.EDU id aa16439; 1 Jun 96 15:41:46 EDT Received: from DST.BOLTZ.CS.CMU.EDU by DST.BOLTZ.CS.CMU.EDU id aa16477; 1 Jun 96 15:41:08 EDT Received: from CS.CMU.EDU by B.GP.CS.CMU.EDU id aa07299; 1 Jun 96 15:09:56 EDT Received: from [206.206.99.100] by CS.CMU.EDU id aa26340; 1 Jun 96 15:09:43 EDT Received: from ives.lacasa.com (ives.lacasa.com [206.206.99.150]) by poincare.lacasa.com (8.6.12/8.6.9) with ESMTP id NAA22124 for ; Sat, 1 Jun 1996 13:09:40 -0600 Received: by ives.lacasa.com (SMI-8.6/SMI-SVR4) id NAA02583; Sat, 1 Jun 1996 13:11:02 -0600 Date: Sat, 1 Jun 1996 13:11:02 -0600 From: "Stephen V. Coggeshall" Message-Id: <199606011911.NAA02583@ives.lacasa.com> To: connectionists@cs.cmu.edu Subject: Message for distribution Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Content-MD5: nVH0dCsrryvWRqDLyRw/UQ== A small start-up company is looking for the right employees to work on a variety of problems in financial and other modeling. The Center for Adaptive Systems Applications (CASA, ~20 employees) has been in existence since June, 95, and is a result of a spin-off from the Los Alamos National Laboratory. The company is based in Los Alamos, NM. We are looking for post doc level researchers with strong skills in computer science (C, C++), math, and adaptive computing (neural nets). Experience in pattern recognition, neural nets, clustering algorithms, radial basis functions, etc. as well as large data set manipulation highly desired. Please send resumes to The Center for Adaptive Systems Applications c/o Frankie Gomez 901 18th Street Los Alamos, NM 87544 mfg@lacasa.com