From hochreit@informatik.tu-muenchen.de Mon Dec 30 12:15:41 1996 Received: from lucy.cs.wisc.edu (lucy.cs.wisc.edu [128.105.2.11]) by sea.cs.wisc.edu (8.6.12/8.6.12) with ESMTP id MAA18639 for ; Mon, 30 Dec 1996 12:15:23 -0600 Received: from TELNET-1.SRV.CS.CMU.EDU (TELNET-1.SRV.CS.CMU.EDU [128.2.254.108]) by lucy.cs.wisc.edu (8.7.6/8.7.3) with SMTP id MAA17790 for ; Mon, 30 Dec 1996 12:15:21 -0600 (CST) Received: from TELNET-1.SRV.CS.CMU.EDU by telnet-1.srv.cs.CMU.EDU id aa03703; 30 Dec 96 12:32:42 EST Received: from DST.BOLTZ.CS.CMU.EDU by TELNET-1.SRV.CS.CMU.EDU id aa03701; 30 Dec 96 12:06:20 EST Received: from DST.BOLTZ.CS.CMU.EDU by DST.BOLTZ.CS.CMU.EDU id aa05810; 30 Dec 96 12:05:53 EST Received: from RI.CMU.EDU by B.GP.CS.CMU.EDU id aa26098; 30 Dec 96 6:33:38 EST Received: from papa.informatik.tu-muenchen.de by RI.CMU.EDU id aa15195; 30 Dec 96 6:33:04 EST Received: from igel.informatik.tu-muenchen.de by papa.informatik.tu-muenchen.de id <49137>; Mon, 30 Dec 1996 12:32:52 +0100 From: Josef Hochreiter To: connectionists@cs.cmu.edu Subject: LSTM paper announcement Message-Id: <96Dec30.123252+0100met_dst.49137+394@papa.informatik.tu-muenchen.de> Date: Mon, 30 Dec 1996 12:32:48 +0100 LONG SHORT-TERM MEMORY Sepp Hochreiter, TUM Juergen Schmidhuber, IDSIA Substantially revised and extended Version 3.0 of TR FKI-207-95 (32 pages 130 KB; formerly 8 pages 50 KB), with numerous additional experiments and details. Abstract. Learning to store information over extended time intervals via recurrent backpropagation takes a very long time, mostly due to insufficient, decaying error back flow. We briefly review Hochreiter's 1991 analysis of this problem, then address it by introducing a novel, efficient method called "Long Short-Term Memory" (LSTM). LSTM can learn to bridge time lags in excess of 1000 steps by enforcing constant error flow through "constant error carrousels" (CECs) within special units. Multiplicative gate units learn to open and close access to CEC. LSTM's update complexity per time step is O(W), where W is the number of weights. In comparisons with RTRL, BPTT, Recurrent Cascade-Correlation, Elman nets, and Neural Sequence Chunking, LSTM leads to many more successful runs, and learns much faster. LSTM also solves complex long time lag tasks that have never been solved by previous recurrent net algorithms. LSTM works with local, distributed, real-valued, and noisy pattern representations. Recent spin-off papers: LSTM can solve hard long time lag problems. To appear in NIPS 9, MIT Press, Cambridge MA, 1997. Bridging long time lags by weight guessing and "Long Short-Term Memory". In F. L. Silva, J. C. Principe, L. B. Almeida, eds., Frontiers in Arti- ficial Intelligence and Applications, Volume 37, pages 65-72, IOS Press, Amsterdam, Netherlands, 1996. _______________________________________________________________________ WWW/FTP pointers: ftp://flop.informatik.tu-muenchen.de/pub/fki/fki-207-95rev.ps.gz ftp://ftp.idsia.ch/pub/juergen/lstm.ps.gz For additional recurrent net papers see our home pages. For instance, the original analysis of recurrent nets' error flow and long time lag problems is in Sepp's 1991 thesis (p. 19-21). http://www7.informatik.tu-muenchen.de/~hochreit/pub.html http://www.idsia.ch/~juergen/onlinepub.html Happy new year! Sepp & Juergen PS: Why don't you stop by at IDSIA and give a talk next time you are near Switzerland or Italy? From angelo@crc.ricoh.com Tue Dec 31 15:04:13 1996 Received: from lucy.cs.wisc.edu (lucy.cs.wisc.edu [128.105.2.11]) by sea.cs.wisc.edu (8.6.12/8.6.12) with ESMTP id PAA26556 for ; Tue, 31 Dec 1996 15:04:04 -0600 Received: from TELNET-1.SRV.CS.CMU.EDU (TELNET-1.SRV.CS.CMU.EDU [128.2.254.108]) by lucy.cs.wisc.edu (8.7.6/8.7.3) with SMTP id PAA27746 for ; Tue, 31 Dec 1996 15:04:03 -0600 (CST) Received: from TELNET-1.SRV.CS.CMU.EDU by telnet-1.srv.cs.CMU.EDU id aa04809; 31 Dec 96 14:23:39 EST Received: from DST.BOLTZ.CS.CMU.EDU by TELNET-1.SRV.CS.CMU.EDU id aa04807; 31 Dec 96 14:02:31 EST Received: from DST.BOLTZ.CS.CMU.EDU by DST.BOLTZ.CS.CMU.EDU id aa07129; 31 Dec 96 14:02:24 EST Received: from EDRC.CMU.EDU by B.GP.CS.CMU.EDU id aa13835; 31 Dec 96 14:00:19 EST Received: from gateway.crc.ricoh.com by EDRC.CMU.EDU id aa01420; 31 Dec 96 13:59:36 EST Received: by gateway.crc.ricoh.com; id AA08162; Tue, 31 Dec 96 10:59:04 PST Received: from congo.crc.ricoh.com(192.80.10.239) by gateway.crc.ricoh.com via smap (3.2) id xma008160; Tue, 31 Dec 96 10:58:54 -0800 Received: from jaguar.crc.ricoh.com (jaguar.crc.ricoh.com [192.80.10.215]) by congo.crc.ricoh.com (8.7.6/8.7.3) with SMTP id KAA08977 for ; Tue, 31 Dec 1996 10:59:50 -0800 (PST) Received: by jaguar.crc.ricoh.com (SMI-8.6/SMI-SVR4) id LAA02756; Tue, 31 Dec 1996 11:00:33 -0800 Date: Tue, 31 Dec 1996 11:00:33 -0800 From: "Michael Angelo (496-5735" MMDF-Warning: Parse error in original version of preceding line at DST.BOLTZ.CS.CMU.EDU Message-Id: <199612311900.LAA02756@jaguar.crc.ricoh.com> To: Connectionists@cs.cmu.edu Subject: Job Opening (in Menlo Park, CA.) The Ricoh California Research Center's Machine Learning and Perception Group invites exceptionally talented candidates to apply for a position as Research scientist in Information Technology Position description: * We seek applicants to join a small team of scientists and engineers exploring the use of machine learning and pattern recognition techniques in the general area of office information systems. Past and ongoing projects include o computer lipreading and speech-based interfaces o theory and application of neural network pruning methods o providing paper and electronic documents with novel functionality o theory for VLSI implementations of learning algorithms o novel human-machine interfaces o applications of the world-wide web * Ricoh CRC is a small center near Stanford University and other Silicon Valley landmarks; the atmosphere is collegial and exciting, and provides opportunities to expand Ricoh's products and services, travel nationally and internationally to professional conferences and presentations, publish in journals, and otherwise participate in the broader technical and professional community. Candidate requirements: * Ph.D. degree in Electrical Engineering, Computer Science or related field. (In exceptional cases, an M.S. degree with relevant work experience will suffice.) * Exceptionally strong C programming and Unix skills (experimental, not necessarily production), with experience in programming mathematical algorithms. C++, Java, Mathematica, MatLab and some parallel language are desirable. * Knowledge of neural networks, statistical and syntactic pattern recognition, image processing, handwriting recognition, natural language processing, and related topics is highly desirable. * Stong communication and organizational skills and the ability to learn quickly and to work both independently with minimal instruction and as part of a small team. Application deadline: * January 30, 1997 (hardcopy required -- see below). ---------------------------------------------------------------------------- RICOH California Research Center (RCRC): RCRC is a small research center in Menlo Park, CA, near the Stanford University campus and other Silicon Valley landmarks. The roughly 20 researchers focus on image compression and processing, pattern recognition, image and document analysis, artificial intelligence, machine learning, electronic service, and novel hardware for implementing computationally expensive algorithms. RCRC is a part of RICOH Corporation, the wholly owned subsidiary of RICOH Company, Ltd. in Japan. RICOH is a pioneer in facsimile, copiers, optical equipment, office automation products and more. Ricoh Corporation is an Equal Employment Opportunity Employer . ---------------------------------------------------------------------------- Please send any questions by e-mail to the address below, and type "Programming job" as your header line. Full applications (which must include a resume and the names and addresses of at least two people familiar with your work) should be sent by surface mail (no e-mail, ftp or html applications will be accepted) to: Dr. David G. Stork Chief Scientist RICOH California Research Center 2882 Sand Hill Road, Suite 115 Menlo Park CA 94025 stork@crc.ricoh.com ---------------------------------------------------------------------------- Web Version: http://www.crc.ricoh.com/jobs/MLPjob.html