From owner-shore-bugs@cs.wisc.edu Mon Jun 4 01:41:15 2001 Received: (from majordom@localhost) by jeeves.cs.wisc.edu (8.9.2/8.9.2) id BAA17321; Mon, 4 Jun 2001 01:41:15 -0500 (CDT) Received: from lucy.cs.wisc.edu (lucy.cs.wisc.edu [128.105.6.15]) by jeeves.cs.wisc.edu (8.9.2/8.9.2) with ESMTP id BAA17317 for ; Mon, 4 Jun 2001 01:41:14 -0500 (CDT) Received: from yaman.serc.iisc.ernet.in (IDENT:root@[144.16.85.188]) by lucy.cs.wisc.edu (8.9.2/8.9.2) with ESMTP id BAA25072 for ; Mon, 4 Jun 2001 01:40:54 -0500 (CDT) Received: from localhost (srikanta@localhost) by yaman.serc.iisc.ernet.in (8.9.3/8.8.7) with ESMTP id MAA17165 for ; Mon, 4 Jun 2001 12:47:13 +0530 X-Authentication-Warning: yaman.serc.iisc.ernet.in: srikanta owned process doing -bs Date: Mon, 4 Jun 2001 12:47:12 +0530 (IST) From: B J Srikanta X-Sender: srikanta@yaman.serc.iisc.ernet.in To: shore-bugs@cs.wisc.edu Subject: Problem with long transactions. Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-shore-bugs@cs.wisc.edu Precedence: list I am trying to build a B-Tree index on a "big" database, but I keep running out of log-space. Probably I could increase the logsize in the options, but I tried the following - I set the lazy flag in the commit function call to true. Thereby no syncing log record is written - I don't mind restarting the B-Tree index building if it fails. But I was wondering if it is a good way of achieving my goal...? I know that this list has been quite inactive, but I hope there is somebody out there who still likes and works with Shore!! -srikanta From owner-shore-bugs@cs.wisc.edu Mon Jun 4 17:32:06 2001 Received: (from majordom@localhost) by jeeves.cs.wisc.edu (8.9.2/8.9.2) id RAA14162; Mon, 4 Jun 2001 17:32:06 -0500 (CDT) X-Authentication-Warning: jeeves.cs.wisc.edu: majordom set sender to owner-shore-bugs@cs.wisc.edu using -f Received: from orca.cs.wisc.edu (orca.cs.wisc.edu [128.105.165.26]) by jeeves.cs.wisc.edu (8.9.2/8.9.2) with ESMTP id RAA14158 for ; Mon, 4 Jun 2001 17:32:05 -0500 (CDT) Received: from cs.wisc.edu (localhost [127.0.0.1]) by orca.cs.wisc.edu (8.9.2/8.9.2) with ESMTP id RAA26627; Mon, 4 Jun 2001 17:32:00 -0500 (CDT) Message-Id: <200106042232.RAA26627@orca.cs.wisc.edu> To: B J Srikanta cc: shore-bugs@cs.wisc.edu Subject: Re: Problem with long transactions. From: Bolo In-reply-to: B J Srikanta's message of Mon, 04 Jun 2001 12:47:12 +0530. Date: Mon, 04 Jun 2001 17:32:00 -0500 Sender: owner-shore-bugs@cs.wisc.edu Precedence: list > I am trying to build a B-Tree index on a "big" database, but I keep > running out of log-space. Probably I could increase the logsize in the > options, but I tried the following - I set the lazy flag in the commit > function call to true. Thereby no syncing log record is written - I don't > mind restarting the B-Tree index building if it fails. One way would indeed be to increase the log space. If you want to do operations on a "live" (querying while building) system with full transactional properties, the log space is the cost you have to pay. I would have to spend more time than I have at the moment to look at this more closely, but the 'lazy' commit doesn't reduce the workload of a commit all that much. It definitely does *not* reduce the logging requirement at all; it just makes the SM a bit less aggressive about writing the log records that are created to disk. > But I was wondering if it is a good way of achieving my goal...? I know > that this list has been quite inactive, but I hope there is somebody out > there who still likes and works with Shore!! I think the most likely thing you want to do is to bulk load the index, rather than "building" it with inserts. When you bulk load the index, there really isn't any log cost, besides the internal space logging in the SM. The btree pages are all written directly, without log records being written. Upon commit, all the pages of the btree in the buffer pool are flushed to disk. Almost no overhead. If something happens (aka crash) before the index is built, or the transaction commit aborts for some reason, then nothing was created and you have to start the process from scratch again. To do this, look at the 'bulkld_index' method in the SM. If your data isn't clustered on the index attribute, you'll also need to sort the data. you can do that using the shore sort utility. You can also do something similar when loading files, to avoid the logging cost of record creations. Create the file as a 'temporary' file, and write records to it. Before committing the transaction, tell shore to change the type of the file from 'temporary' to 'normal'. Then, as above, all the pages for the file are flushed to disk, and you only "pay" for space allocation logging, not for the logging overhead of the individual record creation. There are also similar file types for adding records to existing files; checkout the documentation or the 'store_flag_t' enum in sm_base.h for more info. Or are you doing something completely different? Bolo ps: Do you know about the "interim snapshot" "releases" of Shore? Take a look on the WWW page for info. From owner-shore-bugs@cs.wisc.edu Mon Jun 4 20:04:08 2001 Received: (from majordom@localhost) by jeeves.cs.wisc.edu (8.9.2/8.9.2) id UAA15607; Mon, 4 Jun 2001 20:04:08 -0500 (CDT) Received: from lucy.cs.wisc.edu (lucy.cs.wisc.edu [128.105.6.15]) by jeeves.cs.wisc.edu (8.9.2/8.9.2) with ESMTP id UAA15603 for ; Mon, 4 Jun 2001 20:04:06 -0500 (CDT) Received: from yaman.serc.iisc.ernet.in (IDENT:root@[144.16.85.188]) by lucy.cs.wisc.edu (8.9.2/8.9.2) with ESMTP id UAA21684; Mon, 4 Jun 2001 20:04:03 -0500 (CDT) Received: from localhost (srikanta@localhost) by yaman.serc.iisc.ernet.in (8.9.3/8.8.7) with ESMTP id HAA06397; Tue, 5 Jun 2001 07:10:41 +0530 X-Authentication-Warning: yaman.serc.iisc.ernet.in: srikanta owned process doing -bs Date: Tue, 5 Jun 2001 07:10:41 +0530 (IST) From: B J Srikanta X-Sender: srikanta@yaman.serc.iisc.ernet.in To: Bolo cc: shore-bugs@cs.wisc.edu Subject: More logspace questions. In-Reply-To: <200106042232.RAA26627@orca.cs.wisc.edu> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-shore-bugs@cs.wisc.edu Precedence: list Hi, Earlier problem I had posted, in hindsight was badly posed ;-) and badly solved as well ... I agree that lazy flag does not have any effect on the log size requirement at all. It was a stupid mistake I made - hindsight is great right ;-) ? Thanks Bolo for correcting me on that... But I still have the following problem, if there are large number of transactions arriving at a real fast rate (as of now, my workload is a continous stream of transactions without any delays between them), the logspace is still a problem. (I cannot bulk load them as I am trying to support exactly that kind of transactions - bursty transactions which create number of objects in the repository !). I am not able to figure out how to force checkpointing so that log gets cleared - the log keeps growing even if I commit each transaction (uggh....the performance !) and I run out of logspace even then. Is there any way I can get this to work? I have been looking through the shore sm code to figure out if there is any way of "forcing" the checkpointing - when I hit the ceiling of logspace, or at more frequent intervals than it is done now. Do you have any solutions for me there...? -srikanta From owner-shore-bugs@cs.wisc.edu Mon Jun 4 21:47:24 2001 Received: (from majordom@localhost) by jeeves.cs.wisc.edu (8.9.2/8.9.2) id VAA16128; Mon, 4 Jun 2001 21:47:24 -0500 (CDT) X-Authentication-Warning: jeeves.cs.wisc.edu: majordom set sender to owner-shore-bugs@cs.wisc.edu using -f Received: from orca.cs.wisc.edu (orca.cs.wisc.edu [128.105.165.26]) by jeeves.cs.wisc.edu (8.9.2/8.9.2) with ESMTP id VAA16124 for ; Mon, 4 Jun 2001 21:47:23 -0500 (CDT) Received: from cs.wisc.edu (localhost [127.0.0.1]) by orca.cs.wisc.edu (8.9.2/8.9.2) with ESMTP id VAA27287; Mon, 4 Jun 2001 21:47:19 -0500 (CDT) Message-Id: <200106050247.VAA27287@orca.cs.wisc.edu> To: B J Srikanta cc: shore-bugs@cs.wisc.edu Subject: Re: More logspace questions. From: Bolo In-reply-to: B J Srikanta's message of Tue, 05 Jun 2001 07:10:41 +0530. Date: Mon, 04 Jun 2001 21:47:18 -0500 Sender: owner-shore-bugs@cs.wisc.edu Precedence: list > But I still have the following problem, if there are large number of > transactions arriving at a real fast rate (as of now, my workload is a > continous stream of transactions without any delays between them), the > logspace is still a problem. (I cannot bulk load them as I am trying to > support exactly that kind of transactions - bursty transactions which > create number of objects in the repository !). OK > I am not able to figure out how to force checkpointing so that log gets > cleared - the log keeps growing even if I commit each transaction > (uggh....the performance !) and I run out of logspace even then. > Is there any way I can get this to work? I have been looking through the > shore sm code to figure out if there is any way of "forcing" the > checkpointing - when I hit the ceiling of logspace, or at more frequent > intervals than it is done now. Do you have any solutions for me there...? Faster log checkpointing won't help you any. All that does is force the internal log buffer to be written to disk, and for buffer pool pages belonging to checkpointed logs to be written to disk (for older checkpoints). The log space still needs to exist for the transaction workload. I don't recall any fixed log related problems that match the symptoms you mention. I definitely don't know (as in don't exist) of any such outstanding problems in the current code base. And it has been extensively tested; With really huge workloads with lots of updates; with uptimes of of a month or so continuous; with concurrent (50) multiple request streams, with multiple threads per request stream. :) What your problem *does* sound like is that you have some (even one) long running transaction(s) sitting around. These are tying up the log and preventing the log system from reclaiming the log space they are using. So the log keeps growing and growing, and eventually you run out of space. A simple example assuming only begin/commits use any space in the log [ No nesting of transactions is implied by the indenting, just how many transactions are active simultaneously. ] Log Transactions Inuse in the system ------ -------------------------------------- 0 1 begin xct a 2 | begin xct b 3 | commit xct b 4 | begin xct c 5 | commit xct c 6 | begin xct d 7 | | begin xct e 8 | | commit xct e 9 | commit xct d 9 | 10 commit xct a 0 All those transactions (and whatever work that they do) create log records. However, the SM can't reclaim log space from the time of 'begin xct a' until there is a corresponding commit or abort of xct a. So the log keeps growing without bound. If one of the intermediate transactions becomes slightly longer-running, then the log space used can roll forward until the start of that transaction is encountered in the log: Log Transactions Inuse in the system ------ -------------------------------------- 0 1 begin xct a 2 | begin xct b 3 | commit xct b 4 | begin xct c 5 | commit xct c 6 | begin xct d 7 | | begin xct e 8 | | commit xct e 8 | | 9 | commit xct a 4 | 5 commit xct d 0 The solution to this (short of modifying Shore quite a bit) is to find those long running transactions and get rid of them. 1) The long xcts could be something as simple as a bug, or unintended. Just eliminate them. 2) Do something else instead of having the long running transactions. If the long running transactions are random and an effect of your workload, rather than some other issues, a more invasive solution is required: 3) Abort the oldest running transaction when the system returns a log space warning error (that may be something thing you don't have in your older version of shore). That will allow the log space to be reclaimed and shore will keep on running. 4) If the long running transactions are random and you don't want them to be aborted, then try to "chunk" the long running transactions into some smaller committed pieces, so that their log records don't stay around forever. In other words, have the long running transactions commit themselves every once in a while to free up log space. If you don't know what is causing that transactional behavior off-hand, you should be able to examine the outstanding transactions in your SM instance by the use of dump_xcts(). Bolo From owner-shore-bugs@cs.wisc.edu Mon Jun 18 07:08:19 2001 Received: (from majordom@localhost) by jeeves.cs.wisc.edu (8.9.2/8.9.2) id HAA23537; Mon, 18 Jun 2001 07:08:19 -0500 (CDT) Received: from lucy.cs.wisc.edu (lucy.cs.wisc.edu [128.105.6.15]) by jeeves.cs.wisc.edu (8.9.2/8.9.2) with ESMTP id HAA23533 for ; Mon, 18 Jun 2001 07:08:17 -0500 (CDT) Received: from yaman.serc.iisc.ernet.in (IDENT:root@[144.16.85.188]) by lucy.cs.wisc.edu (8.9.2/8.9.2) with ESMTP id HAA04457; Mon, 18 Jun 2001 07:07:45 -0500 (CDT) Received: from localhost (srikanta@localhost) by yaman.serc.iisc.ernet.in (8.9.3/8.8.7) with ESMTP id RAA05102; Mon, 18 Jun 2001 17:42:03 +0530 X-Authentication-Warning: yaman.serc.iisc.ernet.in: srikanta owned process doing -bs Date: Mon, 18 Jun 2001 17:42:03 +0530 (IST) From: B J Srikanta X-Sender: srikanta@yaman.serc.iisc.ernet.in To: shore-bugs@cs.wisc.edu cc: Bolo Subject: Shore - problems with logging options. Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-shore-bugs@cs.wisc.edu Precedence: list Hi, I am using the SDL/SVAS interface provided by the Shore through lambda-DB (the ODL/OQL compiler/optimizer from Leonidas Fegaras) to load the database with LOTS of data-objects . I had a few problems with disabling sm_logging feature in the shore configuration file. As I had mentioned in the previous posting, I have been facing problems with logspace overflow. I tried looking for if there are any transient transactions, but couldn't locate any. I tried doing some intermediate commits, i.e., after every 2000 objects put into the database I forced a commit and restarted the transaction. But even then, logspace failures persisted. (And obv. I can't afford to commit after each object, can I !?) Hence I tried playing with the sm_logging option in the .shoreconfig file. Obv. I cannot bring up the SVAS server without logging enabled, so I enable the logging, built the repository and then got out of the SVAS, disabled the logging and restarted the SVAS. Now, I have the Shore server running without logging, and I start loading the data - again through the SDL/SVAS layer (not my own VAS - unfortunately lambda-DB doesn't work with SM layer directly I have to go through SVAS/SDL layer :-( ). To test, I loaded about 3000 objects and everything was fine, I committed everything and shutdown the server cleanly. Now if I try restarting the SVAS server, the assertion fails at page.h:445 where the assertion is (surprise!?) "w_assert1(_pp->lsn1 == _pp->lsn2)"; which is evidently to check the log sequence numbers are in order. But why is it doing the logging, when I have disabled it? And if it is not doing the logging, why on the earth it should look for consistency in the log? It looks like both ways I am stuck, changing the lambda-DB code is not a very pleasant task, and I can't figure out how to get around these logging problems...:-( And another question I have is, Why isn't SDL/SVAS layer no longer shipped with Shore? Is it because no Shore user wants to use that layer, or is it that nobody is working on it? There are many features - even in the old Shore I am using - which are not exported to this layer. I did export few things - like Multidimensional indexing etc., but I would like to know if there is somebody working on it or not? All help is greatly valued....!! -- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Srikanta B Jagannath, Ph.D. Student, Database Systems Lab, SERC, IISc Phone: 3600653/54/5 (Extn.302) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ From owner-shore-bugs@cs.wisc.edu Mon Jun 18 17:28:58 2001 Received: (from majordom@localhost) by jeeves.cs.wisc.edu (8.9.2/8.9.2) id RAA15512; Mon, 18 Jun 2001 17:28:58 -0500 (CDT) X-Authentication-Warning: jeeves.cs.wisc.edu: majordom set sender to owner-shore-bugs@cs.wisc.edu using -f Received: from orca.cs.wisc.edu (orca.cs.wisc.edu [128.105.165.26]) by jeeves.cs.wisc.edu (8.9.2/8.9.2) with ESMTP id RAA15508 for ; Mon, 18 Jun 2001 17:28:57 -0500 (CDT) Received: from cs.wisc.edu (localhost [127.0.0.1]) by orca.cs.wisc.edu (8.9.2/8.9.2) with ESMTP id RAA18642; Mon, 18 Jun 2001 17:28:53 -0500 (CDT) Message-Id: <200106182228.RAA18642@orca.cs.wisc.edu> To: B J Srikanta cc: shore-bugs@cs.wisc.edu Subject: Re: Shore - problems with logging options. From: Bolo In-reply-to: B J Srikanta's message of Mon, 18 Jun 2001 17:42:03 +0530. Date: Mon, 18 Jun 2001 17:28:53 -0500 Sender: owner-shore-bugs@cs.wisc.edu Precedence: list > Now if I try restarting the SVAS server, the assertion fails at > page.h:445 where the assertion is (surprise!?) > "w_assert1(_pp->lsn1 == _pp->lsn2)"; > which is evidently to check the log sequence numbers are in > order. But why is it doing the logging, when I have disabled > it? And if it is not doing the logging, why on the earth it > should look for consistency in the log? There are two LSNs kept on each disk page to verify that, when the page was written to disk, that all of it actually made it to disk. For example, in the case of a power failure. One of the lsn's is at the start, and one is at the end of the page. If both are same, then all the disk blocks that make the page hit the disk. This isn't perfect; if the disk ordered the writes of the blocks of the page randomly instead of first..last or last..first then it could be a problem. You have to know the behavior of the hardware to assure correctness, or run small page sizes, or have one LSN stamp per block, or whatever. Anyway ... The assert is saying that the page wasn't updated on disk correctly, and that the two lsns don't match. It has nothing to do with the log itself. It is saying that page in the store is not consistent. > And another question I have is, Why isn't SDL/SVAS layer no longer shipped > with Shore? Is it because no Shore user wants to use that layer, or is it > that nobody is working on it? There are many features - even in the old > Shore I am using - which are not exported to this layer. I did export few > things - like Multidimensional indexing etc., but I would like to know if > there is somebody working on it or not? Shore funding ended and development on Shore for itself stopped quite some time back. At that point I don't believe that anyone was really using the upper layers; interest in that seems to have been something that occured later. Here at the UW, the Shore was used as the storage engine for another project, the Paradise parallel object-relational geo-spatial database system. Paradise (perhaps unfortunately) didn't use the Shore upper layers. However it did use Shore as a storage manager, and all the improvements and performance and correctness work was done as a result. Because of the interest shown by people using the Upper layers of shore, I have been considering to do some work to bring the upper layers in sync with the modern SM. However, any such work is not funded or supported by the University; it comes out of my free time and resources. I have several "back burner" projects (such as the 64 bit support) for Shore that are in such a status, and progress is slow on most of them. However, it is progress! Bolo -- Josef T. Burger From owner-shore-bugs@cs.wisc.edu Thu Jun 28 20:23:33 2001 Received: (from majordom@localhost) by jeeves.cs.wisc.edu (8.9.2/8.9.2) id UAA21602; Thu, 28 Jun 2001 20:23:32 -0500 (CDT) X-Authentication-Warning: jeeves.cs.wisc.edu: majordom set sender to owner-shore-bugs@cs.wisc.edu using -f Received: from lucy.cs.wisc.edu (lucy.cs.wisc.edu [128.105.6.15]) by jeeves.cs.wisc.edu (8.9.2/8.9.2) with ESMTP id UAA21596 for ; Thu, 28 Jun 2001 20:23:31 -0500 (CDT) Received: from korea.com ([211.208.192.221]) by lucy.cs.wisc.edu (8.9.2/8.9.2) with SMTP id UAA23527 for ; Thu, 28 Jun 2001 20:23:29 -0500 (CDT) From: To: shore-bugs@cs.wisc.edu Subject: pc°ü·ÃÁ¤º¸ÀÔ´Ï´Ù Date: ±Ý, 29 6 2001 10:23:57 Message-Id: <137.628513.697493@korea.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Sender: owner-shore-bugs@cs.wisc.edu Precedence: list ÃÖ½ÅÇüPC CPU¡¦¡¦¡¦.933 (ÀÎÅÚ) RAM¡¦¡¦¡¦128 (»ï¼ºÁ¤Ç°133) HDD¡¦¡¦¡¦20 G MONITER.17"(»ï¼º77E) ¸ðµÎ43¸¸¿ø °ü½ÉÀÖÀ¸¼¼¿ä http://comzon.atozpia.com/news/news.html À̸®·Î ¿À¼¼¿ä