RFC: remove DB_PRIVATE flag

[jgarzik]

Wallet backups and other DB examination are easily possible in a safe, atomic, transactional fashion...   if and only if DB_PRIVATE flag is removed.

Code:

--- a/db.cpp
+++ b/db.cpp
@@ -77,7 +77,6 @@ CDB::CDB(const char* pszFile, const char* pszMode) : pdb(NULL)
                              DB_INIT_MPOOL |
                              DB_INIT_TXN   |
                              DB_THREAD     |
-                             DB_PRIVATE    |
                              DB_RECOVER,
                              S_IRUSR | S_IWUSR);

What, if any, problems arise from doing this?  Obviously, this does not cover the non-db4 databases such as the block data file.

[satoshi]

Can you provide more details about what removing DB_PRIVATE does?

I can't remember if I had a specific reason for DB_PRIVATE, or if I just copied the flags from some example code.  Does removing DB_PRIVATE make it safe for other processes to open the database simultaneously?  That may be an improvement, depending what the side effects are.  Does it substantially reduce performance by making it have to write out every change immediately or do other coordination?  Are there additional locking or coordination files then?  What else changes?  You could test by timing an initial block download with and without DB_PRIVATE, preferably -connect-ing to a local machine so network isn't a factor.

Apparently, DB_PRIVATE doesn't do what you would hope it would do, which is prevent other processes from being able to open the database.  It still lets them, it just screws up if they do.  Another option, if there's a way, would be to make it lock the database files so they can't be accessed by other processes.

[jgarzik]

DB_PRIVATE enables a few optimizations by making the assumption that only one process will access the db4 database.  Notably, this flag enables db4 to use pthreads-style mutex locking rather than heavy, operating-system-provided flock and shared memory.  Ref: DB_ENV->open documentation.

The general motivation is that db4 databases can be safely accessed in parallel with bitcoin client, assuming (a) DB_PRIVATE is removed and (b) bitcoin properly uses db4 transactions.  db4 transactions may even be employed to wrap around non-db4 data such as blk0001.dat, if the code is properly architected.

[satoshi]

Trying it without the DB_PRIVATE flag in rev 153.  We need to keep an eye on what's different.

On Windows at least, it creates six __db.001 - __db.006 files with sizes from 24K to 4MB.  It doesn't delete them on exit, it just leaves them behind.

The docs say it uses memory mapped files.  I assume they have the same file permissions as the database files, so the same user access restrictions apply.

Tests on Windows private LAN download of 78500 blocks:
with DB_PRIVATE     20 minutes 51 seconds
without DB_PRIVATE   20 minutes 51 seconds

I wasn't expecting them to come out exactly the same.

[jgarzik]

Great, thanks!  Removing DB_PRIVATE should enable useful tools like gavin's bitcointools.