Hi all,
I'm planning to use 389ds from SD-Card. So I would like to minimise write I/O.
It's absolutely fine if there's only a consistent state saved once a day. For example via some cronjob running ns-slapd db2archive (which I've done so far).
I do not need any database-logs to have even the last seconds in a disaster recovery.
Any hints to tune it for this use case?
There are files constantly written, AFAIK by the Berkeley DB: __db.00* log.000000000* (in /var/lib/dirsrv/slapd-kolab/db/)
Can I get rid of those files? Move them to a Ram Disk?
Regards hede
On 02/24/2014 01:16 PM, hede wrote:
Hi all,
I'm planning to use 389ds from SD-Card. So I would like to minimise write I/O.
It's absolutely fine if there's only a consistent state saved once a day. For example via some cronjob running ns-slapd db2archive (which I've done so far).
I do not need any database-logs to have even the last seconds in a disaster recovery.
Any hints to tune it for this use case?
There are files constantly written, AFAIK by the Berkeley DB: __db.00* log.000000000* (in /var/lib/dirsrv/slapd-kolab/db/)
Can I get rid of those files? Move them to a Ram Disk?
You can move them to a Ram Disk.
You can also completely disable durable transactions.
Regards hede -- 389 users mailing list 389-users@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/389-users
Am Mon, 24 Feb 2014 13:23:33 -0700 schrieb Rich Megginson rmeggins@redhat.com:
You can move them to a Ram Disk.
Found out those files is shared mem written to disk!? So moving it to some ram disk seems ridiculous.
You can also completely disable durable transactions.
Thank you. "durable transactions" is a keyword to find help via internet search.
I've found: https://access.redhat.com/site/documentation/en-US/Red_Hat_Directory_Server/...
So I added "nsslapd-db-durable-transactions: off" to my config and switched also "nsslapd-db-durable-transaction: off" in /etc/dirsrv/slapd-kolab/dse.ldif and checked those values after dirsrv-restart via ldapsearch, see [1].
Values are "off". But it seems the dirsrv is still writing to the files (__db.* / log.* in db-dir).
What am I doing wrong?
###### [1] $ ldapsearch -x -D "cn=directory manager" -W -p 389 -h 192.168.12.46 -b "cn=config" | grep durable Enter LDAP Password: nsslapd-db-durable-transactions: off nsslapd-db-durable-transaction: off
On 02/24/2014 01:51 PM, hede wrote:
Am Mon, 24 Feb 2014 13:23:33 -0700 schrieb Rich Megginson rmeggins@redhat.com:
You can move them to a Ram Disk.
Found out those files is shared mem written to disk!? So moving it to some ram disk seems ridiculous.
Why?
You can also completely disable durable transactions.
Thank you. "durable transactions" is a keyword to find help via internet search.
I've found: https://access.redhat.com/site/documentation/en-US/Red_Hat_Directory_Server/...
So I added "nsslapd-db-durable-transactions: off" to my config and switched also "nsslapd-db-durable-transaction: off" in /etc/dirsrv/slapd-kolab/dse.ldif and checked those values after dirsrv-restart via ldapsearch, see [1].
Values are "off". But it seems the dirsrv is still writing to the files (__db.* / log.* in db-dir).
What am I doing wrong?
I don't know, but you shouldn't have "..-transactions" and "..-transaction".
###### [1] $ ldapsearch -x -D "cn=directory manager" -W -p 389 -h 192.168.12.46 -b "cn=config" | grep durable Enter LDAP Password: nsslapd-db-durable-transactions: off nsslapd-db-durable-transaction: off -- 389 users mailing list 389-users@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/389-users
One thing to say first is that it sounds like you are subjecting the server to write operations (LDAP ops that change the stored data -- adds, deletes, modifies). If this isn't the case (you're expecting to not change the stored data much), then I'd suggest looking into why there are ongoing disk writes. I don't think that should be the case with a quiet server, except for the checkpoint record flushed to the log every so often, and possibly any OS flushing of dirty pages from the page pool to disk. You could use "strace" to get more insight into what is being written when and where.
Disabling durable transactions only prevents a disk write on every LDAP write operation. The writes will still be done, just later sometime. So disabling durable transactions won't I suspect solve your problem. If you want to write to the DIT but not have any disk writes, then you need to put the entire database in ramdisk.
Now, to get rid of the region and log files you would need to run the database in a non-transacted mode, or transacted but single-process. Neither mode is supported by the directory server, so you're stuck with the files and disk writes associated with updates to the DIT, unless you make major modifications to the server source code yourself.
If they're located in ramdisk, the OS is typically smart enough to not allocate the memory twice (can't speak to whether Linux is sufficiently smart but older OSes such as Solaris were).
I'm assuming that you're ok with having your database corrupted which is a distinct possibility if you move the region files into non-persistent storage. Problem I foresee is in determining when it has been corrupted. For that reason it may be appropriate to always assume the database is bad and rebuild it from a backup each time the server is started.
On 2/24/2014 1:51 PM, hede wrote:
Am Mon, 24 Feb 2014 13:23:33 -0700 schrieb Rich Megginson rmeggins@redhat.com:
You can move them to a Ram Disk.
Found out those files is shared mem written to disk!? So moving it to some ram disk seems ridiculous.
You can also completely disable durable transactions.
Thank you. "durable transactions" is a keyword to find help via internet search.
I've found: https://access.redhat.com/site/documentation/en-US/Red_Hat_Directory_Server/...
So I added "nsslapd-db-durable-transactions: off" to my config and switched also "nsslapd-db-durable-transaction: off" in /etc/dirsrv/slapd-kolab/dse.ldif and checked those values after dirsrv-restart via ldapsearch, see [1].
Values are "off". But it seems the dirsrv is still writing to the files (__db.* / log.* in db-dir).
What am I doing wrong?
###### [1] $ ldapsearch -x -D "cn=directory manager" -W -p 389 -h 192.168.12.46 -b "cn=config" | grep durable Enter LDAP Password: nsslapd-db-durable-transactions: off nsslapd-db-durable-transaction: off -- 389 users mailing list 389-users@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/389-users
Am Mon, 24 Feb 2014 14:18:17 -0700 schrieb David Boreham david_list@boreham.org:
Found out those files is shared mem written to disk!? So moving it to some ram disk seems ridiculous.
Why?
If memory gets doubled into a ram disk, then it's ridiculous.
If shared mem and ram disk are shared again, not doubled, then there's nothing wrong it. That would be ok.
And it seems linux is smart enough to not double it. Seems /dev/shm uses tmpfs explicitly for mmap().
Ok, then it's not ridiculous. :-)
Now, to get rid of the region and log files you would need to run the database in a non-transacted mode, or transacted but single-process.
Do you mean with DB_PRIVATE and DB_LOG_IN_MEMORY set to on? This way I would get rid of all those files (on disk).
I can activate the former one via: nsslapd-db-private-mem: on ... in /etc/dirsrv/slapd-kolab/dse.ldif
DB_PRIVATE suppresses __db.00* files, that works.
The later one via: log_set_config DB_LOG_IN_MEMORY ... in /var/lib/dirsrv/slapd-kolab/db/DB_CONFIG
But DB_LOG_IN_MEMORY doesn't work so far. Even if the Database files are consistent, after a clean shutdown. Without the database logs (or empty ones) the database won't start: libdb: file example_com/id2entry.db has LSN 1/3096458, past end of log at 1/140
And with DB_LOG_IN_MEMORY the log is empty with every restart...
I didn't knew those logs are such an important part, and even needed if the database is in a clean state.
Is there a way to reset LSNs in Database files? That would solve my problem.
If there's no way to reset them, there's no difference in having the log in ram (DB_LOG_IN_MEMORY) or having the whole database in ram (/var/lib/dirsrv/slapd-kolab/db/ as tmpfs). If I have to restore the database on every restart in either case, then it doesn't matter if the database files itself get lost or are persistent.
Another solution would be some load_log_into_mem_at_startup()- and save_log_to_disk_on_shutdown()-functions with DB_LOG_IN_MEMORY. With some kind of a "clean"-flag like there is one in filesystems. So with a clean shutdown it could be also started very clean. Then I would only have to recover the database after it has crashed... (e.g. has the "unclean" flag set)
hede
Am Tue, 25 Feb 2014 12:15:10 +0100 schrieb hede adam654@der-he.de:
Is there a way to reset LSNs in Database files? That would solve my problem.
Yes, there is.
$ for i in *db; do db_load -r lsn $i; done
... this does the trick. (in the db-dir)
This has to be done before every start.
389-users@lists.fedoraproject.org
-
David Boreham -
hede -
Rich Megginson