389-commits September 2020

389-commits@lists.fedoraproject.org

2 participants
4 discussions

389-ds-base repository has migrated to GitHub
by Simon Pichugin 13 Sep '20

13 Sep '20

Hi team, so the migration to GitHub was successfully completed! 389 Directory Server repo is now available on: https://github.com/389ds/389-ds-base <https://github.com/389ds/389-ds-base/issues/4317> The issue tracker is here: https://github.com/389ds/389-ds-base/issues Pull-requests are accepted here: https://github.com/389ds/389-ds-base/pulls The old Pagure repo is now closed and read-only. All Bugzillas are updated with the new links and Devel Whiteboard numbers. Commit email notifications still go to: 389-commits(a)lists.fedoraproject.org If you have any questions, please let me know here or on IRC - #dirsec or PM to spichugi. I've sent 389ds organization invitations to all active developers. Please, check your mailboxes. Sincerely, Simon

1 0

[389-ds-base] branch 389-ds-base-1.4.3 updated: Ticket 51190 - SyncRepl plugin provides a wrong cookie
by pagure＠pagure.io 09 Sep '20

09 Sep '20

This is an automated email from the git hooks/post-receive script. tbordaz pushed a commit to branch 389-ds-base-1.4.3 in repository 389-ds-base. The following commit(s) were added to refs/heads/389-ds-base-1.4.3 by this push: new 3533589 Ticket 51190 - SyncRepl plugin provides a wrong cookie 3533589 is described below commit 353358978a121fe61658e684b56c05f292884b91 Author: Thierry Bordaz <tbordaz(a)redhat.com> AuthorDate: Fri Jul 10 10:56:21 2020 +0200 Ticket 51190 - SyncRepl plugin provides a wrong cookie Bug description: A sync repl thread is similar to persistent search thread. The server is communicating with the sync repl thread with an ordered queue of updates. Updates are written in the queue by post op callbacks. Sync repl thread waits/reads the queue, retrieve the updates from the retroCL, checks if target entry matches the request (scope/filter) and send back the entry/update to the sync repl client. Several issues regarding the way order of the updates in the queue: (1) When an update generates nested updates (automemeber, memberof,...) the order of the updates in the queue is not following the order of applied updates. The consequence is that the cookie (containing the update nubmer) can be wrong. It can contains jumps, disorder and invalid number (-1). When an update fails (nested or primary update), none of the updates should be pushed to sync_repl queue (2) The plugin callback on updates are POSTOP, so if there are two direct updates, there is a possibility that the callback of the second update (and its nested updates) are enqueued before the first update. In such case the sync_repl thread may skip some updates and/or fail to retrieve update from retroCL (cookie.update_no=-1) Fix description: The fix does (1) implements a pending list of updates (in the thread private space "get_thread_primary_op"). The first in the pending list is the primary update then the others are the nested updates. A new operation (betxn_preop) registers the operation at the end of the pending list with the state OPERATION_PL_PENDING. It requires to registers new callbacks (sync_betxn_preop_init) During be_postop (see below) callbacks flags the pending updates as OPERATION_PL_SUCCEEDED or OPERATION_PL_FAILED depending on the operatione result. When no more pending updates are OPERATION_PL_PENDING, then depending of the result of the primary update (OPERATION_PL_SUCCEEDED or OPERATION_PL_FAILED) the updates are moved to the sync_repl queue. (2) The postop plugin callbacks are now be_postop https://pagure.io/389-ds-base/issue/51190 Reviewed by: Mark Reynolds, Simon Pichugin, William Brown (Thanks) Platforms tested: F31 --- .../tests/suites/plugins/acceptance_test.py | 4 +- .../tests/suites/syncrepl_plugin/basic_test.py | 553 +++++++++++++++++++++ ldap/servers/plugins/sync/sync.h | 37 ++ ldap/servers/plugins/sync/sync_init.c | 89 +++- ldap/servers/plugins/sync/sync_persist.c | 296 ++++++++++- ldap/servers/plugins/sync/sync_util.c | 56 ++- 6 files changed, 1003 insertions(+), 32 deletions(-) diff --git a/dirsrvtests/tests/suites/plugins/acceptance_test.py b/dirsrvtests/tests/suites/plugins/acceptance_test.py index e9e830d..9a5da25 100644 --- a/dirsrvtests/tests/suites/plugins/acceptance_test.py +++ b/dirsrvtests/tests/suites/plugins/acceptance_test.py @@ -12,6 +12,9 @@ Created on Dec 09, 2014 @author: mreynolds ''' import logging +import threading +from ldap.syncrepl import SyncreplConsumer +from ldap.ldapobject import ReconnectLDAPObject import subprocess import pytest from lib389.utils import * @@ -1546,7 +1549,6 @@ def test_referint(topo, args=None): log.info('test_referint: PASS\n') return - def test_retrocl(topo, args=None): """Test Retro Changelog basic functionality diff --git a/dirsrvtests/tests/suites/syncrepl_plugin/basic_test.py b/dirsrvtests/tests/suites/syncrepl_plugin/basic_test.py new file mode 100644 index 0000000..dc90944 --- /dev/null +++ b/dirsrvtests/tests/suites/syncrepl_plugin/basic_test.py @@ -0,0 +1,553 @@ +# --- BEGIN COPYRIGHT BLOCK --- +# Copyright (C) 2020 William Brown <william(a)blackhats.net.au> +# All rights reserved. +# +# License: GPL (version 3 or any later version). +# See LICENSE for details. +# --- END COPYRIGHT BLOCK --- + +import logging +import ldap +import time +import threading +from ldap.syncrepl import SyncreplConsumer +from ldap.ldapobject import ReconnectLDAPObject +import pytest +from lib389 import DirSrv +from lib389.idm.user import nsUserAccounts, UserAccounts +from lib389.idm.group import Groups +from lib389.topologies import topology_st as topology +from lib389.paths import Paths +from lib389.utils import ds_is_older +from lib389.plugins import RetroChangelogPlugin, ContentSyncPlugin, AutoMembershipPlugin, MemberOfPlugin, MemberOfSharedConfig, AutoMembershipDefinitions +from lib389._constants import * + +from . import ISyncRepl, syncstate_assert + +default_paths = Paths() +pytestmark = pytest.mark.tier1 + +log = logging.getLogger(__name__) + +def test_syncrepl_basic(topology): + """ Test basic functionality of the SyncRepl interface + + :id: f9fea826-8ae2-412a-8e88-b8e0ba939b06 + + :setup: Standalone instance + + :steps: + 1. Enable Retro Changelog + 2. Enable Syncrepl + 3. Run the syncstate test to check refresh, add, delete, mod. + + :expectedresults: + 1. Success + 1. Success + 1. Success + """ + st = topology.standalone + # Enable RetroChangelog. + rcl = RetroChangelogPlugin(st) + rcl.enable() + # Set the default targetid + rcl.replace('nsslapd-attribute', 'nsuniqueid:targetUniqueId') + # Enable sync repl + csp = ContentSyncPlugin(st) + csp.enable() + # Restart DS + st.restart() + # Setup the syncer + sync = ISyncRepl(st) + # Run the checks + syncstate_assert(st, sync) + +class TestSyncer(ReconnectLDAPObject, SyncreplConsumer): + def __init__(self, *args, **kwargs): + self.cookie = None + self.cookies = [] + ldap.ldapobject.ReconnectLDAPObject.__init__(self, *args, **kwargs) + + def syncrepl_set_cookie(self, cookie): + # extract the changenumber from the cookie + self.cookie = cookie + self.cookies.append(cookie.split('#')[2]) + log.info("XXXX Set cookie: %s" % cookie) + + def syncrepl_get_cookie(self): + log.info("XXXX Get cookie: %s" % self.cookie) + return self.cookie + + def syncrepl_present(self, uuids, refreshDeletes=False): + log.info("XXXX syncrepl_present uuids %s %s" % ( uuids, refreshDeletes)) + + def syncrepl_delete(self, uuids): + log.info("XXXX syncrepl_delete uuids %s" % uuids) + + def syncrepl_entry(self, dn, attrs, uuid): + log.info("XXXX syncrepl_entry dn %s" % dn) + + def syncrepl_refreshdone(self): + log.info("XXXX syncrepl_refreshdone") + + def get_cookies(self): + return self.cookies + +class Sync_persist(threading.Thread, ReconnectLDAPObject, SyncreplConsumer): + # This runs a sync_repl client in background + # it registers a result that contain a list of the change numbers (from the cookie) + # that are list as they are received + def __init__(self, inst): + threading.Thread.__init__(self) + self.daemon = True + self.inst = inst + self.cookie = None + self.conn = inst.clone({SER_ROOT_DN: 'cn=directory manager', SER_ROOT_PW: 'password'}) + self.filterstr = '(|(objectClass=groupofnames)(objectClass=person))' + self.attrs = [ + 'objectclass', + 'cn', + 'displayname', + 'gidnumber', + 'givenname', + 'homedirectory', + 'mail', + 'member', + 'memberof', + 'sn', + 'uid', + 'uidnumber', + ] + self.conn.open() + self.result = [] + + def get_result(self): + # used to return the cookies list to the requestor + return self.result + + def run(self): + """Start a sync repl client""" + ldap_connection = TestSyncer(self.inst.toLDAPURL()) + ldap_connection.simple_bind_s('cn=directory manager', 'password') + ldap_search = ldap_connection.syncrepl_search( + "dc=example,dc=com", + ldap.SCOPE_SUBTREE, + mode='refreshAndPersist', + attrlist=self.attrs, + filterstr=self.filterstr, + cookie=None + ) + + try: + while ldap_connection.syncrepl_poll(all=1, msgid=ldap_search): + pass + except (ldap.SERVER_DOWN, ldap.CONNECT_ERROR) as e: + print('syncrepl_poll: LDAP error (%s)', e) + self.result = ldap_connection.get_cookies() + log.info("ZZZ result = %s" % self.result) + self.conn.unbind() + +def test_sync_repl_cookie(topology, request): + """Test sync_repl cookie are progressing is an increasing order + when there are nested updates + + :id: d7fbde25-5702-46ac-b38e-169d7a68e97c + :setup: Standalone Instance + :steps: + 1.: enable retroCL + 2.: configure retroCL to log nsuniqueid as targetUniqueId + 3.: enable content_sync plugin + 4.: enable automember + 5.: create (2) groups. Few groups can help to reproduce the concurrent updates problem. + 6.: configure automember to provision those groups with 'member' + 7.: enable and configure memberof plugin + 8.: enable plugin log level + 9.: restart the server + 10.: create a thread dedicated to run a sync repl client + 11.: Create (9) users that will generate nested updates (automember/memberof) + 12.: stop sync repl client and collect the list of cookie.change_no + 13.: check that cookies.change_no are in increasing order + :expectedresults: + 1.: succeeds + 2.: succeeds + 3.: succeeds + 4.: succeeds + 5.: succeeds + 6.: succeeds + 7.: succeeds + 8.: succeeds + 9.: succeeds + 10.: succeeds + 11.: succeeds + 12.: succeeds + 13.: succeeds + """ + inst = topology[0] + + # Enable/configure retroCL + plugin = RetroChangelogPlugin(inst) + plugin.disable() + plugin.enable() + plugin.set('nsslapd-attribute', 'nsuniqueid:targetuniqueid') + + # Enable sync plugin + plugin = ContentSyncPlugin(inst) + plugin.enable() + + # Enable automember + plugin = AutoMembershipPlugin(inst) + plugin.disable() + plugin.enable() + + # Add the automember group + groups = Groups(inst, DEFAULT_SUFFIX) + group = [] + for i in range(1,3): + group.append(groups.create(properties={'cn': 'group%d' % i})) + + # Add the automember config entry + am_configs = AutoMembershipDefinitions(inst) + for g in group: + am_config = am_configs.create(properties={'cn': 'config %s' % g.get_attr_val_utf8('cn'), + 'autoMemberScope': DEFAULT_SUFFIX, + 'autoMemberFilter': 'uid=*', + 'autoMemberDefaultGroup': g.dn, + 'autoMemberGroupingAttr': 'member:dn'}) + + # Enable and configure memberof plugin + plugin = MemberOfPlugin(inst) + plugin.disable() + plugin.enable() + + plugin.replace_groupattr('member') + + memberof_config = MemberOfSharedConfig(inst, 'cn=memberOf config,{}'.format(DEFAULT_SUFFIX)) + memberof_config.create(properties={'cn': 'memberOf config', + 'memberOfGroupAttr': 'member', + 'memberOfAttr': 'memberof'}) + # Enable plugin log level (usefull for debug) + inst.setLogLevel(65536) + inst.restart() + + # create a sync repl client and wait 5 seconds to be sure it is running + sync_repl = Sync_persist(inst) + sync_repl.start() + time.sleep(5) + + # create users, that automember/memberof will generate nested updates + users = UserAccounts(inst, DEFAULT_SUFFIX) + users_set = [] + for i in range(10001, 10010): + users_set.append(users.create_test_user(uid=i)) + + # stop the server to get the sync_repl result set (exit from while loop). + # Only way I found to acheive that. + # and wait a bit to let sync_repl thread time to set its result before fetching it. + inst.stop() + time.sleep(10) + cookies = sync_repl.get_result() + + # checking that the cookie are in increasing and in an acceptable range (0..1000) + assert len(cookies) > 0 + prev = 0 + for cookie in cookies: + log.info('Check cookie %s' % cookie) + + assert int(cookie) > 0 + assert int(cookie) < 1000 + assert int(cookie) > prev + prev = int(cookie) + sync_repl.join() + log.info('test_sync_repl_cookie: PASS\n') + + def fin(): + inst.restart() + for user in users_set: + try: + user.delete() + except: + pass + for g in group: + try: + g.delete() + except: + pass + + request.addfinalizer(fin) + + return + +def test_sync_repl_cookie_add_del(topology, request): + """Test sync_repl cookie are progressing is an increasing order + when there add and del + + :id: 83e11038-6ed0-4a5b-ac77-e44887ab11e3 + :setup: Standalone Instance + :steps: + 1.: enable retroCL + 2.: configure retroCL to log nsuniqueid as targetUniqueId + 3.: enable content_sync plugin + 4.: enable automember + 5.: create (2) groups. Few groups can help to reproduce the concurrent updates problem. + 6.: configure automember to provision those groups with 'member' + 7.: enable and configure memberof plugin + 8.: enable plugin log level + 9.: restart the server + 10.: create a thread dedicated to run a sync repl client + 11.: Create (3) users that will generate nested updates (automember/memberof) + 12.: Delete (3) users + 13.: stop sync repl client and collect the list of cookie.change_no + 14.: check that cookies.change_no are in increasing order + :expectedresults: + 1.: succeeds + 2.: succeeds + 3.: succeeds + 4.: succeeds + 5.: succeeds + 6.: succeeds + 7.: succeeds + 8.: succeeds + 9.: succeeds + 10.: succeeds + 11.: succeeds + 12.: succeeds + 13.: succeeds + 14.: succeeds + """ + inst = topology[0] + + # Enable/configure retroCL + plugin = RetroChangelogPlugin(inst) + plugin.disable() + plugin.enable() + plugin.set('nsslapd-attribute', 'nsuniqueid:targetuniqueid') + + # Enable sync plugin + plugin = ContentSyncPlugin(inst) + plugin.enable() + + # Enable automember + plugin = AutoMembershipPlugin(inst) + plugin.disable() + plugin.enable() + + # Add the automember group + groups = Groups(inst, DEFAULT_SUFFIX) + group = [] + for i in range(1,3): + group.append(groups.create(properties={'cn': 'group%d' % i})) + + # Add the automember config entry + am_configs = AutoMembershipDefinitions(inst) + for g in group: + am_config = am_configs.create(properties={'cn': 'config %s' % g.get_attr_val_utf8('cn'), + 'autoMemberScope': DEFAULT_SUFFIX, + 'autoMemberFilter': 'uid=*', + 'autoMemberDefaultGroup': g.dn, + 'autoMemberGroupingAttr': 'member:dn'}) + + # Enable and configure memberof plugin + plugin = MemberOfPlugin(inst) + plugin.disable() + plugin.enable() + + plugin.replace_groupattr('member') + + memberof_config = MemberOfSharedConfig(inst, 'cn=memberOf config,{}'.format(DEFAULT_SUFFIX)) + memberof_config.create(properties={'cn': 'memberOf config', + 'memberOfGroupAttr': 'member', + 'memberOfAttr': 'memberof'}) + # Enable plugin log level (usefull for debug) + inst.setLogLevel(65536) + inst.restart() + + # create a sync repl client and wait 5 seconds to be sure it is running + sync_repl = Sync_persist(inst) + sync_repl.start() + time.sleep(5) + + # create users, that automember/memberof will generate nested updates + users = UserAccounts(inst, DEFAULT_SUFFIX) + users_set = [] + for i in range(10001, 10004): + users_set.append(users.create_test_user(uid=i)) + + time.sleep(10) + # delete users, that automember/memberof will generate nested updates + for user in users_set: + user.delete() + # stop the server to get the sync_repl result set (exit from while loop). + # Only way I found to acheive that. + # and wait a bit to let sync_repl thread time to set its result before fetching it. + inst.stop() + cookies = sync_repl.get_result() + + # checking that the cookie are in increasing and in an acceptable range (0..1000) + assert len(cookies) > 0 + prev = 0 + for cookie in cookies: + log.info('Check cookie %s' % cookie) + + assert int(cookie) > 0 + assert int(cookie) < 1000 + assert int(cookie) > prev + prev = int(cookie) + sync_repl.join() + log.info('test_sync_repl_cookie_add_del: PASS\n') + + def fin(): + inst.restart() + for g in group: + try: + g.delete() + except: + pass + + request.addfinalizer(fin) + + return + +def test_sync_repl_cookie_with_failure(topology, request): + """Test sync_repl cookie are progressing is the right order + when there is a failure in nested updates + + :id: e0103448-170e-4080-8f22-c34606447ce2 + :setup: Standalone Instance + :steps: + 1.: enable retroCL + 2.: configure retroCL to log nsuniqueid as targetUniqueId + 3.: enable content_sync plugin + 4.: enable automember + 5.: create (4) groups. + make group2 groupOfUniqueNames so the automember + will fail to add 'member' (uniqueMember expected) + 6.: configure automember to provision those groups with 'member' + 7.: enable and configure memberof plugin + 8.: enable plugin log level + 9.: restart the server + 10.: create a thread dedicated to run a sync repl client + 11.: Create a group that will be the only update received by sync repl client + 12.: Create (9) users that will generate nested updates (automember/memberof) + 13.: stop sync repl client and collect the list of cookie.change_no + 14.: check that the list of cookie.change_no contains only the group 'step 11' + :expectedresults: + 1.: succeeds + 2.: succeeds + 3.: succeeds + 4.: succeeds + 5.: succeeds + 6.: succeeds + 7.: succeeds + 8.: succeeds + 9.: succeeds + 10.: succeeds + 11.: succeeds + 12.: Fails (expected) + 13.: succeeds + 14.: succeeds + """ + inst = topology[0] + + # Enable/configure retroCL + plugin = RetroChangelogPlugin(inst) + plugin.disable() + plugin.enable() + plugin.set('nsslapd-attribute', 'nsuniqueid:targetuniqueid') + + # Enable sync plugin + plugin = ContentSyncPlugin(inst) + plugin.enable() + + # Enable automember + plugin = AutoMembershipPlugin(inst) + plugin.disable() + plugin.enable() + + # Add the automember group + groups = Groups(inst, DEFAULT_SUFFIX) + group = [] + for i in range(1,5): + group.append(groups.create(properties={'cn': 'group%d' % i})) + + # Set group2 as a groupOfUniqueNames so that automember will fail to update that group + # This will trigger a failure in internal MOD and a failure to add member + group[1].replace('objectclass', 'groupOfUniqueNames') + + # Add the automember config entry + am_configs = AutoMembershipDefinitions(inst) + for g in group: + am_config = am_configs.create(properties={'cn': 'config %s' % g.get_attr_val_utf8('cn'), + 'autoMemberScope': DEFAULT_SUFFIX, + 'autoMemberFilter': 'uid=*', + 'autoMemberDefaultGroup': g.dn, + 'autoMemberGroupingAttr': 'member:dn'}) + + # Enable and configure memberof plugin + plugin = MemberOfPlugin(inst) + plugin.disable() + plugin.enable() + + plugin.replace_groupattr('member') + + memberof_config = MemberOfSharedConfig(inst, 'cn=memberOf config,{}'.format(DEFAULT_SUFFIX)) + memberof_config.create(properties={'cn': 'memberOf config', + 'memberOfGroupAttr': 'member', + 'memberOfAttr': 'memberof'}) + + # Enable plugin log level (usefull for debug) + inst.setLogLevel(65536) + inst.restart() + + # create a sync repl client and wait 5 seconds to be sure it is running + sync_repl = Sync_persist(inst) + sync_repl.start() + time.sleep(5) + + # Add a test group just to check that sync_repl receives only one update + group.append(groups.create(properties={'cn': 'group%d' % 10})) + + # create users, that automember/memberof will generate nested updates + users = UserAccounts(inst, DEFAULT_SUFFIX) + users_set = [] + for i in range(1000,1010): + try: + users_set.append(users.create_test_user(uid=i)) + # Automember should fail to add uid=1000 in group2 + assert(False) + except ldap.UNWILLING_TO_PERFORM: + pass + + # stop the server to get the sync_repl result set (exit from while loop). + # Only way I found to acheive that. + # and wait a bit to let sync_repl thread time to set its result before fetching it. + inst.stop() + time.sleep(10) + cookies = sync_repl.get_result() + + # checking that the cookie list contains only one entry + assert len(cookies) == 1 + prev = 0 + for cookie in cookies: + log.info('Check cookie %s' % cookie) + + assert int(cookie) > 0 + assert int(cookie) < 1000 + assert int(cookie) > prev + prev = int(cookie) + sync_repl.join() + log.info('test_sync_repl_cookie_with_failure: PASS\n') + + def fin(): + inst.restart() + for user in users_set: + try: + user.delete() + except: + pass + for g in group: + try: + g.delete() + except: + pass + + request.addfinalizer(fin) diff --git a/ldap/servers/plugins/sync/sync.h b/ldap/servers/plugins/sync/sync.h index a6b54a6..8c367c3 100644 --- a/ldap/servers/plugins/sync/sync.h +++ b/ldap/servers/plugins/sync/sync.h @@ -16,6 +16,8 @@ #include <stdio.h> #include <string.h> +#include <stdbool.h> +#include "slap.h" #include "slapi-plugin.h" #include "slapi-private.h" @@ -25,6 +27,8 @@ #define SYNC_PREOP_DESC "content-sync-preop-subplugin" #define SYNC_POSTOP_DESC "content-sync-postop-subplugin" #define SYNC_INT_POSTOP_DESC "content-sync-int-postop-subplugin" +#define SYNC_BETXN_PREOP_DESC "content-sync-betxn-preop-subplugin" +#define SYNC_BE_POSTOP_DESC "content-sync-be-post-subplugin" #define OP_FLAG_SYNC_PERSIST 0x01 @@ -64,6 +68,37 @@ typedef struct sync_callback Sync_UpdateNode *cb_updates; } Sync_CallBackData; +/* Pending list flags + * OPERATION_PL_PENDING: operation not yet completed + * OPERATION_PL_SUCCEEDED: operation completed successfully + * OPERATION_PL_FAILED: operation completed and failed + * OPERATION_PL_IGNORED: operation completed but with an undefine status + */ +typedef enum _pl_flags { + OPERATION_PL_PENDING = 1, + OPERATION_PL_SUCCEEDED = 2, + OPERATION_PL_FAILED = 3, + OPERATION_PL_IGNORED = 4 +} pl_flags_t; + +/* Pending list operations. + * it contains a list ('next') of nested operations. The + * order the same order that the server applied the operation + * see https://www.port389.org/docs/389ds/design/content-synchronization-plugin.ht… + */ +typedef struct OPERATION_PL_CTX +{ + Operation *op; /* Pending operation, should not be freed as it belongs to the pblock */ + pl_flags_t flags; /* operation is completed (set to TRUE in POST) */ + Slapi_Entry *entry; /* entry to be store in the enqueued node. 1st arg sync_queue_change */ + Slapi_Entry *eprev; /* pre-entry to be stored in the enqueued node. 2nd arg sync_queue_change */ + ber_int_t chgtype; /* change type to be stored in the enqueued node. 3rd arg of sync_queue_change */ + struct OPERATION_PL_CTX *next; /* list of nested operation, the head of the list is the primary operation */ +} OPERATION_PL_CTX_T; + +OPERATION_PL_CTX_T * get_thread_primary_op(void); +void set_thread_primary_op(OPERATION_PL_CTX_T *op); + int sync_register_operation_extension(void); int sync_unregister_operation_entension(void); @@ -75,6 +110,7 @@ int sync_del_persist_post_op(Slapi_PBlock *pb); int sync_mod_persist_post_op(Slapi_PBlock *pb); int sync_modrdn_persist_post_op(Slapi_PBlock *pb); int sync_add_persist_post_op(Slapi_PBlock *pb); +int sync_update_persist_betxn_pre_op(Slapi_PBlock *pb); int sync_parse_control_value(struct berval *psbvp, ber_int_t *mode, int *reload, char **cookie); int sync_create_state_control(Slapi_Entry *e, LDAPControl **ctrlp, int type, Sync_Cookie *cookie); @@ -181,3 +217,4 @@ typedef struct sync_op_info Sync_Cookie *cookie; /* cookie to add in control */ PRThread *tid; /* thread for persistent phase */ } SyncOpInfo; + diff --git a/ldap/servers/plugins/sync/sync_init.c b/ldap/servers/plugins/sync/sync_init.c index 2843b88..4b802d6 100644 --- a/ldap/servers/plugins/sync/sync_init.c +++ b/ldap/servers/plugins/sync/sync_init.c @@ -14,7 +14,10 @@ static int sync_start(Slapi_PBlock *pb); static int sync_close(Slapi_PBlock *pb); static int sync_preop_init(Slapi_PBlock *pb); static int sync_postop_init(Slapi_PBlock *pb); -static int sync_internal_postop_init(Slapi_PBlock *pb); +static int sync_be_postop_init(Slapi_PBlock *pb); +static int sync_betxn_preop_init(Slapi_PBlock *pb); + +static PRUintn thread_primary_op; int sync_init(Slapi_PBlock *pb) @@ -80,17 +83,32 @@ sync_init(Slapi_PBlock *pb) } if (rc == 0) { - char *plugin_type = "internalpostoperation"; + char *plugin_type = "betxnpreoperation"; /* the config change checking post op */ if (slapi_register_plugin(plugin_type, 1, /* Enabled */ "sync_init", /* this function desc */ - sync_internal_postop_init, /* init func for post op */ - SYNC_INT_POSTOP_DESC, /* plugin desc */ + sync_betxn_preop_init, /* init func for post op */ + SYNC_BETXN_PREOP_DESC, /* plugin desc */ NULL, plugin_identity)) { slapi_log_err(SLAPI_LOG_ERR, SYNC_PLUGIN_SUBSYSTEM, - "sync_init - Failed to register internal postop plugin\n"); + "sync_init - Failed to register be_txn_pre_op plugin\n"); + rc = 1; + } + } + if (rc == 0) { + char *plugin_type = "bepostoperation"; + /* the config change checking post op */ + if (slapi_register_plugin(plugin_type, + 1, /* Enabled */ + "sync_init", /* this function desc */ + sync_be_postop_init, /* init func for be_post op */ + SYNC_BE_POSTOP_DESC, /* plugin desc */ + NULL, + plugin_identity)) { + slapi_log_err(SLAPI_LOG_ERR, SYNC_PLUGIN_SUBSYSTEM, + "sync_init - Failed to register be_txn_pre_op plugin\n"); rc = 1; } } @@ -114,25 +132,31 @@ static int sync_postop_init(Slapi_PBlock *pb) { int rc; - rc = slapi_pblock_set(pb, SLAPI_PLUGIN_POST_ADD_FN, (void *)sync_add_persist_post_op); - rc |= slapi_pblock_set(pb, SLAPI_PLUGIN_POST_DELETE_FN, (void *)sync_del_persist_post_op); - rc |= slapi_pblock_set(pb, SLAPI_PLUGIN_POST_MODIFY_FN, (void *)sync_mod_persist_post_op); - rc |= slapi_pblock_set(pb, SLAPI_PLUGIN_POST_MODRDN_FN, (void *)sync_modrdn_persist_post_op); - rc |= slapi_pblock_set(pb, SLAPI_PLUGIN_POST_SEARCH_FN, (void *)sync_srch_refresh_post_search); + rc = slapi_pblock_set(pb, SLAPI_PLUGIN_POST_SEARCH_FN, (void *)sync_srch_refresh_post_search); return (rc); } static int -sync_internal_postop_init(Slapi_PBlock *pb) +sync_be_postop_init(Slapi_PBlock *pb) { int rc; - rc = slapi_pblock_set(pb, SLAPI_PLUGIN_INTERNAL_POST_ADD_FN, (void *)sync_add_persist_post_op); - rc |= slapi_pblock_set(pb, SLAPI_PLUGIN_INTERNAL_POST_DELETE_FN, (void *)sync_del_persist_post_op); - rc |= slapi_pblock_set(pb, SLAPI_PLUGIN_INTERNAL_POST_MODIFY_FN, (void *)sync_mod_persist_post_op); - rc |= slapi_pblock_set(pb, SLAPI_PLUGIN_INTERNAL_POST_MODRDN_FN, (void *)sync_modrdn_persist_post_op); + rc = slapi_pblock_set(pb, SLAPI_PLUGIN_BE_POST_ADD_FN, (void *)sync_add_persist_post_op); + rc |= slapi_pblock_set(pb, SLAPI_PLUGIN_BE_POST_DELETE_FN, (void *)sync_del_persist_post_op); + rc |= slapi_pblock_set(pb, SLAPI_PLUGIN_BE_POST_MODIFY_FN, (void *)sync_mod_persist_post_op); + rc |= slapi_pblock_set(pb, SLAPI_PLUGIN_BE_POST_MODRDN_FN, (void *)sync_modrdn_persist_post_op); return (rc); } +static int +sync_betxn_preop_init(Slapi_PBlock *pb) +{ + int rc; + rc = slapi_pblock_set(pb, SLAPI_PLUGIN_BE_TXN_PRE_ADD_FN, (void *)sync_update_persist_betxn_pre_op); + rc |= slapi_pblock_set(pb, SLAPI_PLUGIN_BE_TXN_PRE_DELETE_FN, (void *)sync_update_persist_betxn_pre_op); + rc |= slapi_pblock_set(pb, SLAPI_PLUGIN_BE_TXN_PRE_MODIFY_FN, (void *)sync_update_persist_betxn_pre_op); + rc |= slapi_pblock_set(pb, SLAPI_PLUGIN_BE_TXN_PRE_MODRDN_FN, (void *)sync_update_persist_betxn_pre_op); + return (rc); +} /* sync_start -------------- @@ -156,6 +180,12 @@ sync_start(Slapi_PBlock *pb) "sync_start - Unable to get arguments\n"); return (-1); } + /* It registers a per thread 'thread_primary_op' variable that is + * a list of pending operations. For simple operation, this list + * only contains one operation. For nested, the list contains the operations + * in the order that they were applied + */ + PR_NewThreadPrivateIndex(&thread_primary_op, NULL); sync_persist_initialize(argc, argv); return (0); @@ -174,3 +204,32 @@ sync_close(Slapi_PBlock *pb __attribute__((unused))) return (0); } + +/* Return the head of the operations list + * the head is the primary operation. + * The list is private to that thread and contains + * all nested operations applied by the thread. + */ +OPERATION_PL_CTX_T * +get_thread_primary_op(void) +{ + OPERATION_PL_CTX_T *prim_op = NULL; + if (thread_primary_op) { + prim_op = (OPERATION_PL_CTX_T *)PR_GetThreadPrivate(thread_primary_op); + } + + return prim_op; +} + +/* It is set with a non NULL op when this is a primary operation + * else it set to NULL when the all pending list has be flushed. + * The list is flushed when no more operations (in that list) are + * pending (OPERATION_PL_PENDING). + */ +void +set_thread_primary_op(OPERATION_PL_CTX_T *op) +{ + if (thread_primary_op) { + PR_SetThreadPrivate(thread_primary_op, (void *)op); + } +} \ No newline at end of file diff --git a/ldap/servers/plugins/sync/sync_persist.c b/ldap/servers/plugins/sync/sync_persist.c index 880163a..5724d7d 100644 --- a/ldap/servers/plugins/sync/sync_persist.c +++ b/ldap/servers/plugins/sync/sync_persist.c @@ -6,6 +6,9 @@ * See LICENSE for details. * END COPYRIGHT BLOCK **/ +#include <nspr4/prlog.h> +#include <bits/stdint-intn.h> + #include "sync.h" /* Main list of established persistent synchronizaton searches */ @@ -29,7 +32,7 @@ static PRUint64 thread_count = 0; static int sync_add_request(SyncRequest *req); static void sync_remove_request(SyncRequest *req); static SyncRequest *sync_request_alloc(void); -void sync_queue_change(Slapi_Entry *e, Slapi_Entry *eprev, ber_int_t chgtype); +void sync_queue_change(OPERATION_PL_CTX_T *operation); static void sync_send_results(void *arg); static void sync_request_wakeup_all(void); static void sync_node_free(SyncQueueNode **node); @@ -37,17 +40,244 @@ static void sync_node_free(SyncQueueNode **node); static int sync_acquire_connection(Slapi_Connection *conn); static int sync_release_connection(Slapi_PBlock *pb, Slapi_Connection *conn, Slapi_Operation *op, int release); +/* This routine appends the operation at the end of the + * per thread pending list of nested operation.. + * being a betxn_preop the pending list has the same order + * that the server received the operation + */ +int +sync_update_persist_betxn_pre_op(Slapi_PBlock *pb) +{ + OPERATION_PL_CTX_T *prim_op; + OPERATION_PL_CTX_T *new_op; + Slapi_DN *sdn; + + if (!SYNC_IS_INITIALIZED()) { + /* not initialized if sync plugin is not started */ + return 0; + } + + /* Create a new pending operation node */ + new_op = (OPERATION_PL_CTX_T *)slapi_ch_calloc(1, sizeof(OPERATION_PL_CTX_T)); + new_op->flags = OPERATION_PL_PENDING; + slapi_pblock_get(pb, SLAPI_OPERATION, &new_op->op); + slapi_pblock_get(pb, SLAPI_TARGET_SDN, &sdn); + + prim_op = get_thread_primary_op(); + if (prim_op) { + /* It already exists a primary operation, so the current + * operation is a nested one that we need to register at the end + * of the pending nested operations + */ + OPERATION_PL_CTX_T *current_op; + for (current_op = prim_op; current_op->next; current_op = current_op->next); + current_op->next = new_op; + slapi_log_err(SLAPI_LOG_PLUGIN, SYNC_PLUGIN_SUBSYSTEM, "sync_update_persist_betxn_pre_op - nested operation targets " + "\"%s\" (0x%lx)\n", + slapi_sdn_get_dn(sdn), (ulong) new_op->op); + } else { + /* The current operation is the first/primary one in the txn + * registers it directly in the thread private data (head) + */ + set_thread_primary_op(new_op); + slapi_log_err(SLAPI_LOG_PLUGIN, SYNC_PLUGIN_SUBSYSTEM, "sync_update_persist_betxn_pre_op - primary operation targets " + "\"%s\" (0x%lx)\n", + slapi_sdn_get_dn(sdn), (ulong) new_op->op); + } + return 0; +} + +/* This operation can not be proceed by sync_repl listener because + * of internal problem. For example, POST entry does not exist + */ +static void +ignore_op_pl(Operation *op) +{ + OPERATION_PL_CTX_T *prim_op, *curr_op; + prim_op = get_thread_primary_op(); + + for (curr_op = prim_op; curr_op; curr_op = curr_op->next) { + if ((curr_op->op == op) && + (curr_op->flags == OPERATION_PL_PENDING)) { /* If by any "chance" a same operation structure was reused in consecutive updates + * we can not only rely on 'op' value + */ + slapi_log_err(SLAPI_LOG_ERR, SYNC_PLUGIN_SUBSYSTEM, "ignore_op_pl operation (0x%lx) from the pending list\n", + (ulong) op); + curr_op->flags = OPERATION_PL_IGNORED; + return; + } + } + slapi_log_err(SLAPI_LOG_ERR, SYNC_PLUGIN_SUBSYSTEM, "ignore_op_pl can not retrieve an operation (0x%lx) in pending list\n", + (ulong) op); +} + +/* This is a generic function that is called by betxn_post of this plugin. + * For the given operation (pb->pb_op) it sets in the pending list the state + * of the completed operation. + * When all operations are completed, if the primary operation is successful it + * flushes (enqueue) the operations to the sync repl queue(s), else it just free + * the pending list (skipping enqueue). + */ +static void +sync_update_persist_op(Slapi_PBlock *pb, Slapi_Entry *e, Slapi_Entry *eprev, ber_int_t op_tag, char *label) +{ + OPERATION_PL_CTX_T *prim_op = NULL, *curr_op; + Operation *pb_op; + Slapi_DN *sdn; + int32_t rc; + + if (!SYNC_IS_INITIALIZED()) { + /* not initialized if sync plugin is not started */ + return; + } + slapi_pblock_get(pb, SLAPI_OPERATION, &pb_op); + slapi_pblock_get(pb, SLAPI_TARGET_SDN, &sdn); + + if (NULL == e) { + /* Ignore this operation (for example case of failure of the operation) */ + ignore_op_pl(pb_op); + return; + } + + /* Retrieve the result of the operation */ + if (slapi_op_internal(pb)) { + slapi_pblock_get(pb, SLAPI_PLUGIN_INTOP_RESULT, &rc); + if (0 != rc) { + /* The internal operation did not succeed */ + slapi_log_err(SLAPI_LOG_PLUGIN, SYNC_PLUGIN_SUBSYSTEM, "internal operation Failed (0x%lx) rc=%d\n", + (ulong) pb_op, rc); + } + } else { + slapi_pblock_get(pb, SLAPI_PLUGIN_OPRETURN, &rc); + if (0 != rc) { + /* The operation did not succeed */ + slapi_log_err(SLAPI_LOG_PLUGIN, SYNC_PLUGIN_SUBSYSTEM, "direct operation Failed (0x%lx) rc=%d\n", + (ulong) pb_op, rc); + } + } + + + prim_op = get_thread_primary_op(); + PR_ASSERT(prim_op); + /* First mark the operation as completed/failed + * the param to be used once the operation will be pushed + * on the listeners queue + */ + for (curr_op = prim_op; curr_op; curr_op = curr_op->next) { + if ((curr_op->op == pb_op) && + (curr_op->flags == OPERATION_PL_PENDING)) { /* If by any "chance" a same operation structure was reused in consecutive updates + * we can not only rely on 'op' value + */ + if (rc == LDAP_SUCCESS) { + curr_op->flags = OPERATION_PL_SUCCEEDED; + curr_op->entry = e ? slapi_entry_dup(e) : NULL; + curr_op->eprev = eprev ? slapi_entry_dup(eprev) : NULL; + curr_op->chgtype = op_tag; + } else { + curr_op->flags = OPERATION_PL_FAILED; + } + break; + } + } + if (!curr_op) { + slapi_log_err(SLAPI_LOG_ERR, SYNC_PLUGIN_SUBSYSTEM, "%s - operation not found on the pendling list\n", label); + PR_ASSERT(curr_op); + } + +#if DEBUG + /* dump the pending queue */ + for (curr_op = prim_op; curr_op; curr_op = curr_op->next) { + char *flags_str; + char * entry_str; + + if (curr_op->entry) { + entry_str = slapi_entry_get_dn(curr_op->entry); + } else if (curr_op->eprev){ + entry_str = slapi_entry_get_dn(curr_op->eprev); + } else { + entry_str = "unknown"; + } + switch (curr_op->flags) { + case OPERATION_PL_SUCCEEDED: + flags_str = "succeeded"; + break; + case OPERATION_PL_FAILED: + flags_str = "failed"; + break; + case OPERATION_PL_IGNORED: + flags_str = "ignored"; + break; + case OPERATION_PL_PENDING: + flags_str = "pending"; + break; + default: + flags_str = "unknown"; + break; + + + } + slapi_log_err(SLAPI_LOG_PLUGIN, SYNC_PLUGIN_SUBSYSTEM, "dump pending list(0x%lx) %s %s\n", + (ulong) curr_op->op, entry_str, flags_str); + } +#endif + + /* Second check if it remains a pending operation in the pending list */ + for (curr_op = prim_op; curr_op; curr_op = curr_op->next) { + if (curr_op->flags == OPERATION_PL_PENDING) { + break; + } + } + if (curr_op) { + slapi_log_err(SLAPI_LOG_PLUGIN, SYNC_PLUGIN_SUBSYSTEM, "%s - It remains a pending operation (0x%lx)\n", label, (ulong) curr_op->op); + } else { + OPERATION_PL_CTX_T *next = NULL; + PRBool enqueue_it = PR_TRUE; + /* all operations on the pending list are completed moved them + * to the listeners queue in the same order as pending list. + * If the primary operation failed, operation are not moved to + * the queue + */ + if (prim_op->flags == OPERATION_PL_FAILED) { + /* if primary update failed, the txn is aborted and none of + * the operations were applied. Just forget this pending list + */ + enqueue_it = PR_FALSE; + } + for (curr_op = prim_op; curr_op; curr_op = next) { + char *entry; + if (curr_op->entry) { + entry = slapi_entry_get_dn(curr_op->entry); + } else if (curr_op->eprev){ + entry = slapi_entry_get_dn(curr_op->eprev); + } else { + entry = "unknown"; + } + slapi_log_err(SLAPI_LOG_PLUGIN, SYNC_PLUGIN_SUBSYSTEM, "Do %s enqueue (0x%lx) %s\n", + enqueue_it ? "" : "not", (ulong) curr_op->op, entry); + if (enqueue_it) { + sync_queue_change(curr_op); + } + + /* now free this pending operation */ + next = curr_op->next; + slapi_entry_free(curr_op->entry); + slapi_entry_free(curr_op->eprev); + slapi_ch_free((void **)&curr_op); + } + /* we consumed all the pending operation, free the pending list*/ + set_thread_primary_op(NULL); + } +} int sync_add_persist_post_op(Slapi_PBlock *pb) { Slapi_Entry *e; - if (!SYNC_IS_INITIALIZED()) { return (0); } slapi_pblock_get(pb, SLAPI_ENTRY_POST_OP, &e); - sync_queue_change(e, NULL, LDAP_REQ_ADD); + sync_update_persist_op(pb, e, NULL, LDAP_REQ_ADD, "sync_add_persist_post_op"); return (0); } @@ -62,7 +292,7 @@ sync_del_persist_post_op(Slapi_PBlock *pb) } slapi_pblock_get(pb, SLAPI_ENTRY_PRE_OP, &e); - sync_queue_change(e, NULL, LDAP_REQ_DELETE); + sync_update_persist_op(pb, e, NULL, LDAP_REQ_DELETE, "sync_del_persist_post_op"); return (0); } @@ -78,7 +308,7 @@ sync_mod_persist_post_op(Slapi_PBlock *pb) slapi_pblock_get(pb, SLAPI_ENTRY_POST_OP, &e); slapi_pblock_get(pb, SLAPI_ENTRY_PRE_OP, &e_prev); - sync_queue_change(e, e_prev, LDAP_REQ_MODIFY); + sync_update_persist_op(pb, e, e_prev, LDAP_REQ_MODIFY, "sync_mod_persist_post_op"); return (0); } @@ -94,19 +324,22 @@ sync_modrdn_persist_post_op(Slapi_PBlock *pb) slapi_pblock_get(pb, SLAPI_ENTRY_POST_OP, &e); slapi_pblock_get(pb, SLAPI_ENTRY_PRE_OP, &e_prev); - sync_queue_change(e, e_prev, LDAP_REQ_MODRDN); + sync_update_persist_op(pb, e, e_prev, LDAP_REQ_MODRDN, "sync_mod_persist_post_op"); return (0); } void -sync_queue_change(Slapi_Entry *e, Slapi_Entry *eprev, ber_int_t chgtype) +sync_queue_change(OPERATION_PL_CTX_T *operation) { SyncRequest *req = NULL; SyncQueueNode *node = NULL; int matched = 0; int prev_match = 0; int cur_match = 0; + Slapi_Entry *e = operation->entry; + Slapi_Entry *eprev = operation->eprev; + ber_int_t chgtype = operation->chgtype; if (!SYNC_IS_INITIALIZED()) { return; @@ -195,16 +428,14 @@ sync_queue_change(Slapi_Entry *e, Slapi_Entry *eprev, ber_int_t chgtype) } else { pOldtail->sync_next = req->ps_eq_tail; } + slapi_log_err(SLAPI_LOG_PLUGIN, SYNC_PLUGIN_SUBSYSTEM, "sync_queue_change - entry " + "\"%s\" \n", + slapi_entry_get_dn_const(node->sync_entry)); PR_Unlock(req->req_lock); } } - - SYNC_UNLOCK_READ(); - /* Were there any matches? */ if (matched) { - /* Notify update threads */ - sync_request_wakeup_all(); slapi_log_err(SLAPI_LOG_TRACE, SYNC_PLUGIN_SUBSYSTEM, "sync_queue_change - enqueued entry " "\"%s\" on %d request listeners\n", slapi_entry_get_dn_const(e), matched); @@ -213,6 +444,13 @@ sync_queue_change(Slapi_Entry *e, Slapi_Entry *eprev, ber_int_t chgtype) "\"%s\" not enqueued on any request search listeners\n", slapi_entry_get_dn_const(e)); } + SYNC_UNLOCK_READ(); + + /* Were there any matches? */ + if (matched) { + /* Notify update threads */ + sync_request_wakeup_all(); + } } /* * Initialize the list structure which contains the list @@ -359,11 +597,12 @@ sync_persist_terminate(PRThread *tid) } /* - * Called when stopping/disabling the plugin + * Called when stopping/disabling the plugin (like shutdown) */ int sync_persist_terminate_all() { + SyncRequest *req = NULL, *next; if (SYNC_IS_INITIALIZED()) { /* signal the threads to stop */ plugin_closing = 1; @@ -377,6 +616,16 @@ sync_persist_terminate_all() slapi_destroy_rwlock(sync_request_list->sync_req_rwlock); PR_DestroyLock(sync_request_list->sync_req_cvarlock); PR_DestroyCondVar(sync_request_list->sync_req_cvar); + + /* it frees the structures, just in case it remained connected sync_repl client */ + for (req = sync_request_list->sync_req_head; NULL != req; req = next) { + next = req->req_next; + slapi_pblock_destroy(req->req_pblock); + req->req_pblock = NULL; + PR_DestroyLock(req->req_lock); + req->req_lock = NULL; + slapi_ch_free((void **)&req); + } slapi_ch_free((void **)&sync_request_list); } @@ -545,6 +794,8 @@ sync_send_results(void *arg) int rc; PRUint64 connid; int opid; + char **attrs_dup; + char *strFilter; slapi_pblock_get(req->req_pblock, SLAPI_CONN_ID, &connid); slapi_pblock_get(req->req_pblock, SLAPI_OPERATION_ID, &opid); @@ -589,9 +840,12 @@ sync_send_results(void *arg) Slapi_Entry *ec; int chg_type = LDAP_SYNC_NONE; - /* deque one element */ + /* dequeue one element */ PR_Lock(req->req_lock); qnode = req->ps_eq_head; + slapi_log_err(SLAPI_LOG_PLUGIN, SYNC_PLUGIN_SUBSYSTEM, "sync_queue_change - dequeue " + "\"%s\" \n", + slapi_entry_get_dn_const(qnode->sync_entry)); req->ps_eq_head = qnode->sync_next; if (NULL == req->ps_eq_head) { req->ps_eq_tail = NULL; @@ -665,10 +919,22 @@ sync_send_results(void *arg) sync_release_connection(req->req_pblock, conn, op, conn_acq_flag == 0); done: + /* This client closed the connection or shutdown, free the req */ sync_remove_request(req); PR_DestroyLock(req->req_lock); req->req_lock = NULL; - slapi_ch_free((void **)&req->req_pblock); + + slapi_pblock_get(req->req_pblock, SLAPI_SEARCH_ATTRS, &attrs_dup); + slapi_ch_array_free(attrs_dup); + slapi_pblock_set(req->req_pblock, SLAPI_SEARCH_ATTRS, NULL); + + slapi_pblock_get(req->req_pblock, SLAPI_SEARCH_STRFILTER, &strFilter); + slapi_ch_free((void **)&strFilter); + slapi_pblock_set(req->req_pblock, SLAPI_SEARCH_STRFILTER, NULL); + + slapi_pblock_destroy(req->req_pblock); + req->req_pblock = NULL; + slapi_ch_free((void **)&req->req_orig_base); slapi_filter_free(req->req_filter, 1); sync_cookie_free(&req->req_cookie); diff --git a/ldap/servers/plugins/sync/sync_util.c b/ldap/servers/plugins/sync/sync_util.c index 0ed7d3c..1070f94 100644 --- a/ldap/servers/plugins/sync/sync_util.c +++ b/ldap/servers/plugins/sync/sync_util.c @@ -7,6 +7,7 @@ * END COPYRIGHT BLOCK **/ #include "sync.h" +#include "slap.h" /* for LDAP_TAG_SK_REVERSE */ static struct berval *create_syncinfo_value(int type, const char *cookie, const char **uuids); static char *sync_cookie_get_server_info(Slapi_PBlock *pb); @@ -411,6 +412,37 @@ sync_cookie_get_client_info(Slapi_PBlock *pb) clientinfo = slapi_ch_smprintf("%s:%s:%s", clientdn, targetdn, strfilter); return (clientinfo); } + +/* This is used with internal search that reverse the order + * of returned entries. So to get + */ +static LDAPControl * +sync_build_sort_control(const char *attr) +{ + LDAPControl *ctrl; + BerElement *ber; + int rc; + + ber = ber_alloc(); + if (NULL == ber) + return NULL; + + rc = ber_printf(ber, "{{stb}}", attr, LDAP_TAG_SK_REVERSE, 1); + if (-1 == rc) { + ber_free(ber, 1); + return NULL; + } + + rc = slapi_build_control(LDAP_CONTROL_SORTREQUEST, ber, 1, &ctrl); + + ber_free(ber, 1); + + if (LDAP_SUCCESS != rc) + return NULL; + + return ctrl; +} + static unsigned long sync_cookie_get_change_number(int lastnr, const char *uniqueid) { @@ -419,11 +451,15 @@ sync_cookie_get_change_number(int lastnr, const char *uniqueid) Slapi_Entry *cl_entry; int rv; unsigned long newnr = SYNC_INVALID_CHANGENUM; + LDAPControl **ctrls = NULL; + + ctrls = (LDAPControl **)slapi_ch_calloc(2, sizeof(LDAPControl *)); char *filter = slapi_ch_smprintf("(&(changenumber>=%d)(targetuniqueid=%s))", lastnr, uniqueid); + ctrls[0] = sync_build_sort_control("changenumber"); srch_pb = slapi_pblock_new(); slapi_search_internal_set_pb(srch_pb, CL_SRCH_BASE, LDAP_SCOPE_SUBTREE, filter, - NULL, 0, NULL, NULL, plugin_get_default_component_id(), 0); + NULL, 0, ctrls, NULL, plugin_get_default_component_id(), 0); slapi_search_internal_pb(srch_pb); slapi_pblock_get(srch_pb, SLAPI_PLUGIN_INTOP_RESULT, &rv); if (rv == LDAP_SUCCESS) { @@ -436,6 +472,24 @@ sync_cookie_get_change_number(int lastnr, const char *uniqueid) slapi_attr_first_value(attr, &val); newnr = sync_number2ulong((char *)slapi_value_get_string(val)); } +#if DEBUG + slapi_log_err(SLAPI_LOG_PLUGIN, SYNC_PLUGIN_SUBSYSTEM, "sync_cookie_get_change_number looks for \"%s\"\n", + filter); + for (size_t i = 0; entries[i]; i++) { + Slapi_Attr *attr; + Slapi_Value *val; + char *entrydn; + unsigned long nr; + slapi_entry_attr_find(entries[i], CL_ATTR_ENTRYDN, &attr); + slapi_attr_first_value(attr, &val); + entrydn = (char *)slapi_value_get_string(val); + slapi_entry_attr_find(entries[i], CL_ATTR_CHANGENUMBER, &attr); + slapi_attr_first_value(attr, &val); + nr = sync_number2ulong((char *)slapi_value_get_string(val)); + slapi_log_err(SLAPI_LOG_PLUGIN, SYNC_PLUGIN_SUBSYSTEM, "sync_cookie_get_change_number after %d: %d %s\n", + lastnr, (int) nr, entrydn); + } +#endif } slapi_free_search_results_internal(srch_pb); -- To stop receiving notification emails like this one, please contact the administrator of this repository.

1 0

[389-ds-base] branch 389-ds-base-1.4.2 updated: Issue 51253 - dscreate should LDAPI to bootstrap the config
by pagure＠pagure.io 01 Sep '20

01 Sep '20

This is an automated email from the git hooks/post-receive script. mreynolds pushed a commit to branch 389-ds-base-1.4.2 in repository 389-ds-base. The following commit(s) were added to refs/heads/389-ds-base-1.4.2 by this push: new f856e52 Issue 51253 - dscreate should LDAPI to bootstrap the config f856e52 is described below commit f856e520a4f90527ff13a1cd364a5d3786976f6a Author: Mark Reynolds <mreynolds(a)redhat.com> AuthorDate: Sun Aug 30 20:17:28 2020 -0400 Issue 51253 - dscreate should LDAPI to bootstrap the config Description: There are cases where DNS is not setup yet, and trying to automate the installation fails. Using LDAPI bypasses this issue and allows for more robust deployment options relates: https://pagure.io/389-ds-base/issue/51253 Reviewed by: minfrin, firstyear, and tbordaz (Thanks!!!) --- ldap/admin/src/defaults.inf.in | 2 ++ ldap/admin/src/scripts/DSCreate.pm.in | 10 ++++++++++ ldap/admin/src/scripts/DSUtil.pm.in | 3 +++ ldap/admin/src/scripts/dscreate.map.in | 3 +++ ldap/admin/src/scripts/dsupdate.map.in | 3 +++ ldap/ldif/template-dse-minimal.ldif.in | 4 ++++ ldap/ldif/template-dse.ldif.in | 4 ++++ src/lib389/lib389/instance/setup.py | 22 +++++++++++----------- 8 files changed, 40 insertions(+), 11 deletions(-) diff --git a/ldap/admin/src/defaults.inf.in b/ldap/admin/src/defaults.inf.in index f749295..52903ce 100644 --- a/ldap/admin/src/defaults.inf.in +++ b/ldap/admin/src/defaults.inf.in @@ -37,6 +37,8 @@ local_state_dir = @localstatedir@ run_dir = @localstatedir@/run/dirsrv # This is the expected location of ldapi. ldapi = @localstatedir@/run/slapd-{instance_name}.socket +ldapi_listen = on +ldapi_autobind = on pid_file = @localstatedir@/run/dirsrv/slapd-{instance_name}.pid inst_dir = @serverdir@/slapd-{instance_name} plugin_dir = @serverplugindir@ diff --git a/ldap/admin/src/scripts/DSCreate.pm.in b/ldap/admin/src/scripts/DSCreate.pm.in index b586cef..6ef9d97 100644 --- a/ldap/admin/src/scripts/DSCreate.pm.in +++ b/ldap/admin/src/scripts/DSCreate.pm.in @@ -952,6 +952,16 @@ sub setDefaults { } } + if (!defined($inf->{slapd}->{ldapi})) { + $inf->{slapd}->{ldapi} = "$localstatedir/run/slapd-$servid.socket"; + } + if (!defined($inf->{slapd}->{ldapi_listen})) { + $inf->{slapd}->{ldapi_listen} = "on"; + } + if (!defined($inf->{slapd}->{ldapi_autobind})) { + $inf->{slapd}->{ldapi_autobind} = "on"; + } + if (!defined($inf->{slapd}->{bak_dir})) { if ("@with_fhs_opt@") { $inf->{slapd}->{bak_dir} = "$localstatedir/@PACKAGE_NAME@/slapd-$servid/bak"; diff --git a/ldap/admin/src/scripts/DSUtil.pm.in b/ldap/admin/src/scripts/DSUtil.pm.in index 197aafa..c994faa 100644 --- a/ldap/admin/src/scripts/DSUtil.pm.in +++ b/ldap/admin/src/scripts/DSUtil.pm.in @@ -964,6 +964,9 @@ sub createInfFromConfig { $inf->{slapd}->{RootDNPwd} = $ent->getValues('nsslapd-rootpw'); $inf->{slapd}->{ServerPort} = $ent->getValues('nsslapd-port'); $inf->{slapd}->{ServerIdentifier} = $id; + $inf->{slapd}->{ldapi} = $ent->getValues('nsslapd-ldapifilepath'); + $inf->{slapd}->{ldapi_listen} = $ent->getValues('nsslapd-ldapilisten'); + $inf->{slapd}->{ldapi_autobind} = $ent->getValues('nsslapd-ldapiautobind'); my $suffix = ""; $ent = $conn->search("cn=ldbm database,cn=plugins,cn=config", diff --git a/ldap/admin/src/scripts/dscreate.map.in b/ldap/admin/src/scripts/dscreate.map.in index fcf6e7a..354d332 100644 --- a/ldap/admin/src/scripts/dscreate.map.in +++ b/ldap/admin/src/scripts/dscreate.map.in @@ -37,3 +37,6 @@ log_dir = log_dir config_dir = config_dir db_dir = db_dir run_dir = run_dir +ldapi_enabled = ldapi_listen +ldapi = ldapi +ldapi_autobind = ldapi_autobind diff --git a/ldap/admin/src/scripts/dsupdate.map.in b/ldap/admin/src/scripts/dsupdate.map.in index 843fd34..e911a94 100644 --- a/ldap/admin/src/scripts/dsupdate.map.in +++ b/ldap/admin/src/scripts/dsupdate.map.in @@ -34,3 +34,6 @@ log_dir = log_dir config_dir = config_dir db_dir = db_dir run_dir = run_dir +ldapi_enabled = ldapi_listen +ldapi = ldapi +ldapi_autobind = ldapi_autobind diff --git a/ldap/ldif/template-dse-minimal.ldif.in b/ldap/ldif/template-dse-minimal.ldif.in index 0be9c17..0084e7e 100644 --- a/ldap/ldif/template-dse-minimal.ldif.in +++ b/ldap/ldif/template-dse-minimal.ldif.in @@ -20,6 +20,10 @@ nsslapd-auditlog: %log_dir%/audit nsslapd-auditfaillog: %log_dir%/audit nsslapd-rootdn: %rootdn% nsslapd-rootpw: %ds_passwd% +nsslapd-ldapilisten: %ldapi_enabled% +nsslapd-ldapifilepath: %ldapi% +nsslapd-ldapiautobind: %ldapi_autobind% +nsslapd-ldapimaprootdn: %rootdn% dn: cn=features,cn=config objectclass: top diff --git a/ldap/ldif/template-dse.ldif.in b/ldap/ldif/template-dse.ldif.in index 9efff58..92adc86 100644 --- a/ldap/ldif/template-dse.ldif.in +++ b/ldap/ldif/template-dse.ldif.in @@ -20,6 +20,10 @@ nsslapd-auditlog: %log_dir%/audit nsslapd-auditfaillog: %log_dir%/audit nsslapd-rootdn: %rootdn% nsslapd-rootpw: %ds_passwd% +nsslapd-ldapilisten: %ldapi_enabled% +nsslapd-ldapifilepath: %ldapi% +nsslapd-ldapiautobind: %ldapi_autobind% +nsslapd-ldapimaprootdn: %rootdn% dn: cn=encryption,cn=config objectClass: top diff --git a/src/lib389/lib389/instance/setup.py b/src/lib389/lib389/instance/setup.py index f0a8857..2d046a3 100644 --- a/src/lib389/lib389/instance/setup.py +++ b/src/lib389/lib389/instance/setup.py @@ -730,6 +730,7 @@ class SetupDs(object): dse += line.replace('%', '{', 1).replace('%', '}', 1) with open(os.path.join(slapd['config_dir'], 'dse.ldif'), 'w') as file_dse: + ldapi_path = os.path.join(slapd['local_state_dir'], "run/slapd-%s.socket" % slapd['instance_name']) file_dse.write(dse.format( schema_dir=slapd['schema_dir'], lock_dir=slapd['lock_dir'], @@ -745,11 +746,15 @@ class SetupDs(object): ds_user=slapd['user'], rootdn=slapd['root_dn'], ds_passwd=self._secure_password, # We set our own password here, so we can connect and mod. - # This is because we never know the users input root password as they can validily give + # This is because we never know the users input root password as they can validly give # us a *hashed* input. ds_suffix=ds_suffix, config_dir=slapd['config_dir'], db_dir=slapd['db_dir'], + db_home_dir=slapd['db_home_dir'], + ldapi_enabled="on", + ldapi=ldapi_path, + ldapi_autobind="on", )) # Create all the needed paths @@ -838,7 +843,7 @@ class SetupDs(object): # it's the only stable and guaranteed way to connect to the instance # at this point. # - # Alternately, we could use ldapi instead, which would prevent the need + # Use ldapi which would prevent the need # to configure a temp root pw in the setup phase. args = { SER_HOST: "localhost", @@ -846,7 +851,10 @@ class SetupDs(object): SER_SERVERID_PROP: slapd['instance_name'], SER_ROOT_DN: slapd['root_dn'], SER_ROOT_PW: self._raw_secure_password, - SER_DEPLOYED_DIR: slapd['prefix'] + SER_DEPLOYED_DIR: slapd['prefix'], + SER_LDAPI_ENABLED: 'on', + SER_LDAPI_SOCKET: ldapi_path, + SER_LDAPI_AUTOBIND: 'on' } ds_instance.allocate(args) @@ -949,14 +957,6 @@ class SetupDs(object): # Unsupported rdn raise ValueError("Suffix RDN '{}' in '{}' is not supported. Supported RDN's are: 'c', 'cn', 'dc', 'o', and 'ou'".format(suffix_rdn_attr, backend['nsslapd-suffix'])) - # Initialise ldapi socket information. IPA expects this .... - ldapi_path = os.path.join(slapd['local_state_dir'], "run/slapd-%s.socket" % slapd['instance_name']) - ds_instance.config.set('nsslapd-ldapifilepath', ldapi_path) - ds_instance.config.set('nsslapd-ldapilisten', 'on') - ds_instance.config.set('nsslapd-ldapiautobind', 'on') - ds_instance.config.set('nsslapd-ldapimaprootdn', slapd['root_dn']) - - # Create all required sasl maps: if we have a single backend ... # our default maps are really really bad, and we should feel bad. # they basically only work with a single backend, and they'll break -- To stop receiving notification emails like this one, please contact the administrator of this repository.

1 0

[389-ds-base] branch 389-ds-base-1.4.3 updated: Issue 51253 - dscreate should LDAPI to bootstrap the config
by pagure＠pagure.io 01 Sep '20

01 Sep '20

This is an automated email from the git hooks/post-receive script. mreynolds pushed a commit to branch 389-ds-base-1.4.3 in repository 389-ds-base. The following commit(s) were added to refs/heads/389-ds-base-1.4.3 by this push: new 65c01bb Issue 51253 - dscreate should LDAPI to bootstrap the config 65c01bb is described below commit 65c01bbae365aeb30f9817d31ec38eead730a22e Author: Mark Reynolds <mreynolds(a)redhat.com> AuthorDate: Sun Aug 30 20:17:28 2020 -0400 Issue 51253 - dscreate should LDAPI to bootstrap the config Description: There are cases where DNS is not setup yet, and trying to automate the installation fails. Using LDAPI bypasses this issue and allows for more robust deployment options relates: https://pagure.io/389-ds-base/issue/51253 Reviewed by: minfrin, firstyear, and tbordaz (Thanks!!!) --- ldap/admin/src/defaults.inf.in | 2 ++ ldap/admin/src/scripts/DSCreate.pm.in | 9 +++++++++ ldap/admin/src/scripts/DSUtil.pm.in | 3 +++ ldap/admin/src/scripts/dscreate.map.in | 3 +++ ldap/admin/src/scripts/dsupdate.map.in | 3 +++ ldap/ldif/template-dse-minimal.ldif.in | 4 ++++ ldap/ldif/template-dse.ldif.in | 4 ++++ src/lib389/lib389/instance/setup.py | 23 +++++++++++------------ 8 files changed, 39 insertions(+), 12 deletions(-) diff --git a/ldap/admin/src/defaults.inf.in b/ldap/admin/src/defaults.inf.in index 2f630f9..e67d65e 100644 --- a/ldap/admin/src/defaults.inf.in +++ b/ldap/admin/src/defaults.inf.in @@ -38,6 +38,8 @@ local_state_dir = @localstatedir@ run_dir = @localstatedir@/run/dirsrv # This is the expected location of ldapi. ldapi = @localstatedir@/run/slapd-{instance_name}.socket +ldapi_listen = on +ldapi_autobind = on pid_file = @localstatedir@/run/dirsrv/slapd-{instance_name}.pid inst_dir = @serverdir@/slapd-{instance_name} plugin_dir = @serverplugindir@ diff --git a/ldap/admin/src/scripts/DSCreate.pm.in b/ldap/admin/src/scripts/DSCreate.pm.in index a2f15f9..ad619db 100644 --- a/ldap/admin/src/scripts/DSCreate.pm.in +++ b/ldap/admin/src/scripts/DSCreate.pm.in @@ -956,6 +956,15 @@ sub setDefaults { if (!defined($inf->{slapd}->{db_home_dir})) { $inf->{slapd}->{db_home_dir} = $inf->{slapd}->{db_dir}; } + if (!defined($inf->{slapd}->{ldapi})) { + $inf->{slapd}->{ldapi} = "$localstatedir/run/slapd-$servid.socket"; + } + if (!defined($inf->{slapd}->{ldapi_listen})) { + $inf->{slapd}->{ldapi_listen} = "on"; + } + if (!defined($inf->{slapd}->{ldapi_autobind})) { + $inf->{slapd}->{ldapi_autobind} = "on"; + } if (!defined($inf->{slapd}->{bak_dir})) { if ("@with_fhs_opt@") { diff --git a/ldap/admin/src/scripts/DSUtil.pm.in b/ldap/admin/src/scripts/DSUtil.pm.in index 197aafa..c994faa 100644 --- a/ldap/admin/src/scripts/DSUtil.pm.in +++ b/ldap/admin/src/scripts/DSUtil.pm.in @@ -964,6 +964,9 @@ sub createInfFromConfig { $inf->{slapd}->{RootDNPwd} = $ent->getValues('nsslapd-rootpw'); $inf->{slapd}->{ServerPort} = $ent->getValues('nsslapd-port'); $inf->{slapd}->{ServerIdentifier} = $id; + $inf->{slapd}->{ldapi} = $ent->getValues('nsslapd-ldapifilepath'); + $inf->{slapd}->{ldapi_listen} = $ent->getValues('nsslapd-ldapilisten'); + $inf->{slapd}->{ldapi_autobind} = $ent->getValues('nsslapd-ldapiautobind'); my $suffix = ""; $ent = $conn->search("cn=ldbm database,cn=plugins,cn=config", diff --git a/ldap/admin/src/scripts/dscreate.map.in b/ldap/admin/src/scripts/dscreate.map.in index 4c47b08..fd6d3e8 100644 --- a/ldap/admin/src/scripts/dscreate.map.in +++ b/ldap/admin/src/scripts/dscreate.map.in @@ -39,3 +39,6 @@ db_dir = db_dir db_home_dir = db_home_dir run_dir = run_dir instance_name = ServerIdentifier +ldapi_enabled = ldapi_listen +ldapi = ldapi +ldapi_autobind = ldapi_autobind diff --git a/ldap/admin/src/scripts/dsupdate.map.in b/ldap/admin/src/scripts/dsupdate.map.in index f6912b6..429b742 100644 --- a/ldap/admin/src/scripts/dsupdate.map.in +++ b/ldap/admin/src/scripts/dsupdate.map.in @@ -35,3 +35,6 @@ config_dir = config_dir db_dir = db_dir db_home_dir = db_home_dir run_dir = run_dir +ldapi_enabled = ldapi_listen +ldapi = ldapi +ldapi_autobind = ldapi_autobind diff --git a/ldap/ldif/template-dse-minimal.ldif.in b/ldap/ldif/template-dse-minimal.ldif.in index 0be9c17..0084e7e 100644 --- a/ldap/ldif/template-dse-minimal.ldif.in +++ b/ldap/ldif/template-dse-minimal.ldif.in @@ -20,6 +20,10 @@ nsslapd-auditlog: %log_dir%/audit nsslapd-auditfaillog: %log_dir%/audit nsslapd-rootdn: %rootdn% nsslapd-rootpw: %ds_passwd% +nsslapd-ldapilisten: %ldapi_enabled% +nsslapd-ldapifilepath: %ldapi% +nsslapd-ldapiautobind: %ldapi_autobind% +nsslapd-ldapimaprootdn: %rootdn% dn: cn=features,cn=config objectclass: top diff --git a/ldap/ldif/template-dse.ldif.in b/ldap/ldif/template-dse.ldif.in index 19abcf8..2cfc985 100644 --- a/ldap/ldif/template-dse.ldif.in +++ b/ldap/ldif/template-dse.ldif.in @@ -20,6 +20,10 @@ nsslapd-auditlog: %log_dir%/audit nsslapd-auditfaillog: %log_dir%/audit nsslapd-rootdn: %rootdn% nsslapd-rootpw: %ds_passwd% +nsslapd-ldapilisten: %ldapi_enabled% +nsslapd-ldapifilepath: %ldapi% +nsslapd-ldapiautobind: %ldapi_autobind% +nsslapd-ldapimaprootdn: %rootdn% dn: cn=encryption,cn=config objectClass: top diff --git a/src/lib389/lib389/instance/setup.py b/src/lib389/lib389/instance/setup.py index e1862f0..7d42ba2 100644 --- a/src/lib389/lib389/instance/setup.py +++ b/src/lib389/lib389/instance/setup.py @@ -731,6 +731,7 @@ class SetupDs(object): dse += line.replace('%', '{', 1).replace('%', '}', 1) with open(os.path.join(slapd['config_dir'], 'dse.ldif'), 'w') as file_dse: + ldapi_path = os.path.join(slapd['local_state_dir'], "run/slapd-%s.socket" % slapd['instance_name']) dse_fmt = dse.format( schema_dir=slapd['schema_dir'], lock_dir=slapd['lock_dir'], @@ -747,12 +748,15 @@ class SetupDs(object): rootdn=slapd['root_dn'], instance_name=slapd['instance_name'], ds_passwd=self._secure_password, # We set our own password here, so we can connect and mod. - # This is because we never know the users input root password as they can validily give + # This is because we never know the users input root password as they can validly give # us a *hashed* input. ds_suffix=ds_suffix, config_dir=slapd['config_dir'], db_dir=slapd['db_dir'], - db_home_dir=slapd['db_home_dir'] + db_home_dir=slapd['db_home_dir'], + ldapi_enabled="on", + ldapi=ldapi_path, + ldapi_autobind="on", ) file_dse.write(dse_fmt) @@ -842,7 +846,7 @@ class SetupDs(object): # it's the only stable and guaranteed way to connect to the instance # at this point. # - # Alternately, we could use ldapi instead, which would prevent the need + # Use ldapi which would prevent the need # to configure a temp root pw in the setup phase. args = { SER_HOST: "localhost", @@ -850,7 +854,10 @@ class SetupDs(object): SER_SERVERID_PROP: slapd['instance_name'], SER_ROOT_DN: slapd['root_dn'], SER_ROOT_PW: self._raw_secure_password, - SER_DEPLOYED_DIR: slapd['prefix'] + SER_DEPLOYED_DIR: slapd['prefix'], + SER_LDAPI_ENABLED: 'on', + SER_LDAPI_SOCKET: ldapi_path, + SER_LDAPI_AUTOBIND: 'on' } ds_instance.allocate(args) @@ -954,14 +961,6 @@ class SetupDs(object): # Unsupported rdn raise ValueError("Suffix RDN '{}' in '{}' is not supported. Supported RDN's are: 'c', 'cn', 'dc', 'o', and 'ou'".format(suffix_rdn_attr, backend['nsslapd-suffix'])) - # Initialise ldapi socket information. IPA expects this .... - ldapi_path = os.path.join(slapd['local_state_dir'], "run/slapd-%s.socket" % slapd['instance_name']) - ds_instance.config.set('nsslapd-ldapifilepath', ldapi_path) - ds_instance.config.set('nsslapd-ldapilisten', 'on') - ds_instance.config.set('nsslapd-ldapiautobind', 'on') - ds_instance.config.set('nsslapd-ldapimaprootdn', slapd['root_dn']) - - # Create all required sasl maps: if we have a single backend ... # our default maps are really really bad, and we should feel bad. # they basically only work with a single backend, and they'll break -- To stop receiving notification emails like this one, please contact the administrator of this repository.

1 0

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

389-commits September 2020