Hi Lukas,
On 23 March 2016 at 15:58, Lukas Slebodnik lslebodn@redhat.com wrote:
On (23/03/16 15:49), Patrick Coleman wrote:
$ grep network_status_change_cb sssd.log | grep '06:59:12' | wc -l 1245
I believe the root cause of our problem is that sssd is receiving more than a thousand netlink notifications a second - because of IPv6 route updates - per above. While this might be something we can reduce on our end, it would be great if there was some way to unsubscribe sssd from receiving these.
Any ideas here appreciated - eg. if netlink is only used to get a more rapid notification of network changes, could we remove the call to setup_netlink in src/monitor/monitor.c in a local build? Does sssd rely on netlink for anything else?
(Wed Mar 23 06:59:07 2016) [sssd] [route_msg_debug_print] (0x1000): route idx 209591 flags 0X200 family 10 addr fd0a:9b09:1f7:0:218:aff:fe33:3b66/128 (Wed Mar 23 06:59:07 2016) [sssd] [network_status_change_cb] (0x2000): A networking status change detected signaling providers to reset offline status
If I interpret this correctly, every single netlink notification seems to be causing sssd to send a message over dbus?
The real error is in sssd_$domain.log
neither sssd.log nor sssd_nss.log will help you.
I see nothing wrong in the domain logs. I've included samples of normal operation before the failure this morning in the domain log below in any case. If there's anything further I can provide let me know.
Cheers,
Patrick
(Wed Mar 23 06:54:04 2016) [sssd[be[ourdomain]]] [sdap_initgr_rfc2307_next_base] (0x0400): Searching for groups with base [dc=meraki,dc=com] (Wed Mar 23 06:54:04 2016) [sssd[be[ourdomain]]] [sdap_get_generic_ext_step] (0x0400): calling ldap_search_ext with [(&(memberuid=blinken)(objectClass=posixGroup)(cn=*)(&(gidNumber=*)(!(gidNumber=0))))][dc=meraki,dc=com]. (Wed Mar 23 06:54:04 2016) [sssd[be[ourdomain]]] [sdap_get_generic_ext_done] (0x0400): Search result: Success(0), no errmsg set (Wed Mar 23 06:54:04 2016) [sssd[be[ourdomain]]] [sdap_get_groups_next_base] (0x0400): Searching for groups with base [dc=meraki,dc=com] (Wed Mar 23 06:54:04 2016) [sssd[be[ourdomain]]] [sdap_get_generic_ext_step] (0x0400): calling ldap_search_ext with [(&(gidNumber=1111)(objectClass=posixGroup)(cn=*)(&(gidNumber=*)(!(gidNumber=0))))][dc=meraki,dc=com]. (Wed Mar 23 06:54:04 2016) [sssd[be[meraki]]] [sdap_get_generic_ext_done] (0x0400): Search result: Success(0), no errmsg set (Wed Mar 23 06:54:04 2016) [sssd[be[ourdomain]]] [sdap_get_groups_process] (0x0400): Search for groups, returned 1 results. (Wed Mar 23 06:54:04 2016) [sssd[be[ourdomain]]] [sdap_attrs_get_sid_str] (0x0080): No [objectSID] attribute while id-mapping. [0][Success] (Wed Mar 23 06:54:04 2016) [sssd[be[ourdomain]]] [sdap_get_primary_name] (0x0400): Processing object blinken (Wed Mar 23 06:54:04 2016) [sssd[be[ourdomain]]] [sdap_save_group] (0x0400): Processing group blinken (Wed Mar 23 06:54:04 2016) [sssd[be[ourdomain]]] [sdap_save_group] (0x0400): Original USN value is not available for [blinken]. (Wed Mar 23 06:54:04 2016) [sssd[be[ourdomain]]] [sdap_process_ghost_members] (0x0400): Group has 0 members (Wed Mar 23 06:54:04 2016) [sssd[be[ourdomain]]] [sdap_save_group] (0x0400): Storing info for group blinken (Wed Mar 23 06:54:04 2016) [sssd[be[ourdomain]]] [sysdb_search_group_by_name] (0x0400): No such entry (Wed Mar 23 06:54:04 2016) [sssd[be[ourdomain]]] [sysdb_search_group_by_gid] (0x0400): No such entry (Wed Mar 23 06:54:04 2016) [sssd[be[ourdomain]]] [acctinfo_callback] (0x0100): Request processed. Returned 0,0,Success (Wed Mar 23 06:54:04 2016) [sssd[be[ourdomain]]] [be_get_account_info] (0x0100): Got request for [4099][1][name=blinken] (Wed Mar 23 06:54:04 2016) [sssd[be[ourdomain]]] [be_req_set_domain] (0x0400): Changing request domain from [ourdomain] to [ourdomain] (Wed Mar 23 06:54:04 2016) [sssd[be[ourdomain]]] [sdap_get_initgr_next_base] (0x0400): Searching for users with base [dc=meraki,dc=com] (Wed Mar 23 06:54:04 2016) [sssd[be[ourdomain]]] [sdap_get_generic_ext_step] (0x0400): calling ldap_search_ext with [(&(uid=blinken)(objectclass=posixAccount)(&(uidNumber=*)(!(uidNumber=0))))][dc=meraki,dc=com]. (Wed Mar 23 06:54:04 2016) [sssd[be[ourdomain]]] [sdap_get_generic_ext_done] (0x0400): Search result: Success(0), no errmsg set (Wed Mar 23 06:54:04 2016) [sssd[be[ourdomain]]] [sdap_save_user] (0x0400): Save user (Wed Mar 23 06:54:04 2016) [sssd[be[ourdomain]]] [sdap_attrs_get_sid_str] (0x0080): No [objectSID] attribute while id-mapping. [0][Success] (Wed Mar 23 06:54:04 2016) [sssd[be[ourdomain]]] [sdap_get_primary_name] (0x0400): Processing object blinken (Wed Mar 23 06:54:04 2016) [sssd[be[ourdomain]]] [sdap_save_user] (0x0400): Processing user blinken (Wed Mar 23 06:54:04 2016) [sssd[be[ourdomain]]] [sdap_save_user] (0x0400): Original memberOf is not available for [blinken]. (Wed Mar 23 06:54:04 2016) [sssd[be[ourdomain]]] [sdap_save_user] (0x0400): Original USN value is not available for [blinken]. (Wed Mar 23 06:54:04 2016) [sssd[be[ourdomain]]] [sdap_save_user] (0x0400): User principal is not available for [blinken]. (Wed Mar 23 06:54:04 2016) [sssd[be[ourdomain]]] [sdap_save_user] (0x0400): Storing info for user blinken (Wed Mar 23 06:54:04 2016) [sssd[be[ourdomain]]] [sysdb_search_user_by_name] (0x0400): No such entry (Wed Mar 23 06:54:04 2016) [sssd[be[ourdomain]]] [sysdb_search_user_by_uid] (0x0400): No such entry (Wed Mar 23 06:54:05 2016) [sssd[be[ourdomain]]] [sdap_initgr_rfc2307_next_base] (0x0400): Searching for groups with base [dc=meraki,dc=com] (Wed Mar 23 06:54:05 2016) [sssd[be[ourdomain]]] [sdap_get_generic_ext_step] (0x0400): calling ldap_search_ext with [(&(memberuid=blinken)(objectClass=posixGroup)(cn=*)(&(gidNumber=*)(!(gidNumber=0))))][dc=meraki,dc=com]. (Wed Mar 23 06:54:05 2016) [sssd[be[ourdomain]]] [sdap_get_generic_ext_done] (0x0400): Search result: Success(0), no errmsg set (Wed Mar 23 06:54:05 2016) [sssd[be[ourdomain]]] [sdap_get_groups_next_base] (0x0400): Searching for groups with base [dc=meraki,dc=com] (Wed Mar 23 06:54:05 2016) [sssd[be[ourdomain]]] [sdap_get_generic_ext_step] (0x0400): calling ldap_search_ext with [(&(gidNumber=1111)(objectClass=posixGroup)(cn=*)(&(gidNumber=*)(!(gidNumber=0))))][dc=meraki,dc=com]. (Wed Mar 23 06:54:05 2016) [sssd[be[ourdomain]]] [sdap_get_generic_ext_done] (0x0400): Search result: Success(0), no errmsg set (Wed Mar 23 06:54:05 2016) [sssd[be[ourdomain]]] [sdap_get_groups_process] (0x0400): Search for groups, returned 1 results. (Wed Mar 23 06:54:05 2016) [sssd[be[ourdomain]]] [sdap_attrs_get_sid_str] (0x0080): No [objectSID] attribute while id-mapping. [0][Success] (Wed Mar 23 06:54:05 2016) [sssd[be[ourdomain]]] [sdap_get_primary_name] (0x0400): Processing object blinken (Wed Mar 23 06:54:05 2016) [sssd[be[ourdomain]]] [sdap_save_group] (0x0400): Processing group blinken (Wed Mar 23 06:54:05 2016) [sssd[be[ourdomain]]] [sdap_save_group] (0x0400): Original USN value is not available for [blinken]. (Wed Mar 23 06:54:05 2016) [sssd[be[ourdomain]]] [sdap_process_ghost_members] (0x0400): Group has 0 members (Wed Mar 23 06:54:05 2016) [sssd[be[ourdomain]]] [sdap_save_group] (0x0400): Storing info for group blinken (Wed Mar 23 06:54:05 2016) [sssd[be[ourdomain]]] [sysdb_search_group_by_name] (0x0400): No such entry (Wed Mar 23 06:54:05 2016) [sssd[be[ourdomain]]] [sysdb_search_group_by_gid] (0x0400): No such entry