cluster-commits March 2013

cluster-commits@lists.stg.fedorahosted.org

6 participants
16 discussions

cluster: RHEL510 - GFS2: Fix permissions on GFS2 man pages
by Bob Peterson 22 Mar '13

22 Mar '13

Gitweb: http://git.fedorahosted.org/git/?p=cluster.git;a=commitdiff;h=fd8905389d16f… Commit: fd8905389d16ff6f5fbaaf525b6fbcdf2d283733 Parent: a2f0c0138865d6df3741feb3d753969aa3495c77 Author: Bob Peterson <rpeterso(a)redhat.com> AuthorDate: Mon Dec 10 14:00:23 2012 -0600 Committer: Bob Peterson <rpeterso(a)redhat.com> CommitterDate: Fri Mar 22 12:22:04 2013 -0700 GFS2: Fix permissions on GFS2 man pages This patch changes the Makefile for the GFS2 man pages so that it installs them with mode 644 rather than the default. rhbz#883864 --- gfs2/man/Makefile | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/gfs2/man/Makefile b/gfs2/man/Makefile index a0f5571..9318fdd 100644 --- a/gfs2/man/Makefile +++ b/gfs2/man/Makefile @@ -34,7 +34,7 @@ include ${top_srcdir}/make/defines.mk install: install -d ${mandir}/man8 - install ${TARGET8} ${mandir}/man8 + install -m 644 ${TARGET8} ${mandir}/man8 ln -f ${mandir}/man8/${FSCK} ${mandir}/man8/gfs2_fsck.8 ln -f ${mandir}/man8/${MOUNT} ${mandir}/man8/gfs2_mount.8

1 0

cluster: RHEL59 - cman|fenced: Fix node killing in case of a 2node cluster that suffers brief network outage
by Fabio M. Di Nitto 20 Mar '13

20 Mar '13

Gitweb: http://git.fedorahosted.org/git/?p=cluster.git;a=commitdiff;h=444a46c80e3ee… Commit: 444a46c80e3ee12544eb8960fdea9f7cefc78d1e Parent: a1e906313b85a6d0451feda7a6222cfd4386caf5 Author: Christine Caulfield <ccaulfie(a)redhat.com> AuthorDate: Fri Jan 11 13:45:50 2013 +0000 Committer: Fabio M. Di Nitto <fdinitto(a)redhat.com> CommitterDate: Wed Mar 20 17:25:16 2013 +0100 cman|fenced: Fix node killing in case of a 2node cluster that suffers brief network outage This patch fixes a rare but nasty condition in cman and fenced. In a 2node cluster if the network splits for a period of time longer than the token timeout but shorter than the time needed to fence a node then both nodes can send 'kill' packets to the other with the effect that both nodes' cmans will quit leaving no operational cluster. This patch adds a check for a 2node cluster and only sends a 'kill' packet to the node with the higher nodeid thus ensuring a predictable response to such events and ensuring that services can continue to run. rhbz#923861 Signed-off-by: Christine Caulfield <ccaulfie(a)redhat.com> Acked-By: Fabio M. Di Nitto <fdinitto(a)redhat.com> --- cman/daemon/commands.c | 27 ++++++++++++++++++++++++--- 1 files changed, 24 insertions(+), 3 deletions(-) diff --git a/cman/daemon/commands.c b/cman/daemon/commands.c index 1fafdac..6f3ec4e 100644 --- a/cman/daemon/commands.c +++ b/cman/daemon/commands.c @@ -1829,10 +1829,31 @@ static void do_process_transition(int nodeid, char *data, int len) /* Don't duplicate messages */ if (node->state != NODESTATE_AISONLY) { if (cluster_is_quorate) { - P_MEMB("Killing node %s because it has rejoined the cluster with existing state", node->name); - log_printf(LOG_CRIT, "Killing node %s because it has rejoined the cluster with existing state", node->name); node->state = NODESTATE_AISONLY; - send_kill(nodeid, CLUSTER_KILL_REJOIN); + /* Oh, this gets even more complicated. Don't send a KILL message if we are in a two_node + * cluster and that node has a lower node ID than us. + * This allows fencing time to startup and caters for the situation where + * a node rejoins REALLY quickly, before fencing has had time to work. + * I've split this up a bit partly for clarity, but mainly so allow us to + * print out helpful messages as to what we are up to here. + */ + if (two_node) { + if (node->node_id > us->node_id) { + log_printf(LOG_CRIT, "Killing node %s because it has rejoined the cluster with existing state and has higher node ID", node->name); + P_MEMB("Killing node %s because it has rejoined the cluster with existing state and has higher node ID", node->name); + send_kill(nodeid, CLUSTER_KILL_REJOIN); + } + else { + log_printf(LOG_CRIT, "Not killing node %s despite it rejoining the cluster with existing state, it has a lower node ID", node->name); + P_MEMB("Not killing node %s despite it rejoining the cluster with existing state, it has a lower node ID", node->name); + } + } + else { + log_printf(LOG_CRIT, "Killing node %s because it has rejoined the cluster with existing state", node->name); + P_MEMB("Killing node %s because it has rejoined the cluster with existing state", node->name); + send_kill(nodeid, CLUSTER_KILL_REJOIN); + } + } else { P_MEMB("Node %s not joined to cman because it has existing state", node->name);

1 0

cluster: STABLE32 - qdiskd: Do not count missed updates from offline nodes
by jruemker 13 Mar '13

13 Mar '13

Gitweb: http://git.fedorahosted.org/git/?p=cluster.git;a=commitdiff;h=6ac07a7ec331e… Commit: 6ac07a7ec331e8f9c1b238e49144731cd48c0150 Parent: 4eb1dcce90bb4d43cf4268d5310f95e12a862ba7 Author: John Ruemker <jruemker(a)redhat.com> AuthorDate: Wed Mar 13 11:17:41 2013 -0400 Committer: John Ruemker <jruemker(a)redhat.com> CommitterDate: Wed Mar 13 11:17:41 2013 -0400 qdiskd: Do not count missed updates from offline nodes After qdiskd initialization, nodes that have not yet been online may have missed updates counted, leading to eviction. This change skips counting missed updates for nodes with state S_NONE. Resolves: rhbz#920358 --- cman/qdisk/main.c | 14 ++++++++------ 1 files changed, 8 insertions(+), 6 deletions(-) diff --git a/cman/qdisk/main.c b/cman/qdisk/main.c index 68e72c7..4d4397a 100644 --- a/cman/qdisk/main.c +++ b/cman/qdisk/main.c @@ -198,12 +198,14 @@ read_node_blocks(qd_ctx *ctx, node_info_t *ni, int max) /* Unchanged timestamp: miss */ if ((sb->ps_timestamp == ni[x].ni_last_seen) && (ni[x].ni_state != S_EXIT)) { - /* XXX check for average + allow grace */ - ni[x].ni_misses++; - if (ni[x].ni_misses > 1) { - logt_print(LOG_DEBUG, - "Node %d missed an update (%d/%d)\n", - x+1, ni[x].ni_misses, ctx->qc_tko); + if (ni[x].ni_state != S_NONE) { + /* XXX check for average + allow grace */ + ni[x].ni_misses++; + if (ni[x].ni_misses > 1) { + logt_print(LOG_DEBUG, + "Node %d missed an update (%d/%d)\n", + x+1, ni[x].ni_misses, ctx->qc_tko); + } } continue; }

1 0

gfs2-utils: master - libgfs2: Remove gfs2_next_rg_meta
by Andrew Price 11 Mar '13

11 Mar '13

Gitweb: http://git.fedorahosted.org/git/?p=gfs2-utils.git;a=commitdiff;h=9c6977dae8… Commit: 9c6977dae8082767f07f96f1c29672556fa062bf Parent: 874f1c26c7398c085bd14fadc2c6b6d524346c92 Author: Andrew Price <anprice(a)redhat.com> AuthorDate: Mon Mar 11 15:07:47 2013 +0000 Committer: Andrew Price <anprice(a)redhat.com> CommitterDate: Mon Mar 11 15:20:58 2013 +0000 libgfs2: Remove gfs2_next_rg_meta This removes gfs2_next_rg_meta and gfs2_next_rg_metatype from libgfs2 and switches gfs2_edit to use lgfs2_bm_scan. The functions are pushed into gfs2_convert rather switching it to lgfs2_bm_scan to provide identical functionality in the absence of a gfs1 fs to test it with. Signed-off-by: Andrew Price <anprice(a)redhat.com> --- gfs2/convert/gfs2_convert.c | 52 +++++++++++++++++++++++++++- gfs2/edit/hexedit.c | 54 +++++++++++++++++++---------- gfs2/edit/savemeta.c | 80 ++++++++++++++----------------------------- gfs2/libgfs2/libgfs2.h | 5 --- gfs2/libgfs2/structures.c | 76 ---------------------------------------- 5 files changed, 113 insertions(+), 154 deletions(-) diff --git a/gfs2/convert/gfs2_convert.c b/gfs2/convert/gfs2_convert.c index 235c95c..d286512 100644 --- a/gfs2/convert/gfs2_convert.c +++ b/gfs2/convert/gfs2_convert.c @@ -999,6 +999,56 @@ static int adjust_inode(struct gfs2_sbd *sbp, struct gfs2_buffer_head *bh) return 0; } /* adjust_inode */ +static int next_rg_meta(struct rgrp_tree *rgd, uint64_t *block, int first) +{ + struct gfs2_bitmap *bits = NULL; + uint32_t length = rgd->ri.ri_length; + uint32_t blk = (first)? 0: (uint32_t)((*block + 1) - rgd->ri.ri_data0); + int i; + + if (!first && (*block < rgd->ri.ri_data0)) { + fprintf(stderr, "next_rg_meta: Start block is outside rgrp bounds.\n"); + exit(1); + } + for (i = 0; i < length; i++){ + bits = &rgd->bits[i]; + if (blk < bits->bi_len * GFS2_NBBY) + break; + blk -= bits->bi_len * GFS2_NBBY; + } + for (; i < length; i++){ + bits = &rgd->bits[i]; + blk = gfs2_bitfit((unsigned char *)rgd->bh[i]->b_data + + bits->bi_offset, bits->bi_len, blk, GFS2_BLKST_DINODE); + if(blk != BFITNOENT){ + *block = blk + (bits->bi_start * GFS2_NBBY) + + rgd->ri.ri_data0; + break; + } + blk = 0; + } + if (i == length) + return -1; + return 0; +} + +static int next_rg_metatype(struct gfs2_sbd *sdp, struct rgrp_tree *rgd, + uint64_t *block, uint32_t type, int first) +{ + struct gfs2_buffer_head *bh = NULL; + + do{ + if (bh) + brelse(bh); + if (next_rg_meta(rgd, block, first)) + return -1; + bh = bread(sdp, *block); + first = 0; + } while(gfs2_check_meta(bh, type)); + brelse(bh); + return 0; +} + /* ------------------------------------------------------------------------- */ /* inode_renumber - renumber the inodes */ /* */ @@ -1046,7 +1096,7 @@ static int inode_renumber(struct gfs2_sbd *sbp, uint64_t root_inode_addr, osi_li /* be "11" (used meta) for both inodes and indirect blocks. */ /* We need to process the inodes and change the indirect blocks */ /* to have a bitmap type of "01" (data). */ - if (gfs2_next_rg_metatype(sbp, rgd, &block, 0, first)) + if (next_rg_metatype(sbp, rgd, &block, 0, first)) break; /* If this is the root inode block, remember it for later: */ if (block == root_inode_addr) { diff --git a/gfs2/edit/hexedit.c b/gfs2/edit/hexedit.c index 4856181..281be5a 100644 --- a/gfs2/edit/hexedit.c +++ b/gfs2/edit/hexedit.c @@ -1569,6 +1569,31 @@ static uint64_t find_metablockoftype_slow(uint64_t startblk, int metatype, int p return blk; } +static int find_rg_metatype(struct rgrp_tree *rgd, uint64_t *blk, uint64_t startblk, int mtype) +{ + int found; + unsigned i, j, m; + struct gfs2_buffer_head *bhp = NULL; + uint64_t *ibuf = malloc(sbd.bsize * GFS2_NBBY * sizeof(uint64_t)); + + for (i = 0; i < rgd->ri.ri_length; i++) { + m = lgfs2_bm_scan(rgd, i, ibuf, GFS2_BLKST_DINODE); + + for (j = 0; j < m; j++) { + *blk = ibuf[j]; + bhp = bread(&sbd, *blk); + found = (*blk > startblk) && !gfs2_check_meta(bhp, mtype); + brelse(bhp); + if (found) { + free(ibuf); + return 0; + } + } + } + free(ibuf); + return -1; +} + /* ------------------------------------------------------------------------ */ /* Find next "metadata in use" block AFTER a given point in the fs */ /* */ @@ -1578,7 +1603,7 @@ static uint64_t find_metablockoftype_slow(uint64_t startblk, int metatype, int p /* ------------------------------------------------------------------------ */ static uint64_t find_metablockoftype_rg(uint64_t startblk, int metatype, int print) { - struct osi_node *n, *next = NULL; + struct osi_node *next = NULL; uint64_t blk, errblk; int first = 1, found = 0; struct rgrp_tree *rgd; @@ -1586,9 +1611,8 @@ static uint64_t find_metablockoftype_rg(uint64_t startblk, int metatype, int pri blk = 0; /* Skip the rgs prior to the block we've been given */ - for (n = osi_first(&sbd.rgtree); n; n = next) { - next = osi_next(n); - rgd = (struct rgrp_tree *)n; + for (next = osi_first(&sbd.rgtree); next; next = osi_next(next)) { + rgd = (struct rgrp_tree *)next; ri = &rgd->ri; if (first && startblk <= ri->ri_data0) { startblk = ri->ri_data0; @@ -1607,25 +1631,19 @@ static uint64_t find_metablockoftype_rg(uint64_t startblk, int metatype, int pri if (print) exit(-1); } - for (; !found && n; n = next){ - next = osi_next(n); - rgd = (struct rgrp_tree *)n; + for (; !found && next; next = osi_next(next)){ + rgd = (struct rgrp_tree *)next; errblk = gfs2_rgrp_read(&sbd, rgd); if (errblk) continue; - first = 1; - do { - if (gfs2_next_rg_metatype(&sbd, rgd, &blk, metatype, - first)) - break; - if (blk > startblk) { - found = 1; - break; - } - first = 0; - } while (blk <= startblk); + + found = !find_rg_metatype(rgd, &blk, startblk, metatype); + if (found) + break; + gfs2_rgrp_relse(rgd); } + if (!found) blk = 0; if (print) { diff --git a/gfs2/edit/savemeta.c b/gfs2/edit/savemeta.c index cfe18eb..986fdbe 100644 --- a/gfs2/edit/savemeta.c +++ b/gfs2/edit/savemeta.c @@ -635,43 +635,34 @@ static void get_journal_inode_blocks(void) } } -static int next_rg_freemeta(struct gfs2_sbd *sdp, struct rgrp_tree *rgd, - uint64_t *nrfblock, int first) +static void save_allocated(struct rgrp_tree *rgd, struct metafd *mfd) { - struct gfs2_bitmap *bits = NULL; - uint32_t length = rgd->ri.ri_length; - uint32_t blk = (first)? 0: (uint32_t)((*nrfblock+1)-rgd->ri.ri_data0); - int i; - struct gfs2_buffer_head *lbh; + int blktype; + unsigned i, j, m; + uint64_t *ibuf = malloc(sbd.bsize * GFS2_NBBY * sizeof(uint64_t)); - if(!first && (*nrfblock < rgd->ri.ri_data0)) { - log_err("next_rg_freemeta: Start block is outside rgrp " - "bounds.\n"); - exit(1); - } - for(i=0; i < length; i++){ - bits = &rgd->bits[i]; - if(blk < bits->bi_len*GFS2_NBBY) - break; - blk -= bits->bi_len*GFS2_NBBY; - } - for(; i < length; i++){ - bits = &rgd->bits[i]; - lbh = bread(sdp, rgd->ri.ri_addr + i); - blk = gfs2_bitfit((unsigned char *)lbh->b_data + - bits->bi_offset, bits->bi_len, blk, - GFS2_BLKST_UNLINKED); - brelse(lbh); - if(blk != BFITNOENT){ - *nrfblock = blk + (bits->bi_start * GFS2_NBBY) + - rgd->ri.ri_data0; - break; + for (i = 0; i < rgd->ri.ri_length; i++) { + m = lgfs2_bm_scan(rgd, i, ibuf, GFS2_BLKST_DINODE); + + for (j = 0; j < m; j++) { + block = ibuf[j]; + warm_fuzzy_stuff(block, FALSE); + blktype = save_block(sbd.device_fd, mfd, block); + if (blktype == GFS2_METATYPE_DI) + save_inode_data(mfd); + } + + if (!sbd.gfs1) + continue; + + /* For gfs1, Save off the free/unlinked meta blocks too. + * If we don't, we may run into metadata allocation issues. */ + m = lgfs2_bm_scan(rgd, i, ibuf, GFS2_BLKST_UNLINKED); + for (j = 0; j < m; j++) { + blktype = save_block(sbd.device_fd, mfd, block); } - blk=0; } - if(i == length) - return -1; - return 0; + free(ibuf); } void savemeta(char *out_fn, int saveoption, int gziplevel) @@ -771,7 +762,6 @@ void savemeta(char *out_fn, int saveoption, int gziplevel) } /* Walk through the resource groups saving everything within */ for (n = osi_first(&sbd.rgtree); n; n = next) { - int first; struct rgrp_tree *rgd; next = osi_next(n); @@ -782,33 +772,15 @@ void savemeta(char *out_fn, int saveoption, int gziplevel) (unsigned long long)rgd->ri.ri_addr, (unsigned long long)rgd->ri.ri_addr, rgd->ri.ri_length); - first = 1; /* Save off the rg and bitmaps */ for (block = rgd->ri.ri_addr; block < rgd->ri.ri_data0; block++) { warm_fuzzy_stuff(block, FALSE); save_block(sbd.device_fd, &mfd, block); } - /* Save off the other metadata: inodes, etc. */ + /* Save off the other metadata: inodes, etc. if mode is not 'savergs' */ if (saveoption != 2) { - int blktype; - - while (!gfs2_next_rg_meta(rgd, &block, first)){ - warm_fuzzy_stuff(block, FALSE); - blktype = save_block(sbd.device_fd, &mfd, - block); - if (blktype == GFS2_METATYPE_DI) - save_inode_data(&mfd); - first = 0; - } - /* Save off the free/unlinked meta blocks too. - If we don't, we may run into metadata - allocation issues. */ - while (!next_rg_freemeta(&sbd, rgd, &block, first)) { - blktype = save_block(sbd.device_fd, - &mfd, block); - first = 0; - } + save_allocated(rgd, &mfd); } gfs2_rgrp_relse(rgd); } diff --git a/gfs2/libgfs2/libgfs2.h b/gfs2/libgfs2/libgfs2.h index 8dd8210..7d2d9ff 100644 --- a/gfs2/libgfs2/libgfs2.h +++ b/gfs2/libgfs2/libgfs2.h @@ -756,11 +756,6 @@ extern int do_init_statfs(struct gfs2_sbd *sdp); extern int gfs2_check_meta(struct gfs2_buffer_head *bh, int type); extern unsigned lgfs2_bm_scan(struct rgrp_tree *rgd, unsigned idx, uint64_t *buf, uint8_t state); -extern int gfs2_next_rg_meta(struct rgrp_tree *rgd, uint64_t *block, - int first); -extern int gfs2_next_rg_metatype(struct gfs2_sbd *sdp, struct rgrp_tree *rgd, - uint64_t *block, uint32_t type, int first); - /* super.c */ extern int check_sb(struct gfs2_sb *sb); extern int read_sb(struct gfs2_sbd *sdp); diff --git a/gfs2/libgfs2/structures.c b/gfs2/libgfs2/structures.c index c60f3e4..46fd8df 100644 --- a/gfs2/libgfs2/structures.c +++ b/gfs2/libgfs2/structures.c @@ -512,79 +512,3 @@ unsigned lgfs2_bm_scan(struct rgrp_tree *rgd, unsigned idx, uint64_t *buf, uint8 return n; } - -/** - * gfs2_next_rg_meta - * @rgd: - * @block: - * @first: if set, start at zero and ignore block - * - * The position to start looking from is *block. When a block - * is found, it is returned in block. - * - * Returns: 0 on success, -1 when finished - */ -static int __gfs2_next_rg_meta(struct rgrp_tree *rgd, uint64_t *block, - int first, unsigned char state) -{ - struct gfs2_bitmap *bits = NULL; - uint32_t length = rgd->ri.ri_length; - uint32_t blk = (first)? 0: (uint32_t)((*block + 1) - rgd->ri.ri_data0); - int i; - - if (!first && (*block < rgd->ri.ri_data0)) { - fprintf(stderr, "next_rg_meta: Start block is outside rgrp bounds.\n"); - exit(1); - } - for (i = 0; i < length; i++){ - bits = &rgd->bits[i]; - if (blk < bits->bi_len * GFS2_NBBY) - break; - blk -= bits->bi_len * GFS2_NBBY; - } - for (; i < length; i++){ - bits = &rgd->bits[i]; - blk = gfs2_bitfit((unsigned char *)rgd->bh[i]->b_data + - bits->bi_offset, bits->bi_len, blk, state); - if(blk != BFITNOENT){ - *block = blk + (bits->bi_start * GFS2_NBBY) + - rgd->ri.ri_data0; - break; - } - blk = 0; - } - if (i == length) - return -1; - return 0; -} - -int gfs2_next_rg_meta(struct rgrp_tree *rgd, uint64_t *block, int first) -{ - return __gfs2_next_rg_meta(rgd, block, first, GFS2_BLKST_DINODE); -} - -/** - * next_rg_metatype - * @rgd: - * @block: - * @type: the type of metadata we're looking for - * @first: if set we should start at block zero and block is ignored - * - * Returns: 0 on success, -1 on error or finished - */ -int gfs2_next_rg_metatype(struct gfs2_sbd *sdp, struct rgrp_tree *rgd, - uint64_t *block, uint32_t type, int first) -{ - struct gfs2_buffer_head *bh = NULL; - - do{ - if (bh) - brelse(bh); - if (gfs2_next_rg_meta(rgd, block, first)) - return -1; - bh = bread(sdp, *block); - first = 0; - } while(gfs2_check_meta(bh, type)); - brelse(bh); - return 0; -}

1 0

dlm: the tag dlm-4.0.1 has been created
by David Teigland 06 Mar '13

06 Mar '13

1 0

dlm: master - release 4.0.1
by David Teigland 06 Mar '13

06 Mar '13

Gitweb: http://git.fedorahosted.org/git/?p=dlm.git;a=commitdiff;h=2b697668c41a12e81… Commit: 2b697668c41a12e816a153dde96dea43bb0094d9 Parent: a83408ad8ee91a3f5d0416dcc3b7e8c50c47c7b9 Author: David Teigland <teigland(a)redhat.com> AuthorDate: Wed Mar 6 12:26:29 2013 -0600 Committer: David Teigland <teigland(a)redhat.com> CommitterDate: Wed Mar 6 12:26:29 2013 -0600 release 4.0.1 Signed-off-by: David Teigland <teigland(a)redhat.com> --- include/copyright.cf | 2 +- include/version.cf | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/include/copyright.cf b/include/copyright.cf index 6a73d4a..714a761 100644 --- a/include/copyright.cf +++ b/include/copyright.cf @@ -1,6 +1,6 @@ #ifndef __COPYRIGHT_DOT_CF__ #define __COPYRIGHT_DOT_CF__ -#define REDHAT_COPYRIGHT "Copyright Red Hat, Inc. 2004-2012" +#define REDHAT_COPYRIGHT "Copyright Red Hat, Inc. 2004-2013" #endif /* __COPYRIGHT_DOT_CF__ */ diff --git a/include/version.cf b/include/version.cf index 5d75ee5..7134537 100644 --- a/include/version.cf +++ b/include/version.cf @@ -1,6 +1,6 @@ #ifndef _RELEASE_VERSION_CF_ #define _RELEASE_VERSION_CF_ -#define RELEASE_VERSION "4.0.0" +#define RELEASE_VERSION "4.0.1" #endif

1 0

dlm: master - dlm_controld: even partition merges
by David Teigland 06 Mar '13

06 Mar '13

Gitweb: http://git.fedorahosted.org/git/?p=dlm.git;a=commitdiff;h=a83408ad8ee91a3f5… Commit: a83408ad8ee91a3f5d0416dcc3b7e8c50c47c7b9 Parent: ee11f54d5448447505239fdff4a68b3922d97f4e Author: David Teigland <teigland(a)redhat.com> AuthorDate: Tue Mar 5 13:36:38 2013 -0600 Committer: David Teigland <teigland(a)redhat.com> CommitterDate: Tue Mar 5 13:36:38 2013 -0600 dlm_controld: even partition merges When a cluster partitions and merges evenly, e.g. 2/2, both sides were killing/fencing each other. In this case, wait for stateful merged nodes to be cleared before fencing. The side with the low nodeid will kill the stateful nodes (from its perspective). Signed-off-by: David Teigland <teigland(a)redhat.com> --- dlm_controld/daemon_cpg.c | 82 ++++++++++++++++++++++++++++++++++++++++++++- dlm_controld/fence.c | 2 +- 2 files changed, 82 insertions(+), 2 deletions(-) diff --git a/dlm_controld/daemon_cpg.c b/dlm_controld/daemon_cpg.c index b3c543d..8c4cff2 100644 --- a/dlm_controld/daemon_cpg.c +++ b/dlm_controld/daemon_cpg.c @@ -72,6 +72,7 @@ struct node_daemon { int need_fence_clear; int need_fencing; int delay_fencing; + int stateful_merge; int fence_pid; int fence_pid_wait; int fence_result_wait; @@ -657,6 +658,20 @@ static void fence_pid_cancel(int nodeid, int pid) nodeid, pid, rv, result); } +static void kick_stateful_merge_members(void) +{ + struct node_daemon *node; + + list_for_each_entry(node, &daemon_nodes, list) { + if (!node->killed && node->stateful_merge) { + log_error("daemon node %d kill stateful merge member", + node->nodeid); + kick_node_from_cluster(node->nodeid); + node->killed = 1; + } + } +} + /* * fence_in_progress_unknown (fipu) * @@ -746,7 +761,8 @@ static void fence_pid_cancel(int nodeid, int pid) static void daemon_fence_work(void) { struct node_daemon *node, *safe; - int rv, nodeid, pid, need, low, actor, result; + int gone_count = 0, part_count = 0, merge_count = 0, clean_count = 0; + int rv, nodeid, pid, need, low = 0, actor, result; int retry = 0; uint32_t flags; @@ -774,6 +790,67 @@ static void daemon_fence_work(void) } /* + * Count different types of nodes + * gone: node not a member + * part: member we've not received a proto message from + * merge: member we received a stateful proto message from + * clean: member we received a clean/new proto message from + * + * A node always views itself as a clean member, not a merge member. + */ + + list_for_each_entry(node, &daemon_nodes, list) { + if (!node->daemon_member) { + gone_count++; + } else { + if (!low || node->nodeid < low) + low = node->nodeid; + + if (node->stateful_merge) + merge_count++; + else if (!node->proto.daemon_max[0]) + part_count++; + else + clean_count++; + } + } + + /* + * Wait for stateful merged members to be removed before moving + * on to fencing. Kill stateful merged members to clear them. + * This section is only relevant to non-two-node, even splits. + * + * With two node splits, they race to fence each other and + * whichever fences successfully then kills corosync on the other + * (in the case where corosync is still running on the fenced node). + * + * With an odd split, the partition that maintained quorum will + * kill stateful merged nodes when their proto message is received. + * + * With an even split, e.g. 2/2, we don't want both sets to + * be fencing each other right after merge, when both sides + * have quorum again and see the other side as statefully merged. + * So, delay fencing until the stateful nodes are cleared on one + * side (by way of the low nodeid killing stateful merged members). + * + * When there are 3 or more partitions that merge, none may see + * enough clean nodes, so the cluster would be stuck here waiting + * for someone to manually reset/restart enough nodes to produce + * sufficient clean nodes (>= merged). + */ + + if (!cluster_two_node && merge_count) { + log_retry(retry_fencing, "fence work wait to clear merge %d clean %d part %d gone %d", + merge_count, clean_count, part_count, gone_count); + + if ((clean_count >= merge_count) && !part_count && (low == our_nodeid)) + kick_stateful_merge_members(); + + retry = 1; + goto out; + } + + /* * startup fencing */ @@ -1608,6 +1685,8 @@ static void receive_protocol(struct dlm_header *hd, int len) (unsigned long long)cluster_quorate_monotime, node->killed); + node->stateful_merge = 1; + if (cluster_quorate && node->daemon_rem_time && cluster_quorate_monotime < node->daemon_rem_time) { if (!node->killed) { @@ -1960,6 +2039,7 @@ static void confchg_cb_daemon(cpg_handle_t handle, node->daemon_member = 0; node->daemon_rem_time = now; node->killed = 0; + node->stateful_merge = 0; /* If we never accepted a valid proto from this node, then it never fully joined and there's no need to diff --git a/dlm_controld/fence.c b/dlm_controld/fence.c index 0094b3a..e4c67e9 100644 --- a/dlm_controld/fence.c +++ b/dlm_controld/fence.c @@ -138,7 +138,7 @@ int fence_result(int nodeid, int pid, int *result) if (rv < 0) { /* shouldn't happen */ log_error("fence result %d pid %d waitpid %d errno %d", - pid, nodeid, rv, errno); + nodeid, pid, rv, errno); return rv; }

1 0

dlm: master - dlm_controld: partition merge handling
by David Teigland 06 Mar '13

06 Mar '13

Gitweb: http://git.fedorahosted.org/git/?p=dlm.git;a=commitdiff;h=ee11f54d544844750… Commit: ee11f54d5448447505239fdff4a68b3922d97f4e Parent: 16ddbe2d61b4edbed4d911c523a82bcf5f325b70 Author: David Teigland <teigland(a)redhat.com> AuthorDate: Thu Feb 28 15:12:33 2013 -0600 Committer: David Teigland <teigland(a)redhat.com> CommitterDate: Thu Feb 28 15:12:33 2013 -0600 dlm_controld: partition merge handling Fix/improve the handling of partition merge cases. This includes fencing and killing merged nodes. Signed-off-by: David Teigland <teigland(a)redhat.com> --- dlm_controld/daemon_cpg.c | 159 +++++++++++++++++++++++++++++++++++---------- dlm_controld/dlm_daemon.h | 1 + dlm_controld/member.c | 15 ++++ 3 files changed, 141 insertions(+), 34 deletions(-) diff --git a/dlm_controld/daemon_cpg.c b/dlm_controld/daemon_cpg.c index 641b189..b3c543d 100644 --- a/dlm_controld/daemon_cpg.c +++ b/dlm_controld/daemon_cpg.c @@ -369,6 +369,18 @@ static int nodes_need_fencing(void) return 0; } +static int nodeid_needs_fencing(int nodeid) +{ + struct node_daemon *node; + + node = get_node_daemon(nodeid); + if (!node) { + log_error("nodeid_needs_fencing %d not found", nodeid); + return 0; + } + return node->need_fencing; +} + static int all_daemon_members_fipu(void) { struct node_daemon *node; @@ -839,7 +851,7 @@ static void daemon_fence_work(void) /* * node has rejoined in clean state */ - log_debug("fence request %d skip member", node->nodeid); + log_debug("fence request %d skip for is_clean_daemon_member", node->nodeid); node->need_fencing = 0; node->delay_fencing = 0; @@ -858,6 +870,10 @@ static void daemon_fence_work(void) continue; } + /* use post_join_delay to avoid fencing a node in the short + time between it joining the cluster (giving cluster quorum) + and joining the daemon cpg, which allows it to bypass fencing */ + if (monotime() - cluster_last_join_monotime < opt(post_join_delay_ind)) { log_debug("fence request %d delay %d from %llu", node->nodeid, opt(post_join_delay_ind), @@ -945,7 +961,7 @@ static void daemon_fence_work(void) * will see and do this, so we don't need to send * a fence result. */ - log_debug("fence wait %d pid %d skip member", nodeid, pid); + log_debug("fence wait %d pid %d skip for is_clean_daemon_member", nodeid, pid); node->need_fencing = 0; node->delay_fencing = 0; @@ -1262,7 +1278,8 @@ static void receive_fence_result(struct dlm_header *hd, int len) } if (!node->need_fencing) { - /* should never happen */ + /* should never happen ... will happen if a manual fence_ack is + done for a node that doesn't need it */ log_error("receive_fence_result %d from %d result %d no need_fencing", fr->nodeid, hd->nodeid, fr->result); return; @@ -1275,20 +1292,23 @@ static void receive_fence_result(struct dlm_header *hd, int len) /* should we ignore and return here? */ } - if (!fr->result && node->daemon_member) { + if (node->daemon_member && + (!fr->result || (fr->result == -ECANCELED))) { /* - * the only time I think this can happen is if there is a - * manual dlm_tool fence_ack for a node that is a member, - * e.g. partition, merge, fence_ack while it's a merged member. - * Ideally it would be killed after merging with state, but - * not necessarily, i.e. it's start message can't be sent or - * received. + * The node was successfully fenced, but is still a member. + * This will happen when there is a partition, storage fencing + * is started, a merge causes the node to become a member + * again, and storage fencing completes successfully. If we + * received a proto message from the node after the merge, then + * we will have detected a stateful merge, and we may have + * already killed it. */ - log_error("receive_fence_result %d from %d result %d node not dead", + log_error("receive_fence_result %d from %d result %d node is daemon_member", fr->nodeid, hd->nodeid, fr->result); - return; + + kick_node_from_cluster(fr->nodeid); } if ((hd->nodeid == our_nodeid) && (fr->result != -ECANCELED)) @@ -1580,18 +1600,62 @@ static void receive_protocol(struct dlm_header *hd, int len) (p->dr_ver.flags & PV_STATEFUL) && (our_protocol.dr_ver.flags & PV_STATEFUL)) { - log_debug("daemon node %d stateful merge", hd->nodeid); - log_debug("daemon node %d join %llu left %llu local quorum %llu", + log_error("daemon node %d stateful merge", hd->nodeid); + log_debug("daemon node %d join %llu left %llu local quorum %llu killed %d", hd->nodeid, (unsigned long long)node->daemon_add_time, (unsigned long long)node->daemon_rem_time, - (unsigned long long)cluster_quorate_monotime); + (unsigned long long)cluster_quorate_monotime, + node->killed); if (cluster_quorate && node->daemon_rem_time && cluster_quorate_monotime < node->daemon_rem_time) { - log_debug("daemon node %d kill due to stateful merge", hd->nodeid); - if (!node->killed) - kick_node_from_cluster(hd->nodeid); + if (!node->killed) { + if (cluster_two_node) { + /* + * When there are two nodes and two_node mode + * is used, both will have quorum throughout + * the partition and subsequent stateful merge. + * + * - both will race to fence each other in + * response to the partition + * + * - both can attempt to kill the cluster + * on the other in response to the stateful + * merge here + * + * - we don't want both nodes to kill the cluster + * on the other, which can happen if the merge + * occurs before power fencing is successful, + * or can happen before/during/after storage + * fencing + * + * - if nodeA successfully fences nodeB (due + * to the partition), we want nodeA to kill + * the cluster on nodeB in response to the + * merge (we don't want nodeB to kill nodeA + * in response to the merge). + * + * So, a node that has successfully fenced the + * other will kill the cluster on it. If fencing + * is still running, we wait until it's + * successfull to kill the cluster on the node + * being fenced. + */ + if (nodeid_needs_fencing(hd->nodeid)) { + /* when fencing completes successfully, + we'll see the node is a daemon member + and kill it */ + log_debug("daemon node %d delay kill for stateful merge", hd->nodeid); + } else { + log_error("daemon node %d kill due to stateful merge", hd->nodeid); + kick_node_from_cluster(hd->nodeid); + } + } else { + log_error("daemon node %d kill due to stateful merge", hd->nodeid); + kick_node_from_cluster(hd->nodeid); + } + } node->killed = 1; } @@ -1808,6 +1872,7 @@ static void confchg_cb_daemon(cpg_handle_t handle, struct node_daemon *node; uint64_t now, now_wall; int nodedown = 0, procdown = 0, leave = 0; + int check_joined_count = 0, check_remove_count = 0, check_member_count = 0; int we_joined = 0; int i, reason, low; @@ -1851,7 +1916,7 @@ static void confchg_cb_daemon(cpg_handle_t handle, } if (nodedown || procdown || leave) - log_debug("%s left nodedown %d procdown %d leave %d", + log_debug("%s left reason nodedown %d procdown %d leave %d", group_name->value, nodedown, procdown, leave); if (nodedown) @@ -1865,6 +1930,8 @@ static void confchg_cb_daemon(cpg_handle_t handle, if (node->daemon_member) continue; + check_joined_count++; + /* node joined daemon cpg */ node->daemon_member = 1; node->daemon_add_time = now; @@ -1878,31 +1945,56 @@ static void confchg_cb_daemon(cpg_handle_t handle, if (node->need_fencing) { /* need_fencing will be cleared if we accept a - valid proto from it */ - log_error("daemon new nodeid %d needs fencing", - node->nodeid); + valid proto from it (is_clean_daemon_member) */ + log_error("daemon joined %d needs fencing", node->nodeid); + } else { + log_debug("daemon joined %d"); } - } else { if (!node->daemon_member) continue; + check_remove_count++; + /* node left daemon cpg */ node->daemon_member = 0; - node->killed = 0; - memset(&node->proto, 0, sizeof(struct protocol)); node->daemon_rem_time = now; + node->killed = 0; - /* tell loop below to look at this node */ - node->recover_setup = 1; + /* If we never accepted a valid proto from this node, + then it never fully joined and there's no need to + recover it. Similary, node_history_lockspace_fail + only sets need_fencing in the lockspace if + node->start_time was non-zero. */ + + if (node->proto.daemon_max[0]) { + /* tell loop below to look at this node */ + node->recover_setup = 1; + } else { + log_debug("daemon remove %d no proto skip recover", node->nodeid); + } + + memset(&node->proto, 0, sizeof(struct protocol)); } } - /* set up recovery work for nodes that just failed */ + list_for_each_entry(node, &daemon_nodes, list) { + if (node->daemon_member) + check_member_count++; + } + + /* when we join, all previous members look like they are joining */ + if (!we_joined && + (daemon_joined_count != check_joined_count || + daemon_remove_count != check_remove_count || + daemon_member_count != check_member_count)) { + log_error("daemon counts joined %d check %d remove %d check %d member %d check %d", + daemon_joined_count, check_joined_count, + daemon_remove_count, check_remove_count, + daemon_member_count, check_member_count); + } - /* TODO: limit to nodes with a valid proto? - * node_history_lockspace_fail() would only set - * need_fencing if node->start_time was non-zero. */ + /* set up recovery work for nodes that just failed (recover_setup set above) */ list_for_each_entry(node, &daemon_nodes, list) { if (!node->recover_setup) @@ -1916,8 +2008,7 @@ static void confchg_cb_daemon(cpg_handle_t handle, continue; if (node->need_fencing) { - log_error("daemon remove nodeid %d already needs fencing", - node->nodeid); + log_error("daemon remove %d already needs fencing", node->nodeid); continue; } @@ -1931,7 +2022,7 @@ static void confchg_cb_daemon(cpg_handle_t handle, if (reason == CPG_REASON_NODEDOWN || reason == CPG_REASON_PROCDOWN) { if (node->fence_pid_wait || node->fence_pid) { /* sanity check, should never happen */ - log_error("daemon remove nodeid %d pid_wait %d pid %d", + log_error("daemon remove %d pid_wait %d pid %d", node->nodeid, node->fence_pid_wait, node->fence_pid); } diff --git a/dlm_controld/dlm_daemon.h b/dlm_controld/dlm_daemon.h index 60d3d46..11a4777 100644 --- a/dlm_controld/dlm_daemon.h +++ b/dlm_controld/dlm_daemon.h @@ -169,6 +169,7 @@ EXTERN int plock_fd; EXTERN int plock_ci; EXTERN struct list_head lockspaces; EXTERN int cluster_quorate; +EXTERN int cluster_two_node; EXTERN uint64_t cluster_last_join_monotime; EXTERN uint64_t cluster_quorate_monotime; EXTERN uint64_t cluster_joined_monotime; diff --git a/dlm_controld/member.c b/dlm_controld/member.c index d195ab1..fca3248 100644 --- a/dlm_controld/member.c +++ b/dlm_controld/member.c @@ -348,6 +348,8 @@ int setup_node_config(void) cmap_handle_t h; cs_error_t err; uint32_t nodeid; + uint8_t two_node = 0; + int node_count = 0; int i; err = cmap_initialize(&h); @@ -365,10 +367,23 @@ int setup_node_config(void) log_debug("node_config %d", nodeid); + node_count++; + if (opt(enable_fencing_ind) && opt(enable_startup_fencing_ind)) add_startup_node(nodeid); } + memset(key, 0, sizeof(key)); + snprintf(key, CMAP_KEYNAME_MAXLEN, "quorum.two_node"); + + err = cmap_get_uint8(h, key, &two_node); + if (err != CS_OK) + goto out; + + if (node_count == 2 && two_node) + cluster_two_node = 1; + + out: cmap_finalize(h); return 0; }

1 0

dlm: master - dlm_controld: ignore fence_ack for member
by David Teigland 06 Mar '13

06 Mar '13

Gitweb: http://git.fedorahosted.org/git/?p=dlm.git;a=commitdiff;h=16ddbe2d61b4edbed… Commit: 16ddbe2d61b4edbed4d911c523a82bcf5f325b70 Parent: f65f3f67b0f87f52398ff1100aaede36e254abda Author: David Teigland <teigland(a)redhat.com> AuthorDate: Wed Feb 6 14:11:15 2013 -0600 Committer: David Teigland <teigland(a)redhat.com> CommitterDate: Wed Feb 6 14:11:15 2013 -0600 dlm_controld: ignore fence_ack for member When a node is still a member of the daemon cpg, ignore a fence_ack for it. Either it never failed and the ack is pointless, or it merged after a partition, fencing hasn't completed and it's still a merged member that needs to be reset. Signed-off-by: David Teigland <teigland(a)redhat.com> --- dlm_controld/daemon_cpg.c | 17 +++++++++++++++++ 1 files changed, 17 insertions(+), 0 deletions(-) diff --git a/dlm_controld/daemon_cpg.c b/dlm_controld/daemon_cpg.c index af9b91d..641b189 100644 --- a/dlm_controld/daemon_cpg.c +++ b/dlm_controld/daemon_cpg.c @@ -1251,6 +1251,7 @@ static void receive_fence_result(struct dlm_header *hd, int len) if (count) { log_debug("receive_fence_result %d from %d clear startup", fr->nodeid, hd->nodeid); + return; } node = get_node_daemon(fr->nodeid); @@ -1274,6 +1275,22 @@ static void receive_fence_result(struct dlm_header *hd, int len) /* should we ignore and return here? */ } + if (!fr->result && node->daemon_member) { + + /* + * the only time I think this can happen is if there is a + * manual dlm_tool fence_ack for a node that is a member, + * e.g. partition, merge, fence_ack while it's a merged member. + * Ideally it would be killed after merging with state, but + * not necessarily, i.e. it's start message can't be sent or + * received. + */ + + log_error("receive_fence_result %d from %d result %d node not dead", + fr->nodeid, hd->nodeid, fr->result); + return; + } + if ((hd->nodeid == our_nodeid) && (fr->result != -ECANCELED)) node->fence_result_wait = 0;

1 0

fence-agents: the annotated tag v4.0.0 has been created
by Marek Grác 06 Mar '13

06 Mar '13

Gitweb: http://git.fedorahosted.org/git/?p=fence-agents.git;a=commitdiff;h=c58fc667… Commit: c58fc667bab72b03527cc95bf791ae36af0fab02 Parent: 0000000000000000000000000000000000000000 Author: Marek 'marx' Grac <mgrac(a)redhat.com> AuthorDate: 2013-03-06 15:16 +0000 Committer: Marek 'marx' Grac <mgrac(a)redhat.com> CommitterDate: 2013-03-06 15:16 +0000 annotated tag: v4.0.0 has been created at c58fc667bab72b03527cc95bf791ae36af0fab02 (tag) tagging 06f7d26340d8f700dc834e8ce5495bf156e20d2e (commit) replaces v3.1.12 v4.0.0 release Jan Pokorný (2): lib: remove static metadata from fence.rng.head fence2rng.xsl: parametrize and make more flexible Marek 'marx' Grac (28): COMPATIBILITY BREAK: remove option -T / --test / test COMPATIBILITY BREAK: fence_drac5 replace --m / --modulename / modulename / module_name with standard -n / --plug / port COMPATIBILITY BREAK: fence_lpar replace --n / --partition / partition with standard -n / --plug / port COMPATIBILITY BREAK: fence_rsb replace -n / telnet_port with standard --ipport / ipport COMPATIBILITY BREAK: Remove obsolete options from STDIN COMPATIBILITY BREAK: replace udpport with ipport, make ipport dependent on ipaddress COMPATIBILITY BREAK: remove -q / quiet fencing: Text in help is changed from foo=<value> to foo=[value] fence agents: Text in help is changed from foo=<value> to foo=[value] fence_vmnware_soap: Fix traceback when hostname cannot be resolved to IP address fence_vmware_soap: Fix previous patch - fix traceback when hostname cannot be resolved fence_cisco_ucs: fence agent does not respect "delay" attribute fence_cisco_ucs: Fix traceback when hostname cannot be resolved to IP address fence_virsh: Replace constant (1) with variable from --power-wait fencing: Testing tool and dummy fence agent fence agents: Fence agents errors are logged to syslog COMPATIBILITY BREAK: remove support for Dell DRAC III and older testing: Add test suite for fence_drac on Dell DRAC IV testing: Add verbose mode to testing library fence_apc: Add support for firmware 5.x COMPATIBILITY BREAK: remove --uuid (use standard -n / --plug instead) fencing: Fix documentation for action 'monitor' fencing: present default values for --ipport in --help fencing: Fix indentation in --help message fencing: All command prompt defaults should be list fencing: Support multiple plugs for actions (e.g. --n 2,3,4) testing: Add test for multiple ports feature fencing: Install non-fence agents to libexec

1 0

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

cluster-commits March 2013