vdsm-patches July 2015

vdsm-patches@lists.fedorahosted.org

44 participants
919 discussions

Change in vdsm[master]: contrib: Simple jsonrpc client
by Nir Soffer 30 Nov '16

30 Nov '16

Nir Soffer has uploaded a new change for review. Change subject: contrib: Simple jsonrpc client ...................................................................... contrib: Simple jsonrpc client This is a simple jsonrpc client for communicating with the jsonrpc server from the command line. Arguments method one of the mehtods described in json schema params optionl json object with message parameters Examples Calling method without arguements: # jsonrpc Host.getVMList { "jsonrpc": "2.0", "id": "0e043d83-294a-4d31-b1b6-6dc2f2747494", "result": [ "b3f6fa00-b315-4ad4-8108-f73da817b5c5" ] } Calling method with arguements: # jsonrpc VM.getStats '{"vmID": "b3f6fa00-b315-4ad4-8108-f73da817b5c5"}' { "jsonrpc": "2.0", "id": "cefd25a3-6250-4123-8a56-d7047899e19e", "result": [ { "status": "Down", "exitMessage": "Admin shut down from the engine", "vmId": "b3f6fa00-b315-4ad4-8108-f73da817b5c5", "exitReason": 6, "timeOffset": "0", "exitCode": 0 } ] } Requires stomp.py library: https://pypi.python.org/pypi/stomp.py Change-Id: Ia6273eabf6f3601602659d1e4e748d8025ae8084 Signed-off-by: Nir Soffer <nsoffer(a)redhat.com> --- A contrib/jsonrpc 1 file changed, 104 insertions(+), 0 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/81/35181/1 diff --git a/contrib/jsonrpc b/contrib/jsonrpc new file mode 100755 index 0000000..3080193 --- /dev/null +++ b/contrib/jsonrpc @@ -0,0 +1,104 @@ +#!/usr/bin/python +# +# Copyright 2014 Red Hat, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA +# +# Refer to the README and COPYING files for full details of the license +# +""" +jsonrpc-cli - Vdsm jsonrpc client +""" + +import json +import os +import signal +import sys +import time +import uuid +import optparse + +import stomp + +# Copied from lib/vdsm/vdscli.py +PKIDIR = '/etc/pki/vdsm' +KEYFILE = os.path.join(PKIDIR, 'keys/vdsmkey.pem') +CERTFILE = os.path.join(PKIDIR, 'certs/vdsmcert.pem') +CACERT = os.path.join(PKIDIR, 'certs/cacert.pem') + +PORT = 54321 +DESTINATION = "/queue/_local/vdsm/requests" + + +class Listener(stomp.ConnectionListener): + + def on_error(self, headers, message): + print 'Error: %s' % message + terminate() + + def on_message(self, headers, message): + msg = json.loads(message) + print json.dumps(msg, indent=4) + terminate() + + +def main(args): + parser = option_parser() + options, args = parser.parse_args(args) + if not args: + parser.error("method required") + + msg = { + "id": str(uuid.uuid4()), + "jsonrpc": "2.0", + "method": args[0] + } + + if len(args) > 1: + msg["params"] = json.loads(args[1]) + + conn = stomp.Connection10( + host_and_ports=((options.host, PORT),), + use_ssl=True, + ssl_key_file=KEYFILE, + ssl_cert_file=CERTFILE, + ssl_ca_certs=CACERT) + + conn.set_listener("", Listener()) + conn.start() + conn.send(body=json.dumps(msg), destination=DESTINATION) + + try: + signal.pause() + except KeyboardInterrupt: + pass + + conn.disconnect() + + +def option_parser(): + parser = optparse.OptionParser(usage='%prog [options] method [params]') + parser.add_option("-a", "--host", dest="host", + help="host address (default localhost)") + parser.set_defaults(host="localhost") + return parser + + +def terminate(): + os.kill(os.getpid(), signal.SIGINT) + + +if __name__ == "__main__": + main(sys.argv[1:]) -- To view, visit http://gerrit.ovirt.org/35181 To unsubscribe, visit http://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Ia6273eabf6f3601602659d1e4e748d8025ae8084 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Nir Soffer <nsoffer(a)redhat.com>

9 53

Change in vdsm[master]: spbackends: simplify over-general evaluation
by Dan Kenigsberg 27 Nov '16

27 Nov '16

Dan Kenigsberg has uploaded a new change for review. Change subject: spbackends: simplify over-general evaluation ...................................................................... spbackends: simplify over-general evaluation The removed code is too clever for the use case of only two values. Change-Id: I775c4a6003a03c6b65bee3abc8559a88e3eaec01 Signed-off-by: Dan Kenigsberg <danken(a)redhat.com> --- M vdsm/storage/spbackends.py 1 file changed, 5 insertions(+), 2 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/53/41253/1 diff --git a/vdsm/storage/spbackends.py b/vdsm/storage/spbackends.py index 953e01e..f6ce02b 100644 --- a/vdsm/storage/spbackends.py +++ b/vdsm/storage/spbackends.py @@ -229,8 +229,11 @@ def setSpmStatus(self, lVer=None, spmId=None): self.invalidateMetadata() - metaParams = dict(filter(lambda kv: kv[1] is not None, - ((PMDK_LVER, lVer), (PMDK_SPM_ID, spmId)))) + metaParams = {} + if lVer is not None: + metaParams[PMDK_LVER] = lVer + if spmId is not None: + metaParams[PMDK_SPM_ID] = spmId self._metadata.update(metaParams) @unsecured -- To view, visit https://gerrit.ovirt.org/41253 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I775c4a6003a03c6b65bee3abc8559a88e3eaec01 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Dan Kenigsberg <danken(a)redhat.com>

5 6

Change in vdsm[master]: virt: enable libgfapi
by ahino＠redhat.com 21 Oct '16

21 Oct '16

Ala Hino has uploaded a new change for review. Change subject: virt: enable libgfapi ...................................................................... virt: enable libgfapi This change is based on Federico's changes: https://gerrit.ovirt.org/33768/ Change-Id: I54b81e87b959b0b49c0f06810f88410e7c75de1d Signed-off-by: Federico Simoncelli <fsimonce(a)redhat.com> Signed-off-by: Ala Hino <ahino(a)redhat.com> --- M vdsm/storage/glusterVolume.py M vdsm/storage/hsm.py M vdsm/virt/vm.py 3 files changed, 22 insertions(+), 5 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/61/44061/1 diff --git a/vdsm/storage/glusterVolume.py b/vdsm/storage/glusterVolume.py index 8f701ba..109d1c9 100644 --- a/vdsm/storage/glusterVolume.py +++ b/vdsm/storage/glusterVolume.py @@ -54,4 +54,5 @@ return {'volType': VmVolumeInfo.TYPE_NETWORK, 'path': glusterPath, 'protocol': 'gluster', 'volPort': volPort, 'volTransport': volTrans, - 'volfileServer': volfileServer} + 'volfileServer': volfileServer, + 'bricks': volInfo[volname]['bricks']} diff --git a/vdsm/storage/hsm.py b/vdsm/storage/hsm.py index f68d3bb..109cdfa 100644 --- a/vdsm/storage/hsm.py +++ b/vdsm/storage/hsm.py @@ -3209,8 +3209,7 @@ path = os.path.join(dom.domaindir, sd.DOMAIN_IMAGES, imgUUID, volUUID) volInfo = {'domainID': sdUUID, 'imageID': imgUUID, - 'volumeID': volUUID, 'path': path, - 'volType': "path"} + 'volumeID': volUUID, 'path': path} leasePath, leaseOffset = dom.getVolumeLease(imgUUID, volUUID) @@ -3221,8 +3220,8 @@ }) imgVolumesInfo.append(volInfo) - if volUUID == leafUUID: - leafInfo = volInfo + + leafInfo = dom.produceVolume(imgUUID, leafUUID).getVmVolumeInfo() return {'path': leafPath, 'info': leafInfo, 'imgVolumesInfo': imgVolumesInfo} diff --git a/vdsm/virt/vm.py b/vdsm/virt/vm.py index 71a74b3..2976f3b 100644 --- a/vdsm/virt/vm.py +++ b/vdsm/virt/vm.py @@ -805,6 +805,23 @@ # A destroy request has been issued, exit early break drive['path'] = self.cif.prepareVolumePath(drive, self.id) + if drive.get('diskType') == DISK_TYPE.NETWORK: + volinfo = drive.get('volumeInfo') + drive['path'] = volinfo['path'] + drive['protocol'] = volinfo['protocol'] + if drive.get('hosts') is None: + """ + THIS CODE COMMENTED OUT DUE TO A BUG IN LIBVIRT, + CANNOT WORK WITH MULTIPLE HOSTS + """ + """ + hosts = [dict(name=brick.split(":")[0], + port='0', transport='tcp') + for brick in volinfo['bricks']] + """ + hosts = [dict(name=volinfo['bricks'][0].split(":")[0], + port='0', transport='tcp')] + drive['hosts'] = hosts else: # Now we got all the resources we needed -- To view, visit https://gerrit.ovirt.org/44061 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I54b81e87b959b0b49c0f06810f88410e7c75de1d Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Ala Hino <ahino(a)redhat.com>

5 45

Change in vdsm[master]: tests: Add basic tests for RWLock
by alitke＠redhat.com 15 Oct '16

15 Oct '16

Adam Litke has uploaded a new change for review. Change subject: tests: Add basic tests for RWLock ...................................................................... tests: Add basic tests for RWLock Change-Id: I77b9ea1cd6d378738fbf14de119c88bf0ebc94e2 Signed-off-by: Adam Litke <alitke(a)redhat.com> --- M tests/miscTests.py 1 file changed, 18 insertions(+), 0 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/72/42772/1 diff --git a/tests/miscTests.py b/tests/miscTests.py index 976b503..c6d9690 100644 --- a/tests/miscTests.py +++ b/tests/miscTests.py @@ -1268,3 +1268,21 @@ def _run(self): self.result = self._func() + + +class RWLockTests(TestCaseBase): + + def test_multiple_acquire(self): + lock = misc.RWLock() + self.assertTrue(lock.acquire(exclusive=True)) + self.assertTrue(lock.acquire(exclusive=True)) + + def test_demote(self): + lock = misc.RWLock() + self.assertTrue(lock.acquire(exclusive=True)) + self.assertTrue(lock.acquire(exclusive=False)) + + def test_promote(self): + lock = misc.RWLock() + self.assertTrue(lock.acquire(False)) + self.assertRaises(RuntimeError, lock.acquire, True) -- To view, visit https://gerrit.ovirt.org/42772 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I77b9ea1cd6d378738fbf14de119c88bf0ebc94e2 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Adam Litke <alitke(a)redhat.com>

4 9

Change in vdsm[master]: lvm: Fail loudly if called with unexptected input.
by Nir Soffer 15 Oct '16

15 Oct '16

Nir Soffer has uploaded a new change for review. Change subject: lvm: Fail loudly if called with unexptected input. ...................................................................... lvm: Fail loudly if called with unexptected input. When creating pvs with the force option, we are very carefull to accept only True. When using the jsonrpc transport, engine was sending "true" and "false", causing the call to fail misteiously. Now we are also carefull about rejecting invlid input, making debugging easier. Change-Id: If9e6754d4aa2efaf894a9309cfaa4595d710063b Signed-off-by: Nir Soffer <nsoffer(a)redhat.com> --- M vdsm/storage/lvm.py 1 file changed, 5 insertions(+), 0 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/29/37329/1 diff --git a/vdsm/storage/lvm.py b/vdsm/storage/lvm.py index aa3c04b..549839a 100644 --- a/vdsm/storage/lvm.py +++ b/vdsm/storage/lvm.py @@ -724,6 +724,11 @@ else: raise + # We must be very carefull here; any value execpt True or False is a user + # error. + if type(force) != bool: + raise ValueError("Invalid value for 'force': %r" % force) + if force is True: options = ("-y", "-ff") _initpvs_removeHolders() -- To view, visit http://gerrit.ovirt.org/37329 To unsubscribe, visit http://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: If9e6754d4aa2efaf894a9309cfaa4595d710063b Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Nir Soffer <nsoffer(a)redhat.com>

6 19

Change in vdsm[master]: hsm: Pep8ize function name
by Nir Soffer 08 Oct '16

08 Oct '16

Nir Soffer has uploaded a new change for review. Change subject: hsm: Pep8ize function name ...................................................................... hsm: Pep8ize function name Remove unneeded _private prefix for inner function and use lowercase name. Change-Id: I54715971378319a8501a49cb89b24f27e50a07f4 Signed-off-by: Nir Soffer <nsoffer(a)redhat.com> --- M vdsm/storage/hsm.py 1 file changed, 3 insertions(+), 3 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/06/39306/1 diff --git a/vdsm/storage/hsm.py b/vdsm/storage/hsm.py index a4bb858..8f75a39 100644 --- a/vdsm/storage/hsm.py +++ b/vdsm/storage/hsm.py @@ -2070,7 +2070,7 @@ boolean :rtype: dict """ - def _isVisible(guid): + def is_visible(guid): path = os.path.join('/dev/mapper', guid) try: st = os.stat(path) @@ -2081,10 +2081,10 @@ else: return (st.st_mode & stat.S_IRUSR) != 0 - visibility = [_isVisible(guid) for guid in guids] + visibility = [is_visible(guid) for guid in guids] if not all(visibility): multipath.rescan() - visibility = [_isVisible(guid) for guid in guids] + visibility = [is_visible(guid) for guid in guids] visibility = dict(zip(guids, visibility)) # After multipath.rescan, existing devices may disapper, and new -- To view, visit https://gerrit.ovirt.org/39306 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I54715971378319a8501a49cb89b24f27e50a07f4 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Nir Soffer <nsoffer(a)redhat.com>

5 9

Change in vdsm[master]: Live Merge: Restore watermark tracking
by alitke＠redhat.com 08 Oct '16

08 Oct '16

Adam Litke has uploaded a new change for review. Change subject: Live Merge: Restore watermark tracking ...................................................................... Live Merge: Restore watermark tracking Change-Id: I632f31e7795ec5d8c6f52a480116b14470c3163f Signed-off-by: Adam Litke <alitke(a)redhat.com> --- M vdsm/virt/vm.py 1 file changed, 108 insertions(+), 10 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/24/36924/1 diff --git a/vdsm/virt/vm.py b/vdsm/virt/vm.py index f22610d..09080b9 100644 --- a/vdsm/virt/vm.py +++ b/vdsm/virt/vm.py @@ -1512,12 +1512,94 @@ with self._confLock: self.conf['timeOffset'] = newTimeOffset + def _getWriteWatermarks(self): + def pathToVolID(drive, path): + for vol in drive.volumeChain: + if os.path.realpath(vol['path']) == os.path.realpath(path): + return vol['volumeID'] + raise LookupError("Unable to find VolumeID for path '%s'", path) + + volAllocMap = {} + statsFlags = self._libvirtBackingChainStatsFlag() + conn = libvirtconnection.get() + blkStats = conn.domainListGetStats([self._dom._dom], + libvirt.VIR_DOMAIN_STATS_BLOCK, + statsFlags)[0][1] + for i in xrange(0, blkStats['block.count']): + name = blkStats['block.%i.name' % i] + try: + drive = self._findDriveByName(name) + except LookupError: + continue + if not drive.blockDev or drive.format != 'cow': + continue + + try: + path = blkStats['block.%i.path' % i] + alloc = blkStats['block.%i.allocation' % i] + except KeyError as e: + self.log.debug("Block stats are missing expected key '%s', " + "skipping volume", e.args[0]) + continue + volID = pathToVolID(drive, path) + volAllocMap[volID] = alloc + return volAllocMap + + def _getLiveMergeExtendCandidates(self): + # The common case is that there are no active jobs. + if not self.conf['_blockJobs'].values(): + return {} + + candidates = {} + watermarks = self._getWriteWatermarks() + for job in self.conf['_blockJobs'].values(): + try: + drive = self._findDriveByUUIDs(job['disk']) + except LookupError: + # After an active layer merge completes the vdsm metadata will + # be out of sync for a brief period. If we cannot find the old + # disk then it's safe to skip it. + continue + + if not drive.blockDev: + continue + + if job['strategy'] == 'commit': + volumeID = job['baseVolume'] + else: + self.log.debug("Unrecognized merge strategy '%s'", + job['strategy']) + continue + res = self.cif.irs.getVolumeInfo(drive.domainID, drive.poolID, + drive.imageID, volumeID) + if res['status']['code'] != 0: + self.log.error("Unable to get the info of volume %s (domain: " + "%s image: %s)", volumeID, drive.domainID, + drive.imageID) + continue + volInfo = res['info'] + + if volInfo['format'].lower() != 'cow': + continue + + if volumeID in watermarks: + self.log.debug("Adding live merge extension candidate: " + "volume=%s allocation=%i", volumeID, + watermarks[volumeID]) + candidates[drive.imageID] = { + 'alloc': watermarks[volumeID], + 'physical': int(volInfo['truesize']), + 'capacity': int(volInfo['apparentsize']), + 'volumeID': volumeID} + else: + self.log.warning("No watermark info available for %s", + volumeID) + return candidates + def _getExtendCandidates(self): ret = [] - # FIXME: mergeCandidates should be a dictionary of candidate volumes - # once libvirt starts reporting watermark information for all volumes. - mergeCandidates = {} + mergeCandidates = self._getLiveMergeExtendCandidates() for drive in self._devices[hwclass.DISK]: if not drive.blockDev or drive.format != 'cow': continue @@ -4771,6 +4853,14 @@ jobsRet[jobID] = entry return jobsRet + def _libvirtBackingChainStatsFlag(self): + # Since libvirt 1.2.13, the virConnectGetAllDomainStats API will return + # block statistics for all volumes in the chain when using a new flag. + try: + return libvirt.VIR_CONNECT_GET_ALL_DOMAINS_STATS_BACKING + except AttributeError: + return 0 + def merge(self, driveSpec, baseVolUUID, topVolUUID, bandwidth, jobUUID): if not caps.getLiveMergeSupport(): self.log.error("Live merge is not supported on this host") @@ -4815,6 +4905,8 @@ if res['info']['voltype'] == 'SHARED': self.log.error("merge: Refusing to merge into a shared volume") return errCode['mergeErr'] + baseSize = int(res['info']['apparentsize']) + baseCow = bool(res['info']['format'].lower() == 'cow') # Indicate that we expect libvirt to maintain the relative paths of # backing files. This is necessary to ensure that a volume chain is @@ -4865,13 +4957,19 @@ # blockCommit will cause data to be written into the base volume. # Perform an initial extension to ensure there is enough space to - # copy all the required data. Normally we'd use monitoring to extend - # the volume on-demand but internal watermark information is not being - # reported by libvirt so we must do the full extension up front. In - # the worst case, we'll need to extend 'base' to the same size as 'top' - # plus a bit more to accomodate additional writes to 'top' during the - # live merge operation. - self.extendDriveVolume(drive, baseVolUUID, topSize) + # copy all the required data. If libvirt supports monitoring of + # backing chain volumes, just extend by one chunk now and monitor + # during the rest of the operation. Otherwise, extend now to + # accomodate the worst case scenario: no intersection between the + # allocated blocks in the base volume and the top volume. + if drive.blockDev and baseCow: + if self._libvirtBackingChainStatsFlag(): + self.extendDrivesIfNeeded() + else: + extendSize = baseSize + topSize + self.log.debug("Preemptively extending volume %s with size %i" + "(job: %s)", baseVolUUID, extendSize, jobUUID) + self.extendDriveVolume(drive, baseVolUUID, extendCurSize) # Trigger the collection of stats before returning so that callers # of getVmStats after this returns will see the new job -- To view, visit http://gerrit.ovirt.org/36924 To unsubscribe, visit http://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I632f31e7795ec5d8c6f52a480116b14470c3163f Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Adam Litke <alitke(a)redhat.com>

6 24

Change in vdsm[master]: GuestIF Refactoring
by Vinzenz Feenstra 08 Oct '16

08 Oct '16

Vinzenz Feenstra has uploaded a new change for review. Change subject: GuestIF Refactoring ...................................................................... GuestIF Refactoring Change-Id: Ib357d770a26ef1dc80b89a32bf6808551a7d622d Signed-off-by: Vinzenz Feenstra <vfeenstr(a)redhat.com> --- M vdsm/guestIF.py 1 file changed, 114 insertions(+), 76 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/18/24618/1 diff --git a/vdsm/guestIF.py b/vdsm/guestIF.py index 229a55d..96ad68c 100644 --- a/vdsm/guestIF.py +++ b/vdsm/guestIF.py @@ -39,6 +39,115 @@ union(set(range(0x86, 0x9F + 1))) +class UnknownMessageError(Exception): + def __init__(self, message, args): + Exception.__init__(self, 'Unknown or unsupported guest agent message ' + '"%s" received with args "%s"' % (message, + str(args))) + + +class MessageHandler(object): + def __init__(self, agent): + self.log = agent.log + self._agent = agent + + def __call__(self, message, args): + handler = self.getattr(self, message.replace('-', '_'), None) + if handler: + handler(args) + else: + raise UnknownMessageError(message, args) + + def applications(self, args): + self._agent.guestInfo['appsList'] = args['applications'] + + def fqdn(self, args): + self._agent.guestInfo['guestFQDN'] = args['fqdn'] + + def host_name(self, args): + self._agent.guestInfo['guestName'] = args['name'] + + def os_version(self, args): + self._agent.guestInfo['guestOs'] = args['version'] + + def session_lock(self, args): + self.agent.guestInfo['session'] = 'Locked' + + def session_logoff(self, args): + self.agent.guestInfo['session'] = 'LoggedOff' + + def session_logon(self, args): + self.agent.guestInfo['session'] = 'UserLoggedOn' + + def session_unlock(self, args): + self.agent.guestInfo['session'] = 'Active' + + def session_shutdown(self, args): + self.log.debug('Guest system shuts down') + + def session_startup(self, args): + self.log.debug('Guest system started or restarted') + + def uninstalled(self, args): + self.log.debug('Guest agent was uninstalled') + self._agent.guestInfo['appsList'] = [] + + def heartbeat(self, args): + self._agent.guestStatus = 'Up' + self._agent.guestInfo['memUsage'] = int(args['free-ram']) + # ovirt-guest-agent reports the following fields in 'memory-stat': + # 'mem_total', 'mem_free', 'mem_unused', 'swap_in', 'swap_out', + # 'pageflt' and 'majflt' + if 'memory-stat' in args: + for (k, v) in args['memory-stat'].iteritems(): + # Convert the value to string since 64-bit integer is not + # supported in XMLRPC + self._agent.guestInfo['memoryStats'][k] = str(v) + + if 'apiVersion' in args: + # The guest agent supports API Versioning + self._agent._handleAPIVersion(args['apiVersion']) + elif self._agent.effectiveApiVersion != _IMPLICIT_API_VERSION_ZERO: + # Older versions of the guest agent (before the introduction + # of API versioning) do not report this field + # Disable the API if not already disabled (e.g. after + # downgrade of the guest agent) + self.log.debug("API versioning no longer reported by guest.") + self._agent.effectiveApiVersion = _IMPLICIT_API_VERSION_ZERO + + def network_interfaces(self, args): + interfaces = [] + old_ips = '' + for iface in args['interfaces']: + iface['inet'] = iface.get('inet', []) + iface['inet6'] = iface.get('inet6', []) + interfaces.append(iface) + # Provide the old information which includes + # only the IP addresses. + old_ips += ' '.join(iface['inet']) + ' ' + self._agent.guestInfo['netIfaces'] = interfaces + self._agent.guestInfo['guestIPs'] = old_ips.strip() + + def active_user(self, args): + currentUser = args['name'] + if ((currentUser != self._agent.guestInfo['username']) and + not (currentUser == 'Unknown' and + self._agent.guestInfo['username'] == 'None')): + self._agent.guestInfo['username'] = currentUser + self._agent.guestInfo['lastLogin'] = time.time() + self.log.debug("username: %s", repr(self.guestInfo['username'])) + + def disks_usage(self, args): + disks = [] + for disk in args['disks']: + # Converting to string because XML-RPC doesn't support 64-bit + # integers. + disk['total'] = str(disk['total']) + disk['used'] = str(disk['used']) + disks.append(disk) + self._agent.guestInfo['disksUsage'] = disks + + def _filterXmlChars(u): """ The set of characters allowed in XML documents is described in @@ -109,6 +218,7 @@ def __init__(self, socketName, channelListener, log, user='Unknown', ips='', connect=True): + self.handler = MessageHandler(self) self.effectiveApiVersion = _IMPLICIT_API_VERSION_ZERO self.log = log self._socketName = socketName @@ -223,82 +333,10 @@ self.log.log(logging.TRACE, "Guest's message %s: %s", message, args) if self.guestStatus is None: self.guestStatus = 'Up' - if message == 'heartbeat': - self.guestStatus = 'Up' - self.guestInfo['memUsage'] = int(args['free-ram']) - # ovirt-guest-agent reports the following fields in 'memory-stat': - # 'mem_total', 'mem_free', 'mem_unused', 'swap_in', 'swap_out', - # 'pageflt' and 'majflt' - if 'memory-stat' in args: - for (k, v) in args['memory-stat'].iteritems(): - # Convert the value to string since 64-bit integer is not - # supported in XMLRPC - self.guestInfo['memoryStats'][k] = str(v) - - if 'apiVersion' in args: - # The guest agent supports API Versioning - self._handleAPIVersion(args['apiVersion']) - elif self.effectiveApiVersion != _IMPLICIT_API_VERSION_ZERO: - # Older versions of the guest agent (before the introduction - # of API versioning) do not report this field - # Disable the API if not already disabled (e.g. after - # downgrade of the guest agent) - self.log.debug("API versioning no longer reported by guest.") - self.effectiveApiVersion = _IMPLICIT_API_VERSION_ZERO - elif message == 'host-name': - self.guestInfo['guestName'] = args['name'] - elif message == 'os-version': - self.guestInfo['guestOs'] = args['version'] - elif message == 'network-interfaces': - interfaces = [] - old_ips = '' - for iface in args['interfaces']: - iface['inet'] = iface.get('inet', []) - iface['inet6'] = iface.get('inet6', []) - interfaces.append(iface) - # Provide the old information which includes - # only the IP addresses. - old_ips += ' '.join(iface['inet']) + ' ' - self.guestInfo['netIfaces'] = interfaces - self.guestInfo['guestIPs'] = old_ips.strip() - elif message == 'applications': - self.guestInfo['appsList'] = args['applications'] - elif message == 'active-user': - currentUser = args['name'] - if ((currentUser != self.guestInfo['username']) and - not (currentUser == 'Unknown' and - self.guestInfo['username'] == 'None')): - self.guestInfo['username'] = currentUser - self.guestInfo['lastLogin'] = time.time() - self.log.debug("username: %s", repr(self.guestInfo['username'])) - elif message == 'session-logon': - self.guestInfo['session'] = "UserLoggedOn" - elif message == 'session-lock': - self.guestInfo['session'] = "Locked" - elif message == 'session-unlock': - self.guestInfo['session'] = "Active" - elif message == 'session-logoff': - self.guestInfo['session'] = "LoggedOff" - elif message == 'uninstalled': - self.log.debug("RHEV agent was uninstalled.") - self.guestInfo['appsList'] = [] - elif message == 'session-startup': - self.log.debug("Guest system is started or restarted.") - elif message == 'fqdn': - self.guestInfo['guestFQDN'] = args['fqdn'] - elif message == 'session-shutdown': - self.log.debug("Guest system shuts down.") - elif message == 'disks-usage': - disks = [] - for disk in args['disks']: - # Converting to string because XML-RPC doesn't support 64-bit - # integers. - disk['total'] = str(disk['total']) - disk['used'] = str(disk['used']) - disks.append(disk) - self.guestInfo['disksUsage'] = disks - else: - self.log.error('Unknown message type %s', message) + try: + self.handler(message, args) + except UnknownMessageError as e: + self.log.error(e) def stop(self): self._stopped = True -- To view, visit http://gerrit.ovirt.org/24618 To unsubscribe, visit http://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Ib357d770a26ef1dc80b89a32bf6808551a7d622d Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Vinzenz Feenstra <vfeenstr(a)redhat.com>

5 13

Change in vdsm[master]: libvirtconnection: Replace assert with AssertionError
by Nir Soffer 25 Sep '16

25 Sep '16

Nir Soffer has uploaded a new change for review. Change subject: libvirtconnection: Replace assert with AssertionError ...................................................................... libvirtconnection: Replace assert with AssertionError The code wrongly assumed that assert always exists. When running in optimized mode, the check would be skipped, and instead of getting an AssertionError, which is the expected error for programmer error (starting the eventloop twice), we could get a confusing RuntimeException or RuntimeError from Thread.start (depending on Python version). RuntimeError misused in the standard library for all kinds of errors that do not have builtin errors. It is particularry bad option when used for usage error. Change-Id: Icf1564f81f4c1fbf77ccaff6d93c047a02d946da Signed-off-by: Nir Soffer <nsoffer(a)redhat.com> --- M lib/vdsm/libvirtconnection.py 1 file changed, 2 insertions(+), 1 deletion(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/64/34364/1 diff --git a/lib/vdsm/libvirtconnection.py b/lib/vdsm/libvirtconnection.py index 5430c82..009f8b7 100644 --- a/lib/vdsm/libvirtconnection.py +++ b/lib/vdsm/libvirtconnection.py @@ -37,7 +37,8 @@ self.__thread = None def start(self): - assert not self.run + if self.run: + raise AssertionError("EventLoop is running") self.__thread = threading.Thread(target=self.__run, name="libvirtEventLoop") self.__thread.setDaemon(True) -- To view, visit http://gerrit.ovirt.org/34364 To unsubscribe, visit http://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Icf1564f81f4c1fbf77ccaff6d93c047a02d946da Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Nir Soffer <nsoffer(a)redhat.com>

6 8

Change in vdsm[master]: cache: Add caching decorator with invalidation
by Nir Soffer 18 Sep '16

18 Sep '16

Nir Soffer has uploaded a new change for review. Change subject: cache: Add caching decorator with invalidation ...................................................................... cache: Add caching decorator with invalidation The new cache.memoized extends utils.memoized, adding invalidation support. Features added: - An optional "validate" argument. This is a callable invoked each time the memoized function is called. When the callable returns False, the cache is invalidated. - Memoized functions have an "invalidate" method, used to invalidate the cache during testing. - file_validator - invalidates the cache when a file changes. Example usage: from vdsm.cache import memoized, file_validator @memoized(file_validator('/bigfile')) def parse_bigfile(): # Expensive code processing '/bigfile' contents Change-Id: I6dd8fb29d94286e3e3a3e29b8218501cbdc5c018 Signed-off-by: Nir Soffer <nsoffer(a)redhat.com> --- M lib/vdsm/Makefile.am A lib/vdsm/cache.py M tests/Makefile.am A tests/cacheTests.py 4 files changed, 366 insertions(+), 0 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/09/34709/1 diff --git a/lib/vdsm/Makefile.am b/lib/vdsm/Makefile.am index b862e71..6f0040d 100644 --- a/lib/vdsm/Makefile.am +++ b/lib/vdsm/Makefile.am @@ -23,6 +23,7 @@ dist_vdsmpylib_PYTHON = \ __init__.py \ + cache.py \ compat.py \ define.py \ exception.py \ diff --git a/lib/vdsm/cache.py b/lib/vdsm/cache.py new file mode 100644 index 0000000..9806e40 --- /dev/null +++ b/lib/vdsm/cache.py @@ -0,0 +1,98 @@ +# +# Copyright 2014 Red Hat, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA +# +# Refer to the README and COPYING files for full details of the license +# + +import errno +import os +import functools + + +def memoized(validate=None): + """ + Return a caching decorator supporting invalidation. + + The decorator accepts an optional validate callable, called each time the + memoized function is called. If the validate callable return True, the + memoized function will use the cache. If the validate callable return + False, the memoized cache is cleared. + + The memoized function may accept multiple positional arguments. The + cache store the result for each combination of arguments. Functions with + kwargs are not supported. + + Memoized functions have an "invalidate" method, used to invalidate the + memoized cache during testing. + + To invalidate the cache when a file changes, use the file_validator from + this module. + + Example usage: + + from vdsm.cache import memoized, file_validator + + @memoized(file_validator('/bigfile')) + def parse_bigfile(): + # Expensive code processing '/bigfile' contents + + """ + def decorator(f): + cache = {} + + @functools.wraps(f) + def wrapper(*args): + if validate is not None and not validate(): + cache.clear() + try: + value = cache[args] + except KeyError: + value = cache[args] = f(*args) + return value + + wrapper.invalidate = cache.clear + return wrapper + + return decorator + + +class file_validator(object): + """ + I'm a validator returning False when a file has changed since the last + validation. + """ + + UNKNOWN = 0 + MISSING = 1 + + def __init__(self, path): + self.path = path + self.stats = self.UNKNOWN + + def __call__(self): + try: + stats = os.stat(self.path) + except OSError as e: + if e.errno != errno.ENOENT: + raise + stats = self.MISSING + else: + stats = stats.st_ino, stats.st_size, stats.st_mtime + if stats != self.stats: + self.stats = stats + return False + return True diff --git a/tests/Makefile.am b/tests/Makefile.am index 36a1cdd..6fa7e64 100644 --- a/tests/Makefile.am +++ b/tests/Makefile.am @@ -26,6 +26,7 @@ alignmentScanTests.py \ blocksdTests.py \ bridgeTests.py \ + cacheTests.py \ cPopenTests.py \ capsTests.py \ clientifTests.py \ diff --git a/tests/cacheTests.py b/tests/cacheTests.py new file mode 100644 index 0000000..8927b39 --- /dev/null +++ b/tests/cacheTests.py @@ -0,0 +1,266 @@ +# +# Copyright 2014 Red Hat, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA +# 02110-1301 USA +# +# Refer to the README and COPYING files for full details of the license +# + +import os +from vdsm.cache import memoized +from vdsm.cache import file_validator +from testlib import VdsmTestCase +from testlib import namedTemporaryDir + + +class Validator(object): + """ I'm a callable returning a boolean value (self.valid) """ + + def __init__(self): + self.valid = True + self.count = 0 + + def __call__(self): + self.count += 1 + return self.valid + + +class Accessor(object): + """ I'm recording how many times a dict was accessed. """ + + def __init__(self, d): + self.d = d + self.count = 0 + + def get(self, key): + self.count += 1 + return self.d[key] + + +class MemoizedTests(VdsmTestCase): + + def setUp(self): + self.values = {'a': 0, 'b': 10, ('a',): 20, ('a', 'b'): 30} + + def test_no_args(self): + accessor = Accessor(self.values) + + @memoized() + def func(key): + return accessor.get(key) + + # Fill the cache + self.assertEqual(func('a'), self.values['a']) + self.assertEqual(func('b'), self.values['b']) + self.assertEqual(accessor.count, 2) + + # Values served now from the cache + self.assertEqual(func('a'), self.values['a']) + self.assertEqual(func('b'), self.values['b']) + self.assertEqual(accessor.count, 2) + + def test_validation(self): + accessor = Accessor(self.values) + validator = Validator() + + @memoized(validator) + def func(key): + return accessor.get(key) + + # Fill the cache + self.assertEqual(func('a'), self.values['a']) + self.assertEqual(func('b'), self.values['b']) + self.assertEqual(accessor.count, 2) + self.assertEqual(validator.count, 2) + + # Values served now from the cache + self.assertEqual(func('a'), self.values['a']) + self.assertEqual(func('b'), self.values['b']) + self.assertEqual(accessor.count, 2) + self.assertEqual(validator.count, 4) + + # Values has changed + self.values['a'] += 1 + self.values['b'] += 1 + + # Next call should clear the cache + validator.valid = False + self.assertEqual(func('a'), self.values['a']) + self.assertEqual(accessor.count, 3) + self.assertEqual(validator.count, 5) + + # Next call should add next value to cache + validator.valid = True + self.assertEqual(func('b'), self.values['b']) + self.assertEqual(accessor.count, 4) + self.assertEqual(validator.count, 6) + + # Values served now from the cache + self.assertEqual(func('a'), self.values['a']) + self.assertEqual(func('b'), self.values['b']) + self.assertEqual(accessor.count, 4) + self.assertEqual(validator.count, 8) + + def test_raise_errors_in_memoized_func(self): + accessor = Accessor(self.values) + validator = Validator() + + @memoized(validator) + def func(key): + return accessor.get(key) + + # First run should fail, second shold fill the cache + self.assertRaises(KeyError, func, 'no such key') + self.assertEqual(func('a'), self.values['a']) + self.assertEqual(accessor.count, 2) + self.assertEqual(validator.count, 2) + + def test_multiple_args(self): + accessor = Accessor(self.values) + + @memoized() + def func(*args): + return accessor.get(args) + + # Fill the cache + self.assertEqual(func('a'), self.values[('a',)]) + self.assertEqual(func('a', 'b'), self.values[('a', 'b')]) + self.assertEqual(accessor.count, 2) + + # Values served now from the cache + self.assertEqual(func('a'), self.values[('a',)]) + self.assertEqual(func('a', 'b'), self.values[('a', 'b')]) + self.assertEqual(accessor.count, 2) + + def test_kwargs_not_supported(self): + @memoized() + def func(a=None, b=None): + pass + self.assertRaises(TypeError, func, a=1, b=2) + + def test_invalidate(self): + accessor = Accessor(self.values) + + @memoized() + def func(key): + return accessor.get(key) + + self.assertEqual(func('a'), self.values['a']) + self.assertEqual(accessor.count, 1) + + func.invalidate() + + self.assertEqual(func('a'), self.values['a']) + self.assertEqual(accessor.count, 2) + + +class FileValidatorTests(VdsmTestCase): + + def test_no_file(self): + with namedTemporaryDir() as tempdir: + path = os.path.join(tempdir, 'data') + validator = file_validator(path) + + # Must be False so memoise call the decorated function + self.assertEqual(validator(), False) + + # Since file state did not change, must remain True + self.assertEqual(validator(), True) + + def test_file_created(self): + with namedTemporaryDir() as tempdir: + path = os.path.join(tempdir, 'data') + validator = file_validator(path) + + self.assertEqual(validator(), False) + self.assertEqual(validator(), True) + + with open(path, 'w') as f: + f.write('data') + + self.assertEqual(validator(), False) + self.assertEqual(validator(), True) + + def test_file_removed(self): + with namedTemporaryDir() as tempdir: + path = os.path.join(tempdir, 'data') + validator = file_validator(path) + + with open(path, 'w') as f: + f.write('data') + + self.assertEqual(validator(), False) + self.assertEqual(validator(), True) + + os.unlink(path) + + self.assertEqual(validator(), False) + self.assertEqual(validator(), True) + + def test_size_changed(self): + with namedTemporaryDir() as tempdir: + path = os.path.join(tempdir, 'data') + validator = file_validator(path) + data = 'old data' + with open(path, 'w') as f: + f.write(data) + + self.assertEqual(validator(), False) + self.assertEqual(validator(), True) + + with open(path, 'w') as f: + f.write(data + ' new data') + + self.assertEqual(validator(), False) + self.assertEqual(validator(), True) + + def test_mtime_changed(self): + with namedTemporaryDir() as tempdir: + path = os.path.join(tempdir, 'data') + validator = file_validator(path) + data = 'old data' + with open(path, 'w') as f: + f.write(data) + + self.assertEqual(validator(), False) + self.assertEqual(validator(), True) + + # Fake timestamp change, as timestamp resolution may not be good + # enough when comparing changes during the test. + atime = mtime = os.path.getmtime(path) + 1 + os.utime(path, (atime, mtime)) + + self.assertEqual(validator(), False) + self.assertEqual(validator(), True) + + def test_ino_changed(self): + with namedTemporaryDir() as tempdir: + path = os.path.join(tempdir, 'data') + validator = file_validator(path) + data = 'old data' + with open(path, 'w') as f: + f.write(data) + + self.assertEqual(validator(), False) + self.assertEqual(validator(), True) + + tmp = path + '.tmp' + with open(tmp, 'w') as f: + f.write(data) + os.rename(tmp, path) + + self.assertEqual(validator(), False) + self.assertEqual(validator(), True) -- To view, visit http://gerrit.ovirt.org/34709 To unsubscribe, visit http://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I6dd8fb29d94286e3e3a3e29b8218501cbdc5c018 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Nir Soffer <nsoffer(a)redhat.com>

9 40

Jump to page:

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

vdsm-patches July 2015