LNST-developers March 2016

lnst-developers@lists.fedorahosted.org

5 participants
32 discussions

[PATCH] simple_netperf: fix typo when running IPv6 UDP_STREAM netperf
by Jiri Prochazka 21 Mar '16

21 Mar '16

We have a typo in the task file, where IPv4 UDP_STREAM version is run instead of IPv6 version. This patch fixes that. Signed-off-by: Jiri Prochazka <jprochaz(a)redhat.com> --- recipes/regression_tests/phase1/simple_netperf.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/recipes/regression_tests/phase1/simple_netperf.py b/recipes/regression_tests/phase1/simple_netperf.py index 8c0d6ef..3eb2a61 100644 --- a/recipes/regression_tests/phase1/simple_netperf.py +++ b/recipes/regression_tests/phase1/simple_netperf.py @@ -246,7 +246,7 @@ for setting in offload_settings: baseline = perf_api.get_baseline_of_result(result_udp) netperf_baseline_template(netperf_cli_udp6, baseline) - udp_res_data = m2.run(netperf_cli_udp, + udp_res_data = m2.run(netperf_cli_udp6, timeout = (netperf_duration + nperf_reserve)*nperf_max_runs) netperf_result_template(result_udp, udp_res_data) -- 2.4.3

3 2

[PATCH v2] SecureSocket: use fallback function if bit_length is not available in Python
by Jan Tluka 21 Mar '16

21 Mar '16

Recently added SecureSocket feature brought dependency on Python 2.7 namely bit_length() method. Since we need to keep compatibility with Python 2.6 that's on RHEL6 I added a fallback if bit_length is not available. I compared the DH_GROUP and SRP_GROUP dicts both on Python2.6 and 2.7 and they had the same content (bit length), so this patch should be safe. Fixes issue 166. Signed-off-by: Jan Tluka <jtluka(a)redhat.com> --- lnst/Common/SecureSocket.py | 25 +++++++++++++++++-------- 1 file changed, 17 insertions(+), 8 deletions(-) diff --git a/lnst/Common/SecureSocket.py b/lnst/Common/SecureSocket.py index b4ee8af..b856238 100644 --- a/lnst/Common/SecureSocket.py +++ b/lnst/Common/SecureSocket.py @@ -34,12 +34,21 @@ DH_GROUP = {"p": int("0xFFFFFFFFFFFFFFFFC90FDAA22168C234C4C6628B80DC1CD1"\ "15728E5A8AACAA68FFFFFFFFFFFFFFFF", 16), "g": 2} +# to support both Python2.6 and 2.7, use following workaround to count +# the bit length of a number; note that the workaround does not work for +# value 0 but we don't use it for such value +def bit_length(i): + try: + return i.bit_length() + except AttributeError: + return len(bin(i)) - 2 + DH_GROUP["q"] = (DH_GROUP["p"]-1)/2 -DH_GROUP["q_size"] = DH_GROUP["q"].bit_length()/8 -if DH_GROUP["q"].bit_length()%8: +DH_GROUP["q_size"] = bit_length(DH_GROUP["q"])/8 +if bit_length(DH_GROUP["q"])%8: DH_GROUP["q_size"] += 1 -DH_GROUP["p_size"] = DH_GROUP["p"].bit_length()/8 -if DH_GROUP["p"].bit_length()%8: +DH_GROUP["p_size"] = bit_length(DH_GROUP["p"])/8 +if bit_length(DH_GROUP["p"])%8: DH_GROUP["p_size"] += 1 SRP_GROUP = {"p": int("0xAC6BDB41324A9A9BF166DE5E1389582FAF72B6651987EE07FC" @@ -55,11 +64,11 @@ SRP_GROUP = {"p": int("0xAC6BDB41324A9A9BF166DE5E1389582FAF72B6651987EE07FC" "g": 2} SRP_GROUP["q"] = (SRP_GROUP["p"]-1)/2 -SRP_GROUP["q_size"] = SRP_GROUP["q"].bit_length()/8 -if SRP_GROUP["q"].bit_length()%8: +SRP_GROUP["q_size"] = bit_length(SRP_GROUP["q"])/8 +if bit_length(SRP_GROUP["q"])%8: SRP_GROUP["q_size"] += 1 -SRP_GROUP["p_size"] = SRP_GROUP["p"].bit_length()/8 -if SRP_GROUP["p"].bit_length()%8: +SRP_GROUP["p_size"] = bit_length(SRP_GROUP["p"])/8 +if bit_length(SRP_GROUP["p"])%8: SRP_GROUP["p_size"] += 1 class SecSocketException(Exception): -- 2.4.3

1 1

[PATCH 1/3] lnst-ctl: add list_pools action
by Jiri Prochazka 21 Mar '16

21 Mar '16

This patch adds new action called list_pools. It has two modes of execution, non-verbose and verbose. Non verbose, executed just with "lnst-ctl list_pools" command, prints out pool names and their paths in pool_name=/path/to/pool format. For verbose mode, option -v or --verbose must be entered. It prints pool names and their paths and in addition, it lists through all .xml files in pool directories and prints their contents in more human readable format. Pools are acquired from all the configs lnst-ctl would use in normal run. Signed-off-by: Jiri Prochazka <jprochaz(a)redhat.com> --- lnst-ctl | 63 +++++++++++++++++++++++++++++++++++++++++++++++++-------------- 1 file changed, 49 insertions(+), 14 deletions(-) diff --git a/lnst-ctl b/lnst-ctl index 6d656f8..5a32f7c 100755 --- a/lnst-ctl +++ b/lnst-ctl @@ -24,6 +24,8 @@ from lnst.Common.Utils import mkdir_p from lnst.Controller.NetTestController import NetTestController, NetTestError from lnst.Controller.NetTestController import NoMatchError from lnst.Controller.NetTestResultSerializer import NetTestResultSerializer +from lnst.Controller.SlaveMachineParser import SlaveMachineParser +from lnst.Controller.SlavePool import SlavePool from lnst.Controller.XmlProcessing import XmlProcessingError RETVAL_PASS = 0 @@ -35,7 +37,8 @@ def usage(retval=0): """ print "Usage: %s [OPTIONS...] ACTION [RECIPES...]" % sys.argv[0] print "" - print "ACTION = [ run | config_only | deconfigure | match_setup ]" + print "ACTION = [ run | config_only | deconfigure | match_setup | " \ + "list_pools]" print "" print "OPTIONS" print " -A, --override-alias name=value define top-level alias that " \ @@ -45,8 +48,8 @@ def usage(retval=0): print " -d, --debug emit debugging messages" print " --dump-config dumps the join of all loaded " \ "configuration files on stdout and exits" - print " --dump-pools dumps the available machine " \ - "pools and exits" + print " -v, --verbose verbose version of list_pools " \ + "command" print " -h, --help print this message" print " -m, --no-colours disable coloured terminal output" print " -o, --disable-pool-checks don't check the availability of " \ @@ -67,6 +70,35 @@ def usage(retval=0): print " -x, --result=FILE file to write xml_result" sys.exit(retval) +def list_pools(): + # iterate over all pools + for pool_name, pool_path in lnst_config.get_pools().items(): + print "%s (%s)" % (pool_name, pool_path) + dentries = os.listdir(pool_path) + # iterate over all slave machine cfgs + for dirent in dentries: + filepath = pool_path + "/" + dirent + dirname, basename = os.path.split(filepath) + if os.path.isfile(filepath) and re.search("\.xml$", filepath, re.I): + parser = SlaveMachineParser(filepath) + xml_data = parser.parse() + # parse XML to dict + res = SlavePool._process_machine_xml_data(SlavePool({}), "", xml_data) + # print in human-readable format + print " ", "Filename: ", basename + if res['params']: + print 3*" ", "Parameters:" + for key in res['params']: + print 7*" ", key+":", res['params'][key] + if res['interfaces']: + print 3*" ", "Interfaces:" + for key in res['interfaces']: + print 5*" ", "id:", key, "\tnetwork:", res['interfaces'][key]['network'] + for param in res['interfaces'][key]['params']: + print 7*" ", param+":", res['interfaces'][key]['params'][param] + print "" + return RETVAL_PASS + def store_alias(alias_def, aliases_dict): try: name, value = alias_def.split("=") @@ -189,7 +221,7 @@ def main(): try: opts, args = getopt.getopt( sys.argv[1:], - "A:a:c:dhmoprs:t:ux:", + "A:a:c:dhmoprs:t:ux:v", [ "override_alias=", "define_alias=", @@ -206,7 +238,7 @@ def main(): "multi-match", "result=", "pools=", - "dump-pools" + "verbose" ] ) except getopt.GetoptError as err: @@ -246,7 +278,7 @@ def main(): reduce_sync = False multi_match = False dump_config = False - dump_pools = False + verbose = False pools = [] for opt, arg in opts: if opt in ("-d", "--debug"): @@ -283,8 +315,8 @@ def main(): dump_config = True elif opt in ("--pools"): pools.extend(arg.split(",")) - elif opt in ("--dump-pools"): - dump_pools = True + elif opt in ("-v", "--verbose"): + verbose= True if xslt_url != None: lnst_config.set_option("environment", "xslt_url", xslt_url) @@ -293,11 +325,6 @@ def main(): print lnst_config.dump_config() return RETVAL_PASS - if dump_pools: - for pool_name, pool_path in lnst_config.get_pools().items(): - print pool_name + ' = ' + pool_path - return RETVAL_PASS - if coloured_output: coloured_output = not lnst_config.get_option("colours", "disable_colours") @@ -315,12 +342,20 @@ def main(): action = args[0] recipes = args[1:] - if not action in ['run', 'config_only', 'deconfigure', 'match_setup']: + if not action in ['run', 'config_only', 'deconfigure', 'match_setup', + 'list_pools']: logging.error("Action '%s' not recognised" % action) usage(RETVAL_ERR) if action == 'deconfigure': NetTestController.remove_saved_machine_config() return 0 + elif action == 'list_pools': + logging.disable(logging.CRITICAL) + for pool_name, pool_path in lnst_config.get_pools().items(): + print pool_name + ' = ' + pool_path + if verbose: + list_pools() + return RETVAL_PASS if recipes == []: logging.error("No recipe specified!") -- 2.4.3

3 8

[PATCH] lnst-ctl: update log_dir properly for multimatch mode
by Jan Tluka 21 Mar '16

21 Mar '16

In multimatch mode the log_dir was not set properly and the initial value (match_1) was used everytime the match was performed. Fixes issue 164. Signed-off-by: Jan Tluka <jtluka(a)redhat.com> --- lnst-ctl | 1 + 1 file changed, 1 insertion(+) diff --git a/lnst-ctl b/lnst-ctl index 6d656f8..f3c2eb2 100755 --- a/lnst-ctl +++ b/lnst-ctl @@ -140,6 +140,7 @@ def get_recipe_result(action, file_path, log_ctl, res_serializer, try: nettestctl.provision_machines() log_ctl.set_recipe(file_path, expand="match_%d" % matches) + log_dir = log_ctl.get_recipe_log_path() recipe_head_log_entry(file_path, log_dir, matches) res_serializer.add_recipe(file_path, matches) nettestctl.print_match_description() -- 2.4.3

2 2

[PATCH 1/4] ExecCmd: include stdout data in ExecCmdFail if command failed
by Jan Tluka 21 Mar '16

21 Mar '16

ExecCmdFail exception is thrown if a command returns with non-zero returncode. When this happens only stderr is saved in the exception. User might want to work with both the stdout and stderr. This patch saves stdout data along with stderr in this case. Signed-off-by: Jan Tluka <jtluka(a)redhat.com> --- lnst/Common/ExecCmd.py | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/lnst/Common/ExecCmd.py b/lnst/Common/ExecCmd.py index d6c92c6..d432c2c 100644 --- a/lnst/Common/ExecCmd.py +++ b/lnst/Common/ExecCmd.py @@ -17,10 +17,12 @@ class ExecCmdFail(Exception): _cmd = None _retval = None _stderr = None + _stdout = None _report_stderr = None - def __init__(self, cmd=None, retval=None, err="", report_stderr=False): - self._stderr = err + def __init__(self, cmd=None, retval=None, outs=["", ""], report_stderr=False): + self._stdout = outs[0] + self._stderr = outs[1] self._retval = retval self._report_stderr = report_stderr @@ -30,6 +32,9 @@ class ExecCmdFail(Exception): def get_stderr(self): return self._stderr + def get_stdout(self): + return self._stdout + def __str__(self): retval = "" stderr = "" @@ -63,7 +68,7 @@ def exec_cmd(cmd, die_on_err=True, log_outputs=True, report_stderr=False): if data_stderr: log_output(logging.debug, "Stderr", data_stderr) if subp.returncode and die_on_err: - err = ExecCmdFail(cmd, subp.returncode, data_stderr,report_stderr) + err = ExecCmdFail(cmd, subp.returncode, [data_stdout, data_stderr], report_stderr) logging.error(err) raise err -- 2.4.3

2 7

[PATCH] TaskAPI: fix exception when perfrepo credentials are not in config
by Jiri Prochazka 21 Mar '16

21 Mar '16

When user does not have PerfRepo installed and tries to use LNST PerfRepo commands, user is informed, PerfRepo commands are ignored and test is run without them. When user does have PerfRepo installed, but does not have url/username/password informations in config file, LNST crashes on PerfRepoRESTAPIException because it tries to connect to URL "". Desired behaviour would be - inform user, that he does not have needed credentials in config and ignore the commands in the same way they are ignored when the PerfRepo is not installed on the machine. This patch fixes that. User gets warning logging messages for all of credentials that are missing and that PerfRepo commands will be ignored and test is run without PerfRepo commands. Signed-off-by: Jiri Prochazka <jprochaz(a)redhat.com> --- lnst/Controller/Task.py | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/lnst/Controller/Task.py b/lnst/Controller/Task.py index 54c9d3f..901d796 100644 --- a/lnst/Controller/Task.py +++ b/lnst/Controller/Task.py @@ -136,7 +136,15 @@ class ControllerAPI(object): username = lnst_config.get_option("perfrepo", "username") if password is None: password = lnst_config.get_option("perfrepo", "password") - self._perf_repo_api.connect(url, username, password) + + if not url: + logging.warn("No PerfRepo URL specified in config file") + if not username: + logging.warn("No PerfRepo username specified in config file") + if not password: + logging.warn("No PerfRepo password specified in config file") + if url and username and password: + self._perf_repo_api.connect(url, username, password) root = Path(None, self._ctl._recipe_path).get_root() path = Path(root, mapping_file) -- 2.4.3

3 3

[PATCH] XmlParser: allow use HTTPS URL as recipe file
by Jiri Prochazka 21 Mar '16

21 Mar '16

Current lxml.etree.parse implementation allows using HTTP, but not HTTPS. When user tries to use HTTPS URL, it crashes on IOException. This patch fixes that by checking if the URL starts with HTTPS and uses urlopen to open the HTTPS page. This allows lxml.etree.parse method to parse the XML from the HTTPS URL. urllib2 is already used Common/Path and is available since Python2.6 so no new dependency is added. Signed-off-by: Jiri Prochazka <jprochaz(a)redhat.com> --- lnst/Controller/XmlParser.py | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/lnst/Controller/XmlParser.py b/lnst/Controller/XmlParser.py index afc2cda..355b5e8 100644 --- a/lnst/Controller/XmlParser.py +++ b/lnst/Controller/XmlParser.py @@ -15,6 +15,7 @@ import re import sys import copy from lxml import etree +from urllib2 import urlopen from lnst.Common.Config import lnst_config from lnst.Controller.XmlTemplates import XmlTemplates from lnst.Controller.XmlProcessing import XmlProcessingError @@ -79,7 +80,10 @@ class XmlParser(object): def _parse(self, path): try: - doc = etree.parse(path) + if path.startswith('https'): + doc = etree.parse(urlopen(path)) + else: + doc = etree.parse(path) except etree.LxmlError as err: # A workaround for cases when lxml (quite strangely) # sets the filename to <string>. -- 2.4.3

3 3

[PATCH] SecureSocket: use fallback function if bit_length is not available in Python
by Jan Tluka 21 Mar '16

21 Mar '16

Recently added SecureSocket feature brought dependency on Python 2.7 namely bit_length() method. Since we need to keep compatibility with Python 2.6 that's on RHEL6 I added a fallback if bit_length is not available. I compared the DH_GROUP and SRP_GROUP dicts both on Python2.6 and 2.7 and they had the same content (bit length), so this patch should be safe. Fixes issue 166. Signed-off-by: Jan Tluka <jtluka(a)redhat.com> --- lnst/Common/SecureSocket.py | 27 +++++++++++++++++++-------- 1 file changed, 19 insertions(+), 8 deletions(-) diff --git a/lnst/Common/SecureSocket.py b/lnst/Common/SecureSocket.py index b4ee8af..734d448 100644 --- a/lnst/Common/SecureSocket.py +++ b/lnst/Common/SecureSocket.py @@ -34,12 +34,23 @@ DH_GROUP = {"p": int("0xFFFFFFFFFFFFFFFFC90FDAA22168C234C4C6628B80DC1CD1"\ "15728E5A8AACAA68FFFFFFFFFFFFFFFF", 16), "g": 2} +# to support both Python2.6 and 2.7, use following workaround to count +# the bit length of a number; note that the workaround does not work for +# value 0 but we don't use it for such value +def bit_length(i): + try: + int(1).bit_length() + except AttributeError: + return len(bin(i)) - 2 + + return i.bit_length() + DH_GROUP["q"] = (DH_GROUP["p"]-1)/2 -DH_GROUP["q_size"] = DH_GROUP["q"].bit_length()/8 -if DH_GROUP["q"].bit_length()%8: +DH_GROUP["q_size"] = bit_length(DH_GROUP["q"])/8 +if bit_length(DH_GROUP["q"])%8: DH_GROUP["q_size"] += 1 -DH_GROUP["p_size"] = DH_GROUP["p"].bit_length()/8 -if DH_GROUP["p"].bit_length()%8: +DH_GROUP["p_size"] = bit_length(DH_GROUP["p"])/8 +if bit_length(DH_GROUP["p"])%8: DH_GROUP["p_size"] += 1 SRP_GROUP = {"p": int("0xAC6BDB41324A9A9BF166DE5E1389582FAF72B6651987EE07FC" @@ -55,11 +66,11 @@ SRP_GROUP = {"p": int("0xAC6BDB41324A9A9BF166DE5E1389582FAF72B6651987EE07FC" "g": 2} SRP_GROUP["q"] = (SRP_GROUP["p"]-1)/2 -SRP_GROUP["q_size"] = SRP_GROUP["q"].bit_length()/8 -if SRP_GROUP["q"].bit_length()%8: +SRP_GROUP["q_size"] = bit_length(SRP_GROUP["q"])/8 +if bit_length(SRP_GROUP["q"])%8: SRP_GROUP["q_size"] += 1 -SRP_GROUP["p_size"] = SRP_GROUP["p"].bit_length()/8 -if SRP_GROUP["p"].bit_length()%8: +SRP_GROUP["p_size"] = bit_length(SRP_GROUP["p"])/8 +if bit_length(SRP_GROUP["p"])%8: SRP_GROUP["p_size"] += 1 class SecSocketException(Exception): -- 2.4.3

2 3

[PATCH] recipes: use variable nperf_max_runs instead of hardcoded value in team test
by Jan Tluka 21 Mar '16

21 Mar '16

Signed-off-by: Jan Tluka <jtluka(a)redhat.com> --- recipes/regression_tests/phase2/team_test.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/recipes/regression_tests/phase2/team_test.py b/recipes/regression_tests/phase2/team_test.py index 6327528..e6e262e 100644 --- a/recipes/regression_tests/phase2/team_test.py +++ b/recipes/regression_tests/phase2/team_test.py @@ -256,7 +256,7 @@ for setting in offload_settings: netperf_baseline_template(netperf_cli_tcp6, baseline) tcp_res_data = m2.run(netperf_cli_tcp6, - timeout = (netperf_duration + nperf_reserve)*5) + timeout = (netperf_duration + nperf_reserve)*nperf_max_runs) netperf_result_template(result_tcp, tcp_res_data) result_tcp.set_comment(pr_comment) -- 2.4.3

2 2

[PATCH v2 1/2] Icmp6Ping: Use correct option name
by Ido Schimmel 14 Mar '16

14 Mar '16

Fixes: 73b24ec5659f ("Icmp6Ping test") Signed-off-by: Ido Schimmel <idosch(a)mellanox.com> --- v1->v2: * No change. --- test_modules/Icmp6Ping.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/test_modules/Icmp6Ping.py b/test_modules/Icmp6Ping.py index 9197b42..405c756 100644 --- a/test_modules/Icmp6Ping.py +++ b/test_modules/Icmp6Ping.py @@ -38,7 +38,7 @@ class Icmp6Ping(TestGeneric): cmd = self.compose_cmd() logging.debug("%s" % cmd) - limit_rate = self.get_opt("limit_rate ", default=80) + limit_rate = self.get_opt("limit_rate", default=80) data_stdout = exec_cmd(cmd, die_on_err=False)[0] stat_pttr1 = r'(\d+) packets transmitted, (\d+) received' -- 2.4.10

2 2

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

LNST-developers March 2016