LNST-developers June 2012

lnst-developers@lists.fedorahosted.org

4 participants
47 discussions

Recipe Parsing Issue
by Radek Pazdera 21 Jun '12

21 Jun '12

Hi, there is one more issue I wanted to discuss with you guys, but I completely forgot to mention it at the meeting we had on Monday. It's the thing with parser implementation that came up after I finished writing libteam support. The problem is that the way the parsing is implemented now allows using template funcions (e.g. {ip(1,1)}) only within a command sequence. The functions can be only used after all machine and net configs were parsed. This makes the libteam implementation practically unusable. It works correctly, but you have to enter the device names of slave interfaces to the teamd config (which is contained within netconfig) by hand. We discussed it a while ago with Jirka and it seems, that there's no easy way around this :-/. The solution is to make the parser work on a line-by-line basis and process actions associated with certain lines right away. Now things are a little optimized, so the controller sends only one big request to each slave with the whole netconfig, which results in this not-very-intuitive behavior of recipes. I personally think, that this could really simplify conditions of using the template functions and aliases from a obscure set of rules (that can be hard to get for people not involved int LNST development) to a simple rule "you can use whatever you see written above this line". Do you think this is worth the work at the moment or I should leave it for now and get back to it later? I don't know if my explanation is clear enough ... I can explain it to you in more detail in person sometime. Radek

3 3

Dropping recipe_eval
by Radek Pazdera 21 Jun '12

21 Jun '12

What is your opinion on keeping recipe_eval options in LNST? The same functionality can be now accessed using a predefined template variable $recipe. For instance: {$recipe["machines"][1]["info"]["hostname"]} So the older way is not needed any more.In my opinion, it would be better to stick to one way of accessing the recipe to maintain consistency among the recipes, but dropping recipe_eval entirely, could break some older ones. What do you think? We could either drop it entirely or keep it in as "undocumented". Radek

3 3

Native packet capturing for LNST
by Radek Pazdera 13 Jun '12

13 Jun '12

Hi, I started working on the packet assert test this week, that we discussed earlier and we discovered, that the idea of packet capturing might have much broader use in LNST. Jirka had an idea, that we could put an optional parameter to the controler to enable debugging mode, in which communication on all interfaces would be captured by tcpdump, logged and then transfered back to controller along with the logs. This could also be useful along with tcpreplay to catch and reproduce some particularly sneaky bugs. tcpreplay can be run as an exec from command sequence to simulate previous situation within the network in later recipe runs. I tried to figure out a way to implement this efficiently, but I can't really come up with a straight-forward solution at the moment. We could use tcpdump to create a *.pcap dump file for each interface and then analyze it with capinfo (it's a part of wireshark) or again with tcpdump to get some information. Or there is a option to bypass tcpdump and use libpcap directly through some of it's python bindings (pypcap, pcapy or pylibpcap) along with dkpt library for parsing packets, but I'm not sure what's the current state of development of these things and how stable they are. I guess we could start simple with tcpdump and optimize it later in case some performance issue comes up. I also looked at tcpdump's code at how they parse the packets' headers into strings and, they have separate C modules for every supported protocol type (called print-<protocol_name>.c), so they are not using any library that we could use easily. Then there's a native capture tool for python called "scapy", but it's supposed to be very slow and buggy (as of what I read about it, I haven't tried running it). The interface in LNST then could look like this: <netconfig> <netdevice id="1" type="eth" phys_id="1"> <addresses> <address value="192.168.122.225/24" /> </addresses> <capture enabled="true"> <options> <option name="filter" value="tcp" /> <option name="format" value="binary" /> <option name="keep_log" value="true" /> <option name="stats" value="true" /> </options> </capture> </netdevice> </netconfig> This config would enable capturing with custom filter to get only tcp traffic, it would keep the log when the recipe is done and transfer it back to controller (the log option) and controller then would read all the files with capinfo and printed some stats. At first, I thought that the stats should be gathered periodically as the packets come in, but it's rather ineffective to bring each packet through an additional process, that would have to run and count them, when we can do this later in postprocessing where performance is not an issue anymore. What do you think? Is this reasonable? Radek

1 0

[lnst] Parsing error fix for short test times
by Jiří Pírko 11 Jun '12

11 Jun '12

commit 812ef52dc6f50ba1000a3b2b48de095e6820c657 Author: Jan Tluka <jtluka(a)redhat.com> Date: Mon Jun 11 15:23:50 2012 +0200 Parsing error fix for short test times Minor fix for the iperf output parsing. Case 1: [ 5] 0.0-10.0 sec 5.21 GBytes 4.47 Gbits/sec Case 2: [ 5] 0.0- 5.0 sec 3.52 GBytes 6.04 Gbits/sec In case #2 the space after dash was not expected and resulted in parsing error. The fix corrects this and handles both cases. Tests/TestIperf.py | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) --- diff --git a/Tests/TestIperf.py b/Tests/TestIperf.py index 9b67d40..f6ba232 100644 --- a/Tests/TestIperf.py +++ b/Tests/TestIperf.py @@ -91,7 +91,7 @@ class TestIperf(TestGeneric): output = client.read_nonblocking() if self.threshold is not None: # check if expected threshold is reached - m = re.search("\[[^0-9]*[0-9]*\]\s*0.0-\d*\.\d sec\s*\d*(\.\d*){0,1}\s*[ kGMT]Bytes\s*(\d*(\.\d*){0,1}\s*[ kGMT]bits\/sec)", output) + m = re.search("\[[^0-9]*[0-9]*\]\s*0.0-\s*\d*\.\d sec\s*\d*(\.\d*){0,1}\s*[ kGMT]Bytes\s*(\d*(\.\d*){0,1}\s*[ kGMT]bits\/sec)", output) if m is None: logging.info("Could not get performance throughput!") return False

1 0

[lnst] Fix the wrong parsing of throughput value
by Jiří Pírko 11 Jun '12

11 Jun '12

commit 3ebbc12d5ad1a35cdd03c8f9645ebeb3acf61793 Author: Jan Tluka <jtluka(a)redhat.com> Date: Mon Jun 11 14:25:48 2012 +0200 Fix the wrong parsing of throughput value Tests/TestIperf.py | 10 ++-------- 1 files changed, 2 insertions(+), 8 deletions(-) --- diff --git a/Tests/TestIperf.py b/Tests/TestIperf.py index cfea973..9b67d40 100644 --- a/Tests/TestIperf.py +++ b/Tests/TestIperf.py @@ -49,15 +49,9 @@ class TestIperf(TestGeneric): r2 = re.match(pattern, rate) rate_units = r2.group(3) - if r1.group(2) != None: - thr_val = float(r1.group(1) + r1.group(2)) - else: - thr_val = float(r1.group(1)) + thr_val = float(r1.group(1)) - if r2.group(2) != None: - rate_val = float(r2.group(1) + r2.group(2)) - else: - rate_val = float(r2.group(1)) + rate_val = float(r2.group(1)) # do the conversion of rate units if thr_units != rate_units:

1 0

[lnst] NetConfig: Adding libteam/teamdev support
by Jiří Pírko 11 Jun '12

11 Jun '12

commit 1a487a1bbda9a5a5a0bcfd8cc9d95decb0847ed2 Author: Radek Pazdera <rpazdera(a)redhat.com> Date: Fri Jun 8 12:19:28 2012 +0200 NetConfig: Adding libteam/teamdev support This commit introduces team device support to LNST. Example net configuration to demonstrate usage of team with LNST was added as well. Signed-off-by: Radek Pazdera <rpazdera(a)redhat.com> NetConfig/NetConfigDevNames.py | 6 ++++ NetConfig/NetConfigDevice.py | 36 +++++++++++++++++++++++++++- NetConfig/NetConfigParse.py | 2 +- example_recipes/net_configs/team1.xml | 42 +++++++++++++++++++++++++++++++++ 4 files changed, 84 insertions(+), 2 deletions(-) --- diff --git a/NetConfig/NetConfigDevNames.py b/NetConfig/NetConfigDevNames.py index a8636f6..8522e00 100644 --- a/NetConfig/NetConfigDevNames.py +++ b/NetConfig/NetConfigDevNames.py @@ -76,6 +76,9 @@ class NetConfigDevNames: def _assign_name_macvlan(self, netdev, config): self._assign_name_generic("t_macvlan", netdev, config) + def _assign_name_team(self, netdev, config): + self._assign_name_generic("t_team", netdev, config) + def _assign_name_vlan(self, netdev, config): real_netdev = config[netdev["slaves"][0]] vlan_tci = get_option(netdev, "vlan_tci") @@ -95,6 +98,9 @@ class NetConfigDevNames: self._assign_name_bridge(netdev, config) elif dev_type == "macvlan": self._assign_name_macvlan(netdev, config) + elif dev_type == "team": + self._assign_name_team(netdev, config) + ''' In second round assign names for vlans as they use previously assigned names diff --git a/NetConfig/NetConfigDevice.py b/NetConfig/NetConfigDevice.py index e024db7..be8e673 100644 --- a/NetConfig/NetConfigDevice.py +++ b/NetConfig/NetConfigDevice.py @@ -212,12 +212,46 @@ class NetConfigDeviceVlan(NetConfigDeviceGeneric): else: exec_cmd("vconfig rem %s" % dev_name) +class NetConfigDeviceTeam(NetConfigDeviceGeneric): + _pidfile = None + + def _slaves_up(self): + for slaveid in get_slaves(self._netdev): + slavenetdev = self._config[slaveid] + NetConfigDevice(slavenetdev, self._config).up() + + def _slaves_down(self): + for slaveid in get_slaves(self._netdev): + slavenetdev = self._config[slaveid] + NetConfigDevice(slavenetdev, self._config).down() + + def configure(self): + self._slaves_down() + + teamd_config = get_option(self._netdev, "teamd_config") + teamd_config = teamd_config.replace('"', '\\"') + + dev_name = self._netdev["name"] + pidfile = "/var/run/teamd_%s.pid" % dev_name + + exec_cmd("teamd -r -d -c \"%s\" -t %s -p %s" % (teamd_config, dev_name, pidfile)) + + self._pidfile = pidfile + + def deconfigure(self): + dev_name = self._netdev["name"] + pidfile = "/var/run/teamd_%s.pid" % dev_name + + exec_cmd("teamd -k -p %s" % pidfile) + self._slaves_up() + type_class_mapping = { "eth": NetConfigDeviceEth, "bond": NetConfigDeviceBond, "bridge": NetConfigDeviceBridge, "macvlan": NetConfigDeviceMacvlan, - "vlan": NetConfigDeviceVlan + "vlan": NetConfigDeviceVlan, + "team": NetConfigDeviceTeam } def NetConfigDevice(netdev, config): diff --git a/NetConfig/NetConfigParse.py b/NetConfig/NetConfigParse.py index ff148e5..9097e88 100644 --- a/NetConfig/NetConfigParse.py +++ b/NetConfig/NetConfigParse.py @@ -192,7 +192,7 @@ class NetConfigParse: if dev_type == "eth": pass - elif dev_type in ["bond", "bridge", "vlan", "macvlan"]: + elif dev_type in ["bond", "bridge", "vlan", "macvlan", "team"]: self._parse_options(netdev, dom_netdev, config) self._parse_slaves(netdev, dom_netdev, config) else: diff --git a/example_recipes/net_configs/team1.xml b/example_recipes/net_configs/team1.xml new file mode 100644 index 0000000..4bebf13 --- /dev/null +++ b/example_recipes/net_configs/team1.xml @@ -0,0 +1,42 @@ +<netconfig> + <netdevice id="1" type="eth" phys_id="1" /> + <netdevice id="2" type="eth" phys_id="2" /> + <netdevice id="5" type="team"> + <options> +  + <option name="teamd_config"> + { + "runner": {"name": "roundrobin"}, + "ports": {"eth1": {}, "eth2": {}} + } + </option> + </options> + <slaves> + <slave id="1" /> + <slave id="2" /> + </slaves> + <addresses> + <address value="192.168.122.200/24" /> + </addresses> + </netdevice> + <netdevice id="3" type="eth" phys_id="3" /> + <netdevice id="4" type="eth" phys_id="4" /> + <netdevice id="6" type="team"> + <options> + <option name="teamd_config"> + { + "runner": {"name": "roundrobin"}, + "ports": {"eth3": {}, "eth4": {}} + } + </option> + </options> + <slaves> + <slave id="3" /> + <slave id="4" /> + </slaves> + <addresses> + <address value="192.168.122.210/24" /> + </addresses> + </netdevice> +</netconfig>

1 0

[lnst] NetTestParse: Remove XML comments before parsing
by Jiří Pírko 11 Jun '12

11 Jun '12

commit 9752c3399e6e1d947c9542f8801c0db24ca97f85 Author: Radek Pazdera <rpazdera(a)redhat.com> Date: Fri Jun 8 12:19:27 2012 +0200 NetTestParse: Remove XML comments before parsing Comments in the XML recipe might interfere with the parser in some cases and result in exceptions. The best way to deal with this is to remove them from the recipe. Signed-off-by: Radek Pazdera <rpazdera(a)redhat.com> Common/XmlPreprocessor.py | 15 +++++++++++++++ NetTest/NetTestParse.py | 2 ++ 2 files changed, 17 insertions(+), 0 deletions(-) --- diff --git a/Common/XmlPreprocessor.py b/Common/XmlPreprocessor.py index 88ae758..b20fec8 100644 --- a/Common/XmlPreprocessor.py +++ b/Common/XmlPreprocessor.py @@ -51,6 +51,21 @@ class XmlPreprocessor: else: raise XmlTemplateError("Alias name '%s' is reserved" % name) + def remove_comments(self, node): + """ + Remove all comment nodes from the tree. + """ + + comments = [] + for child in node.childNodes: + if child.nodeType == node.COMMENT_NODE: + comments.append(child) + else: + self.remove_comments(child) + + for comment in comments: + node.removeChild(comment) + def expand(self, node): """ Traverse DOM tree from `node' down and expand any diff --git a/NetTest/NetTestParse.py b/NetTest/NetTestParse.py index 5b5101a..68c0058 100644 --- a/NetTest/NetTestParse.py +++ b/NetTest/NetTestParse.py @@ -81,6 +81,8 @@ class NetTestParse: dom = parseString(self._recipe_xml_string) xml_prep = self._xml_prep + xml_prep.remove_comments(dom) + self._load_included_parts(dom) dom_nettestrecipe = dom.getElementsByTagName("nettestrecipe")[0]

1 0

[lnst] NetConfigParse: Extending options parsing
by Jiří Pírko 11 Jun '12

11 Jun '12

commit cefbcea3515f7010c955aa5afe783e96d108ea4c Author: Radek Pazdera <rpazdera(a)redhat.com> Date: Fri Jun 8 12:19:25 2012 +0200 NetConfigParse: Extending options parsing This commit makes NetConfigParser accept <netdevice> options in a slightly different format. Previous format of options accepted values only as an 'value' attribute to the <option> tag. With this modification it is possible to define options also with values as a text elements of the option tag. There are now two possibilities:  <option name="opt" value="5" />  <option name="teamd_config"> { "device": "team1", "runner": {"name": "roundrobin"}, "ports": {"eth3": {}, "eth4": {}} } </option> Signed-off-by: Radek Pazdera <rpazdera(a)redhat.com> NetConfig/NetConfigParse.py | 12 +++++++++++- 1 files changed, 11 insertions(+), 1 deletions(-) --- diff --git a/NetConfig/NetConfigParse.py b/NetConfig/NetConfigParse.py index dbd7a6d..ff148e5 100644 --- a/NetConfig/NetConfigParse.py +++ b/NetConfig/NetConfigParse.py @@ -97,7 +97,17 @@ class NetConfigParse: def _parse_options_handler(self, lst, dom_element, config): name = str(dom_element.getAttribute("name")) - value = str(dom_element.getAttribute("value")) + + value = "" + if dom_element.hasAttribute("value"): + value = str(dom_element.getAttribute("value")) + elif dom_element.hasChildNodes(): + node = dom_element.firstChild + try: + value = str(dom_element.firstChild.data) + except: + raise Exception("Invalid option value") + lst.append((name, value)) def _parse_options(self, netdev, dom_netdev, config):

1 0

[lnst] Exclude tarball and git directory from generated tarball
by Jiří Pírko 11 Jun '12

11 Jun '12

commit 3ceb5e8d837997fbbb4e77d5fb6d74437f9522a8 Author: Jan Tluka <jtluka(a)redhat.com> Date: Wed Jun 6 17:06:17 2012 +0200 Exclude tarball and git directory from generated tarball Make tarball smaller excluding git metadata directory and possibly previously generated tarball. Common/SlaveUtils.py | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) --- diff --git a/Common/SlaveUtils.py b/Common/SlaveUtils.py index 3640f94..2c1f043 100644 --- a/Common/SlaveUtils.py +++ b/Common/SlaveUtils.py @@ -27,7 +27,7 @@ def prepare_client_session(host, port, login, passwd=None, command=None, if test_dir is None: test_dir = "lnst" - s = ShellProcess("tar -cjf lnst.tar.bz2 --exclude *.pyc --exclude 'Logs/*' -C '%s' ./" % sys.path[0]) + s = ShellProcess("tar -cjf lnst.tar.bz2 --exclude *.pyc --exclude 'Logs/*' --exclude '.git/*' --exclude 'lnst.tar.bz2' -C '%s' ./" % sys.path[0]) s.wait() scp_to_remote(host, port, login, passwd, "lnst.tar.bz2","/%s/" % (install_path))

1 0

[lnst] Make remote_exec working if executed outside of lnst root directory
by Jiří Pírko 11 Jun '12

11 Jun '12

commit 68063fcbe8c3f120eaa822e059187bcbb60fe88e Author: Jan Tluka <jtluka(a)redhat.com> Date: Wed Jun 6 17:04:34 2012 +0200 Make remote_exec working if executed outside of lnst root directory If controller is executed outside of LNST source directory the generated tarball contains content of current working directory instead of the LNST code. The fix is to change the current directory using -C tar option and use of ./ instead of * wildcard. Common/SlaveUtils.py | 7 ++++--- 1 files changed, 4 insertions(+), 3 deletions(-) --- diff --git a/Common/SlaveUtils.py b/Common/SlaveUtils.py index e7fdf3d..3640f94 100644 --- a/Common/SlaveUtils.py +++ b/Common/SlaveUtils.py @@ -8,7 +8,7 @@ jzupka(a)redhat.com (Jiri Zupka) from Common.SshUtils import scp_to_remote, wait_for_login from Common.ShellProcess import ShellProcess - +import sys def prepare_client_session(host, port, login, passwd=None, command=None, prompt=None, install_path=None, test_dir=None): @@ -26,7 +26,8 @@ def prepare_client_session(host, port, login, passwd=None, command=None, install_path = "/tmp" if test_dir is None: test_dir = "lnst" - s = ShellProcess("tar -cjf lnst.tar.bz2 --exclude *.pyc --exclude 'Logs/*' *") + + s = ShellProcess("tar -cjf lnst.tar.bz2 --exclude *.pyc --exclude 'Logs/*' -C '%s' ./" % sys.path[0]) s.wait() scp_to_remote(host, port, login, passwd, "lnst.tar.bz2","/%s/" % (install_path)) @@ -39,4 +40,4 @@ def prepare_client_session(host, port, login, passwd=None, command=None, prompt = "Started" command = "cd /%s/%s/ && ./%s" % (install_path, test_dir, command) return wait_for_login(host, port, login, passwd, prompt, - command=command, timeout=10) \ No newline at end of file + command=command, timeout=10)

1 0

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

LNST-developers June 2012