LNST-developers October 2012

lnst-developers@lists.fedorahosted.org

6 participants
82 discussions

[PATCH 1/4] logging: add new file LoggingHandler
by Ondrej Lichtner 05 Oct '12

05 Oct '12

From: Ondrej Lichtner <olichtne(a)redhat.com> This file defines a new class ServerHandler. This class is a logging handler and therefore is a decendant of the class logging.Handler. This new class contains a TCP server socket listening on a specified port. Until an incoming connection is recieved emitted messages are stored in a buffer. After recieving a connection the stored messages are all sent to the client and the buffer is flushed. To avoid using threads or a new process for this server, the server socket is created as nonblocking and we check for incoming connections only when a new log message is emitted. This handler was created because there is no suitable alternative for this functionality in the python standard library. However this handler is inspired by the SocketHandler and uses some of it's code. Signed-off-by: Ondrej Lichtner <olichtne(a)redhat.com> --- Common/LoggingHandler.py | 108 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 108 insertions(+) create mode 100644 Common/LoggingHandler.py diff --git a/Common/LoggingHandler.py b/Common/LoggingHandler.py new file mode 100644 index 0000000..9b08f9d --- /dev/null +++ b/Common/LoggingHandler.py @@ -0,0 +1,108 @@ +""" +Server-like logging handler. +Stores logged messages in a buffer. Every time a new message is emitted it +checks for incoming connections. If a connection is established it flushes the +messages stored in the buffer to the connecting client. + +Copyright 2012 Red Hat, Inc. +Licensed under the GNU General Public License, version 2 as +published by the Free Software Foundation; see COPYING for details. +""" + +__autor__ = """ +olichtne(a)redhat.com (Ondrej Lichtner) +""" + +import socket, struct, pickle +import logging + +DEFAULT_LOG_PORT = 9998 + +class ServerHandler(logging.Handler): + def __init__(self, port=DEFAULT_LOG_PORT): + logging.Handler.__init__(self) + self.port = port + + self.sock = None + self.buf = [] + + self.server_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM) + self.server_socket.setblocking(0) + self.server_socket.bind(('0.0.0.0', self.port)) + self.server_socket.listen(1) + + def makePickle(self, record): + """ + Pickles the record in binary format with a length prefix, and + returns it ready for transmission across the socket. + + Function taken from class SocketHandler from standard python + library logging.handlers + """ + d = dict(record.__dict__) + d['msg'] = record.getMessage() + d['args'] = None + d['exc_info'] = None + s = pickle.dumps(d, 1) + slen = struct.pack(">L", len(s)) + return slen + s + + def emit(self, record): + try: + s = self.makePickle(record) + self.buf.append(s) + self.send_all() + except (KeyboardInterrupt, SystemExit): + raise + except: + logging.Handler.handleError(self, record) + + def client_connection(self): + if self.sock == None: + try: + self.sock = self.server_socket.accept()[0] + return True + except: + self.sock = None + return False + else: + return True + + def send_all(self): + if self.client_connection(): + sent = len(self.buf) + for record in self.buf: + if not self.send(record): + sent = self.buf.index(record) + break + + self.buf = self.buf[sent:] + + def send(self, record): + try: + if hasattr(self.sock, "sendall"): + self.sock.sendall(record) + else: + sentsofar = 0 + left = len(record) + while left > 0: + sent = self.sock.send(record[sentsofar:]) + sentsofar = sentsofar + sent + left = left - sent + return True + except socket.error: + self.sock.close() + self.sock = None + return False + + def close(self): + if self.sock: + self.sock.close() + self.sock = None + + self.server_socket.close() + self.server_socket = None + + self.buf = [] + + logging.Handler.close(self) -- 1.7.11.4

2 6

[lnst trac] #36: add documentation for each possible Test (options, results, etc)
by fedora-badges 04 Oct '12

04 Oct '12

#36: add documentation for each possible Test (options, results, etc) -------------------------+----------------------- Reporter: jirka | Owner: somebody Type: enhancement | Status: new Priority: major | Milestone: Component: component1 | Version: Keywords: | Blocked By: Blocking: | -------------------------+----------------------- -- Ticket URL: <https://fedorahosted.org/lnst/ticket/36> lnst <http://example.org/> My example project

1 0

[lnst] recipes: team: add recipe for testing queue_mapping
by Jiří Pírko 04 Oct '12

04 Oct '12

commit 6df8464b15d9bcec126d70982d96a0f49f5b9758 Author: Jiri Pirko <jiri(a)resnulli.us> Date: Thu Oct 4 17:44:49 2012 +0200 recipes: team: add recipe for testing queue_mapping Signed-off-by: Jiri Pirko <jiri(a)resnulli.us> recipes/team/netconfig-br_queue_mapping.xml | 18 ++++++++ recipes/team/netconfig-team_rr_queue_mapping.xml | 50 ++++++++++++++++++++++ recipes/team/recipey_rr_queue_mapping.xml | 21 +++++++++ recipes/team/sequence_queue_mapping.xml | 43 +++++++++++++++++++ 4 files changed, 132 insertions(+), 0 deletions(-) --- diff --git a/recipes/team/netconfig-br_queue_mapping.xml b/recipes/team/netconfig-br_queue_mapping.xml new file mode 100644 index 0000000..0331142 --- /dev/null +++ b/recipes/team/netconfig-br_queue_mapping.xml @@ -0,0 +1,18 @@ +<netconfig> + <interface id="1" phys_id="1" type="eth"/> + <interface id="2" phys_id="2" type="eth"/> + <interface id="3" phys_id="3" type="eth"/> + <interface id="testiface" type="bridge"> + <slaves> + <slave id="1"/> + <slave id="2"/> + <slave id="3"/> + </slaves> + <addresses> + <address value="172.0.1.2/24"/> + <address value="172.0.2.2/24"/> + <address value="172.0.3.2/24"/> + <address value="172.0.4.2/24"/> + </addresses> + </interface> +</netconfig> diff --git a/recipes/team/netconfig-team_rr_queue_mapping.xml b/recipes/team/netconfig-team_rr_queue_mapping.xml new file mode 100644 index 0000000..b07f521 --- /dev/null +++ b/recipes/team/netconfig-team_rr_queue_mapping.xml @@ -0,0 +1,50 @@ +<netconfig> + <interface id="1" phys_id="1" type="eth"> + <options> + <option name="teamd_port_config"> + { + "queue_id": 1 + } + </option> + </options> + </interface> + <interface id="2" phys_id="2" type="eth"> + <options> + <option name="teamd_port_config"> + { + "queue_id": 2 + } + </option> + </options> + </interface> + <interface id="3" phys_id="3" type="eth"> + <options> + <option name="teamd_port_config"> + { + "queue_id": 3 + } + </option> + </options> + </interface> + <interface id="testiface" type="team"> + <options> + <option name="teamd_config"> + { + "hwaddr": "00:11:22:33:44:55", + "runner": {"name": "roundrobin"} + } + </option> + </options> + <slaves> + <slave id="1"/> + <slave id="2"/> + <slave id="3"/> + </slaves> + <addresses> + <address value="172.0.1.1/24"/> + <address value="172.0.2.1/24"/> + <address value="172.0.3.1/24"/> + <address value="172.0.4.1/24"/> + </addresses> + </interface> +</netconfig> diff --git a/recipes/team/recipey_rr_queue_mapping.xml b/recipes/team/recipey_rr_queue_mapping.xml new file mode 100644 index 0000000..4250837 --- /dev/null +++ b/recipes/team/recipey_rr_queue_mapping.xml @@ -0,0 +1,21 @@ +<nettestrecipe> + <machines> + <define> + <alias name="testip" value="192.168.111.1/24"/> + <alias name="testip6" value="fe01::1/64"/> + </define> + <machine id="1"> + <machineconfig source="machineconfig-peanut.xml"/> + <netconfig source="netconfig-team_rr_queue_mapping.xml"/> + </machine> + <define> + <alias name="testip" value="192.168.111.2/24"/> + <alias name="testip6" value="fe01::2/64"/> + </define> + <machine id="2"> + <machineconfig source="machineconfig-dhcp-37-128.xml"/> + <netconfig source="netconfig-br_queue_mapping.xml"/> + </machine> + </machines> + <command_sequence source="sequence_queue_mapping.xml"/> +</nettestrecipe> diff --git a/recipes/team/sequence_queue_mapping.xml b/recipes/team/sequence_queue_mapping.xml new file mode 100644 index 0000000..d3999f7 --- /dev/null +++ b/recipes/team/sequence_queue_mapping.xml @@ -0,0 +1,43 @@ +<command_sequence> + <command type="exec" value="sleep 4"/> + <command machine_id="1" type="exec" value="tc qdisc add dev {devname(1,testiface)} handle 1 root multiq"/> + <command machine_id="1" type="exec" value="tc filter add dev {devname(1,testiface)} protocol ip parent 1: prio 1 u32 match ip dst {ip(2,testiface,0)} action skbedit queue_mapping 1"/> + <command machine_id="1" type="exec" value="tc filter add dev {devname(1,testiface)} protocol ip parent 1: prio 1 u32 match ip dst {ip(2,testiface,1)} action skbedit queue_mapping 2"/> + <command machine_id="1" type="exec" value="tc filter add dev {devname(1,testiface)} protocol ip parent 1: prio 1 u32 match ip dst {ip(2,testiface,2)} action skbedit queue_mapping 3"/> + <command machine_id="1" timeout="60" type="test" value="IcmpPing" bg_id="1"> + <options> + <option name="addr" value="{ip(2,testiface,0)}"/> + <option name="count" value="100"/> + <option name="interval" value="0.01"/> + <option name="limit_rate" value="95"/> + </options> + </command> + <command machine_id="1" timeout="60" type="test" value="IcmpPing" bg_id="2"> + <options> + <option name="addr" value="{ip(2,testiface,1)}"/> + <option name="count" value="100"/> + <option name="interval" value="0.01"/> + <option name="limit_rate" value="95"/> + </options> + </command> + <command machine_id="1" timeout="60" type="test" value="IcmpPing" bg_id="3"> + <options> + <option name="addr" value="{ip(2,testiface,2)}"/> + <option name="count" value="100"/> + <option name="interval" value="0.01"/> + <option name="limit_rate" value="95"/> + </options> + </command> + <command machine_id="1" timeout="60" type="test" value="IcmpPing" bg_id="4"> + <options> + <option name="addr" value="{ip(2,testiface,3)}"/> + <option name="count" value="100"/> + <option name="interval" value="0.01"/> + <option name="limit_rate" value="95"/> + </options> + </command> + <command machine_id="1" type="wait" value="1"/> + <command machine_id="1" type="wait" value="2"/> + <command machine_id="1" type="wait" value="3"/> + <command machine_id="1" type="wait" value="4"/> +</command_sequence>

1 0

[lnst] recipes: add active loadbalance recipes
by Jiří Pírko 04 Oct '12

04 Oct '12

commit 8f0572f95b8dd2b13c0560c6f1c9b25edd0a6a03 Author: Jiri Pirko <jiri(a)resnulli.us> Date: Thu Oct 4 10:01:31 2012 +0200 recipes: add active loadbalance recipes Signed-off-by: Jiri Pirko <jiri(a)resnulli.us> recipes/team/netconfig-team_lb_act.xml | 30 ++++++++++++++++++++++++++ recipes/team/recipex_lb_act_001.xml | 21 ++++++++++++++++++ recipes/team/recipex_lb_act_002.xml | 36 ++++++++++++++++++++++++++++++++ recipes/team/recipex_lb_act_003.xml | 21 ++++++++++++++++++ recipes/team/recipex_lb_act_004.xml | 21 ++++++++++++++++++ 5 files changed, 129 insertions(+), 0 deletions(-) --- diff --git a/recipes/team/netconfig-team_lb_act.xml b/recipes/team/netconfig-team_lb_act.xml new file mode 100644 index 0000000..fca08ce --- /dev/null +++ b/recipes/team/netconfig-team_lb_act.xml @@ -0,0 +1,30 @@ +<netconfig> + <interface id="1" phys_id="1" type="eth"/> + <interface id="2" phys_id="2" type="eth"/> + <interface id="3" phys_id="3" type="eth"/> + <interface id="testiface" type="team"> + <options> + <option name="teamd_config"> + { + "hwaddr": "00:11:22:33:44:55", + "runner": { + "name": "loadbalance", + "tx_hash": ["eth", "ipv4", "ipv6"], + "tx_balancer": { + "name": "basic" + } + } + } + </option> + </options> + <slaves> + <slave id="1"/> + <slave id="2"/> + <slave id="3"/> + </slaves> + <addresses> + <address value="{$testip}"/> + <address value="{$testip6}"/> + </addresses> + </interface> +</netconfig> diff --git a/recipes/team/recipex_lb_act_001.xml b/recipes/team/recipex_lb_act_001.xml new file mode 100644 index 0000000..24ec354 --- /dev/null +++ b/recipes/team/recipex_lb_act_001.xml @@ -0,0 +1,21 @@ +<nettestrecipe> + <machines> + <define> + <alias name="testip" value="192.168.111.1/24"/> + <alias name="testip6" value="fe01::1/64"/> + </define> + <machine id="1"> + <machineconfig source="machineconfig-peanut.xml"/> + <netconfig source="netconfig-team_lb_act.xml"/> + </machine> + <define> + <alias name="testip" value="192.168.111.2/24"/> + <alias name="testip6" value="fe01::2/64"/> + </define> + <machine id="2"> + <machineconfig source="machineconfig-dhcp-37-128.xml"/> + <netconfig source="netconfig-simple_br.xml"/> + </machine> + </machines> + <command_sequence source="sequence_ping_simple.xml"/> +</nettestrecipe> diff --git a/recipes/team/recipex_lb_act_002.xml b/recipes/team/recipex_lb_act_002.xml new file mode 100644 index 0000000..bdc5e7e --- /dev/null +++ b/recipes/team/recipex_lb_act_002.xml @@ -0,0 +1,36 @@ +<nettestrecipe> + <machines> + <define> + <alias name="testip" value="192.168.111.1/24"/> + <alias name="testip6" value="fe01::1/64"/> + </define> + <machine id="1"> + <machineconfig source="machineconfig-peanut.xml"/> + <netconfig source="netconfig-team_lb_act.xml"/> + </machine> + <define> + <alias name="testip" value="192.168.111.2/24"/> + <alias name="testip6" value="fe01::2/64"/> + </define> + <machine id="2"> + <machineconfig source="machineconfig-dhcp-37-128.xml"/> + <netconfig source="netconfig-simple_br.xml"/> + </machine> + </machines> + <define> + <alias name="multicast_group" value="239.1.2.3"/> + <alias name="port" value="1337"/> + <alias name="test_duration" value="10"/> + <alias name="send_delay" value="0.1"/> + <alias name="nonexistent_ip" value="127.0.0.200"/> + </define> + + <command_sequence source="../multicast/cmd_sequences/max_groups.xml"/> + <command_sequence source="../multicast/cmd_sequences/block_source.xml"/> + <command_sequence source="../multicast/cmd_sequences/source_membership.xml"/> + <command_sequence source="../multicast/cmd_sequences/membership.xml"/> + <command_sequence source="../multicast/cmd_sequences/if.xml"/> + <command_sequence source="../multicast/cmd_sequences/ttl.xml"/> + <command_sequence source="../multicast/cmd_sequences/loop.xml"/> + <command_sequence source="../multicast/cmd_sequences/simple.xml"/> +</nettestrecipe> diff --git a/recipes/team/recipex_lb_act_003.xml b/recipes/team/recipex_lb_act_003.xml new file mode 100644 index 0000000..1383240 --- /dev/null +++ b/recipes/team/recipex_lb_act_003.xml @@ -0,0 +1,21 @@ +<nettestrecipe> + <machines> + <define> + <alias name="testip" value="192.168.111.1/24"/> + <alias name="testip6" value="fe01::1/64"/> + </define> + <machine id="1"> + <machineconfig source="machineconfig-peanut.xml"/> + <netconfig source="netconfig-team_lb_act.xml"/> + </machine> + <define> + <alias name="testip" value="192.168.111.2/24"/> + <alias name="testip6" value="fe01::2/64"/> + </define> + <machine id="2"> + <machineconfig source="machineconfig-dhcp-37-128.xml"/> + <netconfig source="netconfig-simple.xml"/> + </machine> + </machines> + <command_sequence source="sequence_pktgen.xml"/> +</nettestrecipe> diff --git a/recipes/team/recipex_lb_act_004.xml b/recipes/team/recipex_lb_act_004.xml new file mode 100644 index 0000000..b436dc4 --- /dev/null +++ b/recipes/team/recipex_lb_act_004.xml @@ -0,0 +1,21 @@ +<nettestrecipe> + <machines> + <define> + <alias name="testip" value="192.168.111.1/24"/> + <alias name="testip6" value="fe01::1/64"/> + </define> + <machine id="1"> + <machineconfig source="machineconfig-peanut.xml"/> + <netconfig source="netconfig-team_lb_act.xml"/> + </machine> + <define> + <alias name="testip" value="192.168.111.2/24"/> + <alias name="testip6" value="fe01::2/64"/> + </define> + <machine id="2"> + <machineconfig source="machineconfig-dhcp-37-128.xml"/> + <netconfig source="netconfig-simple_br.xml"/> + </machine> + </machines> + <command_sequence source="sequence_iperf.xml"/> +</nettestrecipe>

1 0

[lnst] Fix wrong path resolution
by Jiří Pírko 04 Oct '12

04 Oct '12

commit 27e953afc31a3745e83c923fe2727b626ebe1792 Author: Jan Tluka <jtluka(a)redhat.com> Date: Tue Oct 2 16:56:44 2012 +0200 Fix wrong path resolution In case when the recipe path was specified with prefix './' or none at all, the root was incorrectly resolved to an empty string causing further errors in path resolution. To fix this the supplied recipe file path is converted to absolute path. Signed-off-by: Jan Tluka <jtluka(a)redhat.com> Common/RecipePath.py | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) --- diff --git a/Common/RecipePath.py b/Common/RecipePath.py index b5f650c..60d035d 100644 --- a/Common/RecipePath.py +++ b/Common/RecipePath.py @@ -20,7 +20,7 @@ def get_recipepath_class(root, path): return HttpRecipePath(root, path) else: if os.access(path, os.R_OK): - return FileRecipePath(root, path) + return FileRecipePath(None, os.path.realpath(path)) else: raise Exception("Recipe path does not exist \"%s\"!" % path)

1 0

[lnst] controller: changes required by the new LoggingServer
by Jiří Pírko 04 Oct '12

04 Oct '12

commit 91d8843f589b0ebef34cb2d69cf3c44f5cac8bf5 Author: Ondrej Lichtner <olichtne(a)redhat.com> Date: Tue Oct 2 13:42:13 2012 +0200 controller: changes required by the new LoggingServer NetTestController now needs to have access to the logging server so nettestctl now passes it as an argument to the constructor. The function _init_slave_logging now doesn't communicate with the slave at all. Instead it sends the slaves address to the logging server which then tries to connect to it. Signed-off-by: Ondrej Lichtner <olichtne(a)redhat.com> NetTest/NetTestController.py | 18 +++++++++--------- nettestctl.py | 13 ++++++------- 2 files changed, 15 insertions(+), 16 deletions(-) --- diff --git a/NetTest/NetTestController.py b/NetTest/NetTestController.py index e9735b4..2629c40 100644 --- a/NetTest/NetTestController.py +++ b/NetTest/NetTestController.py @@ -26,6 +26,7 @@ from Common.LoggingServer import LoggingServer from Common.VirtUtils import VirtNetCtl, VirtDomainCtl, BridgeCtl from Common.Utils import wait_for from NetTest.MachinePool import MachinePool +from Common.LoggingHandler import DEFAULT_LOG_PORT class NetTestError(Exception): pass @@ -35,12 +36,13 @@ def ignore_event(**kwarg): class NetTestController: def __init__(self, recipe_path, remoteexec=False, cleanup=False, - res_serializer=None, config=None): + res_serializer=None, config=None, logServer=None): self._remoteexec = remoteexec self._docleanup = cleanup self._res_serializer = res_serializer self._remote_capture_files = {} self._config = config + self._logServer = logServer self._command_context = NetTestCommandContext() self._machine_pool = MachinePool(config.get_option('environment', 'pool_dirs')) @@ -257,15 +259,13 @@ class NetTestController: def _init_slave_logging(self, machine_id): info = self._get_machineinfo(machine_id) + logServer = self._logServer + hostname = info["hostname"] - logging.info("Setting logging server on machine %s", hostname) - rpc = self._get_machinerpc(machine_id) - ip_addr = get_corespond_local_ip(hostname) - if not rpc.set_logging(ip_addr, self._config.get_option('log', 'port')): - logging.error("==================================================") - logging.error("Machine %s is unable to connect to the logging "\ - "server! Check your firewall settings." % hostname) - logging.error("==================================================") + port = str(DEFAULT_LOG_PORT) + + logging.info("Connecting to the logging server on machine %s", hostname) + logServer.addSlave(hostname, port) def _deconfigure_slaves(self): if 'machines' not in self._recipe: diff --git a/nettestctl.py b/nettestctl.py index f9973e4..c4154d4 100755 --- a/nettestctl.py +++ b/nettestctl.py @@ -45,12 +45,12 @@ def usage(): print " -x, --result=FILE file to write xml_result" sys.exit() -def process_recipe(action, file_path, remoteexec, cleanup, - res_serializer, packet_capture, config): +def process_recipe(action, file_path, remoteexec, cleanup, res_serializer, + packet_capture, config, loggingServer): nettestctl = NetTestController(file_path, remoteexec=remoteexec, cleanup=cleanup, res_serializer=res_serializer, - config=config) + config=config, logServer=loggingServer) if action == "run": return nettestctl.run_recipe(packet_capture) elif action == "dump": @@ -75,11 +75,10 @@ def get_recipe_result(args, file_path, remoteexec, cleanup, res_serializer, packet_capture, config): res_serializer.add_recipe(file_path) Logs.set_logging_root_path(file_path) - loggingServer = LoggingServer(config.get_option('log', 'port'), - Logs.root_path, Logs.debug) + loggingServer = LoggingServer(Logs.root_path, Logs.debug) loggingServer.start() - res = process_recipe(args, file_path, remoteexec, cleanup, - res_serializer, packet_capture, config) + res = process_recipe(args, file_path, remoteexec, cleanup, res_serializer, + packet_capture, config, loggingServer) loggingServer.stop() return ((file_path, res))

1 0

[lnst] Logs: slaves use Server handler instead of SocketHandler
by Jiří Pírko 04 Oct '12

04 Oct '12

commit 4c0fec81335d53547f9a0b1e20322041a5f5fd36 Author: Ondrej Lichtner <olichtne(a)redhat.com> Date: Tue Oct 2 13:42:12 2012 +0200 Logs: slaves use Server handler instead of SocketHandler This commit makes slaves use the newly implemented ServerHandler instead of a combination of a MemoryHandler and a SocketHandler. This is another one of the changes required for the reversal of the communication between controller and slave logging. I also removed the function append_network_hadler since it won't be used from now Signed-off-by: Ondrej Lichtner <olichtne(a)redhat.com> Common/Logs.py | 14 +++----------- 1 files changed, 3 insertions(+), 11 deletions(-) --- diff --git a/Common/Logs.py b/Common/Logs.py index bf20e80..85099d4 100644 --- a/Common/Logs.py +++ b/Common/Logs.py @@ -13,6 +13,7 @@ import os, sys, shutil, datetime from logging import Formatter import logging.handlers import traceback +from Common.LoggingHandler import ServerHandler LOCAL_IP = "(127.0.0.1)" @@ -214,14 +215,6 @@ class Logs: cls.root_path = cls.prepare_logging(debug, waitForNet, recipe_path, to_display) - - @staticmethod - def append_network_hadler(address, port): - """ - Append to log network handler. - """ - logging.net_handler.setTarget(logging.handlers.SocketHandler(address, port)) - @classmethod def clean_root_log_folder(cls, logRootPath): try: @@ -297,9 +290,8 @@ class Logs: root_logger.addHandler(display) if waitForNet: - memory_handler = logging.handlers.MemoryHandler(1) - root_logger.addHandler(memory_handler) - logging.net_handler = memory_handler + server_handler = ServerHandler() + root_logger.addHandler(server_handler) log_root_folder = cls.set_logging_root_path(recipe_path)

1 0

[lnst] LoggingServer: invert the direction of connections
by Jiří Pírko 04 Oct '12

04 Oct '12

commit 6076f8431f5ef71d1c2b88343862da0ccc6a815a Author: Ondrej Lichtner <olichtne(a)redhat.com> Date: Tue Oct 2 13:42:11 2012 +0200 LoggingServer: invert the direction of connections This commit changes the LoggingServer so that it doesn't listen for incoming connections and instead initiates them. This solves the problem of slave logs not arriving when the controller is using a firewall or is located behind a NAT. To connect a new slave its ip address and the port number are sent to the server process through a pipe. The server then attempts to create a new connection. This expects that there is a listening server socket on the other side, which for us means that the slave is using the ServerHandler. Signed-off-by: Ondrej Lichtner <olichtne(a)redhat.com> Common/LoggingServer.py | 59 ++++++++++++++++++++++++---------------------- 1 files changed, 31 insertions(+), 28 deletions(-) --- diff --git a/Common/LoggingServer.py b/Common/LoggingServer.py index 4bd4f19..7aab431 100644 --- a/Common/LoggingServer.py +++ b/Common/LoggingServer.py @@ -18,26 +18,24 @@ from Common.Utils import die_when_parent_die class LoggingServer: - DEFAULT_PORT = 9998 - def __init__(self, port, root_path, debug): - self.port = port + def __init__(self, root_path, debug): self.pid = None - self.socket = None self.stopped = False self.root_path = root_path self.debug = debug self.childSocket = {} + self.read_pipe = None + self.write_pipe = None def server_stop_handler(self, sig, frame): """ Call function. Used for signal handle. """ + os.close(self.read_pipe) if (sig == signal.SIGTERM): for sock in self.childSocket.itervalues(): sock[2].close() - self.socket.shutdown(socket.SHUT_RDWR) - self.socket.close() sys.exit() @@ -48,26 +46,20 @@ class LoggingServer: @param port: Port on which logging server listen. @return: Pid of logging process. """ - self.socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM) - self.socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) - try: - self.socket.bind(('0.0.0.0',self.port)) - except socket.error, e: - if (e.errno == 98): - logging.error("Another Logging server listen" - " on port %d" % self.port) - raise + self.read_pipe, self.write_pipe = os.pipe() self.pid = os.fork() if not self.pid: + os.close(self.write_pipe) die_when_parent_die() signal.signal(signal.SIGTERM, self.server_stop_handler) self._forked() else: + os.close(self.read_pipe) time.sleep(0.5) def prepare_logging(self, root_path, sock): - address = sock[1][0] + address = sock.getpeername()[0] slave_root_path = os.path.join(root_path, address) try: os.mkdir(slave_root_path) @@ -77,7 +69,7 @@ class LoggingServer: logger = logging.getLogger(address) Logs(self.debug, False, logger, log_root=slave_root_path, to_display=False, date="") - return (logger, address, sock[0]) + return (logger, address, sock) def recv_slave_log(self, logger, address, sock): @@ -111,23 +103,34 @@ class LoggingServer: @param port: Port for listening. """ - self.socket.listen(100) - wait_socket = [self.socket.fileno()] + wait_socket = [self.read_pipe] while True: (r,w,e) = select.select(wait_socket, [], []) - if (self.socket.fileno() in r): - csock = self.socket.accept() - slave = self.prepare_logging(self.root_path, csock) - self.childSocket[csock[0].fileno()] = slave - wait_socket.append(slave[2].fileno()) - r.remove(self.socket.fileno()) + if (self.read_pipe in r): + slave_ip = os.read(self.read_pipe, 4096) + host, port = slave_ip.split() + + try: + csock = socket.socket(socket.AF_INET, socket.SOCK_STREAM) + csock.connect((host, int(port))) + except socket.error: + logging.debug("Failed to connect to slave: "+ slave_ip) + else: + slave = self.prepare_logging(self.root_path, csock) + self.childSocket[csock.fileno()] = slave + wait_socket.append(slave[2].fileno()) + finally: + r.remove(self.read_pipe) + for so in r: if not self.recv_slave_log(*self.childSocket[so]): self.childSocket[so][2].close() del self.childSocket[so] wait_socket.remove(so) - self.socket.close() - sys.exit() + + + def addSlave(self, hostname, port): + os.write(self.write_pipe, hostname+' '+port) def stop(self): @@ -149,7 +152,7 @@ if __name__ == '__main__': logger = logging.getLogger() c = logging.FileHandler("out.txt") logger.addHandler(c) - l = LoggingServer(LoggingServer.DEFAULT_PORT) + l = LoggingServer() l.start() raw_input() l.stop()

1 0

[lnst] logging: add new file LoggingHandler
by Jiří Pírko 04 Oct '12

04 Oct '12

commit eb14765ab34352cabcd51387403c0863b72c126e Author: Ondrej Lichtner <olichtne(a)redhat.com> Date: Tue Oct 2 13:42:10 2012 +0200 logging: add new file LoggingHandler This file defines a new class ServerHandler. This class is a logging handler and therefore is a decendant of the class logging.Handler. This new class contains a TCP server socket listening on a specified port. Until an incoming connection is recieved emitted messages are stored in a buffer. After recieving a connection the stored messages are all sent to the client and the buffer is flushed. To avoid using threads or a new process for this server, the server socket is created as nonblocking and we check for incoming connections only when a new log message is emitted. This handler was created because there is no suitable alternative for this functionality in the python standard library. However this handler is inspired by the SocketHandler and uses some of it's code. Signed-off-by: Ondrej Lichtner <olichtne(a)redhat.com> Common/LoggingHandler.py | 108 ++++++++++++++++++++++++++++++++++++++++++++++ 1 files changed, 108 insertions(+), 0 deletions(-) --- diff --git a/Common/LoggingHandler.py b/Common/LoggingHandler.py new file mode 100644 index 0000000..9b08f9d --- /dev/null +++ b/Common/LoggingHandler.py @@ -0,0 +1,108 @@ +""" +Server-like logging handler. +Stores logged messages in a buffer. Every time a new message is emitted it +checks for incoming connections. If a connection is established it flushes the +messages stored in the buffer to the connecting client. + +Copyright 2012 Red Hat, Inc. +Licensed under the GNU General Public License, version 2 as +published by the Free Software Foundation; see COPYING for details. +""" + +__autor__ = """ +olichtne(a)redhat.com (Ondrej Lichtner) +""" + +import socket, struct, pickle +import logging + +DEFAULT_LOG_PORT = 9998 + +class ServerHandler(logging.Handler): + def __init__(self, port=DEFAULT_LOG_PORT): + logging.Handler.__init__(self) + self.port = port + + self.sock = None + self.buf = [] + + self.server_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM) + self.server_socket.setblocking(0) + self.server_socket.bind(('0.0.0.0', self.port)) + self.server_socket.listen(1) + + def makePickle(self, record): + """ + Pickles the record in binary format with a length prefix, and + returns it ready for transmission across the socket. + + Function taken from class SocketHandler from standard python + library logging.handlers + """ + d = dict(record.__dict__) + d['msg'] = record.getMessage() + d['args'] = None + d['exc_info'] = None + s = pickle.dumps(d, 1) + slen = struct.pack(">L", len(s)) + return slen + s + + def emit(self, record): + try: + s = self.makePickle(record) + self.buf.append(s) + self.send_all() + except (KeyboardInterrupt, SystemExit): + raise + except: + logging.Handler.handleError(self, record) + + def client_connection(self): + if self.sock == None: + try: + self.sock = self.server_socket.accept()[0] + return True + except: + self.sock = None + return False + else: + return True + + def send_all(self): + if self.client_connection(): + sent = len(self.buf) + for record in self.buf: + if not self.send(record): + sent = self.buf.index(record) + break + + self.buf = self.buf[sent:] + + def send(self, record): + try: + if hasattr(self.sock, "sendall"): + self.sock.sendall(record) + else: + sentsofar = 0 + left = len(record) + while left > 0: + sent = self.sock.send(record[sentsofar:]) + sentsofar = sentsofar + sent + left = left - sent + return True + except socket.error: + self.sock.close() + self.sock = None + return False + + def close(self): + if self.sock: + self.sock.close() + self.sock = None + + self.server_socket.close() + self.server_socket = None + + self.buf = [] + + logging.Handler.close(self)

1 0

machineconfig templates: mapping of id's
by Radek Pazdera 04 Oct '12

04 Oct '12

Hi! I'm currently working on incorporating the matching algorithm code (more precisely a class) into LNST. I had to modify some parts of NetTestParser, but I have those parts basically ready. The matching algorithm is already wrapped-up in a class and ready as well. But I can't figure out how to work with the matched ID's. When a template topology's been successfully matched, we need to take the <machineconfig>'s and put it into the recipe in place of the teplates. But there will be different phys_id's, so we need to change them somehow. We could replace them according to the matches, but it turns out we can't do that so easily :(. In case of a successful partial match, there could be conflicts between id's from template namespace and pool namespace. To illustrate this, let's match a template A on a machine B: * template A with 2 interfaces, id's 1 and 2 * pool machine B with 3 interfaces, id's 1, 2, 3 They could be matched like this: A B ----- ----- 1 --> 3 2 --> 2 unused 1 In this case, after replacing matched id's we'd get a machine with devices 1,1,2. One is unused, so we could just remove it, but I don't think it's a good idea. If the device is unused for testing (not referenced from the <netconfig>), LNST should make sure it is DOWN, so it cannot affect the test results in some very obscure way. We could also do it the other way around. Replace the references to phys_id in <netconfig>'s. But that could be a little confusing at times. In case the values are ever used for error reporting. In my opinion, the best way would be to count in both id's and somehow work with them both. Maybe create some abstract type for id? I don't know ... But it's hard to do such changes to the structure of the recipe variable, because it is used on many places across a lot of modules. What do you think? -Radek

2 1

← Newer
1
2
3
4
5
6
7
8
9
Older →

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

LNST-developers October 2012