From: Ondrej Lichtner olichtne@redhat.com
Hi all,
the following patch is a proposal for how to do documentation for LNST-next. To review there are two parts:
* review the patch source code changes as usual, if there's something that doesn't make sense code wise or similar...
* please also apply the patch locally and generate the documentation as follows:
cd docs/ make html
Then open the generated html index file located at build/html/index.html with your favourite browser.
To generate the documentation you will need "sphinx" installed, for me that is provided by the python-sphinx package on Arch Linux, there should be an equivalent package on other distrubutions or you can install it via PyPI:
pip install -U sphinx
Thanks, -Ondrej
Ondrej Lichtner (1): Sphinx based documentation
README.md | 25 +- docs/Makefile | 20 ++ docs/source/ControllerAPI.rst | 5 + docs/source/Parameters.rst | 5 + docs/source/RecipeAPI.rst | 5 + docs/source/RecipeRequirementsAPI.rst | 5 + docs/source/SimpleNetworkRecipe.rst | 6 + docs/source/TesterAPI.rst | 10 + docs/source/base_enrt_class.rst | 8 + docs/source/conf.py | 64 +++++ docs/source/config_mixins.rst | 3 + docs/source/enrt_recipes.rst | 12 + docs/source/index.rst | 25 ++ docs/source/installation.rst | 133 +++++++++ docs/source/recipe_packages.rst | 7 + docs/source/specific_scenarios.rst | 5 + lnst/Common/Parameters.py | 23 +- lnst/Controller/Controller.py | 92 +++--- lnst/Controller/Recipe.py | 71 ++--- lnst/Controller/Requirements.py | 80 ++++-- lnst/Controller/__init__.py | 7 + lnst/Recipes/ENRT/BaseEnrtRecipe.py | 312 ++++++++++++++++++++- lnst/Recipes/ENRT/ConfigMixins/README.md | 3 + lnst/Recipes/ENRT/ConfigMixins/__init__.py | 9 + lnst/Recipes/ENRT/README.md | 47 ++++ lnst/Recipes/ENRT/SimpleNetworkRecipe.py | 49 ++++ lnst/Recipes/ENRT/__init__.py | 52 ++++ 27 files changed, 965 insertions(+), 118 deletions(-) create mode 100644 docs/Makefile create mode 100644 docs/source/ControllerAPI.rst create mode 100644 docs/source/Parameters.rst create mode 100644 docs/source/RecipeAPI.rst create mode 100644 docs/source/RecipeRequirementsAPI.rst create mode 100644 docs/source/SimpleNetworkRecipe.rst create mode 100644 docs/source/TesterAPI.rst create mode 100644 docs/source/base_enrt_class.rst create mode 100644 docs/source/conf.py create mode 100644 docs/source/config_mixins.rst create mode 100644 docs/source/enrt_recipes.rst create mode 100644 docs/source/index.rst create mode 100644 docs/source/installation.rst create mode 100644 docs/source/recipe_packages.rst create mode 100644 docs/source/specific_scenarios.rst create mode 100644 lnst/Recipes/ENRT/ConfigMixins/README.md create mode 100644 lnst/Recipes/ENRT/README.md
From: Ondrej Lichtner olichtne@redhat.com
Creating the docs/ directory where we should work on creating documentation. The standard for Python projects is to use "Sphinx" to generate various formats (e.g. pdf, html or others) of documentation from ReStructuredText (rst) source. The source is either stored as ".rst" standalone files or as docstrings with Python source code.
This is the first version that should cover some of the major topics and exported APIs available for the tester. It is by no means complete yet but should be enough to get a feel for how the overall structure should look like.
Signed-off-by: Ondrej Lichtner olichtne@redhat.com --- README.md | 25 +- docs/Makefile | 20 ++ docs/source/ControllerAPI.rst | 5 + docs/source/Parameters.rst | 5 + docs/source/RecipeAPI.rst | 5 + docs/source/RecipeRequirementsAPI.rst | 5 + docs/source/SimpleNetworkRecipe.rst | 6 + docs/source/TesterAPI.rst | 10 + docs/source/base_enrt_class.rst | 8 + docs/source/conf.py | 64 +++++ docs/source/config_mixins.rst | 3 + docs/source/enrt_recipes.rst | 12 + docs/source/index.rst | 25 ++ docs/source/installation.rst | 133 +++++++++ docs/source/recipe_packages.rst | 7 + docs/source/specific_scenarios.rst | 5 + lnst/Common/Parameters.py | 23 +- lnst/Controller/Controller.py | 92 +++--- lnst/Controller/Recipe.py | 71 ++--- lnst/Controller/Requirements.py | 80 ++++-- lnst/Controller/__init__.py | 7 + lnst/Recipes/ENRT/BaseEnrtRecipe.py | 312 ++++++++++++++++++++- lnst/Recipes/ENRT/ConfigMixins/README.md | 3 + lnst/Recipes/ENRT/ConfigMixins/__init__.py | 9 + lnst/Recipes/ENRT/README.md | 47 ++++ lnst/Recipes/ENRT/SimpleNetworkRecipe.py | 49 ++++ lnst/Recipes/ENRT/__init__.py | 52 ++++ 27 files changed, 965 insertions(+), 118 deletions(-) create mode 100644 docs/Makefile create mode 100644 docs/source/ControllerAPI.rst create mode 100644 docs/source/Parameters.rst create mode 100644 docs/source/RecipeAPI.rst create mode 100644 docs/source/RecipeRequirementsAPI.rst create mode 100644 docs/source/SimpleNetworkRecipe.rst create mode 100644 docs/source/TesterAPI.rst create mode 100644 docs/source/base_enrt_class.rst create mode 100644 docs/source/conf.py create mode 100644 docs/source/config_mixins.rst create mode 100644 docs/source/enrt_recipes.rst create mode 100644 docs/source/index.rst create mode 100644 docs/source/installation.rst create mode 100644 docs/source/recipe_packages.rst create mode 100644 docs/source/specific_scenarios.rst create mode 100644 lnst/Recipes/ENRT/ConfigMixins/README.md create mode 100644 lnst/Recipes/ENRT/README.md
diff --git a/README.md b/README.md index 256f79e..dec9aba 100644 --- a/README.md +++ b/README.md @@ -12,7 +12,7 @@ backwards compatibility are yet in place.
This also means that many of our documentation resources outlining how to write recipes on the wiki are also out of date. We'll soon start working on these but -please be paitent with us. +please be patient with us.
If you're interested in helping out we accept code contributions via Patches submitted to our mailing list lnst-developers@lists.fedorahosted.org. @@ -33,13 +33,13 @@ Internet Resources bellow).
## Install
-LNST can be installed using python's distutils. +Installation and a simple Hello world example is available at +[Installation](docs/source/installation.rst)
-```bash -su -./setup.py install -``` +## Documentation
+Documentation is available in the `docs/` directory, you can build it with +`make html` using *Sphinx*.
## Authors/Contributors
@@ -53,18 +53,15 @@ su * Jiri Zupka (not active anymore) * Radek Pazdera (not active anymore)
+## How to contact us
-## Internet Resources - -* Project Wiki: https://github.com/jpirko/lnst/wiki (currently out of date) -* Documentation: https://github.com/jpirko/lnst/wiki#learn (currently out of date) -* Git Source Tree: https://github.com/jpirko/lnst -* Mailing List: lnst-developers@lists.fedorahosted.org - +* Git Source Tree: https://github.com/jpirko/lnst +* Mailing List: lnst-developers@lists.fedorahosted.org +* IRC channel: #lnst @ freenode.net
## License
-**Copyright (C) 2011-2019 Red Hat, Inc.** +**Copyright (C) 2011-2020 Red Hat, Inc.**
LNST is distributed under GNU General Public License version 2. See the file "COPYING" in the source distribution for information on terms & conditions diff --git a/docs/Makefile b/docs/Makefile new file mode 100644 index 0000000..d0c3cbf --- /dev/null +++ b/docs/Makefile @@ -0,0 +1,20 @@ +# Minimal makefile for Sphinx documentation +# + +# You can set these variables from the command line, and also +# from the environment for the first two. +SPHINXOPTS ?= +SPHINXBUILD ?= sphinx-build +SOURCEDIR = source +BUILDDIR = build + +# Put it first so that "make" without argument is like "make help". +help: + @$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) + +.PHONY: help Makefile + +# Catch-all target: route all unknown targets to Sphinx using the new +# "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS). +%: Makefile + @$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) diff --git a/docs/source/ControllerAPI.rst b/docs/source/ControllerAPI.rst new file mode 100644 index 0000000..f80d840 --- /dev/null +++ b/docs/source/ControllerAPI.rst @@ -0,0 +1,5 @@ +Controller API +^^^^^^^^^^^^^^ + +.. automodule:: lnst.Controller.Controller + :members: Controller diff --git a/docs/source/Parameters.rst b/docs/source/Parameters.rst new file mode 100644 index 0000000..9646b01 --- /dev/null +++ b/docs/source/Parameters.rst @@ -0,0 +1,5 @@ +Parameters +^^^^^^^^^^ + +.. automodule:: lnst.Common.Parameters + :members: diff --git a/docs/source/RecipeAPI.rst b/docs/source/RecipeAPI.rst new file mode 100644 index 0000000..32e7b4c --- /dev/null +++ b/docs/source/RecipeAPI.rst @@ -0,0 +1,5 @@ +Recipe API +^^^^^^^^^^ + +.. automodule:: lnst.Controller.Recipe + :members: BaseRecipe, RecipeRun diff --git a/docs/source/RecipeRequirementsAPI.rst b/docs/source/RecipeRequirementsAPI.rst new file mode 100644 index 0000000..4cf1c59 --- /dev/null +++ b/docs/source/RecipeRequirementsAPI.rst @@ -0,0 +1,5 @@ +Recipe Requirements +^^^^^^^^^^^^^^^^^^^ + +.. automodule:: lnst.Controller.Requirements + :members: HostReq, DeviceReq diff --git a/docs/source/SimpleNetworkRecipe.rst b/docs/source/SimpleNetworkRecipe.rst new file mode 100644 index 0000000..1f08feb --- /dev/null +++ b/docs/source/SimpleNetworkRecipe.rst @@ -0,0 +1,6 @@ +SimpleNetworkRecipe +^^^^^^^^^^^^^^^^^^^ + +.. autoclass:: lnst.Recipes.ENRT.SimpleNetworkRecipe.SimpleNetworkRecipe + :members: + :show-inheritance: diff --git a/docs/source/TesterAPI.rst b/docs/source/TesterAPI.rst new file mode 100644 index 0000000..9261171 --- /dev/null +++ b/docs/source/TesterAPI.rst @@ -0,0 +1,10 @@ +Test developer API +================== + +.. toctree:: + :maxdepth: 2 + + RecipeRequirementsAPI + RecipeAPI + ControllerAPI + Parameters diff --git a/docs/source/base_enrt_class.rst b/docs/source/base_enrt_class.rst new file mode 100644 index 0000000..32dd6fd --- /dev/null +++ b/docs/source/base_enrt_class.rst @@ -0,0 +1,8 @@ +BaseEnrtRecipe class +==================== + +.. autoclass:: lnst.Recipes.ENRT.BaseEnrtRecipe.BaseEnrtRecipe + :members: + :show-inheritance: + +.. autoclass:: lnst.Recipes.ENRT.BaseEnrtRecipe.EnrtConfiguration diff --git a/docs/source/conf.py b/docs/source/conf.py new file mode 100644 index 0000000..7c59490 --- /dev/null +++ b/docs/source/conf.py @@ -0,0 +1,64 @@ +# Configuration file for the Sphinx documentation builder. +# +# This file only contains a selection of the most common options. For a full +# list see the documentation: +# https://www.sphinx-doc.org/en/master/usage/configuration.html + +# -- Path setup -------------------------------------------------------------- + +# If extensions (or modules to document with autodoc) are in another directory, +# add these directories to sys.path here. If the directory is relative to the +# documentation root, use os.path.abspath to make it absolute, like shown here. +# +# import os +# import sys +# sys.path.insert(0, os.path.abspath('.')) + + +# -- Project information ----------------------------------------------------- + +project = 'lnst' +copyright = '2020, Jiri Pirko' +author = 'Jiri Pirko' + + +# -- General configuration --------------------------------------------------- + +# Add any Sphinx extension module names here, as strings. They can be +# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom +# ones. +extensions = [ + 'sphinx.ext.autodoc', +] + +import sys +sys.path = ['../'] + sys.path + +# Add any paths that contain templates here, relative to this directory. +templates_path = ['_templates'] + +# List of patterns, relative to source directory, that match files and +# directories to ignore when looking for source files. +# This pattern also affects html_static_path and html_extra_path. +exclude_patterns = [] + + +# -- Options for HTML output ------------------------------------------------- + +# The theme to use for HTML and HTML Help pages. See the documentation for +# a list of builtin themes. +# +html_theme = 'classic' +html_theme_options = { + #"body_min_width": "100%", + "body_max_width": "100%", +} + +# Add any paths that contain custom static files (such as style sheets) here, +# relative to this directory. They are copied after the builtin static files, +# so a file named "default.css" will overwrite the builtin "default.css". +html_static_path = ['_static'] + +autodoc_default_options = { + 'member-order': 'bysource', +} diff --git a/docs/source/config_mixins.rst b/docs/source/config_mixins.rst new file mode 100644 index 0000000..66119b6 --- /dev/null +++ b/docs/source/config_mixins.rst @@ -0,0 +1,3 @@ +.. ENRT Config Mixins + +.. automodule:: lnst.Recipes.ENRT.ConfigMixins diff --git a/docs/source/enrt_recipes.rst b/docs/source/enrt_recipes.rst new file mode 100644 index 0000000..af8935a --- /dev/null +++ b/docs/source/enrt_recipes.rst @@ -0,0 +1,12 @@ +Early Network Regression Testing (ENRT) +======================================= + +.. automodule:: lnst.Recipes.ENRT + +.. toctree:: + :maxdepth: 2 + :caption: ENRT Components + + base_enrt_class + config_mixins + specific_scenarios diff --git a/docs/source/index.rst b/docs/source/index.rst new file mode 100644 index 0000000..e5f2dd5 --- /dev/null +++ b/docs/source/index.rst @@ -0,0 +1,25 @@ +LNST - Linux Network Stack Test +=============================== + +Linux Network Stack Test is a framework that supports development and execution +of automated and portable network tests which usually involve multiple test +systems. This repository contains the implementation of the library to write +:any:`Recipes<BaseRecipe>`, library functions to implement executable script +files which run your tests as well as the code for the "LNST Slave" +application. + +.. toctree:: + :maxdepth: 2 + :caption: General topics: + + installation + TesterAPI + recipe_packages + + +Indices and tables +================== + +* :ref:`genindex` +* :ref:`modindex` +* :ref:`search` diff --git a/docs/source/installation.rst b/docs/source/installation.rst new file mode 100644 index 0000000..ede73a2 --- /dev/null +++ b/docs/source/installation.rst @@ -0,0 +1,133 @@ +Install LNST and Hello world +============================ + +LNST is logically split into two separate application use cases: + +* Controller - something that controlls the execution of your :any:`Test + Recipes<BaseRecipe>` +* Slave - a server application running on all hosts available for testing, + executes remote procedure calls from the Controller to either run tests or + configure the test machine + +Codebases for both use cases are developed in this repository and as we +currently don't have a stable release yet, the recommended method of +installation involves the following steps: + +.. code-block:: bash + + git clone https://github.com/jpirko/lnst + cd lnst + pip3 install --requirements requirements.txt + pip3 install . + + +This installs both the Controller and the Slave code, and you'll need to run +this on all the test machines that you want to use as well as the machine which +you want to use as the Controller. Optionally a Controller and a Slave CAN run +on the same machine. + +You can start your Slave application immediatelly by running:: + + lnst-slave + +Because the lnst-slave application takes care of network configuration, it +**requires** to be executed with root privileges. This is **A BIG SECURITY +RISK** so make sure you only run this application on test machines that are not +publicly accessible or don't contain any sensitive data. + +The Controller is a bit more complicated and requires you to: + +* create an executable test script +* create a slave machine pool + +Creating an executable "HelloWorld" test script +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +LNST currently doesn't come with a CLI application for the Controller, instead +we need to create an executable python script ourselves that takes care of +creating an instance of a Controller class and an instance of the Test Recipe +that we want to run and calling the Controller.run() method to execute it. + +A minimal "hello world" example of an executable test script looks like this: + +.. code-block:: python + + from lnst.Controller import Controller, HostReq, DeviceReq, BaseRecipe + + class HelloWorldRecipe(BaseRecipe): + machine1 = HostReq() + machine1.nic1 = DeviceReq(label="net1") + + machine2 = HostReq() + machine2.nic1 = DeviceReq(label="net1") + + def test(self): + self.matched.m1.nic1.ip_add("192.168.1.1/24") + self.matched.m1.nic1.up() + self.matched.m2.nic1.ip_add("192.168.1.2/24") + self.matched.m2.nic1.up() + + self.matched.m1.run("ping 192.168.1.2") + + ctl = Controller() + recipe_instance = HelloWorldRecipe() + ctl.run(recipe_instance) + + +This test requires that you have 2 test machines that are directly connected to +each other. +If you write this into a ``hello_world.py`` file you should now be able to +execute this script by running:: + + python3 hello_world.py + +And you'll end up receiving an error about being unable to find a match in your +configured pools, since we didn't configure any yet, this is quite expected. But +running this script did take care of creating a default configuration file and +directory where we'll now be able to create our machine pool. + +Creating a simple machine pool +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The default location for the Controller config file is ``~/.lnst/lnst-ctl.conf``. +At this point in time, you don't need to change anything inside this file. + +At the same time, the default location for a machine pool is ``~/.lnst/pool/``, +to create a pool you'll need to put XML files that describe your test machines +where the ``lnst-slave`` application is running, and how they're connected. You +need to create one file per test machine, so to satisfy the +**HelloWorldRecipe** requirements, we need to create two files: + +.. code-block:: bash + + touch ~/.lnst/pool/test_machine1.xml + touch ~/.lnst/pool/test_machine2.xml + +For the contents of the files you can use the following template: + +.. code-block:: xml + + <slavemachine> + <params> + <param name="hostname" value="HOSTNAME"/> + <param name="rpc_port" value="9999"/> + </params> + <interfaces> + <eth label="A" id="1"> + <params> + <param name="hwaddr" value="MAC_ADDRESS"/> + </params> + </eth> + </interfaces> + </slavemachine> + +You'll need to edit the template and replace the **HOSTNAME** and +**MAC_ADDRESS** strings with values that correspond to the hostname which the +controller can use to connet to the slave, and the mac address of a network +interface usable for testing. This **MUST** be a different interface than the +one used for the Controller-Slave connection, as it's configuration will change +during test execution, the Controller-Slave connection would break if you used +the same interface. + +After creating your pool, you should now be able to run the ``hello_world.py`` +script successfully and receive back some logs about what happened. diff --git a/docs/source/recipe_packages.rst b/docs/source/recipe_packages.rst new file mode 100644 index 0000000..ccd6269 --- /dev/null +++ b/docs/source/recipe_packages.rst @@ -0,0 +1,7 @@ +Supported Recipe packages +========================= + +.. toctree:: + :maxdepth: 2 + + enrt_recipes diff --git a/docs/source/specific_scenarios.rst b/docs/source/specific_scenarios.rst new file mode 100644 index 0000000..2f13d02 --- /dev/null +++ b/docs/source/specific_scenarios.rst @@ -0,0 +1,5 @@ +Specific ENRT scenarios +======================= + +.. toctree:: + SimpleNetworkRecipe diff --git a/lnst/Common/Parameters.py b/lnst/Common/Parameters.py index e56c32d..f73fd3a 100644 --- a/lnst/Common/Parameters.py +++ b/lnst/Common/Parameters.py @@ -4,10 +4,6 @@ This module defines the Param class, it's type specific derivatives Param instances. This can be used by a BaseRecipe class to specify optional/mandatory parameters for the entire test, or by HostReq and DeviceReq classes to define specific parameters needed for the matching algorithm. - -Copyright 2017 Red Hat, Inc. -Licensed under the GNU General Public License, version 2 as -published by the Free Software Foundation; see COPYING for details. """
__author__ = """ @@ -24,12 +20,31 @@ class ParamError(LnstError): pass
class Param(object): + """Base Parameter class + + Can beused to define your own specific parameter type. Param derived classes + serve as *type checkers* to enable earlier failure of the recipe. + + :param mandatory: if `True`, marks the parameter as mandatory + :type mandatory: bool + + :param default: the default value for the parameter, is also type-checked, + immediately at Param object creation + """ def __init__(self, mandatory=False, **kwargs): self.mandatory = mandatory if "default" in kwargs: self.default = self.type_check(kwargs["default"])
def type_check(self, value): + """The type check method + + Implementation depends on the specific Param derived class. + + :return: the type checked or converted value + + :raises: :any:`ParamError` if the type check or conversion is invalid + """ return value
class IntParam(Param): diff --git a/lnst/Controller/Controller.py b/lnst/Controller/Controller.py index ace0caf..d992b27 100644 --- a/lnst/Controller/Controller.py +++ b/lnst/Controller/Controller.py @@ -2,10 +2,6 @@ This module defines the Controller class that brings together individual implementation parts of an LNST Controller. When instantiated, it allows the tester to configure and run his own recipes with the LNST 'infrastructure'. - -Copyright 2017 Red Hat, Inc. -Licensed under the GNU General Public License, version 2 as -published by the Free Software Foundation; see COPYING for details. """
__author__ = """ @@ -33,34 +29,59 @@ from lnst.Controller.Recipe import BaseRecipe, RecipeRun from lnst.Controller.RecipeControl import RecipeControl
class Controller(object): - """The LNST Controller class - - Most importantly allows the tester to run instantiated Recipe tests using - the LNST infrastructure. - - Can be configured with custom implementation of several objects used for - setting up the infrastructure. + """Allows to run LNST Recipe instances + + This is the main mechanism that allows users to create their own executable + test scripts that execute LNST Recipes. + + The Controller class implementation provides the most common default values + for various parameters that can significantly change the way that Recipes + are executed. This includes custom implementations of classes that are used + for setting up the testing infrastructure such as the PoolManager or the + MachineMapper. + + :param poolMgr: + class that implements the + :py:class:`lnst.Controller.SlavePoolManager.SlavePoolManager` interface + will be instantiated by the Controller to provide the mapper with pools + available for matching, also handles the creation of + :py:class:`Machine` objects (internal LNST class used to access the + slave hosts) + :type poolMgr: :py:class:`lnst.Controller.MachineMapper.MachineMapper` + + :param mapper: + class that implements the + :py:class:`lnst.Controller.MachineMapper.MachineMapper` interface will + be instantiated by the Controller to match Recipe requirements to the + available pools + :type mapper: :py:class:`lnst.Controller.MachineMapper.MachineMapper` + + :param config: + optional LNST configuration object, if None the Controller will + load it's own configuration from default paths + :type config: :py:class:`lnst.Controller.Config.CtlConfig` + + :param pools: + a list of pool names to restrict the used pool directories + :type pools: List[str] + + :param pool_checks: + if False, will disable checking the online status of Slaves + :type pool_checks: boolean (default True) + + :param debug: + sets the debug level of LNST + :type debug: integer (default 0) + + Example:: + + lnst_controller = Controller() + recipe_instance = MyRecipe(test_parameter=123) + lnst_controller.run(recipe_instance) """
def __init__(self, poolMgr=SlavePoolManager, mapper=MachineMapper, config=None, pools=[], pool_checks=True, debug=0): - """ - Args: - poolMgr -- class that implements the SlavePoolManager interface - will be instantiated by the Controller to provide the mapper - with pools available for matching, also handles the creation - of Machine objects (internal LNST class used to access the - slave hosts) - mapper -- class that implements the MachineMapper interface - will be instantiated by the Controller to match Recipe - requirements to the available pools - config -- optional LNST configuration object, if None the - Controller will load it's own configuration from default paths - pools -- a list of pool names to restrict the used pool directories - pool_checks -- boolean (default True), if False will disable - checking online status of Slaves - debug -- integer (default 0), sets debug level of LNST - """ self._config = self._load_ctl_config(config) config = self._config
@@ -96,13 +117,17 @@ class Controller(object): def run(self, recipe, **kwargs): """Execute the provided Recipe
- This method takes care of both finding a Slave hosts matching the Recipe - requirements, provisioning them and calling the 'test' method of the + This method takes care of both finding Slave hosts matching the Recipe + requirements, provisioning them and calling the *test* method of the Recipe object with proper references to the mapped Hosts
- Args: - recipe -- an instantiated Recipe object (isinstance BaseRecipe) - kwargs -- optional keyword arguments passed to the configured Mapper + :param recipe: + an instantiated Recipe object + :type recipe: :py:class:`lnst.Controller.Recipe.BaseRecipe` + + :param kwargs: + optional keyword arguments passed to the configured Mapper + :type kwargs: Dict[str, Any] """ if not isinstance(recipe, BaseRecipe): raise ControllerError("recipe argument must be a BaseRecipe instance.") @@ -134,7 +159,6 @@ class Controller(object): finally: self._cleanup_slaves()
- def _map_match(self, match, requested, recipe): self._machines = {} self._hosts = Hosts() diff --git a/lnst/Controller/Recipe.py b/lnst/Controller/Recipe.py index 42f4e3d..26e0737 100644 --- a/lnst/Controller/Recipe.py +++ b/lnst/Controller/Recipe.py @@ -1,9 +1,5 @@ """ Module implementing the BaseRecipe class. - -Copyright 2017 Red Hat, Inc. -Licensed under the GNU General Public License, version 2 as -published by the Free Software Foundation; see COPYING for details. """
__author__ = """ @@ -23,27 +19,30 @@ class RecipeError(ControllerError): pass
class BaseRecipe(object): - """BaseRecipe class + """Base class for LNST Recipe definition.
Every LNST Recipe written by testers should be inherited from this class. An LNST Recipe is composed of several parts: + * Requirements definition - you define recipe requirements in a derived - class by defining class attributes of the HostReq type. You can further - specify Ethernet Device requirements by defining DeviceReq attributes - of the HostReq object. - Example: - m1 = HostReq(arch="x86_64") - m1.eth0 = DeviceReq(driver="ixgbe") - * Parameter definition (optional) - you can define paramaters of your Recipe - by defining class attributes of the Param type (or inherited). These - parameters can then be accessed from the test() method to change it's - behaviour. Parameter validity (type) is checked during the - instantiation of the Recipe object by the base __init__ method. - You can define your own __init__ method to implement more complex - Parameter checking if needed, but you MUST call the base __init__ - method first. - Example: - MyRecipe(BaseRecipe): + class by defining class attributes of the HostReq type. You can further + specify Ethernet Device requirements by defining DeviceReq attributes of + the :py:class:`lnst.Controller.Requirements.HostReq` object. Example:: + + class MyRecipe(BaseRecipe): + m1 = HostReq(arch="x86_64") + m1.eth0 = DeviceReq(driver="ixgbe") + + * Parameter definition (optional) - you can define paramaters of your + Recipe by defining class attributes of the :any:`Param` type (or + inherited). These parameters can then be accessed from the test() method + to change it's behaviour. Parameter validity (type) is checked during the + instantiation of the Recipe object by the base __init__ method. You can + define your own __init__ method to implement more complex Parameter + checking if needed, but you MUST call the base __init__ method first. + Example:: + + class MyRecipe(BaseRecipe): int_param = IntParam(mandatory=True) optional_param = IntParam()
@@ -55,18 +54,24 @@ class BaseRecipe(object): MyRecipe(int_param = 2, optional_param = 3)
* Test definition - this is done by defining the test() method, in this - method the tester has direct access to mapped LNST slave Hosts, can - manipulate them and implement his tests. - - Attributes: - matched -- when running the Recipe the Controller will fill this - attribute with a Hosts object after the Mapper finds suitable slave - hosts. - req -- instantiated Requirements object, you can optionally change the - Recipe requirements through this object during runtime (e.g. - variable number of hosts or devices of a host based on a Parameter) - params -- instantiated Parameters object, can be used to access the - calculated parameters during Recipe initialization/execution + method the tester has direct access to mapped LNST slave Hosts, can + manipulate them and implement his tests. + + :ivar matched: + When running the Recipe the Controller will fill this attribute with a + Hosts object after the Mapper finds suitable slave hosts. + :type matched: :py:class:`lnst.Controller.Host.Hosts` + + :ivar req: + Instantiated Requirements object, you can optionally change the Recipe + requirements through this object during runtime (e.g. variable number + of hosts or devices of a host based on a Parameter) + :type req: :py:class:`lnst.Controller.Requirements._Requirements` + + :ivar params: + Instantiated Parameters object, can be used to access the calculated + parameters during Recipe initialization/execution + :type params: :py:class:`lnst.Common.Parameters.Parameters` """ def __init__(self, **kwargs): """ diff --git a/lnst/Controller/Requirements.py b/lnst/Controller/Requirements.py index 342454f..ed61996 100644 --- a/lnst/Controller/Requirements.py +++ b/lnst/Controller/Requirements.py @@ -1,19 +1,19 @@ """ -This module defines the DeviceReq and HostReq classes, which can be used to +This module defines the :py:class:`lnst.Controller.Requirements.DeviceReq` and +:py:class:`lnst.Controller.Requirements.HostReq` classes, which can be used to create a global description of Requirements for a network test. You can use -these to define class attributes of a BaseRecipe derived class to specify +these to define class attributes of a +:py:class:`lnst.Controller.Recipe.BaseRecipe` derived class to specify "general" requirements for that Recipe, or you can add them to an instance of a Recipe derived class based on it's parameters to define requirements "specific" for that single test run.
-The module also specifies a Requirements class which serves as a container for -HostReq objects, while HostReq classes also serve as containers for DeviceReq -objects. The object tree created this way is translated to a dictionary used by -the internal LNST matching algorithm against available machines. - -Copyright 2017 Red Hat, Inc. -Licensed under the GNU General Public License, version 2 as -published by the Free Software Foundation; see COPYING for details. +The module also specifies a +:py:class:`lnst.Controller.Requirements._Requirements` class (currently for +internal use only) which serves as a container for HostReq objects, while +HostReq classes also serve as containers for DeviceReq objects. The object tree +created this way is translated to a dictionary used by the internal LNST +matching algorithm against available machines. """
__author__ = """ @@ -63,15 +63,22 @@ class HostReq(BaseReq):
To define a Host requirement you assign a HostReq instance to a class attribute of a BaseRecipe derived class. - Example: - class MyRecipe(BaseRecipe): - m1 = HostReq() - - Args: - kwargs -- any other arguments will be treated as arbitrary string - parameters that will be matched to parameters of Slave machines - which can define their parameter values based on the implementation - of the SlaveMachineParser + + :param kwargs: + any argument will be treated as arbitrary string parameters that will + be matched to parameters of Slave machines which can define their + parameter values based on the implementation of the SlaveMachineParser + + A special case is the use of a + :py:mod:`lnst.Controller.Requirements.RecipeParam` instance as value. + This is used to link to a value provided as a Parameter to the Recipe. + :type kwargs: Dict[str, Any] + + Example:: + + class MyRecipe(BaseRecipe): + m1 = HostReq() + m2 = HostReq(architecture="x86_64") """ def reinit_with_params(self, recipe_params): super(HostReq, self).reinit_with_params(recipe_params) @@ -93,21 +100,32 @@ class HostReq(BaseReq): return res
class DeviceReq(BaseReq): - """Specifies an Ethernet Device requirement + """Specifies a static test network Device requirement + + This will be used to find a matching test machine in the configured slave + machine pools, specifically this will be used to match against a test + device on a slave machine that is "statically" present on the machine. In + other words an actual REAL network device connected to a network usable for + testing.
To define a Device requirement you assign a DeviceReq instance to a HostReq instance in a BaseRecipe derived class. - Example: - class MyRecipe(BaseRecipe): - m1 = HostReq() - m1.eth0 = DeviceReq(label="net1") - - Args: - label -- string value indicating the network the Device is connected to - kwargs -- any other arguments will be treated as arbitrary string - parameters that will be matched to parameters of Slave machines - which can define their parameter values based on the implementation - of the SlaveMachineParser + + :param label: + string value indicating the network the Device is connected to + :type label: string + + :param kwargs: + any other arguments will be treated as arbitrary string parameters that + will be matched to parameters of Slave machines which can define their + parameter values based on the implementation of the SlaveMachineParser + :type kwargs: Dict[str, Any] + + Example:: + + class MyRecipe(BaseRecipe): + m1 = HostReq() + m1.eth0 = DeviceReq(label="net1") """ def __init__(self, label, **kwargs): self.label = label diff --git a/lnst/Controller/__init__.py b/lnst/Controller/__init__.py index 106bb91..cb0392d 100644 --- a/lnst/Controller/__init__.py +++ b/lnst/Controller/__init__.py @@ -1,3 +1,10 @@ +""" +Controller package +================== + +This package exposes public facing APIs that can be used to create Recipes, as +well as executable test scripts. +""" from lnst.Controller.Controller import Controller from lnst.Controller.Recipe import BaseRecipe from lnst.Controller.Requirements import HostReq, DeviceReq, RecipeParam diff --git a/lnst/Recipes/ENRT/BaseEnrtRecipe.py b/lnst/Recipes/ENRT/BaseEnrtRecipe.py index a90c837..d139546 100644 --- a/lnst/Recipes/ENRT/BaseEnrtRecipe.py +++ b/lnst/Recipes/ENRT/BaseEnrtRecipe.py @@ -2,7 +2,7 @@ import pprint from contextlib import contextmanager
from lnst.Common.LnstError import LnstError -from lnst.Common.Parameters import Param, IntParam, StrParam, BoolParam, ListParam +from lnst.Common.Parameters import Param, IntParam, StrParam, BoolParam, ListParam, FloatParam from lnst.Common.IpAddress import AF_INET, AF_INET6
from lnst.Recipes.ENRT.ConfigMixins.BaseSubConfigMixin import BaseSubConfigMixin @@ -16,10 +16,150 @@ from lnst.RecipeCommon.Perf.Measurements import StatCPUMeasurement from lnst.RecipeCommon.Perf.Evaluators import NonzeroFlowEvaluator
class EnrtConfiguration(object): + """Container object for configuration + + Intentionally left empty as it is intended to be used as a container to + store any values relevant to configuration being applied during the lifetime + of the Recipe. + """ pass
class BaseEnrtRecipe(BaseSubConfigMixin, PingTestAndEvaluate, PerfRecipe): - #common requirements parameters + """Base Recipe class for the ENRT recipe package + + This class defines the shared *test* method defining the common test + procedure in a very generic way. This common test procedure involves a + single main *test_wide* configuration that is different for every specific + scenario. After the main configuration there is usually a loop of several + minor *sub* configrations types that can take different values to slightly + change the tested use cases. + + Finally, for each combination of a **test_wide** + **sub** configuration we + do a several ping connection test and several performance measurement tests. + + **test_wide** and **sub** configurations are implemented with **context + manager** methods which ensure that if any exceptions are raised (for + example because of a bug in the recipe) that deconfiguration is called. + + Both **test_wide** and **sub** configurations are to be implemented in + different classes, the BaseEnrtRecipe class only defines the common API and + the base versions of the relevant methods. + + Test wide configuration is implemented via the following methods: + + * :any:`test_wide_configuration` + * :any:`test_wide_deconfiguration` + * :any:`generate_test_wide_description` + + Sub configurations are **mixed in** to classes defining the specific + scenario that is being tested. Various sub configurations are implemented as + individual Python **Mixin** classes in the + :any:`ConfigMixins<config_mixins>` package. These make use of Pythons + collaborative inheritance by calling the `super` function in a specific way. + The "machinery" for that defined in the :any:`BaseSubConfigMixin` class and + used in this class from the `test` method loop that generates the combined + Sub configuration and the `_sub_context` context manager method that takes + care of applying and removing the configuration. + + :param driver: + The driver parameter is used to modify the hw network requirements, + specifically to request Devices using the specified driver. This is + common enough in the Enrt recipes that it can be part of the Base class. + + :type driver: :any:`StrParam` (default "ixgbe") + + :param ip_versions: + Parameter that determines which IP protocol versions will be tested. + :type ip_versions: Tuple[Str] (default ("ipv4", "ipv6")) + + :param ping_parallel: + Parameter used by the :any:`generate_ping_configurations` generator. + Tells the generator method to create :any:`PingConf` objects that will + be run in parallel. + :type ping_parallel: :any:`BoolParam` (default False) + + :param ping_bidirect: + Parameter used by the :any:`generate_ping_configurations` generator. + Tells the generator method to create :any:`PingConf` objects for both + directions between the ping endpoints. + :type ping_bidirect: :any:`BoolParam` (default False) + + :param ping_count: + Parameter used by the :any:`generate_ping_configurations` generator. + Tells the generator how many pings should be sent for each ping test. + :type ping_count: :any:`IntParam` (default 100) + + :param ping_interval: + Parameter used by the :any:`generate_ping_configurations` generator. + Tells the generator how fast should the pings be sent in each ping test. + :type ping_interval: :any:`FloatParam` (default 0.2) + + :param ping_psize: + Parameter used by the :any:`generate_ping_configurations` generator. + Tells the generator how big should the pings packets be in each ping + test. + :type ping_psize: :any:`IntParam` (default None) + + :param perf_tests: + Parameter used by the :any:`generate_flow_combinations` generator. + Tells the generator what types of network flow measurements to generate + perf test configurations for. + :type perf_tests: Tuple[str] (default ("tcp_stream", "udp_stream", + "sctp_stream")) + + :param perf_tool_cpu: + Parameter used by the :any:`generate_flow_combinations` generator. To + indicate that the flow measurement should be pinned to a specific CPU + core. + :type perf_tool_cpu: :any:`IntParam` (optional parameter) + + :param perf_duration: + Parameter used by the :any:`generate_perf_configurations` generator. To + specify the duration of the performance measurements, in seconds. + :type perf_duration: :any:`IntParam` (default 60) + + :param perf_iterations: + Parameter used by the :any:`generate_perf_configurations` generator. To + specify how many times should each performance measurement be repeated + to generate cumulative results which can be statistically analyzed. + :type perf_iterations: :any:`IntParam` (default 5) + + :param perf_parallel_streams: + Parameter used by the :any:`generate_flow_combinations` generator. To + specify how many parallel streams of the same network flow should be + measured at the same time. + :type perf_parallel_streams: :any:`IntParam` (default 1) + + :param perf_msg_sizes: + Parameter used by the :any:`generate_flow_combinations` generator. To + specify what different message sizes (in bytes) used generated for the + network flow should be tested - each message size resulting in a + separate performance measurement. + :type perf_msg_sizes: List[Int] (default [123]) + + :param perf_reverse: + Parameter used by the :any:`generate_flow_combinations` generator. To + specify that both directions between the endpoints of a network flow + should be measured for the same test. + :type perf_reverse: :any:`BoolParam` (default False) + + + :param net_perf_tool: + Specifies a network flow measurement class that accepts :any:`PerfFlow` + objects and can be used to measure those specified flows + Parameter used by the :any:`generate_perf_configurations` generator to + create a PerfRecipeConf object. + :type net_perf_tool: :any:`BaseFlowMeasurement` (default + IperfFlowMeasurement) + + :param cpu_perf_tool: + Specifies a cpu measurement class that can be used to measure CPU + utilization on specified hosts. + Parameter used by the :any:`generate_perf_configurations` generator to + create a PerfRecipeConf object. + :type cpu_perf_tool: :any:`BaseCPUMeasurement` (default StatCPUMeasurement) + """ + driver = StrParam(default="ixgbe")
#common test parameters @@ -45,6 +185,20 @@ class BaseEnrtRecipe(BaseSubConfigMixin, PingTestAndEvaluate, PerfRecipe): cpu_perf_tool = Param(default=StatCPUMeasurement)
def test(self): + """Main test loop shared by all the Enrt recipes + + The test loop involves a single application of a **test_wide** + configuration, then a loop over multiple **sub** configurations that + involves: + + * creating the combined sub configuration of all available SubConfig + Mixin classes via :any:`generate_sub_configurations` + * applying the generated sub configuration via the :any:`_sub_context` + context manager method + * running tests + * removing the current sub configuration via the :any:`_sub_context` + context manager method + """ with self._test_wide_context() as main_config: for sub_config in self.generate_sub_configurations(main_config): with self._sub_context(sub_config) as recipe_config: @@ -60,19 +214,90 @@ class BaseEnrtRecipe(BaseSubConfigMixin, PingTestAndEvaluate, PerfRecipe): self.test_wide_deconfiguration(config)
def test_wide_configuration(self): + """Creates an empty :any:`EnrtConfiguration` object + + This is again used in potential collaborative inheritance design that + may potentially be useful for Enrt recipes. Derived classes will each + individually add their own values to the instance created here. This way + the complete test wide configuration is tracked in a single object. + + :return: returns a config object that tracks the applied configuration + that can be used during testing to inspect the current state and + make test decisions based on it. + :rtype: :any:`EnrtConfiguration` + + Example:: + + class Derived: + def test_wide_configuration(): + config = super().test_wide_configuration() + + # ... configure something + config.something = what_was_configured + + return config + """ return EnrtConfiguration()
def test_wide_deconfiguration(self, config): + """Base deconfiguration method. + + In the base class this should maybe only check if there's any leftover + configuration and warn about it. In derived classes this can be + overriden to take care of deconfiguring what was configured in the + respective test_wide_configuration method. + + Example:: + + class Derived: + def test_wide_deconfiguration(config): + # ... deconfigure something + del config.something #cleanup tracking + + return super().test_wide_deconfiguration() + """ #TODO check if anything is still applied and throw exception? return
def describe_test_wide_configuration(self, config): + """Describes the current test wide configuration + + Creates a new result object that tracks a *successful* result, but + contains the description of the full test wide configuration applied by + all the :any:`test_wide_configuration` methods in the class hierarchy. + + The description needs to be generated by the + :any:`generate_test_wide_description` method. Additionally the + description contains the state of all the parameters and their values + passed to the recipe class instance during initialization. + """ description = self.generate_test_wide_description(config) self.add_result(True, "Summary of used Recipe parameters:\n{}".format( pprint.pformat(self.params._to_dict()))) self.add_result(True, "\n".join(description))
def generate_test_wide_description(self, config): + """Generates the test wide configuration description + + Another class inteded to be used with the collaborative version of the + `super` method to cumulatively desribe the full test wide configuration + that was applied through multiple classes. + + The base class version of this method creates the initial list of + strings containing just the header line. Each string added to this list + will later be printed on its own line. + + :return: list of strings, each representing a single line + :rtype: List[str] + + Example:: + + class Derived: + def generate_sub_configuration_description(config): + desc = super().generate_sub_configuration_description(config) + desc.append("Configured something: {}".format(config.something)) + return desc + """ return [ "Testwide configuration for recipe {} description:".format( self.__class__.__name__ @@ -93,20 +318,47 @@ class BaseEnrtRecipe(BaseSubConfigMixin, PingTestAndEvaluate, PerfRecipe): self.add_result(True, "\n".join(description))
def do_tests(self, recipe_config): + """Entry point for actual tests + + The common scenario is to do ping and performance tests, however the + method can be overriden to add more tests if needed. + """ self.do_ping_tests(recipe_config) self.do_perf_tests(recipe_config)
def do_ping_tests(self, recipe_config): + """Ping testing loop + + Loops over all various ping configurations generated by the + :any:`generate_ping_configurations` method, then uses the PingRecipe + methods to execute, report and evaluate the results. + """ for ping_config in self.generate_ping_configurations(recipe_config): result = self.ping_test(ping_config) self.ping_evaluate_and_report(ping_config, result)
def do_perf_tests(self, recipe_config): + """Performance testing loop + + Loops over all various perf configurations generated by the + :any:`generate_perf_configurations` method, then uses the PerfRecipe + methods to execute, report and evaluate the results. + """ for perf_config in self.generate_perf_configurations(recipe_config): result = self.perf_test(perf_config) self.perf_report_and_evaluate(result)
def generate_ping_configurations(self, config): + """Base ping test configuration generator + + The generator loops over all endpoint pairs to test ping between + (generated by the :any:`generate_ping_endpoints` method) then over all + the selected :any:`ip_versions` and finally over all the IP addresses + that fit those criteria. + + :return: list of Ping configurations to test in parallel + :rtype: List[:any:`PingConf`] + """ for endpoint1, endpoint2 in self.generate_ping_endpoints(config): for ipv in self.params.ip_versions: ip_filter = {} @@ -144,9 +396,30 @@ class BaseEnrtRecipe(BaseSubConfigMixin, PingTestAndEvaluate, PerfRecipe): yield ping_conf_list
def generate_ping_endpoints(self, config): + """Generator for ping endpoints + + To be overriden by a derived class. + + :return: list of device pairs + :rtype: List[Tuple[:any:`Device`, :any:`Device`]] + """ return []
def generate_perf_configurations(self, config): + """Base perf test configuration generator + + The generator loops over all flow combinations to measure performance + for (generated by the :any:`generate_flow_combinations` method). In + addition to that during each flow combination measurement we add CPU + utilization measurement to run on the background. + + Finally for each generated perf test configuration we register + measurement evaluators based on the :any:`cpu_perf_evaluators` and + :any:`net_perf_evaluators` properties. + + :return: list of Perf test configurations + :rtype: List[:any:`PerfRecipeConf`] + """ for flows in self.generate_flow_combinations(config): perf_recipe_conf=dict( recipe_config=config, @@ -183,6 +456,18 @@ class BaseEnrtRecipe(BaseSubConfigMixin, PingTestAndEvaluate, PerfRecipe): yield perf_conf
def generate_flow_combinations(self, config): + """Base flow combination generator + + The generator loops over all endpoint pairs to test performance between + (generated by the :any:`generate_perf_endpoints` method) then over all + the selected :any:`ip_versions` and uses the first IP address fitting + these criteria. Then the generator loops over the selected performance + tests as selected via :any:`perf_tests`, then message sizes from + :any:`msg_sizes`. + + :return: list of Flow combinations to measure in parallel + :rtype: List[:any:`PerfFlow`] + """ for client_nic, server_nic in self.generate_perf_endpoints(config): for ipv in self.params.ip_versions: if ipv == "ipv4": @@ -213,14 +498,37 @@ class BaseEnrtRecipe(BaseSubConfigMixin, PingTestAndEvaluate, PerfRecipe): yield [reverse_flow]
def generate_perf_endpoints(self, config): + """Generator for perf endpoints + + To be overriden by a derived class. + + :return: list of device pairs + :rtype: List[Tuple[:any:`Device`, :any:`Device`]] + """ return []
@property def cpu_perf_evaluators(self): + """CPU measurement evaluators + + To be overriden by a derived class. Returns the list of evaluators to + use for CPU utilization measurement evaluation. + + :return: a list of cpu evaluator objects + :rtype: List[BaseEvaluator] + """ return []
@property def net_perf_evaluators(self): + """Network flow measurement evaluators + + To be overriden bby a derived class. Returns the list of evaluators to + use for Network flow measurement evaluation. + + :return: a list of flow evaluator objects + :rtype: List[BaseEvaluator] + """ return [NonzeroFlowEvaluator()]
def _create_reverse_flow(self, flow): diff --git a/lnst/Recipes/ENRT/ConfigMixins/README.md b/lnst/Recipes/ENRT/ConfigMixins/README.md new file mode 100644 index 0000000..0472be0 --- /dev/null +++ b/lnst/Recipes/ENRT/ConfigMixins/README.md @@ -0,0 +1,3 @@ +# SubConfig Mixins + +TODO diff --git a/lnst/Recipes/ENRT/ConfigMixins/__init__.py b/lnst/Recipes/ENRT/ConfigMixins/__init__.py index e69de29..7ae6450 100644 --- a/lnst/Recipes/ENRT/ConfigMixins/__init__.py +++ b/lnst/Recipes/ENRT/ConfigMixins/__init__.py @@ -0,0 +1,9 @@ +""" +ENRT Config Mixins +================== + +.. autoclass:: lnst.Recipes.ENRT.ConfigMixins.BaseSubConfigMixin.BaseSubConfigMixin + +""" + +from lnst.Recipes.ENRT.ConfigMixins.BaseSubConfigMixin import BaseSubConfigMixin diff --git a/lnst/Recipes/ENRT/README.md b/lnst/Recipes/ENRT/README.md new file mode 100644 index 0000000..3500cdc --- /dev/null +++ b/lnst/Recipes/ENRT/README.md @@ -0,0 +1,47 @@ +# ENRT Recipes + +ENRT stands for Early Network Regression Testing, which was the name of the +project a couple of years back when we started developing this set of tests +when we were still using the Legacy LNST API. + +This package aims to reimplement the same set of tests, using the new LNST-next +APIs, while fully utilizing Python to address the largest problems with the old +implementation: +* large amounts of code duplication +* copy paste errors caused by code duplication +* very hard to maintain the test set and fix bugs +* very hard to add new tests due to limitations of the old LNST Framework + +With that in mind, the main goals for this reimplementation were as follows: +* reduce code duplication as much as possible by utilizing class inheritance +* separate individual types of configurations into smaller Mixin classes that + can be mixed and matched based on what a specific Test scenario requires +* writing new test scenarios should be very quick because it should be possible + to reuse most of the functionality already defined +* it should be possible to very easily extend the recipes with new features such + as more types of parallel or sequential measurements, more types of + Evaluations for different types of measurements, or to be able to switch out + and use different measurement tools + +The resulting design is split into several parts that interact with each other +in a specific way: + +* [BaseEnrtRecipe](BaseEnrtRecipe.py) serves as the base class for all the + specific test scenarios we want to test. Defines the common test loop and the + default implementation for configuration generators used in this common test + loop. +* [ConfigMixins](ConfigMixins/README.md) package contains the various types of + "SubConfig" mixin classes, these are classes that implement some form of + configuration on test machine(s) which can be reused between mutliple recipes, + but can often times be looped over to try different variations of the + configuration. A good example is configuration of hardware offloads, it's + relevant to test it for many different test scenarios, but only some + combinations of offloads make sense based on the scenario and we often time + want to test more than one combination +* Specific test scenario implementation that defines the requirements and the + main configuration for the specific scenario that we want to test. It also + defines the combination of various "SubConfig" configurations that we want to + include by adding them to it's inheritance tree. If required the recipe can + also override any of the default functionality defined by it's parent classes, + a good example could be the generator method for creating various + configurations for flow performance measurement. diff --git a/lnst/Recipes/ENRT/SimpleNetworkRecipe.py b/lnst/Recipes/ENRT/SimpleNetworkRecipe.py index 911a996..2c8e12b 100644 --- a/lnst/Recipes/ENRT/SimpleNetworkRecipe.py +++ b/lnst/Recipes/ENRT/SimpleNetworkRecipe.py @@ -12,6 +12,25 @@ from lnst.Recipes.ENRT.ConfigMixins.CommonHWSubConfigMixin import ( class SimpleNetworkRecipe( CommonHWSubConfigMixin, OffloadSubConfigMixin, BaseEnrtRecipe ): + """ + This recipe implements Enrt testing for a simple network scenario that looks + as follows + + .. code-block:: + + +--------+ + +------+ switch +-----+ + | +--------+ | + +--+-+ +-+--+ + +-|eth0|-+ +-|eth0|-+ + | +----+ | | +----+ | + | host1 | | host2 | + +--------+ +--------+ + + All sub configurations are included via Mixin classes. + + The actual test machinery is implemented in the :any:`BaseEnrtRecipe` class. + """ host1 = HostReq() host1.eth0 = DeviceReq(label="net1", driver=RecipeParam("driver"))
@@ -26,6 +45,14 @@ class SimpleNetworkRecipe( dict(gro="on", gso="on", tso="on", tx="on", rx="off")))
def test_wide_configuration(self): + """ + Test wide configuration for this recipe involves just adding an IPv4 and + IPv6 address to the matched eth0 nics on both hosts. + + host1.eth0 = 192.168.101.1/24 and fc00::1/64 + + host2.eth0 = 192.168.101.2/24 and fc00::2/64 + """ host1, host2 = self.matched.host1, self.matched.host2 configuration = super().test_wide_configuration() configuration.test_wide_devices = [] @@ -41,6 +68,9 @@ class SimpleNetworkRecipe( return configuration
def generate_test_wide_description(self, config): + """ + Test wide description is extended with the configured addresses + """ desc = super().generate_test_wide_description(config) desc += [ "Configured {}.{}.ips = {}".format( @@ -51,14 +81,33 @@ class SimpleNetworkRecipe( return desc
def test_wide_deconfiguration(self, config): + "" # overriding the parent docstring del config.test_wide_devices
super().test_wide_deconfiguration(config)
def generate_ping_endpoints(self, config): + """ + The ping endpoints for this recipe are simply the two matched NICs: + + host1.eth0 and host2.eth0 + + Returned as:: + + [(self.matched.host1.eth0, self.matched.host2.eth0)] + """ return [(self.matched.host1.eth0, self.matched.host2.eth0)]
def generate_perf_endpoints(self, config): + """ + The perf endpoints for this recipe are simply the two matched NICs: + + host1.eth0 and host2.eth0 + + Returned as:: + + [(self.matched.host1.eth0, self.matched.host2.eth0)] + """ return [(self.matched.host1.eth0, self.matched.host2.eth0)]
def wait_tentative_ips(self, devices): diff --git a/lnst/Recipes/ENRT/__init__.py b/lnst/Recipes/ENRT/__init__.py index ea5aaf0..a591ef2 100644 --- a/lnst/Recipes/ENRT/__init__.py +++ b/lnst/Recipes/ENRT/__init__.py @@ -1,3 +1,55 @@ +''' +ENRT Recipes +------------ + +ENRT stands for Early Network Regression Testing, which was the name of the +project a couple of years back when we started developing this set of tests +when we were still using the Legacy LNST API. + +This package aims to reimplement the same set of tests, using the new LNST-next +APIs, while fully utilizing Python to address the largest problems with the old +implementation: + +* large amounts of code duplication +* copy paste errors caused by code duplication +* very hard to maintain the test set and fix bugs +* very hard to add new tests due to limitations of the old LNST Framework + +With that in mind, the main goals for this reimplementation were as follows: + +* reduce code duplication as much as possible by utilizing class inheritance +* separate individual types of configurations into smaller Mixin classes that + can be mixed and matched based on what a specific Test scenario requires +* writing new test scenarios should be very quick because it should be possible + to reuse most of the functionality already defined +* it should be possible to very easily extend the recipes with new features + such as more types of parallel or sequential measurements, more types of + Evaluations for different types of measurements, or to be able to switch out + and use different measurement tools + +The resulting design is split into several parts that interact with each other +in a specific way: + +* BaseEnrtRecipe serves as the base class for all the specific test scenarios + we want to test. Defines the common test loop and the default implementation + for configuration generators used in this common test loop. +* ConfigMixins package contains the various types of "SubConfig" mixin classes, + these are classes that implement some form of configuration on test + machine(s) which can be reused between mutliple recipes, but can often times + be looped over to try different variations of the configuration. A good + example is configuration of hardware offloads, it's relevant to test it for + many different test scenarios, but only some combinations of offloads make + sense based on the scenario and we often time want to test more than one + combination +* Specific test scenario implementation that defines the requirements and the + main configuration for the specific scenario that we want to test. It also + defines the combination of various "SubConfig" configurations that we want to + include by adding them to it's inheritance tree. If required the recipe can + also override any of the default functionality defined by it's parent + classes, a good example could be the generator method for creating various + configurations for flow performance measurement. + +''' from lnst.Recipes.ENRT.SimpleNetworkRecipe import SimpleNetworkRecipe from lnst.Recipes.ENRT.BondRecipe import BondRecipe from lnst.Recipes.ENRT.DoubleBondRecipe import DoubleBondRecipe
Mon, Mar 16, 2020 at 11:43:03AM CET, olichtne@redhat.com wrote:
From: Ondrej Lichtner olichtne@redhat.com
Creating the docs/ directory where we should work on creating documentation. The standard for Python projects is to use "Sphinx" to generate various formats (e.g. pdf, html or others) of documentation from ReStructuredText (rst) source. The source is either stored as ".rst" standalone files or as docstrings with Python source code.
This is the first version that should cover some of the major topics and exported APIs available for the tester. It is by no means complete yet but should be enough to get a feel for how the overall structure should look like.
Signed-off-by: Ondrej Lichtner olichtne@redhat.com
README.md | 25 +- docs/Makefile | 20 ++ docs/source/ControllerAPI.rst | 5 + docs/source/Parameters.rst | 5 + docs/source/RecipeAPI.rst | 5 + docs/source/RecipeRequirementsAPI.rst | 5 + docs/source/SimpleNetworkRecipe.rst | 6 + docs/source/TesterAPI.rst | 10 + docs/source/base_enrt_class.rst | 8 + docs/source/conf.py | 64 +++++ docs/source/config_mixins.rst | 3 + docs/source/enrt_recipes.rst | 12 + docs/source/index.rst | 25 ++ docs/source/installation.rst | 133 +++++++++ docs/source/recipe_packages.rst | 7 + docs/source/specific_scenarios.rst | 5 + lnst/Common/Parameters.py | 23 +- lnst/Controller/Controller.py | 92 +++--- lnst/Controller/Recipe.py | 71 ++--- lnst/Controller/Requirements.py | 80 ++++-- lnst/Controller/__init__.py | 7 + lnst/Recipes/ENRT/BaseEnrtRecipe.py | 312 ++++++++++++++++++++- lnst/Recipes/ENRT/ConfigMixins/README.md | 3 + lnst/Recipes/ENRT/ConfigMixins/__init__.py | 9 + lnst/Recipes/ENRT/README.md | 47 ++++ lnst/Recipes/ENRT/SimpleNetworkRecipe.py | 49 ++++ lnst/Recipes/ENRT/__init__.py | 52 ++++ 27 files changed, 965 insertions(+), 118 deletions(-) create mode 100644 docs/Makefile create mode 100644 docs/source/ControllerAPI.rst create mode 100644 docs/source/Parameters.rst create mode 100644 docs/source/RecipeAPI.rst create mode 100644 docs/source/RecipeRequirementsAPI.rst create mode 100644 docs/source/SimpleNetworkRecipe.rst create mode 100644 docs/source/TesterAPI.rst create mode 100644 docs/source/base_enrt_class.rst create mode 100644 docs/source/conf.py create mode 100644 docs/source/config_mixins.rst create mode 100644 docs/source/enrt_recipes.rst create mode 100644 docs/source/index.rst create mode 100644 docs/source/installation.rst create mode 100644 docs/source/recipe_packages.rst create mode 100644 docs/source/specific_scenarios.rst create mode 100644 lnst/Recipes/ENRT/ConfigMixins/README.md create mode 100644 lnst/Recipes/ENRT/README.md
diff --git a/README.md b/README.md index 256f79e..dec9aba 100644 --- a/README.md +++ b/README.md @@ -12,7 +12,7 @@ backwards compatibility are yet in place.
This also means that many of our documentation resources outlining how to write recipes on the wiki are also out of date. We'll soon start working on these but -please be paitent with us. +please be patient with us.
If you're interested in helping out we accept code contributions via Patches submitted to our mailing list lnst-developers@lists.fedorahosted.org. @@ -33,13 +33,13 @@ Internet Resources bellow).
## Install
-LNST can be installed using python's distutils. +Installation and a simple Hello world example is available at +[Installation](docs/source/installation.rst)
-```bash -su -./setup.py install -``` +## Documentation
+Documentation is available in the `docs/` directory, you can build it with +`make html` using *Sphinx*.
## Authors/Contributors
@@ -53,18 +53,15 @@ su
- Jiri Zupka (not active anymore)
- Radek Pazdera (not active anymore)
+## How to contact us
-## Internet Resources
-* Project Wiki: https://github.com/jpirko/lnst/wiki (currently out of date) -* Documentation: https://github.com/jpirko/lnst/wiki#learn (currently out of date) -* Git Source Tree: https://github.com/jpirko/lnst -* Mailing List: lnst-developers@lists.fedorahosted.org
+* Git Source Tree: https://github.com/jpirko/lnst +* Mailing List: lnst-developers@lists.fedorahosted.org +* IRC channel: #lnst @ freenode.net
## License
-**Copyright (C) 2011-2019 Red Hat, Inc.** +**Copyright (C) 2011-2020 Red Hat, Inc.**
LNST is distributed under GNU General Public License version 2. See the file "COPYING" in the source distribution for information on terms & conditions diff --git a/docs/Makefile b/docs/Makefile new file mode 100644 index 0000000..d0c3cbf --- /dev/null +++ b/docs/Makefile @@ -0,0 +1,20 @@ +# Minimal makefile for Sphinx documentation +#
+# You can set these variables from the command line, and also +# from the environment for the first two. +SPHINXOPTS ?= +SPHINXBUILD ?= sphinx-build +SOURCEDIR = source +BUILDDIR = build
+# Put it first so that "make" without argument is like "make help". +help:
- @$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
+.PHONY: help Makefile
+# Catch-all target: route all unknown targets to Sphinx using the new +# "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS). +%: Makefile
- @$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
diff --git a/docs/source/ControllerAPI.rst b/docs/source/ControllerAPI.rst new file mode 100644 index 0000000..f80d840 --- /dev/null +++ b/docs/source/ControllerAPI.rst @@ -0,0 +1,5 @@ +Controller API +^^^^^^^^^^^^^^
+.. automodule:: lnst.Controller.Controller
- :members: Controller
diff --git a/docs/source/Parameters.rst b/docs/source/Parameters.rst new file mode 100644 index 0000000..9646b01 --- /dev/null +++ b/docs/source/Parameters.rst @@ -0,0 +1,5 @@ +Parameters +^^^^^^^^^^
+.. automodule:: lnst.Common.Parameters
- :members:
diff --git a/docs/source/RecipeAPI.rst b/docs/source/RecipeAPI.rst new file mode 100644 index 0000000..32e7b4c --- /dev/null +++ b/docs/source/RecipeAPI.rst @@ -0,0 +1,5 @@ +Recipe API +^^^^^^^^^^
+.. automodule:: lnst.Controller.Recipe
- :members: BaseRecipe, RecipeRun
diff --git a/docs/source/RecipeRequirementsAPI.rst b/docs/source/RecipeRequirementsAPI.rst new file mode 100644 index 0000000..4cf1c59 --- /dev/null +++ b/docs/source/RecipeRequirementsAPI.rst @@ -0,0 +1,5 @@ +Recipe Requirements +^^^^^^^^^^^^^^^^^^^
+.. automodule:: lnst.Controller.Requirements
- :members: HostReq, DeviceReq
diff --git a/docs/source/SimpleNetworkRecipe.rst b/docs/source/SimpleNetworkRecipe.rst new file mode 100644 index 0000000..1f08feb --- /dev/null +++ b/docs/source/SimpleNetworkRecipe.rst @@ -0,0 +1,6 @@ +SimpleNetworkRecipe +^^^^^^^^^^^^^^^^^^^
+.. autoclass:: lnst.Recipes.ENRT.SimpleNetworkRecipe.SimpleNetworkRecipe
- :members:
- :show-inheritance:
diff --git a/docs/source/TesterAPI.rst b/docs/source/TesterAPI.rst new file mode 100644 index 0000000..9261171 --- /dev/null +++ b/docs/source/TesterAPI.rst @@ -0,0 +1,10 @@ +Test developer API +==================
+.. toctree::
- :maxdepth: 2
- RecipeRequirementsAPI
- RecipeAPI
- ControllerAPI
- Parameters
diff --git a/docs/source/base_enrt_class.rst b/docs/source/base_enrt_class.rst new file mode 100644 index 0000000..32dd6fd --- /dev/null +++ b/docs/source/base_enrt_class.rst @@ -0,0 +1,8 @@ +BaseEnrtRecipe class +====================
+.. autoclass:: lnst.Recipes.ENRT.BaseEnrtRecipe.BaseEnrtRecipe
- :members:
- :show-inheritance:
+.. autoclass:: lnst.Recipes.ENRT.BaseEnrtRecipe.EnrtConfiguration diff --git a/docs/source/conf.py b/docs/source/conf.py new file mode 100644 index 0000000..7c59490 --- /dev/null +++ b/docs/source/conf.py @@ -0,0 +1,64 @@ +# Configuration file for the Sphinx documentation builder. +# +# This file only contains a selection of the most common options. For a full +# list see the documentation: +# https://www.sphinx-doc.org/en/master/usage/configuration.html
+# -- Path setup --------------------------------------------------------------
+# If extensions (or modules to document with autodoc) are in another directory, +# add these directories to sys.path here. If the directory is relative to the +# documentation root, use os.path.abspath to make it absolute, like shown here. +# +# import os +# import sys +# sys.path.insert(0, os.path.abspath('.'))
+# -- Project information -----------------------------------------------------
+project = 'lnst' +copyright = '2020, Jiri Pirko' +author = 'Jiri Pirko'
+# -- General configuration ---------------------------------------------------
+# Add any Sphinx extension module names here, as strings. They can be +# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom +# ones. +extensions = [
- 'sphinx.ext.autodoc',
+]
+import sys +sys.path = ['../'] + sys.path
+# Add any paths that contain templates here, relative to this directory. +templates_path = ['_templates']
+# List of patterns, relative to source directory, that match files and +# directories to ignore when looking for source files. +# This pattern also affects html_static_path and html_extra_path. +exclude_patterns = []
+# -- Options for HTML output -------------------------------------------------
+# The theme to use for HTML and HTML Help pages. See the documentation for +# a list of builtin themes. +# +html_theme = 'classic' +html_theme_options = {
- #"body_min_width": "100%",
- "body_max_width": "100%",
+}
+# Add any paths that contain custom static files (such as style sheets) here, +# relative to this directory. They are copied after the builtin static files, +# so a file named "default.css" will overwrite the builtin "default.css". +html_static_path = ['_static']
+autodoc_default_options = {
- 'member-order': 'bysource',
+}
On Fedora 30 I had to add this line: master_doc = 'index'
This should not be an issue for distros that have sphinx >= 2.0
Still may be worth to have it added.
diff --git a/docs/source/config_mixins.rst b/docs/source/config_mixins.rst new file mode 100644 index 0000000..66119b6 --- /dev/null +++ b/docs/source/config_mixins.rst @@ -0,0 +1,3 @@ +.. ENRT Config Mixins
+.. automodule:: lnst.Recipes.ENRT.ConfigMixins diff --git a/docs/source/enrt_recipes.rst b/docs/source/enrt_recipes.rst new file mode 100644 index 0000000..af8935a --- /dev/null +++ b/docs/source/enrt_recipes.rst @@ -0,0 +1,12 @@ +Early Network Regression Testing (ENRT) +=======================================
+.. automodule:: lnst.Recipes.ENRT
+.. toctree::
- :maxdepth: 2
- :caption: ENRT Components
- base_enrt_class
- config_mixins
- specific_scenarios
diff --git a/docs/source/index.rst b/docs/source/index.rst new file mode 100644 index 0000000..e5f2dd5 --- /dev/null +++ b/docs/source/index.rst @@ -0,0 +1,25 @@ +LNST - Linux Network Stack Test +===============================
+Linux Network Stack Test is a framework that supports development and execution +of automated and portable network tests which usually involve multiple test +systems. This repository contains the implementation of the library to write +:any:`Recipes<BaseRecipe>`, library functions to implement executable script +files which run your tests as well as the code for the "LNST Slave" +application.
+.. toctree::
- :maxdepth: 2
- :caption: General topics:
- installation
- TesterAPI
- recipe_packages
Just a question here. Having some of the docs with capital and others with lower case is to distinguish between classes and generic documentation?
+Indices and tables +==================
+* :ref:`genindex` +* :ref:`modindex` +* :ref:`search` diff --git a/docs/source/installation.rst b/docs/source/installation.rst new file mode 100644 index 0000000..ede73a2 --- /dev/null +++ b/docs/source/installation.rst @@ -0,0 +1,133 @@ +Install LNST and Hello world +============================
+LNST is logically split into two separate application use cases:
+* Controller - something that controlls the execution of your :any:`Test
- Recipes<BaseRecipe>`
+* Slave - a server application running on all hosts available for testing,
- executes remote procedure calls from the Controller to either run tests or
- configure the test machine
+Codebases for both use cases are developed in this repository and as we +currently don't have a stable release yet, the recommended method of +installation involves the following steps:
+.. code-block:: bash
- git clone https://github.com/jpirko/lnst
- cd lnst
- pip3 install --requirements requirements.txt
- pip3 install .
+This installs both the Controller and the Slave code, and you'll need to run +this on all the test machines that you want to use as well as the machine which +you want to use as the Controller. Optionally a Controller and a Slave CAN run +on the same machine.
+You can start your Slave application immediatelly by running::
- lnst-slave
+Because the lnst-slave application takes care of network configuration, it +**requires** to be executed with root privileges. This is **A BIG SECURITY +RISK** so make sure you only run this application on test machines that are not +publicly accessible or don't contain any sensitive data.
+The Controller is a bit more complicated and requires you to:
+* create an executable test script +* create a slave machine pool
+Creating an executable "HelloWorld" test script +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+LNST currently doesn't come with a CLI application for the Controller, instead +we need to create an executable python script ourselves that takes care of +creating an instance of a Controller class and an instance of the Test Recipe +that we want to run and calling the Controller.run() method to execute it.
+A minimal "hello world" example of an executable test script looks like this:
+.. code-block:: python
- from lnst.Controller import Controller, HostReq, DeviceReq, BaseRecipe
- class HelloWorldRecipe(BaseRecipe):
machine1 = HostReq()machine1.nic1 = DeviceReq(label="net1")machine2 = HostReq()machine2.nic1 = DeviceReq(label="net1")def test(self):self.matched.m1.nic1.ip_add("192.168.1.1/24")self.matched.m1.nic1.up()self.matched.m2.nic1.ip_add("192.168.1.2/24")self.matched.m2.nic1.up()self.matched.m1.run("ping 192.168.1.2")- ctl = Controller()
- recipe_instance = HelloWorldRecipe()
- ctl.run(recipe_instance)
+This test requires that you have 2 test machines that are directly connected to +each other. +If you write this into a ``hello_world.py`` file you should now be able to +execute this script by running::
- python3 hello_world.py
+And you'll end up receiving an error about being unable to find a match in your +configured pools, since we didn't configure any yet, this is quite expected. But +running this script did take care of creating a default configuration file and +directory where we'll now be able to create our machine pool.
+Creating a simple machine pool +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+The default location for the Controller config file is ``~/.lnst/lnst-ctl.conf``. +At this point in time, you don't need to change anything inside this file.
+At the same time, the default location for a machine pool is ``~/.lnst/pool/``, +to create a pool you'll need to put XML files that describe your test machines +where the ``lnst-slave`` application is running, and how they're connected. You +need to create one file per test machine, so to satisfy the +**HelloWorldRecipe** requirements, we need to create two files:
+.. code-block:: bash
- touch ~/.lnst/pool/test_machine1.xml
- touch ~/.lnst/pool/test_machine2.xml
+For the contents of the files you can use the following template:
+.. code-block:: xml
<slavemachine>
<params><param name="hostname" value="HOSTNAME"/><param name="rpc_port" value="9999"/></params><interfaces><eth label="A" id="1"><params><param name="hwaddr" value="MAC_ADDRESS"/></params></eth></interfaces></slavemachine>
+You'll need to edit the template and replace the **HOSTNAME** and +**MAC_ADDRESS** strings with values that correspond to the hostname which the +controller can use to connet to the slave, and the mac address of a network +interface usable for testing. This **MUST** be a different interface than the +one used for the Controller-Slave connection, as it's configuration will change +during test execution, the Controller-Slave connection would break if you used +the same interface.
+After creating your pool, you should now be able to run the ``hello_world.py`` +script successfully and receive back some logs about what happened. diff --git a/docs/source/recipe_packages.rst b/docs/source/recipe_packages.rst new file mode 100644 index 0000000..ccd6269 --- /dev/null +++ b/docs/source/recipe_packages.rst @@ -0,0 +1,7 @@ +Supported Recipe packages +=========================
+.. toctree::
- :maxdepth: 2
- enrt_recipes
diff --git a/docs/source/specific_scenarios.rst b/docs/source/specific_scenarios.rst new file mode 100644 index 0000000..2f13d02 --- /dev/null +++ b/docs/source/specific_scenarios.rst @@ -0,0 +1,5 @@ +Specific ENRT scenarios +=======================
+.. toctree::
- SimpleNetworkRecipe
diff --git a/lnst/Common/Parameters.py b/lnst/Common/Parameters.py index e56c32d..f73fd3a 100644 --- a/lnst/Common/Parameters.py +++ b/lnst/Common/Parameters.py @@ -4,10 +4,6 @@ This module defines the Param class, it's type specific derivatives Param instances. This can be used by a BaseRecipe class to specify optional/mandatory parameters for the entire test, or by HostReq and DeviceReq classes to define specific parameters needed for the matching algorithm.
-Copyright 2017 Red Hat, Inc. -Licensed under the GNU General Public License, version 2 as -published by the Free Software Foundation; see COPYING for details. """
__author__ = """ @@ -24,12 +20,31 @@ class ParamError(LnstError): pass
class Param(object):
"""Base Parameter class
Can beused to define your own specific parameter type. Param derived classes
serve as *type checkers* to enable earlier failure of the recipe.
:param mandatory: if `True`, marks the parameter as mandatory
:type mandatory: bool
:param default: the default value for the parameter, is also type-checked,
immediately at Param object creation""" def __init__(self, mandatory=False, **kwargs): self.mandatory = mandatory if "default" in kwargs: self.default = self.type_check(kwargs["default"])
def type_check(self, value):
"""The type check methodImplementation depends on the specific Param derived class.:return: the type checked or converted value:raises: :any:`ParamError` if the type check or conversion is invalid""" return valueclass IntParam(Param): diff --git a/lnst/Controller/Controller.py b/lnst/Controller/Controller.py index ace0caf..d992b27 100644 --- a/lnst/Controller/Controller.py +++ b/lnst/Controller/Controller.py @@ -2,10 +2,6 @@ This module defines the Controller class that brings together individual implementation parts of an LNST Controller. When instantiated, it allows the tester to configure and run his own recipes with the LNST 'infrastructure'.
-Copyright 2017 Red Hat, Inc. -Licensed under the GNU General Public License, version 2 as -published by the Free Software Foundation; see COPYING for details. """
__author__ = """ @@ -33,34 +29,59 @@ from lnst.Controller.Recipe import BaseRecipe, RecipeRun from lnst.Controller.RecipeControl import RecipeControl
class Controller(object):
- """The LNST Controller class
- Most importantly allows the tester to run instantiated Recipe tests using
- the LNST infrastructure.
- Can be configured with custom implementation of several objects used for
- setting up the infrastructure.
- """Allows to run LNST Recipe instances
- This is the main mechanism that allows users to create their own executable
- test scripts that execute LNST Recipes.
- The Controller class implementation provides the most common default values
- for various parameters that can significantly change the way that Recipes
- are executed. This includes custom implementations of classes that are used
- for setting up the testing infrastructure such as the PoolManager or the
- MachineMapper.
- :param poolMgr:
class that implements the:py:class:`lnst.Controller.SlavePoolManager.SlavePoolManager` interfacewill be instantiated by the Controller to provide the mapper with poolsavailable for matching, also handles the creation of:py:class:`Machine` objects (internal LNST class used to access theslave hosts)- :type poolMgr: :py:class:`lnst.Controller.MachineMapper.MachineMapper`
^^^^ Should be lnst.Controller.SlavePoolManager.SlavePoolManager.
Also I'd add a note that a default PoolManager is used if not specified. Same goes for the MachineMapper below.
:param mapper:
class that implements the:py:class:`lnst.Controller.MachineMapper.MachineMapper` interface willbe instantiated by the Controller to match Recipe requirements to theavailable pools:type mapper: :py:class:`lnst.Controller.MachineMapper.MachineMapper`
:param config:
optional LNST configuration object, if None the Controller willload it's own configuration from default paths:type config: :py:class:`lnst.Controller.Config.CtlConfig`
:param pools:
a list of pool names to restrict the used pool directories:type pools: List[str]
:param pool_checks:
if False, will disable checking the online status of Slaves:type pool_checks: boolean (default True)
:param debug:
sets the debug level of LNST:type debug: integer (default 0)
Example::
lnst_controller = Controller()recipe_instance = MyRecipe(test_parameter=123)lnst_controller.run(recipe_instance)"""
def __init__(self, poolMgr=SlavePoolManager, mapper=MachineMapper, config=None, pools=[], pool_checks=True, debug=0):
"""Args:poolMgr -- class that implements the SlavePoolManager interfacewill be instantiated by the Controller to provide the mapperwith pools available for matching, also handles the creationof Machine objects (internal LNST class used to access theslave hosts)mapper -- class that implements the MachineMapper interfacewill be instantiated by the Controller to match Reciperequirements to the available poolsconfig -- optional LNST configuration object, if None theController will load it's own configuration from default pathspools -- a list of pool names to restrict the used pool directoriespool_checks -- boolean (default True), if False will disablechecking online status of Slavesdebug -- integer (default 0), sets debug level of LNST""" self._config = self._load_ctl_config(config) config = self._config@@ -96,13 +117,17 @@ class Controller(object): def run(self, recipe, **kwargs): """Execute the provided Recipe
This method takes care of both finding a Slave hosts matching the Reciperequirements, provisioning them and calling the 'test' method of the
This method takes care of both finding Slave hosts matching the Reciperequirements, provisioning them and calling the *test* method of the Recipe object with proper references to the mapped Hosts
Args:recipe -- an instantiated Recipe object (isinstance BaseRecipe)kwargs -- optional keyword arguments passed to the configured Mapper
:param recipe:an instantiated Recipe object:type recipe: :py:class:`lnst.Controller.Recipe.BaseRecipe`:param kwargs:optional keyword arguments passed to the configured Mapper:type kwargs: Dict[str, Any] """ if not isinstance(recipe, BaseRecipe): raise ControllerError("recipe argument must be a BaseRecipe instance.")@@ -134,7 +159,6 @@ class Controller(object): finally: self._cleanup_slaves()
- def _map_match(self, match, requested, recipe): self._machines = {} self._hosts = Hosts()
diff --git a/lnst/Controller/Recipe.py b/lnst/Controller/Recipe.py index 42f4e3d..26e0737 100644 --- a/lnst/Controller/Recipe.py +++ b/lnst/Controller/Recipe.py @@ -1,9 +1,5 @@ """ Module implementing the BaseRecipe class.
-Copyright 2017 Red Hat, Inc. -Licensed under the GNU General Public License, version 2 as -published by the Free Software Foundation; see COPYING for details. """
__author__ = """ @@ -23,27 +19,30 @@ class RecipeError(ControllerError): pass
class BaseRecipe(object):
- """BaseRecipe class
"""Base class for LNST Recipe definition.
Every LNST Recipe written by testers should be inherited from this class. An LNST Recipe is composed of several parts:
- Requirements definition - you define recipe requirements in a derived
class by defining class attributes of the HostReq type. You can furtherspecify Ethernet Device requirements by defining DeviceReq attributesof the HostReq object.Example:m1 = HostReq(arch="x86_64")m1.eth0 = DeviceReq(driver="ixgbe")
- Parameter definition (optional) - you can define paramaters of your Recipe
by defining class attributes of the Param type (or inherited). Theseparameters can then be accessed from the test() method to change it'sbehaviour. Parameter validity (type) is checked during theinstantiation of the Recipe object by the base __init__ method.You can define your own __init__ method to implement more complexParameter checking if needed, but you MUST call the base __init__method first.Example:MyRecipe(BaseRecipe):
class by defining class attributes of the HostReq type. You can furtherspecify Ethernet Device requirements by defining DeviceReq attributes ofthe :py:class:`lnst.Controller.Requirements.HostReq` object. Example::class MyRecipe(BaseRecipe):m1 = HostReq(arch="x86_64")m1.eth0 = DeviceReq(driver="ixgbe")
- Parameter definition (optional) - you can define paramaters of your
Recipe by defining class attributes of the :any:`Param` type (orinherited). These parameters can then be accessed from the test() methodto change it's behaviour. Parameter validity (type) is checked during theinstantiation of the Recipe object by the base __init__ method. You candefine your own __init__ method to implement more complex Parameterchecking if needed, but you MUST call the base __init__ method first.Example::class MyRecipe(BaseRecipe): int_param = IntParam(mandatory=True) optional_param = IntParam()@@ -55,18 +54,24 @@ class BaseRecipe(object): MyRecipe(int_param = 2, optional_param = 3)
* Test definition - this is done by defining the test() method, in this
method the tester has direct access to mapped LNST slave Hosts, canmanipulate them and implement his tests.- Attributes:
matched -- when running the Recipe the Controller will fill thisattribute with a Hosts object after the Mapper finds suitable slavehosts.req -- instantiated Requirements object, you can optionally change theRecipe requirements through this object during runtime (e.g.variable number of hosts or devices of a host based on a Parameter)params -- instantiated Parameters object, can be used to access thecalculated parameters during Recipe initialization/execution
method the tester has direct access to mapped LNST slave Hosts, canmanipulate them and implement his tests.- :ivar matched:
When running the Recipe the Controller will fill this attribute with aHosts object after the Mapper finds suitable slave hosts.- :type matched: :py:class:`lnst.Controller.Host.Hosts`
- :ivar req:
Instantiated Requirements object, you can optionally change the Reciperequirements through this object during runtime (e.g. variable numberof hosts or devices of a host based on a Parameter)- :type req: :py:class:`lnst.Controller.Requirements._Requirements`
- :ivar params:
Instantiated Parameters object, can be used to access the calculatedparameters during Recipe initialization/execution- :type params: :py:class:`lnst.Common.Parameters.Parameters` """ def __init__(self, **kwargs): """
diff --git a/lnst/Controller/Requirements.py b/lnst/Controller/Requirements.py index 342454f..ed61996 100644 --- a/lnst/Controller/Requirements.py +++ b/lnst/Controller/Requirements.py @@ -1,19 +1,19 @@ """ -This module defines the DeviceReq and HostReq classes, which can be used to +This module defines the :py:class:`lnst.Controller.Requirements.DeviceReq` and +:py:class:`lnst.Controller.Requirements.HostReq` classes, which can be used to create a global description of Requirements for a network test. You can use -these to define class attributes of a BaseRecipe derived class to specify +these to define class attributes of a +:py:class:`lnst.Controller.Recipe.BaseRecipe` derived class to specify "general" requirements for that Recipe, or you can add them to an instance of a Recipe derived class based on it's parameters to define requirements "specific" for that single test run.
-The module also specifies a Requirements class which serves as a container for -HostReq objects, while HostReq classes also serve as containers for DeviceReq -objects. The object tree created this way is translated to a dictionary used by -the internal LNST matching algorithm against available machines.
-Copyright 2017 Red Hat, Inc. -Licensed under the GNU General Public License, version 2 as -published by the Free Software Foundation; see COPYING for details. +The module also specifies a +:py:class:`lnst.Controller.Requirements._Requirements` class (currently for +internal use only) which serves as a container for HostReq objects, while +HostReq classes also serve as containers for DeviceReq objects. The object tree +created this way is translated to a dictionary used by the internal LNST +matching algorithm against available machines. """
__author__ = """ @@ -63,15 +63,22 @@ class HostReq(BaseReq):
To define a Host requirement you assign a HostReq instance to a class attribute of a BaseRecipe derived class.
- Example:
- class MyRecipe(BaseRecipe):
m1 = HostReq()- Args:
kwargs -- any other arguments will be treated as arbitrary stringparameters that will be matched to parameters of Slave machineswhich can define their parameter values based on the implementationof the SlaveMachineParser
- :param kwargs:
any argument will be treated as arbitrary string parameters that willbe matched to parameters of Slave machines which can define theirparameter values based on the implementation of the SlaveMachineParserA special case is the use of a:py:mod:`lnst.Controller.Requirements.RecipeParam` instance as value.This is used to link to a value provided as a Parameter to the Recipe.- :type kwargs: Dict[str, Any]
- Example::
class MyRecipe(BaseRecipe):m1 = HostReq()m2 = HostReq(architecture="x86_64")@@ -93,21 +100,32 @@ class HostReq(BaseReq): return res
class DeviceReq(BaseReq):
- """Specifies an Ethernet Device requirement
"""Specifies a static test network Device requirement
This will be used to find a matching test machine in the configured slave
machine pools, specifically this will be used to match against a test
device on a slave machine that is "statically" present on the machine. In
other words an actual REAL network device connected to a network usable for
testing.
To define a Device requirement you assign a DeviceReq instance to a HostReq instance in a BaseRecipe derived class.
- Example:
- class MyRecipe(BaseRecipe):
m1 = HostReq()m1.eth0 = DeviceReq(label="net1")- Args:
label -- string value indicating the network the Device is connected tokwargs -- any other arguments will be treated as arbitrary stringparameters that will be matched to parameters of Slave machineswhich can define their parameter values based on the implementationof the SlaveMachineParser
- :param label:
string value indicating the network the Device is connected to- :type label: string
- :param kwargs:
any other arguments will be treated as arbitrary string parameters thatwill be matched to parameters of Slave machines which can define theirparameter values based on the implementation of the SlaveMachineParser- :type kwargs: Dict[str, Any]
- Example::
class MyRecipe(BaseRecipe):m1 = HostReq()m1.eth0 = DeviceReq(label="net1")diff --git a/lnst/Controller/__init__.py b/lnst/Controller/__init__.py index 106bb91..cb0392d 100644 --- a/lnst/Controller/__init__.py +++ b/lnst/Controller/__init__.py @@ -1,3 +1,10 @@ +""" +Controller package +==================
+This package exposes public facing APIs that can be used to create Recipes, as +well as executable test scripts. +""" from lnst.Controller.Controller import Controller from lnst.Controller.Recipe import BaseRecipe from lnst.Controller.Requirements import HostReq, DeviceReq, RecipeParam diff --git a/lnst/Recipes/ENRT/BaseEnrtRecipe.py b/lnst/Recipes/ENRT/BaseEnrtRecipe.py index a90c837..d139546 100644 --- a/lnst/Recipes/ENRT/BaseEnrtRecipe.py +++ b/lnst/Recipes/ENRT/BaseEnrtRecipe.py @@ -2,7 +2,7 @@ import pprint from contextlib import contextmanager
from lnst.Common.LnstError import LnstError -from lnst.Common.Parameters import Param, IntParam, StrParam, BoolParam, ListParam +from lnst.Common.Parameters import Param, IntParam, StrParam, BoolParam, ListParam, FloatParam from lnst.Common.IpAddress import AF_INET, AF_INET6
from lnst.Recipes.ENRT.ConfigMixins.BaseSubConfigMixin import BaseSubConfigMixin @@ -16,10 +16,150 @@ from lnst.RecipeCommon.Perf.Measurements import StatCPUMeasurement from lnst.RecipeCommon.Perf.Evaluators import NonzeroFlowEvaluator
class EnrtConfiguration(object):
- """Container object for configuration
- Intentionally left empty as it is intended to be used as a container to
- store any values relevant to configuration being applied during the lifetime
- of the Recipe.
- """ pass
class BaseEnrtRecipe(BaseSubConfigMixin, PingTestAndEvaluate, PerfRecipe):
- #common requirements parameters
"""Base Recipe class for the ENRT recipe package
This class defines the shared *test* method defining the common test
procedure in a very generic way. This common test procedure involves a
single main *test_wide* configuration that is different for every specific
scenario. After the main configuration there is usually a loop of several
minor *sub* configrations types that can take different values to slightly
change the tested use cases.
Finally, for each combination of a **test_wide** + **sub** configuration we
do a several ping connection test and several performance measurement tests.
**test_wide** and **sub** configurations are implemented with **context
manager** methods which ensure that if any exceptions are raised (for
example because of a bug in the recipe) that deconfiguration is called.
Both **test_wide** and **sub** configurations are to be implemented in
different classes, the BaseEnrtRecipe class only defines the common API and
the base versions of the relevant methods.
Test wide configuration is implemented via the following methods:
- :any:`test_wide_configuration`
- :any:`test_wide_deconfiguration`
- :any:`generate_test_wide_description`
Sub configurations are **mixed in** to classes defining the specific
scenario that is being tested. Various sub configurations are implemented as
individual Python **Mixin** classes in the
:any:`ConfigMixins<config_mixins>` package. These make use of Pythons
collaborative inheritance by calling the `super` function in a specific way.
The "machinery" for that defined in the :any:`BaseSubConfigMixin` class and
used in this class from the `test` method loop that generates the combined
Sub configuration and the `_sub_context` context manager method that takes
care of applying and removing the configuration.
:param driver:
The driver parameter is used to modify the hw network requirements,specifically to request Devices using the specified driver. This iscommon enough in the Enrt recipes that it can be part of the Base class.:type driver: :any:`StrParam` (default "ixgbe")
:param ip_versions:
Parameter that determines which IP protocol versions will be tested.:type ip_versions: Tuple[Str] (default ("ipv4", "ipv6"))
:param ping_parallel:
Parameter used by the :any:`generate_ping_configurations` generator.Tells the generator method to create :any:`PingConf` objects that willbe run in parallel.:type ping_parallel: :any:`BoolParam` (default False)
:param ping_bidirect:
Parameter used by the :any:`generate_ping_configurations` generator.Tells the generator method to create :any:`PingConf` objects for bothdirections between the ping endpoints.:type ping_bidirect: :any:`BoolParam` (default False)
:param ping_count:
Parameter used by the :any:`generate_ping_configurations` generator.Tells the generator how many pings should be sent for each ping test.:type ping_count: :any:`IntParam` (default 100)
:param ping_interval:
Parameter used by the :any:`generate_ping_configurations` generator.Tells the generator how fast should the pings be sent in each ping test.:type ping_interval: :any:`FloatParam` (default 0.2)
:param ping_psize:
Parameter used by the :any:`generate_ping_configurations` generator.Tells the generator how big should the pings packets be in each pingtest.:type ping_psize: :any:`IntParam` (default None)
:param perf_tests:
Parameter used by the :any:`generate_flow_combinations` generator.Tells the generator what types of network flow measurements to generateperf test configurations for.:type perf_tests: Tuple[str] (default ("tcp_stream", "udp_stream",
"sctp_stream")):param perf_tool_cpu:
Parameter used by the :any:`generate_flow_combinations` generator. Toindicate that the flow measurement should be pinned to a specific CPUcore.:type perf_tool_cpu: :any:`IntParam` (optional parameter)
:param perf_duration:
Parameter used by the :any:`generate_perf_configurations` generator. Tospecify the duration of the performance measurements, in seconds.:type perf_duration: :any:`IntParam` (default 60)
:param perf_iterations:
Parameter used by the :any:`generate_perf_configurations` generator. Tospecify how many times should each performance measurement be repeatedto generate cumulative results which can be statistically analyzed.:type perf_iterations: :any:`IntParam` (default 5)
:param perf_parallel_streams:
Parameter used by the :any:`generate_flow_combinations` generator. Tospecify how many parallel streams of the same network flow should bemeasured at the same time.:type perf_parallel_streams: :any:`IntParam` (default 1)
:param perf_msg_sizes:
Parameter used by the :any:`generate_flow_combinations` generator. Tospecify what different message sizes (in bytes) used generated for thenetwork flow should be tested - each message size resulting in aseparate performance measurement.:type perf_msg_sizes: List[Int] (default [123])
:param perf_reverse:
Parameter used by the :any:`generate_flow_combinations` generator. Tospecify that both directions between the endpoints of a network flowshould be measured for the same test.:type perf_reverse: :any:`BoolParam` (default False)
:param net_perf_tool:
Specifies a network flow measurement class that accepts :any:`PerfFlow`objects and can be used to measure those specified flowsParameter used by the :any:`generate_perf_configurations` generator tocreate a PerfRecipeConf object.:type net_perf_tool: :any:`BaseFlowMeasurement` (default
IperfFlowMeasurement):param cpu_perf_tool:
Specifies a cpu measurement class that can be used to measure CPUutilization on specified hosts.Parameter used by the :any:`generate_perf_configurations` generator tocreate a PerfRecipeConf object.:type cpu_perf_tool: :any:`BaseCPUMeasurement` (default StatCPUMeasurement)
"""
driver = StrParam(default="ixgbe")
#common test parameters
@@ -45,6 +185,20 @@ class BaseEnrtRecipe(BaseSubConfigMixin, PingTestAndEvaluate, PerfRecipe): cpu_perf_tool = Param(default=StatCPUMeasurement)
def test(self):
"""Main test loop shared by all the Enrt recipesThe test loop involves a single application of a **test_wide**configuration, then a loop over multiple **sub** configurations thatinvolves:* creating the combined sub configuration of all available SubConfigMixin classes via :any:`generate_sub_configurations`* applying the generated sub configuration via the :any:`_sub_context`context manager method* running tests* removing the current sub configuration via the :any:`_sub_context`context manager method""" with self._test_wide_context() as main_config: for sub_config in self.generate_sub_configurations(main_config): with self._sub_context(sub_config) as recipe_config:@@ -60,19 +214,90 @@ class BaseEnrtRecipe(BaseSubConfigMixin, PingTestAndEvaluate, PerfRecipe): self.test_wide_deconfiguration(config)
def test_wide_configuration(self):
"""Creates an empty :any:`EnrtConfiguration` objectThis is again used in potential collaborative inheritance design thatmay potentially be useful for Enrt recipes. Derived classes will eachindividually add their own values to the instance created here. This waythe complete test wide configuration is tracked in a single object.:return: returns a config object that tracks the applied configurationthat can be used during testing to inspect the current state andmake test decisions based on it.:rtype: :any:`EnrtConfiguration`Example::class Derived:def test_wide_configuration():config = super().test_wide_configuration()# ... configure somethingconfig.something = what_was_configuredreturn config""" return EnrtConfiguration()def test_wide_deconfiguration(self, config):
"""Base deconfiguration method.In the base class this should maybe only check if there's any leftoverconfiguration and warn about it. In derived classes this can beoverriden to take care of deconfiguring what was configured in therespective test_wide_configuration method.Example::class Derived:def test_wide_deconfiguration(config):# ... deconfigure somethingdel config.something #cleanup trackingreturn super().test_wide_deconfiguration()""" #TODO check if anything is still applied and throw exception? returndef describe_test_wide_configuration(self, config):
"""Describes the current test wide configurationCreates a new result object that tracks a *successful* result, butcontains the description of the full test wide configuration applied byall the :any:`test_wide_configuration` methods in the class hierarchy.The description needs to be generated by the:any:`generate_test_wide_description` method. Additionally thedescription contains the state of all the parameters and their valuespassed to the recipe class instance during initialization.""" description = self.generate_test_wide_description(config) self.add_result(True, "Summary of used Recipe parameters:\n{}".format( pprint.pformat(self.params._to_dict()))) self.add_result(True, "\n".join(description))def generate_test_wide_description(self, config):
"""Generates the test wide configuration descriptionAnother class inteded to be used with the collaborative version of the`super` method to cumulatively desribe the full test wide configurationthat was applied through multiple classes.The base class version of this method creates the initial list ofstrings containing just the header line. Each string added to this listwill later be printed on its own line.:return: list of strings, each representing a single line:rtype: List[str]Example::class Derived:def generate_sub_configuration_description(config):desc = super().generate_sub_configuration_description(config)desc.append("Configured something: {}".format(config.something))return desc""" return [ "Testwide configuration for recipe {} description:".format( self.__class__.__name__@@ -93,20 +318,47 @@ class BaseEnrtRecipe(BaseSubConfigMixin, PingTestAndEvaluate, PerfRecipe): self.add_result(True, "\n".join(description))
def do_tests(self, recipe_config):
"""Entry point for actual testsThe common scenario is to do ping and performance tests, however themethod can be overriden to add more tests if needed.""" self.do_ping_tests(recipe_config) self.do_perf_tests(recipe_config)def do_ping_tests(self, recipe_config):
"""Ping testing loopLoops over all various ping configurations generated by the:any:`generate_ping_configurations` method, then uses the PingRecipemethods to execute, report and evaluate the results.""" for ping_config in self.generate_ping_configurations(recipe_config): result = self.ping_test(ping_config) self.ping_evaluate_and_report(ping_config, result)def do_perf_tests(self, recipe_config):
"""Performance testing loopLoops over all various perf configurations generated by the:any:`generate_perf_configurations` method, then uses the PerfRecipemethods to execute, report and evaluate the results.""" for perf_config in self.generate_perf_configurations(recipe_config): result = self.perf_test(perf_config) self.perf_report_and_evaluate(result)def generate_ping_configurations(self, config):
"""Base ping test configuration generatorThe generator loops over all endpoint pairs to test ping between(generated by the :any:`generate_ping_endpoints` method) then over allthe selected :any:`ip_versions` and finally over all the IP addressesthat fit those criteria.:return: list of Ping configurations to test in parallel:rtype: List[:any:`PingConf`]""" for endpoint1, endpoint2 in self.generate_ping_endpoints(config): for ipv in self.params.ip_versions: ip_filter = {}@@ -144,9 +396,30 @@ class BaseEnrtRecipe(BaseSubConfigMixin, PingTestAndEvaluate, PerfRecipe): yield ping_conf_list
def generate_ping_endpoints(self, config):
"""Generator for ping endpointsTo be overriden by a derived class.:return: list of device pairs:rtype: List[Tuple[:any:`Device`, :any:`Device`]]""" return []def generate_perf_configurations(self, config):
"""Base perf test configuration generatorThe generator loops over all flow combinations to measure performancefor (generated by the :any:`generate_flow_combinations` method). Inaddition to that during each flow combination measurement we add CPUutilization measurement to run on the background.Finally for each generated perf test configuration we registermeasurement evaluators based on the :any:`cpu_perf_evaluators` and:any:`net_perf_evaluators` properties.:return: list of Perf test configurations:rtype: List[:any:`PerfRecipeConf`]""" for flows in self.generate_flow_combinations(config): perf_recipe_conf=dict( recipe_config=config,@@ -183,6 +456,18 @@ class BaseEnrtRecipe(BaseSubConfigMixin, PingTestAndEvaluate, PerfRecipe): yield perf_conf
def generate_flow_combinations(self, config):
"""Base flow combination generatorThe generator loops over all endpoint pairs to test performance between(generated by the :any:`generate_perf_endpoints` method) then over allthe selected :any:`ip_versions` and uses the first IP address fittingthese criteria. Then the generator loops over the selected performancetests as selected via :any:`perf_tests`, then message sizes from:any:`msg_sizes`.:return: list of Flow combinations to measure in parallel:rtype: List[:any:`PerfFlow`]""" for client_nic, server_nic in self.generate_perf_endpoints(config): for ipv in self.params.ip_versions: if ipv == "ipv4":@@ -213,14 +498,37 @@ class BaseEnrtRecipe(BaseSubConfigMixin, PingTestAndEvaluate, PerfRecipe): yield [reverse_flow]
def generate_perf_endpoints(self, config):
"""Generator for perf endpointsTo be overriden by a derived class.:return: list of device pairs:rtype: List[Tuple[:any:`Device`, :any:`Device`]]""" return []@property def cpu_perf_evaluators(self):
"""CPU measurement evaluatorsTo be overriden by a derived class. Returns the list of evaluators touse for CPU utilization measurement evaluation.:return: a list of cpu evaluator objects:rtype: List[BaseEvaluator]""" return []@property def net_perf_evaluators(self):
"""Network flow measurement evaluatorsTo be overriden bby a derived class. Returns the list of evaluators touse for Network flow measurement evaluation.:return: a list of flow evaluator objects:rtype: List[BaseEvaluator]""" return [NonzeroFlowEvaluator()]def _create_reverse_flow(self, flow):
diff --git a/lnst/Recipes/ENRT/ConfigMixins/README.md b/lnst/Recipes/ENRT/ConfigMixins/README.md new file mode 100644 index 0000000..0472be0 --- /dev/null +++ b/lnst/Recipes/ENRT/ConfigMixins/README.md @@ -0,0 +1,3 @@ +# SubConfig Mixins
+TODO diff --git a/lnst/Recipes/ENRT/ConfigMixins/__init__.py b/lnst/Recipes/ENRT/ConfigMixins/__init__.py index e69de29..7ae6450 100644 --- a/lnst/Recipes/ENRT/ConfigMixins/__init__.py +++ b/lnst/Recipes/ENRT/ConfigMixins/__init__.py @@ -0,0 +1,9 @@ +""" +ENRT Config Mixins +==================
+.. autoclass:: lnst.Recipes.ENRT.ConfigMixins.BaseSubConfigMixin.BaseSubConfigMixin
+"""
+from lnst.Recipes.ENRT.ConfigMixins.BaseSubConfigMixin import BaseSubConfigMixin diff --git a/lnst/Recipes/ENRT/README.md b/lnst/Recipes/ENRT/README.md new file mode 100644 index 0000000..3500cdc --- /dev/null +++ b/lnst/Recipes/ENRT/README.md @@ -0,0 +1,47 @@ +# ENRT Recipes
+ENRT stands for Early Network Regression Testing, which was the name of the +project a couple of years back when we started developing this set of tests +when we were still using the Legacy LNST API.
+This package aims to reimplement the same set of tests, using the new LNST-next +APIs, while fully utilizing Python to address the largest problems with the old +implementation: +* large amounts of code duplication +* copy paste errors caused by code duplication +* very hard to maintain the test set and fix bugs +* very hard to add new tests due to limitations of the old LNST Framework
+With that in mind, the main goals for this reimplementation were as follows: +* reduce code duplication as much as possible by utilizing class inheritance +* separate individual types of configurations into smaller Mixin classes that
- can be mixed and matched based on what a specific Test scenario requires
+* writing new test scenarios should be very quick because it should be possible
- to reuse most of the functionality already defined
+* it should be possible to very easily extend the recipes with new features such
- as more types of parallel or sequential measurements, more types of
- Evaluations for different types of measurements, or to be able to switch out
- and use different measurement tools
+The resulting design is split into several parts that interact with each other +in a specific way:
+* [BaseEnrtRecipe](BaseEnrtRecipe.py) serves as the base class for all the
- specific test scenarios we want to test. Defines the common test loop and the
- default implementation for configuration generators used in this common test
- loop.
+* [ConfigMixins](ConfigMixins/README.md) package contains the various types of
- "SubConfig" mixin classes, these are classes that implement some form of
- configuration on test machine(s) which can be reused between mutliple recipes,
- but can often times be looped over to try different variations of the
- configuration. A good example is configuration of hardware offloads, it's
- relevant to test it for many different test scenarios, but only some
- combinations of offloads make sense based on the scenario and we often time
- want to test more than one combination
+* Specific test scenario implementation that defines the requirements and the
- main configuration for the specific scenario that we want to test. It also
- defines the combination of various "SubConfig" configurations that we want to
- include by adding them to it's inheritance tree. If required the recipe can
- also override any of the default functionality defined by it's parent classes,
- a good example could be the generator method for creating various
- configurations for flow performance measurement.
diff --git a/lnst/Recipes/ENRT/SimpleNetworkRecipe.py b/lnst/Recipes/ENRT/SimpleNetworkRecipe.py index 911a996..2c8e12b 100644 --- a/lnst/Recipes/ENRT/SimpleNetworkRecipe.py +++ b/lnst/Recipes/ENRT/SimpleNetworkRecipe.py @@ -12,6 +12,25 @@ from lnst.Recipes.ENRT.ConfigMixins.CommonHWSubConfigMixin import ( class SimpleNetworkRecipe( CommonHWSubConfigMixin, OffloadSubConfigMixin, BaseEnrtRecipe ):
- """
- This recipe implements Enrt testing for a simple network scenario that looks
- as follows
- .. code-block::
Though it's not required with the sphinx version >= 2.0, please specify no language used by: code-block:: none
+--------++------+ switch +-----+| +--------+ |+--+-+ +-+--++-|eth0|-+ +-|eth0|-+| +----+ | | +----+ || host1 | | host2 |+--------+ +--------+- All sub configurations are included via Mixin classes.
- The actual test machinery is implemented in the :any:`BaseEnrtRecipe` class.
- """ host1 = HostReq() host1.eth0 = DeviceReq(label="net1", driver=RecipeParam("driver"))
@@ -26,6 +45,14 @@ class SimpleNetworkRecipe( dict(gro="on", gso="on", tso="on", tx="on", rx="off")))
def test_wide_configuration(self):
"""Test wide configuration for this recipe involves just adding an IPv4 andIPv6 address to the matched eth0 nics on both hosts.host1.eth0 = 192.168.101.1/24 and fc00::1/64host2.eth0 = 192.168.101.2/24 and fc00::2/64""" host1, host2 = self.matched.host1, self.matched.host2 configuration = super().test_wide_configuration() configuration.test_wide_devices = []@@ -41,6 +68,9 @@ class SimpleNetworkRecipe( return configuration
def generate_test_wide_description(self, config):
"""Test wide description is extended with the configured addresses""" desc = super().generate_test_wide_description(config) desc += [ "Configured {}.{}.ips = {}".format(@@ -51,14 +81,33 @@ class SimpleNetworkRecipe( return desc
def test_wide_deconfiguration(self, config):
"" # overriding the parent docstring del config.test_wide_devices super().test_wide_deconfiguration(config)def generate_ping_endpoints(self, config):
"""The ping endpoints for this recipe are simply the two matched NICs:host1.eth0 and host2.eth0Returned as::[(self.matched.host1.eth0, self.matched.host2.eth0)]""" return [(self.matched.host1.eth0, self.matched.host2.eth0)]def generate_perf_endpoints(self, config):
"""The perf endpoints for this recipe are simply the two matched NICs:host1.eth0 and host2.eth0Returned as::[(self.matched.host1.eth0, self.matched.host2.eth0)]""" return [(self.matched.host1.eth0, self.matched.host2.eth0)]def wait_tentative_ips(self, devices):
diff --git a/lnst/Recipes/ENRT/__init__.py b/lnst/Recipes/ENRT/__init__.py index ea5aaf0..a591ef2 100644 --- a/lnst/Recipes/ENRT/__init__.py +++ b/lnst/Recipes/ENRT/__init__.py @@ -1,3 +1,55 @@ +''' +ENRT Recipes +------------
+ENRT stands for Early Network Regression Testing, which was the name of the +project a couple of years back when we started developing this set of tests +when we were still using the Legacy LNST API.
+This package aims to reimplement the same set of tests, using the new LNST-next +APIs, while fully utilizing Python to address the largest problems with the old +implementation:
+* large amounts of code duplication +* copy paste errors caused by code duplication +* very hard to maintain the test set and fix bugs +* very hard to add new tests due to limitations of the old LNST Framework
+With that in mind, the main goals for this reimplementation were as follows:
+* reduce code duplication as much as possible by utilizing class inheritance +* separate individual types of configurations into smaller Mixin classes that
- can be mixed and matched based on what a specific Test scenario requires
+* writing new test scenarios should be very quick because it should be possible
- to reuse most of the functionality already defined
+* it should be possible to very easily extend the recipes with new features
- such as more types of parallel or sequential measurements, more types of
- Evaluations for different types of measurements, or to be able to switch out
- and use different measurement tools
+The resulting design is split into several parts that interact with each other +in a specific way:
+* BaseEnrtRecipe serves as the base class for all the specific test scenarios
- we want to test. Defines the common test loop and the default implementation
- for configuration generators used in this common test loop.
+* ConfigMixins package contains the various types of "SubConfig" mixin classes,
- these are classes that implement some form of configuration on test
- machine(s) which can be reused between mutliple recipes, but can often times
- be looped over to try different variations of the configuration. A good
- example is configuration of hardware offloads, it's relevant to test it for
- many different test scenarios, but only some combinations of offloads make
- sense based on the scenario and we often time want to test more than one
- combination
+* Specific test scenario implementation that defines the requirements and the
- main configuration for the specific scenario that we want to test. It also
- defines the combination of various "SubConfig" configurations that we want to
- include by adding them to it's inheritance tree. If required the recipe can
- also override any of the default functionality defined by it's parent
- classes, a good example could be the generator method for creating various
- configurations for flow performance measurement.
+''' from lnst.Recipes.ENRT.SimpleNetworkRecipe import SimpleNetworkRecipe from lnst.Recipes.ENRT.BondRecipe import BondRecipe from lnst.Recipes.ENRT.DoubleBondRecipe import DoubleBondRecipe -- 2.25.1 _______________________________________________ LNST-developers mailing list -- lnst-developers@lists.fedorahosted.org To unsubscribe send an email to lnst-developers-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/lnst-developers@lists.fedorahos...
Additional comments, inline.
Mon, Mar 16, 2020 at 11:43:03AM CET, olichtne@redhat.com wrote:
From: Ondrej Lichtner olichtne@redhat.com
Creating the docs/ directory where we should work on creating documentation. The standard for Python projects is to use "Sphinx" to generate various formats (e.g. pdf, html or others) of documentation from ReStructuredText (rst) source. The source is either stored as ".rst" standalone files or as docstrings with Python source code.
This is the first version that should cover some of the major topics and exported APIs available for the tester. It is by no means complete yet but should be enough to get a feel for how the overall structure should look like.
Signed-off-by: Ondrej Lichtner olichtne@redhat.com
README.md | 25 +- docs/Makefile | 20 ++ docs/source/ControllerAPI.rst | 5 + docs/source/Parameters.rst | 5 + docs/source/RecipeAPI.rst | 5 + docs/source/RecipeRequirementsAPI.rst | 5 + docs/source/SimpleNetworkRecipe.rst | 6 + docs/source/TesterAPI.rst | 10 + docs/source/base_enrt_class.rst | 8 + docs/source/conf.py | 64 +++++ docs/source/config_mixins.rst | 3 + docs/source/enrt_recipes.rst | 12 + docs/source/index.rst | 25 ++ docs/source/installation.rst | 133 +++++++++ docs/source/recipe_packages.rst | 7 + docs/source/specific_scenarios.rst | 5 + lnst/Common/Parameters.py | 23 +- lnst/Controller/Controller.py | 92 +++--- lnst/Controller/Recipe.py | 71 ++--- lnst/Controller/Requirements.py | 80 ++++-- lnst/Controller/__init__.py | 7 + lnst/Recipes/ENRT/BaseEnrtRecipe.py | 312 ++++++++++++++++++++- lnst/Recipes/ENRT/ConfigMixins/README.md | 3 + lnst/Recipes/ENRT/ConfigMixins/__init__.py | 9 + lnst/Recipes/ENRT/README.md | 47 ++++ lnst/Recipes/ENRT/SimpleNetworkRecipe.py | 49 ++++ lnst/Recipes/ENRT/__init__.py | 52 ++++ 27 files changed, 965 insertions(+), 118 deletions(-) create mode 100644 docs/Makefile create mode 100644 docs/source/ControllerAPI.rst create mode 100644 docs/source/Parameters.rst create mode 100644 docs/source/RecipeAPI.rst create mode 100644 docs/source/RecipeRequirementsAPI.rst create mode 100644 docs/source/SimpleNetworkRecipe.rst create mode 100644 docs/source/TesterAPI.rst create mode 100644 docs/source/base_enrt_class.rst create mode 100644 docs/source/conf.py create mode 100644 docs/source/config_mixins.rst create mode 100644 docs/source/enrt_recipes.rst create mode 100644 docs/source/index.rst create mode 100644 docs/source/installation.rst create mode 100644 docs/source/recipe_packages.rst create mode 100644 docs/source/specific_scenarios.rst create mode 100644 lnst/Recipes/ENRT/ConfigMixins/README.md create mode 100644 lnst/Recipes/ENRT/README.md
diff --git a/README.md b/README.md index 256f79e..dec9aba 100644 --- a/README.md +++ b/README.md @@ -12,7 +12,7 @@ backwards compatibility are yet in place.
This also means that many of our documentation resources outlining how to write recipes on the wiki are also out of date. We'll soon start working on these but -please be paitent with us. +please be patient with us.
If you're interested in helping out we accept code contributions via Patches submitted to our mailing list lnst-developers@lists.fedorahosted.org. @@ -33,13 +33,13 @@ Internet Resources bellow).
## Install
-LNST can be installed using python's distutils. +Installation and a simple Hello world example is available at +[Installation](docs/source/installation.rst)
-```bash -su -./setup.py install -``` +## Documentation
+Documentation is available in the `docs/` directory, you can build it with +`make html` using *Sphinx*.
## Authors/Contributors
@@ -53,18 +53,15 @@ su
- Jiri Zupka (not active anymore)
- Radek Pazdera (not active anymore)
+## How to contact us
-## Internet Resources
-* Project Wiki: https://github.com/jpirko/lnst/wiki (currently out of date) -* Documentation: https://github.com/jpirko/lnst/wiki#learn (currently out of date) -* Git Source Tree: https://github.com/jpirko/lnst -* Mailing List: lnst-developers@lists.fedorahosted.org
+* Git Source Tree: https://github.com/jpirko/lnst +* Mailing List: lnst-developers@lists.fedorahosted.org +* IRC channel: #lnst @ freenode.net
## License
-**Copyright (C) 2011-2019 Red Hat, Inc.** +**Copyright (C) 2011-2020 Red Hat, Inc.**
LNST is distributed under GNU General Public License version 2. See the file "COPYING" in the source distribution for information on terms & conditions diff --git a/docs/Makefile b/docs/Makefile new file mode 100644 index 0000000..d0c3cbf --- /dev/null +++ b/docs/Makefile @@ -0,0 +1,20 @@ +# Minimal makefile for Sphinx documentation +#
+# You can set these variables from the command line, and also +# from the environment for the first two. +SPHINXOPTS ?= +SPHINXBUILD ?= sphinx-build +SOURCEDIR = source +BUILDDIR = build
+# Put it first so that "make" without argument is like "make help". +help:
- @$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
+.PHONY: help Makefile
+# Catch-all target: route all unknown targets to Sphinx using the new +# "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS). +%: Makefile
- @$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
diff --git a/docs/source/ControllerAPI.rst b/docs/source/ControllerAPI.rst new file mode 100644 index 0000000..f80d840 --- /dev/null +++ b/docs/source/ControllerAPI.rst @@ -0,0 +1,5 @@ +Controller API +^^^^^^^^^^^^^^
+.. automodule:: lnst.Controller.Controller
- :members: Controller
diff --git a/docs/source/Parameters.rst b/docs/source/Parameters.rst new file mode 100644 index 0000000..9646b01 --- /dev/null +++ b/docs/source/Parameters.rst @@ -0,0 +1,5 @@ +Parameters +^^^^^^^^^^
+.. automodule:: lnst.Common.Parameters
- :members:
diff --git a/docs/source/RecipeAPI.rst b/docs/source/RecipeAPI.rst new file mode 100644 index 0000000..32e7b4c --- /dev/null +++ b/docs/source/RecipeAPI.rst @@ -0,0 +1,5 @@ +Recipe API +^^^^^^^^^^
+.. automodule:: lnst.Controller.Recipe
- :members: BaseRecipe, RecipeRun
diff --git a/docs/source/RecipeRequirementsAPI.rst b/docs/source/RecipeRequirementsAPI.rst new file mode 100644 index 0000000..4cf1c59 --- /dev/null +++ b/docs/source/RecipeRequirementsAPI.rst @@ -0,0 +1,5 @@ +Recipe Requirements +^^^^^^^^^^^^^^^^^^^
+.. automodule:: lnst.Controller.Requirements
- :members: HostReq, DeviceReq
diff --git a/docs/source/SimpleNetworkRecipe.rst b/docs/source/SimpleNetworkRecipe.rst new file mode 100644 index 0000000..1f08feb --- /dev/null +++ b/docs/source/SimpleNetworkRecipe.rst @@ -0,0 +1,6 @@ +SimpleNetworkRecipe +^^^^^^^^^^^^^^^^^^^
+.. autoclass:: lnst.Recipes.ENRT.SimpleNetworkRecipe.SimpleNetworkRecipe
- :members:
- :show-inheritance:
diff --git a/docs/source/TesterAPI.rst b/docs/source/TesterAPI.rst new file mode 100644 index 0000000..9261171 --- /dev/null +++ b/docs/source/TesterAPI.rst @@ -0,0 +1,10 @@ +Test developer API +==================
+.. toctree::
- :maxdepth: 2
- RecipeRequirementsAPI
- RecipeAPI
- ControllerAPI
- Parameters
diff --git a/docs/source/base_enrt_class.rst b/docs/source/base_enrt_class.rst new file mode 100644 index 0000000..32dd6fd --- /dev/null +++ b/docs/source/base_enrt_class.rst @@ -0,0 +1,8 @@ +BaseEnrtRecipe class +====================
+.. autoclass:: lnst.Recipes.ENRT.BaseEnrtRecipe.BaseEnrtRecipe
- :members:
- :show-inheritance:
+.. autoclass:: lnst.Recipes.ENRT.BaseEnrtRecipe.EnrtConfiguration diff --git a/docs/source/conf.py b/docs/source/conf.py new file mode 100644 index 0000000..7c59490 --- /dev/null +++ b/docs/source/conf.py @@ -0,0 +1,64 @@ +# Configuration file for the Sphinx documentation builder. +# +# This file only contains a selection of the most common options. For a full +# list see the documentation: +# https://www.sphinx-doc.org/en/master/usage/configuration.html
+# -- Path setup --------------------------------------------------------------
+# If extensions (or modules to document with autodoc) are in another directory, +# add these directories to sys.path here. If the directory is relative to the +# documentation root, use os.path.abspath to make it absolute, like shown here. +# +# import os +# import sys +# sys.path.insert(0, os.path.abspath('.'))
+# -- Project information -----------------------------------------------------
+project = 'lnst' +copyright = '2020, Jiri Pirko' +author = 'Jiri Pirko'
+# -- General configuration ---------------------------------------------------
+# Add any Sphinx extension module names here, as strings. They can be +# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom +# ones. +extensions = [
- 'sphinx.ext.autodoc',
+]
+import sys +sys.path = ['../'] + sys.path
+# Add any paths that contain templates here, relative to this directory. +templates_path = ['_templates']
+# List of patterns, relative to source directory, that match files and +# directories to ignore when looking for source files. +# This pattern also affects html_static_path and html_extra_path. +exclude_patterns = []
+# -- Options for HTML output -------------------------------------------------
+# The theme to use for HTML and HTML Help pages. See the documentation for +# a list of builtin themes. +# +html_theme = 'classic' +html_theme_options = {
- #"body_min_width": "100%",
- "body_max_width": "100%",
+}
+# Add any paths that contain custom static files (such as style sheets) here, +# relative to this directory. They are copied after the builtin static files, +# so a file named "default.css" will overwrite the builtin "default.css". +html_static_path = ['_static']
+autodoc_default_options = {
- 'member-order': 'bysource',
+} diff --git a/docs/source/config_mixins.rst b/docs/source/config_mixins.rst new file mode 100644 index 0000000..66119b6 --- /dev/null +++ b/docs/source/config_mixins.rst @@ -0,0 +1,3 @@ +.. ENRT Config Mixins
+.. automodule:: lnst.Recipes.ENRT.ConfigMixins diff --git a/docs/source/enrt_recipes.rst b/docs/source/enrt_recipes.rst new file mode 100644 index 0000000..af8935a --- /dev/null +++ b/docs/source/enrt_recipes.rst @@ -0,0 +1,12 @@ +Early Network Regression Testing (ENRT) +=======================================
+.. automodule:: lnst.Recipes.ENRT
+.. toctree::
- :maxdepth: 2
- :caption: ENRT Components
- base_enrt_class
- config_mixins
- specific_scenarios
diff --git a/docs/source/index.rst b/docs/source/index.rst new file mode 100644 index 0000000..e5f2dd5 --- /dev/null +++ b/docs/source/index.rst @@ -0,0 +1,25 @@ +LNST - Linux Network Stack Test +===============================
+Linux Network Stack Test is a framework that supports development and execution +of automated and portable network tests which usually involve multiple test +systems. This repository contains the implementation of the library to write +:any:`Recipes<BaseRecipe>`, library functions to implement executable script +files which run your tests as well as the code for the "LNST Slave" +application.
+.. toctree::
- :maxdepth: 2
- :caption: General topics:
- installation
- TesterAPI
- recipe_packages
+Indices and tables +==================
+* :ref:`genindex` +* :ref:`modindex` +* :ref:`search` diff --git a/docs/source/installation.rst b/docs/source/installation.rst new file mode 100644 index 0000000..ede73a2 --- /dev/null +++ b/docs/source/installation.rst @@ -0,0 +1,133 @@ +Install LNST and Hello world +============================
+LNST is logically split into two separate application use cases:
+* Controller - something that controlls the execution of your :any:`Test
- Recipes<BaseRecipe>`
+* Slave - a server application running on all hosts available for testing,
- executes remote procedure calls from the Controller to either run tests or
- configure the test machine
+Codebases for both use cases are developed in this repository and as we +currently don't have a stable release yet, the recommended method of +installation involves the following steps:
+.. code-block:: bash
- git clone https://github.com/jpirko/lnst
- cd lnst
- pip3 install --requirements requirements.txt
- pip3 install .
+This installs both the Controller and the Slave code, and you'll need to run +this on all the test machines that you want to use as well as the machine which +you want to use as the Controller. Optionally a Controller and a Slave CAN run +on the same machine.
+You can start your Slave application immediatelly by running::
- lnst-slave
+Because the lnst-slave application takes care of network configuration, it +**requires** to be executed with root privileges. This is **A BIG SECURITY +RISK** so make sure you only run this application on test machines that are not +publicly accessible or don't contain any sensitive data.
+The Controller is a bit more complicated and requires you to:
+* create an executable test script +* create a slave machine pool
+Creating an executable "HelloWorld" test script +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+LNST currently doesn't come with a CLI application for the Controller, instead +we need to create an executable python script ourselves that takes care of +creating an instance of a Controller class and an instance of the Test Recipe +that we want to run and calling the Controller.run() method to execute it.
+A minimal "hello world" example of an executable test script looks like this:
+.. code-block:: python
- from lnst.Controller import Controller, HostReq, DeviceReq, BaseRecipe
- class HelloWorldRecipe(BaseRecipe):
machine1 = HostReq()machine1.nic1 = DeviceReq(label="net1")machine2 = HostReq()machine2.nic1 = DeviceReq(label="net1")def test(self):self.matched.m1.nic1.ip_add("192.168.1.1/24")self.matched.m1.nic1.up()self.matched.m2.nic1.ip_add("192.168.1.2/24")self.matched.m2.nic1.up()self.matched.m1.run("ping 192.168.1.2")- ctl = Controller()
- recipe_instance = HelloWorldRecipe()
- ctl.run(recipe_instance)
+This test requires that you have 2 test machines that are directly connected to +each other. +If you write this into a ``hello_world.py`` file you should now be able to +execute this script by running::
- python3 hello_world.py
+And you'll end up receiving an error about being unable to find a match in your +configured pools, since we didn't configure any yet, this is quite expected. But +running this script did take care of creating a default configuration file and +directory where we'll now be able to create our machine pool.
+Creating a simple machine pool +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+The default location for the Controller config file is ``~/.lnst/lnst-ctl.conf``. +At this point in time, you don't need to change anything inside this file.
+At the same time, the default location for a machine pool is ``~/.lnst/pool/``, +to create a pool you'll need to put XML files that describe your test machines +where the ``lnst-slave`` application is running, and how they're connected. You +need to create one file per test machine, so to satisfy the +**HelloWorldRecipe** requirements, we need to create two files:
+.. code-block:: bash
- touch ~/.lnst/pool/test_machine1.xml
- touch ~/.lnst/pool/test_machine2.xml
+For the contents of the files you can use the following template:
+.. code-block:: xml
<slavemachine>
<params><param name="hostname" value="HOSTNAME"/><param name="rpc_port" value="9999"/></params><interfaces><eth label="A" id="1"><params><param name="hwaddr" value="MAC_ADDRESS"/></params></eth></interfaces></slavemachine>
+You'll need to edit the template and replace the **HOSTNAME** and +**MAC_ADDRESS** strings with values that correspond to the hostname which the +controller can use to connet to the slave, and the mac address of a network +interface usable for testing. This **MUST** be a different interface than the +one used for the Controller-Slave connection, as it's configuration will change +during test execution, the Controller-Slave connection would break if you used +the same interface.
+After creating your pool, you should now be able to run the ``hello_world.py`` +script successfully and receive back some logs about what happened. diff --git a/docs/source/recipe_packages.rst b/docs/source/recipe_packages.rst new file mode 100644 index 0000000..ccd6269 --- /dev/null +++ b/docs/source/recipe_packages.rst @@ -0,0 +1,7 @@ +Supported Recipe packages +=========================
+.. toctree::
- :maxdepth: 2
- enrt_recipes
diff --git a/docs/source/specific_scenarios.rst b/docs/source/specific_scenarios.rst new file mode 100644 index 0000000..2f13d02 --- /dev/null +++ b/docs/source/specific_scenarios.rst @@ -0,0 +1,5 @@ +Specific ENRT scenarios +=======================
+.. toctree::
- SimpleNetworkRecipe
diff --git a/lnst/Common/Parameters.py b/lnst/Common/Parameters.py index e56c32d..f73fd3a 100644 --- a/lnst/Common/Parameters.py +++ b/lnst/Common/Parameters.py @@ -4,10 +4,6 @@ This module defines the Param class, it's type specific derivatives Param instances. This can be used by a BaseRecipe class to specify optional/mandatory parameters for the entire test, or by HostReq and DeviceReq classes to define specific parameters needed for the matching algorithm.
-Copyright 2017 Red Hat, Inc. -Licensed under the GNU General Public License, version 2 as -published by the Free Software Foundation; see COPYING for details. """
__author__ = """ @@ -24,12 +20,31 @@ class ParamError(LnstError): pass
class Param(object):
"""Base Parameter class
Can beused to define your own specific parameter type. Param derived classes
serve as *type checkers* to enable earlier failure of the recipe.
:param mandatory: if `True`, marks the parameter as mandatory
:type mandatory: bool
:param default: the default value for the parameter, is also type-checked,
immediately at Param object creation""" def __init__(self, mandatory=False, **kwargs): self.mandatory = mandatory if "default" in kwargs: self.default = self.type_check(kwargs["default"])
def type_check(self, value):
"""The type check methodImplementation depends on the specific Param derived class.:return: the type checked or converted value:raises: :any:`ParamError` if the type check or conversion is invalid""" return valueclass IntParam(Param): diff --git a/lnst/Controller/Controller.py b/lnst/Controller/Controller.py index ace0caf..d992b27 100644 --- a/lnst/Controller/Controller.py +++ b/lnst/Controller/Controller.py @@ -2,10 +2,6 @@ This module defines the Controller class that brings together individual implementation parts of an LNST Controller. When instantiated, it allows the tester to configure and run his own recipes with the LNST 'infrastructure'.
-Copyright 2017 Red Hat, Inc. -Licensed under the GNU General Public License, version 2 as -published by the Free Software Foundation; see COPYING for details. """
__author__ = """ @@ -33,34 +29,59 @@ from lnst.Controller.Recipe import BaseRecipe, RecipeRun from lnst.Controller.RecipeControl import RecipeControl
class Controller(object):
- """The LNST Controller class
- Most importantly allows the tester to run instantiated Recipe tests using
- the LNST infrastructure.
- Can be configured with custom implementation of several objects used for
- setting up the infrastructure.
"""Allows to run LNST Recipe instances
This is the main mechanism that allows users to create their own executable
test scripts that execute LNST Recipes.
The Controller class implementation provides the most common default values
for various parameters that can significantly change the way that Recipes
are executed. This includes custom implementations of classes that are used
for setting up the testing infrastructure such as the PoolManager or the
MachineMapper.
:param poolMgr:
class that implements the:py:class:`lnst.Controller.SlavePoolManager.SlavePoolManager` interfacewill be instantiated by the Controller to provide the mapper with poolsavailable for matching, also handles the creation of:py:class:`Machine` objects (internal LNST class used to access theslave hosts):type poolMgr: :py:class:`lnst.Controller.MachineMapper.MachineMapper`
:param mapper:
class that implements the:py:class:`lnst.Controller.MachineMapper.MachineMapper` interface willbe instantiated by the Controller to match Recipe requirements to theavailable pools:type mapper: :py:class:`lnst.Controller.MachineMapper.MachineMapper`
:param config:
optional LNST configuration object, if None the Controller willload it's own configuration from default paths:type config: :py:class:`lnst.Controller.Config.CtlConfig`
:param pools:
a list of pool names to restrict the used pool directories:type pools: List[str]
:param pool_checks:
if False, will disable checking the online status of Slaves:type pool_checks: boolean (default True)
:param debug:
sets the debug level of LNST:type debug: integer (default 0)
Example::
lnst_controller = Controller()recipe_instance = MyRecipe(test_parameter=123)lnst_controller.run(recipe_instance)"""
def __init__(self, poolMgr=SlavePoolManager, mapper=MachineMapper, config=None, pools=[], pool_checks=True, debug=0):
"""Args:poolMgr -- class that implements the SlavePoolManager interfacewill be instantiated by the Controller to provide the mapperwith pools available for matching, also handles the creationof Machine objects (internal LNST class used to access theslave hosts)mapper -- class that implements the MachineMapper interfacewill be instantiated by the Controller to match Reciperequirements to the available poolsconfig -- optional LNST configuration object, if None theController will load it's own configuration from default pathspools -- a list of pool names to restrict the used pool directoriespool_checks -- boolean (default True), if False will disablechecking online status of Slavesdebug -- integer (default 0), sets debug level of LNST""" self._config = self._load_ctl_config(config) config = self._config@@ -96,13 +117,17 @@ class Controller(object): def run(self, recipe, **kwargs): """Execute the provided Recipe
This method takes care of both finding a Slave hosts matching the Reciperequirements, provisioning them and calling the 'test' method of the
This method takes care of both finding Slave hosts matching the Reciperequirements, provisioning them and calling the *test* method of the Recipe object with proper references to the mapped Hosts
Args:recipe -- an instantiated Recipe object (isinstance BaseRecipe)kwargs -- optional keyword arguments passed to the configured Mapper
:param recipe:an instantiated Recipe object:type recipe: :py:class:`lnst.Controller.Recipe.BaseRecipe`:param kwargs:optional keyword arguments passed to the configured Mapper:type kwargs: Dict[str, Any] """ if not isinstance(recipe, BaseRecipe): raise ControllerError("recipe argument must be a BaseRecipe instance.")@@ -134,7 +159,6 @@ class Controller(object): finally: self._cleanup_slaves()
- def _map_match(self, match, requested, recipe): self._machines = {} self._hosts = Hosts()
diff --git a/lnst/Controller/Recipe.py b/lnst/Controller/Recipe.py index 42f4e3d..26e0737 100644 --- a/lnst/Controller/Recipe.py +++ b/lnst/Controller/Recipe.py @@ -1,9 +1,5 @@ """ Module implementing the BaseRecipe class.
-Copyright 2017 Red Hat, Inc. -Licensed under the GNU General Public License, version 2 as -published by the Free Software Foundation; see COPYING for details. """
__author__ = """ @@ -23,27 +19,30 @@ class RecipeError(ControllerError): pass
class BaseRecipe(object):
- """BaseRecipe class
"""Base class for LNST Recipe definition.
Every LNST Recipe written by testers should be inherited from this class. An LNST Recipe is composed of several parts:
- Requirements definition - you define recipe requirements in a derived
class by defining class attributes of the HostReq type. You can furtherspecify Ethernet Device requirements by defining DeviceReq attributesof the HostReq object.Example:m1 = HostReq(arch="x86_64")m1.eth0 = DeviceReq(driver="ixgbe")
- Parameter definition (optional) - you can define paramaters of your Recipe
by defining class attributes of the Param type (or inherited). Theseparameters can then be accessed from the test() method to change it'sbehaviour. Parameter validity (type) is checked during theinstantiation of the Recipe object by the base __init__ method.You can define your own __init__ method to implement more complexParameter checking if needed, but you MUST call the base __init__method first.Example:MyRecipe(BaseRecipe):
class by defining class attributes of the HostReq type. You can furtherspecify Ethernet Device requirements by defining DeviceReq attributes ofthe :py:class:`lnst.Controller.Requirements.HostReq` object. Example::class MyRecipe(BaseRecipe):m1 = HostReq(arch="x86_64")m1.eth0 = DeviceReq(driver="ixgbe")
- Parameter definition (optional) - you can define paramaters of your
Recipe by defining class attributes of the :any:`Param` type (orinherited). These parameters can then be accessed from the test() methodto change it's behaviour. Parameter validity (type) is checked during theinstantiation of the Recipe object by the base __init__ method. You candefine your own __init__ method to implement more complex Parameterchecking if needed, but you MUST call the base __init__ method first.Example::class MyRecipe(BaseRecipe): int_param = IntParam(mandatory=True) optional_param = IntParam()@@ -55,18 +54,24 @@ class BaseRecipe(object): MyRecipe(int_param = 2, optional_param = 3)
* Test definition - this is done by defining the test() method, in this
method the tester has direct access to mapped LNST slave Hosts, canmanipulate them and implement his tests.- Attributes:
matched -- when running the Recipe the Controller will fill thisattribute with a Hosts object after the Mapper finds suitable slavehosts.req -- instantiated Requirements object, you can optionally change theRecipe requirements through this object during runtime (e.g.variable number of hosts or devices of a host based on a Parameter)params -- instantiated Parameters object, can be used to access thecalculated parameters during Recipe initialization/execution
method the tester has direct access to mapped LNST slave Hosts, canmanipulate them and implement his tests.- :ivar matched:
When running the Recipe the Controller will fill this attribute with aHosts object after the Mapper finds suitable slave hosts.- :type matched: :py:class:`lnst.Controller.Host.Hosts`
- :ivar req:
Instantiated Requirements object, you can optionally change the Reciperequirements through this object during runtime (e.g. variable numberof hosts or devices of a host based on a Parameter)- :type req: :py:class:`lnst.Controller.Requirements._Requirements`
- :ivar params:
Instantiated Parameters object, can be used to access the calculatedparameters during Recipe initialization/execution- :type params: :py:class:`lnst.Common.Parameters.Parameters` """ def __init__(self, **kwargs): """
diff --git a/lnst/Controller/Requirements.py b/lnst/Controller/Requirements.py index 342454f..ed61996 100644 --- a/lnst/Controller/Requirements.py +++ b/lnst/Controller/Requirements.py @@ -1,19 +1,19 @@ """ -This module defines the DeviceReq and HostReq classes, which can be used to +This module defines the :py:class:`lnst.Controller.Requirements.DeviceReq` and +:py:class:`lnst.Controller.Requirements.HostReq` classes, which can be used to create a global description of Requirements for a network test. You can use -these to define class attributes of a BaseRecipe derived class to specify +these to define class attributes of a +:py:class:`lnst.Controller.Recipe.BaseRecipe` derived class to specify "general" requirements for that Recipe, or you can add them to an instance of a Recipe derived class based on it's parameters to define requirements "specific" for that single test run.
-The module also specifies a Requirements class which serves as a container for -HostReq objects, while HostReq classes also serve as containers for DeviceReq -objects. The object tree created this way is translated to a dictionary used by -the internal LNST matching algorithm against available machines.
-Copyright 2017 Red Hat, Inc. -Licensed under the GNU General Public License, version 2 as -published by the Free Software Foundation; see COPYING for details. +The module also specifies a +:py:class:`lnst.Controller.Requirements._Requirements` class (currently for +internal use only) which serves as a container for HostReq objects, while +HostReq classes also serve as containers for DeviceReq objects. The object tree +created this way is translated to a dictionary used by the internal LNST +matching algorithm against available machines. """
__author__ = """ @@ -63,15 +63,22 @@ class HostReq(BaseReq):
To define a Host requirement you assign a HostReq instance to a class attribute of a BaseRecipe derived class.
- Example:
- class MyRecipe(BaseRecipe):
m1 = HostReq()- Args:
kwargs -- any other arguments will be treated as arbitrary stringparameters that will be matched to parameters of Slave machineswhich can define their parameter values based on the implementationof the SlaveMachineParser
- :param kwargs:
any argument will be treated as arbitrary string parameters that willbe matched to parameters of Slave machines which can define theirparameter values based on the implementation of the SlaveMachineParserA special case is the use of a:py:mod:`lnst.Controller.Requirements.RecipeParam` instance as value.This is used to link to a value provided as a Parameter to the Recipe.- :type kwargs: Dict[str, Any]
- Example::
class MyRecipe(BaseRecipe):m1 = HostReq()m2 = HostReq(architecture="x86_64")@@ -93,21 +100,32 @@ class HostReq(BaseReq): return res
class DeviceReq(BaseReq):
- """Specifies an Ethernet Device requirement
"""Specifies a static test network Device requirement
This will be used to find a matching test machine in the configured slave
machine pools, specifically this will be used to match against a test
device on a slave machine that is "statically" present on the machine. In
other words an actual REAL network device connected to a network usable for
testing.
To define a Device requirement you assign a DeviceReq instance to a HostReq instance in a BaseRecipe derived class.
- Example:
- class MyRecipe(BaseRecipe):
m1 = HostReq()m1.eth0 = DeviceReq(label="net1")- Args:
label -- string value indicating the network the Device is connected tokwargs -- any other arguments will be treated as arbitrary stringparameters that will be matched to parameters of Slave machineswhich can define their parameter values based on the implementationof the SlaveMachineParser
- :param label:
string value indicating the network the Device is connected to- :type label: string
- :param kwargs:
any other arguments will be treated as arbitrary string parameters thatwill be matched to parameters of Slave machines which can define theirparameter values based on the implementation of the SlaveMachineParser- :type kwargs: Dict[str, Any]
- Example::
class MyRecipe(BaseRecipe):m1 = HostReq()m1.eth0 = DeviceReq(label="net1")diff --git a/lnst/Controller/__init__.py b/lnst/Controller/__init__.py index 106bb91..cb0392d 100644 --- a/lnst/Controller/__init__.py +++ b/lnst/Controller/__init__.py @@ -1,3 +1,10 @@ +""" +Controller package +==================
+This package exposes public facing APIs that can be used to create Recipes, as +well as executable test scripts. +""" from lnst.Controller.Controller import Controller from lnst.Controller.Recipe import BaseRecipe from lnst.Controller.Requirements import HostReq, DeviceReq, RecipeParam diff --git a/lnst/Recipes/ENRT/BaseEnrtRecipe.py b/lnst/Recipes/ENRT/BaseEnrtRecipe.py index a90c837..d139546 100644 --- a/lnst/Recipes/ENRT/BaseEnrtRecipe.py +++ b/lnst/Recipes/ENRT/BaseEnrtRecipe.py @@ -2,7 +2,7 @@ import pprint from contextlib import contextmanager
from lnst.Common.LnstError import LnstError -from lnst.Common.Parameters import Param, IntParam, StrParam, BoolParam, ListParam +from lnst.Common.Parameters import Param, IntParam, StrParam, BoolParam, ListParam, FloatParam from lnst.Common.IpAddress import AF_INET, AF_INET6
from lnst.Recipes.ENRT.ConfigMixins.BaseSubConfigMixin import BaseSubConfigMixin @@ -16,10 +16,150 @@ from lnst.RecipeCommon.Perf.Measurements import StatCPUMeasurement from lnst.RecipeCommon.Perf.Evaluators import NonzeroFlowEvaluator
class EnrtConfiguration(object):
- """Container object for configuration
- Intentionally left empty as it is intended to be used as a container to
- store any values relevant to configuration being applied during the lifetime
- of the Recipe.
- """ pass
class BaseEnrtRecipe(BaseSubConfigMixin, PingTestAndEvaluate, PerfRecipe):
- #common requirements parameters
- """Base Recipe class for the ENRT recipe package
- This class defines the shared *test* method defining the common test
- procedure in a very generic way. This common test procedure involves a
- single main *test_wide* configuration that is different for every specific
- scenario. After the main configuration there is usually a loop of several
- minor *sub* configrations types that can take different values to slightly
- change the tested use cases.
- Finally, for each combination of a **test_wide** + **sub** configuration we
- do a several ping connection test and several performance measurement tests.
- **test_wide** and **sub** configurations are implemented with **context
- manager** methods which ensure that if any exceptions are raised (for
- example because of a bug in the recipe) that deconfiguration is called.
- Both **test_wide** and **sub** configurations are to be implemented in
- different classes, the BaseEnrtRecipe class only defines the common API and
- the base versions of the relevant methods.
- Test wide configuration is implemented via the following methods:
- :any:`test_wide_configuration`
- :any:`test_wide_deconfiguration`
- :any:`generate_test_wide_description`
- Sub configurations are **mixed in** to classes defining the specific
- scenario that is being tested. Various sub configurations are implemented as
- individual Python **Mixin** classes in the
- :any:`ConfigMixins<config_mixins>` package. These make use of Pythons
- collaborative inheritance by calling the `super` function in a specific way.
- The "machinery" for that defined in the :any:`BaseSubConfigMixin` class and
- used in this class from the `test` method loop that generates the combined
- Sub configuration and the `_sub_context` context manager method that takes
- care of applying and removing the configuration.
The last sentence is way too long and I simply do not understand what is meant here. Could you simplify/split into smaller parts?
- :param driver:
The driver parameter is used to modify the hw network requirements,specifically to request Devices using the specified driver. This iscommon enough in the Enrt recipes that it can be part of the Base class.- :type driver: :any:`StrParam` (default "ixgbe")
- :param ip_versions:
Parameter that determines which IP protocol versions will be tested.- :type ip_versions: Tuple[Str] (default ("ipv4", "ipv6"))
- :param ping_parallel:
Parameter used by the :any:`generate_ping_configurations` generator.Tells the generator method to create :any:`PingConf` objects that willbe run in parallel.- :type ping_parallel: :any:`BoolParam` (default False)
- :param ping_bidirect:
Parameter used by the :any:`generate_ping_configurations` generator.Tells the generator method to create :any:`PingConf` objects for bothdirections between the ping endpoints.- :type ping_bidirect: :any:`BoolParam` (default False)
- :param ping_count:
Parameter used by the :any:`generate_ping_configurations` generator.Tells the generator how many pings should be sent for each ping test.- :type ping_count: :any:`IntParam` (default 100)
- :param ping_interval:
Parameter used by the :any:`generate_ping_configurations` generator.Tells the generator how fast should the pings be sent in each ping test.- :type ping_interval: :any:`FloatParam` (default 0.2)
- :param ping_psize:
Parameter used by the :any:`generate_ping_configurations` generator.Tells the generator how big should the pings packets be in each pingtest.- :type ping_psize: :any:`IntParam` (default None)
- :param perf_tests:
Parameter used by the :any:`generate_flow_combinations` generator.Tells the generator what types of network flow measurements to generateperf test configurations for.- :type perf_tests: Tuple[str] (default ("tcp_stream", "udp_stream",
"sctp_stream"))- :param perf_tool_cpu:
Parameter used by the :any:`generate_flow_combinations` generator. Toindicate that the flow measurement should be pinned to a specific CPUcore.- :type perf_tool_cpu: :any:`IntParam` (optional parameter)
- :param perf_duration:
Parameter used by the :any:`generate_perf_configurations` generator. Tospecify the duration of the performance measurements, in seconds.- :type perf_duration: :any:`IntParam` (default 60)
- :param perf_iterations:
Parameter used by the :any:`generate_perf_configurations` generator. Tospecify how many times should each performance measurement be repeatedto generate cumulative results which can be statistically analyzed.- :type perf_iterations: :any:`IntParam` (default 5)
- :param perf_parallel_streams:
Parameter used by the :any:`generate_flow_combinations` generator. Tospecify how many parallel streams of the same network flow should bemeasured at the same time.- :type perf_parallel_streams: :any:`IntParam` (default 1)
- :param perf_msg_sizes:
Parameter used by the :any:`generate_flow_combinations` generator. Tospecify what different message sizes (in bytes) used generated for thenetwork flow should be tested - each message size resulting in aseparate performance measurement.- :type perf_msg_sizes: List[Int] (default [123])
- :param perf_reverse:
Parameter used by the :any:`generate_flow_combinations` generator. Tospecify that both directions between the endpoints of a network flowshould be measured for the same test.- :type perf_reverse: :any:`BoolParam` (default False)
- :param net_perf_tool:
Specifies a network flow measurement class that accepts :any:`PerfFlow`objects and can be used to measure those specified flowsParameter used by the :any:`generate_perf_configurations` generator tocreate a PerfRecipeConf object.- :type net_perf_tool: :any:`BaseFlowMeasurement` (default
IperfFlowMeasurement)- :param cpu_perf_tool:
Specifies a cpu measurement class that can be used to measure CPUutilization on specified hosts.Parameter used by the :any:`generate_perf_configurations` generator tocreate a PerfRecipeConf object.
Swap the two sentences in both net_perf_tool and cpu_perf_tool to match the pattern in other parameters.
:type cpu_perf_tool: :any:`BaseCPUMeasurement` (default StatCPUMeasurement)
"""
driver = StrParam(default="ixgbe")
#common test parameters
@@ -45,6 +185,20 @@ class BaseEnrtRecipe(BaseSubConfigMixin, PingTestAndEvaluate, PerfRecipe): cpu_perf_tool = Param(default=StatCPUMeasurement)
def test(self):
"""Main test loop shared by all the Enrt recipesThe test loop involves a single application of a **test_wide**configuration, then a loop over multiple **sub** configurations thatinvolves:* creating the combined sub configuration of all available SubConfigMixin classes via :any:`generate_sub_configurations`* applying the generated sub configuration via the :any:`_sub_context`context manager method* running tests* removing the current sub configuration via the :any:`_sub_context`context manager method""" with self._test_wide_context() as main_config: for sub_config in self.generate_sub_configurations(main_config): with self._sub_context(sub_config) as recipe_config:@@ -60,19 +214,90 @@ class BaseEnrtRecipe(BaseSubConfigMixin, PingTestAndEvaluate, PerfRecipe): self.test_wide_deconfiguration(config)
def test_wide_configuration(self):
"""Creates an empty :any:`EnrtConfiguration` objectThis is again used in potential collaborative inheritance design thatmay potentially be useful for Enrt recipes. Derived classes will eachindividually add their own values to the instance created here. This waythe complete test wide configuration is tracked in a single object.:return: returns a config object that tracks the applied configurationthat can be used during testing to inspect the current state andmake test decisions based on it.:rtype: :any:`EnrtConfiguration`Example::class Derived:def test_wide_configuration():config = super().test_wide_configuration()# ... configure somethingconfig.something = what_was_configuredreturn config""" return EnrtConfiguration()def test_wide_deconfiguration(self, config):
"""Base deconfiguration method.In the base class this should maybe only check if there's any leftoverconfiguration and warn about it. In derived classes this can beoverriden to take care of deconfiguring what was configured in therespective test_wide_configuration method.Example::class Derived:def test_wide_deconfiguration(config):# ... deconfigure somethingdel config.something #cleanup trackingreturn super().test_wide_deconfiguration()""" #TODO check if anything is still applied and throw exception? returndef describe_test_wide_configuration(self, config):
"""Describes the current test wide configurationCreates a new result object that tracks a *successful* result, but
Not quite sure what 'successful result' means.
contains the description of the full test wide configuration applied byall the :any:`test_wide_configuration` methods in the class hierarchy.The description needs to be generated by the:any:`generate_test_wide_description` method. Additionally thedescription contains the state of all the parameters and their valuespassed to the recipe class instance during initialization.""" description = self.generate_test_wide_description(config) self.add_result(True, "Summary of used Recipe parameters:\n{}".format( pprint.pformat(self.params._to_dict()))) self.add_result(True, "\n".join(description))def generate_test_wide_description(self, config):
"""Generates the test wide configuration descriptionAnother class inteded to be used with the collaborative version of the`super` method to cumulatively desribe the full test wide configurationthat was applied through multiple classes.The base class version of this method creates the initial list ofstrings containing just the header line. Each string added to this listwill later be printed on its own line.:return: list of strings, each representing a single line:rtype: List[str]Example::class Derived:def generate_sub_configuration_description(config):desc = super().generate_sub_configuration_description(config)desc.append("Configured something: {}".format(config.something))return desc""" return [ "Testwide configuration for recipe {} description:".format( self.__class__.__name__@@ -93,20 +318,47 @@ class BaseEnrtRecipe(BaseSubConfigMixin, PingTestAndEvaluate, PerfRecipe): self.add_result(True, "\n".join(description))
def do_tests(self, recipe_config):
"""Entry point for actual testsThe common scenario is to do ping and performance tests, however themethod can be overriden to add more tests if needed.""" self.do_ping_tests(recipe_config) self.do_perf_tests(recipe_config)def do_ping_tests(self, recipe_config):
"""Ping testing loopLoops over all various ping configurations generated by the:any:`generate_ping_configurations` method, then uses the PingRecipemethods to execute, report and evaluate the results.""" for ping_config in self.generate_ping_configurations(recipe_config): result = self.ping_test(ping_config) self.ping_evaluate_and_report(ping_config, result)def do_perf_tests(self, recipe_config):
"""Performance testing loopLoops over all various perf configurations generated by the:any:`generate_perf_configurations` method, then uses the PerfRecipemethods to execute, report and evaluate the results.""" for perf_config in self.generate_perf_configurations(recipe_config): result = self.perf_test(perf_config) self.perf_report_and_evaluate(result)def generate_ping_configurations(self, config):
"""Base ping test configuration generatorThe generator loops over all endpoint pairs to test ping between(generated by the :any:`generate_ping_endpoints` method) then over allthe selected :any:`ip_versions` and finally over all the IP addressesthat fit those criteria.:return: list of Ping configurations to test in parallel:rtype: List[:any:`PingConf`]""" for endpoint1, endpoint2 in self.generate_ping_endpoints(config): for ipv in self.params.ip_versions: ip_filter = {}@@ -144,9 +396,30 @@ class BaseEnrtRecipe(BaseSubConfigMixin, PingTestAndEvaluate, PerfRecipe): yield ping_conf_list
def generate_ping_endpoints(self, config):
"""Generator for ping endpointsTo be overriden by a derived class.:return: list of device pairs:rtype: List[Tuple[:any:`Device`, :any:`Device`]]""" return []def generate_perf_configurations(self, config):
"""Base perf test configuration generatorThe generator loops over all flow combinations to measure performancefor (generated by the :any:`generate_flow_combinations` method). Inaddition to that during each flow combination measurement we add CPUutilization measurement to run on the background.Finally for each generated perf test configuration we registermeasurement evaluators based on the :any:`cpu_perf_evaluators` and:any:`net_perf_evaluators` properties.:return: list of Perf test configurations:rtype: List[:any:`PerfRecipeConf`]""" for flows in self.generate_flow_combinations(config): perf_recipe_conf=dict( recipe_config=config,@@ -183,6 +456,18 @@ class BaseEnrtRecipe(BaseSubConfigMixin, PingTestAndEvaluate, PerfRecipe): yield perf_conf
def generate_flow_combinations(self, config):
"""Base flow combination generatorThe generator loops over all endpoint pairs to test performance between(generated by the :any:`generate_perf_endpoints` method) then over allthe selected :any:`ip_versions` and uses the first IP address fittingthese criteria. Then the generator loops over the selected performancetests as selected via :any:`perf_tests`, then message sizes from:any:`msg_sizes`.:return: list of Flow combinations to measure in parallel:rtype: List[:any:`PerfFlow`]""" for client_nic, server_nic in self.generate_perf_endpoints(config): for ipv in self.params.ip_versions: if ipv == "ipv4":@@ -213,14 +498,37 @@ class BaseEnrtRecipe(BaseSubConfigMixin, PingTestAndEvaluate, PerfRecipe): yield [reverse_flow]
def generate_perf_endpoints(self, config):
"""Generator for perf endpointsTo be overriden by a derived class.:return: list of device pairs:rtype: List[Tuple[:any:`Device`, :any:`Device`]]""" return []@property def cpu_perf_evaluators(self):
"""CPU measurement evaluatorsTo be overriden by a derived class. Returns the list of evaluators touse for CPU utilization measurement evaluation.:return: a list of cpu evaluator objects:rtype: List[BaseEvaluator]""" return []@property def net_perf_evaluators(self):
"""Network flow measurement evaluatorsTo be overriden bby a derived class. Returns the list of evaluators touse for Network flow measurement evaluation.:return: a list of flow evaluator objects:rtype: List[BaseEvaluator]""" return [NonzeroFlowEvaluator()]def _create_reverse_flow(self, flow):
diff --git a/lnst/Recipes/ENRT/ConfigMixins/README.md b/lnst/Recipes/ENRT/ConfigMixins/README.md new file mode 100644 index 0000000..0472be0 --- /dev/null +++ b/lnst/Recipes/ENRT/ConfigMixins/README.md @@ -0,0 +1,3 @@ +# SubConfig Mixins
+TODO diff --git a/lnst/Recipes/ENRT/ConfigMixins/__init__.py b/lnst/Recipes/ENRT/ConfigMixins/__init__.py index e69de29..7ae6450 100644 --- a/lnst/Recipes/ENRT/ConfigMixins/__init__.py +++ b/lnst/Recipes/ENRT/ConfigMixins/__init__.py @@ -0,0 +1,9 @@ +""" +ENRT Config Mixins +==================
+.. autoclass:: lnst.Recipes.ENRT.ConfigMixins.BaseSubConfigMixin.BaseSubConfigMixin
+"""
+from lnst.Recipes.ENRT.ConfigMixins.BaseSubConfigMixin import BaseSubConfigMixin diff --git a/lnst/Recipes/ENRT/README.md b/lnst/Recipes/ENRT/README.md new file mode 100644 index 0000000..3500cdc --- /dev/null +++ b/lnst/Recipes/ENRT/README.md @@ -0,0 +1,47 @@ +# ENRT Recipes
+ENRT stands for Early Network Regression Testing, which was the name of the +project a couple of years back when we started developing this set of tests +when we were still using the Legacy LNST API.
+This package aims to reimplement the same set of tests, using the new LNST-next +APIs, while fully utilizing Python to address the largest problems with the old +implementation: +* large amounts of code duplication +* copy paste errors caused by code duplication +* very hard to maintain the test set and fix bugs +* very hard to add new tests due to limitations of the old LNST Framework
+With that in mind, the main goals for this reimplementation were as follows: +* reduce code duplication as much as possible by utilizing class inheritance +* separate individual types of configurations into smaller Mixin classes that
- can be mixed and matched based on what a specific Test scenario requires
+* writing new test scenarios should be very quick because it should be possible
- to reuse most of the functionality already defined
+* it should be possible to very easily extend the recipes with new features such
- as more types of parallel or sequential measurements, more types of
- Evaluations for different types of measurements, or to be able to switch out
- and use different measurement tools
+The resulting design is split into several parts that interact with each other +in a specific way:
+* [BaseEnrtRecipe](BaseEnrtRecipe.py) serves as the base class for all the
- specific test scenarios we want to test. Defines the common test loop and the
- default implementation for configuration generators used in this common test
- loop.
+* [ConfigMixins](ConfigMixins/README.md) package contains the various types of
- "SubConfig" mixin classes, these are classes that implement some form of
- configuration on test machine(s) which can be reused between mutliple recipes,
- but can often times be looped over to try different variations of the
- configuration. A good example is configuration of hardware offloads, it's
- relevant to test it for many different test scenarios, but only some
- combinations of offloads make sense based on the scenario and we often time
- want to test more than one combination
+* Specific test scenario implementation that defines the requirements and the
- main configuration for the specific scenario that we want to test. It also
- defines the combination of various "SubConfig" configurations that we want to
- include by adding them to it's inheritance tree. If required the recipe can
- also override any of the default functionality defined by it's parent classes,
- a good example could be the generator method for creating various
- configurations for flow performance measurement.
diff --git a/lnst/Recipes/ENRT/SimpleNetworkRecipe.py b/lnst/Recipes/ENRT/SimpleNetworkRecipe.py index 911a996..2c8e12b 100644 --- a/lnst/Recipes/ENRT/SimpleNetworkRecipe.py +++ b/lnst/Recipes/ENRT/SimpleNetworkRecipe.py @@ -12,6 +12,25 @@ from lnst.Recipes.ENRT.ConfigMixins.CommonHWSubConfigMixin import ( class SimpleNetworkRecipe( CommonHWSubConfigMixin, OffloadSubConfigMixin, BaseEnrtRecipe ):
- """
- This recipe implements Enrt testing for a simple network scenario that looks
- as follows
- .. code-block::
+--------++------+ switch +-----+| +--------+ |+--+-+ +-+--++-|eth0|-+ +-|eth0|-+| +----+ | | +----+ || host1 | | host2 |+--------+ +--------+- All sub configurations are included via Mixin classes.
- The actual test machinery is implemented in the :any:`BaseEnrtRecipe` class.
- """ host1 = HostReq() host1.eth0 = DeviceReq(label="net1", driver=RecipeParam("driver"))
@@ -26,6 +45,14 @@ class SimpleNetworkRecipe( dict(gro="on", gso="on", tso="on", tx="on", rx="off")))
def test_wide_configuration(self):
"""Test wide configuration for this recipe involves just adding an IPv4 andIPv6 address to the matched eth0 nics on both hosts.host1.eth0 = 192.168.101.1/24 and fc00::1/64host2.eth0 = 192.168.101.2/24 and fc00::2/64""" host1, host2 = self.matched.host1, self.matched.host2 configuration = super().test_wide_configuration() configuration.test_wide_devices = []@@ -41,6 +68,9 @@ class SimpleNetworkRecipe( return configuration
def generate_test_wide_description(self, config):
"""Test wide description is extended with the configured addresses""" desc = super().generate_test_wide_description(config) desc += [ "Configured {}.{}.ips = {}".format(@@ -51,14 +81,33 @@ class SimpleNetworkRecipe( return desc
def test_wide_deconfiguration(self, config):
"" # overriding the parent docstring del config.test_wide_devices super().test_wide_deconfiguration(config)def generate_ping_endpoints(self, config):
"""The ping endpoints for this recipe are simply the two matched NICs:host1.eth0 and host2.eth0Returned as::[(self.matched.host1.eth0, self.matched.host2.eth0)]""" return [(self.matched.host1.eth0, self.matched.host2.eth0)]def generate_perf_endpoints(self, config):
"""The perf endpoints for this recipe are simply the two matched NICs:host1.eth0 and host2.eth0Returned as::[(self.matched.host1.eth0, self.matched.host2.eth0)]""" return [(self.matched.host1.eth0, self.matched.host2.eth0)]def wait_tentative_ips(self, devices):
diff --git a/lnst/Recipes/ENRT/__init__.py b/lnst/Recipes/ENRT/__init__.py index ea5aaf0..a591ef2 100644 --- a/lnst/Recipes/ENRT/__init__.py +++ b/lnst/Recipes/ENRT/__init__.py @@ -1,3 +1,55 @@ +''' +ENRT Recipes +------------
+ENRT stands for Early Network Regression Testing, which was the name of the +project a couple of years back when we started developing this set of tests +when we were still using the Legacy LNST API.
+This package aims to reimplement the same set of tests, using the new LNST-next +APIs, while fully utilizing Python to address the largest problems with the old +implementation:
+* large amounts of code duplication +* copy paste errors caused by code duplication +* very hard to maintain the test set and fix bugs +* very hard to add new tests due to limitations of the old LNST Framework
+With that in mind, the main goals for this reimplementation were as follows:
+* reduce code duplication as much as possible by utilizing class inheritance +* separate individual types of configurations into smaller Mixin classes that
- can be mixed and matched based on what a specific Test scenario requires
+* writing new test scenarios should be very quick because it should be possible
- to reuse most of the functionality already defined
+* it should be possible to very easily extend the recipes with new features
- such as more types of parallel or sequential measurements, more types of
- Evaluations for different types of measurements, or to be able to switch out
- and use different measurement tools
+The resulting design is split into several parts that interact with each other +in a specific way:
+* BaseEnrtRecipe serves as the base class for all the specific test scenarios
- we want to test. Defines the common test loop and the default implementation
- for configuration generators used in this common test loop.
+* ConfigMixins package contains the various types of "SubConfig" mixin classes,
- these are classes that implement some form of configuration on test
- machine(s) which can be reused between mutliple recipes, but can often times
- be looped over to try different variations of the configuration. A good
- example is configuration of hardware offloads, it's relevant to test it for
- many different test scenarios, but only some combinations of offloads make
- sense based on the scenario and we often time want to test more than one
- combination
+* Specific test scenario implementation that defines the requirements and the
- main configuration for the specific scenario that we want to test. It also
- defines the combination of various "SubConfig" configurations that we want to
- include by adding them to it's inheritance tree. If required the recipe can
- also override any of the default functionality defined by it's parent
- classes, a good example could be the generator method for creating various
- configurations for flow performance measurement.
+''' from lnst.Recipes.ENRT.SimpleNetworkRecipe import SimpleNetworkRecipe from lnst.Recipes.ENRT.BondRecipe import BondRecipe from lnst.Recipes.ENRT.DoubleBondRecipe import DoubleBondRecipe -- 2.25.1 _______________________________________________ LNST-developers mailing list -- lnst-developers@lists.fedorahosted.org To unsubscribe send an email to lnst-developers-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/lnst-developers@lists.fedorahos...
I got an error because sphinx wasn't installed. Do we want to add it to requirements.txt (or perhaps something like requirements_extra.txt?)
On Tue, Mar 17, 2020 at 11:52 AM Jan Tluka jtluka@redhat.com wrote:
Additional comments, inline.
Mon, Mar 16, 2020 at 11:43:03AM CET, olichtne@redhat.com wrote:
From: Ondrej Lichtner olichtne@redhat.com
Creating the docs/ directory where we should work on creating documentation. The standard for Python projects is to use "Sphinx" to generate various formats (e.g. pdf, html or others) of documentation from ReStructuredText (rst) source. The source is either stored as ".rst" standalone files or as docstrings with Python source code.
This is the first version that should cover some of the major topics and exported APIs available for the tester. It is by no means complete yet but should be enough to get a feel for how the overall structure should look like.
Signed-off-by: Ondrej Lichtner olichtne@redhat.com
README.md | 25 +- docs/Makefile | 20 ++ docs/source/ControllerAPI.rst | 5 + docs/source/Parameters.rst | 5 + docs/source/RecipeAPI.rst | 5 + docs/source/RecipeRequirementsAPI.rst | 5 + docs/source/SimpleNetworkRecipe.rst | 6 + docs/source/TesterAPI.rst | 10 + docs/source/base_enrt_class.rst | 8 + docs/source/conf.py | 64 +++++ docs/source/config_mixins.rst | 3 + docs/source/enrt_recipes.rst | 12 + docs/source/index.rst | 25 ++ docs/source/installation.rst | 133 +++++++++ docs/source/recipe_packages.rst | 7 + docs/source/specific_scenarios.rst | 5 + lnst/Common/Parameters.py | 23 +- lnst/Controller/Controller.py | 92 +++--- lnst/Controller/Recipe.py | 71 ++--- lnst/Controller/Requirements.py | 80 ++++-- lnst/Controller/__init__.py | 7 + lnst/Recipes/ENRT/BaseEnrtRecipe.py | 312 ++++++++++++++++++++- lnst/Recipes/ENRT/ConfigMixins/README.md | 3 + lnst/Recipes/ENRT/ConfigMixins/__init__.py | 9 + lnst/Recipes/ENRT/README.md | 47 ++++ lnst/Recipes/ENRT/SimpleNetworkRecipe.py | 49 ++++ lnst/Recipes/ENRT/__init__.py | 52 ++++ 27 files changed, 965 insertions(+), 118 deletions(-) create mode 100644 docs/Makefile create mode 100644 docs/source/ControllerAPI.rst create mode 100644 docs/source/Parameters.rst create mode 100644 docs/source/RecipeAPI.rst create mode 100644 docs/source/RecipeRequirementsAPI.rst create mode 100644 docs/source/SimpleNetworkRecipe.rst create mode 100644 docs/source/TesterAPI.rst create mode 100644 docs/source/base_enrt_class.rst create mode 100644 docs/source/conf.py create mode 100644 docs/source/config_mixins.rst create mode 100644 docs/source/enrt_recipes.rst create mode 100644 docs/source/index.rst create mode 100644 docs/source/installation.rst create mode 100644 docs/source/recipe_packages.rst create mode 100644 docs/source/specific_scenarios.rst create mode 100644 lnst/Recipes/ENRT/ConfigMixins/README.md create mode 100644 lnst/Recipes/ENRT/README.md
diff --git a/README.md b/README.md index 256f79e..dec9aba 100644 --- a/README.md +++ b/README.md @@ -12,7 +12,7 @@ backwards compatibility are yet in place.
This also means that many of our documentation resources outlining how
to write
recipes on the wiki are also out of date. We'll soon start working on
these but
-please be paitent with us. +please be patient with us.
If you're interested in helping out we accept code contributions via
Patches
submitted to our mailing list lnst-developers@lists.fedorahosted.org. @@ -33,13 +33,13 @@ Internet Resources bellow).
## Install
-LNST can be installed using python's distutils. +Installation and a simple Hello world example is available at +[Installation](docs/source/installation.rst)
-```bash -su -./setup.py install -``` +## Documentation
+Documentation is available in the `docs/` directory, you can build it
with
+`make html` using *Sphinx*.
## Authors/Contributors
@@ -53,18 +53,15 @@ su
- Jiri Zupka (not active anymore)
- Radek Pazdera (not active anymore)
+## How to contact us
-## Internet Resources
-* Project Wiki: https://github.com/jpirko/lnst/wiki (currently out
of date)
-* Documentation: https://github.com/jpirko/lnst/wiki#learn
(currently out of date)
-* Git Source Tree: https://github.com/jpirko/lnst -* Mailing List: lnst-developers@lists.fedorahosted.org
+* Git Source Tree: https://github.com/jpirko/lnst +* Mailing List: lnst-developers@lists.fedorahosted.org +* IRC channel: #lnst @ freenode.net
## License
-**Copyright (C) 2011-2019 Red Hat, Inc.** +**Copyright (C) 2011-2020 Red Hat, Inc.**
LNST is distributed under GNU General Public License version 2. See the
file
"COPYING" in the source distribution for information on terms &
conditions
diff --git a/docs/Makefile b/docs/Makefile new file mode 100644 index 0000000..d0c3cbf --- /dev/null +++ b/docs/Makefile @@ -0,0 +1,20 @@ +# Minimal makefile for Sphinx documentation +#
+# You can set these variables from the command line, and also +# from the environment for the first two. +SPHINXOPTS ?= +SPHINXBUILD ?= sphinx-build +SOURCEDIR = source +BUILDDIR = build
+# Put it first so that "make" without argument is like "make help". +help:
@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS)$(O)
+.PHONY: help Makefile
+# Catch-all target: route all unknown targets to Sphinx using the new +# "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS). +%: Makefile
@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS)$(O)
diff --git a/docs/source/ControllerAPI.rst b/docs/source/ControllerAPI.rst new file mode 100644 index 0000000..f80d840 --- /dev/null +++ b/docs/source/ControllerAPI.rst @@ -0,0 +1,5 @@ +Controller API +^^^^^^^^^^^^^^
+.. automodule:: lnst.Controller.Controller
- :members: Controller
diff --git a/docs/source/Parameters.rst b/docs/source/Parameters.rst new file mode 100644 index 0000000..9646b01 --- /dev/null +++ b/docs/source/Parameters.rst @@ -0,0 +1,5 @@ +Parameters +^^^^^^^^^^
+.. automodule:: lnst.Common.Parameters
- :members:
diff --git a/docs/source/RecipeAPI.rst b/docs/source/RecipeAPI.rst new file mode 100644 index 0000000..32e7b4c --- /dev/null +++ b/docs/source/RecipeAPI.rst @@ -0,0 +1,5 @@ +Recipe API +^^^^^^^^^^
+.. automodule:: lnst.Controller.Recipe
- :members: BaseRecipe, RecipeRun
diff --git a/docs/source/RecipeRequirementsAPI.rst
b/docs/source/RecipeRequirementsAPI.rst
new file mode 100644 index 0000000..4cf1c59 --- /dev/null +++ b/docs/source/RecipeRequirementsAPI.rst @@ -0,0 +1,5 @@ +Recipe Requirements +^^^^^^^^^^^^^^^^^^^
+.. automodule:: lnst.Controller.Requirements
- :members: HostReq, DeviceReq
diff --git a/docs/source/SimpleNetworkRecipe.rst
b/docs/source/SimpleNetworkRecipe.rst
new file mode 100644 index 0000000..1f08feb --- /dev/null +++ b/docs/source/SimpleNetworkRecipe.rst @@ -0,0 +1,6 @@ +SimpleNetworkRecipe +^^^^^^^^^^^^^^^^^^^
+.. autoclass:: lnst.Recipes.ENRT.SimpleNetworkRecipe.SimpleNetworkRecipe
- :members:
- :show-inheritance:
diff --git a/docs/source/TesterAPI.rst b/docs/source/TesterAPI.rst new file mode 100644 index 0000000..9261171 --- /dev/null +++ b/docs/source/TesterAPI.rst @@ -0,0 +1,10 @@ +Test developer API +==================
+.. toctree::
- :maxdepth: 2
- RecipeRequirementsAPI
- RecipeAPI
- ControllerAPI
- Parameters
diff --git a/docs/source/base_enrt_class.rst
b/docs/source/base_enrt_class.rst
new file mode 100644 index 0000000..32dd6fd --- /dev/null +++ b/docs/source/base_enrt_class.rst @@ -0,0 +1,8 @@ +BaseEnrtRecipe class +====================
+.. autoclass:: lnst.Recipes.ENRT.BaseEnrtRecipe.BaseEnrtRecipe
- :members:
- :show-inheritance:
+.. autoclass:: lnst.Recipes.ENRT.BaseEnrtRecipe.EnrtConfiguration diff --git a/docs/source/conf.py b/docs/source/conf.py new file mode 100644 index 0000000..7c59490 --- /dev/null +++ b/docs/source/conf.py @@ -0,0 +1,64 @@ +# Configuration file for the Sphinx documentation builder. +# +# This file only contains a selection of the most common options. For a
full
+# list see the documentation: +# https://www.sphinx-doc.org/en/master/usage/configuration.html
+# -- Path setup
+# If extensions (or modules to document with autodoc) are in another
directory,
+# add these directories to sys.path here. If the directory is relative
to the
+# documentation root, use os.path.abspath to make it absolute, like
shown here.
+# +# import os +# import sys +# sys.path.insert(0, os.path.abspath('.'))
+# -- Project information
+project = 'lnst' +copyright = '2020, Jiri Pirko' +author = 'Jiri Pirko'
+# -- General configuration
+# Add any Sphinx extension module names here, as strings. They can be +# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom +# ones. +extensions = [
- 'sphinx.ext.autodoc',
+]
+import sys +sys.path = ['../'] + sys.path
+# Add any paths that contain templates here, relative to this directory. +templates_path = ['_templates']
+# List of patterns, relative to source directory, that match files and +# directories to ignore when looking for source files. +# This pattern also affects html_static_path and html_extra_path. +exclude_patterns = []
+# -- Options for HTML output
+# The theme to use for HTML and HTML Help pages. See the documentation
for
+# a list of builtin themes. +# +html_theme = 'classic' +html_theme_options = {
- #"body_min_width": "100%",
- "body_max_width": "100%",
+}
+# Add any paths that contain custom static files (such as style sheets)
here,
+# relative to this directory. They are copied after the builtin static
files,
+# so a file named "default.css" will overwrite the builtin "default.css". +html_static_path = ['_static']
+autodoc_default_options = {
- 'member-order': 'bysource',
+} diff --git a/docs/source/config_mixins.rst b/docs/source/config_mixins.rst new file mode 100644 index 0000000..66119b6 --- /dev/null +++ b/docs/source/config_mixins.rst @@ -0,0 +1,3 @@ +.. ENRT Config Mixins
+.. automodule:: lnst.Recipes.ENRT.ConfigMixins diff --git a/docs/source/enrt_recipes.rst b/docs/source/enrt_recipes.rst new file mode 100644 index 0000000..af8935a --- /dev/null +++ b/docs/source/enrt_recipes.rst @@ -0,0 +1,12 @@ +Early Network Regression Testing (ENRT) +=======================================
+.. automodule:: lnst.Recipes.ENRT
+.. toctree::
- :maxdepth: 2
- :caption: ENRT Components
- base_enrt_class
- config_mixins
- specific_scenarios
diff --git a/docs/source/index.rst b/docs/source/index.rst new file mode 100644 index 0000000..e5f2dd5 --- /dev/null +++ b/docs/source/index.rst @@ -0,0 +1,25 @@ +LNST - Linux Network Stack Test +===============================
+Linux Network Stack Test is a framework that supports development and
execution
+of automated and portable network tests which usually involve multiple
test
+systems. This repository contains the implementation of the library to
write
+:any:`Recipes<BaseRecipe>`, library functions to implement executable
script
+files which run your tests as well as the code for the "LNST Slave" +application.
+.. toctree::
- :maxdepth: 2
- :caption: General topics:
- installation
- TesterAPI
- recipe_packages
+Indices and tables +==================
+* :ref:`genindex` +* :ref:`modindex` +* :ref:`search` diff --git a/docs/source/installation.rst b/docs/source/installation.rst new file mode 100644 index 0000000..ede73a2 --- /dev/null +++ b/docs/source/installation.rst @@ -0,0 +1,133 @@ +Install LNST and Hello world +============================
+LNST is logically split into two separate application use cases:
+* Controller - something that controlls the execution of your :any:`Test
- Recipes<BaseRecipe>`
+* Slave - a server application running on all hosts available for
testing,
- executes remote procedure calls from the Controller to either run
tests or
- configure the test machine
+Codebases for both use cases are developed in this repository and as we +currently don't have a stable release yet, the recommended method of +installation involves the following steps:
+.. code-block:: bash
- git clone https://github.com/jpirko/lnst
- cd lnst
- pip3 install --requirements requirements.txt
- pip3 install .
+This installs both the Controller and the Slave code, and you'll need to
run
+this on all the test machines that you want to use as well as the
machine which
+you want to use as the Controller. Optionally a Controller and a Slave
CAN run
+on the same machine.
+You can start your Slave application immediatelly by running::
- lnst-slave
+Because the lnst-slave application takes care of network configuration,
it
+**requires** to be executed with root privileges. This is **A BIG
SECURITY
+RISK** so make sure you only run this application on test machines that
are not
+publicly accessible or don't contain any sensitive data.
+The Controller is a bit more complicated and requires you to:
+* create an executable test script +* create a slave machine pool
+Creating an executable "HelloWorld" test script +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+LNST currently doesn't come with a CLI application for the Controller,
instead
+we need to create an executable python script ourselves that takes care
of
+creating an instance of a Controller class and an instance of the Test
Recipe
+that we want to run and calling the Controller.run() method to execute
it.
+A minimal "hello world" example of an executable test script looks like
this:
+.. code-block:: python
- from lnst.Controller import Controller, HostReq, DeviceReq,
BaseRecipe
- class HelloWorldRecipe(BaseRecipe):
machine1 = HostReq()machine1.nic1 = DeviceReq(label="net1")machine2 = HostReq()machine2.nic1 = DeviceReq(label="net1")def test(self):self.matched.m1.nic1.ip_add("192.168.1.1/24")self.matched.m1.nic1.up()self.matched.m2.nic1.ip_add("192.168.1.2/24")self.matched.m2.nic1.up()self.matched.m1.run("ping 192.168.1.2")- ctl = Controller()
- recipe_instance = HelloWorldRecipe()
- ctl.run(recipe_instance)
+This test requires that you have 2 test machines that are directly
connected to
+each other. +If you write this into a ``hello_world.py`` file you should now be able
to
+execute this script by running::
- python3 hello_world.py
+And you'll end up receiving an error about being unable to find a match
in your
+configured pools, since we didn't configure any yet, this is quite
expected. But
+running this script did take care of creating a default configuration
file and
+directory where we'll now be able to create our machine pool.
+Creating a simple machine pool +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+The default location for the Controller config file is
``~/.lnst/lnst-ctl.conf``.
+At this point in time, you don't need to change anything inside this
file.
+At the same time, the default location for a machine pool is
``~/.lnst/pool/``,
+to create a pool you'll need to put XML files that describe your test
machines
+where the ``lnst-slave`` application is running, and how they're
connected. You
+need to create one file per test machine, so to satisfy the +**HelloWorldRecipe** requirements, we need to create two files:
+.. code-block:: bash
- touch ~/.lnst/pool/test_machine1.xml
- touch ~/.lnst/pool/test_machine2.xml
+For the contents of the files you can use the following template:
+.. code-block:: xml
<slavemachine>
<params><param name="hostname" value="HOSTNAME"/><param name="rpc_port" value="9999"/></params><interfaces><eth label="A" id="1"><params><param name="hwaddr" value="MAC_ADDRESS"/></params></eth></interfaces></slavemachine>
+You'll need to edit the template and replace the **HOSTNAME** and +**MAC_ADDRESS** strings with values that correspond to the hostname
which the
+controller can use to connet to the slave, and the mac address of a
network
+interface usable for testing. This **MUST** be a different interface
than the
+one used for the Controller-Slave connection, as it's configuration will
change
+during test execution, the Controller-Slave connection would break if
you used
+the same interface.
+After creating your pool, you should now be able to run the
``hello_world.py``
+script successfully and receive back some logs about what happened. diff --git a/docs/source/recipe_packages.rst
b/docs/source/recipe_packages.rst
new file mode 100644 index 0000000..ccd6269 --- /dev/null +++ b/docs/source/recipe_packages.rst @@ -0,0 +1,7 @@ +Supported Recipe packages +=========================
+.. toctree::
- :maxdepth: 2
- enrt_recipes
diff --git a/docs/source/specific_scenarios.rst
b/docs/source/specific_scenarios.rst
new file mode 100644 index 0000000..2f13d02 --- /dev/null +++ b/docs/source/specific_scenarios.rst @@ -0,0 +1,5 @@ +Specific ENRT scenarios +=======================
+.. toctree::
- SimpleNetworkRecipe
diff --git a/lnst/Common/Parameters.py b/lnst/Common/Parameters.py index e56c32d..f73fd3a 100644 --- a/lnst/Common/Parameters.py +++ b/lnst/Common/Parameters.py @@ -4,10 +4,6 @@ This module defines the Param class, it's type specific
derivatives
Param instances. This can be used by a BaseRecipe class to specify optional/mandatory parameters for the entire test, or by HostReq and
DeviceReq
classes to define specific parameters needed for the matching algorithm.
-Copyright 2017 Red Hat, Inc. -Licensed under the GNU General Public License, version 2 as -published by the Free Software Foundation; see COPYING for details. """
__author__ = """ @@ -24,12 +20,31 @@ class ParamError(LnstError): pass
class Param(object):
- """Base Parameter class
- Can beused to define your own specific parameter type. Param derived
classes
- serve as *type checkers* to enable earlier failure of the recipe.
- :param mandatory: if `True`, marks the parameter as mandatory
- :type mandatory: bool
- :param default: the default value for the parameter, is also
type-checked,
immediately at Param object creation""" def __init__(self, mandatory=False, **kwargs): self.mandatory = mandatory if "default" in kwargs: self.default = self.type_check(kwargs["default"])
def type_check(self, value):
"""The type check methodImplementation depends on the specific Param derived class.:return: the type checked or converted value:raises: :any:`ParamError` if the type check or conversion isinvalid
""" return valueclass IntParam(Param): diff --git a/lnst/Controller/Controller.py b/lnst/Controller/Controller.py index ace0caf..d992b27 100644 --- a/lnst/Controller/Controller.py +++ b/lnst/Controller/Controller.py @@ -2,10 +2,6 @@ This module defines the Controller class that brings together individual implementation parts of an LNST Controller. When instantiated, it allows
the
tester to configure and run his own recipes with the LNST
'infrastructure'.
-Copyright 2017 Red Hat, Inc. -Licensed under the GNU General Public License, version 2 as -published by the Free Software Foundation; see COPYING for details. """
__author__ = """ @@ -33,34 +29,59 @@ from lnst.Controller.Recipe import BaseRecipe,
RecipeRun
from lnst.Controller.RecipeControl import RecipeControl
class Controller(object):
- """The LNST Controller class
- Most importantly allows the tester to run instantiated Recipe tests
using
- the LNST infrastructure.
- Can be configured with custom implementation of several objects used
for
- setting up the infrastructure.
- """Allows to run LNST Recipe instances
- This is the main mechanism that allows users to create their own
executable
- test scripts that execute LNST Recipes.
- The Controller class implementation provides the most common default
values
- for various parameters that can significantly change the way that
Recipes
- are executed. This includes custom implementations of classes that
are used
- for setting up the testing infrastructure such as the PoolManager or
the
- MachineMapper.
- :param poolMgr:
class that implements the:py:class:`lnst.Controller.SlavePoolManager.SlavePoolManager`interface
will be instantiated by the Controller to provide the mapperwith pools
available for matching, also handles the creation of:py:class:`Machine` objects (internal LNST class used to accessthe
slave hosts)- :type poolMgr:
:py:class:`lnst.Controller.MachineMapper.MachineMapper`
- :param mapper:
class that implements the:py:class:`lnst.Controller.MachineMapper.MachineMapper`interface will
be instantiated by the Controller to match Recipe requirementsto the
available pools:type mapper: :py:class:`lnst.Controller.MachineMapper.MachineMapper`
:param config:
optional LNST configuration object, if None the Controller willload it's own configuration from default paths:type config: :py:class:`lnst.Controller.Config.CtlConfig`
:param pools:
a list of pool names to restrict the used pool directories:type pools: List[str]
:param pool_checks:
if False, will disable checking the online status of Slaves:type pool_checks: boolean (default True)
:param debug:
sets the debug level of LNST:type debug: integer (default 0)
Example::
lnst_controller = Controller()recipe_instance = MyRecipe(test_parameter=123)lnst_controller.run(recipe_instance)"""
def __init__(self, poolMgr=SlavePoolManager, mapper=MachineMapper, config=None, pools=[], pool_checks=True, debug=0):
"""Args:poolMgr -- class that implements the SlavePoolManagerinterface
will be instantiated by the Controller to provide themapper
with pools available for matching, also handles thecreation
of Machine objects (internal LNST class used to accessthe
slave hosts)mapper -- class that implements the MachineMapper interfacewill be instantiated by the Controller to match Reciperequirements to the available poolsconfig -- optional LNST configuration object, if None theController will load it's own configuration from defaultpaths
pools -- a list of pool names to restrict the used pooldirectories
pool_checks -- boolean (default True), if False will disablechecking online status of Slavesdebug -- integer (default 0), sets debug level of LNST""" self._config = self._load_ctl_config(config) config = self._config@@ -96,13 +117,17 @@ class Controller(object): def run(self, recipe, **kwargs): """Execute the provided Recipe
This method takes care of both finding a Slave hosts matchingthe Recipe
requirements, provisioning them and calling the 'test' method ofthe
This method takes care of both finding Slave hosts matching theRecipe
requirements, provisioning them and calling the *test* method ofthe
Recipe object with proper references to the mapped Hosts
Args:recipe -- an instantiated Recipe object (isinstanceBaseRecipe)
kwargs -- optional keyword arguments passed to theconfigured Mapper
:param recipe:an instantiated Recipe object:type recipe: :py:class:`lnst.Controller.Recipe.BaseRecipe`:param kwargs:optional keyword arguments passed to the configured Mapper:type kwargs: Dict[str, Any] """ if not isinstance(recipe, BaseRecipe): raise ControllerError("recipe argument must be a BaseRecipeinstance.")
@@ -134,7 +159,6 @@ class Controller(object): finally: self._cleanup_slaves()
- def _map_match(self, match, requested, recipe): self._machines = {} self._hosts = Hosts()
diff --git a/lnst/Controller/Recipe.py b/lnst/Controller/Recipe.py index 42f4e3d..26e0737 100644 --- a/lnst/Controller/Recipe.py +++ b/lnst/Controller/Recipe.py @@ -1,9 +1,5 @@ """ Module implementing the BaseRecipe class.
-Copyright 2017 Red Hat, Inc. -Licensed under the GNU General Public License, version 2 as -published by the Free Software Foundation; see COPYING for details. """
__author__ = """ @@ -23,27 +19,30 @@ class RecipeError(ControllerError): pass
class BaseRecipe(object):
- """BaseRecipe class
"""Base class for LNST Recipe definition.
Every LNST Recipe written by testers should be inherited from this
class.
An LNST Recipe is composed of several parts:
- Requirements definition - you define recipe requirements in a
derived
class by defining class attributes of the HostReq type. You canfurther
specify Ethernet Device requirements by defining DeviceReqattributes
of the HostReq object.Example:m1 = HostReq(arch="x86_64")m1.eth0 = DeviceReq(driver="ixgbe")
- Parameter definition (optional) - you can define paramaters of
your Recipe
by defining class attributes of the Param type (or inherited).These
parameters can then be accessed from the test() method to changeit's
behaviour. Parameter validity (type) is checked during theinstantiation of the Recipe object by the base __init__ method.You can define your own __init__ method to implement more complexParameter checking if needed, but you MUST call the base __init__method first.Example:MyRecipe(BaseRecipe):
class by defining class attributes of the HostReq type. You canfurther
specify Ethernet Device requirements by defining DeviceReqattributes of
the :py:class:`lnst.Controller.Requirements.HostReq` object.Example::
class MyRecipe(BaseRecipe):m1 = HostReq(arch="x86_64")m1.eth0 = DeviceReq(driver="ixgbe")
- Parameter definition (optional) - you can define paramaters of your
Recipe by defining class attributes of the :any:`Param` type (orinherited). These parameters can then be accessed from the test()method
to change it's behaviour. Parameter validity (type) is checkedduring the
instantiation of the Recipe object by the base __init__ method.You can
define your own __init__ method to implement more complex Parameterchecking if needed, but you MUST call the base __init__ methodfirst.
Example::class MyRecipe(BaseRecipe): int_param = IntParam(mandatory=True) optional_param = IntParam()@@ -55,18 +54,24 @@ class BaseRecipe(object): MyRecipe(int_param = 2, optional_param = 3)
* Test definition - this is done by defining the test() method, inthis
method the tester has direct access to mapped LNST slave Hosts,can
manipulate them and implement his tests.- Attributes:
matched -- when running the Recipe the Controller will fill thisattribute with a Hosts object after the Mapper findssuitable slave
hosts.req -- instantiated Requirements object, you can optionallychange the
Recipe requirements through this object during runtime (e.g.variable number of hosts or devices of a host based on aParameter)
params -- instantiated Parameters object, can be used to accessthe
calculated parameters during Recipe initialization/execution
method the tester has direct access to mapped LNST slave Hosts, canmanipulate them and implement his tests.- :ivar matched:
When running the Recipe the Controller will fill this attributewith a
Hosts object after the Mapper finds suitable slave hosts.- :type matched: :py:class:`lnst.Controller.Host.Hosts`
- :ivar req:
Instantiated Requirements object, you can optionally change theRecipe
requirements through this object during runtime (e.g. variablenumber
of hosts or devices of a host based on a Parameter)- :type req: :py:class:`lnst.Controller.Requirements._Requirements`
- :ivar params:
Instantiated Parameters object, can be used to access thecalculated
parameters during Recipe initialization/execution- :type params: :py:class:`lnst.Common.Parameters.Parameters` """ def __init__(self, **kwargs): """
diff --git a/lnst/Controller/Requirements.py
b/lnst/Controller/Requirements.py
index 342454f..ed61996 100644 --- a/lnst/Controller/Requirements.py +++ b/lnst/Controller/Requirements.py @@ -1,19 +1,19 @@ """ -This module defines the DeviceReq and HostReq classes, which can be used
to
+This module defines the
:py:class:`lnst.Controller.Requirements.DeviceReq` and
+:py:class:`lnst.Controller.Requirements.HostReq` classes, which can be
used to
create a global description of Requirements for a network test. You can
use
-these to define class attributes of a BaseRecipe derived class to specify +these to define class attributes of a +:py:class:`lnst.Controller.Recipe.BaseRecipe` derived class to specify "general" requirements for that Recipe, or you can add them to an
instance of a
Recipe derived class based on it's parameters to define requirements
"specific"
for that single test run.
-The module also specifies a Requirements class which serves as a
container for
-HostReq objects, while HostReq classes also serve as containers for
DeviceReq
-objects. The object tree created this way is translated to a dictionary
used by
-the internal LNST matching algorithm against available machines.
-Copyright 2017 Red Hat, Inc. -Licensed under the GNU General Public License, version 2 as -published by the Free Software Foundation; see COPYING for details. +The module also specifies a +:py:class:`lnst.Controller.Requirements._Requirements` class (currently
for
+internal use only) which serves as a container for HostReq objects, while +HostReq classes also serve as containers for DeviceReq objects. The
object tree
+created this way is translated to a dictionary used by the internal LNST +matching algorithm against available machines. """
__author__ = """ @@ -63,15 +63,22 @@ class HostReq(BaseReq):
To define a Host requirement you assign a HostReq instance to a class attribute of a BaseRecipe derived class.
- Example:
- class MyRecipe(BaseRecipe):
m1 = HostReq()- Args:
kwargs -- any other arguments will be treated as arbitrary stringparameters that will be matched to parameters of Slavemachines
which can define their parameter values based on theimplementation
of the SlaveMachineParser
- :param kwargs:
any argument will be treated as arbitrary string parameters thatwill
be matched to parameters of Slave machines which can define theirparameter values based on the implementation of theSlaveMachineParser
A special case is the use of a:py:mod:`lnst.Controller.Requirements.RecipeParam` instance asvalue.
This is used to link to a value provided as a Parameter to theRecipe.
- :type kwargs: Dict[str, Any]
- Example::
class MyRecipe(BaseRecipe):m1 = HostReq()m2 = HostReq(architecture="x86_64")@@ -93,21 +100,32 @@ class HostReq(BaseReq): return res
class DeviceReq(BaseReq):
- """Specifies an Ethernet Device requirement
- """Specifies a static test network Device requirement
- This will be used to find a matching test machine in the configured
slave
- machine pools, specifically this will be used to match against a test
- device on a slave machine that is "statically" present on the
machine. In
- other words an actual REAL network device connected to a network
usable for
testing.
To define a Device requirement you assign a DeviceReq instance to a
HostReq
instance in a BaseRecipe derived class.
- Example:
- class MyRecipe(BaseRecipe):
m1 = HostReq()m1.eth0 = DeviceReq(label="net1")- Args:
label -- string value indicating the network the Device isconnected to
kwargs -- any other arguments will be treated as arbitrary stringparameters that will be matched to parameters of Slavemachines
which can define their parameter values based on theimplementation
of the SlaveMachineParser
- :param label:
string value indicating the network the Device is connected to- :type label: string
- :param kwargs:
any other arguments will be treated as arbitrary stringparameters that
will be matched to parameters of Slave machines which can definetheir
parameter values based on the implementation of theSlaveMachineParser
- :type kwargs: Dict[str, Any]
- Example::
class MyRecipe(BaseRecipe):m1 = HostReq()m1.eth0 = DeviceReq(label="net1")diff --git a/lnst/Controller/__init__.py b/lnst/Controller/__init__.py index 106bb91..cb0392d 100644 --- a/lnst/Controller/__init__.py +++ b/lnst/Controller/__init__.py @@ -1,3 +1,10 @@ +""" +Controller package +==================
+This package exposes public facing APIs that can be used to create
Recipes, as
+well as executable test scripts. +""" from lnst.Controller.Controller import Controller from lnst.Controller.Recipe import BaseRecipe from lnst.Controller.Requirements import HostReq, DeviceReq, RecipeParam diff --git a/lnst/Recipes/ENRT/BaseEnrtRecipe.py
b/lnst/Recipes/ENRT/BaseEnrtRecipe.py
index a90c837..d139546 100644 --- a/lnst/Recipes/ENRT/BaseEnrtRecipe.py +++ b/lnst/Recipes/ENRT/BaseEnrtRecipe.py @@ -2,7 +2,7 @@ import pprint from contextlib import contextmanager
from lnst.Common.LnstError import LnstError -from lnst.Common.Parameters import Param, IntParam, StrParam, BoolParam,
ListParam
+from lnst.Common.Parameters import Param, IntParam, StrParam, BoolParam,
ListParam, FloatParam
from lnst.Common.IpAddress import AF_INET, AF_INET6
from lnst.Recipes.ENRT.ConfigMixins.BaseSubConfigMixin import
BaseSubConfigMixin
@@ -16,10 +16,150 @@ from lnst.RecipeCommon.Perf.Measurements import
StatCPUMeasurement
from lnst.RecipeCommon.Perf.Evaluators import NonzeroFlowEvaluator
class EnrtConfiguration(object):
- """Container object for configuration
- Intentionally left empty as it is intended to be used as a container
to
- store any values relevant to configuration being applied during the
lifetime
- of the Recipe.
- """ pass
class BaseEnrtRecipe(BaseSubConfigMixin, PingTestAndEvaluate,
PerfRecipe):
- #common requirements parameters
- """Base Recipe class for the ENRT recipe package
- This class defines the shared *test* method defining the common test
- procedure in a very generic way. This common test procedure involves
a
- single main *test_wide* configuration that is different for every
specific
- scenario. After the main configuration there is usually a loop of
several
- minor *sub* configrations types that can take different values to
slightly
- change the tested use cases.
- Finally, for each combination of a **test_wide** + **sub**
configuration we
- do a several ping connection test and several performance
measurement tests.
- **test_wide** and **sub** configurations are implemented with
**context
- manager** methods which ensure that if any exceptions are raised (for
- example because of a bug in the recipe) that deconfiguration is
called.
- Both **test_wide** and **sub** configurations are to be implemented
in
- different classes, the BaseEnrtRecipe class only defines the common
API and
- the base versions of the relevant methods.
- Test wide configuration is implemented via the following methods:
- :any:`test_wide_configuration`
- :any:`test_wide_deconfiguration`
- :any:`generate_test_wide_description`
- Sub configurations are **mixed in** to classes defining the specific
- scenario that is being tested. Various sub configurations are
implemented as
- individual Python **Mixin** classes in the
- :any:`ConfigMixins<config_mixins>` package. These make use of Pythons
- collaborative inheritance by calling the `super` function in a
specific way.
- The "machinery" for that defined in the :any:`BaseSubConfigMixin`
class and
- used in this class from the `test` method loop that generates the
combined
- Sub configuration and the `_sub_context` context manager method that
takes
- care of applying and removing the configuration.
The last sentence is way too long and I simply do not understand what is meant here. Could you simplify/split into smaller parts?
- :param driver:
The driver parameter is used to modify the hw networkrequirements,
specifically to request Devices using the specified driver. Thisis
common enough in the Enrt recipes that it can be part of theBase class.
- :type driver: :any:`StrParam` (default "ixgbe")
- :param ip_versions:
Parameter that determines which IP protocol versions will betested.
- :type ip_versions: Tuple[Str] (default ("ipv4", "ipv6"))
- :param ping_parallel:
Parameter used by the :any:`generate_ping_configurations`generator.
Tells the generator method to create :any:`PingConf` objectsthat will
be run in parallel.- :type ping_parallel: :any:`BoolParam` (default False)
- :param ping_bidirect:
Parameter used by the :any:`generate_ping_configurations`generator.
Tells the generator method to create :any:`PingConf` objects forboth
directions between the ping endpoints.- :type ping_bidirect: :any:`BoolParam` (default False)
- :param ping_count:
Parameter used by the :any:`generate_ping_configurations`generator.
Tells the generator how many pings should be sent for each pingtest.
- :type ping_count: :any:`IntParam` (default 100)
- :param ping_interval:
Parameter used by the :any:`generate_ping_configurations`generator.
Tells the generator how fast should the pings be sent in eachping test.
- :type ping_interval: :any:`FloatParam` (default 0.2)
- :param ping_psize:
Parameter used by the :any:`generate_ping_configurations`generator.
Tells the generator how big should the pings packets be in eachping
test.- :type ping_psize: :any:`IntParam` (default None)
- :param perf_tests:
Parameter used by the :any:`generate_flow_combinations`generator.
Tells the generator what types of network flow measurements togenerate
perf test configurations for.- :type perf_tests: Tuple[str] (default ("tcp_stream", "udp_stream",
"sctp_stream"))- :param perf_tool_cpu:
Parameter used by the :any:`generate_flow_combinations`generator. To
indicate that the flow measurement should be pinned to aspecific CPU
core.- :type perf_tool_cpu: :any:`IntParam` (optional parameter)
- :param perf_duration:
Parameter used by the :any:`generate_perf_configurations`generator. To
specify the duration of the performance measurements, in seconds.- :type perf_duration: :any:`IntParam` (default 60)
- :param perf_iterations:
Parameter used by the :any:`generate_perf_configurations`generator. To
specify how many times should each performance measurement berepeated
to generate cumulative results which can be statisticallyanalyzed.
- :type perf_iterations: :any:`IntParam` (default 5)
- :param perf_parallel_streams:
Parameter used by the :any:`generate_flow_combinations`generator. To
specify how many parallel streams of the same network flowshould be
measured at the same time.- :type perf_parallel_streams: :any:`IntParam` (default 1)
- :param perf_msg_sizes:
Parameter used by the :any:`generate_flow_combinations`generator. To
specify what different message sizes (in bytes) used generatedfor the
network flow should be tested - each message size resulting in aseparate performance measurement.- :type perf_msg_sizes: List[Int] (default [123])
- :param perf_reverse:
Parameter used by the :any:`generate_flow_combinations`generator. To
specify that both directions between the endpoints of a networkflow
should be measured for the same test.- :type perf_reverse: :any:`BoolParam` (default False)
- :param net_perf_tool:
Specifies a network flow measurement class that accepts:any:`PerfFlow`
objects and can be used to measure those specified flowsParameter used by the :any:`generate_perf_configurations`generator to
create a PerfRecipeConf object.- :type net_perf_tool: :any:`BaseFlowMeasurement` (default
IperfFlowMeasurement)- :param cpu_perf_tool:
Specifies a cpu measurement class that can be used to measure CPUutilization on specified hosts.Parameter used by the :any:`generate_perf_configurations`generator to
create a PerfRecipeConf object.Swap the two sentences in both net_perf_tool and cpu_perf_tool to match the pattern in other parameters.
- :type cpu_perf_tool: :any:`BaseCPUMeasurement` (default
StatCPUMeasurement)
"""
driver = StrParam(default="ixgbe")
#common test parameters
@@ -45,6 +185,20 @@ class BaseEnrtRecipe(BaseSubConfigMixin,
PingTestAndEvaluate, PerfRecipe):
cpu_perf_tool = Param(default=StatCPUMeasurement) def test(self):
"""Main test loop shared by all the Enrt recipesThe test loop involves a single application of a **test_wide**configuration, then a loop over multiple **sub** configurationsthat
involves:* creating the combined sub configuration of all availableSubConfig
Mixin classes via :any:`generate_sub_configurations`* applying the generated sub configuration via the:any:`_sub_context`
context manager method* running tests* removing the current sub configuration via the:any:`_sub_context`
context manager method""" with self._test_wide_context() as main_config: for sub_config inself.generate_sub_configurations(main_config):
with self._sub_context(sub_config) as recipe_config:@@ -60,19 +214,90 @@ class BaseEnrtRecipe(BaseSubConfigMixin,
PingTestAndEvaluate, PerfRecipe):
self.test_wide_deconfiguration(config) def test_wide_configuration(self):
"""Creates an empty :any:`EnrtConfiguration` objectThis is again used in potential collaborative inheritance designthat
may potentially be useful for Enrt recipes. Derived classes willeach
individually add their own values to the instance created here.This way
the complete test wide configuration is tracked in a singleobject.
:return: returns a config object that tracks the appliedconfiguration
that can be used during testing to inspect the current stateand
make test decisions based on it.:rtype: :any:`EnrtConfiguration`Example::class Derived:def test_wide_configuration():config = super().test_wide_configuration()# ... configure somethingconfig.something = what_was_configuredreturn config""" return EnrtConfiguration()def test_wide_deconfiguration(self, config):
"""Base deconfiguration method.In the base class this should maybe only check if there's anyleftover
configuration and warn about it. In derived classes this can beoverriden to take care of deconfiguring what was configured inthe
respective test_wide_configuration method.Example::class Derived:def test_wide_deconfiguration(config):# ... deconfigure somethingdel config.something #cleanup trackingreturn super().test_wide_deconfiguration()""" #TODO check if anything is still applied and throw exception? returndef describe_test_wide_configuration(self, config):
"""Describes the current test wide configurationCreates a new result object that tracks a *successful* result,but
Not quite sure what 'successful result' means.
contains the description of the full test wide configurationapplied by
all the :any:`test_wide_configuration` methods in the classhierarchy.
The description needs to be generated by the:any:`generate_test_wide_description` method. Additionally thedescription contains the state of all the parameters and theirvalues
passed to the recipe class instance during initialization.""" description = self.generate_test_wide_description(config) self.add_result(True, "Summary of used Recipeparameters:\n{}".format(
pprint.pformat(self.params._to_dict()))) self.add_result(True, "\n".join(description)) def generate_test_wide_description(self, config):
"""Generates the test wide configuration descriptionAnother class inteded to be used with the collaborative versionof the
`super` method to cumulatively desribe the full test wideconfiguration
that was applied through multiple classes.The base class version of this method creates the initial list ofstrings containing just the header line. Each string added tothis list
will later be printed on its own line.:return: list of strings, each representing a single line:rtype: List[str]Example::class Derived:def generate_sub_configuration_description(config):desc =super().generate_sub_configuration_description(config)
desc.append("Configured something:{}".format(config.something))
return desc""" return [ "Testwide configuration for recipe {} description:".format( self.__class__.__name__@@ -93,20 +318,47 @@ class BaseEnrtRecipe(BaseSubConfigMixin,
PingTestAndEvaluate, PerfRecipe):
self.add_result(True, "\n".join(description)) def do_tests(self, recipe_config):
"""Entry point for actual testsThe common scenario is to do ping and performance tests, howeverthe
method can be overriden to add more tests if needed.""" self.do_ping_tests(recipe_config) self.do_perf_tests(recipe_config)def do_ping_tests(self, recipe_config):
"""Ping testing loopLoops over all various ping configurations generated by the:any:`generate_ping_configurations` method, then uses thePingRecipe
methods to execute, report and evaluate the results.""" for ping_config inself.generate_ping_configurations(recipe_config):
result = self.ping_test(ping_config) self.ping_evaluate_and_report(ping_config, result) def do_perf_tests(self, recipe_config):
"""Performance testing loopLoops over all various perf configurations generated by the:any:`generate_perf_configurations` method, then uses thePerfRecipe
methods to execute, report and evaluate the results.""" for perf_config inself.generate_perf_configurations(recipe_config):
result = self.perf_test(perf_config) self.perf_report_and_evaluate(result) def generate_ping_configurations(self, config):
"""Base ping test configuration generatorThe generator loops over all endpoint pairs to test ping between(generated by the :any:`generate_ping_endpoints` method) thenover all
the selected :any:`ip_versions` and finally over all the IPaddresses
that fit those criteria.:return: list of Ping configurations to test in parallel:rtype: List[:any:`PingConf`]""" for endpoint1, endpoint2 in self.generate_ping_endpoints(config): for ipv in self.params.ip_versions: ip_filter = {}@@ -144,9 +396,30 @@ class BaseEnrtRecipe(BaseSubConfigMixin,
PingTestAndEvaluate, PerfRecipe):
yield ping_conf_list def generate_ping_endpoints(self, config):
"""Generator for ping endpointsTo be overriden by a derived class.:return: list of device pairs:rtype: List[Tuple[:any:`Device`, :any:`Device`]]""" return []def generate_perf_configurations(self, config):
"""Base perf test configuration generatorThe generator loops over all flow combinations to measureperformance
for (generated by the :any:`generate_flow_combinations` method).In
addition to that during each flow combination measurement we addCPU
utilization measurement to run on the background.Finally for each generated perf test configuration we registermeasurement evaluators based on the :any:`cpu_perf_evaluators`and
:any:`net_perf_evaluators` properties.:return: list of Perf test configurations:rtype: List[:any:`PerfRecipeConf`]""" for flows in self.generate_flow_combinations(config): perf_recipe_conf=dict( recipe_config=config,@@ -183,6 +456,18 @@ class BaseEnrtRecipe(BaseSubConfigMixin,
PingTestAndEvaluate, PerfRecipe):
yield perf_conf def generate_flow_combinations(self, config):
"""Base flow combination generatorThe generator loops over all endpoint pairs to test performancebetween
(generated by the :any:`generate_perf_endpoints` method) thenover all
the selected :any:`ip_versions` and uses the first IP addressfitting
these criteria. Then the generator loops over the selectedperformance
tests as selected via :any:`perf_tests`, then message sizes from:any:`msg_sizes`.:return: list of Flow combinations to measure in parallel:rtype: List[:any:`PerfFlow`]""" for client_nic, server_nic inself.generate_perf_endpoints(config):
for ipv in self.params.ip_versions: if ipv == "ipv4":@@ -213,14 +498,37 @@ class BaseEnrtRecipe(BaseSubConfigMixin,
PingTestAndEvaluate, PerfRecipe):
yield [reverse_flow] def generate_perf_endpoints(self, config):
"""Generator for perf endpointsTo be overriden by a derived class.:return: list of device pairs:rtype: List[Tuple[:any:`Device`, :any:`Device`]]""" return []@property def cpu_perf_evaluators(self):
"""CPU measurement evaluatorsTo be overriden by a derived class. Returns the list ofevaluators to
use for CPU utilization measurement evaluation.:return: a list of cpu evaluator objects:rtype: List[BaseEvaluator]""" return []@property def net_perf_evaluators(self):
"""Network flow measurement evaluatorsTo be overriden bby a derived class. Returns the list ofevaluators to
use for Network flow measurement evaluation.:return: a list of flow evaluator objects:rtype: List[BaseEvaluator]""" return [NonzeroFlowEvaluator()]def _create_reverse_flow(self, flow):
diff --git a/lnst/Recipes/ENRT/ConfigMixins/README.md
b/lnst/Recipes/ENRT/ConfigMixins/README.md
new file mode 100644 index 0000000..0472be0 --- /dev/null +++ b/lnst/Recipes/ENRT/ConfigMixins/README.md @@ -0,0 +1,3 @@ +# SubConfig Mixins
+TODO diff --git a/lnst/Recipes/ENRT/ConfigMixins/__init__.py
b/lnst/Recipes/ENRT/ConfigMixins/__init__.py
index e69de29..7ae6450 100644 --- a/lnst/Recipes/ENRT/ConfigMixins/__init__.py +++ b/lnst/Recipes/ENRT/ConfigMixins/__init__.py @@ -0,0 +1,9 @@ +""" +ENRT Config Mixins +==================
+.. autoclass::
lnst.Recipes.ENRT.ConfigMixins.BaseSubConfigMixin.BaseSubConfigMixin
+"""
+from lnst.Recipes.ENRT.ConfigMixins.BaseSubConfigMixin import
BaseSubConfigMixin
diff --git a/lnst/Recipes/ENRT/README.md b/lnst/Recipes/ENRT/README.md new file mode 100644 index 0000000..3500cdc --- /dev/null +++ b/lnst/Recipes/ENRT/README.md @@ -0,0 +1,47 @@ +# ENRT Recipes
+ENRT stands for Early Network Regression Testing, which was the name of
the
+project a couple of years back when we started developing this set of
tests
+when we were still using the Legacy LNST API.
+This package aims to reimplement the same set of tests, using the new
LNST-next
+APIs, while fully utilizing Python to address the largest problems with
the old
+implementation: +* large amounts of code duplication +* copy paste errors caused by code duplication +* very hard to maintain the test set and fix bugs +* very hard to add new tests due to limitations of the old LNST Framework
+With that in mind, the main goals for this reimplementation were as
follows:
+* reduce code duplication as much as possible by utilizing class
inheritance
+* separate individual types of configurations into smaller Mixin classes
that
- can be mixed and matched based on what a specific Test scenario
requires
+* writing new test scenarios should be very quick because it should be
possible
- to reuse most of the functionality already defined
+* it should be possible to very easily extend the recipes with new
features such
- as more types of parallel or sequential measurements, more types of
- Evaluations for different types of measurements, or to be able to
switch out
- and use different measurement tools
+The resulting design is split into several parts that interact with each
other
+in a specific way:
+* [BaseEnrtRecipe](BaseEnrtRecipe.py) serves as the base class for all
the
- specific test scenarios we want to test. Defines the common test loop
and the
- default implementation for configuration generators used in this
common test
- loop.
+* [ConfigMixins](ConfigMixins/README.md) package contains the various
types of
- "SubConfig" mixin classes, these are classes that implement some form
of
- configuration on test machine(s) which can be reused between mutliple
recipes,
- but can often times be looped over to try different variations of the
- configuration. A good example is configuration of hardware offloads,
it's
- relevant to test it for many different test scenarios, but only some
- combinations of offloads make sense based on the scenario and we often
time
- want to test more than one combination
+* Specific test scenario implementation that defines the requirements
and the
- main configuration for the specific scenario that we want to test. It
also
- defines the combination of various "SubConfig" configurations that we
want to
- include by adding them to it's inheritance tree. If required the
recipe can
- also override any of the default functionality defined by it's parent
classes,
- a good example could be the generator method for creating various
- configurations for flow performance measurement.
diff --git a/lnst/Recipes/ENRT/SimpleNetworkRecipe.py
b/lnst/Recipes/ENRT/SimpleNetworkRecipe.py
index 911a996..2c8e12b 100644 --- a/lnst/Recipes/ENRT/SimpleNetworkRecipe.py +++ b/lnst/Recipes/ENRT/SimpleNetworkRecipe.py @@ -12,6 +12,25 @@ from
lnst.Recipes.ENRT.ConfigMixins.CommonHWSubConfigMixin import (
class SimpleNetworkRecipe( CommonHWSubConfigMixin, OffloadSubConfigMixin, BaseEnrtRecipe ):
- """
- This recipe implements Enrt testing for a simple network scenario
that looks
- as follows
- .. code-block::
+--------++------+ switch +-----+| +--------+ |+--+-+ +-+--++-|eth0|-+ +-|eth0|-+| +----+ | | +----+ || host1 | | host2 |+--------+ +--------+- All sub configurations are included via Mixin classes.
- The actual test machinery is implemented in the
:any:`BaseEnrtRecipe` class.
- """ host1 = HostReq() host1.eth0 = DeviceReq(label="net1", driver=RecipeParam("driver"))
@@ -26,6 +45,14 @@ class SimpleNetworkRecipe( dict(gro="on", gso="on", tso="on", tx="on", rx="off")))
def test_wide_configuration(self):
"""Test wide configuration for this recipe involves just adding anIPv4 and
IPv6 address to the matched eth0 nics on both hosts.host1.eth0 = 192.168.101.1/24 and fc00::1/64host2.eth0 = 192.168.101.2/24 and fc00::2/64""" host1, host2 = self.matched.host1, self.matched.host2 configuration = super().test_wide_configuration() configuration.test_wide_devices = []@@ -41,6 +68,9 @@ class SimpleNetworkRecipe( return configuration
def generate_test_wide_description(self, config):
"""Test wide description is extended with the configured addresses""" desc = super().generate_test_wide_description(config) desc += [ "Configured {}.{}.ips = {}".format(@@ -51,14 +81,33 @@ class SimpleNetworkRecipe( return desc
def test_wide_deconfiguration(self, config):
"" # overriding the parent docstring del config.test_wide_devices super().test_wide_deconfiguration(config)def generate_ping_endpoints(self, config):
"""The ping endpoints for this recipe are simply the two matchedNICs:
host1.eth0 and host2.eth0Returned as::[(self.matched.host1.eth0, self.matched.host2.eth0)]""" return [(self.matched.host1.eth0, self.matched.host2.eth0)]def generate_perf_endpoints(self, config):
"""The perf endpoints for this recipe are simply the two matchedNICs:
host1.eth0 and host2.eth0Returned as::[(self.matched.host1.eth0, self.matched.host2.eth0)]""" return [(self.matched.host1.eth0, self.matched.host2.eth0)]def wait_tentative_ips(self, devices):
diff --git a/lnst/Recipes/ENRT/__init__.py b/lnst/Recipes/ENRT/__init__.py index ea5aaf0..a591ef2 100644 --- a/lnst/Recipes/ENRT/__init__.py +++ b/lnst/Recipes/ENRT/__init__.py @@ -1,3 +1,55 @@ +''' +ENRT Recipes +------------
+ENRT stands for Early Network Regression Testing, which was the name of
the
+project a couple of years back when we started developing this set of
tests
+when we were still using the Legacy LNST API.
+This package aims to reimplement the same set of tests, using the new
LNST-next
+APIs, while fully utilizing Python to address the largest problems with
the old
+implementation:
+* large amounts of code duplication +* copy paste errors caused by code duplication +* very hard to maintain the test set and fix bugs +* very hard to add new tests due to limitations of the old LNST Framework
+With that in mind, the main goals for this reimplementation were as
follows:
+* reduce code duplication as much as possible by utilizing class
inheritance
+* separate individual types of configurations into smaller Mixin classes
that
- can be mixed and matched based on what a specific Test scenario
requires
+* writing new test scenarios should be very quick because it should be
possible
- to reuse most of the functionality already defined
+* it should be possible to very easily extend the recipes with new
features
- such as more types of parallel or sequential measurements, more types
of
- Evaluations for different types of measurements, or to be able to
switch out
- and use different measurement tools
+The resulting design is split into several parts that interact with each
other
+in a specific way:
+* BaseEnrtRecipe serves as the base class for all the specific test
scenarios
- we want to test. Defines the common test loop and the default
implementation
- for configuration generators used in this common test loop.
+* ConfigMixins package contains the various types of "SubConfig" mixin
classes,
- these are classes that implement some form of configuration on test
- machine(s) which can be reused between mutliple recipes, but can often
times
- be looped over to try different variations of the configuration. A good
- example is configuration of hardware offloads, it's relevant to test
it for
- many different test scenarios, but only some combinations of offloads
make
- sense based on the scenario and we often time want to test more than
one
- combination
+* Specific test scenario implementation that defines the requirements
and the
- main configuration for the specific scenario that we want to test. It
also
- defines the combination of various "SubConfig" configurations that we
want to
- include by adding them to it's inheritance tree. If required the
recipe can
- also override any of the default functionality defined by it's parent
- classes, a good example could be the generator method for creating
various
- configurations for flow performance measurement.
+''' from lnst.Recipes.ENRT.SimpleNetworkRecipe import SimpleNetworkRecipe from lnst.Recipes.ENRT.BondRecipe import BondRecipe from lnst.Recipes.ENRT.DoubleBondRecipe import DoubleBondRecipe -- 2.25.1 _______________________________________________ LNST-developers mailing list -- lnst-developers@lists.fedorahosted.org To unsubscribe send an email to
lnst-developers-leave@lists.fedorahosted.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives:
https://lists.fedorahosted.org/archives/list/lnst-developers@lists.fedorahos... _______________________________________________ LNST-developers mailing list -- lnst-developers@lists.fedorahosted.org To unsubscribe send an email to lnst-developers-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/lnst-developers@lists.fedorahos...
On Tue, Mar 17, 2020 at 02:43:51PM -0400, Perry Gagne wrote:
I got an error because sphinx wasn't installed. Do we want to add it to requirements.txt (or perhaps something like requirements_extra.txt?)
Not sure, it's not a dependency for running LNST... tried googling it and it seems it's not standard to do that:
https://stackoverflow.com/questions/18426342/should-i-include-sphinx-and-or-...
It might be worth just noting it in the main README file that documentation is located in the docs/ directory, and that building it requires sphinx to be installed. Does that sound ok?
-Ondrej
lnst-developers@lists.fedorahosted.org
-
Jan Tluka -
olichtne＠redhat.com -
Ondrej Lichtner
-
Perry Gagne