On Tue, Jun 23, 2020 at 11:43:17AM +0200, Adrian Moreno wrote:

On 6/23/20 11:31 AM, Ondrej Lichtner wrote:

...

Hi Adrian,

adding Cc to lnst-developers as the discussed changes may be relevant to a wider audience.

replies inline.

On Tue, Jun 23, 2020 at 10:38:20AM +0200, Adrian Moreno wrote:

...

Hi Ondrej,

I'm having a problem (which I think it's the second time I hit), when moving the TRex library files.

My current file hierarchy

lnst/Tests/ total 64K -rw-r--r--. 1 amorenoz adrian 3.8K Aug 29 2019 BaseTestModule.py -rw-r--r--. 1 amorenoz adrian 3.8K Aug 29 2019 CPUStatMonitor.py -rw-r--r-- 1 amorenoz adrian 636 Jun 23 10:32 __init__.py -rw-r--r--. 1 amorenoz adrian 4.4K Aug 29 2019 Iperf.py -rw-r--r--. 1 amorenoz adrian 23K Aug 29 2019 Netperf.py -rw-r--r--. 1 amorenoz adrian 3.3K Aug 29 2019 PacketAssert.py -rw-r--r-- 1 amorenoz adrian 3.8K Apr 16 18:04 Ping.py -rw-r--r--. 1 amorenoz adrian 1.8K Oct 23 2019 TestPMD.py drwxr-xr-x 3 amorenoz adrian 4.0K Jun 23 09:16 TRex

lnst/Tests/TRex/ total 24K -rw-r--r-- 1 amorenoz adrian 0 Jun 23 08:46 __init__.py -rw-r--r-- 1 amorenoz adrian 1.8K Jun 23 08:33 TestModule.py <- The wrapper -rw-r--r-- 1 amorenoz adrian 7.1K Jun 23 09:07 TRexLib.py <- The library -rw-r--r-- 1 amorenoz adrian 1.4K Jun 23 09:07 UDPMultiflow.py -rw-r--r-- 1 amorenoz adrian 989 Jun 23 09:07 UDPSimple.py

The problem is that when I:

from lnst.Tests.TRex.TRexLib import TRexCli, TRexSrv, TRexParams

I get: Traceback (most recent call last): File "/usr/lib64/python3.6/pdb.py", line 1667, in main pdb._runscript(mainpyfile) File "/usr/lib64/python3.6/pdb.py", line 1548, in _runscript self.run(statement) File "/usr/lib64/python3.6/bdb.py", line 434, in run exec(cmd, globals, locals) File "<string>", line 1, in <module> File "/root/lnst/test_tools/tperf/tperf", line 26, in <module> from lnst.Tests.TRex.TRexLib import TRexCli, TRexSrv, TRexParams File "/root/lnst/test_tools/tperf/../../lnst/Tests/__init__.py", line 15, in

<module> from lnst.Tests.Ping import Ping File "/root/lnst/test_tools/tperf/../../lnst/Tests/Ping.py", line 8, in <module> from lnst.Tests.BaseTestModule import BaseTestModule, TestModuleError File "/root/lnst/test_tools/tperf/../../lnst/Tests/BaseTestModule.py", line 19, in <module> from lnst.Common.Logs import log_exc_traceback File "/root/lnst/test_tools/tperf/../../lnst/Common/Logs.py", line 16, in <module> from lnst.Common.LoggingHandler import TransmitHandler File "/root/lnst/test_tools/tperf/../../lnst/Common/LoggingHandler.py", line 20, in <module> from lnst.Common.ConnectionHandler import send_data File "/root/lnst/test_tools/tperf/../../lnst/Common/ConnectionHandler.py", line 19, in <module> from pyroute2 import IPRSocket ModuleNotFoundError: No module named 'pyroute2'

I checked and we can at least remove the pyroute2 import from the ConnectionHandler, that seems to be unused and probably left-over from when I changed some stuff here...

...

The reason is that

lnst.Tests.__init__.py contains:

from lnst.Tests.Ping import Ping from lnst.Tests.PacketAssert import PacketAssert from lnst.Tests.Iperf import IperfClient, IperfServer

Why is that needed and, can we remove them?

The idea here was so that in a Recipe you can just do:

from lnst.Tests import Ping

Just removing the __init__.py contents will mean that all the recipes that import this way also need to be updated so I'm not sure that's the best approach.

I have just now learned that what I attempted to do here is now called "namespace package", I was also not aware that the __init__.py of a package is called even if you're only importing a subpackage, that seems rather unfortunate... but makes sense from what I see in python docs.

Thinking about it more this looks like it was a bad idea because it makes the test modules "depend on each other" - even if you only want to run Ping, you still need to parse source of all the other modules, in turn requiring all of their dependencies as well.

I would still like to be able to do

from lnst.Tests import Ping, TRexClient

and have it in such a way that only doing

from lnst.Tests import TRexClient

will not even look at the ping source code... I think I'll have to look a bit more into how to do this thing dynamically.

So to improve the situation for now:

• remove pyroute2 from ConnectionHandler.py
• maybe move log_exc_traceback to somewhere where it doesn't end up importing anything wrt. lnst connections at all... maybe lnst.Common.Utils

Would this improve the situation? Do you maybe have some better ideas?

Not really as the whole point of the work I'm doing (related to the RFC I sent recently) is to have lnst/Tests/Trex/TrexLib.py not depend on anything else. So as a workaround I would rather move the LNST-independent files out of lnst/Tests (maybe lnst/External/?).

Other ways could be, have a separate namespace module: lnst/Tests/ __init__.py: from lnst.TestModules.Ping include Ping from lnst.TestModules.TRex include TRexClient lnst/TestModules/ Ping.py TRexClient.py

Thanks

Adrián Moreno

Attaching a patch to lnst/Tests/__init__.py as a rough example of what I had in mind with the lazy loading stuff. I think that's resolves an additional step to the reported issue... the implementation can maybe be improved, though I'm not sure if anything nicer is possible.

I tried to replicate the file structure you described in the original email and printed out what is imported at the top of the tperf main method with:

print(sys.modules.keys())

I then grepped for "lnst" and only the TRexLib hierarchy was imported, this does include the lnst.Tests.TRex package though, so if you were to add anything to the lnst/Tests/TRex/__init__.py file, that would still run.

So even though the patch probably resolves your issues, I'm not sure if it resolves the whole situation. If not, then I'm starting to lean towards the lnst/External/TRexLib* solution.

-Ondrej