It is some time now that we are discussing an eventual repository reorganization for vdsm. In fact I'm sure that we all experienced at least once the discomfort of having several modules scattered around the tree. The main goal of the reorganization would be to place the modules in their proper location so that they can be used (imported) without any special change (or hack) even when the code is executed inside the development repository (e.g. tests).
Recently there has been an initial proposal about moving some of these modules:
http://gerrit.ovirt.org/#/c/11858/
That spawned an interesting discussion that must involve the entire community; in fact before starting any work we should try to converge on a decision for the final repository structure in order to minimize the discomfort for the contributors that will be forced to rebase their pending gerrit patches. Even if the full reorganization won't happen in a short time I think we should plan the entire structure now and then eventually move only few relevant modules to their final location.
To start the discussion I'm attaching here a sketch of the vdsm repository structure that I envision:
. |-- client | |-- [...] | `-- vdsClient.py |-- common | |-- [...] | |-- betterPopen | | `-- [...] | `-- vdsm | |-- [...] | `-- config.py |-- contrib | |-- [...] | |-- nfs-check.py | `-- sos |-- daemon | |-- [...] | |-- supervdsm.py | `-- vdsmd `-- tool |-- [...] `-- vdsm-tool
This would allow any component (daemon, client, tool, etc...) to run in place with the only addition of PYTHONPATH=$(top_srcdir)/common. The tests would need also the additional component path but that is a given since it's what they should be testing.
Once the components are in the proper place we can eventually start using distutils inside the Makefile.am files in order to simplify the installation process (and also as verification that our repository structure is complying with the python standards).
Please get involved, share your feedback and proposals. Thanks,
On Mon, Feb 11, 2013 at 12:17:39PM -0500, Federico Simoncelli wrote:
It is some time now that we are discussing an eventual repository reorganization for vdsm. In fact I'm sure that we all experienced at least once the discomfort of having several modules scattered around the tree. The main goal of the reorganization would be to place the modules in their proper location so that they can be used (imported) without any special change (or hack) even when the code is executed inside the development repository (e.g. tests).
Recently there has been an initial proposal about moving some of these modules:
http://gerrit.ovirt.org/#/c/11858/
That spawned an interesting discussion that must involve the entire community; in fact before starting any work we should try to converge on a decision for the final repository structure in order to minimize the discomfort for the contributors that will be forced to rebase their pending gerrit patches. Even if the full reorganization won't happen in a short time I think we should plan the entire structure now and then eventually move only few relevant modules to their final location.
To start the discussion I'm attaching here a sketch of the vdsm repository structure that I envision:
. |-- client | |-- [...] | `-- vdsClient.py |-- common | |-- [...] | |-- betterPopen | | `-- [...] | `-- vdsm | |-- [...] | `-- config.py |-- contrib | |-- [...] | |-- nfs-check.py | `-- sos |-- daemon | |-- [...] | |-- supervdsm.py | `-- vdsmd `-- tool |-- [...] `-- vdsm-tool
Wondering if common should be named lib since you'll most likely want to install it to the python lib dir anyway. Minor bikeshedding.
Why the distinction between client and tool? Do you expect other scripts to be added and give them all their own directory? I'd expect that most of their code would mostly live in vdsm and the actual executables are thin wrappers. This would match up with the scripts dir distutils has.
Where do the hooks go?
This would allow any component (daemon, client, tool, etc...) to run in place with the only addition of PYTHONPATH=$(top_srcdir)/common. The tests would need also the additional component path but that is a given since it's what they should be testing.
+1 on fewer hacks to load the environment.
Once the components are in the proper place we can eventually start using distutils inside the Makefile.am files in order to simplify the installation process (and also as verification that our repository structure is complying with the python standards).
+1 IMHO use of auto* should be minimized when developing python.
On Tue, Feb 12, 2013 at 01:58:01AM +0100, Ewoud Kohl van Wijngaarden wrote:
On Mon, Feb 11, 2013 at 12:17:39PM -0500, Federico Simoncelli wrote:
It is some time now that we are discussing an eventual repository reorganization for vdsm. In fact I'm sure that we all experienced at least once the discomfort of having several modules scattered around the tree. The main goal of the reorganization would be to place the modules in their proper location so that they can be used (imported) without any special change (or hack) even when the code is executed inside the development repository (e.g. tests).
Recently there has been an initial proposal about moving some of these modules:
http://gerrit.ovirt.org/#/c/11858/
That spawned an interesting discussion that must involve the entire community; in fact before starting any work we should try to converge on a decision for the final repository structure in order to minimize the discomfort for the contributors that will be forced to rebase their pending gerrit patches. Even if the full reorganization won't happen in a short time I think we should plan the entire structure now and then eventually move only few relevant modules to their final location.
To start the discussion I'm attaching here a sketch of the vdsm repository structure that I envision:
. |-- client | |-- [...] | `-- vdsClient.py |-- common | |-- [...] | |-- betterPopen | | `-- [...] | `-- vdsm | |-- [...] | `-- config.py |-- contrib | |-- [...] | |-- nfs-check.py | `-- sos |-- daemon | |-- [...] | |-- supervdsm.py | `-- vdsmd `-- tool |-- [...] `-- vdsm-tool
Wondering if common should be named lib since you'll most likely want to install it to the python lib dir anyway. Minor bikeshedding.
Why the distinction between client and tool? Do you expect other scripts to be added and give them all their own directory? I'd expect that most of their code would mostly live in vdsm and the actual executables are thin wrappers. This would match up with the scripts dir distutils has.
vdsm-tool is to be shipped with the server only. It is a tool that helps the admin (and the sysv/systemd service) configure the host for vdsm. It's code is part of common (or lib) only to make it easier to use it with the default sys.path.
Where do the hooks go?
They, too, are part of the daemon, but I think they can stay as a top-level directory as they are now.
This would allow any component (daemon, client, tool, etc...) to run in place with the only addition of PYTHONPATH=$(top_srcdir)/common. The tests would need also the additional component path but that is a given since it's what they should be testing.
+1 on fewer hacks to load the environment.
Once the components are in the proper place we can eventually start using distutils inside the Makefile.am files in order to simplify the installation process (and also as verification that our repository structure is complying with the python standards).
+1 IMHO use of auto* should be minimized when developing python. _______________________________________________ vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel
On Mon, Feb 11, 2013 at 12:17:39PM -0500, Federico Simoncelli wrote:
It is some time now that we are discussing an eventual repository reorganization for vdsm. In fact I'm sure that we all experienced at least once the discomfort of having several modules scattered around the tree. The main goal of the reorganization would be to place the modules in their proper location so that they can be used (imported) without any special change (or hack) even when the code is executed inside the development repository (e.g. tests).
Recently there has been an initial proposal about moving some of these modules:
http://gerrit.ovirt.org/#/c/11858/
That spawned an interesting discussion that must involve the entire community; in fact before starting any work we should try to converge on a decision for the final repository structure in order to minimize the discomfort for the contributors that will be forced to rebase their pending gerrit patches. Even if the full reorganization won't happen in a short time I think we should plan the entire structure now and then eventually move only few relevant modules to their final location.
To start the discussion I'm attaching here a sketch of the vdsm repository structure that I envision:
. |-- client | |-- [...] | `-- vdsClient.py |-- common | |-- [...] | |-- betterPopen | | `-- [...] | `-- vdsm | |-- [...] | `-- config.py |-- contrib | |-- [...] | |-- nfs-check.py | `-- sos |-- daemon | |-- [...] | |-- supervdsm.py | `-- vdsmd `-- tool |-- [...] `-- vdsm-tool
The schema file vdsmapi-schema.json (as well as the python module to parse it) are needed by the server and clients. Initially I'd think it should be installed in 'common', but a client does not need things like betterPopen. Any recommendation on where the schema/API definition should live?
----- Original Message -----
From: "Adam Litke" agl@us.ibm.com To: "Federico Simoncelli" fsimonce@redhat.com Cc: "VDSM Project Development" vdsm-devel@lists.fedorahosted.org, "Dan Kenigsberg" danken@redhat.com, "Saggi Mizrahi" smizrahi@redhat.com, "Vinzenz Feenstra" vfeenstr@redhat.com, "Ayal Baron" abaron@redhat.com Sent: Tuesday, February 12, 2013 9:08:09 PM Subject: Re: VDSM Repository Reorganization
On Mon, Feb 11, 2013 at 12:17:39PM -0500, Federico Simoncelli wrote:
It is some time now that we are discussing an eventual repository reorganization for vdsm. In fact I'm sure that we all experienced at least once the discomfort of having several modules scattered around the tree. The main goal of the reorganization would be to place the modules in their proper location so that they can be used (imported) without any special change (or hack) even when the code is executed inside the development repository (e.g. tests).
Recently there has been an initial proposal about moving some of these modules:
http://gerrit.ovirt.org/#/c/11858/
That spawned an interesting discussion that must involve the entire community; in fact before starting any work we should try to converge on a decision for the final repository structure in order to minimize the discomfort for the contributors that will be forced to rebase their pending gerrit patches. Even if the full reorganization won't happen in a short time I think we should plan the entire structure now and then eventually move only few relevant modules to their final location.
To start the discussion I'm attaching here a sketch of the vdsm repository structure that I envision:
. |-- client | |-- [...] | `-- vdsClient.py |-- common | |-- [...] | |-- betterPopen | | `-- [...] | `-- vdsm | |-- [...] | `-- config.py |-- contrib | |-- [...] | |-- nfs-check.py | `-- sos |-- daemon | |-- [...] | |-- supervdsm.py | `-- vdsmd `-- tool |-- [...] `-- vdsm-tool
The schema file vdsmapi-schema.json (as well as the python module to parse it) are needed by the server and clients. Initially I'd think it should be installed in 'common', but a client does not need things like betterPopen. Any recommendation on where the schema/API definition should live?
The problem is that betterPopen is misplaced as it's used only by the daemon.
Anyway let's not mix the three different aspects:
- repository structure - installation location - packaging
That said in my opinion it can remain in "common" (which can be renamed to "lib" as Ewoud suggested) and be installed in the site-lib *but* be packaged only with the daemon (while we're waiting for it to become fully independent and be moved out of the repo).
I suppose that the code dealing with the schema will be shared by both the client and the daemon and it will be placed in "common" (or "lib") and the schema itself can go there too.
To be honest I started envisioning client/common/daemon/tool as pure-python directories (as they are once installed) so maybe the schema could be placed somewhere else with other miscellaneous files (./schema? ./config?), but I wouldn't stress over this if it's impractical.
First lets illustrate the current repository structure: - tests - vdsm - storage - gluster - betterPopen ... and more unused directories - vdsm_api - vdsm_cli - vdsm_hooks - vdsm_log - vdsm_reg - vdsm_tools - vds_bootstrap - contrib
As I understand your proposal, you added 'common' and 'deamon' and left the rest the way it is. So vdsm can be renamed to vdsm_core and can include sub folders for deamon and common\lib python files. It'll be something like: - vdsm_core - deamon - vdsmd .. - more deamon scripts - lib - independent tools -> can also be part of vdsm_tools, as it doesn't part of the vdsm_core - utils.py - betterPopen - dmidecode ... - more unrelated utils that vdsm uses - core - vdsm - API.py - clientIF.py - config.py defines.py constants.py ... should be part of vdsm_core - libvirtconnection.py ... core files of vdsm - storage ... as it now without files that can be and should be moved to lib folder (e.g. threadPool, task, taskManager and inc..)
the rest we'll stay the same.
about the installation location it something else.. In my opinion this arrangement is nice to have, but not a must - we'll still need to set PYTHONPATH when we run from different locations (what you called making an hack), so as I see it, it just for comfort.. no? Maybe I missed something..
(And tests can be split to subfolders for each part - core\tests lib\tests and inc..)
----- Original Message ----- From: "Federico Simoncelli" fsimonce@redhat.com To: "Adam Litke" agl@us.ibm.com Cc: "VDSM Project Development" vdsm-devel@lists.fedorahosted.org Sent: Wednesday, February 13, 2013 1:06:55 AM Subject: Re: [vdsm] VDSM Repository Reorganization
----- Original Message -----
From: "Adam Litke" agl@us.ibm.com To: "Federico Simoncelli" fsimonce@redhat.com Cc: "VDSM Project Development" vdsm-devel@lists.fedorahosted.org, "Dan Kenigsberg" danken@redhat.com, "Saggi Mizrahi" smizrahi@redhat.com, "Vinzenz Feenstra" vfeenstr@redhat.com, "Ayal Baron" abaron@redhat.com Sent: Tuesday, February 12, 2013 9:08:09 PM Subject: Re: VDSM Repository Reorganization
On Mon, Feb 11, 2013 at 12:17:39PM -0500, Federico Simoncelli wrote:
It is some time now that we are discussing an eventual repository reorganization for vdsm. In fact I'm sure that we all experienced at least once the discomfort of having several modules scattered around the tree. The main goal of the reorganization would be to place the modules in their proper location so that they can be used (imported) without any special change (or hack) even when the code is executed inside the development repository (e.g. tests).
Recently there has been an initial proposal about moving some of these modules:
http://gerrit.ovirt.org/#/c/11858/
That spawned an interesting discussion that must involve the entire community; in fact before starting any work we should try to converge on a decision for the final repository structure in order to minimize the discomfort for the contributors that will be forced to rebase their pending gerrit patches. Even if the full reorganization won't happen in a short time I think we should plan the entire structure now and then eventually move only few relevant modules to their final location.
To start the discussion I'm attaching here a sketch of the vdsm repository structure that I envision:
. |-- client | |-- [...] | `-- vdsClient.py |-- common | |-- [...] | |-- betterPopen | | `-- [...] | `-- vdsm | |-- [...] | `-- config.py |-- contrib | |-- [...] | |-- nfs-check.py | `-- sos |-- daemon | |-- [...] | |-- supervdsm.py | `-- vdsmd `-- tool |-- [...] `-- vdsm-tool
The schema file vdsmapi-schema.json (as well as the python module to parse it) are needed by the server and clients. Initially I'd think it should be installed in 'common', but a client does not need things like betterPopen. Any recommendation on where the schema/API definition should live?
The problem is that betterPopen is misplaced as it's used only by the daemon.
Anyway let's not mix the three different aspects:
- repository structure - installation location - packaging
That said in my opinion it can remain in "common" (which can be renamed to "lib" as Ewoud suggested) and be installed in the site-lib *but* be packaged only with the daemon (while we're waiting for it to become fully independent and be moved out of the repo).
I suppose that the code dealing with the schema will be shared by both the client and the daemon and it will be placed in "common" (or "lib") and the schema itself can go there too.
To be honest I started envisioning client/common/daemon/tool as pure-python directories (as they are once installed) so maybe the schema could be placed somewhere else with other miscellaneous files (./schema? ./config?), but I wouldn't stress over this if it's impractical.
On 02/13/2013 03:08 PM, Yaniv Bronheim wrote:
First lets illustrate the current repository structure:
- tests
- vdsm
- storage
- gluster
- betterPopen
... and more unused directories
- vdsm_api
- vdsm_cli
- vdsm_hooks
- vdsm_log
- vdsm_reg
- vdsm_tools
- vds_bootstrap
- contrib
As I understand your proposal, you added 'common' and 'deamon' and left the rest the way it is. So vdsm can be renamed to vdsm_core and can include sub folders for deamon and common\lib python files. It'll be something like:
- vdsm_core
- deamon
.. - more deamon scripts
- vdsmd
- lib - independent tools -> can also be part of vdsm_tools, as it doesn't part of the vdsm_core
... - more unrelated utils that vdsm uses
- utils.py
- betterPopen
- dmidecode
- core
... core files of vdsm
- vdsm
- API.py
- clientIF.py
- config.py defines.py constants.py ... should be part of vdsm_core
- libvirtconnection.py
- storage ... as it now without files that can be and should be moved to lib folder (e.g. threadPool, task, taskManager and inc..)
the rest we'll stay the same.
about the installation location it something else.. In my opinion this arrangement is nice to have, but not a must - we'll still need to set PYTHONPATH when we run from different locations (what you called making an hack), so as I see it, it just for comfort.. no? Maybe I missed something..
(And tests can be split to subfolders for each part - core\tests lib\tests and inc..)
----- Original Message ----- From: "Federico Simoncelli" fsimonce@redhat.com To: "Adam Litke" agl@us.ibm.com Cc: "VDSM Project Development" vdsm-devel@lists.fedorahosted.org Sent: Wednesday, February 13, 2013 1:06:55 AM Subject: Re: [vdsm] VDSM Repository Reorganization
----- Original Message -----
From: "Adam Litke" agl@us.ibm.com To: "Federico Simoncelli" fsimonce@redhat.com Cc: "VDSM Project Development" vdsm-devel@lists.fedorahosted.org, "Dan Kenigsberg" danken@redhat.com, "Saggi Mizrahi" smizrahi@redhat.com, "Vinzenz Feenstra" vfeenstr@redhat.com, "Ayal Baron" abaron@redhat.com Sent: Tuesday, February 12, 2013 9:08:09 PM Subject: Re: VDSM Repository Reorganization
On Mon, Feb 11, 2013 at 12:17:39PM -0500, Federico Simoncelli wrote:
It is some time now that we are discussing an eventual repository reorganization for vdsm. In fact I'm sure that we all experienced at least once the discomfort of having several modules scattered around the tree. The main goal of the reorganization would be to place the modules in their proper location so that they can be used (imported) without any special change (or hack) even when the code is executed inside the development repository (e.g. tests).
Recently there has been an initial proposal about moving some of these modules:
http://gerrit.ovirt.org/#/c/11858/
That spawned an interesting discussion that must involve the entire community; in fact before starting any work we should try to converge on a decision for the final repository structure in order to minimize the discomfort for the contributors that will be forced to rebase their pending gerrit patches. Even if the full reorganization won't happen in a short time I think we should plan the entire structure now and then eventually move only few relevant modules to their final location.
To start the discussion I'm attaching here a sketch of the vdsm repository structure that I envision:
. |-- client | |-- [...] | `-- vdsClient.py |-- common | |-- [...] | |-- betterPopen | | `-- [...] | `-- vdsm | |-- [...] | `-- config.py |-- contrib | |-- [...] | |-- nfs-check.py | `-- sos |-- daemon | |-- [...] | |-- supervdsm.py | `-- vdsmd `-- tool |-- [...] `-- vdsm-tool
The schema file vdsmapi-schema.json (as well as the python module to parse it) are needed by the server and clients. Initially I'd think it should be installed in 'common', but a client does not need things like betterPopen. Any recommendation on where the schema/API definition should live?
The problem is that betterPopen is misplaced as it's used only by the daemon.
Anyway let's not mix the three different aspects:
+1
- repository structure
Well that's what is supposed to be the topic.
- installation location
- packaging
Those are supposed to be off topic considerint $SUBJECT
That said in my opinion it can remain in "common" (which can be renamed to "lib" as Ewoud suggested) and be installed in the site-lib *but* be packaged only with the daemon (while we're waiting for it to become fully independent and be moved out of the repo).
Where such things could be put in a folder 'extra' for now. (As an example)
I suppose that the code dealing with the schema will be shared by both the client and the daemon and it will be placed in "common" (or "lib") and the schema itself can go there too.
To be honest I started envisioning client/common/daemon/tool as pure-python directories (as they are once installed) so maybe the schema could be placed somewhere else with other miscellaneous files (./schema? ./config?), but I wouldn't stress over this if it's impractical.
I would like to see also pure python directories.
A rough quick n dirty overview how I could imagine it to look like:
topdir/ build-aux/* contrib/* deployment/ bootstrap/ log/ reg/ doc/ vdsm api cli hooks/* m4/* tests/* scripts/ vdsm/* storage/* network/* vdsm/ api/ json/ jsonrpc/ BindingJsonRpc.py xml/ BindingXMLRPC.py Bridge.py vdsmapi-schema.json process-schema.py vdsmapi.py cli/ vdsClient.py VdsClientGluster.py extra/ betterPopen/ shared/ vdsm/ SecureXMLRPCServer.py config.py.in constants.py.in utils.py vdscli.py.in* utils.py tool/ daemon/ storage/* network/* vdsmd/ vdsm *
That said, I need to assert that I am not yet fully aware of all connections between the files and I guess the maintainers will know best about their respective parts of responsibility.
I'd like to add that it'd be probably a good idea to start having network and storage split off the main daemon for maintenance reasons, even if they interface each other. I personally even could see a scenario where network and storage are working as standalone daemons. However that's off topic for know I guess.
Hi everyone,
It would be nice to come to an agreement any time soon. I would like to apply all the change as soon as possible. I would not like to see this go into the depth of a dead hole.
Thanks.
On 02/13/2013 03:54 PM, Vinzenz Feenstra wrote:
On 02/13/2013 03:08 PM, Yaniv Bronheim wrote:
First lets illustrate the current repository structure:
- tests
- vdsm
- storage
- gluster
- betterPopen
... and more unused directories
- vdsm_api
- vdsm_cli
- vdsm_hooks
- vdsm_log
- vdsm_reg
- vdsm_tools
- vds_bootstrap
- contrib
As I understand your proposal, you added 'common' and 'deamon' and left the rest the way it is. So vdsm can be renamed to vdsm_core and can include sub folders for deamon and common\lib python files. It'll be something like:
- vdsm_core
- deamon
.. - more deamon scripts
- vdsmd
- lib - independent tools -> can also be part of vdsm_tools, as it
doesn't part of the vdsm_core - utils.py - betterPopen - dmidecode ... - more unrelated utils that vdsm uses
- core
... core files of vdsm
- vdsm
- API.py
- clientIF.py
- config.py defines.py constants.py ... should be part of vdsm_core
- libvirtconnection.py
- storage ... as it now without files that can be and should be moved to
lib folder (e.g. threadPool, task, taskManager and inc..)
the rest we'll stay the same.
about the installation location it something else.. In my opinion this arrangement is nice to have, but not a must - we'll still need to set PYTHONPATH when we run from different locations (what you called making an hack), so as I see it, it just for comfort.. no? Maybe I missed something..
(And tests can be split to subfolders for each part - core\tests lib\tests and inc..)
----- Original Message ----- From: "Federico Simoncelli" fsimonce@redhat.com To: "Adam Litke" agl@us.ibm.com Cc: "VDSM Project Development" vdsm-devel@lists.fedorahosted.org Sent: Wednesday, February 13, 2013 1:06:55 AM Subject: Re: [vdsm] VDSM Repository Reorganization
----- Original Message -----
From: "Adam Litke" agl@us.ibm.com To: "Federico Simoncelli" fsimonce@redhat.com Cc: "VDSM Project Development" vdsm-devel@lists.fedorahosted.org, "Dan Kenigsberg" danken@redhat.com, "Saggi Mizrahi" smizrahi@redhat.com, "Vinzenz Feenstra" vfeenstr@redhat.com, "Ayal Baron" abaron@redhat.com Sent: Tuesday, February 12, 2013 9:08:09 PM Subject: Re: VDSM Repository Reorganization
On Mon, Feb 11, 2013 at 12:17:39PM -0500, Federico Simoncelli wrote:
It is some time now that we are discussing an eventual repository reorganization for vdsm. In fact I'm sure that we all experienced at least once the discomfort of having several modules scattered around the tree. The main goal of the reorganization would be to place the modules in their proper location so that they can be used (imported) without any special change (or hack) even when the code is executed inside the development repository (e.g. tests).
Recently there has been an initial proposal about moving some of these modules:
http://gerrit.ovirt.org/#/c/11858/
That spawned an interesting discussion that must involve the entire community; in fact before starting any work we should try to converge on a decision for the final repository structure in order to minimize the discomfort for the contributors that will be forced to rebase their pending gerrit patches. Even if the full reorganization won't happen in a short time I think we should plan the entire structure now and then eventually move only few relevant modules to their final location.
To start the discussion I'm attaching here a sketch of the vdsm repository structure that I envision:
. |-- client | |-- [...] | `-- vdsClient.py |-- common | |-- [...] | |-- betterPopen | | `-- [...] | `-- vdsm | |-- [...] | `-- config.py |-- contrib | |-- [...] | |-- nfs-check.py | `-- sos |-- daemon | |-- [...] | |-- supervdsm.py | `-- vdsmd `-- tool |-- [...] `-- vdsm-tool
The schema file vdsmapi-schema.json (as well as the python module to parse it) are needed by the server and clients. Initially I'd think it should be installed in 'common', but a client does not need things like betterPopen. Any recommendation on where the schema/API definition should live?
The problem is that betterPopen is misplaced as it's used only by the daemon.
Anyway let's not mix the three different aspects:
+1
- repository structure
Well that's what is supposed to be the topic.
- installation location
- packaging
Those are supposed to be off topic considerint $SUBJECT
That said in my opinion it can remain in "common" (which can be renamed to "lib" as Ewoud suggested) and be installed in the site-lib *but* be packaged only with the daemon (while we're waiting for it to become fully independent and be moved out of the repo).
Where such things could be put in a folder 'extra' for now. (As an example)
I suppose that the code dealing with the schema will be shared by both the client and the daemon and it will be placed in "common" (or "lib") and the schema itself can go there too.
To be honest I started envisioning client/common/daemon/tool as pure-python directories (as they are once installed) so maybe the schema could be placed somewhere else with other miscellaneous files (./schema? ./config?), but I wouldn't stress over this if it's impractical.
I would like to see also pure python directories.
A rough quick n dirty overview how I could imagine it to look like:
topdir/ build-aux/* contrib/* deployment/ bootstrap/ log/ reg/ doc/ vdsm api cli hooks/* m4/* tests/* scripts/ vdsm/* storage/* network/* vdsm/ api/ json/ jsonrpc/ BindingJsonRpc.py xml/ BindingXMLRPC.py Bridge.py vdsmapi-schema.json process-schema.py vdsmapi.py cli/ vdsClient.py VdsClientGluster.py extra/ betterPopen/ shared/ vdsm/ SecureXMLRPCServer.py config.py.in constants.py.in utils.py vdscli.py.in* utils.py tool/ daemon/ storage/* network/* vdsmd/ vdsm *
That said, I need to assert that I am not yet fully aware of all connections between the files and I guess the maintainers will know best about their respective parts of responsibility.
I'd like to add that it'd be probably a good idea to start having network and storage split off the main daemon for maintenance reasons, even if they interface each other. I personally even could see a scenario where network and storage are working as standalone daemons. However that's off topic for know I guess.
Hi Vinzenz,
On 02/18/2013 05:43 PM, Vinzenz Feenstra wrote:
It would be nice to come to an agreement any time soon. I would like to apply all the change as soon as possible. I would not like to see this go into the depth of a dead hole.
The absence of feedback typically means one of 6 things:
* No-one read it * No-one understood it * No-one cared * No-one is paying attention to you any more * It's so ridiculous it's unworthy of comment * Everyone who read it agrees
I suggest you assume the last one, and proceed until someone starts shouting at you ;-)
Thanks, Dave.
于 2013-2-13 22:54, Vinzenz Feenstra 写道:
On 02/13/2013 03:08 PM, Yaniv Bronheim wrote:
First lets illustrate the current repository structure:
- tests
- vdsm
- storage
- gluster
- betterPopen
... and more unused directories
- vdsm_api
- vdsm_cli
- vdsm_hooks
- vdsm_log
- vdsm_reg
- vdsm_tools
- vds_bootstrap
- contrib
As I understand your proposal, you added 'common' and 'deamon' and left the rest the way it is. So vdsm can be renamed to vdsm_core and can include sub folders for deamon and common\lib python files. It'll be something like:
- vdsm_core
- deamon
.. - more deamon scripts
- vdsmd
- lib - independent tools -> can also be part of vdsm_tools, as it
doesn't part of the vdsm_core - utils.py - betterPopen - dmidecode ... - more unrelated utils that vdsm uses
- core
... core files of vdsm
- vdsm
- API.py
- clientIF.py
- config.py defines.py constants.py ... should be part of vdsm_core
- libvirtconnection.py
- storage ... as it now without files that can be and should be moved to
lib folder (e.g. threadPool, task, taskManager and inc..)
the rest we'll stay the same.
about the installation location it something else.. In my opinion this arrangement is nice to have, but not a must - we'll still need to set PYTHONPATH when we run from different locations (what you called making an hack), so as I see it, it just for comfort.. no? Maybe I missed something..
(And tests can be split to subfolders for each part - core\tests lib\tests and inc..)
----- Original Message ----- From: "Federico Simoncelli" fsimonce@redhat.com To: "Adam Litke" agl@us.ibm.com Cc: "VDSM Project Development" vdsm-devel@lists.fedorahosted.org Sent: Wednesday, February 13, 2013 1:06:55 AM Subject: Re: [vdsm] VDSM Repository Reorganization
----- Original Message -----
From: "Adam Litke" agl@us.ibm.com To: "Federico Simoncelli" fsimonce@redhat.com Cc: "VDSM Project Development" vdsm-devel@lists.fedorahosted.org, "Dan Kenigsberg" danken@redhat.com, "Saggi Mizrahi" smizrahi@redhat.com, "Vinzenz Feenstra" vfeenstr@redhat.com, "Ayal Baron" abaron@redhat.com Sent: Tuesday, February 12, 2013 9:08:09 PM Subject: Re: VDSM Repository Reorganization
On Mon, Feb 11, 2013 at 12:17:39PM -0500, Federico Simoncelli wrote:
It is some time now that we are discussing an eventual repository reorganization for vdsm. In fact I'm sure that we all experienced at least once the discomfort of having several modules scattered around the tree. The main goal of the reorganization would be to place the modules in their proper location so that they can be used (imported) without any special change (or hack) even when the code is executed inside the development repository (e.g. tests).
Recently there has been an initial proposal about moving some of these modules:
http://gerrit.ovirt.org/#/c/11858/
That spawned an interesting discussion that must involve the entire community; in fact before starting any work we should try to converge on a decision for the final repository structure in order to minimize the discomfort for the contributors that will be forced to rebase their pending gerrit patches. Even if the full reorganization won't happen in a short time I think we should plan the entire structure now and then eventually move only few relevant modules to their final location.
To start the discussion I'm attaching here a sketch of the vdsm repository structure that I envision:
. |-- client | |-- [...] | `-- vdsClient.py |-- common | |-- [...] | |-- betterPopen | | `-- [...] | `-- vdsm | |-- [...] | `-- config.py |-- contrib | |-- [...] | |-- nfs-check.py | `-- sos |-- daemon | |-- [...] | |-- supervdsm.py | `-- vdsmd `-- tool |-- [...] `-- vdsm-tool
The schema file vdsmapi-schema.json (as well as the python module to parse it) are needed by the server and clients. Initially I'd think it should be installed in 'common', but a client does not need things like betterPopen. Any recommendation on where the schema/API definition should live?
The problem is that betterPopen is misplaced as it's used only by the daemon.
Anyway let's not mix the three different aspects:
+1
- repository structure
Well that's what is supposed to be the topic.
- installation location
- packaging
Those are supposed to be off topic considerint $SUBJECT
That said in my opinion it can remain in "common" (which can be renamed to "lib" as Ewoud suggested) and be installed in the site-lib *but* be packaged only with the daemon (while we're waiting for it to become fully independent and be moved out of the repo).
Where such things could be put in a folder 'extra' for now. (As an example)
I suppose that the code dealing with the schema will be shared by both the client and the daemon and it will be placed in "common" (or "lib") and the schema itself can go there too.
To be honest I started envisioning client/common/daemon/tool as pure-python directories (as they are once installed) so maybe the schema could be placed somewhere else with other miscellaneous files (./schema? ./config?), but I wouldn't stress over this if it's impractical.
I would like to see also pure python directories.
A rough quick n dirty overview how I could imagine it to look like:
topdir/ build-aux/* contrib/* deployment/ bootstrap/ log/ reg/ doc/ vdsm api cli hooks/* m4/* tests/* scripts/ vdsm/* storage/* network/* vdsm/ api/ json/ jsonrpc/ BindingJsonRpc.py xml/ BindingXMLRPC.py Bridge.py vdsmapi-schema.json process-schema.py vdsmapi.py cli/ vdsClient.py VdsClientGluster.py extra/ betterPopen/ shared/ vdsm/ SecureXMLRPCServer.py config.py.in constants.py.in utils.py vdscli.py.in* utils.py tool/ daemon/ storage/* network/* vdsmd/ vdsm *
I would like to make all of the service under vdsm/daemon directory only reference the files under vdsm/shared. And back-reference from shared/vdsm to vdsm/daemon is not allowed. In current VDSM repository, the files under vdsm/storage and the files under vdsm are cross-referenced on each other, that is a disaster to the effort trying to make stand-alone service from storage service outside of vdsmd. In addition, the files under vdsm/cli should also only depend on the files under vdsm/shared.
BTW: Another service under vdsm/daemon should be supervdsm, so the daemon should like: ... daemon/ storage/* network/* vdsmd/ supervdsmd/ ...
That said, I need to assert that I am not yet fully aware of all connections between the files and I guess the maintainers will know best about their respective parts of responsibility.
I'd like to add that it'd be probably a good idea to start having network and storage split off the main daemon for maintenance reasons, even if they interface each other. I personally even could see a scenario where network and storage are working as standalone daemons. However that's off topic for know I guess.
----- Original Message -----
From: "Adam Litke" agl@us.ibm.com To: "Federico Simoncelli" fsimonce@redhat.com Cc: "VDSM Project Development" vdsm-devel@lists.fedorahosted.org, "Dan Kenigsberg" danken@redhat.com, "Saggi Mizrahi" smizrahi@redhat.com, "Vinzenz Feenstra" vfeenstr@redhat.com, "Ayal Baron" abaron@redhat.com Sent: Tuesday, February 12, 2013 3:08:09 PM Subject: Re: VDSM Repository Reorganization
On Mon, Feb 11, 2013 at 12:17:39PM -0500, Federico Simoncelli wrote:
It is some time now that we are discussing an eventual repository reorganization for vdsm. In fact I'm sure that we all experienced at least once the discomfort of having several modules scattered around the tree. The main goal of the reorganization would be to place the modules in their proper location so that they can be used (imported) without any special change (or hack) even when the code is executed inside the development repository (e.g. tests).
Recently there has been an initial proposal about moving some of these modules:
http://gerrit.ovirt.org/#/c/11858/
That spawned an interesting discussion that must involve the entire community; in fact before starting any work we should try to converge on a decision for the final repository structure in order to minimize the discomfort for the contributors that will be forced to rebase their pending gerrit patches. Even if the full reorganization won't happen in a short time I think we should plan the entire structure now and then eventually move only few relevant modules to their final location.
To start the discussion I'm attaching here a sketch of the vdsm repository structure that I envision:
. |-- client | |-- [...] | `-- vdsClient.py |-- common | |-- [...] | |-- betterPopen | | `-- [...] | `-- vdsm | |-- [...] | `-- config.py |-- contrib | |-- [...] | |-- nfs-check.py | `-- sos |-- daemon | |-- [...] | |-- supervdsm.py | `-- vdsmd `-- tool |-- [...] `-- vdsm-tool
The schema file vdsmapi-schema.json (as well as the python module to parse it) are needed by the server and clients. Initially I'd think it should be installed in 'common', but a client does not need things like betterPopen. Any recommendation on where the schema/API definition should live?
Well they both should have the file but when installed both should have their own version of the file depending on the version installed of the client or the server. This is so that vdsm-cli doesn't depend on vdsm or vice-versa. You can't have them share the file since if one is installed with a version of the schema where the schema syntax changed the client\server will fail to parse the schema.
As for development, I think the least bad solution is to put it in contrib with symlinks that have relative paths. |--daemon | |-- [...] | `-- vdsmapi-schema.json -> ../contrib/vdsmapi-schema.json |--client | |-- [...] | `-- vdsmapi-schema.json -> ../contrib/vdsmapi-schema.json |--contrib | |-- [...] | `-- vdsmapi-schema.json : .
Git knows how to handle symlinks and symlinks are relative to the location of the symlink.
We could also just select the daemon or the client folder and put the real file there and a symlink in the other but I feel it's like choosing which one of your children is the main user of a schema file.
-- Adam Litke agl@us.ibm.com IBM Linux Technology Center
----- Original Message -----
From: "Saggi Mizrahi" smizrahi@redhat.com To: "Adam Litke" agl@us.ibm.com Cc: "VDSM Project Development" vdsm-devel@lists.fedorahosted.org, "Dan Kenigsberg" danken@redhat.com, "Vinzenz Feenstra" vfeenstr@redhat.com, "Ayal Baron" abaron@redhat.com, "Federico Simoncelli" fsimonce@redhat.com Sent: Monday, February 18, 2013 8:50:30 PM Subject: Re: VDSM Repository Reorganization
----- Original Message -----
From: "Adam Litke" agl@us.ibm.com To: "Federico Simoncelli" fsimonce@redhat.com Cc: "VDSM Project Development" vdsm-devel@lists.fedorahosted.org, "Dan Kenigsberg" danken@redhat.com, "Saggi Mizrahi" smizrahi@redhat.com, "Vinzenz Feenstra" vfeenstr@redhat.com, "Ayal Baron" abaron@redhat.com Sent: Tuesday, February 12, 2013 3:08:09 PM Subject: Re: VDSM Repository Reorganization
On Mon, Feb 11, 2013 at 12:17:39PM -0500, Federico Simoncelli wrote:
It is some time now that we are discussing an eventual repository reorganization for vdsm. In fact I'm sure that we all experienced at least once the discomfort of having several modules scattered around the tree. The main goal of the reorganization would be to place the modules in their proper location so that they can be used (imported) without any special change (or hack) even when the code is executed inside the development repository (e.g. tests).
Recently there has been an initial proposal about moving some of these modules:
http://gerrit.ovirt.org/#/c/11858/
That spawned an interesting discussion that must involve the entire community; in fact before starting any work we should try to converge on a decision for the final repository structure in order to minimize the discomfort for the contributors that will be forced to rebase their pending gerrit patches. Even if the full reorganization won't happen in a short time I think we should plan the entire structure now and then eventually move only few relevant modules to their final location.
To start the discussion I'm attaching here a sketch of the vdsm repository structure that I envision:
. |-- client | |-- [...] | `-- vdsClient.py |-- common | |-- [...] | |-- betterPopen | | `-- [...] | `-- vdsm | |-- [...] | `-- config.py |-- contrib | |-- [...] | |-- nfs-check.py | `-- sos |-- daemon | |-- [...] | |-- supervdsm.py | `-- vdsmd `-- tool |-- [...] `-- vdsm-tool
The schema file vdsmapi-schema.json (as well as the python module to parse it) are needed by the server and clients. Initially I'd think it should be installed in 'common', but a client does not need things like betterPopen. Any recommendation on where the schema/API definition should live?
Well they both should have the file but when installed both should have their own version of the file depending on the version installed of the client or the server. This is so that vdsm-cli doesn't depend on vdsm or vice-versa. You can't have them share the file since if one is installed with a version of the schema where the schema syntax changed the client\server will fail to parse the schema.
I'm not sure what's the purpose of having different versions of the client/server on the same machine. The software repository is one and it should provide both (as they're built from the same source). This is the standard way of delivering client/server applications in all the distributions. We can change that but we must have a good reason.
As for development, I think the least bad solution is to put it in contrib with symlinks that have relative paths.
|--daemon | |-- [...] | `-- vdsmapi-schema.json -> ../contrib/vdsmapi-schema.json |--client | |-- [...] | `-- vdsmapi-schema.json -> ../contrib/vdsmapi-schema.json |--contrib | |-- [...] | `-- vdsmapi-schema.json : .
Git knows how to handle symlinks and symlinks are relative to the location of the symlink.
+1.
----- Original Message -----
From: "Federico Simoncelli" fsimonce@redhat.com To: "Saggi Mizrahi" smizrahi@redhat.com Cc: "VDSM Project Development" vdsm-devel@lists.fedorahosted.org, "Dan Kenigsberg" danken@redhat.com, "Vinzenz Feenstra" vfeenstr@redhat.com, "Ayal Baron" abaron@redhat.com, "Adam Litke" agl@us.ibm.com Sent: Tuesday, February 19, 2013 11:27:59 AM Subject: Re: VDSM Repository Reorganization
----- Original Message -----
From: "Saggi Mizrahi" smizrahi@redhat.com To: "Adam Litke" agl@us.ibm.com Cc: "VDSM Project Development" vdsm-devel@lists.fedorahosted.org, "Dan Kenigsberg" danken@redhat.com, "Vinzenz Feenstra" vfeenstr@redhat.com, "Ayal Baron" abaron@redhat.com, "Federico Simoncelli" fsimonce@redhat.com Sent: Monday, February 18, 2013 8:50:30 PM Subject: Re: VDSM Repository Reorganization
----- Original Message -----
From: "Adam Litke" agl@us.ibm.com To: "Federico Simoncelli" fsimonce@redhat.com Cc: "VDSM Project Development" vdsm-devel@lists.fedorahosted.org, "Dan Kenigsberg" danken@redhat.com, "Saggi Mizrahi" smizrahi@redhat.com, "Vinzenz Feenstra" vfeenstr@redhat.com, "Ayal Baron" abaron@redhat.com Sent: Tuesday, February 12, 2013 3:08:09 PM Subject: Re: VDSM Repository Reorganization
On Mon, Feb 11, 2013 at 12:17:39PM -0500, Federico Simoncelli wrote:
It is some time now that we are discussing an eventual repository reorganization for vdsm. In fact I'm sure that we all experienced at least once the discomfort of having several modules scattered around the tree. The main goal of the reorganization would be to place the modules in their proper location so that they can be used (imported) without any special change (or hack) even when the code is executed inside the development repository (e.g. tests).
Recently there has been an initial proposal about moving some of these modules:
http://gerrit.ovirt.org/#/c/11858/
That spawned an interesting discussion that must involve the entire community; in fact before starting any work we should try to converge on a decision for the final repository structure in order to minimize the discomfort for the contributors that will be forced to rebase their pending gerrit patches. Even if the full reorganization won't happen in a short time I think we should plan the entire structure now and then eventually move only few relevant modules to their final location.
To start the discussion I'm attaching here a sketch of the vdsm repository structure that I envision:
. |-- client | |-- [...] | `-- vdsClient.py |-- common | |-- [...] | |-- betterPopen | | `-- [...] | `-- vdsm | |-- [...] | `-- config.py |-- contrib | |-- [...] | |-- nfs-check.py | `-- sos |-- daemon | |-- [...] | |-- supervdsm.py | `-- vdsmd `-- tool |-- [...] `-- vdsm-tool
The schema file vdsmapi-schema.json (as well as the python module to parse it) are needed by the server and clients. Initially I'd think it should be installed in 'common', but a client does not need things like betterPopen. Any recommendation on where the schema/API definition should live?
Well they both should have the file but when installed both should have their own version of the file depending on the version installed of the client or the server. This is so that vdsm-cli doesn't depend on vdsm or vice-versa. You can't have them share the file since if one is installed with a version of the schema where the schema syntax changed the client\server will fail to parse the schema.
I'm not sure what's the purpose of having different versions of the client/server on the same machine. The software repository is one and it should provide both (as they're built from the same source). This is the standard way of delivering client/server applications in all the distributions. We can change that but we must have a good reason.
There isn't really a reason. But, as I said, you don't want them to depend on each other or have the schema in it's own rpm. This means that you have to distribute them separately.
I also want to allow to update the client on a host without updating the server. This is because you may want to have a script that works across the cluster but not update all the hosts.
Now, even though you will use only old methods, the schema itself might become unparsable by old implementations.
As for development, I think the least bad solution is to put it in contrib with symlinks that have relative paths.
|--daemon | |-- [...] | `-- vdsmapi-schema.json -> ../contrib/vdsmapi-schema.json |--client | |-- [...] | `-- vdsmapi-schema.json -> ../contrib/vdsmapi-schema.json |--contrib | |-- [...] | `-- vdsmapi-schema.json : .
Git knows how to handle symlinks and symlinks are relative to the location of the symlink.
+1.
-- Federico
On Tue, Feb 19, 2013 at 03:53:46PM -0500, Saggi Mizrahi wrote:
I'm not sure what's the purpose of having different versions of the client/server on the same machine. The software repository is one and it should provide both (as they're built from the same source). This is the standard way of delivering client/server applications in all the distributions. We can change that but we must have a good reason.
There isn't really a reason. But, as I said, you don't want them to depend on each other or have the schema in it's own rpm. This means that you have to distribute them separately.
I also want to allow to update the client on a host without updating the server. This is because you may want to have a script that works across the cluster but not update all the hosts.
Now, even though you will use only old methods, the schema itself might become unparsable by old implementations.
This should never happen. Right now each "symbol" in the schema is represented by a single OrderedDict and the parsing code just loads the schema file into the a list of these dicts. Once loaded, the vdsmapi module categorizes symbols according to the top-level keys. Unrecognized symbol types are simply skipped.
In order to make progress on the file reorg, I want to summarize the discussion and propose that a consensus has been reached regarding placement of the schema file.
The current code has a routine find_schema() that can locate the schema file in the development source tree or in an installed location. Therefore, it only needs to appear in the source tree in a single location and we will not need any symlinks for this purpose. Recently, the API handling code (schema and parsing module) have been split into their own rpm. This should solve the installation problem since any entity that needs access to the schema and parser should simply install the vdsm-api rpm.
vdsm-devel@lists.stg.fedorahosted.org
-
agl＠us.ibm.com -
Dan Kenigsberg -
Dave Neary -
ewoud＠kohlvanwijngaarden.nl -
Federico Simoncelli -
shuming＠linux.vnet.ibm.com -
smizrahi＠redhat.com -
Vinzenz Feenstra -
ybronhei＠redhat.com