Greetings,
This message to the Fedora Development Community is an attempt to start a useful discussion on where Fedora/RHEL is today and where it is going in the future, on the subject of OS management. My apologies in advance for the long message.
OS management is a very broad topic, in specific I will focus on the elements that I consider foundational weaknesses (compared to what it could be). This is not to imply these perceived "weaknesses" out weigh its strengths or that other OS's are overall superior in this regard.
A bit of background to put my technical comments in perspective:
I remember when I was first introduced to Unix (1992) and then later Linux (1993) compared to what I was used to at the time they combined a flexibility, functionality and a form of simplicity (in design) that when combined made all other operating systems seem hallow. For the most part I still feel this way today, but over the years having worked in enterprise environments where Linux is the up and coming challenger to entrenched non-unix platforms (VMS, Windows, zOS, etc...) I have come to better appreciate some of the elements involved in limiting Linux's adoption.
I was recently tasked with standardizing/documenting our Linux server build/configuration process and as part of this process began training some of our Windows/Novell administrators to be able to carry out some of these basic tasks. It was this second part of helping train people that caused me to examine and re-examine the processes I used for this standardization.
My goal was simple in theory, I would use pxe+kickstart and enforce standard configurations and policies via a series of post scripts while attempting to keep readability, simplicity and supportability from a "non-unix guru" perspective as my guiding light (A wise man once said: Make it as simple as possible but no simpler). For those interested an example of what I came up with it can be obtained here: http://geeklords.org/~shane/kickstart
The process turned out to be not quite as simple as the theory but very educational none the less. To start with I had a number of ways I could manipulate my fresh Linux install.
1) I could create a custom RPM that had all of the changes imbedded in config files and rpm scripts that merely overwrote the pre-existing ones. Problem being this approach hides the complexity of what all was being changed and why.
2) Use cfengine after the kickstart install. This of course has some advantages over the rpm method but it also hides the same complexity and adds some of its own.
3) Document on paper all of the steps required and how to perform them from the console, taking advantage of the guis when available or command line when required. I felt that was not only a big waste of time but left far too much to human error.
4) Create a series of simple scripts using basic OS applications/tools (sed, echo, cp, mv, authtool, postconf etc...) to perform all of the required modifications, documenting what and why along the way.
I choose method 4.
My conclusion having completed this process is that the number of variables, complexity and dependencies that must be understood and taken into account when modifying configuration files is much higher than it needs to be. I think this should be clear when reading the example scripts mentioned above.
Some things I learned:
Sed is a wonderful tool, but it is highly limited by the fact its user must take into account whole file(s) for each expression, this is further complicated when one must consider the file may change over time. The complexity and readability of regular expression tools is much higher than should be required to change OS/application variables.
Creating new files or appending to the end of existing files with "echo" only takes one so far. This also tends to have the cost of hurting readability as it is often the case you would prefer putting data somewhere else in the file (i.e. sed).
The nature of flat configuration files where each application has its own format is such where recovery and/or applying changes only if they have not been already applied is too complex and hurts readability far to much to be attempted in a simple shell script.
Many applications have their own command line driven configuration utilities. However each has their own oddities and complexities (authtool, gconftool-2, postconf etc...) Oddly enough out of all of the command line configuration tools I have used, I find gconftool-2 to be closest to The Right Way (TM), which is odd as it is by far one of the most complex syntax's to use (God help anyone who tries to make major edits to gnome xml files using sed) example:
"gconftool-2 --direct --config-source xml:readwrite:/etc/gconf/gconf.xml.mandator y --type float --set /desktop/gnome/peripherals/mouse/motion_acceleration 1"
Even a complex (yet powerful) tool can meet the golden rule of "Make it as simple as possible but no simpler" when compared against a bunch of tools that are less powerful (and perhaps less complex when viewed in isolation).
It is my opinion that we as a community need to find a Better Way (tm) to manage configuration changes for Linux subsystems and applications. Everything should be a file, but that does not mean every configuration file needs to be formatted and managed differently.
In closing, a change this fundamental must be designed by the best minds and must have a strong champion for it to have a chance to succeed. The Fedora community is likely the only Linux entity with the brain power and influence to address this issue.
Cheers, Shane
On 3/25/06, Shane Stixrud shane@geeklords.org wrote:
Greetings,
....
I was recently tasked with standardizing/documenting our Linux server build/configuration process and as part of this process began training some of our Windows/Novell administrators to be able to carry out some of these basic tasks. It was this second part of helping train people that caused me to examine and re-examine the processes I used for this standardization.
My goal was simple in theory, I would use pxe+kickstart and enforce standard configurations and policies via a series of post scripts while attempting to keep readability, simplicity and supportability from a "non-unix guru" perspective as my guiding light (A wise man once said: Make it as simple as possible but no simpler). For those interested an example of what I came up with it can be obtained here: http://geeklords.org/~shane/kickstart
The process turned out to be not quite as simple as the theory but very educational none the less. To start with I had a number of ways I could manipulate my fresh Linux install.
- I could create a custom RPM that had all of the changes imbedded in
config files and rpm scripts that merely overwrote the pre-existing ones. Problem being this approach hides the complexity of what all was being changed and why.
- Use cfengine after the kickstart install. This of course has some
advantages over the rpm method but it also hides the same complexity and adds some of its own.
I am completely missing the point of this letter I realize.. but doesnt your "The Right Way" also hide all the complex things too. It either does that or removes a lot of functionality that some admin needs to complete a job in a slightly different environment... which means enforcing that everyone's enviromental needs matches what you are laying out. I have to cover 8 different field environments and tasks that require that the same tools do something slightly different each time. Now I can build 8 versions of the same tool that reuses 90% of the same code or I can increase the complexity of the each tool to cover all cases.
-- Stephen J Smoogen. CSIRT/Linux System Administrator
I am completely missing the point of this letter I realize.. but doesnt your "The Right Way" also hide all the complex things too. It either does that or removes a lot of functionality that some admin needs to complete a job in a slightly different environment... which means enforcing that everyone's enviromental needs matches what you are laying out. I have to cover 8 different field environments and tasks that require that the same tools do something slightly different each time. Now I can build 8 versions of the same tool that reuses 90% of the same code or I can increase the complexity of the each tool to cover all cases.
I did not say gconftool-2 should be the the solution, only that it shows how a single tool can be very powerful when it has a standard file configuration file format to work with. I am by no means advocating using xml as the standard configuration format, it is way over kill in my opinion.
Nothing is hidden by standardizing config files, I am not implying they should be hidden in a binary file.... nor am I implying they couldn't be edited with sed or vi although that would likely be inefficient when compared to using a tool designed to manage configuration data.
Please provide an example of how this would harm your ability to: "cover 8 different field environments and tasks that require that the same tools do something slightly different each time"
Perhaps you are just confusing my story of what lead to my conclusions with my actual suggestion?
Cheers, Shane
I find it hard to believe no one else sees the current complexity of system/application configuration as undesirable? Is this problem too big to be addressed? Is there too much historical precedence involved? Or is there a solid technical reason for not having a standardized configuration file format?
Linux is perceived to be "difficult" to configure and managed by many, this is not because we lack guis and wizards, rather because one must discover the process each subsystem and application uses to accept configure directives from its users.
I see no reason why it would not be possible and desirable to have a plain text compatible configuration file standard that is universal between subsystems, services and applications (besides it being a lot of work). It seems to me that the complexities of existing configuration tools (authconfig, system-config-samba etc...) are largely due to this same fundamental issue.
Cheers, Shane
On Sat, 25 Mar 2006, Shane Stixrud wrote:
Greetings,
This message to the Fedora Development Community is an attempt to start a useful discussion on where Fedora/RHEL is today and where it is going in the future, on the subject of OS management. My apologies in advance for the long message.
OS management is a very broad topic, in specific I will focus on the elements that I consider foundational weaknesses (compared to what it could be). This is not to imply these perceived "weaknesses" out weigh its strengths or that other OS's are overall superior in this regard.
A bit of background to put my technical comments in perspective:
I remember when I was first introduced to Unix (1992) and then later Linux (1993) compared to what I was used to at the time they combined a flexibility, functionality and a form of simplicity (in design) that when combined made all other operating systems seem hallow. For the most part I still feel this way today, but over the years having worked in enterprise environments where Linux is the up and coming challenger to entrenched non-unix platforms (VMS, Windows, zOS, etc...) I have come to better appreciate some of the elements involved in limiting Linux's adoption.
I was recently tasked with standardizing/documenting our Linux server build/configuration process and as part of this process began training some of our Windows/Novell administrators to be able to carry out some of these basic tasks. It was this second part of helping train people that caused me to examine and re-examine the processes I used for this standardization.
My goal was simple in theory, I would use pxe+kickstart and enforce standard configurations and policies via a series of post scripts while attempting to keep readability, simplicity and supportability from a "non-unix guru" perspective as my guiding light (A wise man once said: Make it as simple as possible but no simpler). For those interested an example of what I came up with it can be obtained here: http://geeklords.org/~shane/kickstart
The process turned out to be not quite as simple as the theory but very educational none the less. To start with I had a number of ways I could manipulate my fresh Linux install.
- I could create a custom RPM that had all of the changes imbedded in config
files and rpm scripts that merely overwrote the pre-existing ones. Problem being this approach hides the complexity of what all was being changed and why.
- Use cfengine after the kickstart install. This of course has some
advantages over the rpm method but it also hides the same complexity and adds some of its own.
- Document on paper all of the steps required and how to perform them from
the console, taking advantage of the guis when available or command line when required. I felt that was not only a big waste of time but left far too much to human error.
- Create a series of simple scripts using basic OS applications/tools (sed,
echo, cp, mv, authtool, postconf etc...) to perform all of the required modifications, documenting what and why along the way.
I choose method 4.
My conclusion having completed this process is that the number of variables, complexity and dependencies that must be understood and taken into account when modifying configuration files is much higher than it needs to be. I think this should be clear when reading the example scripts mentioned above.
Some things I learned:
Sed is a wonderful tool, but it is highly limited by the fact its user must take into account whole file(s) for each expression, this is further complicated when one must consider the file may change over time. The complexity and readability of regular expression tools is much higher than should be required to change OS/application variables.
Creating new files or appending to the end of existing files with "echo" only takes one so far. This also tends to have the cost of hurting readability as it is often the case you would prefer putting data somewhere else in the file (i.e. sed).
The nature of flat configuration files where each application has its own format is such where recovery and/or applying changes only if they have not been already applied is too complex and hurts readability far to much to be attempted in a simple shell script.
Many applications have their own command line driven configuration utilities. However each has their own oddities and complexities (authtool, gconftool-2, postconf etc...) Oddly enough out of all of the command line configuration tools I have used, I find gconftool-2 to be closest to The Right Way (TM), which is odd as it is by far one of the most complex syntax's to use (God help anyone who tries to make major edits to gnome xml files using sed) example:
"gconftool-2 --direct --config-source xml:readwrite:/etc/gconf/gconf.xml.mandator y --type float --set /desktop/gnome/peripherals/mouse/motion_acceleration 1"
Even a complex (yet powerful) tool can meet the golden rule of "Make it as simple as possible but no simpler" when compared against a bunch of tools that are less powerful (and perhaps less complex when viewed in isolation).
It is my opinion that we as a community need to find a Better Way (tm) to manage configuration changes for Linux subsystems and applications. Everything should be a file, but that does not mean every configuration file needs to be formatted and managed differently.
In closing, a change this fundamental must be designed by the best minds and must have a strong champion for it to have a chance to succeed. The Fedora community is likely the only Linux entity with the brain power and influence to address this issue.
Cheers, Shane
Shane Stixrud (shane@geeklords.org) said:
I see no reason why it would not be possible and desirable to have a plain text compatible configuration file standard that is universal between subsystems, services and applications (besides it being a lot of work). It seems to me that the complexities of existing configuration tools (authconfig, system-config-samba etc...) are largely due to this same fundamental issue.
Basically, the problem arises in trying to move the inertia of 50 upstream applications to move to a "new, better" format versus the pain of trying to consistently maintain such patches for 50 applications outside of upstream.
Bill
On Mon, 27 Mar 2006, Bill Nottingham wrote:
Basically, the problem arises in trying to move the inertia of 50 upstream applications to move to a "new, better" format versus the pain of trying to consistently maintain such patches for 50 applications outside of upstream.
I understand that. So the equation seems to be is the effort worth the long term benefits? I can tell you from the meetings and hall way conversations I have been involved in, this issue is a large part of the equation when an enterprise is comparing Windows vs Linux for x86 desktop and server deployments. It is not the command line that turns Netware / Windows administrators off (even if they think it is) it is the lack of consistency and thus the increased complexity involved in configuration the system.
Shane
On Mon, 27 Mar 2006 12:33:50 -0800 (PST) Shane Stixrud shane@geeklords.org wrote:
I find it hard to believe no one else sees the current complexity of system/application configuration as undesirable? Is this problem too big to be addressed? Is there too much historical precedence involved? Or is there a solid technical reason for not having a standardized configuration file format?
Linux is perceived to be "difficult" to configure and managed by many, this is not because we lack guis and wizards, rather because one must discover the process each subsystem and application uses to accept configure directives from its users.
I see no reason why it would not be possible and desirable to have a plain text compatible configuration file standard that is universal between subsystems, services and applications (besides it being a lot of work). It seems to me that the complexities of existing configuration tools (authconfig, system-config-samba etc...) are largely due to this same fundamental issue.
GUI users don't want to be hunting through text files anyway, they want nice settings windows and wizards. Anyone hand editing config files better know what's going on anyway; the current situation isn't too bad there.
gconf already provides a reasonable way to change settings from the command line and via GUI tools. What would you change?
No matter what you come up with though, it will be many years before you see wide spread adoption. If anything, you might consider a project to create a system-wide config editor that knows all the different formats etc and provides a consistent CLI/GUI interface.
Sean
On Mon, 27 Mar 2006, sean wrote:
GUI users don't want to be hunting through text files anyway, they want nice settings windows and wizards. Anyone hand editing config files better know what's going on anyway; the current situation isn't too bad there.
This is not a gui issue, nor is it just an "end user issue". This attitude of "anyone hand editing config files better know what's going on anyways" becomes largely invalid when a standard methodology exists.
gconf already provides a reasonable way to change settings from the command line and via GUI tools. What would you change?
Gconf shows the gnome people realized early on having a standard method for storing and modifying configuration data is important, to the gnome platform... We are not talking about JUST the gnome platform and my guess is gconf would not meet the needs of Fedora as a whole.
No matter what you come up with though, it will be many years before you see wide spread adoption. If anything, you might consider a project to create a system-wide config editor that knows all the different formats etc and provides a consistent CLI/GUI interface.
Projects already exist http://www.libelektra.org/Main_Page for example. The problem is not that code doesn't exist, the problem is one of getting everyone to: a) agree its desirable b) Agreeing/creating an implementation c) having a plan for getting where we want to be "many years" later.
I don't see how a few people can make this happen, it is going to take some serious influence to make any real world progress.
Dnia 27-03-2006, pon o godzinie 13:08 -0800, Shane Stixrud napisał(a):
No matter what you come up with though, it will be many years before you see wide spread adoption. If anything, you might consider a project to create a system-wide config editor that knows all the different formats etc and provides a consistent CLI/GUI interface.
Projects already exist http://www.libelektra.org/Main_Page for example.
No, this concentrates on creating a single API for many programs to store their configs. Parent meant something understanding currently existent configuration formats. Something like that exists, too: http://www.solucorp.qc.ca/linuxconf/
Lam
On Mon, 27 Mar 2006, Leszek Matok wrote:
Dnia 27-03-2006, pon o godzinie 13:08 -0800, Shane Stixrud napisaÿÿ(a):
No matter what you come up with though, it will be many years before you see wide spread adoption. If anything, you might consider a project to create a system-wide config editor that knows all the different formats etc and provides a consistent CLI/GUI interface.
Projects already exist http://www.libelektra.org/Main_Page for example.
No, this concentrates on creating a single API for many programs to store their configs. Parent meant something understanding currently existent configuration formats. Something like that exists, too: http://www.solucorp.qc.ca/linuxconf/
The above project has the concept of "backends" that can be made to understand specific configuration formats. There is an example where Xorg.conf can be imported and exported on demand.
On Mon, 27 Mar 2006 13:33:17 -0800 (PST) Shane Stixrud shane@geeklords.org wrote:
The above project has the concept of "backends" that can be made to understand specific configuration formats. There is an example where Xorg.conf can be imported and exported on demand.
If you think this package is going to be the long term answer, why not start by packaging it up for extras?
Sean
On Mon, 27 Mar 2006, sean wrote:
On Mon, 27 Mar 2006 13:33:17 -0800 (PST) Shane Stixrud shane@geeklords.org wrote:
The above project has the concept of "backends" that can be made to understand specific configuration formats. There is an example where Xorg.conf can be imported and exported on demand.
If you think this package is going to be the long term answer, why not start by packaging it up for extras?
I have not said that. I am interested in the right solution and on something this important and massive I can't see it working without rough community agreement on what that thing would look like. If the community felt elektra was a good direction then I would be more than happy to package it up, although I am sure the authors would beat me too it.
Shane
On Mon, 27 Mar 2006 13:08:57 -0800 (PST) Shane Stixrud shane@geeklords.org wrote:
This is not a gui issue, nor is it just an "end user issue". This attitude of "anyone hand editing config files better know what's going on anyways" becomes largely invalid when a standard methodology exists.
I actually don't buy that. You don't change anything when you go to a standard config file format. Anyone editing at the file/gconf/ registry level had better know what the heck they're doing.
Everyone else wants a nice GUI that is logically consistent with the problem space they're interested in and provides wizards etc to explain the configuration process.
Gconf shows the gnome people realized early on having a standard method for storing and modifying configuration data is important, to the gnome platform... We are not talking about JUST the gnome platform and my guess is gconf would not meet the needs of Fedora as a whole.
And that's really the problem isn't it, how do you get KDE, Gnome, every CLI program, and every other window manager etc to all agree on one format. The way it will happen is gradually everyone will see that someone is winning the fight to provide a backend config system and will use it. This might be a good project for freedesktop.org.
Projects already exist http://www.libelektra.org/Main_Page for example. The problem is not that code doesn't exist, the problem is one of getting everyone to: a) agree its desirable b) Agreeing/creating an implementation c) having a plan for getting where we want to be "many years" later.
But gconf and gconf-2 and others existed before the package you cite. Perhaps this one is the end all and be all of config backends. Dunno.
I don't see how a few people can make this happen, it is going to take some serious influence to make any real world progress.
It might help if the Fedora project "blessed" one particular backend config system, but I suspect even getting that decision made would be a big challenge.
Sean
On Mon, 27 Mar 2006, sean wrote:
This is not a gui issue, nor is it just an "end user issue". This attitude of "anyone hand editing config files better know what's going on anyways" becomes largely invalid when a standard methodology exists.
I actually don't buy that. You don't change anything when you go to a standard config file format. Anyone editing at the file/gconf/ registry level had better know what the heck they're doing.
I do not see it this cut and dry. There is no line between those people who "know what they are doing" and those who do not.
Everyone else wants a nice GUI that is logically consistent with the problem space they're interested in and provides wizards etc to explain the configuration process.
I am not saying wizards and interfaces that present multiple change values via am interrogated interface is not valuable. I am saying their value is greatly amplified by the needless complexities at the lower layers. I am also saying providing these wizards and interfaces are much more difficult to build pragmatically due to these same complexities.
Projects already exist http://www.libelektra.org/Main_Page for example. The problem is not that code doesn't exist, the problem is one of getting
But gconf and gconf-2 and others existed before the package you cite. Perhaps this one is the end all and be all of config backends. Dunno.
A package designed to take into account other systems can help with the transition, Elektra appears to do this for gconf and I think kde's backend is in the works. I am not saying Elektra is or isn't the solution, only that a solution like it is desirable.
On Mon, 27 Mar 2006 13:43:38 -0800 (PST) Shane Stixrud shane@geeklords.org wrote:
I do not see it this cut and dry. There is no line between those people who "know what they are doing" and those who do not.
Well the line is somewhere around the area of those who can look at a config file and figure it out in a minute or two, and those who can't. Anyone who can't deal with today's config files, isn't going to be any better off with a nice gui version of that same process.
Another way of saying this is, it's not the config file format that is the barrier to people configuring their system today. It's the contents of those files and understanding what all those options do and why you might want to use one or the other.
That's why even with Windows where you have a registry you still need configuration screens and wizards etc.
I am not saying wizards and interfaces that present multiple change values via am interrogated interface is not valuable. I am saying their value is greatly amplified by the needless complexities at the lower layers. I am also saying providing these wizards and interfaces are much more difficult to build pragmatically due to these same complexities.
Well you're right that it would be one less thing a developer would have to worry about if their was a common library for dealing with any and all config files.
A package designed to take into account other systems can help with the transition, Elektra appears to do this for gconf and I think kde's backend is in the works. I am not saying Elektra is or isn't the solution, only that a solution like it is desirable.
I hadn't heard of Elektra before you mentioned it and yeah, it looks pretty good.
Sean
On Mon, 27 Mar 2006, sean wrote:
Well the line is somewhere around the area of those who can look at a config file and figure it out in a minute or two, and those who can't. Anyone who can't deal with today's config files, isn't going to be any better off with a nice gui version of that same process.
I would be insulted if I didn't know you were just making a generalized statement :). Who here hasn't spent more than 2 minutes trying to figure a missing period in a named conf file, or more than two minutes setting up ldap for the first time, or more than two minutes figuring out where to find a specific option and value for /etc/modules.conf etc... The problem is each time you touch something you haven't touched in the past one must spend significant time figuring out how to make the change even if they already know what they want to change. This is not true for all configuration files, but it is for many. The amount of neurons I have dedicated to configuration syntax and where the lists of values and their descriptions are stored is many more than I would like.
The "non-initiated" see this as complex and on this specific issue I agree, it is more complex than it needs to be.. I am just used to it and shrug it off.
If for every "key": its default, possible values (on, off string, float etc..) is readily accessible and for every key there is a simple description much of this complexity is eliminated.
On 3/27/06, Shane Stixrud shane@geeklords.org wrote:
If for every "key": its default, possible values (on, off string, float etc..) is readily accessible and for every key there is a simple description much of this complexity is eliminated.
Yes, I think this is what's missing: if for each config file there was a schema things would be better. For example I've pasted below a clip from a schema of a web microsite generator I'm working on. This not only describes the data, it allows me to make a web based editor on the fly, by specifying the appropriate widget to edit an item an and help text.
Joe.
<global-schema> <field id="theme" datatype="enum" name="Theme"> <option id="autumn" name="Autumn" /> <option id="spring" name="Spring" /> <option id="winter" name="Winter" /> <option id="test" name="Test" /> </field> <field id="title" datatype="string" name="Title" helptext="The title of the website. It is used in the header of very page." /> <field id="slogan" datatype="string" name="Slogan" helptext="A slogan or strap line which appears under the site title on every page."/>
<field id="left_image" datatype="image" name="Left block image." helptext="The left block image. 300x200 pixels." />
<field id="custom_css" datatype="string" name="Custom CSS" widget="widget:textarea;rows:8;cols:60" helptext="Custom CSS" />
</global-sc
On Mon, 27 Mar 2006 14:36:11 -0800 (PST) Shane Stixrud shane@geeklords.org wrote:
I would be insulted if I didn't know you were just making a generalized statement :). Who here hasn't spent more than 2 minutes trying to figure a missing period in a named conf file, or more than two minutes setting up ldap for the first time, or more than two minutes figuring out where to find a specific option and value for /etc/modules.conf etc... The problem is each time you touch something you haven't touched in the past one must spend significant time figuring out how to make the change even if they already know what they want to change. This is not true for all configuration files, but it is for many. The amount of neurons I have dedicated to configuration syntax and where the lists of values and their descriptions are stored is many more than I would like.
lol, naw i think you mistook my meaning. I'm not saying that all config editing can be handled in under two minutes. What i am saying though is that no more time than that is needed to comprehend the config file format. Once you've seen one config file, you've seen em all; more or less.
A standard config file format isn't going to lessen the burden if understanding what all those config options etc mean. Even if that config file is presented in a pretty gui gconf-editor like tool.
The "non-initiated" see this as complex and on this specific issue I agree, it is more complex than it needs to be.. I am just used to it and shrug it off.
If for every "key": its default, possible values (on, off string, float etc..) is readily accessible and for every key there is a simple description much of this complexity is eliminated.
I just don't think this is really the significant part of what makes configuring a system difficult. Happy to be proved wrong.
Sean
Le Mar 28 mars 2006 00:36, Shane Stixrud a écrit :
On Mon, 27 Mar 2006, sean wrote:
Well the line is somewhere around the area of those who can look at a config file and figure it out in a minute or two, and those who can't. Anyone who can't deal with today's config files, isn't going to be any better off with a nice gui version of that same process.
I would be insulted if I didn't know you were just making a generalized statement :). Who here hasn't spent more than 2 minutes trying to figure a missing period in a named conf file, or more than two minutes setting up ldap for the first time, or more than two minutes figuring out where to find a specific option and value for /etc/modules.conf etc...
The difference between a traditional conf file and a gconf file is not so much the XML syntax. Rather traditional conf files authors try very hard to find key names which are easy to understand and key values which can be entered by a human, while gconf authors dump garbage in their files since they expect it to be hidden from users and kept "private"
Seems to me, that someone with the drive and IQ needs to develop a system so flexible and easy to use (eg. having prebuilt modules for diff. languages) that many developers with decide to use it instead of rolling their own. I would not advocate XML for this however.
On Tue, 28 Mar 2006 02:43:27 -0600 "Arthur Pemberton" pemboa@gmail.com wrote:
Seems to me, that someone with the drive and IQ needs to develop a system so flexible and easy to use (eg. having prebuilt modules for diff. languages) that many developers with decide to use it instead of rolling their own. I would not advocate XML for this however.
Think you've actually hit the nail on the head Arthur; if a good solution exists developers will use it. Although it might help a bit to get some exposure for the solution on freedesktop.org or other developer sites.
As for XML, with a proper API there's no reason to worry much about the format of the configuration data files. The applications would never look directly inside any config file anyway. Some people might want to use a central SQL server for config files, someone else might want to use traditional /etc flat files etc..
Sean
I think the only way this will happen is for a consortium of interests in the Linux (and similar OSes) world to come up with a formal standard (file format, best practices and API bindings for common languages). Then cast it in concrete by submitting it to a standards body eg EMCA.
If the standard is good then I believe people will use it. Parsing config files is tedious and error prone. Often developers forget about things like international characters and have to change the format after a few releases.
At it's simplest this standard could be a simple name-value pair text file. But it should also cater for complex configurations and allow a schema to define and describe the file (and perhaps the GUI used to edit it).
To ease migration, adapter modules could be written to dynamically translate existing config files to/from the new format.
This would be a huge step forward for Linux. Right now the /etc directory is littered with different formats. Many are not documented outside of the source coded used to read them.
Joe.
On 3/27/06, Shane Stixrud shane@geeklords.org wrote:
On Mon, 27 Mar 2006, sean wrote:
GUI users don't want to be hunting through text files anyway, they want nice settings windows and wizards. Anyone hand editing config files better know what's going on anyway; the current situation isn't too bad there.
This is not a gui issue, nor is it just an "end user issue". This attitude of "anyone hand editing config files better know what's going on anyways" becomes largely invalid when a standard methodology exists.
gconf already provides a reasonable way to change settings from the command line and via GUI tools. What would you change?
Gconf shows the gnome people realized early on having a standard method for storing and modifying configuration data is important, to the gnome platform... We are not talking about JUST the gnome platform and my guess is gconf would not meet the needs of Fedora as a whole.
No matter what you come up with though, it will be many years before you see wide spread adoption. If anything, you might consider a project to create a system-wide config editor that knows all the different formats etc and provides a consistent CLI/GUI interface.
Projects already exist http://www.libelektra.org/Main_Page for example. The problem is not that code doesn't exist, the problem is one of getting everyone to: a) agree its desirable b) Agreeing/creating an implementation c) having a plan for getting where we want to be "many years" later.
I don't see how a few people can make this happen, it is going to take some serious influence to make any real world progress.
-- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
On Mon, 27 Mar 2006 23:20:37 +0100 "Joe Desbonnet" jdesbonnet@gmail.com wrote:
I think the only way this will happen is for a consortium of interests in the Linux (and similar OSes) world to come up with a formal standard (file format, best practices and API bindings for common languages). Then cast it in concrete by submitting it to a standards body eg EMCA.
Sure, or freedesktop.org, freestandards.org etc.. But really all that is needed is a defacto standard based on a prevalent solution. So talking about it and trying to form a committee etc. is likely the long way to a solution.
If the standard is good then I believe people will use it. Parsing config files is tedious and error prone. Often developers forget about things like international characters and have to change the format after a few releases.
No doubt.
At it's simplest this standard could be a simple name-value pair text file. But it should also cater for complex configurations and allow a schema to define and describe the file (and perhaps the GUI used to edit it).
The backend used to store the configuration database isn't really all that important once you have a standard API which can be used by applications and CLI/GUI config editors.
To ease migration, adapter modules could be written to dynamically translate existing config files to/from the new format.
Yup, and the package Shane mentioned (Elektra) has this.
This would be a huge step forward for Linux. Right now the /etc directory is littered with different formats. Many are not documented outside of the source coded used to read them.
I don't think this will actually be a huge step forward or make the difference to anyone deciding whether to embrace Linux or not. But there's no doubt it would be at least nice incremental improvement.
Sean
Shane Stixrud wrote:
I find it hard to believe no one else sees the current complexity of system/application configuration as undesirable?
Just my $0.02, since this is a question which I often hear from Unix newcomers, especially from Windows ... the current complexity is more apparent than real, it still undesiderable, but not to the point that just any apparent solution to said complexity would be adopted willy-nilly. The problem is not that the current situation is optimal, but that all proposed alternatives are subpar. At least as far as I can tell from my personal experience thus far, YMMV.
Is this problem too big to be addressed? Is there too much historical precedence involved? Or is there a solid technical reason for not having a standardized configuration file format?
There isn't, because the problem is not with the technical approach IMHO but in the users, developers and administrators using the system, so the current approach is best *social* compromise for an enviroment where none of the involved parties is able to dictate policy.
Developers don't want to devote resources to configuration file handling, because they force the use of configuration to paper over deficiencies in their code in the first place; administrators don't want to risk a system which they cannot control and fix; users don't want to learn configuration files at all. So users change as little as necessary in textual files which the administrators trust they can fix over a 9600 baud cellphone ssh session in a format the developer never bothered to design in the first place.
Thank you for your consideration, Davide Bolcioni
On 3/26/06, Shane Stixrud shane@geeklords.org wrote:
Sed is a wonderful tool, but it is highly limited by the fact its user must take into account whole file(s) for each expression, this is further complicated when one must consider the file may change over time. The complexity and readability of regular expression tools is much higher than should be required to change OS/application variables.
Creating new files or appending to the end of existing files with "echo" only takes one so far. This also tends to have the cost of hurting readability as it is often the case you would prefer putting data somewhere else in the file (i.e. sed).
The nature of flat configuration files where each application has its own format is such where recovery and/or applying changes only if they have not been already applied is too complex and hurts readability far to much to be attempted in a simple shell script.
One tool I did not see you list is `patch`. A context diff can still work even when the file has changed significantly, and the format is extremely readable (just + or -.) Maintaining a collection of diffs is going to be significantly easier to manage than dealing with a large number of cp/mv/echo/sed commands in shell files. There are also a variety of tools made to deal with diffs and can show them in graphical, friendly ways.
There might be a good reason not to use diff/patch for doing what you want, but I think that it could save a few headaches.
n0dalus.
On Sat, 2006-03-25 at 15:37 -0800, Shane Stixrud wrote:
- I could create a custom RPM that had all of the changes imbedded in
config files and rpm scripts that merely overwrote the pre-existing ones. Problem being this approach hides the complexity of what all was being changed and why.
- Use cfengine after the kickstart install. This of course has some
advantages over the rpm method but it also hides the same complexity and adds some of its own.
- Document on paper all of the steps required and how to perform them
from the console, taking advantage of the guis when available or command line when required. I felt that was not only a big waste of time but left far too much to human error.
- Create a series of simple scripts using basic OS applications/tools
(sed, echo, cp, mv, authtool, postconf etc...) to perform all of the required modifications, documenting what and why along the way.
The problem with (4) is that you mix describing what you want to be done with how it is done, which has a number of disadvantages. One of the goals of config mgmt systems like cfengine is to separate these two concerns. You are right though that cfengine is not entirely trivial to use. I have been using another config mgmt system, puppet, (http://reductivelabs.com/projects/puppet) I wrote up an example of how a config mgmt tool could help in solving the kind of problem you are looking at: http://people.redhat.com/dlutter/puppet-app.html
My conclusion having completed this process is that the number of variables, complexity and dependencies that must be understood and taken into account when modifying configuration files is much higher than it needs to be. I think this should be clear when reading the example scripts mentioned above.
One of the reasons to separate the what from the how: there are complexities in both areas, and using a declarative approach to this problem lets you separate these two kinds of cpmplexities nicely.
It is my opinion that we as a community need to find a Better Way (tm) to manage configuration changes for Linux subsystems and applications.
I completely agree.
Everything should be a file, but that does not mean every configuration file needs to be formatted and managed differently.
While I agree in theory, the reality of config files with different formats makes it extremely hard to get there. One approach that is being pushed by a number of players to achieve uniform management of configs is CIM/WBEM, an enormous standard that IMHO too much tool-specific knowledge to see wide adoption.
A lighter-weight approach seems more promising: encapsulate most of the config-file specific knowledge in simple script wrappers that can be controlled by a declarative description of the configuration you want to achieve and the logical interdependencies between them. This is what puppet does, and why I find it very attractive.
David
On Tue, 28 Mar 2006 17:51:14 -0800 David Lutterkort dlutter@redhat.com wrote:
While I agree in theory, the reality of config files with different formats makes it extremely hard to get there. One approach that is being pushed by a number of players to achieve uniform management of configs is CIM/WBEM, an enormous standard that IMHO too much tool-specific knowledge to see wide adoption.
A lighter-weight approach seems more promising: encapsulate most of the config-file specific knowledge in simple script wrappers that can be controlled by a declarative description of the configuration you want to achieve and the logical interdependencies between them. This is what puppet does, and why I find it very attractive.
Okay, so your script wrappers do all the config-file specific work etc. We talked about making a tool earlier in this conversation that would know how to handle all the different config file formats, so i'm with you up to there.
But what is the advantage of building an entire new config language around these wrappers? Couldn't they be manipulated just as easily with shell or python? Can the administrator make simple ad-hoc command line config file changes?
Sean
On 3/28/06, sean seanlkml@sympatico.ca wrote:
But what is the advantage of building an entire new config language around these wrappers? Couldn't they be manipulated just as easily with shell or python? Can the administrator make simple ad-hoc command line config file changes?
Experience shows that is more difficult and less effective to write a lot of different configuration parsers, then patching the original softwares to make them read configuration atoms from other sources and formats.
Avi
On Tue, 28 Mar 2006, David Lutterkort wrote:
The problem with (4) is that you mix describing what you want to be done with how it is done, which has a number of disadvantages.
I agree, the biggest disadvantages with 4) IMO are:
1) Overhead of keeping the build scripts up to date 2) Determining what should be scripted vs done manually 3) Roll back / change revision is very difficult. 4) This approach can easily cause confusion on "objective" vs "implementation" for a 3rd party.
It is my opinion that in an ideal world the OS/configuration engine would be able to largely eliminate these disadvantages by tracking the history of all configuration changes from inital system build to current state, independent of how the change was made (scripted, distributed, manually at the console, etc...). It seems to me this would be trivial if every configuration element was easily identifiable by path (filesystem), key name (filename) and it's value (content). Obviously there are practical considerations that make this difficult, performance, number of files, what should be a directory vs a file vs content etc...
I have been using another config mgmt system, puppet, (http://reductivelabs.com/projects/puppet) I wrote up an example of how a config mgmt tool could help in solving the kind of problem you are looking at: http://people.redhat.com/dlutter/puppet-app.html
Puppet looks like a very promising centralized change distribution system. As you mentioned this separates the "how" changes are applied from the "what" is actually changing. If you combine the change re-visioning concept mentioned above with a puppet like centralized distribution system that can detect client changes we now have something quite impressive. In fact it resembles very much what I have with my Cisco equipment + management software.
A lighter-weight approach seems more promising: encapsulate most of the config-file specific knowledge in simple script wrappers that can be controlled by a declarative description of the configuration you want to achieve and the logical interdependencies between them. This is what puppet does, and why I find it very attractive.
I mostly agree, although very elegant compared to what exists today I would still classify it as a work-around.
Cheers, Shane
On Tue, 2006-03-28 at 19:20 -0800, Shane Stixrud wrote:
I have been using another config mgmt system, puppet, (http://reductivelabs.com/projects/puppet) I wrote up an example of how a config mgmt tool could help in solving the kind of problem you are looking at: http://people.redhat.com/dlutter/puppet-app.html
Puppet looks like a very promising centralized change distribution system. As you mentioned this separates the "how" changes are applied from the "what" is actually changing. If you combine the change re-visioning concept mentioned above with a puppet like centralized distribution system that can detect client changes we now have something quite impressive.
Yes, one of the things I am very interested in is combining puppet with a distributed SCM like mercurial, both for distribution of canned configs, and for tracking changes to those configs over time. Puppet right now logs what changes it applied to each machine, but that should probably be expanded to produce a more focused history.
In fact it resembles very much what I have with my Cisco equipment + management software.
Interesting. Any pointers where I can look at docs ?
David
On Wed, 29 Mar 2006, David Lutterkort wrote:
In fact it resembles very much what I have with my Cisco equipment + management software.
Interesting. Any pointers where I can look at docs ?
Well not comprehensive docs per say, but I can outline how it works.
Here is an example of an active config state on a cisco device:
http://geeklords.org/~shane/cisco.cfg
It is important to note from a admin perspective the configuration directives in the above file IS equal to the function that device has been instructed to perform.
Now I am not proposing this type of configuration interface would work for Linux, Cisco has much less to consider and even so there are things about their configuration process I think could be improved. But their semi-standardized plain text directives allows their management software (cisco resource manager) to easily track system changes, apply system changes and revert system changes across multiple platforms using the same "command directives" that the end user would manually enter.
Being able to easily do the following on Linux would be very nice.
1) Show all configuration directives that are not at their default values i.e.:
cfg_report -keys -nondefaults /system/dhcpd
system/dhcpd/option/domain-name/ system/dhcpd/option/domain-name-servers/ system/dhcpd/subnet/172.20.0.0 netmask 255.255.255.0/ system/dhcpd/subnet/172.20.64.0 netmask 255.255.254.0/ etc...
2) Save/restore keys and values
cfg_mgr -export .ini,xml,tar -nodefaults / >/root/changes.tar cfg_mgr -import /root/changes.tar /
cfg_mgr -export file / >/root/hostname.cfg (very large path/key/value text file)
cfg_mgr -import /root/dhcpd.tar /system/dhcpd
3) Display and edit all ntpd's configuration keys/values a standard editor
cfg_edit -export plaintext -editor <editor> /system/ntpd
*note* this would include /etc/ntp/keys /etc/ntp/step-tickers etc..
4) revert dhcpd to a specific date/time
cfg_mgr -revert 03/05/2006 /system/dhcpd
If/when the above is possible centralized config policy management becomes a no brainer. We just create a cron job where we can optionally sync changed keys/values prior to downloading the devices new/active directives.
cfg_mgr -push -pull -server host.domain.top
Cheers, Shane
On 3/28/06, David Lutterkort dlutter@redhat.com wrote:
A lighter-weight approach seems more promising: encapsulate most of the config-file specific knowledge in simple script wrappers that can be controlled by a declarative description of the configuration you want to achieve and the logical interdependencies between them. This is what puppet does, and why I find it very attractive.
Hummm, never heard about puppet. Seems interesting. The problem I see with this approach is the additional layer that manages the configuration files syntax. And again: ALL configuration files can be represented by an hierarchy of key/value pairs. They are different because they received a considerable amount of syntax fat to make them look nicer to your human eyes. And what puppet seems to do is try to work with this fat.
Elektra completely removes that fat and reduces the configuration parameters to what they really are: key/value pairs.
Avi
Avi Alkalay napsal(a):
And again: ALL configuration files can be represented by an hierarchy of key/value pairs.
Sure, any finite object can be represented by one long bit string :)
Can all configuration files be _usefully_ represented by a hierarchy? The natural format (= format prefered for human editing) for some applications uses m4 (sendmail, SELinux policy) or even arbitrary PHP/perl code. Mirek
On 3/30/06, Avi Alkalay avi@unix.sh wrote:
Hummm, never heard about puppet. Seems interesting. The problem I see with this approach is the additional layer that manages the configuration files syntax. And again: ALL configuration files can be represented by an hierarchy of key/value pairs. They are different because they received a considerable amount of syntax fat to make them look nicer to your human eyes. And what puppet seems to do is try to work with this fat.
What would be really cool is to have a file-system based configuration system, where a config file is mounted into a user space or kernel file-system driver. Then programs could use whatever config file format they like, and as long as there is a translation module for it, people can change/view things with simple echo/cat commands. Like /proc/sys/. Eg: cat 80 > /conf/httpd/Listen
Probably just madness, n0dalus.
n0dalus wrote:
On 3/30/06, Avi Alkalay avi@unix.sh wrote:
Hummm, never heard about puppet. Seems interesting. The problem I see with this approach is the additional layer that manages the configuration files syntax. And again: ALL configuration files can be represented by an hierarchy of key/value pairs. They are different because they received a considerable amount of syntax fat to make them look nicer to your human eyes. And what puppet seems to do is try to work with this fat.
What would be really cool is to have a file-system based configuration system, where a config file is mounted into a user space or kernel file-system driver. Then programs could use whatever config file format they like, and as long as there is a translation module for it, people can change/view things with simple echo/cat commands. Like /proc/sys/. Eg: cat 80 > /conf/httpd/Listen
Actually this is a (great) idea, with fuse (userspace filesystems) this should be doable, I think fuse <-> gconf bridge would be a nice start for this.
What do others think? This feels very much like the unix way. (Yeah yet another filesystem, howmany can we get / do we need?
Regards,
Hans
On Thursday 30 March 2006 04:53, Avi Alkalay wrote:
Hummm, never heard about puppet. Seems interesting. The problem I see with this approach is the additional layer that manages the configuration files syntax. And again: ALL configuration files can be represented by an hierarchy of key/value pairs. They are different because they received a considerable amount of syntax fat to make them look nicer to your human eyes. And what puppet seems to do is try to work with this fat.
Elektra completely removes that fat and reduces the configuration parameters to what they really are: key/value pairs.
Dunno where you got this obsession, but just because you can represent data as "key/value pairs" doesn't mean that's always the best layout.
There's a reason why programs aren't written for the old Turing Machine, and that's that however well it might be able to represent any possible program, it's incredibly verbose.
The examples which have appeared in this thread have all made things *less* clear afaics.
Bill"somewhat sick of this thread but suspecting if people don't reply the lunatics will end up running the asylum"Crawford.
devel@lists.stg.fedoraproject.org