Hi,
I would like to propose enhancing the VDSM source to emit message IDs in log messages with levels greater than DEBUG (i.e. INFO, ERROR, WARN, EXCEPTION, and CRITICAL). Messages IDs have been used successfully in many products and they can aid in debugging, documentation, root cause analysis, and internationalization.
I have scanned the source and have determined that there are roughly 525 calls to log.[error|warn|info|critical|exception] and I am volunteering to go through and add IDs to them all but, before I do I want to get buy-in from the project maintainers. While adding IDs to the messages I would also enable them for i18n.
I have written a rough plan for adding messages IDs to VDSM below and I've provided some examples of products/projects that use message IDs.
Examples of Projects that Use Message IDs: - JBoss 7 is adopting Message IDs for all of its logging. See the annotations (e.g. @Message(id = 12000, ) in the following piece of source: https://github.com/jbossas/jboss-as/blob/master/process-controller/src/main/...
- Almost all IBM products use message IDs and they do a fairly good job of documenting them. Google has indexed them all and you can simply type them in to a search and get a description of the problem that is associated with the ID. http://publib.boulder.ibm.com/infocenter/tivihelp/v42r1/topic/com.ibm.omegam...
Example of a Log Statement with a Message ID in Python: FORMAT = '%(asctime)s %(lineno)s %(levelname)s(%(messageid)s): %(message)s' logging.basicConfig(format=FORMAT) logger = logging.getLogger() logger.warning('NFS problem. Unable to mount.', extra={ 'messageid' : 'VDSM1000'})
// Produces 2011-10-05 13:24:43,180 11 WARNING(VDSM1000): NFS problem. Unable to mount.
Example of a Log Statement with a Message ID in Python with i18n: logger.warning(_('NFS problem. Unable to mount.'), extra={ 'messageid' : 'VDSM1000'})
Example of an i18n Log Message: Chinese: 2011年10月5日13:24:43,18011警告（VDSM1000）：NFS问题。无法安装。 English: 2011-10-05 13:24:43,180 11 WARNING(VDSM1000): NFS problem. Unable to mount.
A Rough Plan for adding Message IDs to VDSM:
Step 0: Decide on a logging format of the ID itself: - JBoss is using "JBAS#####". I am thinking that VDSM could probably do something similar "VDSM####".
Step 1: Decide where to list and document all of the IDs. - It could be as simple as a page on VDSM's fedorahosted wiki.
Step 2: Pin the format of the log messages: - Currently VDSM allows a user configurable Python log format. We would need to pin the format so that it includes a substitution variable for the message ID. Example: '%(asctime)s %(lineno)s %(levelname)s(%(messageid)s): %(message)s'
Step 4: Plow through the VDSM source and add message IDs.
Cheers, Keith
On Wed 05 Oct 2011 08:24:13 PM IST, Keith Robertson wrote:
Hi,
I would like to propose enhancing the VDSM source to emit message IDs in log messages with levels greater than DEBUG (i.e. INFO, ERROR, WARN, EXCEPTION, and CRITICAL). Messages IDs have been used successfully in many products and they can aid in debugging, documentation, root cause analysis, and internationalization.
I have scanned the source and have determined that there are roughly 525 calls to log.[error|warn|info|critical|exception] and I am volunteering to go through and add IDs to them all but, before I do I want to get buy-in from the project maintainers. While adding IDs to the messages I would also enable them for i18n.
I have written a rough plan for adding messages IDs to VDSM below and I've provided some examples of products/projects that use message IDs.
Examples of Projects that Use Message IDs:
- JBoss 7 is adopting Message IDs for all of its logging. See the
annotations (e.g. @Message(id = 12000, ) in the following piece of source: https://github.com/jbossas/jboss-as/blob/master/process-controller/src/main/...
- Almost all IBM products use message IDs and they do a fairly good job
of documenting them. Google has indexed them all and you can simply type them in to a search and get a description of the problem that is associated with the ID. http://publib.boulder.ibm.com/infocenter/tivihelp/v42r1/topic/com.ibm.omegam...
Example of a Log Statement with a Message ID in Python: FORMAT = '%(asctime)s %(lineno)s %(levelname)s(%(messageid)s): %(message)s' logging.basicConfig(format=FORMAT) logger = logging.getLogger() logger.warning('NFS problem. Unable to mount.', extra={ 'messageid' : 'VDSM1000'})
// Produces 2011-10-05 13:24:43,180 11 WARNING(VDSM1000): NFS problem. Unable to mount.
Example of a Log Statement with a Message ID in Python with i18n: logger.warning(_('NFS problem. Unable to mount.'), extra={ 'messageid' : 'VDSM1000'})
Example of an i18n Log Message: Chinese: 2011年10月5日13:24:43,18011警告（VDSM1000）：NFS问题。无法安装。 English: 2011-10-05 13:24:43,180 11 WARNING(VDSM1000): NFS problem. Unable to mount.
A Rough Plan for adding Message IDs to VDSM:
Step 0: Decide on a logging format of the ID itself:
- JBoss is using "JBAS#####". I am thinking that VDSM could probably do
something similar "VDSM####".
Step 1: Decide where to list and document all of the IDs.
- It could be as simple as a page on VDSM's fedorahosted wiki.
Step 2: Pin the format of the log messages:
- Currently VDSM allows a user configurable Python log format. We would
need to pin the format so that it includes a substitution variable for the message ID. Example: '%(asctime)s %(lineno)s %(levelname)s(%(messageid)s): %(message)s'
Step 4: Plow through the VDSM source and add message IDs.
Cheers, Keith
vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/vdsm-devel
Seems like a nice idea. But how does it affect developers. How do they choose mIDs? How do you prevent collisions? How are they different from simple gettext?
Message IDs do require some discipline on the part of the developer to ensure uniqueness. They also force the developer to think about whether or not the text that is being emitted is actually worthy of [INFO|ERROR|WARN|EXCEPTION|CRITICAL] instead of the the lowly DEBUG.
As to how it is done, I have seen this done a few different ways but the most popular seems to be to assign a range of numbers to a particular component/package. In this manner, a developer can quickly scan that package and choose the next higher number. Some projects, like my last one, were fairly monolithic in design and we just used an up-counter for the whole source tree (a little harder but not impossible with grep and sort).
All solutions generally involve a central document/registry where the IDs are listed and *can* be documented (generally the documentation evolves over time). For VDSM, I would probably recommend a simple up-counter. Take a look at how JBoss is breaking up their product. http://community.jboss.org/wiki/LoggingIds
As for gettext, message IDs don't really affect gettext. gettext is a vehicle that is used to enable i18n. Message IDs are typically coded as symbols directly in the source while gettext handles the internationalization of all things that are not substitution variables (you don't want to translate them). I've attached the makefile that I wrote for the rhevm-iso-uploader to show the flow for using gettext in a project.
Here is an example of a logging statement that would work with gettext and a message ID: // Assume the log format is this... FORMAT = '%(asctime)s %(lineno)s %(levelname)s(%(messageid)s): %(message)s' // An invocation logger.warning(_("A log message with a substitution string, %s, that doesn't get translated.") % "woweee", extra={ 'messageid' : 'VDSM1000'}) // The result 2011-10-06 11:13:09,458 14 WARNING(VDSM1000): A log message with a substitution string, woweee, that doesn't get translated.
Things to notice: 1. The substitution variable "woweee" doesn't get translated or picked up by gettext because it isn't surrounded by _(...) which is what you want. 2. The message ID (i.e. VDSM1000) doesn't get translated either for the same reason.
Cheers, Keith
On 10/06/2011 10:02 AM, Saggi Mizrahi wrote:
On Wed 05 Oct 2011 08:24:13 PM IST, Keith Robertson wrote:
Hi,
I would like to propose enhancing the VDSM source to emit message IDs in log messages with levels greater than DEBUG (i.e. INFO, ERROR, WARN, EXCEPTION, and CRITICAL). Messages IDs have been used successfully in many products and they can aid in debugging, documentation, root cause analysis, and internationalization.
I have scanned the source and have determined that there are roughly 525 calls to log.[error|warn|info|critical|exception] and I am volunteering to go through and add IDs to them all but, before I do I want to get buy-in from the project maintainers. While adding IDs to the messages I would also enable them for i18n.
I have written a rough plan for adding messages IDs to VDSM below and I've provided some examples of products/projects that use message IDs.
Examples of Projects that Use Message IDs:
- JBoss 7 is adopting Message IDs for all of its logging. See the
annotations (e.g. @Message(id = 12000, ) in the following piece of source: https://github.com/jbossas/jboss-as/blob/master/process-controller/src/main/...
- Almost all IBM products use message IDs and they do a fairly good job
of documenting them. Google has indexed them all and you can simply type them in to a search and get a description of the problem that is associated with the ID. http://publib.boulder.ibm.com/infocenter/tivihelp/v42r1/topic/com.ibm.omegam...
Example of a Log Statement with a Message ID in Python: FORMAT = '%(asctime)s %(lineno)s %(levelname)s(%(messageid)s): %(message)s' logging.basicConfig(format=FORMAT) logger = logging.getLogger() logger.warning('NFS problem. Unable to mount.', extra={ 'messageid' : 'VDSM1000'})
// Produces 2011-10-05 13:24:43,180 11 WARNING(VDSM1000): NFS problem. Unable to mount.
Example of a Log Statement with a Message ID in Python with i18n: logger.warning(_('NFS problem. Unable to mount.'), extra={ 'messageid' : 'VDSM1000'})
Example of an i18n Log Message: Chinese: 2011年10月5日13:24:43,18011警告（VDSM1000）：NFS问题。无法安 装。 English: 2011-10-05 13:24:43,180 11 WARNING(VDSM1000): NFS problem. Unable to mount.
A Rough Plan for adding Message IDs to VDSM:
Step 0: Decide on a logging format of the ID itself:
- JBoss is using "JBAS#####". I am thinking that VDSM could probably do
something similar "VDSM####".
Step 1: Decide where to list and document all of the IDs.
- It could be as simple as a page on VDSM's fedorahosted wiki.
Step 2: Pin the format of the log messages:
- Currently VDSM allows a user configurable Python log format. We would
need to pin the format so that it includes a substitution variable for the message ID. Example: '%(asctime)s %(lineno)s %(levelname)s(%(messageid)s): %(message)s'
Step 4: Plow through the VDSM source and add message IDs.
Cheers, Keith
vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/vdsm-devel
Seems like a nice idea. But how does it affect developers. How do they choose mIDs? How do you prevent collisions? How are they different from simple gettext?
On Thu 06 Oct 2011 05:22:02 PM IST, Keith Robertson wrote:
Message IDs do require some discipline on the part of the developer to ensure uniqueness. They also force the developer to think about whether or not the text that is being emitted is actually worthy of [INFO|ERROR|WARN|EXCEPTION|CRITICAL] instead of the the lowly DEBUG.
As to how it is done, I have seen this done a few different ways but the most popular seems to be to assign a range of numbers to a particular component/package. In this manner, a developer can quickly scan that package and choose the next higher number. Some projects, like my last one, were fairly monolithic in design and we just used an up-counter for the whole source tree (a little harder but not impossible with grep and sort).
All solutions generally involve a central document/registry where the IDs are listed and *can* be documented (generally the documentation evolves over time). For VDSM, I would probably recommend a simple up-counter. Take a look at how JBoss is breaking up their product. http://community.jboss.org/wiki/LoggingIds
As for gettext, message IDs don't really affect gettext. gettext is a vehicle that is used to enable i18n. Message IDs are typically coded as symbols directly in the source while gettext handles the internationalization of all things that are not substitution variables (you don't want to translate them). I've attached the makefile that I wrote for the rhevm-iso-uploader to show the flow for using gettext in a project.
Here is an example of a logging statement that would work with gettext and a message ID: // Assume the log format is this... FORMAT = '%(asctime)s %(lineno)s %(levelname)s(%(messageid)s): %(message)s' // An invocation logger.warning(_("A log message with a substitution string, %s, that doesn't get translated.") % "woweee", extra={ 'messageid' : 'VDSM1000'}) // The result 2011-10-06 11:13:09,458 14 WARNING(VDSM1000): A log message with a substitution string, woweee, that doesn't get translated.
Things to notice:
- The substitution variable "woweee" doesn't get translated or picked
up by gettext because it isn't surrounded by _(...) which is what you want. 2. The message ID (i.e. VDSM1000) doesn't get translated either for the same reason.
Cheers, Keith
On 10/06/2011 10:02 AM, Saggi Mizrahi wrote:
On Wed 05 Oct 2011 08:24:13 PM IST, Keith Robertson wrote:
Hi,
I would like to propose enhancing the VDSM source to emit message IDs in log messages with levels greater than DEBUG (i.e. INFO, ERROR, WARN, EXCEPTION, and CRITICAL). Messages IDs have been used successfully in many products and they can aid in debugging, documentation, root cause analysis, and internationalization.
I have scanned the source and have determined that there are roughly 525 calls to log.[error|warn|info|critical|exception] and I am volunteering to go through and add IDs to them all but, before I do I want to get buy-in from the project maintainers. While adding IDs to the messages I would also enable them for i18n.
I have written a rough plan for adding messages IDs to VDSM below and I've provided some examples of products/projects that use message IDs.
Examples of Projects that Use Message IDs:
- JBoss 7 is adopting Message IDs for all of its logging. See the
annotations (e.g. @Message(id = 12000, ) in the following piece of source: https://github.com/jbossas/jboss-as/blob/master/process-controller/src/main/...
- Almost all IBM products use message IDs and they do a fairly good job
of documenting them. Google has indexed them all and you can simply type them in to a search and get a description of the problem that is associated with the ID. http://publib.boulder.ibm.com/infocenter/tivihelp/v42r1/topic/com.ibm.omegam...
Example of a Log Statement with a Message ID in Python: FORMAT = '%(asctime)s %(lineno)s %(levelname)s(%(messageid)s): %(message)s' logging.basicConfig(format=FORMAT) logger = logging.getLogger() logger.warning('NFS problem. Unable to mount.', extra={ 'messageid' : 'VDSM1000'})
// Produces 2011-10-05 13:24:43,180 11 WARNING(VDSM1000): NFS problem. Unable to mount.
Example of a Log Statement with a Message ID in Python with i18n: logger.warning(_('NFS problem. Unable to mount.'), extra={ 'messageid' : 'VDSM1000'})
Example of an i18n Log Message: Chinese: 2011年10月5日13:24:43,18011警告（VDSM1000）：NFS问题。无法 安 装。 English: 2011-10-05 13:24:43,180 11 WARNING(VDSM1000): NFS problem. Unable to mount.
A Rough Plan for adding Message IDs to VDSM:
Step 0: Decide on a logging format of the ID itself:
- JBoss is using "JBAS#####". I am thinking that VDSM could probably do
something similar "VDSM####".
Step 1: Decide where to list and document all of the IDs.
- It could be as simple as a page on VDSM's fedorahosted wiki.
Step 2: Pin the format of the log messages:
- Currently VDSM allows a user configurable Python log format. We would
need to pin the format so that it includes a substitution variable for the message ID. Example: '%(asctime)s %(lineno)s %(levelname)s(%(messageid)s): %(message)s'
Step 4: Plow through the VDSM source and add message IDs.
Cheers, Keith
vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/vdsm-devel
Seems like a nice idea. But how does it affect developers. How do they choose mIDs? How do you prevent collisions? How are they different from simple gettext?
As I never used this myself so I can't really imagine the all benefits of using such a system, at least enough to warrant all the tedious work. If you could just list out some of the use cases it'll be most helpful.
For instance I assume grepping for an ID is easier then grepping for a message especially in multiple versions where the message might have changed.
On 10/06/2011 12:14 PM, Saggi Mizrahi wrote:
On Thu 06 Oct 2011 05:22:02 PM IST, Keith Robertson wrote:
Message IDs do require some discipline on the part of the developer to ensure uniqueness. They also force the developer to think about whether or not the text that is being emitted is actually worthy of [INFO|ERROR|WARN|EXCEPTION|CRITICAL] instead of the the lowly DEBUG.
As to how it is done, I have seen this done a few different ways but the most popular seems to be to assign a range of numbers to a particular component/package. In this manner, a developer can quickly scan that package and choose the next higher number. Some projects, like my last one, were fairly monolithic in design and we just used an up-counter for the whole source tree (a little harder but not impossible with grep and sort).
All solutions generally involve a central document/registry where the IDs are listed and *can* be documented (generally the documentation evolves over time). For VDSM, I would probably recommend a simple up-counter. Take a look at how JBoss is breaking up their product. http://community.jboss.org/wiki/LoggingIds
As for gettext, message IDs don't really affect gettext. gettext is a vehicle that is used to enable i18n. Message IDs are typically coded as symbols directly in the source while gettext handles the internationalization of all things that are not substitution variables (you don't want to translate them). I've attached the makefile that I wrote for the rhevm-iso-uploader to show the flow for using gettext in a project.
Here is an example of a logging statement that would work with gettext and a message ID: // Assume the log format is this... FORMAT = '%(asctime)s %(lineno)s %(levelname)s(%(messageid)s): %(message)s' // An invocation logger.warning(_("A log message with a substitution string, %s, that doesn't get translated.") % "woweee", extra={ 'messageid' : 'VDSM1000'}) // The result 2011-10-06 11:13:09,458 14 WARNING(VDSM1000): A log message with a substitution string, woweee, that doesn't get translated.
Things to notice:
- The substitution variable "woweee" doesn't get translated or
picked up by gettext because it isn't surrounded by _(...) which is what you want. 2. The message ID (i.e. VDSM1000) doesn't get translated either for the same reason.
Cheers, Keith
On 10/06/2011 10:02 AM, Saggi Mizrahi wrote:
On Wed 05 Oct 2011 08:24:13 PM IST, Keith Robertson wrote:
Hi,
I would like to propose enhancing the VDSM source to emit message IDs in log messages with levels greater than DEBUG (i.e. INFO, ERROR, WARN, EXCEPTION, and CRITICAL). Messages IDs have been used successfully in many products and they can aid in debugging, documentation, root cause analysis, and internationalization.
I have scanned the source and have determined that there are roughly 525 calls to log.[error|warn|info|critical|exception] and I am volunteering to go through and add IDs to them all but, before I do I want to get buy-in from the project maintainers. While adding IDs to the messages I would also enable them for i18n.
I have written a rough plan for adding messages IDs to VDSM below and I've provided some examples of products/projects that use message IDs.
Examples of Projects that Use Message IDs:
- JBoss 7 is adopting Message IDs for all of its logging. See the
annotations (e.g. @Message(id = 12000, ) in the following piece of source: https://github.com/jbossas/jboss-as/blob/master/process-controller/src/main/...
- Almost all IBM products use message IDs and they do a fairly good
job of documenting them. Google has indexed them all and you can simply type them in to a search and get a description of the problem that is associated with the ID. http://publib.boulder.ibm.com/infocenter/tivihelp/v42r1/topic/com.ibm.omegam...
Example of a Log Statement with a Message ID in Python: FORMAT = '%(asctime)s %(lineno)s %(levelname)s(%(messageid)s): %(message)s' logging.basicConfig(format=FORMAT) logger = logging.getLogger() logger.warning('NFS problem. Unable to mount.', extra={ 'messageid' : 'VDSM1000'})
// Produces 2011-10-05 13:24:43,180 11 WARNING(VDSM1000): NFS problem. Unable to mount.
Example of a Log Statement with a Message ID in Python with i18n: logger.warning(_('NFS problem. Unable to mount.'), extra={ 'messageid' : 'VDSM1000'})
Example of an i18n Log Message: Chinese: 2011年10月5日13:24:43,18011警告（VDSM1000）：NFS问题。无法 安 装。 English: 2011-10-05 13:24:43,180 11 WARNING(VDSM1000): NFS problem. Unable to mount.
A Rough Plan for adding Message IDs to VDSM:
Step 0: Decide on a logging format of the ID itself:
- JBoss is using "JBAS#####". I am thinking that VDSM could
probably do something similar "VDSM####".
Step 1: Decide where to list and document all of the IDs.
- It could be as simple as a page on VDSM's fedorahosted wiki.
Step 2: Pin the format of the log messages:
- Currently VDSM allows a user configurable Python log format. We
would need to pin the format so that it includes a substitution variable for the message ID. Example: '%(asctime)s %(lineno)s %(levelname)s(%(messageid)s): %(message)s'
Step 4: Plow through the VDSM source and add message IDs.
Cheers, Keith
vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/vdsm-devel
Seems like a nice idea. But how does it affect developers. How do they choose mIDs? How do you prevent collisions? How are they different from simple gettext?
As I never used this myself so I can't really imagine the all benefits of using such a system, at least enough to warrant all the tedious work. If you could just list out some of the use cases it'll be most helpful.
For instance I assume grepping for an ID is easier then grepping for a message especially in multiple versions where the message might have changed.
Sure. As far as the tedious part, yes, adding message IDs to an existing source tree is a bit of a PITA but the 525 invocations in VDSM isn't all that bad. I've seen much worse. Plus, once it's done the effort is much smaller (new lines of code only).
As for the value (I've tried to itemize a few common reasons for doing it):
Value: Message IDs allow you to use gettext globalize your messages. If you just used gettext and pulled the strings out of your code for globalization then you might end up with something like this in vdsm.log... 错误：问题安装驱动器 the English translation (according to google) is: ERROR: Problem mounting drive Unless you can read Chinese, then the message above is completely unhelpful to a support engineer. The converse is true also. Consider a non-english speaking person trying to decipher "ERROR: Problem mounting drive". With a message ID; however, you get this... 错误（VDSM1000）：安装驱动问题
Value: Message IDs allow other applications to perform actions based on what is in your logs. On the mainframe, this is extremely common. You'll have one program watching the logs for a particular ID(s). When the program sees the ID it will look in a dictionary and decide what (if anything) needs to be done. On linux, logwatch, is frequently used for this purpose and message IDs make this process much simpler because the program doesn't have to write complicated REGEXs to look for arbitrary strings it can just scan for an ID.
Value: Message IDs generally have the same meaning across releases of the product. This means that the developer can change the text of the log message without worrying about breaking someone's automation. Example: Version 1 of the product- ERROR(VDSM1000): Problem mounting drive. Version 2 of the product- ERROR(VDSM1000): unable to contact 127.0.0.1 for mounting. Try blah blah.
Value: If you do it right search engines will index your message IDs and take users to a page where they can hopefully get more detail on the issue and potentially "self solve" the problem without calling support. Put this into a search engine... "KDFAP0020S"
Value: Message IDs allow you to easily describe flows. You might be able to say... "if you saw ID123 then look farther back for ID124 and if you see ID124 then the problem is _probably_ XYZ."
I am probably not doing enough justice to the value of message IDs but, I will say that lots of very large companies use them in complex software applications to solve these and other issues.
Cheers, Keith
On 10/06/2011 12:18 PM, Keith Robertson wrote:
On 10/06/2011 12:14 PM, Saggi Mizrahi wrote:
On Thu 06 Oct 2011 05:22:02 PM IST, Keith Robertson wrote:
Message IDs do require some discipline on the part of the developer to ensure uniqueness. They also force the developer to think about whether or not the text that is being emitted is actually worthy of [INFO|ERROR|WARN|EXCEPTION|CRITICAL] instead of the the lowly DEBUG.
As to how it is done, I have seen this done a few different ways but the most popular seems to be to assign a range of numbers to a particular component/package. In this manner, a developer can quickly scan that package and choose the next higher number. Some projects, like my last one, were fairly monolithic in design and we just used an up-counter for the whole source tree (a little harder but not impossible with grep and sort).
All solutions generally involve a central document/registry where the IDs are listed and *can* be documented (generally the documentation evolves over time). For VDSM, I would probably recommend a simple up-counter. Take a look at how JBoss is breaking up their product. http://community.jboss.org/wiki/LoggingIds
As for gettext, message IDs don't really affect gettext. gettext is a vehicle that is used to enable i18n. Message IDs are typically coded as symbols directly in the source while gettext handles the internationalization of all things that are not substitution variables (you don't want to translate them). I've attached the makefile that I wrote for the rhevm-iso-uploader to show the flow for using gettext in a project.
Here is an example of a logging statement that would work with gettext and a message ID: // Assume the log format is this... FORMAT = '%(asctime)s %(lineno)s %(levelname)s(%(messageid)s): %(message)s' // An invocation logger.warning(_("A log message with a substitution string, %s, that doesn't get translated.") % "woweee", extra={ 'messageid' : 'VDSM1000'}) // The result 2011-10-06 11:13:09,458 14 WARNING(VDSM1000): A log message with a substitution string, woweee, that doesn't get translated.
Things to notice:
- The substitution variable "woweee" doesn't get translated or
picked up by gettext because it isn't surrounded by _(...) which is what you want. 2. The message ID (i.e. VDSM1000) doesn't get translated either for the same reason.
Cheers, Keith
On 10/06/2011 10:02 AM, Saggi Mizrahi wrote:
On Wed 05 Oct 2011 08:24:13 PM IST, Keith Robertson wrote:
Hi,
I would like to propose enhancing the VDSM source to emit message IDs in log messages with levels greater than DEBUG (i.e. INFO, ERROR, WARN, EXCEPTION, and CRITICAL). Messages IDs have been used successfully in many products and they can aid in debugging, documentation, root cause analysis, and internationalization.
I have scanned the source and have determined that there are roughly 525 calls to log.[error|warn|info|critical|exception] and I am volunteering to go through and add IDs to them all but, before I do I want to get buy-in from the project maintainers. While adding IDs to the messages I would also enable them for i18n.
I have written a rough plan for adding messages IDs to VDSM below and I've provided some examples of products/projects that use message IDs.
Examples of Projects that Use Message IDs:
- JBoss 7 is adopting Message IDs for all of its logging. See the
annotations (e.g. @Message(id = 12000, ) in the following piece of source: https://github.com/jbossas/jboss-as/blob/master/process-controller/src/main/...
- Almost all IBM products use message IDs and they do a fairly good
job of documenting them. Google has indexed them all and you can simply type them in to a search and get a description of the problem that is associated with the ID. http://publib.boulder.ibm.com/infocenter/tivihelp/v42r1/topic/com.ibm.omegam...
Example of a Log Statement with a Message ID in Python: FORMAT = '%(asctime)s %(lineno)s %(levelname)s(%(messageid)s): %(message)s' logging.basicConfig(format=FORMAT) logger = logging.getLogger() logger.warning('NFS problem. Unable to mount.', extra={ 'messageid' : 'VDSM1000'})
// Produces 2011-10-05 13:24:43,180 11 WARNING(VDSM1000): NFS problem. Unable to mount.
Example of a Log Statement with a Message ID in Python with i18n: logger.warning(_('NFS problem. Unable to mount.'), extra={ 'messageid' : 'VDSM1000'})
Example of an i18n Log Message: Chinese: 2011年10月5日13:24:43,18011警告（VDSM1000）：NFS问题。无法 安 装。 English: 2011-10-05 13:24:43,180 11 WARNING(VDSM1000): NFS problem. Unable to mount.
A Rough Plan for adding Message IDs to VDSM:
Step 0: Decide on a logging format of the ID itself:
- JBoss is using "JBAS#####". I am thinking that VDSM could
probably do something similar "VDSM####".
Step 1: Decide where to list and document all of the IDs.
- It could be as simple as a page on VDSM's fedorahosted wiki.
Step 2: Pin the format of the log messages:
- Currently VDSM allows a user configurable Python log format. We
would need to pin the format so that it includes a substitution variable for the message ID. Example: '%(asctime)s %(lineno)s %(levelname)s(%(messageid)s): %(message)s'
Step 4: Plow through the VDSM source and add message IDs.
Cheers, Keith
vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/vdsm-devel
Seems like a nice idea. But how does it affect developers. How do they choose mIDs? How do you prevent collisions? How are they different from simple gettext?
As I never used this myself so I can't really imagine the all benefits of using such a system, at least enough to warrant all the tedious work. If you could just list out some of the use cases it'll be most helpful.
For instance I assume grepping for an ID is easier then grepping for a message especially in multiple versions where the message might have changed.
Sure. As far as the tedious part, yes, adding message IDs to an existing source tree is a bit of a PITA but the 525 invocations in VDSM isn't all that bad. I've seen much worse. Plus, once it's done the effort is much smaller (new lines of code only).
As for the value (I've tried to itemize a few common reasons for doing it):
Value: Message IDs allow you to use gettext globalize your messages. If you just used gettext and pulled the strings out of your code for globalization then you might end up with something like this in vdsm.log... 错误：问题安装驱动器 the English translation (according to google) is: ERROR: Problem mounting drive Unless you can read Chinese, then the message above is completely unhelpful to a support engineer. The converse is true also. Consider a non-english speaking person trying to decipher "ERROR: Problem mounting drive". With a message ID; however, you get this... 错误（VDSM1000）：安装驱动问题
Is it possible to just hash the main string before internationalization?
So basically:
fprintf(f, "[%s] ERROR(VDSM%04x) %s", date_str, crc32(message), message);
Or the equivalent Python.
Value: Message IDs allow other applications to perform actions based on what is in your logs. On the mainframe, this is extremely common. You'll have one program watching the logs for a particular ID(s). When the program sees the ID it will look in a dictionary and decide what (if anything) needs to be done. On linux, logwatch, is frequently used for this purpose and message IDs make this process much simpler because the program doesn't have to write complicated REGEXs to look for arbitrary strings it can just scan for an ID.
Value: Message IDs generally have the same meaning across releases of the product. This means that the developer can change the text of the log message without worrying about breaking someone's automation. Example: Version 1 of the product- ERROR(VDSM1000): Problem mounting drive. Version 2 of the product- ERROR(VDSM1000): unable to contact 127.0.0.1 for mounting. Try blah blah.
Does this mean that logging messages are an ABI that needs to be maintained over time? That's probably very difficult to do right.
Regards,
Anthony Liguori
Value: If you do it right search engines will index your message IDs and take users to a page where they can hopefully get more detail on the issue and potentially "self solve" the problem without calling support. Put this into a search engine... "KDFAP0020S"
Value: Message IDs allow you to easily describe flows. You might be able to say... "if you saw ID123 then look farther back for ID124 and if you see ID124 then the problem is _probably_ XYZ."
I am probably not doing enough justice to the value of message IDs but, I will say that lots of very large companies use them in complex software applications to solve these and other issues.
Cheers, Keith
vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/vdsm-devel
On 10/06/2011 01:55 PM, Anthony Liguori wrote:
On 10/06/2011 12:18 PM, Keith Robertson wrote:
On 10/06/2011 12:14 PM, Saggi Mizrahi wrote:
On Thu 06 Oct 2011 05:22:02 PM IST, Keith Robertson wrote:
Message IDs do require some discipline on the part of the developer to ensure uniqueness. They also force the developer to think about whether or not the text that is being emitted is actually worthy of [INFO|ERROR|WARN|EXCEPTION|CRITICAL] instead of the the lowly DEBUG.
As to how it is done, I have seen this done a few different ways but the most popular seems to be to assign a range of numbers to a particular component/package. In this manner, a developer can quickly scan that package and choose the next higher number. Some projects, like my last one, were fairly monolithic in design and we just used an up-counter for the whole source tree (a little harder but not impossible with grep and sort).
All solutions generally involve a central document/registry where the IDs are listed and *can* be documented (generally the documentation evolves over time). For VDSM, I would probably recommend a simple up-counter. Take a look at how JBoss is breaking up their product. http://community.jboss.org/wiki/LoggingIds
As for gettext, message IDs don't really affect gettext. gettext is a vehicle that is used to enable i18n. Message IDs are typically coded as symbols directly in the source while gettext handles the internationalization of all things that are not substitution variables (you don't want to translate them). I've attached the makefile that I wrote for the rhevm-iso-uploader to show the flow for using gettext in a project.
Here is an example of a logging statement that would work with gettext and a message ID: // Assume the log format is this... FORMAT = '%(asctime)s %(lineno)s %(levelname)s(%(messageid)s): %(message)s' // An invocation logger.warning(_("A log message with a substitution string, %s, that doesn't get translated.") % "woweee", extra={ 'messageid' : 'VDSM1000'}) // The result 2011-10-06 11:13:09,458 14 WARNING(VDSM1000): A log message with a substitution string, woweee, that doesn't get translated.
Things to notice:
- The substitution variable "woweee" doesn't get translated or
picked up by gettext because it isn't surrounded by _(...) which is what you want. 2. The message ID (i.e. VDSM1000) doesn't get translated either for the same reason.
Cheers, Keith
On 10/06/2011 10:02 AM, Saggi Mizrahi wrote:
On Wed 05 Oct 2011 08:24:13 PM IST, Keith Robertson wrote:
Hi,
I would like to propose enhancing the VDSM source to emit message IDs in log messages with levels greater than DEBUG (i.e. INFO, ERROR, WARN, EXCEPTION, and CRITICAL). Messages IDs have been used successfully in many products and they can aid in debugging, documentation, root cause analysis, and internationalization.
I have scanned the source and have determined that there are roughly 525 calls to log.[error|warn|info|critical|exception] and I am volunteering to go through and add IDs to them all but, before I do I want to get buy-in from the project maintainers. While adding IDs to the messages I would also enable them for i18n.
I have written a rough plan for adding messages IDs to VDSM below and I've provided some examples of products/projects that use message IDs.
Examples of Projects that Use Message IDs:
- JBoss 7 is adopting Message IDs for all of its logging. See the
annotations (e.g. @Message(id = 12000, ) in the following piece of source: https://github.com/jbossas/jboss-as/blob/master/process-controller/src/main/...
- Almost all IBM products use message IDs and they do a fairly good
job of documenting them. Google has indexed them all and you can simply type them in to a search and get a description of the problem that is associated with the ID. http://publib.boulder.ibm.com/infocenter/tivihelp/v42r1/topic/com.ibm.omegam...
Example of a Log Statement with a Message ID in Python: FORMAT = '%(asctime)s %(lineno)s %(levelname)s(%(messageid)s): %(message)s' logging.basicConfig(format=FORMAT) logger = logging.getLogger() logger.warning('NFS problem. Unable to mount.', extra={ 'messageid' : 'VDSM1000'})
// Produces 2011-10-05 13:24:43,180 11 WARNING(VDSM1000): NFS problem. Unable to mount.
Example of a Log Statement with a Message ID in Python with i18n: logger.warning(_('NFS problem. Unable to mount.'), extra={ 'messageid' : 'VDSM1000'})
Example of an i18n Log Message: Chinese: 2011年10月5日13:24:43,18011警告（VDSM1000）：NFS问题。无法 安 装。 English: 2011-10-05 13:24:43,180 11 WARNING(VDSM1000): NFS problem. Unable to mount.
A Rough Plan for adding Message IDs to VDSM:
Step 0: Decide on a logging format of the ID itself:
- JBoss is using "JBAS#####". I am thinking that VDSM could
probably do something similar "VDSM####".
Step 1: Decide where to list and document all of the IDs.
- It could be as simple as a page on VDSM's fedorahosted wiki.
Step 2: Pin the format of the log messages:
- Currently VDSM allows a user configurable Python log format. We
would need to pin the format so that it includes a substitution variable for the message ID. Example: '%(asctime)s %(lineno)s %(levelname)s(%(messageid)s): %(message)s'
Step 4: Plow through the VDSM source and add message IDs.
Cheers, Keith
vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/vdsm-devel
Seems like a nice idea. But how does it affect developers. How do they choose mIDs? How do you prevent collisions? How are they different from simple gettext?
As I never used this myself so I can't really imagine the all benefits of using such a system, at least enough to warrant all the tedious work. If you could just list out some of the use cases it'll be most helpful.
For instance I assume grepping for an ID is easier then grepping for a message especially in multiple versions where the message might have changed.
Sure. As far as the tedious part, yes, adding message IDs to an existing source tree is a bit of a PITA but the 525 invocations in VDSM isn't all that bad. I've seen much worse. Plus, once it's done the effort is much smaller (new lines of code only).
As for the value (I've tried to itemize a few common reasons for doing it):
Value: Message IDs allow you to use gettext globalize your messages. If you just used gettext and pulled the strings out of your code for globalization then you might end up with something like this in vdsm.log... 错误：问题安装驱动器 the English translation (according to google) is: ERROR: Problem mounting drive Unless you can read Chinese, then the message above is completely unhelpful to a support engineer. The converse is true also. Consider a non-english speaking person trying to decipher "ERROR: Problem mounting drive". With a message ID; however, you get this... 错误（VDSM1000）：安装驱动问题
Is it possible to just hash the main string before internationalization?
So basically:
fprintf(f, "[%s] ERROR(VDSM%04x) %s", date_str, crc32(message), message);
Yes, a hash of the string is possible. However, would make it hard for someone to document the ID because they would have to hash the string to get the ID.
One of the features of message ID/codes is that they are documentable.
Or the equivalent Python.
Value: Message IDs allow other applications to perform actions based on what is in your logs. On the mainframe, this is extremely common. You'll have one program watching the logs for a particular ID(s). When the program sees the ID it will look in a dictionary and decide what (if anything) needs to be done. On linux, logwatch, is frequently used for this purpose and message IDs make this process much simpler because the program doesn't have to write complicated REGEXs to look for arbitrary strings it can just scan for an ID.
Value: Message IDs generally have the same meaning across releases of the product. This means that the developer can change the text of the log message without worrying about breaking someone's automation. Example: Version 1 of the product- ERROR(VDSM1000): Problem mounting drive. Version 2 of the product- ERROR(VDSM1000): unable to contact 127.0.0.1 for mounting. Try blah blah.
Does this mean that logging messages are an ABI that needs to be maintained over time?
The degree to which message IDs need to be maintained is generally a function of how dynamic the code is and how much you want to document what the ID actually means. If, on the WIKI wherein you define messages, you elect to say [1] then this "ABI" doesn't need any maintenance whatsoever.
[1] VDSM1000 == "Call support or post to the forum to figure out what the heck the program just did"
That's probably very difficult to do right.
Lots of other software programs have been successful at it [1], [2]. I would prefer to think of it in terms of how easy we want to make it for people to self diagnose. Also, without message IDs supporting source that is i18n is nearly impossible unless everyone is multilingual.
[1] http://www.cisco.com/en/US/products/sw/netmgtsw/ps1982/products_command_refe... [2] http://mail.google.com/support/bin/answer.py?answer=12336
Regards,
Anthony Liguori
Value: If you do it right search engines will index your message IDs and take users to a page where they can hopefully get more detail on the issue and potentially "self solve" the problem without calling support. Put this into a search engine... "KDFAP0020S"
Value: Message IDs allow you to easily describe flows. You might be able to say... "if you saw ID123 then look farther back for ID124 and if you see ID124 then the problem is _probably_ XYZ."
I am probably not doing enough justice to the value of message IDs but, I will say that lots of very large companies use them in complex software applications to solve these and other issues.
Cheers, Keith
vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/vdsm-devel
> I would like to propose enhancing the VDSM source to emit > message > IDs in > log messages with levels greater than DEBUG (i.e. INFO, ERROR, > WARN, > EXCEPTION, and CRITICAL). Messages IDs have been used > successfully in > many products and they can aid in debugging, documentation, > root > cause > analysis, and internationalization. > > I have scanned the source and have determined that there are > roughly 525 > calls to log.[error|warn|info|critical|exception] and I am > volunteering > to go through and add IDs to them all but, before I do I want > to get > buy-in from the project maintainers. While adding IDs to the
wrt i18n, I'm not entirely convinced. Every line we log has the module name and line (never localized) which logged it so effectively we already have this info. However, the ID would remain constant where lines will change place between versions so it makes it easier. I also agree about the lookup and being able to write simpler parsers.
So I'm in.
> messages I > would also enable them for i18n. > > I have written a rough plan for adding messages IDs to VDSM > below > and > I've provided some examples of products/projects that use > message > IDs. > > Examples of Projects that Use Message IDs: > - JBoss 7 is adopting Message IDs for all of its logging. See > the > annotations (e.g. @Message(id = 12000, ) in the following > piece of > source: > https://github.com/jbossas/jboss-as/blob/master/process-controller/src/main/... > > > > - Almost all IBM products use message IDs and they do a fairly > good > job > of documenting them. Google has indexed them all and you can > simply > type > them in to a search and get a description of the problem that > is > associated with the ID. > http://publib.boulder.ibm.com/infocenter/tivihelp/v42r1/topic/com.ibm.omegam... > > > > > > Example of a Log Statement with a Message ID in Python: > FORMAT = '%(asctime)s %(lineno)s %(levelname)s(%(messageid)s): > %(message)s' > logging.basicConfig(format=FORMAT) > logger = logging.getLogger() > logger.warning('NFS problem. Unable to mount.', extra={ > 'messageid' : > 'VDSM1000'}) > > // Produces > 2011-10-05 13:24:43,180 11 WARNING(VDSM1000): NFS problem. > Unable > to mount. > > > Example of a Log Statement with a Message ID in Python with > i18n: > logger.warning(_('NFS problem. Unable to mount.'), extra={ > 'messageid' : > 'VDSM1000'}) > > Example of an i18n Log Message: > Chinese: 2011年10月5日13:24:43,18011警告（VDSM1000）：NFS问题。无法 > 安 装。 > English: 2011-10-05 13:24:43,180 11 WARNING(VDSM1000): NFS > problem. > Unable to mount. > > > > A Rough Plan for adding Message IDs to VDSM: > > Step 0: Decide on a logging format of the ID itself: > - JBoss is using "JBAS#####". I am thinking that VDSM could > probably do > something similar "VDSM####". > > Step 1: Decide where to list and document all of the IDs. > - It could be as simple as a page on VDSM's fedorahosted wiki. > > Step 2: Pin the format of the log messages: > - Currently VDSM allows a user configurable Python log format. > We > would > need to pin the format so that it includes a substitution > variable for > the message ID. > Example: '%(asctime)s %(lineno)s %(levelname)s(%(messageid)s): > %(message)s' > > Step 4: Plow through the VDSM source and add message IDs. > > > Cheers, > Keith > > > > > > _______________________________________________ > vdsm-devel mailing list > vdsm-devel@lists.fedorahosted.org > https://fedorahosted.org/mailman/listinfo/vdsm-devel
Seems like a nice idea. But how does it affect developers. How do they choose mIDs? How do you prevent collisions? How are they different from simple gettext?
As I never used this myself so I can't really imagine the all benefits of using such a system, at least enough to warrant all the tedious work. If you could just list out some of the use cases it'll be most helpful.
For instance I assume grepping for an ID is easier then grepping for a message especially in multiple versions where the message might have changed.
Sure. As far as the tedious part, yes, adding message IDs to an existing source tree is a bit of a PITA but the 525 invocations in VDSM isn't all that bad. I've seen much worse. Plus, once it's done the effort is much smaller (new lines of code only).
As for the value (I've tried to itemize a few common reasons for doing it):
Value: Message IDs allow you to use gettext globalize your messages. If you just used gettext and pulled the strings out of your code for globalization then you might end up with something like this in vdsm.log... 错误：问题安装驱动器 the English translation (according to google) is: ERROR: Problem mounting drive Unless you can read Chinese, then the message above is completely unhelpful to a support engineer. The converse is true also. Consider a non-english speaking person trying to decipher "ERROR: Problem mounting drive". With a message ID; however, you get this... 错误（VDSM1000）：安装驱动问题
Is it possible to just hash the main string before internationalization?
So basically:
fprintf(f, "[%s] ERROR(VDSM%04x) %s", date_str, crc32(message), message);
Yes, a hash of the string is possible. However, would make it hard for someone to document the ID because they would have to hash the string to get the ID.
One of the features of message ID/codes is that they are documentable.
Or the equivalent Python.
Value: Message IDs allow other applications to perform actions based on what is in your logs. On the mainframe, this is extremely common. You'll have one program watching the logs for a particular ID(s). When the program sees the ID it will look in a dictionary and decide what (if anything) needs to be done. On linux, logwatch, is frequently used for this purpose and message IDs make this process much simpler because the program doesn't have to write complicated REGEXs to look for arbitrary strings it can just scan for an ID.
Value: Message IDs generally have the same meaning across releases of the product. This means that the developer can change the text of the log message without worrying about breaking someone's automation. Example: Version 1 of the product- ERROR(VDSM1000): Problem mounting drive. Version 2 of the product- ERROR(VDSM1000): unable to contact 127.0.0.1 for mounting. Try blah blah.
Does this mean that logging messages are an ABI that needs to be maintained over time?
The degree to which message IDs need to be maintained is generally a function of how dynamic the code is and how much you want to document what the ID actually means. If, on the WIKI wherein you define messages, you elect to say [1] then this "ABI" doesn't need any maintenance whatsoever.
[1] VDSM1000 == "Call support or post to the forum to figure out what the heck the program just did"
That's probably very difficult to do right.
Lots of other software programs have been successful at it [1], [2]. I would prefer to think of it in terms of how easy we want to make it for people to self diagnose. Also, without message IDs supporting source that is i18n is nearly impossible unless everyone is multilingual.
[1] http://www.cisco.com/en/US/products/sw/netmgtsw/ps1982/products_command_refe... [2] http://mail.google.com/support/bin/answer.py?answer=12336
Regards,
Anthony Liguori
Value: If you do it right search engines will index your message IDs and take users to a page where they can hopefully get more detail on the issue and potentially "self solve" the problem without calling support. Put this into a search engine... "KDFAP0020S"
Value: Message IDs allow you to easily describe flows. You might be able to say... "if you saw ID123 then look farther back for ID124 and if you see ID124 then the problem is _probably_ XYZ."
I am probably not doing enough justice to the value of message IDs but, I will say that lots of very large companies use them in complex software applications to solve these and other issues.
Cheers, Keith
vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/vdsm-devel
vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/vdsm-devel
On 10/06/2011 03:11 PM, Ayal Baron wrote:
>> I would like to propose enhancing the VDSM source to emit >> message >> IDs in >> log messages with levels greater than DEBUG (i.e. INFO, ERROR, >> WARN, >> EXCEPTION, and CRITICAL). Messages IDs have been used >> successfully in >> many products and they can aid in debugging, documentation, >> root >> cause >> analysis, and internationalization. >> >> I have scanned the source and have determined that there are >> roughly 525 >> calls to log.[error|warn|info|critical|exception] and I am >> volunteering >> to go through and add IDs to them all but, before I do I want >> to get >> buy-in from the project maintainers. While adding IDs to the
wrt i18n, I'm not entirely convinced.
Yeah, i18n is one reason to do it but it is only relevant if you take the time to create lang-packs for different nationalities (which can be expensive). However, it is a cakewalk to code for in Python _("string here"). So you can code for it ... and if someone ever wanted to take the time create a lang-pack you'd be set.
Every line we log has the module name and line (never localized) which logged it so effectively we already have this info. However, the ID would remain constant where lines will change place between versions so it makes it easier. I also agree about the lookup and being able to write simpler parsers.
So I'm in.
Cool.
>> messages I >> would also enable them for i18n. >> >> I have written a rough plan for adding messages IDs to VDSM >> below >> and >> I've provided some examples of products/projects that use >> message >> IDs. >> >> Examples of Projects that Use Message IDs: >> - JBoss 7 is adopting Message IDs for all of its logging. See >> the >> annotations (e.g. @Message(id = 12000, ) in the following >> piece of >> source: >> https://github.com/jbossas/jboss-as/blob/master/process-controller/src/main/... >> >> >> >> - Almost all IBM products use message IDs and they do a fairly >> good >> job >> of documenting them. Google has indexed them all and you can >> simply >> type >> them in to a search and get a description of the problem that >> is >> associated with the ID. >> http://publib.boulder.ibm.com/infocenter/tivihelp/v42r1/topic/com.ibm.omegam... >> >> >> >> >> >> Example of a Log Statement with a Message ID in Python: >> FORMAT = '%(asctime)s %(lineno)s %(levelname)s(%(messageid)s): >> %(message)s' >> logging.basicConfig(format=FORMAT) >> logger = logging.getLogger() >> logger.warning('NFS problem. Unable to mount.', extra={ >> 'messageid' : >> 'VDSM1000'}) >> >> // Produces >> 2011-10-05 13:24:43,180 11 WARNING(VDSM1000): NFS problem. >> Unable >> to mount. >> >> >> Example of a Log Statement with a Message ID in Python with >> i18n: >> logger.warning(_('NFS problem. Unable to mount.'), extra={ >> 'messageid' : >> 'VDSM1000'}) >> >> Example of an i18n Log Message: >> Chinese: 2011年10月5日13:24:43,18011警告（VDSM1000）：NFS问题。无法 >> 安 装。 >> English: 2011-10-05 13:24:43,180 11 WARNING(VDSM1000): NFS >> problem. >> Unable to mount. >> >> >> >> A Rough Plan for adding Message IDs to VDSM: >> >> Step 0: Decide on a logging format of the ID itself: >> - JBoss is using "JBAS#####". I am thinking that VDSM could >> probably do >> something similar "VDSM####". >> >> Step 1: Decide where to list and document all of the IDs. >> - It could be as simple as a page on VDSM's fedorahosted wiki. >> >> Step 2: Pin the format of the log messages: >> - Currently VDSM allows a user configurable Python log format. >> We >> would >> need to pin the format so that it includes a substitution >> variable for >> the message ID. >> Example: '%(asctime)s %(lineno)s %(levelname)s(%(messageid)s): >> %(message)s' >> >> Step 4: Plow through the VDSM source and add message IDs. >> >> >> Cheers, >> Keith >> >> >> >> >> >> _______________________________________________ >> vdsm-devel mailing list >> vdsm-devel@lists.fedorahosted.org >> https://fedorahosted.org/mailman/listinfo/vdsm-devel > Seems like a nice idea. But how does it affect developers. How > do > they choose mIDs? How do you prevent collisions? How are they > different from simple gettext?
As I never used this myself so I can't really imagine the all benefits of using such a system, at least enough to warrant all the tedious work. If you could just list out some of the use cases it'll be most helpful.
For instance I assume grepping for an ID is easier then grepping for a message especially in multiple versions where the message might have changed.
Sure. As far as the tedious part, yes, adding message IDs to an existing source tree is a bit of a PITA but the 525 invocations in VDSM isn't all that bad. I've seen much worse. Plus, once it's done the effort is much smaller (new lines of code only).
As for the value (I've tried to itemize a few common reasons for doing it):
Value: Message IDs allow you to use gettext globalize your messages. If you just used gettext and pulled the strings out of your code for globalization then you might end up with something like this in vdsm.log... 错误：问题安装驱动器 the English translation (according to google) is: ERROR: Problem mounting drive Unless you can read Chinese, then the message above is completely unhelpful to a support engineer. The converse is true also. Consider a non-english speaking person trying to decipher "ERROR: Problem mounting drive". With a message ID; however, you get this... 错误（VDSM1000）：安装驱动问题
Is it possible to just hash the main string before internationalization?
So basically:
fprintf(f, "[%s] ERROR(VDSM%04x) %s", date_str, crc32(message), message);
Yes, a hash of the string is possible. However, would make it hard for someone to document the ID because they would have to hash the string to get the ID.
One of the features of message ID/codes is that they are documentable.
Or the equivalent Python.
Value: Message IDs allow other applications to perform actions based on what is in your logs. On the mainframe, this is extremely common. You'll have one program watching the logs for a particular ID(s). When the program sees the ID it will look in a dictionary and decide what (if anything) needs to be done. On linux, logwatch, is frequently used for this purpose and message IDs make this process much simpler because the program doesn't have to write complicated REGEXs to look for arbitrary strings it can just scan for an ID.
Value: Message IDs generally have the same meaning across releases of the product. This means that the developer can change the text of the log message without worrying about breaking someone's automation. Example: Version 1 of the product- ERROR(VDSM1000): Problem mounting drive. Version 2 of the product- ERROR(VDSM1000): unable to contact 127.0.0.1 for mounting. Try blah blah.
Does this mean that logging messages are an ABI that needs to be maintained over time?
The degree to which message IDs need to be maintained is generally a function of how dynamic the code is and how much you want to document what the ID actually means. If, on the WIKI wherein you define messages, you elect to say [1] then this "ABI" doesn't need any maintenance whatsoever.
[1] VDSM1000 == "Call support or post to the forum to figure out what the heck the program just did"
That's probably very difficult to do right.
Lots of other software programs have been successful at it [1], [2]. I would prefer to think of it in terms of how easy we want to make it for people to self diagnose. Also, without message IDs supporting source that is i18n is nearly impossible unless everyone is multilingual.
[1] http://www.cisco.com/en/US/products/sw/netmgtsw/ps1982/products_command_refe... [2] http://mail.google.com/support/bin/answer.py?answer=12336
Regards,
Anthony Liguori
Value: If you do it right search engines will index your message IDs and take users to a page where they can hopefully get more detail on the issue and potentially "self solve" the problem without calling support. Put this into a search engine... "KDFAP0020S"
Value: Message IDs allow you to easily describe flows. You might be able to say... "if you saw ID123 then look farther back for ID124 and if you see ID124 then the problem is _probably_ XYZ."
I am probably not doing enough justice to the value of message IDs but, I will say that lots of very large companies use them in complex software applications to solve these and other issues.
Cheers, Keith
vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/vdsm-devel
vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/vdsm-devel
vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/vdsm-devel
OK, unless anyone has anything else to add I'm going to assume that the project maintainers are OK with message IDs and I'll start the process.
To summarize: 1. I'm going to do some minor surgery to the logger so that the log format is pinned an not user modifiable. This is necessary to ensure that message IDs can be substituted into the string. 2. Message IDs will have the following format: VDSM##### 3. Message IDs will just be a simple up-counter across all of VDSM. 4. Existing strings will be converted to translatable strings. The string itself won't be changed it will just be wrapped by _(...) so that gettext will work. 5. Message IDs may be documented, somewhere not sure where yet, with a brief explanation (if an explanation is appropriate). This part might take some and; hopefully, the explanations will evolve. I expect many of the IDs to not have explanations right away though and I definitely don't want to put any bad explanations up or ones that haven't been vetted.
Cheers, Keith
On 10/06/2011 03:29 PM, Keith Robertson wrote:
On 10/06/2011 03:11 PM, Ayal Baron wrote:
>>> I would like to propose enhancing the VDSM source to emit >>> message >>> IDs in >>> log messages with levels greater than DEBUG (i.e. INFO, ERROR, >>> WARN, >>> EXCEPTION, and CRITICAL). Messages IDs have been used >>> successfully in >>> many products and they can aid in debugging, documentation, >>> root >>> cause >>> analysis, and internationalization. >>> >>> I have scanned the source and have determined that there are >>> roughly 525 >>> calls to log.[error|warn|info|critical|exception] and I am >>> volunteering >>> to go through and add IDs to them all but, before I do I want >>> to get >>> buy-in from the project maintainers. While adding IDs to the
wrt i18n, I'm not entirely convinced.
Yeah, i18n is one reason to do it but it is only relevant if you take the time to create lang-packs for different nationalities (which can be expensive). However, it is a cakewalk to code for in Python _("string here"). So you can code for it ... and if someone ever wanted to take the time create a lang-pack you'd be set.
Every line we log has the module name and line (never localized) which logged it so effectively we already have this info. However, the ID would remain constant where lines will change place between versions so it makes it easier. I also agree about the lookup and being able to write simpler parsers.
So I'm in.
Cool.
>>> messages I >>> would also enable them for i18n. >>> >>> I have written a rough plan for adding messages IDs to VDSM >>> below >>> and >>> I've provided some examples of products/projects that use >>> message >>> IDs. >>> >>> Examples of Projects that Use Message IDs: >>> - JBoss 7 is adopting Message IDs for all of its logging. See >>> the >>> annotations (e.g. @Message(id = 12000, ) in the following >>> piece of >>> source: >>> https://github.com/jbossas/jboss-as/blob/master/process-controller/src/main/... >>> >>> >>> >>> >>> - Almost all IBM products use message IDs and they do a fairly >>> good >>> job >>> of documenting them. Google has indexed them all and you can >>> simply >>> type >>> them in to a search and get a description of the problem that >>> is >>> associated with the ID. >>> http://publib.boulder.ibm.com/infocenter/tivihelp/v42r1/topic/com.ibm.omegam... >>> >>> >>> >>> >>> >>> >>> Example of a Log Statement with a Message ID in Python: >>> FORMAT = '%(asctime)s %(lineno)s %(levelname)s(%(messageid)s): >>> %(message)s' >>> logging.basicConfig(format=FORMAT) >>> logger = logging.getLogger() >>> logger.warning('NFS problem. Unable to mount.', extra={ >>> 'messageid' : >>> 'VDSM1000'}) >>> >>> // Produces >>> 2011-10-05 13:24:43,180 11 WARNING(VDSM1000): NFS problem. >>> Unable >>> to mount. >>> >>> >>> Example of a Log Statement with a Message ID in Python with >>> i18n: >>> logger.warning(_('NFS problem. Unable to mount.'), extra={ >>> 'messageid' : >>> 'VDSM1000'}) >>> >>> Example of an i18n Log Message: >>> Chinese: 2011年10月5日13:24:43,18011警告（VDSM1000）：NFS问 >>> 题。无法 >>> 安 装。 >>> English: 2011-10-05 13:24:43,180 11 WARNING(VDSM1000): NFS >>> problem. >>> Unable to mount. >>> >>> >>> >>> A Rough Plan for adding Message IDs to VDSM: >>> >>> Step 0: Decide on a logging format of the ID itself: >>> - JBoss is using "JBAS#####". I am thinking that VDSM could >>> probably do >>> something similar "VDSM####". >>> >>> Step 1: Decide where to list and document all of the IDs. >>> - It could be as simple as a page on VDSM's fedorahosted wiki. >>> >>> Step 2: Pin the format of the log messages: >>> - Currently VDSM allows a user configurable Python log format. >>> We >>> would >>> need to pin the format so that it includes a substitution >>> variable for >>> the message ID. >>> Example: '%(asctime)s %(lineno)s %(levelname)s(%(messageid)s): >>> %(message)s' >>> >>> Step 4: Plow through the VDSM source and add message IDs. >>> >>> >>> Cheers, >>> Keith >>> >>> >>> >>> >>> >>> _______________________________________________ >>> vdsm-devel mailing list >>> vdsm-devel@lists.fedorahosted.org >>> https://fedorahosted.org/mailman/listinfo/vdsm-devel >> Seems like a nice idea. But how does it affect developers. How >> do >> they choose mIDs? How do you prevent collisions? How are they >> different from simple gettext? As I never used this myself so I can't really imagine the all benefits of using such a system, at least enough to warrant all the tedious work. If you could just list out some of the use cases it'll be most helpful.
For instance I assume grepping for an ID is easier then grepping for a message especially in multiple versions where the message might have changed.
Sure. As far as the tedious part, yes, adding message IDs to an existing source tree is a bit of a PITA but the 525 invocations in VDSM isn't all that bad. I've seen much worse. Plus, once it's done the effort is much smaller (new lines of code only).
As for the value (I've tried to itemize a few common reasons for doing it):
Value: Message IDs allow you to use gettext globalize your messages. If you just used gettext and pulled the strings out of your code for globalization then you might end up with something like this in vdsm.log... 错误：问题安装驱动器 the English translation (according to google) is: ERROR: Problem mounting drive Unless you can read Chinese, then the message above is completely unhelpful to a support engineer. The converse is true also. Consider a non-english speaking person trying to decipher "ERROR: Problem mounting drive". With a message ID; however, you get this... 错误（VDSM1000）：安装驱动问题
Is it possible to just hash the main string before internationalization?
So basically:
fprintf(f, "[%s] ERROR(VDSM%04x) %s", date_str, crc32(message), message);
Yes, a hash of the string is possible. However, would make it hard for someone to document the ID because they would have to hash the string to get the ID.
One of the features of message ID/codes is that they are documentable.
Or the equivalent Python.
Value: Message IDs allow other applications to perform actions based on what is in your logs. On the mainframe, this is extremely common. You'll have one program watching the logs for a particular ID(s). When the program sees the ID it will look in a dictionary and decide what (if anything) needs to be done. On linux, logwatch, is frequently used for this purpose and message IDs make this process much simpler because the program doesn't have to write complicated REGEXs to look for arbitrary strings it can just scan for an ID.
Value: Message IDs generally have the same meaning across releases of the product. This means that the developer can change the text of the log message without worrying about breaking someone's automation. Example: Version 1 of the product- ERROR(VDSM1000): Problem mounting drive. Version 2 of the product- ERROR(VDSM1000): unable to contact 127.0.0.1 for mounting. Try blah blah.
Does this mean that logging messages are an ABI that needs to be maintained over time?
The degree to which message IDs need to be maintained is generally a function of how dynamic the code is and how much you want to document what the ID actually means. If, on the WIKI wherein you define messages, you elect to say [1] then this "ABI" doesn't need any maintenance whatsoever.
[1] VDSM1000 == "Call support or post to the forum to figure out what the heck the program just did"
That's probably very difficult to do right.
Lots of other software programs have been successful at it [1], [2]. I would prefer to think of it in terms of how easy we want to make it for people to self diagnose. Also, without message IDs supporting source that is i18n is nearly impossible unless everyone is multilingual.
[1] http://www.cisco.com/en/US/products/sw/netmgtsw/ps1982/products_command_refe...
[2] http://mail.google.com/support/bin/answer.py?answer=12336
Regards,
Anthony Liguori
Value: If you do it right search engines will index your message IDs and take users to a page where they can hopefully get more detail on the issue and potentially "self solve" the problem without calling support. Put this into a search engine... "KDFAP0020S"
Value: Message IDs allow you to easily describe flows. You might be able to say... "if you saw ID123 then look farther back for ID124 and if you see ID124 then the problem is _probably_ XYZ."
I am probably not doing enough justice to the value of message IDs but, I will say that lots of very large companies use them in complex software applications to solve these and other issues.
Cheers, Keith
vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/vdsm-devel
vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/vdsm-devel
vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/vdsm-devel
----- Original Message -----
OK, unless anyone has anything else to add I'm going to assume that the project maintainers are OK with message IDs and I'll start the process.
Danken, where do you stand on this?
To summarize:
- I'm going to do some minor surgery to the logger so that the log
format is pinned an not user modifiable. This is necessary to ensure that message IDs can be substituted into the string. 2. Message IDs will have the following format: VDSM##### 3. Message IDs will just be a simple up-counter across all of VDSM. 4. Existing strings will be converted to translatable strings. The string itself won't be changed it will just be wrapped by _(...) so that gettext will work. 5. Message IDs may be documented, somewhere not sure where yet, with a brief explanation (if an explanation is appropriate). This part might take some and; hopefully, the explanations will evolve. I expect many of the IDs to not have explanations right away though and I definitely don't want to put any bad explanations up or ones that haven't been vetted.
Cheers, Keith
On 10/06/2011 03:29 PM, Keith Robertson wrote:
On 10/06/2011 03:11 PM, Ayal Baron wrote:
>>>> I would like to propose enhancing the VDSM source to emit >>>> message >>>> IDs in >>>> log messages with levels greater than DEBUG (i.e. INFO, >>>> ERROR, >>>> WARN, >>>> EXCEPTION, and CRITICAL). Messages IDs have been used >>>> successfully in >>>> many products and they can aid in debugging, documentation, >>>> root >>>> cause >>>> analysis, and internationalization. >>>> >>>> I have scanned the source and have determined that there >>>> are >>>> roughly 525 >>>> calls to log.[error|warn|info|critical|exception] and I am >>>> volunteering >>>> to go through and add IDs to them all but, before I do I >>>> want >>>> to get >>>> buy-in from the project maintainers. While adding IDs to >>>> the
wrt i18n, I'm not entirely convinced.
Yeah, i18n is one reason to do it but it is only relevant if you take the time to create lang-packs for different nationalities (which can be expensive). However, it is a cakewalk to code for in Python _("string here"). So you can code for it ... and if someone ever wanted to take the time create a lang-pack you'd be set.
Every line we log has the module name and line (never localized) which logged it so effectively we already have this info. However, the ID would remain constant where lines will change place between versions so it makes it easier. I also agree about the lookup and being able to write simpler parsers.
So I'm in.
Cool.
>>>> messages I >>>> would also enable them for i18n. >>>> >>>> I have written a rough plan for adding messages IDs to VDSM >>>> below >>>> and >>>> I've provided some examples of products/projects that use >>>> message >>>> IDs. >>>> >>>> Examples of Projects that Use Message IDs: >>>> - JBoss 7 is adopting Message IDs for all of its logging. >>>> See >>>> the >>>> annotations (e.g. @Message(id = 12000, ) in the following >>>> piece of >>>> source: >>>> https://github.com/jbossas/jboss-as/blob/master/process-controller/src/main/... >>>> >>>> >>>> >>>> >>>> - Almost all IBM products use message IDs and they do a >>>> fairly >>>> good >>>> job >>>> of documenting them. Google has indexed them all and you >>>> can >>>> simply >>>> type >>>> them in to a search and get a description of the problem >>>> that >>>> is >>>> associated with the ID. >>>> http://publib.boulder.ibm.com/infocenter/tivihelp/v42r1/topic/com.ibm.omegam... >>>> >>>> >>>> >>>> >>>> >>>> >>>> Example of a Log Statement with a Message ID in Python: >>>> FORMAT = '%(asctime)s %(lineno)s >>>> %(levelname)s(%(messageid)s): >>>> %(message)s' >>>> logging.basicConfig(format=FORMAT) >>>> logger = logging.getLogger() >>>> logger.warning('NFS problem. Unable to mount.', extra={ >>>> 'messageid' : >>>> 'VDSM1000'}) >>>> >>>> // Produces >>>> 2011-10-05 13:24:43,180 11 WARNING(VDSM1000): NFS problem. >>>> Unable >>>> to mount. >>>> >>>> >>>> Example of a Log Statement with a Message ID in Python with >>>> i18n: >>>> logger.warning(_('NFS problem. Unable to mount.'), extra={ >>>> 'messageid' : >>>> 'VDSM1000'}) >>>> >>>> Example of an i18n Log Message: >>>> Chinese: 2011年10月5日13:24:43,18011警告（VDSM1000）：NFS问 >>>> 题。无法 >>>> 安 装。 >>>> English: 2011-10-05 13:24:43,180 11 WARNING(VDSM1000): NFS >>>> problem. >>>> Unable to mount. >>>> >>>> >>>> >>>> A Rough Plan for adding Message IDs to VDSM: >>>> >>>> Step 0: Decide on a logging format of the ID itself: >>>> - JBoss is using "JBAS#####". I am thinking that VDSM could >>>> probably do >>>> something similar "VDSM####". >>>> >>>> Step 1: Decide where to list and document all of the IDs. >>>> - It could be as simple as a page on VDSM's fedorahosted >>>> wiki. >>>> >>>> Step 2: Pin the format of the log messages: >>>> - Currently VDSM allows a user configurable Python log >>>> format. >>>> We >>>> would >>>> need to pin the format so that it includes a substitution >>>> variable for >>>> the message ID. >>>> Example: '%(asctime)s %(lineno)s >>>> %(levelname)s(%(messageid)s): >>>> %(message)s' >>>> >>>> Step 4: Plow through the VDSM source and add message IDs. >>>> >>>> >>>> Cheers, >>>> Keith >>>> >>>> >>>> >>>> >>>> >>>> _______________________________________________ >>>> vdsm-devel mailing list >>>> vdsm-devel@lists.fedorahosted.org >>>> https://fedorahosted.org/mailman/listinfo/vdsm-devel >>> Seems like a nice idea. But how does it affect developers. >>> How >>> do >>> they choose mIDs? How do you prevent collisions? How are >>> they >>> different from simple gettext? > As I never used this myself so I can't really imagine the all > benefits > of using such a system, at least enough to warrant all the > tedious > work. If you could just list out some of the use cases it'll > be > most > helpful. > > For instance I assume grepping for an ID is easier then > grepping > for a > message especially in multiple versions where the message > might > have > changed. Sure. As far as the tedious part, yes, adding message IDs to an existing source tree is a bit of a PITA but the 525 invocations in VDSM isn't all that bad. I've seen much worse. Plus, once it's done the effort is much smaller (new lines of code only).
As for the value (I've tried to itemize a few common reasons for doing it):
Value: Message IDs allow you to use gettext globalize your messages. If you just used gettext and pulled the strings out of your code for globalization then you might end up with something like this in vdsm.log... 错误：问题安装驱动器 the English translation (according to google) is: ERROR: Problem mounting drive Unless you can read Chinese, then the message above is completely unhelpful to a support engineer. The converse is true also. Consider a non-english speaking person trying to decipher "ERROR: Problem mounting drive". With a message ID; however, you get this... 错误（VDSM1000）：安装驱动问题
Is it possible to just hash the main string before internationalization?
So basically:
fprintf(f, "[%s] ERROR(VDSM%04x) %s", date_str, crc32(message), message);
Yes, a hash of the string is possible. However, would make it hard for someone to document the ID because they would have to hash the string to get the ID.
One of the features of message ID/codes is that they are documentable.
Or the equivalent Python.
Value: Message IDs allow other applications to perform actions based on what is in your logs. On the mainframe, this is extremely common. You'll have one program watching the logs for a particular ID(s). When the program sees the ID it will look in a dictionary and decide what (if anything) needs to be done. On linux, logwatch, is frequently used for this purpose and message IDs make this process much simpler because the program doesn't have to write complicated REGEXs to look for arbitrary strings it can just scan for an ID.
Value: Message IDs generally have the same meaning across releases of the product. This means that the developer can change the text of the log message without worrying about breaking someone's automation. Example: Version 1 of the product- ERROR(VDSM1000): Problem mounting drive. Version 2 of the product- ERROR(VDSM1000): unable to contact 127.0.0.1 for mounting. Try blah blah.
Does this mean that logging messages are an ABI that needs to be maintained over time?
The degree to which message IDs need to be maintained is generally a function of how dynamic the code is and how much you want to document what the ID actually means. If, on the WIKI wherein you define messages, you elect to say [1] then this "ABI" doesn't need any maintenance whatsoever.
[1] VDSM1000 == "Call support or post to the forum to figure out what the heck the program just did"
That's probably very difficult to do right.
Lots of other software programs have been successful at it [1], [2]. I would prefer to think of it in terms of how easy we want to make it for people to self diagnose. Also, without message IDs supporting source that is i18n is nearly impossible unless everyone is multilingual.
[1] http://www.cisco.com/en/US/products/sw/netmgtsw/ps1982/products_command_refe...
[2] http://mail.google.com/support/bin/answer.py?answer=12336
Regards,
Anthony Liguori
Value: If you do it right search engines will index your message IDs and take users to a page where they can hopefully get more detail on the issue and potentially "self solve" the problem without calling support. Put this into a search engine... "KDFAP0020S"
Value: Message IDs allow you to easily describe flows. You might be able to say... "if you saw ID123 then look farther back for ID124 and if you see ID124 then the problem is _probably_ XYZ."
I am probably not doing enough justice to the value of message IDs but, I will say that lots of very large companies use them in complex software applications to solve these and other issues.
Cheers, Keith
vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/vdsm-devel
vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/vdsm-devel
vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/vdsm-devel
vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/vdsm-devel
On Fri, Oct 07, 2011 at 10:28:17AM -0400, Keith Robertson wrote:
OK, unless anyone has anything else to add I'm going to assume that the project maintainers are OK with message IDs and I'll start the process.
I must say that from my perspective, this adds a lot of work for a very small gain. I suspect that your perspective, of having to correlate different logs, is slightly different.
Even though you've volunteered to do the first giant leap for man kind, it is going to be quite tedious for me to maintain it for each and every future log-using patch. I shiver of the thought of one patch adding VDSM1234 in parallel to another one - I hate to be the synchronization mechanism of this. Surely, we could use git hooks to help, but no matter how you look at it, it adds pain to development process. Even the simple NACK for "dude, you've forgot _(bla)" is counter-effective.
However, if consensus is reached that this is truly helpful for users trying to figure out what went wrong, I am willing to to dive in.
To summarize:
- I'm going to do some minor surgery to the logger so that the log
format is pinned an not user modifiable. This is necessaory to ensure that message IDs can be substituted into the string.
Do you mean a log adapter? Something else?
- Message IDs will have the following format: VDSM#####
- Message IDs will just be a simple up-counter across all of VDSM.
and a `make check` test to find collisions, and a `make nextID` rule to find the next free message ID.
- Existing strings will be converted to translatable strings. The
string itself won't be changed it will just be wrapped by _(...) so that gettext will work. 5. Message IDs may be documented, somewhere not sure where yet, with a brief explanation (if an explanation is appropriate). This part might take some and; hopefully, the explanations will evolve. I expect many of the IDs to not have explanations right away though and I definitely don't want to put any bad explanations up or ones that haven't been vetted.
Since the explanations are going to sit remotely from the explained code bit, they would be quite susceptible to comment rot.
As you can see, I'm not yet thrilled about this suggested project. Let's see how a proof-of-concept of this looks like.
Regards, Dan.
----- Original Message -----
On Fri, Oct 07, 2011 at 10:28:17AM -0400, Keith Robertson wrote:
OK, unless anyone has anything else to add I'm going to assume that the project maintainers are OK with message IDs and I'll start the process.
I must say that from my perspective, this adds a lot of work for a very small gain. I suspect that your perspective, of having to correlate different logs, is slightly different.
Even though you've volunteered to do the first giant leap for man kind, it is going to be quite tedious for me to maintain it for each and every future log-using patch. I shiver of the thought of one patch adding VDSM1234 in parallel to another one - I hate to be the synchronization mechanism of this. Surely, we could use git hooks to help, but no matter how you look at it, it adds pain to development process. Even the simple NACK for "dude, you've forgot _(bla)" is counter-effective.
This can be easily and *entirely* automated. In fact it could be autoresolved so you wouldn't even have to nack it (found a conflict with msgId? automatically change that line in the patch, should be quite simple even).
However, if consensus is reached that this is truly helpful for users trying to figure out what went wrong, I am willing to to dive in.
To summarize:
- I'm going to do some minor surgery to the logger so that the log
format is pinned an not user modifiable. This is necessaory to ensure that message IDs can be substituted into the string.
Do you mean a log adapter? Something else?
- Message IDs will have the following format: VDSM#####
- Message IDs will just be a simple up-counter across all of VDSM.
and a `make check` test to find collisions, and a `make nextID` rule to find the next free message ID.
- Existing strings will be converted to translatable strings. The
string itself won't be changed it will just be wrapped by _(...) so that gettext will work. 5. Message IDs may be documented, somewhere not sure where yet, with a brief explanation (if an explanation is appropriate). This part might take some and; hopefully, the explanations will evolve. I expect many of the IDs to not have explanations right away though and I definitely don't want to put any bad explanations up or ones that haven't been vetted.
Since the explanations are going to sit remotely from the explained code bit, they would be quite susceptible to comment rot.
As you can see, I'm not yet thrilled about this suggested project. Let's see how a proof-of-concept of this looks like.
Regards, Dan. _______________________________________________ vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/vdsm-devel
On 10/09/2011 05:53 PM, Ayal Baron wrote:
----- Original Message -----
On Fri, Oct 07, 2011 at 10:28:17AM -0400, Keith Robertson wrote:
OK, unless anyone has anything else to add I'm going to assume that the project maintainers are OK with message IDs and I'll start the process.
I must say that from my perspective, this adds a lot of work for a very small gain. I suspect that your perspective, of having to correlate different logs, is slightly different.
Even though you've volunteered to do the first giant leap for man kind, it is going to be quite tedious for me to maintain it for each and every future log-using patch. I shiver of the thought of one patch adding VDSM1234 in parallel to another one - I hate to be the synchronization mechanism of this. Surely, we could use git hooks to help, but no matter how you look at it, it adds pain to development process. Even the simple NACK for "dude, you've forgot _(bla)" is counter-effective.
This can be easily and *entirely* automated. In fact it could be autoresolved so you wouldn't even have to nack it (found a conflict with msgId? automatically change that line in the patch, should be quite simple even).
Continuing, Ayal's theme, I will do the following to make this easier for everyone... 1. Create a git hook that checks incoming changes for existing IDs and emits a warning. Maybe, I'll even get it to suggest the next ID. 2. Enhance the Makefile with a couple of new targets: 2.1: One target to check for duplicate IDs 2.2: One target to suggest the next ID. 2.3: Target(s) to do gettext (see the makefile I sent earlier.)
Cheers, Keith
However, if consensus is reached that this is truly helpful for users trying to figure out what went wrong, I am willing to to dive in.
To summarize:
- I'm going to do some minor surgery to the logger so that the log
format is pinned an not user modifiable. This is necessaory to ensure that message IDs can be substituted into the string.
Do you mean a log adapter? Something else?
- Message IDs will have the following format: VDSM#####
- Message IDs will just be a simple up-counter across all of VDSM.
and a `make check` test to find collisions, and a `make nextID` rule to find the next free message ID.
- Existing strings will be converted to translatable strings. The
string itself won't be changed it will just be wrapped by _(...) so that gettext will work. 5. Message IDs may be documented, somewhere not sure where yet, with a brief explanation (if an explanation is appropriate). This part might take some and; hopefully, the explanations will evolve. I expect many of the IDs to not have explanations right away though and I definitely don't want to put any bad explanations up or ones that haven't been vetted.
Since the explanations are going to sit remotely from the explained code bit, they would be quite susceptible to comment rot.
As you can see, I'm not yet thrilled about this suggested project. Let's see how a proof-of-concept of this looks like.
Regards, Dan. _______________________________________________ vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/vdsm-devel
vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/vdsm-devel
----- Original Message -----
On 10/09/2011 05:53 PM, Ayal Baron wrote:
----- Original Message -----
On Fri, Oct 07, 2011 at 10:28:17AM -0400, Keith Robertson wrote:
OK, unless anyone has anything else to add I'm going to assume that the project maintainers are OK with message IDs and I'll start the process.
I must say that from my perspective, this adds a lot of work for a very small gain. I suspect that your perspective, of having to correlate different logs, is slightly different.
Even though you've volunteered to do the first giant leap for man kind, it is going to be quite tedious for me to maintain it for each and every future log-using patch. I shiver of the thought of one patch adding VDSM1234 in parallel to another one - I hate to be the synchronization mechanism of this. Surely, we could use git hooks to help, but no matter how you look at it, it adds pain to development process. Even the simple NACK for "dude, you've forgot _(bla)" is counter-effective.
This can be easily and *entirely* automated. In fact it could be autoresolved so you wouldn't even have to nack it (found a conflict with msgId? automatically change that line in the patch, should be quite simple even).
Continuing, Ayal's theme, I will do the following to make this easier for everyone...
- Create a git hook that checks incoming changes for existing IDs
and emits a warning. Maybe, I'll even get it to suggest the next ID. 2. Enhance the Makefile with a couple of new targets: 2.1: One target to check for duplicate IDs 2.2: One target to suggest the next ID. 2.3: Target(s) to do gettext (see the makefile I sent earlier.)
Please wait with actually working on this, I think there is some contention about the value/cost ratio of this change. I would like to see comments from a few extra people and I would hate for you to do all this work just for it to get nack'd.
Cheers, Keith
However, if consensus is reached that this is truly helpful for users trying to figure out what went wrong, I am willing to to dive in.
To summarize:
- I'm going to do some minor surgery to the logger so that the
log format is pinned an not user modifiable. This is necessaory to ensure that message IDs can be substituted into the string.
Do you mean a log adapter? Something else?
- Message IDs will have the following format: VDSM#####
- Message IDs will just be a simple up-counter across all of
VDSM.
and a `make check` test to find collisions, and a `make nextID` rule to find the next free message ID.
- Existing strings will be converted to translatable strings.
The string itself won't be changed it will just be wrapped by _(...) so that gettext will work. 5. Message IDs may be documented, somewhere not sure where yet, with a brief explanation (if an explanation is appropriate). This part might take some and; hopefully, the explanations will evolve. I expect many of the IDs to not have explanations right away though and I definitely don't want to put any bad explanations up or ones that haven't been vetted.
Since the explanations are going to sit remotely from the explained code bit, they would be quite susceptible to comment rot.
As you can see, I'm not yet thrilled about this suggested project. Let's see how a proof-of-concept of this looks like.
Regards, Dan. _______________________________________________ vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/vdsm-devel
vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/vdsm-devel
vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/vdsm-devel
On 10/10/2011 08:44 AM, Ayal Baron wrote:
----- Original Message -----
On 10/09/2011 05:53 PM, Ayal Baron wrote:
----- Original Message -----
On Fri, Oct 07, 2011 at 10:28:17AM -0400, Keith Robertson wrote:
OK, unless anyone has anything else to add I'm going to assume that the project maintainers are OK with message IDs and I'll start the process.
I must say that from my perspective, this adds a lot of work for a very small gain. I suspect that your perspective, of having to correlate different logs, is slightly different.
Even though you've volunteered to do the first giant leap for man kind, it is going to be quite tedious for me to maintain it for each and every future log-using patch. I shiver of the thought of one patch adding VDSM1234 in parallel to another one - I hate to be the synchronization mechanism of this. Surely, we could use git hooks to help, but no matter how you look at it, it adds pain to development process. Even the simple NACK for "dude, you've forgot _(bla)" is counter-effective.
This can be easily and *entirely* automated. In fact it could be autoresolved so you wouldn't even have to nack it (found a conflict with msgId? automatically change that line in the patch, should be quite simple even).
Continuing, Ayal's theme, I will do the following to make this easier for everyone...
- Create a git hook that checks incoming changes for existing IDs
and emits a warning. Maybe, I'll even get it to suggest the next ID. 2. Enhance the Makefile with a couple of new targets: 2.1: One target to check for duplicate IDs 2.2: One target to suggest the next ID. 2.3: Target(s) to do gettext (see the makefile I sent earlier.)
Please wait with actually working on this, I think there is some contention about the value/cost ratio of this change. I would like to see comments from a few extra people and I would hate for you to do all this work just for it to get nack'd.
Understood. Can I get an ETA though?
Cheers, Keith
However, if consensus is reached that this is truly helpful for users trying to figure out what went wrong, I am willing to to dive in.
To summarize:
- I'm going to do some minor surgery to the logger so that the
log format is pinned an not user modifiable. This is necessaory to ensure that message IDs can be substituted into the string.
Do you mean a log adapter? Something else?
- Message IDs will have the following format: VDSM#####
- Message IDs will just be a simple up-counter across all of
VDSM.
and a `make check` test to find collisions, and a `make nextID` rule to find the next free message ID.
- Existing strings will be converted to translatable strings.
The string itself won't be changed it will just be wrapped by _(...) so that gettext will work. 5. Message IDs may be documented, somewhere not sure where yet, with a brief explanation (if an explanation is appropriate). This part might take some and; hopefully, the explanations will evolve. I expect many of the IDs to not have explanations right away though and I definitely don't want to put any bad explanations up or ones that haven't been vetted.
Since the explanations are going to sit remotely from the explained code bit, they would be quite susceptible to comment rot.
As you can see, I'm not yet thrilled about this suggested project. Let's see how a proof-of-concept of this looks like.
Regards, Dan. _______________________________________________ vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/vdsm-devel
vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/vdsm-devel
vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/vdsm-devel
vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/vdsm-devel
----- Original Message -----
On 10/10/2011 08:44 AM, Ayal Baron wrote:
----- Original Message -----
On 10/09/2011 05:53 PM, Ayal Baron wrote:
----- Original Message -----
On Fri, Oct 07, 2011 at 10:28:17AM -0400, Keith Robertson wrote:
OK, unless anyone has anything else to add I'm going to assume that the project maintainers are OK with message IDs and I'll start the process.
I must say that from my perspective, this adds a lot of work for a very small gain. I suspect that your perspective, of having to correlate different logs, is slightly different.
Even though you've volunteered to do the first giant leap for man kind, it is going to be quite tedious for me to maintain it for each and every future log-using patch. I shiver of the thought of one patch adding VDSM1234 in parallel to another one - I hate to be the synchronization mechanism of this. Surely, we could use git hooks to help, but no matter how you look at it, it adds pain to development process. Even the simple NACK for "dude, you've forgot _(bla)" is counter-effective.
This can be easily and *entirely* automated. In fact it could be autoresolved so you wouldn't even have to nack it (found a conflict with msgId? automatically change that line in the patch, should be quite simple even).
Continuing, Ayal's theme, I will do the following to make this easier for everyone...
- Create a git hook that checks incoming changes for existing IDs
and emits a warning. Maybe, I'll even get it to suggest the next ID. 2. Enhance the Makefile with a couple of new targets: 2.1: One target to check for duplicate IDs 2.2: One target to suggest the next ID. 2.3: Target(s) to do gettext (see the makefile I sent earlier.)
Please wait with actually working on this, I think there is some contention about the value/cost ratio of this change. I would like to see comments from a few extra people and I would hate for you to do all this work just for it to get nack'd.
Understood. Can I get an ETA though?
Need to solicit responses on the list. Dan, Saggi, Edu, Federico, please reply with your thoughts on this.
We already know that in the next 2 versions the code is going to change *a lot*. So it's not clear what the value of doing this at the present moment is (adds pains for something that will change 10 times in the next year or so).
Cheers, Keith
However, if consensus is reached that this is truly helpful for users trying to figure out what went wrong, I am willing to to dive in.
To summarize:
- I'm going to do some minor surgery to the logger so that the
log format is pinned an not user modifiable. This is necessaory to ensure that message IDs can be substituted into the string.
Do you mean a log adapter? Something else?
- Message IDs will have the following format: VDSM#####
- Message IDs will just be a simple up-counter across all of
VDSM.
and a `make check` test to find collisions, and a `make nextID` rule to find the next free message ID.
- Existing strings will be converted to translatable strings.
The string itself won't be changed it will just be wrapped by _(...) so that gettext will work. 5. Message IDs may be documented, somewhere not sure where yet, with a brief explanation (if an explanation is appropriate). This part might take some and; hopefully, the explanations will evolve. I expect many of the IDs to not have explanations right away though and I definitely don't want to put any bad explanations up or ones that haven't been vetted.
Since the explanations are going to sit remotely from the explained code bit, they would be quite susceptible to comment rot.
As you can see, I'm not yet thrilled about this suggested project. Let's see how a proof-of-concept of this looks like.
Regards, Dan. _______________________________________________ vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/vdsm-devel
vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/vdsm-devel
vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/vdsm-devel
vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/vdsm-devel
vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/vdsm-devel
Need to solicit responses on the list. Dan, Saggi, Edu, Federico, please reply with your thoughts on this.
We already know that in the next 2 versions the code is going to change *a lot*. So it's not clear what the value of doing this at the present moment is (adds pains for something that will change 10 times in the next year or so).
If you are going with codes, it makes sense to add msg ids from the beginning, not as a retrofit. Yes, some message numbers may not be used when code changes discards a message, the overall process should be more consistent.
-subhendu
----- Original Message -----
From: "Subhendu Ghosh" sghosh@redhat.com To: "VDSM Project Development" vdsm-devel@lists.fedorahosted.org Sent: Wednesday, October 12, 2011 4:57:00 AM Subject: Re: Proposal to add message IDs to vdsm.log
Need to solicit responses on the list. Dan, Saggi, Edu, Federico, please reply with your thoughts on this.
We already know that in the next 2 versions the code is going to change *a lot*. So it's not clear what the value of doing this at the present moment is (adds pains for something that will change 10 times in the next year or so).
If you are going with codes, it makes sense to add msg ids from the beginning, not as a retrofit. Yes, some message numbers may not be used when code changes discards a message, the overall process should be more consistent.
My concern about this is that to be useful we must commit to the meaning of the IDs and in a case where not only the code changes a lot but also the internal flow changes (even though the APIs are consistent) we'll often need to provide new IDs (and remove old ones where needed). Therefore I feel that I won't be using IDs at all if I need to check what vdsm version I'm using to grep for an ID in vdsm.log.
We could postpone the message IDs into the future. In my opinion the only problem at the moment is bad timing. If there are plans to implement also an application that is going to collect the message IDs, analyze them and deal with their constant change then I'll agree on having them now.
Why not put our effort into more urgent matters as for example make vdsm/vdsmd.in 100% compatible with fedora using better tools. (PS. if anyone is interested in this please contact me)
We should maintain a list of changes that we would like to make so that we can discuss them and rate them.
On Wed, Oct 12, 2011 at 04:14:24AM -0400, Federico Simoncelli wrote:
We should maintain a list of changes that we would like to make so that we can discuss them and rate them.
+1 to maintaining a TODO list. I think this will help us to organize and prioritize. It will also enable us to make efficient use of what is sure to be an expanding developer base. Does vdsm have a home on the oVirt wiki yet?
On 10/12/2011 04:14 AM, Federico Simoncelli wrote:
----- Original Message -----
From: "Subhendu Ghosh"sghosh@redhat.com To: "VDSM Project Development"vdsm-devel@lists.fedorahosted.org Sent: Wednesday, October 12, 2011 4:57:00 AM Subject: Re: Proposal to add message IDs to vdsm.log
Need to solicit responses on the list. Dan, Saggi, Edu, Federico, please reply with your thoughts on this.
We already know that in the next 2 versions the code is going to change *a lot*. So it's not clear what the value of doing this at the present moment is (adds pains for something that will change 10 times in the next year or so).
If you are going with codes, it makes sense to add msg ids from the beginning, not as a retrofit. Yes, some message numbers may not be used when code changes discards a message, the overall process should be more consistent.
My concern about this is that to be useful we must commit to the meaning of the IDs and in a case where not only the code changes a lot but also the internal flow changes (even though the APIs are consistent) we'll often need to provide new IDs (and remove old ones where needed). Therefore I feel that I won't be using IDs at all if I need to check what vdsm version I'm using to grep for an ID in vdsm.log.
We could postpone the message IDs into the future. In my opinion the only problem at the moment is bad timing. If there are plans to implement also an application that is going to collect the message IDs, analyze them and deal with their constant change then I'll agree on having them now.
Why not put our effort into more urgent matters as for example make vdsm/vdsmd.in 100% compatible with fedora using better tools. (PS. if anyone is interested in this please contact me)
We should maintain a list of changes that we would like to make so that we can discuss them and rate them.
For some reference: http://cee.mitre.org/
Common Event Expression
vdsm-devel@lists.stg.fedorahosted.org
-
abaron＠redhat.com -
agl＠us.ibm.com -
Anthony Liguori -
Dan Kenigsberg -
Federico Simoncelli -
kroberts＠redhat.com -
smizrahi＠redhat.com -
Subhendu Ghosh