Hallo,
With koji-1.4 running on RHEL6-beta2, some email notifications are not sent out correctly:
Traceback (most recent call last): File "/usr/sbin/kojid", line 1437, in runTask response = (handler.run(),) File "/usr/sbin/kojid", line 1513, in run return self.handler(*self.params,**self.opts) File "/usr/sbin/kojid", line 3649, in handler message = message.encode('utf-8') UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 546: ordinal not in range(128)
Thie seems to happen e.g. on dejavu-fonts, ctags, crontabs, cronie, amanda, akonadi.
regards,
Florian La Roche
Thie seems to happen e.g. on dejavu-fonts, ctags, crontabs, cronie, amanda, akonadi.
The following change seems to work (tested with ctags and crontabs):
--- builder/kojid +++ builder/kojid @@ -3455,7 +3455,7 @@ Status: %(status)s\r
message = self.message_templ % locals() # ensure message is in UTF-8 - message = message.encode('utf-8') + message = koji.fixEncoding(message)
server = smtplib.SMTP(options.smtphost) #server.set_debuglevel(True) @@ -3646,7 +3646,7 @@ Build Info: %(weburl)s/buildinfo?buildID subject = self.subject_templ % locals() message = self.message_templ % locals() # ensure message is in UTF-8 - message = message.encode('utf-8') + message = koji.fixEncoding(message)
server = smtplib.SMTP(options.smtphost) # server.set_debuglevel(True)
I'll put this into a bugzilla and also upload my current koji rpm to http://jur-linux.org/rpms/el-updates/6/ again.
regards,
Florian La Roche
Florian La Roche píše v Čt 15. 07. 2010 v 06:00 +0200:
Hallo,
With koji-1.4 running on RHEL6-beta2, some email notifications are not sent out correctly:
Traceback (most recent call last): File "/usr/sbin/kojid", line 1437, in runTask response = (handler.run(),) File "/usr/sbin/kojid", line 1513, in run return self.handler(*self.params,**self.opts) File "/usr/sbin/kojid", line 3649, in handler message = message.encode('utf-8') UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 546: ordinal not in range(128)
Thie seems to happen e.g. on dejavu-fonts, ctags, crontabs, cronie, amanda, akonadi.
And that's the reason why I didn't received any notification from Fedora Koji about successful builds ...
Dan
On Wednesday 21 July 2010, Dan Horák wrote:
Florian La Roche píše v Čt 15. 07. 2010 v 06:00 +0200:
Hallo,
With koji-1.4 running on RHEL6-beta2, some email notifications are not sent out correctly:
Traceback (most recent call last): File "/usr/sbin/kojid", line 1437, in runTask
response = (handler.run(),)File "/usr/sbin/kojid", line 1513, in run
return self.handler(*self.params,**self.opts)File "/usr/sbin/kojid", line 3649, in handler
message = message.encode('utf-8')UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 546: ordinal not in range(128)
Thie seems to happen e.g. on dejavu-fonts, ctags, crontabs, cronie, amanda, akonadi.
And that's the reason why I didn't received any notification from Fedora Koji about successful builds ...
Neither have I, and not getting those mails has already started to interfere with my workflow which I'd rather not change... when can we expect a fix for this in the Fedora koji instance?
On Wednesday 21 July 2010, Ville Skyttä wrote:
On Wednesday 21 July 2010, Dan Horák wrote:
Florian La Roche píše v Čt 15. 07. 2010 v 06:00 +0200:
Hallo,
With koji-1.4 running on RHEL6-beta2, some email notifications are not sent out correctly:
Traceback (most recent call last): File "/usr/sbin/kojid", line 1437, in runTask
response = (handler.run(),)File "/usr/sbin/kojid", line 1513, in run
return self.handler(*self.params,**self.opts)File "/usr/sbin/kojid", line 3649, in handler
message = message.encode('utf-8')UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 546: ordinal not in range(128)
Thie seems to happen e.g. on dejavu-fonts, ctags, crontabs, cronie, amanda, akonadi.
And that's the reason why I didn't received any notification from Fedora Koji about successful builds ...
Neither have I, and not getting those mails has already started to interfere with my workflow which I'd rather not change... when can we expect a fix for this in the Fedora koji instance?
Ping? Still no notification mails for me, for example the modplugtools builds done yesterday, and I haven't seen any comments when this might get fixed.
On Wed, Aug 04, 2010 at 09:12:16AM +0300, Ville Skyttä wrote:
On Wednesday 21 July 2010, Ville Skyttä wrote:
On Wednesday 21 July 2010, Dan Horák wrote:
Florian La Roche píše v Čt 15. 07. 2010 v 06:00 +0200:
Hallo,
With koji-1.4 running on RHEL6-beta2, some email notifications are not sent out correctly:
Traceback (most recent call last): File "/usr/sbin/kojid", line 1437, in runTask
response = (handler.run(),)File "/usr/sbin/kojid", line 1513, in run
return self.handler(*self.params,**self.opts)File "/usr/sbin/kojid", line 3649, in handler
message = message.encode('utf-8')UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 546: ordinal not in range(128)
Thie seems to happen e.g. on dejavu-fonts, ctags, crontabs, cronie, amanda, akonadi.
And that's the reason why I didn't received any notification from Fedora Koji about successful builds ...
Neither have I, and not getting those mails has already started to interfere with my workflow which I'd rather not change... when can we expect a fix for this in the Fedora koji instance?
Ping? Still no notification mails for me, for example the modplugtools builds done yesterday, and I haven't seen any comments when this might get fixed.
I know practically nothing about the koji code but that code seems very wrong. This is earlier in the class::
message_templ = \ """From: %(from_addr)s\r Subject: %(nvr)s %(result)s %(operation)s by %(user_name)s\r To: %(to_addrs)s\r X-Koji-Package: %(pkg_name)s\r X-Koji-NVR: %(nvr)s\r X-Koji-User: %(user_name)s\r X-Koji-Status: %(status)s\r %(tag_headers)s\r \r Package: %(pkg_name)s\r NVR: %(nvr)s\r User: %(user_name)s\r Status: %(status)s\r %(operation_details)s\r %(nvr)s %(result)s %(operation)s by %(user_name)s\r %(failure_info)s\r """
Then in this method:: message = self.message_templ % locals() # ensure message is in UTF-8 message = message.encode('utf-8')
Notice that message_templ is a byte string? If all of the variables in locals() is also a byte string then message will be a byte string (if locals() contains unicode strings it will either become a unicode string or throw a traceback if it can't decode some other byte string to unicode).
The encode method of byte strings is pretty much useless. It allows you to take a byte string that's all ascii and transform it into another byte string. You can use it if you want to rot13 it but not much else. If the byte string contains any bytes that aren't in the ascii subset, it'll throw the error given above.
How to solve this depends on what you're actually trying to achieve. If it's really to make sure that the byte string only contains valid utf8 data and you don't care what the non-utf8 data becomes I'd do it like this:
# Note this will be unsafe if locals contains strings of type <unicode> message = self.message_templ % locals() message = unicode(message, encoding='utf8', errors='replace').encode('utf8')
That takes the byte string that is the message, turns it into a unicode string by treating the bytes as 'utf8', replacing any invalid bytes with �, and then reencoding it to utf8.
If you can't guarantee that you start off with only byte strings, you could use kitchen.text.converters.to_unicode to do the work or copy the essentials of what it does::
message = self.message_templ % locals() if isinstance(message, unicode): message = message.encode('utf8') else: message = unicode(message, encoding='utf8', errors='replace').encode('utf8')
-Toshio
On Friday 13 August 2010, Toshio Kuratomi wrote:
On Wed, Aug 04, 2010 at 09:12:16AM +0300, Ville Skyttä wrote:
On Wednesday 21 July 2010, Ville Skyttä wrote:
On Wednesday 21 July 2010, Dan Horák wrote:
Florian La Roche píše v Čt 15. 07. 2010 v 06:00 +0200:
Hallo,
With koji-1.4 running on RHEL6-beta2, some email notifications are not sent out correctly:
Traceback (most recent call last): File "/usr/sbin/kojid", line 1437, in runTask
response = (handler.run(),)File "/usr/sbin/kojid", line 1513, in run
return self.handler(*self.params,**self.opts)File "/usr/sbin/kojid", line 3649, in handler
message = message.encode('utf-8')UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 546: ordinal not in range(128)
Thie seems to happen e.g. on dejavu-fonts, ctags, crontabs, cronie, amanda, akonadi.
And that's the reason why I didn't received any notification from Fedora Koji about successful builds ...
Neither have I, and not getting those mails has already started to interfere with my workflow which I'd rather not change... when can we expect a fix for this in the Fedora koji instance?
Ping? Still no notification mails for me, for example the modplugtools builds done yesterday, and I haven't seen any comments when this might get fixed.
I know practically nothing about the koji code but that code seems very wrong. This is earlier in the class:: [...]
Thanks for taking the time to look into this. Maybe some changes have been made already - at least I received a build failure mail from today for a failed javasqlite build. I didn't receive one for example for successful seeker and rpmlint builds yesterday though :(
On Fri, Aug 20, 2010 at 05:06:33PM +0300, Ville Skyttä wrote:
On Friday 13 August 2010, Toshio Kuratomi wrote:
On Wed, Aug 04, 2010 at 09:12:16AM +0300, Ville Skyttä wrote:
Ping? Still no notification mails for me, for example the modplugtools builds done yesterday, and I haven't seen any comments when this might get fixed.
I know practically nothing about the koji code but that code seems very wrong. This is earlier in the class:: [...]
Thanks for taking the time to look into this. Maybe some changes have been made already - at least I received a build failure mail from today for a failed javasqlite build. I didn't receive one for example for successful seeker and rpmlint builds yesterday though :(
dgilmore says that we haven't applied any changes to the koji running in Fedora Infrastructure but that he needs to pull in a patch to hotfix our instance with. Since I didn't see a patch in the upstream git repository, I'm attaching one here.
-Toshio
Hello Toshio,
koji already has a generic routing called fixEncoding() and I am currently running koji with the patch from https://bugzilla.redhat.com/show_bug.cgi?id=614767.
Maybe your below code to use "errors='replace'" should go into the generic koji function? (Haven't looked into this myself if it makes sense...)
regards,
Florian La Roche
dgilmore says that we haven't applied any changes to the koji running in Fedora Infrastructure but that he needs to pull in a patch to hotfix our instance with. Since I didn't see a patch in the upstream git repository, I'm attaching one here.
-Toshio
From a3c0248a10c5ddbe9a34f76b995fbf10c34e87f5 Mon Sep 17 00:00:00 2001 From: Toshio Kuratomi toshio@fedoraproject.org Date: Fri, 20 Aug 2010 11:17:43 -0400 Subject: [PATCH] Fix Unicode traceback with notification email.
builder/kojid | 5 ++++- 1 files changed, 4 insertions(+), 1 deletions(-)
diff --git a/builder/kojid b/builder/kojid index 2f7f480..77ded36 100755 --- a/builder/kojid +++ b/builder/kojid @@ -3455,7 +3455,10 @@ Status: %(status)s\r
message = self.message_templ % locals() # ensure message is in UTF-8
message = message.encode('utf-8')
if isinstance(message, unicode):message = message.encode('utf-8')else:message = unicode(message, encoding='utf8', errors='replace').encode('utf8') server = smtplib.SMTP(options.smtphost) #server.set_debuglevel(True)-- 1.7.2.1
On Tue, Aug 31, 2010 at 09:54:07AM +0200, Florian La Roche wrote:
Hello Toshio,
koji already has a generic routing called fixEncoding() and I am currently running koji with the patch from https://bugzilla.redhat.com/show_bug.cgi?id=614767.
Maybe your below code to use "errors='replace'" should go into the generic koji function? (Haven't looked into this myself if it makes sense...)
Took a look -- fixEncoding() is almost as robust as my code[1]_. I think I prefer my code but the result of either transformation will be mangled byte strings. Basically we're being handed byte strings without a guarantee that the strings are utf8. So we first assume that they are utf8. If not, both pieces of code try someway to coerce the output to utf8. My code substitutes a replacement character for every illegal byte. So a string could end up looking like this 'café' (in some non-utf8 encoding): 'caf�'.
The fixEncoding() code fallsback to an encoding that covers the whole spectrum of possible bytes. So the output will substitute a random character (or characters if the invalid characters are a multi-byte sequence) in place of the unknown bytes.
It probably makes sense to use fixEncoding() everywhere as the patch in your bug report does. It could make sense to change fixEncoding to incorporate ideas from my code. The result would look like this:
import warnings def fixEncoding(value, from_encoding=None, fallback=None): # fallback is used for backwards compatibility if not from_encoding: if fallback: warnings.warn('fixEncoding() no longer takes a fallback' ' keyword arg. Use from_encoding instead.', DeprecationWarning, stacklevel=2) from_encoding = fallback else: from_encoding = 'utf8'
if isinstance(value, unicode): # value is already unicode, so just convert it # to a utf8-encoded str # Note: with python3, this can fail unless you use an error # argument because a unicode string could have been created using # the surrogateescape error handler. return value.encode('utf8', 'replace') else: # value is a str but may not be valid utf8 (encoded in latin1, for # instance). Note that the string is almost certain to be mangled # in these instances unless you know what encoding the string is in # and have set from_encoding to that encoding. return value.decode(from_encoding, 'replace').encode('utf8', 'replace')
[1]: fixEncoding's ``if not value: return value`` statement will lead to unicode errors in circumstances like this::
monkeywrench = fixEncoding(u'') name = fixEncoding(u'café') combined = monkeywrench + name
The empty string in monkeywrench won't be converted into a byte string. When you then try to combine the unicode string in monkeywrench with the byte string in name python will try to convert name into a unicode string with the ascii codec and fail.
-Toshio
regards,
Florian La Roche
dgilmore says that we haven't applied any changes to the koji running in Fedora Infrastructure but that he needs to pull in a patch to hotfix our instance with. Since I didn't see a patch in the upstream git repository, I'm attaching one here.
-Toshio
From a3c0248a10c5ddbe9a34f76b995fbf10c34e87f5 Mon Sep 17 00:00:00 2001 From: Toshio Kuratomi toshio@fedoraproject.org Date: Fri, 20 Aug 2010 11:17:43 -0400 Subject: [PATCH] Fix Unicode traceback with notification email.
builder/kojid | 5 ++++- 1 files changed, 4 insertions(+), 1 deletions(-)
diff --git a/builder/kojid b/builder/kojid index 2f7f480..77ded36 100755 --- a/builder/kojid +++ b/builder/kojid @@ -3455,7 +3455,10 @@ Status: %(status)s\r
message = self.message_templ % locals() # ensure message is in UTF-8
message = message.encode('utf-8')
if isinstance(message, unicode):message = message.encode('utf-8')else:message = unicode(message, encoding='utf8', errors='replace').encode('utf8') server = smtplib.SMTP(options.smtphost) #server.set_debuglevel(True)-- 1.7.2.1
-- buildsys mailing list buildsys@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/buildsys
On Friday 20 August 2010, Toshio Kuratomi wrote:
On Fri, Aug 20, 2010 at 05:06:33PM +0300, Ville Skyttä wrote:
On Friday 13 August 2010, Toshio Kuratomi wrote:
On Wed, Aug 04, 2010 at 09:12:16AM +0300, Ville Skyttä wrote:
Ping? Still no notification mails for me, for example the modplugtools builds done yesterday, and I haven't seen any comments when this might get fixed.
I know practically nothing about the koji code but that code seems very wrong. This is earlier in the class:: [...]
Thanks for taking the time to look into this. Maybe some changes have been made already - at least I received a build failure mail from today for a failed javasqlite build. I didn't receive one for example for successful seeker and rpmlint builds yesterday though :(
dgilmore says that we haven't applied any changes to the koji running in Fedora Infrastructure but that he needs to pull in a patch to hotfix our instance with. Since I didn't see a patch in the upstream git repository, I'm attaching one here.
Thanks again. I'm still not receiving any mails for successful builds, hopefully someone can do something about that soon.
On Wed, Jul 21, 2010 at 04:34:11PM +0200, Dan Horák wrote:
Florian La Roche píše v Čt 15. 07. 2010 v 06:00 +0200:
Hallo,
With koji-1.4 running on RHEL6-beta2, some email notifications are not sent out correctly:
Traceback (most recent call last): File "/usr/sbin/kojid", line 1437, in runTask response = (handler.run(),) File "/usr/sbin/kojid", line 1513, in run return self.handler(*self.params,**self.opts) File "/usr/sbin/kojid", line 3649, in handler message = message.encode('utf-8') UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 546: ordinal not in range(128)
Thie seems to happen e.g. on dejavu-fonts, ctags, crontabs, cronie, amanda, akonadi.
And that's the reason why I didn't received any notification from Fedora Koji about successful builds ...
Hello Dan,
right, nobody normally looks into the web pages to see this failure, but the log file also contains a "WARNING" about these and I've wanted a clean rebuild of all rpm packages.
regards,
Florian La Roche
buildsys@lists.fedoraproject.org
-
Dan Horák -
Florian La Roche -
Toshio Kuratomi -
Ville Skyttä