Hi folks,
Recently I wrote a Twisted-based chat bot that can interact with a Koji hub.
To do this, I wrote a "txkoji" client library: https://github.com/ktdreyer/txkoji
(the bot code is separate, this is just the koji client library)
txkoji allows me to interact with a Koji Hub in an event-driven way, without blocking the reactor for my program. This works well for client programs that must be responsive to input from several sources (like IRC and a message bus and many other inputs). I return the XML-RPC responses as Munch (dict-like) objects because I ran across Munch in a couple other Fedora infra projects and I like the syntax.
If Koji gets a REST API, I could convert this to use treq instead of Twisted's XML-RPC client.
- Ken
Neat!
Note that an authenticated koji session will probably not play nice with async use because of the callnum data.
I've never used twisted. I presume its xmlrpc lib handles the nil extension (otherwise you would almost certainly have seen lots of errors). I wonder if it can handle an i8 tag, which kojihub will now emit for large integers in some contexts.
Koji's faults can have many different fault codes, though yes 1000 is the most likely. The full list in near the beginning of koji/__init__.py.
On Tue, Dec 12, 2017 at 5:28 PM, Ken Dreyer ktdreyer@ktdreyer.com wrote:
Hi folks,
Recently I wrote a Twisted-based chat bot that can interact with a Koji hub.
To do this, I wrote a "txkoji" client library: https://github.com/ktdreyer/txkoji
(the bot code is separate, this is just the koji client library)
txkoji allows me to interact with a Koji Hub in an event-driven way, without blocking the reactor for my program. This works well for client programs that must be responsive to input from several sources (like IRC and a message bus and many other inputs). I return the XML-RPC responses as Munch (dict-like) objects because I ran across Munch in a couple other Fedora infra projects and I like the syntax.
If Koji gets a REST API, I could convert this to use treq instead of Twisted's XML-RPC client.
- Ken
koji-devel mailing list -- koji-devel@lists.fedorahosted.org To unsubscribe send an email to koji-devel-leave@lists.fedorahosted.org
On Tue, Dec 12, 2017 at 7:37 PM, Michael McLean mikem@redhat.com wrote:
Neat!
Note that an authenticated koji session will probably not play nice with async use because of the callnum data.
Thanks, I see the callnum code in Koji now. What is the purpose of the callnum? Is it a security feature, or just a way to maintain write operation order?
I've never used twisted. I presume its xmlrpc lib handles the nil extension (otherwise you would almost certainly have seen lots of errors). I wonder if it can handle an i8 tag, which kojihub will now emit for large integers in some contexts.
Koji's faults can have many different fault codes, though yes 1000 is the most likely. The full list in near the beginning of koji/__init__.py.
Thanks Mike for all these pointers. I checked Twisted and it uses xmlrpclib.loads() to parse the responses. I tested <nil/> and it translates fine to None, and the Koji git log indicates Python's xmlrpclib will read <i8> too. I can't find a live Koji instance where we've overflowed the 32-bit int yet to verify, though!
- Ken
On Wed, Dec 13, 2017 at 11:30 AM, Ken Dreyer ktdreyer@ktdreyer.com wrote: I can't find a live Koji instance where
we've overflowed the 32-bit int yet to verify, though!
Now I'm confused. https://pagure.io/koji/issue/464 mentions the webkitgtk4-debuginfo RPMs, but when I check one with getRPM(10230453), the size is a string, not i8:
$ cat xmlrpc <?xml version="1.0"?> <methodCall> <methodName>getRPM</methodName> <params> <param> <value> <int>10230453</int> </value> </param> </params> </methodCall>
$ curl -d @xmlrpc https://koji.fedoraproject.org/kojihub | tidy -q -xml -i - ... <member> <name>size</name> <value><string>2753783486</string></value> </member> ...
- Ken
So, over the years we addressed the int overflow problem in different ways. Initially, we had some targeted code that would convert integers over the limit into strings. The was done only in a few places that were known to return large ints.
The most recent case of overflow was a bit messier, and we went with a more systematic approach. We also discovered the i8 tag, which was had not previously known about (and may not have existed when we first implemented the earlier solution).
When we started using the i8 tags, it was late in our release cycle, and we didn't want to break existing code that might rely on those values being strings. So, we opted to leave those older parts alone for the time being.
If you want a quick test, the echo call will give you whatever data you pass in.
$ koji --noauth call echo 9223372036854775807 [9223372036854775807]
On Wed, Dec 13, 2017 at 1:58 PM, Ken Dreyer ktdreyer@ktdreyer.com wrote:
On Wed, Dec 13, 2017 at 11:30 AM, Ken Dreyer ktdreyer@ktdreyer.com wrote: I can't find a live Koji instance where
we've overflowed the 32-bit int yet to verify, though!
Now I'm confused. https://pagure.io/koji/issue/464 mentions the webkitgtk4-debuginfo RPMs, but when I check one with getRPM(10230453), the size is a string, not i8:
$ cat xmlrpc
<?xml version="1.0"?>
<methodCall> <methodName>getRPM</methodName> <params> <param> <value> <int>10230453</int> </value> </param> </params> </methodCall>
$ curl -d @xmlrpc https://koji.fedoraproject.org/kojihub | tidy -q -xml -i - ...
<member> <name>size</name> <value><string>2753783486</string></value> </member> ...
- Ken
koji-devel mailing list -- koji-devel@lists.fedorahosted.org To unsubscribe send an email to koji-devel-leave@lists.fedorahosted.org
On Wed, Dec 13, 2017 at 12:55 PM, Michael McLean mikem@redhat.com wrote:
If you want a quick test, the echo call will give you whatever data you pass in.
$ koji --noauth call echo 9223372036854775807 [9223372036854775807]
Thanks Mike!
That echo call definitely breaks my client (OverflowError, "int exceeds XML-RPC limits" from xmlrpclib). I added custom Marshaller overrides to deal with this.
- Ken
Odds are it's the encoding part that's breaking. You might also check that you can encode None.
On Wed, Dec 13, 2017 at 4:55 PM, Ken Dreyer ktdreyer@ktdreyer.com wrote:
On Wed, Dec 13, 2017 at 12:55 PM, Michael McLean mikem@redhat.com wrote:
If you want a quick test, the echo call will give you whatever data you
pass
in.
$ koji --noauth call echo 9223372036854775807 [9223372036854775807]
Thanks Mike!
That echo call definitely breaks my client (OverflowError, "int exceeds XML-RPC limits" from xmlrpclib). I added custom Marshaller overrides to deal with this.
- Ken
koji-devel mailing list -- koji-devel@lists.fedorahosted.org To unsubscribe send an email to koji-devel-leave@lists.fedorahosted.org
On Wed, Dec 13, 2017 at 3:23 PM, Michael McLean mikem@redhat.com wrote:
Odds are it's the encoding part that's breaking. You might also check that you can encode None.
To clarify: with my latest commit to txkoji to add the custom Marshaller, that echo works now!
It gave me some ideas for optimizing koji/xmlrpcplus.py , https://pagure.io/koji/pull-request/751
- Ken
koji-devel@lists.stg.fedorahosted.org
-
Ken Dreyer -
Michael McLean