I've done two builds for rawhide this morning.
On the first the armv7hl and ppc64le builds failed because the source tar file could not be unpacked.
On the second the aarch64 build failed because the source tar file could not be unpacked.
All the other arches built successfully.
----- Original Message -----
I've done two builds for rawhide this morning.
On the first the armv7hl and ppc64le builds failed because the source tar file could not be unpacked.
On the second the aarch64 build failed because the source tar file could not be unpacked.
All the other arches built successfully.
--
Kaleb _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org
I am encountering the same problem even on i686 (when trying to scratch build graphviz [1, 2]), so there is probably something wrong with the builders
thanks & regards
Jaroslav
[1] https://koji.fedoraproject.org/koji/taskinfo?taskID=17302165 [2] https://koji.fedoraproject.org/koji/taskinfo?taskID=17302557
On Mon, 16 Jan 2017 09:38:25 -0500 (EST) Jaroslav Skarvada jskarvad@redhat.com wrote:
----- Original Message -----
I've done two builds for rawhide this morning.
On the first the armv7hl and ppc64le builds failed because the source tar file could not be unpacked.
On the second the aarch64 build failed because the source tar file could not be unpacked.
All the other arches built successfully.
--
Kaleb _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org
I am encountering the same problem even on i686 (when trying to scratch build graphviz [1, 2]), so there is probably something wrong with the builders
Yes, there's been ongoing issues since last week:
* This issue (which seems new in the last few days) where srpm isn't unpacking correctly. https://pagure.io/fedora-infrastructure/issue/5694
* Sometimes downloads of packages for the mock chroot are failing. Even though there's nothing at all wrong with the squid cache (and In fact I added a second one this weekend). https://pagure.io/fedora-infrastructure/issue/5689
* buildvm's are unstable and sometimes reboot or have odd kernel blowups. (The 4.9.x kernel bug is https://bugzilla.redhat.com/show_bug.cgi?id=1413314 but it might not be a kernel bug at all, since 4.8.x kernels are doing the same now as well.
I guess my monday is all mapped out for me. ;)
will keep the above bugs posted and report back here when I figure anything out. ;(
kevin
On 01/16/2017 05:16 PM, Kevin Fenzi wrote:
I guess my monday is all mapped out for me. ;)
will keep the above bugs posted and report back here when I figure anything out. ;(
And I have another case for you:
Buildroots fail to find/download packages:
From https://koji.fedoraproject.org/koji/taskinfo?taskID=17311012 rsp. https://kojipkgs.fedoraproject.org//work/tasks/1021/17311021/root.log ... DEBUG util.py:435: Error: Error downloading packages: DEBUG util.py:435: Cannot download toplink/packages/freeglut/3.0.0/3.fc24/x86_64/freeglut-3.0.0-3.fc24.x86_64.rpm: All mirrors were tried ...
Ralf
Python 3 started failing on i686, x86_64 and arm only.
https://koji.fedoraproject.org/koji/taskinfo?taskID=17313924
The failures come from the test_socket.py [0], test_aead_aes_gcm [1] and more specifically this line [2] with the message: OSError: [Errno 22] Invalid argument
This test is checking Python's interface for the kernel crypto API (added in 3.6 [3]).
[0] https://hg.python.org/cpython/file/3.6/Lib/test/test_socket.py#l5473 [1] https://hg.python.org/cpython/file/3.6/Lib/test/test_socket.py#l5472 [2] https://hg.python.org/cpython/file/3.6/Lib/test/test_socket.py#l5497 [3] http://bugs.python.org/issue27744
Charalampos Stratakis Associate Software Engineer Python Maintenance Team, Red Hat
----- Original Message ----- From: "Kevin Fenzi" kevin@scrye.com To: devel@lists.fedoraproject.org Sent: Monday, January 16, 2017 5:16:40 PM Subject: Re: Is there something wrong with the Koji builders?
On Mon, 16 Jan 2017 09:38:25 -0500 (EST) Jaroslav Skarvada jskarvad@redhat.com wrote:
----- Original Message -----
I've done two builds for rawhide this morning.
On the first the armv7hl and ppc64le builds failed because the source tar file could not be unpacked.
On the second the aarch64 build failed because the source tar file could not be unpacked.
All the other arches built successfully.
--
Kaleb _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org
I am encountering the same problem even on i686 (when trying to scratch build graphviz [1, 2]), so there is probably something wrong with the builders
Yes, there's been ongoing issues since last week:
* This issue (which seems new in the last few days) where srpm isn't unpacking correctly. https://pagure.io/fedora-infrastructure/issue/5694
* Sometimes downloads of packages for the mock chroot are failing. Even though there's nothing at all wrong with the squid cache (and In fact I added a second one this weekend). https://pagure.io/fedora-infrastructure/issue/5689
* buildvm's are unstable and sometimes reboot or have odd kernel blowups. (The 4.9.x kernel bug is https://bugzilla.redhat.com/show_bug.cgi?id=1413314 but it might not be a kernel bug at all, since 4.8.x kernels are doing the same now as well.
I guess my monday is all mapped out for me. ;)
will keep the above bugs posted and report back here when I figure anything out. ;(
kevin
_______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org
On Wed, 18 Jan 2017 07:22:49 -0500 (EST) Charalampos Stratakis cstratak@redhat.com wrote:
Python 3 started failing on i686, x86_64 and arm only.
https://koji.fedoraproject.org/koji/taskinfo?taskID=17313924
The failures come from the test_socket.py [0], test_aead_aes_gcm [1] and more specifically this line [2] with the message: OSError: [Errno 22] Invalid argument
This test is checking Python's interface for the kernel crypto API (added in 3.6 [3]).
[0] https://hg.python.org/cpython/file/3.6/Lib/test/test_socket.py#l5473 [1] https://hg.python.org/cpython/file/3.6/Lib/test/test_socket.py#l5472 [2] https://hg.python.org/cpython/file/3.6/Lib/test/test_socket.py#l5497 [3] http://bugs.python.org/issue27744
Thats nothing to do with the buildsystem. Or at least if it is, it's not any of the known issues I was working on.
As far as I know now everything should be back to working/normal.
There were 3 issues:
1. The buildvm's were updated and started crashing under load. They would spew oopses and finally reboot. This would result in builds on them getting restarted or dying in strange ways. This seems to be something particular to their hardware/setup that changed in later 4.8.x and early 4.9.x kernels. We are currently running 4.10rc4 on them and it's been stable (knock on wood).
2. Sometimes, very rarely dnf would fail downloading packages for the build root. We have worked around this by passing dnf via koji several mirrors where it can download from, so if it fails on one it should fall back to the next and so on. https://pagure.io/fedora-infrastructure/issue/5689
3. In some configurations (where there was more than I host to download from) koji would fail to download the src.rpm correctly and error out. We are working around this now by just pointing koji at one place for these downloads for now until we can get things fixed. https://pagure.io/fedora-infrastructure/issue/5694
Sorry for all the instability the last few days.
If you see any further issues like the above, let me know.
kevin
On 01/18/2017 07:55 PM, Kevin Fenzi wrote:
If you see any further issues like the above, let me know.
https://koji.fedoraproject.org/koji/taskinfo?taskID=17327268 seems to hang on arm7vhl and i386.
The x86_64 built finished ca. 45 minutes after the built was started.
However, I can not spot any activity in the i386 logs https://koji.fedoraproject.org/koji/taskinfo?taskID=17327272 nor the arm's logs https://koji.fedoraproject.org/koji/taskinfo?taskID=17327270
ATM, it's > 10 hours since these builts have been started, with me expecting them to take approx. 4 hours (The time, fc24 and fc26 builds took), but ...
Ralf
After some discussions with Christian Heimes, it seems this has something to do with the latest kernel (4.9), as the kernel's api used in this test changed [0][1]. This will need to be addressed in python, but it is weird that it happens only on specific architectures.
[0] https://github.com/smuellerDD/libkcapi/commit/be242c387b7030cbccae2c183107ef... [1] http://bugs.python.org/issue29324
Anyway I don't think any more attention is required by infra for this specific case.
Regards,
Charalampos Stratakis Associate Software Engineer Python Maintenance Team, Red Hat
----- Original Message ----- From: "Kevin Fenzi" kevin@scrye.com To: devel@lists.fedoraproject.org Sent: Wednesday, January 18, 2017 7:55:38 PM Subject: Re: Is there something wrong with the Koji builders?
On Wed, 18 Jan 2017 07:22:49 -0500 (EST) Charalampos Stratakis cstratak@redhat.com wrote:
Python 3 started failing on i686, x86_64 and arm only.
https://koji.fedoraproject.org/koji/taskinfo?taskID=17313924
The failures come from the test_socket.py [0], test_aead_aes_gcm [1] and more specifically this line [2] with the message: OSError: [Errno 22] Invalid argument
This test is checking Python's interface for the kernel crypto API (added in 3.6 [3]).
[0] https://hg.python.org/cpython/file/3.6/Lib/test/test_socket.py#l5473 [1] https://hg.python.org/cpython/file/3.6/Lib/test/test_socket.py#l5472 [2] https://hg.python.org/cpython/file/3.6/Lib/test/test_socket.py#l5497 [3] http://bugs.python.org/issue27744
Thats nothing to do with the buildsystem. Or at least if it is, it's not any of the known issues I was working on.
As far as I know now everything should be back to working/normal.
There were 3 issues:
1. The buildvm's were updated and started crashing under load. They would spew oopses and finally reboot. This would result in builds on them getting restarted or dying in strange ways. This seems to be something particular to their hardware/setup that changed in later 4.8.x and early 4.9.x kernels. We are currently running 4.10rc4 on them and it's been stable (knock on wood).
2. Sometimes, very rarely dnf would fail downloading packages for the build root. We have worked around this by passing dnf via koji several mirrors where it can download from, so if it fails on one it should fall back to the next and so on. https://pagure.io/fedora-infrastructure/issue/5689
3. In some configurations (where there was more than I host to download from) koji would fail to download the src.rpm correctly and error out. We are working around this now by just pointing koji at one place for these downloads for now until we can get things fixed. https://pagure.io/fedora-infrastructure/issue/5694
Sorry for all the instability the last few days.
If you see any further issues like the above, let me know.
kevin
_______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org
On Wed, Jan 18, 2017 at 11:55:38AM -0700, Kevin Fenzi wrote:
On Wed, 18 Jan 2017 07:22:49 -0500 (EST) Charalampos Stratakis cstratak@redhat.com wrote:
Python 3 started failing on i686, x86_64 and arm only.
https://koji.fedoraproject.org/koji/taskinfo?taskID=17313924
The failures come from the test_socket.py [0], test_aead_aes_gcm [1] and more specifically this line [2] with the message: OSError: [Errno 22] Invalid argument
This test is checking Python's interface for the kernel crypto API (added in 3.6 [3]).
[0] https://hg.python.org/cpython/file/3.6/Lib/test/test_socket.py#l5473 [1] https://hg.python.org/cpython/file/3.6/Lib/test/test_socket.py#l5472 [2] https://hg.python.org/cpython/file/3.6/Lib/test/test_socket.py#l5497 [3] http://bugs.python.org/issue27744
Thats nothing to do with the buildsystem. Or at least if it is, it's not any of the known issues I was working on.
As far as I know now everything should be back to working/normal.
There were 3 issues:
- The buildvm's were updated and started crashing under load. They
would spew oopses and finally reboot. This would result in builds on them getting restarted or dying in strange ways. This seems to be something particular to their hardware/setup that changed in later 4.8.x and early 4.9.x kernels. We are currently running 4.10rc4 on them and it's been stable (knock on wood).
- Sometimes, very rarely dnf would fail downloading packages for the
build root. We have worked around this by passing dnf via koji several mirrors where it can download from, so if it fails on one it should fall back to the next and so on. https://pagure.io/fedora-infrastructure/issue/5689
- In some configurations (where there was more than I host to download
from) koji would fail to download the src.rpm correctly and error out. We are working around this now by just pointing koji at one place for these downloads for now until we can get things fixed. https://pagure.io/fedora-infrastructure/issue/5694
Sorry for all the instability the last few days.
If you see any further issues like the above, let me know.
I'm still seeing quite alot of problems today - randomly the arm7 or ppc task will just sit there in "open" status doing nothing for hours. If I cancel & rerun the build it works fine, presumably because it hit a different build instance.
Now I've just seen some builds which succeeded in the build phase get marked as failed because mock throws an exception handling the results
https://kojipkgs.fedoraproject.org//work/tasks/1011/17331011/mock_output.log
Finish: run ERROR: list index out of range Traceback (most recent call last): File "/usr/libexec/mock/mock", line 885, in <module> raise File "/usr/lib/python3.5/site-packages/mockbuild/trace_decorator.py", line 89, in trace result = func(*args, **kw) File "/usr/libexec/mock/mock", line 704, in main File "/usr/lib/python3.5/site-packages/mockbuild/trace_decorator.py", line 89, in trace result = func(*args, **kw) File "/usr/lib/python3.5/site-packages/mockbuild/buildroot.py", line 545, in finalize self._unlock_buildroot() File "/usr/lib/python3.5/site-packages/mockbuild/trace_decorator.py", line 89, in trace result = func(*args, **kw) File "/usr/lib/python3.5/site-packages/mockbuild/mounts.py", line 162, in umountall File "/usr/lib/python3.5/site-packages/mockbuild/trace_decorator.py", line 81, in trace frame = inspect.getouterframes(inspect.currentframe())[1][0] File "/usr/lib/python3.5/inspect.py", line 1441, in getouterframes frameinfo = (frame,) + getframeinfo(frame, context) File "/usr/lib/python3.5/inspect.py", line 1414, in getframeinfo lines, lnum = findsource(frame) File "/usr/lib/python3.5/inspect.py", line 804, in findsource if pat.match(lines[lnum]): break IndexError: list index out of range
Regards, Daniel
On 01/19/2017 06:37 PM, Daniel P. Berrange wrote:
Now I've just seen some builds which succeeded in the build phase get marked as failed because mock throws an exception handling the results
https://kojipkgs.fedoraproject.org//work/tasks/1011/17331011/mock_output.log
Finish: run ERROR: list index out of range Traceback (most recent call last): File "/usr/libexec/mock/mock", line 885, in <module> raise File "/usr/lib/python3.5/site-packages/mockbuild/trace_decorator.py", line 89, in trace result = func(*args, **kw) File "/usr/libexec/mock/mock", line 704, in main File "/usr/lib/python3.5/site-packages/mockbuild/trace_decorator.py", line 89, in trace result = func(*args, **kw) File "/usr/lib/python3.5/site-packages/mockbuild/buildroot.py", line 545, in finalize self._unlock_buildroot() File "/usr/lib/python3.5/site-packages/mockbuild/trace_decorator.py", line 89, in trace result = func(*args, **kw) File "/usr/lib/python3.5/site-packages/mockbuild/mounts.py", line 162, in umountall File "/usr/lib/python3.5/site-packages/mockbuild/trace_decorator.py", line 81, in trace frame = inspect.getouterframes(inspect.currentframe())[1][0] File "/usr/lib/python3.5/inspect.py", line 1441, in getouterframes frameinfo = (frame,) + getframeinfo(frame, context) File "/usr/lib/python3.5/inspect.py", line 1414, in getframeinfo lines, lnum = findsource(frame) File "/usr/lib/python3.5/inspect.py", line 804, in findsource if pat.match(lines[lnum]): break IndexError: list index out of range
This is what I meanwhile am seeing in before mentioned "hanging" builds, as well:
https://kojipkgs.fedoraproject.org//work/tasks/7272/17327272/mock_output.log
Ralf
On Thu, 19 Jan 2017 17:37:23 +0000 "Daniel P. Berrange" berrange@redhat.com wrote:
I'm still seeing quite alot of problems today - randomly the arm7 or ppc task will just sit there in "open" status doing nothing for hours. If I cancel & rerun the build it works fine, presumably because it hit a different build instance.
Well, not sure thats the case yet. This is a NEW AND EXCITING failure I am looking at this morning.
I freed all the tasks that were stuck this way and am investigating.
Now I've just seen some builds which succeeded in the build phase get marked as failed because mock throws an exception handling the results
https://kojipkgs.fedoraproject.org//work/tasks/1011/17331011/mock_output.log
Please post links to the top level task. (in this case: https://koji.fedoraproject.org/koji/taskinfo?taskID=17331011 )
This was my fault. I was downgrading mock on the builders in an attempt to see if it was causing the above new and exciting bug, but unfortunately it caused in progress builds to explode. ;(
So, resubmit and it should work (unless it gets stuck not processing in which case I will come along and free it up).
kevin
On 01/19/2017 06:55 PM, Kevin Fenzi wrote:
Please post links to the top level task.
My case:
https://koji.fedoraproject.org/koji/taskinfo?taskID=17327268
https://kojipkgs.fedoraproject.org//work/tasks/7272/17327272/mock_output.log
Approx. at the same time the i686 raised the error above, the arm build-job seems to have started.
Ralf
ok. After a bunch of debugging and digging today it sure looks like the underlying problem was squid operating in smp mode.
We switched the squid servers back to non smp mode around 00:00 and I've not seen any of these failures since then.
I sure hope everything is back to normal now.
kevin
On 01/19/17 at 10:24pm, Kevin Fenzi wrote:
ok. After a bunch of debugging and digging today it sure looks like the underlying problem was squid operating in smp mode.
We switched the squid servers back to non smp mode around 00:00 and I've not seen any of these failures since then.
I sure hope everything is back to normal now.
kevin
A scratch build failed for me: https://koji.fedoraproject.org/koji/taskinfo?taskID=17388293
I noticed below error in ppc64 build log: DEBUG util.py:435: Error: package pkgconf-pkg-config-1.2.0-1.fc26.ppc64 conflicts with pkgconfig < 1:0.29.1-2 provided by pkgconfig-1:0.29.1-1.fc25.ppc64
Thanks Dave
On Mon, Jan 23, 2017 at 8:30 AM, Dave Young dyoung@redhat.com wrote:
On 01/19/17 at 10:24pm, Kevin Fenzi wrote:
ok. After a bunch of debugging and digging today it sure looks like the underlying problem was squid operating in smp mode.
We switched the squid servers back to non smp mode around 00:00 and I've not seen any of these failures since then.
I sure hope everything is back to normal now.
kevin
A scratch build failed for me: https://koji.fedoraproject.org/koji/taskinfo?taskID=17388293
I noticed below error in ppc64 build log: DEBUG util.py:435: Error: package pkgconf-pkg-config-1.2.0-1.fc26.ppc64 conflicts with pkgconfig < 1:0.29.1-2 provided by pkgconfig-1:0.29.1-1.fc25.ppc64
Known issue, working on it now.
A scratch build failed for me, https://koji.fedoraproject.org/koji/taskinfo?taskID=17934589
This was only a couple of minutes ago.
On Mon, Jan 23, 2017 at 5:15 PM, Peter Robinson pbrobinson@gmail.com wrote:
On Mon, Jan 23, 2017 at 8:30 AM, Dave Young dyoung@redhat.com wrote:
On 01/19/17 at 10:24pm, Kevin Fenzi wrote:
ok. After a bunch of debugging and digging today it sure looks like the underlying problem was squid operating in smp mode.
We switched the squid servers back to non smp mode around 00:00 and I've not seen any of these failures since then.
I sure hope everything is back to normal now.
kevin
A scratch build failed for me: https://koji.fedoraproject.org/koji/taskinfo?taskID=17388293
I noticed below error in ppc64 build log: DEBUG util.py:435: Error: package pkgconf-pkg-config-1.2.0-1.fc26.ppc64 conflicts with pkgconfig < 1:0.29.1-2 provided by pkgconfig-1:0.29.1-1.fc25.ppc64
Known issue, working on it now. _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org
On Sat, 18 Feb 2017 22:30:08 +0800 Kalpa Welivitigoda callkalpa@gmail.com wrote:
A scratch build failed for me, https://koji.fedoraproject.org/koji/taskinfo?taskID=17934589
This was only a couple of minutes ago.
But your build failure wasn't anything to do with this threads build issues:
+ /usr/lib/rpm/check-buildroot /builddir/build/BUILDROOT/sugar-fractionbounce-25-1.fc26.noarch/usr/share/applications/org.sugarlabs.FractionBounceActivity.activity.desktop:Icon = /builddir/build/BUILDROOT/sugar-fractionbounce-25-1.fc26.noarch/usr/share/sugar/activities/FractionBounce.activity/activity/activity-fractionbounce.svg /builddir/build/BUILDROOT/sugar-fractionbounce-25-1.fc26.noarch/usr/share/applications/org.sugarlabs.FractionBounceActivity.activity.desktop:Path = /builddir/build/BUILDROOT/sugar-fractionbounce-25-1.fc26.noarch/usr/share/sugar/activities/FractionBounce.activity Found '/builddir/build/BUILDROOT/sugar-fractionbounce-25-1.fc26.noarch' in installed files; aborting RPM build errors: error: Bad exit status from /var/tmp/rpm-tmp.8fezSB (%install) Bad exit status from /var/tmp/rpm-tmp.8fezSB (%install)
Somewhere you have files that have the buildroot in them instead of the end place the files will be in the filesystem.
kevin
On Sat, Feb 18, 2017 at 11:36 PM, Kevin Fenzi kevin@scrye.com wrote:
On Sat, 18 Feb 2017 22:30:08 +0800 Kalpa Welivitigoda callkalpa@gmail.com wrote:
A scratch build failed for me, https://koji.fedoraproject.org/koji/taskinfo?taskID=17934589
This was only a couple of minutes ago.
But your build failure wasn't anything to do with this threads build issues:
- /usr/lib/rpm/check-buildroot
/builddir/build/BUILDROOT/sugar-fractionbounce-25-1.fc26.noarch/usr/share/ applications/org.sugarlabs.FractionBounceActivity.activity.desktop:Icon = /builddir/build/BUILDROOT/sugar-fractionbounce-25-1. fc26.noarch/usr/share/sugar/activities/FractionBounce. activity/activity/activity-fractionbounce.svg /builddir/build/BUILDROOT/sugar-fractionbounce-25-1.fc26.noarch/usr/share/ applications/org.sugarlabs.FractionBounceActivity.activity.desktop:Path = /builddir/build/BUILDROOT/sugar-fractionbounce-25-1. fc26.noarch/usr/share/sugar/activities/FractionBounce.activity Found '/builddir/build/BUILDROOT/sugar-fractionbounce-25-1.fc26.noarch' in installed files; aborting RPM build errors: error: Bad exit status from /var/tmp/rpm-tmp.8fezSB (%install) Bad exit status from /var/tmp/rpm-tmp.8fezSB (%install)
Extremely sorry for the noise.
Somewhere you have files that have the buildroot in them instead of the end place the files will be in the filesystem.
Exactly, that's the reason. As in the error message itself, org.sugarlabs. FractionBounceActivity.activity.desktop contains BUILDROOT which fails the check. The generation of *.desktop file seems to be newly introduced with sugar-toolkit-gtk3 and it make use of the prefix passed in the install phase, the prefix has the BUILDROOT.
Anyway that's a separate discussion and thank you for pointing out the issue here.
kevin
devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org
On 01/16/2017 09:16 AM, Kevin Fenzi wrote:
Yes, there's been ongoing issues since last week:
- This issue (which seems new in the last few days) where srpm isn't unpacking correctly. https://pagure.io/fedora-infrastructure/issue/5694
Still seeing this: https://koji.fedoraproject.org/koji/taskinfo?taskID=17332860
On Fri, 20 Jan 2017 10:22:01 -0700 Orion Poplawski orion@cora.nwra.com wrote:
On 01/16/2017 09:16 AM, Kevin Fenzi wrote:
Yes, there's been ongoing issues since last week:
- This issue (which seems new in the last few days) where srpm isn't unpacking correctly. https://pagure.io/fedora-infrastructure/issue/5694
Still seeing this: https://koji.fedoraproject.org/koji/taskinfo?taskID=17332860
You mean you were seeing it back before I fixed it. ;)
Please retry any builds that were before 2017-01-20 00:00 UTC.
Thats about when the last fix went out, so anything before then wasn't subject to that.
kevin
devel@lists.stg.fedoraproject.org
-
Charalampos Stratakis -
Daniel P. Berrange -
Dave Young -
Jaroslav Skarvada -
Kaleb S. KEITHLEY -
Kalpa Welivitigoda -
Kevin Fenzi -
Orion Poplawski -
Peter Robinson -
Ralf Corsepius