Hi all, Sorry to bother, but I couldn't figure this problem out. If anyone could provide help, that's very appreciated! Tuan Hoang and I are trying to build cores image on s390x, we did some changes on coreos-assembler, mantle, fedora-coreos-config and some others, the first steps of 'docker build ...', 'cosa init ...', 'cosa fetch' succeeded, but we failed on 'cosa build' and the error is: [ 93.278169] anaconda[1637]: Receiving objects: 99% (30655/30656) 520.4 MB^M^M [ 93.278204] anaconda[1637]: Preparing deployment of fedora/29/s390x/coreos^M^M [ 93.278242] anaconda[1637]: Deployment starting: fedora/29/s390x/coreos^M^M [ 93.278277] anaconda[1637]: Deployment complete: fedora/29/s390x/coreos^M^M [ 94.099939] anaconda[1637]: .^M^M [ 94.100135] anaconda[1637]: Configuring storage^M^M [ 275.623279] anaconda[1637]: .--- Logging error ---^M^M [ 275.623530] anaconda[1637]: Traceback (most recent call last):^M^M [ 275.623651] anaconda[1637]: File "/usr/lib64/python3.7/logging/init.py", line 985, in emit^M^M [ 275.623770] anaconda[1637]: stream.write(msg)^M^M [ 275.623881] anaconda[1637]: File "/usr/lib64/python3.7/site-packages/pyanaconda/anaconda_logging.py", line 120, in write^M^M [ 275.623994] anaconda[1637]: stream.write(*args, **kwargs)^M^M [ 275.624105] anaconda[1637]: ValueError: underlying buffer has been detached^M^M [ 275.624214] anaconda[1637]: Call stack:^M^M [ 275.624325] anaconda[1637]: File "/usr/sbin/anaconda", line 642, in ^M^M [ 275.624436] anaconda[1637]: display.setup_display(anaconda, opts, addon_paths=addon_paths)^M^M [ 275.624548] anaconda[1637]: File "/usr/lib64/python3.7/site-packages/pyanaconda/display.py", line 273, in setup_display^M^M [ 275.624660] anaconda[1637]: stdout_log.warning(error_message)^M^M [ 275.624767] anaconda[1637]: File "/usr/lib64/python3.7/logging/init.py", line 1344, in warning^M^M [ 275.624879] anaconda[1637]: self._log(WARNING, msg, args, **kwargs)^M^M [ 275.624988] anaconda[1637]: File "/usr/lib64/python3.7/logging/init.py", line 1468, in _log^M^M [ 275.625101] anaconda[1637]: self.handle(record)^M^M [ 275.625216] anaconda[1637]: File "/usr/lib64/python3.7/logging/init.py", line 1478, in handle^M^M [ 275.625320] anaconda[1637]: self.callHandlers(record)^M^M [ 275.625431] anaconda[1637]: File "/usr/lib64/python3.7/logging/init.py", line 1540, in callHandlers^M^M [ 275.625542] anaconda[1637]: hdlr.handle(record)^M^M [ 275.625645] anaconda[1637]: File "/usr/lib64/python3.7/site-packages/pyanaconda/anaconda_logging.py", line 101, in handle^M^M [ 275.625747] anaconda[1637]: self.emit(record) # pylint: disable=no-member^M^M [ 275.625852] anaconda[1637]: Message: 'Not asking for VNC because of an automated install'^M^M [ 275.625954] anaconda[1637]: Arguments: ()^M^M [ 275.626054] anaconda[1637]: --- Logging error ---^M^M [ 275.626166] anaconda[1637]: Traceback (most recent call last):^M^M [ 275.626282] anaconda[1637]: File "/usr/lib64/python3.7/logging/init.py", line 985, in emit^M^M [ 275.626385] anaconda[1637]: stream.write(msg)^M^M [ 275.626487] anaconda[1637]: File "/usr/lib64/python3.7/site-packages/pyanaconda/anaconda_logging.py", line 120, in write^M^M [ 275.626598] anaconda[1637]: stream.write(*args, **kwargs)^M^M [ 275.626712] anaconda[1637]: ValueError: underlying buffer has been detached^M^M [ 275.626811] anaconda[1637]: Call stack:^M^M [ 275.626926] anaconda[1637]: File "/usr/sbin/anaconda", line 642, in ^M^M [ 275.627040] anaconda[1637]: display.setup_display(anaconda, opts, addon_paths=addon_paths)^M^M [ 275.627144] anaconda[1637]: File "/usr/lib64/python3.7/site-packages/pyanaconda/display.py", line 273, in setup_display^M^M [ 275.627205] anaconda[1637]: stdout_log.warning(error_message)^M^M [ 275.627244] anaconda[1637]: File "/usr/lib64/python3.7/logging/init.py", line 1344, in warning^M^M [ 275.627283] anaconda[1637]: self._log(WARNING, msg, args, **kwargs)^M^M [ 275.627392] anaconda[1637]: File "/usr/lib64/python3.7/logging/init.py", line 1468, in _log^M^M [ 275.627489] anaconda[1637]: self.handle(record)^M^M [ 275.627593] anaconda[1637]: File "/usr/lib64/python3.7/logging/init.py", line 1478, in handle^M^M [ 275.627692] anaconda[1637]: self.callHandlers(record)^M^M [ 275.627801] anaconda[1637]: File "/usr/lib64/python3.7/logging/init.py", line 1540, in callHandlers^M^M [ 275.627916] anaconda[1637]: hdlr.handle(record)^M^M [ 275.628016] anaconda[1637]: File "/usr/lib64/python3.7/site-packages/pyanaconda/anaconda_logging.py", line 101, in handle^M^M [ 275.628122] anaconda[1637]: self.emit(record) # pylint: disable=no-member^M^M [ 275.628220] anaconda[1637]: Message: 'Not asking for VNC because text mode was explicitly asked for in kickstart'^M^M [ 275.628336] anaconda[1637]: Arguments: ()^M^M [ 275.628442] anaconda[1637]: Installing boot loader^M^M [ 275.628541] anaconda[1637]: Running in cmdline mode, no interactive debugging allowed.^M^M [ 275.628640] anaconda[1637]: The exact error message is:^M^M [ 275.628740] anaconda[1637]: 'NoneType' object has no attribute 'short_label'.^M^M [ 275.628839] anaconda[1637]: The installer will now terminate.^M^M [^[[0;32m OK ^[[0m] Stopped coreos-virtinstall-logs.service.^M^M^M
However, Tuan succeeded to run 'cosa build' last month, so we doubt there might be something changed on upstream which broke this, or we need any adoption for s390x? or is that because of python 3.7 incompatible for s390x in this case? Any tip is welcome. thanks a lot!
Looks to me the installer failed to install the bootloader, based on the following messages
... [ 275.628442] anaconda[1637]: Installing boot loader [ 275.628541] anaconda[1637]: Running in cmdline mode, no interactive debugging allowed. [ 275.628640] anaconda[1637]: The exact error message is: [ 275.628740] anaconda[1637]: 'NoneType' object has no attribute 'short_label'. [ 275.628839] anaconda[1637]: The installer will now terminate.
Would be useful to see the kickstart used or the full log.
Dan
----- Original Message -----
From: "Dan Horák" sharkcz@fedoraproject.org To: coreos@lists.fedoraproject.org Sent: Thursday, April 18, 2019 12:44:05 PM Subject: [CoreOS] Re: Help: 'cosa build' failed on s390x
Looks to me the installer failed to install the bootloader, based on the following messages
... [ 275.628442] anaconda[1637]: Installing boot loader [ 275.628541] anaconda[1637]: Running in cmdline mode, no interactive debugging allowed. [ 275.628640] anaconda[1637]: The exact error message is: [ 275.628740] anaconda[1637]: 'NoneType' object has no attribute 'short_label'. [ 275.628839] anaconda[1637]: The installer will now terminate.
Would be useful to see the kickstart used or the full log.
If I'm not mistaken the ks should be https://github.com/coreos/coreos-assembler/blob/master/src/image-base.ks and it seems that there has been some changes recently.
JC
Dan _______________________________________________ CoreOS mailing list -- coreos@lists.fedoraproject.org To unsubscribe send an email to coreos-leave@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/coreos@lists.fedoraproject.org
在 2019/4/18 18:50, Jakub Cajka 写道:
----- Original Message -----
From: "Dan Horák" sharkcz@fedoraproject.org To: coreos@lists.fedoraproject.org Sent: Thursday, April 18, 2019 12:44:05 PM Subject: [CoreOS] Re: Help: 'cosa build' failed on s390x
Looks to me the installer failed to install the bootloader, based on the following messages
... [ 275.628442] anaconda[1637]: Installing boot loader [ 275.628541] anaconda[1637]: Running in cmdline mode, no interactive debugging allowed. [ 275.628640] anaconda[1637]: The exact error message is: [ 275.628740] anaconda[1637]: 'NoneType' object has no attribute 'short_label'. [ 275.628839] anaconda[1637]: The installer will now terminate.
Would be useful to see the kickstart used or the full log.
If I'm not mistaken the ks should be https://github.com/coreos/coreos-assembler/blob/master/src/image-base.ks and it seems that there has been some changes recently.
JC
Dan and JC, thanks a lot for your tip! Please check the attachment build-th.log for full build log. Tuan changed 'clearpart --initlabel --all --disklabel=msdos' in image-base.ks because the default gpt even breaks s390x 'cosa fetch'. And I also tried to build fedora 30 of coreos-assembler which failed in ahead (not reaching the step as f29). Thanks!
Dan _______________________________________________ CoreOS mailing list -- coreos@lists.fedoraproject.org To unsubscribe send an email to coreos-leave@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/coreos@lists.fedoraproject.org
CoreOS mailing list -- coreos@lists.fedoraproject.org To unsubscribe send an email to coreos-leave@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/coreos@lists.fedoraproject.org
On 4/23/19 10:43 AM, QingFeng Hao wrote:
在 2019/4/18 18:50, Jakub Cajka 写道:
----- Original Message -----
From: "Dan Horák" sharkcz@fedoraproject.org To: coreos@lists.fedoraproject.org Sent: Thursday, April 18, 2019 12:44:05 PM Subject: [CoreOS] Re: Help: 'cosa build' failed on s390x
Looks to me the installer failed to install the bootloader, based on the following messages
... [ 275.628442] anaconda[1637]: Installing boot loader [ 275.628541] anaconda[1637]: Running in cmdline mode, no interactive debugging allowed. [ 275.628640] anaconda[1637]: The exact error message is: [ 275.628740] anaconda[1637]: 'NoneType' object has no attribute 'short_label'. [ 275.628839] anaconda[1637]: The installer will now terminate.
Would be useful to see the kickstart used or the full log.
If I'm not mistaken the ks should be https://github.com/coreos/coreos-assembler/blob/master/src/image-base.ks and it seems that there has been some changes recently.
JC
Dan and JC, thanks a lot for your tip! Please check the attachment build-th.log for full build log. Tuan changed 'clearpart --initlabel --all --disklabel=msdos' in image-base.ks because the default gpt even breaks s390x 'cosa fetch'. And I also tried to build fedora 30 of coreos-assembler which failed in ahead (not reaching the step as f29). Thanks!
I just start a fresh build and refresh my memory a little bit about my changes.
The 'msdos' change in image-base.ks file is simply wrong because after all we still need gpt partition scheme. This change simply allows `cosa build` to finish. I have not hit the python3 exception you are having but I do now. Maybe turning on anaconda debugging flags would provide more info. However, I think it would not make sense to do this because it might be obsolete when we have gpt support.
The important part I think is to make anaconda (specifically the block device library it uses [1]) to support gpt on s390x, which at first I thought it requires patching s390-tools/zipl but Dan suggested it's anaconda thing. The stack would be libblockdev -> blivet -> anaconda (Thanks Dan!).
Next steps could be: - Manually install any s390x Linux distro on a virtual/virtio disk using gpt partitioning scheme to make sure zipl supports gpt. - Patch anaconda/blivet/libblockdev to allow gpt on s390x (alongside msdos and dasd).
[1] https://github.com/storaged-project/blivet'
Tuan
On 4/23/19 11:38 PM, Tuan Hoang wrote:
On 4/23/19 10:43 AM, QingFeng Hao wrote:
在 2019/4/18 18:50, Jakub Cajka 写道:
----- Original Message -----
From: "Dan Horák" sharkcz@fedoraproject.org To: coreos@lists.fedoraproject.org Sent: Thursday, April 18, 2019 12:44:05 PM Subject: [CoreOS] Re: Help: 'cosa build' failed on s390x
Looks to me the installer failed to install the bootloader, based on the following messages
... [ 275.628442] anaconda[1637]: Installing boot loader [ 275.628541] anaconda[1637]: Running in cmdline mode, no interactive debugging allowed. [ 275.628640] anaconda[1637]: The exact error message is: [ 275.628740] anaconda[1637]: 'NoneType' object has no attribute 'short_label'. [ 275.628839] anaconda[1637]: The installer will now terminate.
Would be useful to see the kickstart used or the full log.
If I'm not mistaken the ks should be https://github.com/coreos/coreos-assembler/blob/master/src/image-base.ks and it seems that there has been some changes recently.
JC
Dan and JC, thanks a lot for your tip! Please check the attachment build-th.log for full build log. Tuan changed 'clearpart --initlabel --all --disklabel=msdos' in image-base.ks because the default gpt even breaks s390x 'cosa fetch'. And I also tried to build fedora 30 of coreos-assembler which failed in ahead (not reaching the step as f29). Thanks!
I just start a fresh build and refresh my memory a little bit about my changes.
The 'msdos' change in image-base.ks file is simply wrong because after all we still need gpt partition scheme. This change simply allows `cosa build` to finish. I have not hit the python3 exception you are having but I do now. Maybe turning on anaconda debugging flags would provide more info. However, I think it would not make sense to do this because it might be obsolete when we have gpt support.
The important part I think is to make anaconda (specifically the block device library it uses [1]) to support gpt on s390x, which at first I thought it requires patching s390-tools/zipl but Dan suggested it's anaconda thing. The stack would be libblockdev -> blivet -> anaconda (Thanks Dan!).
Next steps could be:
- Manually install any s390x Linux distro on a virtual/virtio disk using
gpt partitioning scheme to make sure zipl supports gpt.
- Patch anaconda/blivet/libblockdev to allow gpt on s390x (alongside
msdos and dasd).
[1] https://github.com/storaged-project/blivet'
Tuan
Very good news. I could install Alpine Linux s390x on virtual GPT-ed disk and boot just fine. Should be working with other Linux distros. So the next step is to look into anaconda code.
在 2019/4/24 9:46, Tuan Hoang 写道:
On 4/23/19 11:38 PM, Tuan Hoang wrote:
On 4/23/19 10:43 AM, QingFeng Hao wrote:
在 2019/4/18 18:50, Jakub Cajka 写道:
----- Original Message -----
From: "Dan Horák" sharkcz@fedoraproject.org To: coreos@lists.fedoraproject.org Sent: Thursday, April 18, 2019 12:44:05 PM Subject: [CoreOS] Re: Help: 'cosa build' failed on s390x
Looks to me the installer failed to install the bootloader, based on the following messages
... [ 275.628442] anaconda[1637]: Installing boot loader [ 275.628541] anaconda[1637]: Running in cmdline mode, no interactive debugging allowed. [ 275.628640] anaconda[1637]: The exact error message is: [ 275.628740] anaconda[1637]: 'NoneType' object has no attribute 'short_label'. [ 275.628839] anaconda[1637]: The installer will now terminate.
Would be useful to see the kickstart used or the full log.
If I'm not mistaken the ks should be https://github.com/coreos/coreos-assembler/blob/master/src/image-base.ks and it seems that there has been some changes recently.
JC
Dan and JC, thanks a lot for your tip! Please check the attachment build-th.log for full build log. Tuan changed 'clearpart --initlabel --all --disklabel=msdos' in image-base.ks because the default gpt even breaks s390x 'cosa fetch'. And I also tried to build fedora 30 of coreos-assembler which failed in ahead (not reaching the step as f29). Thanks!
I just start a fresh build and refresh my memory a little bit about my changes.
The 'msdos' change in image-base.ks file is simply wrong because after all we still need gpt partition scheme. This change simply allows `cosa build` to finish. I have not hit the python3 exception you are having but I do now. Maybe turning on anaconda debugging flags would provide more info. However, I think it would not make sense to do this because it might be obsolete when we have gpt support.
The important part I think is to make anaconda (specifically the block device library it uses [1]) to support gpt on s390x, which at first I thought it requires patching s390-tools/zipl but Dan suggested it's anaconda thing. The stack would be libblockdev -> blivet -> anaconda (Thanks Dan!).
Next steps could be:
- Manually install any s390x Linux distro on a virtual/virtio disk using
gpt partitioning scheme to make sure zipl supports gpt.
- Patch anaconda/blivet/libblockdev to allow gpt on s390x (alongside
msdos and dasd).
[1] https://github.com/storaged-project/blivet'
Tuan
Very good news. I could install Alpine Linux s390x on virtual GPT-ed disk and boot just fine. Should be working with other Linux distros. So the next step is to look into anaconda code.
Cool! Another problem is how to debug anaconda in the VM coreos-inst-12054-1554889268 launched by 'cosa build'. thanks
On Wed, 24 Apr 2019 14:50:22 +0800 QingFeng Hao haoqf@linux.vnet.ibm.com wrote:
在 2019/4/24 9:46, Tuan Hoang 写道:
On 4/23/19 11:38 PM, Tuan Hoang wrote:
On 4/23/19 10:43 AM, QingFeng Hao wrote:
在 2019/4/18 18:50, Jakub Cajka 写道:
----- Original Message -----
From: "Dan Horák" sharkcz@fedoraproject.org To: coreos@lists.fedoraproject.org Sent: Thursday, April 18, 2019 12:44:05 PM Subject: [CoreOS] Re: Help: 'cosa build' failed on s390x
Looks to me the installer failed to install the bootloader, based on the following messages
... [ 275.628442] anaconda[1637]: Installing boot loader [ 275.628541] anaconda[1637]: Running in cmdline mode, no interactive debugging allowed. [ 275.628640] anaconda[1637]: The exact error message is: [ 275.628740] anaconda[1637]: 'NoneType' object has no attribute 'short_label'. [ 275.628839] anaconda[1637]: The installer will now terminate.
Would be useful to see the kickstart used or the full log.
If I'm not mistaken the ks should be https://github.com/coreos/coreos-assembler/blob/master/src/image-base.ks and it seems that there has been some changes recently.
JC
Dan and JC, thanks a lot for your tip! Please check the attachment build-th.log for full build log. Tuan changed 'clearpart --initlabel --all --disklabel=msdos' in image-base.ks because the default gpt even breaks s390x 'cosa fetch'. And I also tried to build fedora 30 of coreos-assembler which failed in ahead (not reaching the step as f29). Thanks!
I just start a fresh build and refresh my memory a little bit about my changes.
The 'msdos' change in image-base.ks file is simply wrong because after all we still need gpt partition scheme. This change simply allows `cosa build` to finish. I have not hit the python3 exception you are having but I do now. Maybe turning on anaconda debugging flags would provide more info. However, I think it would not make sense to do this because it might be obsolete when we have gpt support.
The important part I think is to make anaconda (specifically the block device library it uses [1]) to support gpt on s390x, which at first I thought it requires patching s390-tools/zipl but Dan suggested it's anaconda thing. The stack would be libblockdev -> blivet -> anaconda (Thanks Dan!).
Next steps could be:
- Manually install any s390x Linux distro on a virtual/virtio disk
using gpt partitioning scheme to make sure zipl supports gpt.
- Patch anaconda/blivet/libblockdev to allow gpt on s390x
(alongside msdos and dasd).
[1] https://github.com/storaged-project/blivet'
Tuan
Very good news. I could install Alpine Linux s390x on virtual GPT-ed disk and boot just fine. Should be working with other Linux distros. So the next step is to look into anaconda code.
Cool! Another problem is how to debug anaconda in the VM coreos-inst-12054-1554889268 launched by 'cosa build'. thanks
you should be able to use the image-base kickstart outside coreos to install a regular VM and look what's going on, capture logs, etc.
Dan
On Wed, Apr 24, 2019, at 9:20 AM, Dan Horák wrote:
you should be able to use the image-base kickstart outside coreos to install a regular VM and look what's going on, capture logs, etc.
That shouldn't be necessary, see https://github.com/coreos/coreos-assembler/pull/491
(You don't need that PR, the log extraction functionality works today, it just tells you where to find it)
Also on this topic, what bootloader is s390x using nowadays? So far the Fedora+ostree integration is mostly tested with grub2; see also https://github.com/ostreedev/ostree/issues/1801 though.
It looks like whatever is going wrong is in the s390x bootloader code in Anaconda.
On 4/25/19 10:31 AM, Colin Walters wrote:
On Wed, Apr 24, 2019, at 9:20 AM, Dan Horák wrote:
you should be able to use the image-base kickstart outside coreos to install a regular VM and look what's going on, capture logs, etc.
That shouldn't be necessary, see https://github.com/coreos/coreos-assembler/pull/491
(You don't need that PR, the log extraction functionality works today, it just tells you where to find it)
Also on this topic, what bootloader is s390x using nowadays? So far the Fedora+ostree integration is mostly tested with grub2; see also https://github.com/ostreedev/ostree/issues/1801 though.
It looks like whatever is going wrong is in the s390x bootloader code in Anaconda.
s390x uses zipl as usual.
Yes Anaconda is not recognizing GPT-ed SCSI disks as allowed booting media even though zipl supports both MBR and GPT SCSI disks. https://github.com/rhinstaller/anaconda/blob/master/pyanaconda/platform.py#L...
在 2019/4/25 21:11, Tuan Hoang 写道:
On 4/25/19 10:31 AM, Colin Walters wrote:
On Wed, Apr 24, 2019, at 9:20 AM, Dan Horák wrote:
you should be able to use the image-base kickstart outside coreos to install a regular VM and look what's going on, capture logs, etc.
Thanks Dan, could you pleaes give more details about this?
That shouldn't be necessary, see https://github.com/coreos/coreos-assembler/pull/491
I am not sure if I understand this, but is it just for check anaconda's log? What if I want to check anaconda's code to deep dive? thanks!
(You don't need that PR, the log extraction functionality works today, it just tells you where to find it)
Also on this topic, what bootloader is s390x using nowadays? So far the Fedora+ostree integration is mostly tested with grub2; see also https://github.com/ostreedev/ostree/issues/1801 though.
It looks like whatever is going wrong is in the s390x bootloader code in Anaconda.
Right, that's why I want to deep dive in anaconda's code. thanks
s390x uses zipl as usual.
Yes Anaconda is not recognizing GPT-ed SCSI disks as allowed booting media even though zipl supports both MBR and GPT SCSI disks. https://github.com/rhinstaller/anaconda/blob/master/pyanaconda/platform.py#L... _______________________________________________ CoreOS mailing list -- coreos@lists.fedoraproject.org To unsubscribe send an email to coreos-leave@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/coreos@lists.fedoraproject.org
在 2019/4/25 21:11, Tuan Hoang 写道:
On 4/25/19 10:31 AM, Colin Walters wrote:
On Wed, Apr 24, 2019, at 9:20 AM, Dan Horák wrote:
you should be able to use the image-base kickstart outside coreos to install a regular VM and look what's going on, capture logs, etc.
That shouldn't be necessary, see https://github.com/coreos/coreos-assembler/pull/491
(You don't need that PR, the log extraction functionality works today, it just tells you where to find it)
Also on this topic, what bootloader is s390x using nowadays? So far the Fedora+ostree integration is mostly tested with grub2; see also https://github.com/ostreedev/ostree/issues/1801 though.
It looks like whatever is going wrong is in the s390x bootloader code in Anaconda.
s390x uses zipl as usual.
Yes Anaconda is not recognizing GPT-ed SCSI disks as allowed booting media even though zipl supports both MBR and GPT SCSI disks. https://github.com/rhinstaller/anaconda/blob/master/pyanaconda/platform.py#L...
How to deal with it then? thanks!
CoreOS mailing list -- coreos@lists.fedoraproject.org To unsubscribe send an email to coreos-leave@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/coreos@lists.fedoraproject.org
coreos@lists.fedoraproject.org
-
Colin Walters -
Dan Horák -
Dan Horák -
Jakub Cajka -
QingFeng Hao -
Tuan Hoang