Hullo
It's been a while since I've spun up fedoraiot, so I thought I'd see where we'd got to, as I've now got a concrete usecase monitoring some solar panels. Unfortunately, when I put Fedora-IoT-32-20200603.0.aarch64.raw.xz onto an SDCard, the pi (3b+) boots and gets to a login prompt, but I cannot login; neither as `root` with no password on the console, nor ssh'ing as `root` after using the Zezere portal. I'm pretty sure that I'm using the right key (although the documentation is a bit light on what a good key content looks like). The `/portal/devices/` page shows that there's an action to upload a key, with a `cancel runrequest` button, but no indication that more is happening.
Any hints or tips / troubleshooting approaches?
Tim
I managed to fix the access by mounting /dev/sdb3 and copying a known good `authorized_keys` to `ostree/deploy/fedora-iot/var/roothome/.ssh/authorized_keys`. The version on the image looked like a version of the public key that I thought I'd deleted on zerere before trying to claim this device. It was just the key, and lacked the key type at the start of the line and userid at the end.
Hi Tim,
It's been a while since I've spun up fedoraiot, so I thought I'd see where we'd got to, as I've now got a concrete usecase monitoring some solar panels. Unfortunately, when I put Fedora-IoT-32-20200603.0.aarch64.raw.xz onto an SDCard, the pi (3b+) boots and gets to a login prompt, but I cannot login; neither as `root` with no password on the console, nor ssh'ing as `root` after using the Zezere portal. I'm pretty sure that I'm using the right key (although the documentation is a bit light on what a good key content looks like). The `/portal/devices/` page shows that there's an action to upload a key, with a `cancel runrequest` button, but no indication that more is happening.
Any hints or tips / troubleshooting approaches?
There's details in the docs for key provisioning: https://docs.fedoraproject.org/en-US/iot/ignition/
Hi Peter
Thanks. I had spotted that, but couldn’t find the detail that I needed (eg definition of the key - does it include the key type? is it just a copy of an entry in ~/.ssh/authorized_keys); should all keys be dropped into /root/.ssh/authorized_keys? What triggers the upload? etc. As a user, with a pi that you cannot log into, there’s not a lot to go on.
Tim
On 21 Aug 2020, at 07:34, Peter Robinson pbrobinson@gmail.com wrote:
Hi Tim,
It's been a while since I've spun up fedoraiot, so I thought I'd see where we'd got to, as I've now got a concrete usecase monitoring some solar panels. Unfortunately, when I put Fedora-IoT-32-20200603.0.aarch64.raw.xz onto an SDCard, the pi (3b+) boots and gets to a login prompt, but I cannot login; neither as `root` with no password on the console, nor ssh'ing as `root` after using the Zezere portal. I'm pretty sure that I'm using the right key (although the documentation is a bit light on what a good key content looks like). The `/portal/devices/` page shows that there's an action to upload a key, with a `cancel runrequest` button, but no indication that more is happening.
Any hints or tips / troubleshooting approaches?
There's details in the docs for key provisioning: https://docs.fedoraproject.org/en-US/iot/ignition/ _______________________________________________ IoT mailing list -- iot@lists.fedoraproject.org To unsubscribe send an email to iot-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/iot@lists.fedoraproject.org
Hullo
I’m sure that I’m being a bit dim here and that I’ve seen this working. However.
Starting with a downloaded image for a Pi (3b+), after a `rpm-ostree upgrade`, I seem unable to deploy layered packages. They seem to install, but after the reboot, they’re not there. As an example, I tried to drop `toolbox` onto the image so that I could build some of my own rpms to run under systemd, using the approach here: https://bit.ly/3b8Ivri. The initial status before installation looks like this: ``` rpm-ostree status State: idle Deployments: * ostree://fedora-iot:fedora/stable/aarch64/iot Version: 32.20200817.0 (2020-08-17T15:28:18Z) Commit: ace3685f47d39f5d4b6065d49554cf9ada92da685541640298b53576895176ea GPGSignature: Valid signature by 97A1AE57C3A2372CCA3A4ABA6C13026D12C944D0
ostree://fedora-iot:fedora/stable/aarch64/iot Version: 32.20200603.0 (2020-06-03T10:45:43Z) Commit: c02bd26925b4e849fd0e53f3645e97b5cb22f47d7614c5a047d6200c64b3421b GPGSignature: Valid signature by 7D22D5867F2A4236474BF7B850CB390B3C3359C4 ```
`sudo rpm-ostree install toolbox` progresses as I’d expect, creating a new bootable deployment: ``` [tim@localhost ~]$ sudo rpm-ostree install toolbox Checking out tree ace3685... done Enabled rpm-md repositories: fedora-cisco-openh264 updates fedora fedora-modular updates-modular rpm-md repo 'fedora-cisco-openh264' (cached); generated: 2020-03-17T20:10:43Z rpm-md repo 'updates' (cached); generated: 2020-08-27T14:18:53Z rpm-md repo 'fedora' (cached); generated: 2020-04-22T22:22:16Z rpm-md repo 'fedora-modular' (cached); generated: 2020-04-22T20:58:40Z rpm-md repo 'updates-modular' (cached); generated: 2020-08-27T14:52:01Z Importing rpm-md... done Resolving dependencies... done Checking out packages... done Running pre scripts... done Running post scripts... done Running posttrans scripts... done Writing rpmdb... done Writing OSTree commit... done ⠉ Staging deployment... Staging deployment... done
Freed: 354.4?MB (pkgcache branches: 217) Added: flatpak-session-helper-1.6.5-1.fc32.aarch64 toolbox-0.0.93-1.fc32.aarch64 Run "systemctl reboot" to start a reboot
```
and the status looks ok (ignoring the niggling error): ``` [tim@localhost ~]$ rpm-ostree status error: Timeout was reached [tim@localhost ~]$ rpm-ostree status State: idle Deployments: ostree://fedora-iot:fedora/stable/aarch64/iot Version: 32.20200817.0 (2020-08-17T15:28:18Z) BaseCommit: ace3685f47d39f5d4b6065d49554cf9ada92da685541640298b53576895176ea GPGSignature: Valid signature by 97A1AE57C3A2372CCA3A4ABA6C13026D12C944D0 Diff: 2 added LayeredPackages: toolbox
* ostree://fedora-iot:fedora/stable/aarch64/iot Version: 32.20200817.0 (2020-08-17T15:28:18Z) Commit: ace3685f47d39f5d4b6065d49554cf9ada92da685541640298b53576895176ea GPGSignature: Valid signature by 97A1AE57C3A2372CCA3A4ABA6C13026D12C944D0
ostree://fedora-iot:fedora/stable/aarch64/iot Version: 32.20200603.0 (2020-06-03T10:45:43Z) Commit: c02bd26925b4e849fd0e53f3645e97b5cb22f47d7614c5a047d6200c64b3421b GPGSignature: Valid signature by 7D22D5867F2A4236474BF7B850CB390B3C3359C4 ```
But when I reboot, that deployment isn’t the image used and it’s disappeared from rpm-ostree’s world: ``` [tim@localhost ~]$ sudo rpm-ostree status State: idle Deployments: * ostree://fedora-iot:fedora/stable/aarch64/iot Version: 32.20200817.0 (2020-08-17T15:28:18Z) Commit: ace3685f47d39f5d4b6065d49554cf9ada92da685541640298b53576895176ea GPGSignature: Valid signature by 97A1AE57C3A2372CCA3A4ABA6C13026D12C944D0
ostree://fedora-iot:fedora/stable/aarch64/iot Version: 32.20200603.0 (2020-06-03T10:45:43Z) Commit: c02bd26925b4e849fd0e53f3645e97b5cb22f47d7614c5a047d6200c64b3421b GPGSignature: Valid signature by 7D22D5867F2A4236474BF7B850CB390B3C3359C4 ```
Did I forget how this type of layered installation should work?
Tim
On 8/28/20 5:43 PM, TimC wrote:
Hullo
I’m sure that I’m being a bit dim here and that I’ve seen this working. However.
Starting with a downloaded image for a Pi (3b+), after a `rpm-ostree upgrade`, I seem unable to deploy layered packages. They seem to install, but after the reboot, they’re not there. As an example, I tried to drop `toolbox` onto the image so that I could build some of my own rpms to run under systemd, using the approach here: https://bit.ly/3b8Ivri. The initial status before installation looks like this:
rpm-ostree status State: idle Deployments: * ostree://fedora-iot:fedora/stable/aarch64/iot Version: 32.20200817.0 (2020-08-17T15:28:18Z) Commit: ace3685f47d39f5d4b6065d49554cf9ada92da685541640298b53576895176ea GPGSignature: Valid signature by 97A1AE57C3A2372CCA3A4ABA6C13026D12C944D0 ostree://fedora-iot:fedora/stable/aarch64/iot Version: 32.20200603.0 (2020-06-03T10:45:43Z) Commit: c02bd26925b4e849fd0e53f3645e97b5cb22f47d7614c5a047d6200c64b3421b GPGSignature: Valid signature by 7D22D5867F2A4236474BF7B850CB390B3C3359C4
`sudo rpm-ostree install toolbox` progresses as I’d expect, creating a new bootable deployment:
[tim@localhost ~]$ sudo rpm-ostree install toolbox Checking out tree ace3685... done Enabled rpm-md repositories: fedora-cisco-openh264 updates fedora fedora-modular updates-modular rpm-md repo 'fedora-cisco-openh264' (cached); generated: 2020-03-17T20:10:43Z rpm-md repo 'updates' (cached); generated: 2020-08-27T14:18:53Z rpm-md repo 'fedora' (cached); generated: 2020-04-22T22:22:16Z rpm-md repo 'fedora-modular' (cached); generated: 2020-04-22T20:58:40Z rpm-md repo 'updates-modular' (cached); generated: 2020-08-27T14:52:01Z Importing rpm-md... done Resolving dependencies... done Checking out packages... done Running pre scripts... done Running post scripts... done Running posttrans scripts... done Writing rpmdb... done Writing OSTree commit... done ⠉ Staging deployment... Staging deployment... done Freed: 354.4?MB (pkgcache branches: 217) Added: flatpak-session-helper-1.6.5-1.fc32.aarch64 toolbox-0.0.93-1.fc32.aarch64 Run "systemctl reboot" to start a reboot
and the status looks ok (ignoring the niggling error):
[tim@localhost ~]$ rpm-ostree status error: Timeout was reached [tim@localhost ~]$ rpm-ostree status State: idle Deployments: ostree://fedora-iot:fedora/stable/aarch64/iot Version: 32.20200817.0 (2020-08-17T15:28:18Z) BaseCommit: ace3685f47d39f5d4b6065d49554cf9ada92da685541640298b53576895176ea GPGSignature: Valid signature by 97A1AE57C3A2372CCA3A4ABA6C13026D12C944D0 Diff: 2 added LayeredPackages: toolbox * ostree://fedora-iot:fedora/stable/aarch64/iot Version: 32.20200817.0 (2020-08-17T15:28:18Z) Commit: ace3685f47d39f5d4b6065d49554cf9ada92da685541640298b53576895176ea GPGSignature: Valid signature by 97A1AE57C3A2372CCA3A4ABA6C13026D12C944D0 ostree://fedora-iot:fedora/stable/aarch64/iot Version: 32.20200603.0 (2020-06-03T10:45:43Z) Commit: c02bd26925b4e849fd0e53f3645e97b5cb22f47d7614c5a047d6200c64b3421b GPGSignature: Valid signature by 7D22D5867F2A4236474BF7B850CB390B3C3359C4
But when I reboot, that deployment isn’t the image used and it’s disappeared from rpm-ostree’s world:
[tim@localhost ~]$ sudo rpm-ostree status State: idle Deployments: * ostree://fedora-iot:fedora/stable/aarch64/iot Version: 32.20200817.0 (2020-08-17T15:28:18Z) Commit: ace3685f47d39f5d4b6065d49554cf9ada92da685541640298b53576895176ea GPGSignature: Valid signature by 97A1AE57C3A2372CCA3A4ABA6C13026D12C944D0 ostree://fedora-iot:fedora/stable/aarch64/iot Version: 32.20200603.0 (2020-06-03T10:45:43Z) Commit: c02bd26925b4e849fd0e53f3645e97b5cb22f47d7614c5a047d6200c64b3421b GPGSignature: Valid signature by 7D22D5867F2A4236474BF7B850CB390B3C3359C4
Did I forget how this type of layered installation should work?
Seems like it should be working. This vaguely sounds like the deployment not getting finalized.
Does `sudo journalctl -u ostree-finalize-staged.service` show any errors? If it is timing out for some reason you could try to manually run `sudo ostree admin finalize-staged` *before* you type reboot to reboot into the new deployment so that you know it finishes.
Dusty
Hi Dusty
Is this a likely culprit? ``` -- Reboot -- Aug 28 20:56:24 localhost.localdomain systemd[1]: Finished OSTree Finalize Staged Deployment. Aug 28 21:07:23 localhost.localdomain systemd[1]: Stopping OSTree Finalize Staged Deployment... Aug 28 21:07:23 localhost.localdomain ostree[14561]: Finalizing staged deployment Aug 28 21:07:32 localhost.localdomain ostree[14561]: Copying /etc changes: 10 modified, 0 removed, 34 added Aug 28 21:07:32 localhost.localdomain ostree[14561]: Copying /etc changes: 10 modified, 0 removed, 34 added Aug 28 21:08:30 localhost.localdomain ostree[14561]: ** Aug 28 21:08:30 localhost.localdomain ostree[14561]: OSTree:ERROR:src/libostree/ostree-sysroot-deploy.c:1775:install_deployment_kernel: assertion failed (kernel_layout->bootcsum == bootcsum): ("02ec07fa832b4485887f6dbac19d0311e3> Aug 28 21:08:30 localhost.localdomain ostree[14561]: Bail out! OSTree:ERROR:src/libostree/ostree-sysroot-deploy.c:1775:install_deployment_kernel: assertion failed (kernel_layout->bootcsum == bootcsum): ("02ec07fa832b4485887f6dba> Aug 28 21:08:30 localhost.localdomain systemd[1]: ostree-finalize-staged.service: Control process exited, code=dumped, status=6/ABRT Aug 28 21:08:30 localhost.localdomain systemd[1]: ostree-finalize-staged.service: Failed with result 'core-dump'. Aug 28 21:08:30 localhost.localdomain systemd[1]: Stopped OSTree Finalize Staged Deployment. Aug 28 21:08:30 localhost.localdomain systemd[1]: ostree-finalize-staged.service: Consumed 8.211s CPU time. ```
Note that this is from yesterday’s fail. When I tried again today, I only get this (before reboot): ``` -- Reboot -- Aug 29 11:12:12 localhost.localdomain systemd[1]: Finished OSTree Finalize Staged Deployment. ```
When I run `sudo ostree admin finalize-staged`, it fails, thus: ``` sudo ostree admin finalize-staged Copying /etc changes: 10 modified, 0 removed, 34 added ** OSTree:ERROR:src/libostree/ostree-sysroot-deploy.c:1775:install_deployment_kernel: assertion failed (kernel_layout->bootcsum == bootcsum): ("02ec07fa832b4485887f6dbac19d0311e3039855cc6ebd57aefaafa4a3997d1e" == "fc8629fdb6f5355dd5359350296b619216cbc32e19887511caf75ee9d6aafbad") Bail out! OSTree:ERROR:src/libostree/ostree-sysroot-deploy.c:1775:install_deployment_kernel: assertion failed (kernel_layout->bootcsum == bootcsum): ("02ec07fa832b4485887f6dbac19d0311e3039855cc6ebd57aefaafa4a3997d1e" == "fc8629fdb6f5355dd5359350296b619216cbc32e19887511caf75ee9d6aafbad") Aborted
```
I’d guess that’s a fail originating somehow in my trying to increase the image and filesystem in the download as my first attempts to layer packages just ran out of space. The approach worked with Fedora-IoT 29 and was based on decompressing, appending 0’s to the end of the file and then resizing the filesystem.
Would it make sense to confirm the cause by layering a single package onto the original download, if that’s useful? I think that I will need some mechanism to increase the image size, tho’ or I won’t be able to get many tools onto the pi.
Tim
On 29 Aug 2020, at 06:02, Dusty Mabe dusty@dustymabe.com wrote:
On 8/28/20 5:43 PM, TimC wrote:
Hullo
I’m sure that I’m being a bit dim here and that I’ve seen this working. However.
Starting with a downloaded image for a Pi (3b+), after a `rpm-ostree upgrade`, I seem unable to deploy layered packages. They seem to install, but after the reboot, they’re not there. As an example, I tried to drop `toolbox` onto the image so that I could build some of my own rpms to run under systemd, using the approach here: https://bit.ly/3b8Ivri. The initial status before installation looks like this:
rpm-ostree status State: idle Deployments: * ostree://fedora-iot:fedora/stable/aarch64/iot Version: 32.20200817.0 (2020-08-17T15:28:18Z) Commit: ace3685f47d39f5d4b6065d49554cf9ada92da685541640298b53576895176ea GPGSignature: Valid signature by 97A1AE57C3A2372CCA3A4ABA6C13026D12C944D0 ostree://fedora-iot:fedora/stable/aarch64/iot Version: 32.20200603.0 (2020-06-03T10:45:43Z) Commit: c02bd26925b4e849fd0e53f3645e97b5cb22f47d7614c5a047d6200c64b3421b GPGSignature: Valid signature by 7D22D5867F2A4236474BF7B850CB390B3C3359C4
`sudo rpm-ostree install toolbox` progresses as I’d expect, creating a new bootable deployment:
[tim@localhost ~]$ sudo rpm-ostree install toolbox Checking out tree ace3685... done Enabled rpm-md repositories: fedora-cisco-openh264 updates fedora fedora-modular updates-modular rpm-md repo 'fedora-cisco-openh264' (cached); generated: 2020-03-17T20:10:43Z rpm-md repo 'updates' (cached); generated: 2020-08-27T14:18:53Z rpm-md repo 'fedora' (cached); generated: 2020-04-22T22:22:16Z rpm-md repo 'fedora-modular' (cached); generated: 2020-04-22T20:58:40Z rpm-md repo 'updates-modular' (cached); generated: 2020-08-27T14:52:01Z Importing rpm-md... done Resolving dependencies... done Checking out packages... done Running pre scripts... done Running post scripts... done Running posttrans scripts... done Writing rpmdb... done Writing OSTree commit... done ⠉ Staging deployment... Staging deployment... done Freed: 354.4?MB (pkgcache branches: 217) Added: flatpak-session-helper-1.6.5-1.fc32.aarch64 toolbox-0.0.93-1.fc32.aarch64 Run "systemctl reboot" to start a reboot
and the status looks ok (ignoring the niggling error):
[tim@localhost ~]$ rpm-ostree status error: Timeout was reached [tim@localhost ~]$ rpm-ostree status State: idle Deployments: ostree://fedora-iot:fedora/stable/aarch64/iot Version: 32.20200817.0 (2020-08-17T15:28:18Z) BaseCommit: ace3685f47d39f5d4b6065d49554cf9ada92da685541640298b53576895176ea GPGSignature: Valid signature by 97A1AE57C3A2372CCA3A4ABA6C13026D12C944D0 Diff: 2 added LayeredPackages: toolbox * ostree://fedora-iot:fedora/stable/aarch64/iot Version: 32.20200817.0 (2020-08-17T15:28:18Z) Commit: ace3685f47d39f5d4b6065d49554cf9ada92da685541640298b53576895176ea GPGSignature: Valid signature by 97A1AE57C3A2372CCA3A4ABA6C13026D12C944D0 ostree://fedora-iot:fedora/stable/aarch64/iot Version: 32.20200603.0 (2020-06-03T10:45:43Z) Commit: c02bd26925b4e849fd0e53f3645e97b5cb22f47d7614c5a047d6200c64b3421b GPGSignature: Valid signature by 7D22D5867F2A4236474BF7B850CB390B3C3359C4
But when I reboot, that deployment isn’t the image used and it’s disappeared from rpm-ostree’s world:
[tim@localhost ~]$ sudo rpm-ostree status State: idle Deployments: * ostree://fedora-iot:fedora/stable/aarch64/iot Version: 32.20200817.0 (2020-08-17T15:28:18Z) Commit: ace3685f47d39f5d4b6065d49554cf9ada92da685541640298b53576895176ea GPGSignature: Valid signature by 97A1AE57C3A2372CCA3A4ABA6C13026D12C944D0 ostree://fedora-iot:fedora/stable/aarch64/iot Version: 32.20200603.0 (2020-06-03T10:45:43Z) Commit: c02bd26925b4e849fd0e53f3645e97b5cb22f47d7614c5a047d6200c64b3421b GPGSignature: Valid signature by 7D22D5867F2A4236474BF7B850CB390B3C3359C4
Did I forget how this type of layered installation should work?
Seems like it should be working. This vaguely sounds like the deployment not getting finalized.
Does `sudo journalctl -u ostree-finalize-staged.service` show any errors? If it is timing out for some reason you could try to manually run `sudo ostree admin finalize-staged` *before* you type reboot to reboot into the new deployment so that you know it finishes.
Dusty
fwiw, there may also be an issue with the log entries being visible to journalctl: after a reboot, there are more log entries, that, I think, originate from before the reboot:
…[last entry after yesterday’s reboot, followed by the logs post `sudo rpm-ostree install toolbox`, then `sudo ostree admin finalize-staged`] ``` Aug 28 21:08:30 localhost.localdomain systemd[1]: ostree-finalize-staged.service: Consumed 8.211s CPU time. -- Reboot -- Aug 29 11:12:12 localhost.localdomain systemd[1]: Finished OSTree Finalize Staged Deployment. Aug 29 11:32:23 localhost.localdomain systemd[1]: Stopping OSTree Finalize Staged Deployment... Aug 29 11:32:23 localhost.localdomain ostree[26417]: No deployment staged for finalization Aug 29 11:32:25 localhost.localdomain systemd[1]: ostree-finalize-staged.service: Succeeded. Aug 29 11:32:25 localhost.localdomain systemd[1]: Stopped OSTree Finalize Staged Deployment. ```
On 29 Aug 2020, at 12:34, TimC tim+fedoraproject.org@coote.org wrote:
Hi Dusty
Is this a likely culprit?
-- Reboot -- Aug 28 20:56:24 localhost.localdomain systemd[1]: Finished OSTree Finalize Staged Deployment. Aug 28 21:07:23 localhost.localdomain systemd[1]: Stopping OSTree Finalize Staged Deployment... Aug 28 21:07:23 localhost.localdomain ostree[14561]: Finalizing staged deployment Aug 28 21:07:32 localhost.localdomain ostree[14561]: Copying /etc changes: 10 modified, 0 removed, 34 added Aug 28 21:07:32 localhost.localdomain ostree[14561]: Copying /etc changes: 10 modified, 0 removed, 34 added Aug 28 21:08:30 localhost.localdomain ostree[14561]: ** Aug 28 21:08:30 localhost.localdomain ostree[14561]: OSTree:ERROR:src/libostree/ostree-sysroot-deploy.c:1775:install_deployment_kernel: assertion failed (kernel_layout->bootcsum == bootcsum): ("02ec07fa832b4485887f6dbac19d0311e3> Aug 28 21:08:30 localhost.localdomain ostree[14561]: Bail out! OSTree:ERROR:src/libostree/ostree-sysroot-deploy.c:1775:install_deployment_kernel: assertion failed (kernel_layout->bootcsum == bootcsum): ("02ec07fa832b4485887f6dba> Aug 28 21:08:30 localhost.localdomain systemd[1]: ostree-finalize-staged.service: Control process exited, code=dumped, status=6/ABRT Aug 28 21:08:30 localhost.localdomain systemd[1]: ostree-finalize-staged.service: Failed with result 'core-dump'. Aug 28 21:08:30 localhost.localdomain systemd[1]: Stopped OSTree Finalize Staged Deployment. Aug 28 21:08:30 localhost.localdomain systemd[1]: ostree-finalize-staged.service: Consumed 8.211s CPU time.
Note that this is from yesterday’s fail. When I tried again today, I only get this (before reboot):
-- Reboot -- Aug 29 11:12:12 localhost.localdomain systemd[1]: Finished OSTree Finalize Staged Deployment.
When I run `sudo ostree admin finalize-staged`, it fails, thus:
sudo ostree admin finalize-staged Copying /etc changes: 10 modified, 0 removed, 34 added ** OSTree:ERROR:src/libostree/ostree-sysroot-deploy.c:1775:install_deployment_kernel: assertion failed (kernel_layout->bootcsum == bootcsum): ("02ec07fa832b4485887f6dbac19d0311e3039855cc6ebd57aefaafa4a3997d1e" == "fc8629fdb6f5355dd5359350296b619216cbc32e19887511caf75ee9d6aafbad") Bail out! OSTree:ERROR:src/libostree/ostree-sysroot-deploy.c:1775:install_deployment_kernel: assertion failed (kernel_layout->bootcsum == bootcsum): ("02ec07fa832b4485887f6dbac19d0311e3039855cc6ebd57aefaafa4a3997d1e" == "fc8629fdb6f5355dd5359350296b619216cbc32e19887511caf75ee9d6aafbad") Aborted
I’d guess that’s a fail originating somehow in my trying to increase the image and filesystem in the download as my first attempts to layer packages just ran out of space. The approach worked with Fedora-IoT 29 and was based on decompressing, appending 0’s to the end of the file and then resizing the filesystem.
Would it make sense to confirm the cause by layering a single package onto the original download, if that’s useful? I think that I will need some mechanism to increase the image size, tho’ or I won’t be able to get many tools onto the pi.
Tim
On 29 Aug 2020, at 06:02, Dusty Mabe dusty@dustymabe.com wrote:
On 8/28/20 5:43 PM, TimC wrote:
Hullo
I’m sure that I’m being a bit dim here and that I’ve seen this working. However.
Starting with a downloaded image for a Pi (3b+), after a `rpm-ostree upgrade`, I seem unable to deploy layered packages. They seem to install, but after the reboot, they’re not there. As an example, I tried to drop `toolbox` onto the image so that I could build some of my own rpms to run under systemd, using the approach here: https://bit.ly/3b8Ivri. The initial status before installation looks like this:
rpm-ostree status State: idle Deployments: * ostree://fedora-iot:fedora/stable/aarch64/iot Version: 32.20200817.0 (2020-08-17T15:28:18Z) Commit: ace3685f47d39f5d4b6065d49554cf9ada92da685541640298b53576895176ea GPGSignature: Valid signature by 97A1AE57C3A2372CCA3A4ABA6C13026D12C944D0 ostree://fedora-iot:fedora/stable/aarch64/iot Version: 32.20200603.0 (2020-06-03T10:45:43Z) Commit: c02bd26925b4e849fd0e53f3645e97b5cb22f47d7614c5a047d6200c64b3421b GPGSignature: Valid signature by 7D22D5867F2A4236474BF7B850CB390B3C3359C4
`sudo rpm-ostree install toolbox` progresses as I’d expect, creating a new bootable deployment:
[tim@localhost ~]$ sudo rpm-ostree install toolbox Checking out tree ace3685... done Enabled rpm-md repositories: fedora-cisco-openh264 updates fedora fedora-modular updates-modular rpm-md repo 'fedora-cisco-openh264' (cached); generated: 2020-03-17T20:10:43Z rpm-md repo 'updates' (cached); generated: 2020-08-27T14:18:53Z rpm-md repo 'fedora' (cached); generated: 2020-04-22T22:22:16Z rpm-md repo 'fedora-modular' (cached); generated: 2020-04-22T20:58:40Z rpm-md repo 'updates-modular' (cached); generated: 2020-08-27T14:52:01Z Importing rpm-md... done Resolving dependencies... done Checking out packages... done Running pre scripts... done Running post scripts... done Running posttrans scripts... done Writing rpmdb... done Writing OSTree commit... done ⠉ Staging deployment... Staging deployment... done Freed: 354.4?MB (pkgcache branches: 217) Added: flatpak-session-helper-1.6.5-1.fc32.aarch64 toolbox-0.0.93-1.fc32.aarch64 Run "systemctl reboot" to start a reboot
and the status looks ok (ignoring the niggling error):
[tim@localhost ~]$ rpm-ostree status error: Timeout was reached [tim@localhost ~]$ rpm-ostree status State: idle Deployments: ostree://fedora-iot:fedora/stable/aarch64/iot Version: 32.20200817.0 (2020-08-17T15:28:18Z) BaseCommit: ace3685f47d39f5d4b6065d49554cf9ada92da685541640298b53576895176ea GPGSignature: Valid signature by 97A1AE57C3A2372CCA3A4ABA6C13026D12C944D0 Diff: 2 added LayeredPackages: toolbox * ostree://fedora-iot:fedora/stable/aarch64/iot Version: 32.20200817.0 (2020-08-17T15:28:18Z) Commit: ace3685f47d39f5d4b6065d49554cf9ada92da685541640298b53576895176ea GPGSignature: Valid signature by 97A1AE57C3A2372CCA3A4ABA6C13026D12C944D0 ostree://fedora-iot:fedora/stable/aarch64/iot Version: 32.20200603.0 (2020-06-03T10:45:43Z) Commit: c02bd26925b4e849fd0e53f3645e97b5cb22f47d7614c5a047d6200c64b3421b GPGSignature: Valid signature by 7D22D5867F2A4236474BF7B850CB390B3C3359C4
But when I reboot, that deployment isn’t the image used and it’s disappeared from rpm-ostree’s world:
[tim@localhost ~]$ sudo rpm-ostree status State: idle Deployments: * ostree://fedora-iot:fedora/stable/aarch64/iot Version: 32.20200817.0 (2020-08-17T15:28:18Z) Commit: ace3685f47d39f5d4b6065d49554cf9ada92da685541640298b53576895176ea GPGSignature: Valid signature by 97A1AE57C3A2372CCA3A4ABA6C13026D12C944D0 ostree://fedora-iot:fedora/stable/aarch64/iot Version: 32.20200603.0 (2020-06-03T10:45:43Z) Commit: c02bd26925b4e849fd0e53f3645e97b5cb22f47d7614c5a047d6200c64b3421b GPGSignature: Valid signature by 7D22D5867F2A4236474BF7B850CB390B3C3359C4
Did I forget how this type of layered installation should work?
Seems like it should be working. This vaguely sounds like the deployment not getting finalized.
Does `sudo journalctl -u ostree-finalize-staged.service` show any errors? If it is timing out for some reason you could try to manually run `sudo ostree admin finalize-staged` *before* you type reboot to reboot into the new deployment so that you know it finishes.
Dusty
IoT mailing list -- iot@lists.fedoraproject.org To unsubscribe send an email to iot-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/iot@lists.fedoraproject.org
On 8/29/20 7:34 AM, TimC wrote:
Hi Dusty
Is this a likely culprit?
-- Reboot -- Aug 28 20:56:24 localhost.localdomain systemd[1]: Finished OSTree Finalize Staged Deployment. Aug 28 21:07:23 localhost.localdomain systemd[1]: Stopping OSTree Finalize Staged Deployment... Aug 28 21:07:23 localhost.localdomain ostree[14561]: Finalizing staged deployment Aug 28 21:07:32 localhost.localdomain ostree[14561]: Copying /etc changes: 10 modified, 0 removed, 34 added Aug 28 21:07:32 localhost.localdomain ostree[14561]: Copying /etc changes: 10 modified, 0 removed, 34 added Aug 28 21:08:30 localhost.localdomain ostree[14561]: ** Aug 28 21:08:30 localhost.localdomain ostree[14561]: OSTree:ERROR:src/libostree/ostree-sysroot-deploy.c:1775:install_deployment_kernel: assertion failed (kernel_layout->bootcsum == bootcsum): ("02ec07fa832b4485887f6dbac19d0311e3> Aug 28 21:08:30 localhost.localdomain ostree[14561]: Bail out! OSTree:ERROR:src/libostree/ostree-sysroot-deploy.c:1775:install_deployment_kernel: assertion failed (kernel_layout->bootcsum == bootcsum): ("02ec07fa832b4485887f6dba> Aug 28 21:08:30 localhost.localdomain systemd[1]: ostree-finalize-staged.service: Control process exited, code=dumped, status=6/ABRT Aug 28 21:08:30 localhost.localdomain systemd[1]: ostree-finalize-staged.service: Failed with result 'core-dump'. Aug 28 21:08:30 localhost.localdomain systemd[1]: Stopped OSTree Finalize Staged Deployment. Aug 28 21:08:30 localhost.localdomain systemd[1]: ostree-finalize-staged.service: Consumed 8.211s CPU time.
I would say so. It's either a bug or the things you did to modify the system (mentioned below) caused some corruption maybe.
Note that this is from yesterday’s fail. When I tried again today, I only get this (before reboot):
-- Reboot -- Aug 29 11:12:12 localhost.localdomain systemd[1]: Finished OSTree Finalize Staged Deployment.
When I run `sudo ostree admin finalize-staged`, it fails, thus:
sudo ostree admin finalize-staged Copying /etc changes: 10 modified, 0 removed, 34 added ** OSTree:ERROR:src/libostree/ostree-sysroot-deploy.c:1775:install_deployment_kernel: assertion failed (kernel_layout->bootcsum == bootcsum): ("02ec07fa832b4485887f6dbac19d0311e3039855cc6ebd57aefaafa4a3997d1e" == "fc8629fdb6f5355dd5359350296b619216cbc32e19887511caf75ee9d6aafbad") Bail out! OSTree:ERROR:src/libostree/ostree-sysroot-deploy.c:1775:install_deployment_kernel: assertion failed (kernel_layout->bootcsum == bootcsum): ("02ec07fa832b4485887f6dbac19d0311e3039855cc6ebd57aefaafa4a3997d1e" == "fc8629fdb6f5355dd5359350296b619216cbc32e19887511caf75ee9d6aafbad") Aborted
I’d guess that’s a fail originating somehow in my trying to increase the image and filesystem in the download as my first attempts to layer packages just ran out of space. The approach worked with Fedora-IoT 29 and was based on decompressing, appending 0’s to the end of the file and then resizing the filesystem.
I'm not sure if that is OK or not. I haven't used IoT much, but I do know the rpm-ostree ecosystem a bit. Somebody else might be able to answer better than I.
Would it make sense to confirm the cause by layering a single package onto the original download, if that’s useful? I think that I will need some mechanism to increase the image size, tho’ or I won’t be able to get many tools onto the pi.
That would be good information.
Tim
I’d guess that’s a fail originating somehow in my trying to increase the image and filesystem in the download as my first attempts to layer packages just ran out of space. The approach worked with Fedora-IoT 29 and was based on decompressing, appending 0’s to the end of the file and then resizing the filesystem.
How are you running this image? Is it a VM image or on a physical piece of hardware?
If a VM you should be able to do this: https://nullr0ute.com/2018/08/increasing-a-libvirt-kvm-virtual-machine-disk-...