https://bugzilla.redhat.com/show_bug.cgi?id=1210364
Bug ID: 1210364 Summary: Embedded kubelet cadvisor fails to enumerate docker containers Product: Fedora Version: 22 Component: kubernetes Assignee: jchaloup@redhat.com Reporter: stefw@redhat.com QA Contact: extras-qa@fedoraproject.org CC: eparis@redhat.com, golang@lists.fedoraproject.org, jchaloup@redhat.com, lsm5@redhat.com, nhorman@redhat.com, vbatts@redhat.com
Description of problem:
The cAdvisor embedded in kubelet fails to enumerate docker containers. It fails with the message:
failed to get all Docker containers with error: unable to find data for container /system.slice/docker-....scope
Version-Release number of selected component (if applicable):
kubernetes-0.13.2-0.5.git8d94c43.fc22.x86_64
How reproducible:
Every time. Tried restarting Docker.
Steps to Reproduce: 1. http://localhost:4194/api/v1.2/docker 1. Or just go to http://localhost:4194 and click on 'Docker Containers' to see the same thing.
Actual results:
failed to get all Docker containers with error: unable to find data for container /system.slice/docker-9dade0a01bf0368b6c0dad722f7e85af98593dde00873e6abb2cfd86d2d2b7a9.scope
Expected results:
Output like this:
{"/system.slice/docker-1a0bda816225dc1fcae4e9f0583f982664675f66d523095513ff01694c3311e1.scope":{"name":"/system.slice/docker-1a0bda816225dc1fcae4e9f0583f982664675f66d523095513ff01694c3311e1.scope","aliases":["k8s_ruby-helloworld.8ec853e6_frontend-58wav_default_19192267-d9c8-11e4-b2d6-10c37bdb8410_025ed744","1a0bda816225dc1fcae4e9f0583f982664675f66d523095513ff01694c3311e1"],"namespace":"docker","spec":{"creation_time":"0001-01-01T00:00:00Z","has_c...
Additional info:
When running standalone cadvisor I get the above expected output.
cadvisor-0.10.1-0.1.gitef7dddf.fc22.x86_64
https://bugzilla.redhat.com/show_bug.cgi?id=1210364
--- Comment #1 from Stef Walter stefw@redhat.com --- Obviously when switching between kubelet and cadvisor you have to:
$ sudo systemctl stop kubelet $ sudo systemctl start cadvisor
Or the both try to listen on the same port.
https://bugzilla.redhat.com/show_bug.cgi?id=1210364
--- Comment #2 from Stef Walter stefw@redhat.com --- Confirming this is an upstream bug in cadvisor 0.10.1 ... if I build master it goes away.
https://bugzilla.redhat.com/show_bug.cgi?id=1210364
--- Comment #3 from Stef Walter stefw@redhat.com --- git bisect is hard since various cAdvisor commits either OOM the entire machine, or don't build at all ... but:
* 0.11.0 fails with the above error message * 0.12.0 succeeds in this situation
https://bugzilla.redhat.com/show_bug.cgi?id=1210364
--- Comment #4 from Jan Chaloupka jchaloup@redhat.com --- As kubernetes integrated cadvisor, I have obsoleted cadvisor.
Rawhide already obsoletes cadvisor, f22-f20 will obsolete at the end of this week.
https://bugzilla.redhat.com/show_bug.cgi?id=1210364
--- Comment #5 from Stef Walter stefw@redhat.com --- The integrated cadvisor in kubernetes has this same problem. The component above is 'kubernetes'.
https://bugzilla.redhat.com/show_bug.cgi?id=1210364
--- Comment #6 from Stef Walter stefw@redhat.com --- Work around is to run cadvisor standalone, and put this in /etc/kubernetes/kubelet
KUBELET_ARGS="--cadvisor_port=4192"
https://bugzilla.redhat.com/show_bug.cgi?id=1210364
--- Comment #7 from Stef Walter stefw@redhat.com --- Seems to be fixed by: kubernetes-0.15.0-8.fc22
https://bugzilla.redhat.com/show_bug.cgi?id=1210364
Stef Walter stefw@redhat.com changed:
What |Removed |Added ---------------------------------------------------------------------------- Fixed In Version| |kubernetes-0.15.0-8.fc22
https://bugzilla.redhat.com/show_bug.cgi?id=1210364
--- Comment #8 from Jan Chaloupka jchaloup@redhat.com --- Thanks Stef for the message.
https://bugzilla.redhat.com/show_bug.cgi?id=1210364
--- Comment #9 from Stef Walter stefw@redhat.com --- Unfortunately I spoke too soon. I'm seeing this error again with an updated Fedora Atomic 22 that includes kubernetes-0.15.0-8.fc22
failed to get container "" with error: unable to find data for container /system.slice/docker-3ca350475bcc4b2fdb7dbb49b3dab55f7b47505dfcb93b4952d13a397dc4f07a.scope
https://bugzilla.redhat.com/show_bug.cgi?id=1210364
--- Comment #10 from Jan Chaloupka jchaloup@redhat.com --- kubernetes-0.17.0 was released day ago.
Stef, can you check it out if this issue is still valid?
The latest build in koji is kubernetes-0.17.0-3.fc23.
https://bugzilla.redhat.com/show_bug.cgi?id=1210364
--- Comment #11 from Stef Walter stefw@redhat.com --- Running 0.17.0 now. Will let you know if i run the issue again.
https://bugzilla.redhat.com/show_bug.cgi?id=1210364
--- Comment #12 from Stef Walter stefw@redhat.com --- 0.17.0 doesn't start containers for me. Since you asked, I'll post here, but happy to break this out into another bug if you want.
Mai 13 21:03:50 falcon.thewalter.lan docker[10292]: time="2015-05-13T21:03:50+02:00" level=error msg="Handler for POST /containers/{name:.*}/start returned error: Cannot start container 750db29d128126ac8f32ed6c9a89abc7c30616280c6b8e32fbf3e5ad6fbda04b: [8] System error: Unit name /-750db29d128126ac8f32ed6c9a89abc7c30616280c6b8e32fbf3e5ad6fbda04b.scope is not valid." Mai 13 21:03:50 falcon.thewalter.lan kubelet[10510]: E0513 21:03:50.861443 10510 manager.go:1436] Failed to create pod infra container: API error (500): Cannot start container 750db29d128126ac8f32ed6c9a89abc7c30616280c6b8e32fbf3e5ad6fbda04b: [8] System error: Unit name /-750db29d128126ac8f32ed6c9a89abc7c30616280c6b8e32fbf3e5ad6fbda04b.scope is not valid.
docker-1.6.0-3.git9d26a07.fc22.x86_64 kubernetes-0.17.0-3.fc23.x86_64 etcd-2.0.8-0.1.fc22.x86_64
State: Image: mysql is ready, container is creating
https://bugzilla.redhat.com/show_bug.cgi?id=1210364
--- Comment #13 from Jan Chaloupka jchaloup@redhat.com --- Great :) Thanks.
What does "Unit name /-750db29d128126ac8f32ed6c9a89abc7c30616280c6b8e32fbf3e5ad6fbda04b.scope is not valid." mean?
Looks like another bug, correct? Could be missing support of docker for this unit name?
happy to break this out into another bug if you want
What component?
https://bugzilla.redhat.com/show_bug.cgi?id=1210364
--- Comment #14 from Stef Walter stefw@redhat.com --- It's likely related to https://github.com/docker/docker/issues/7015
But cAdvisor (and/or kubelet) should account for that.
https://bugzilla.redhat.com/show_bug.cgi?id=1210364
Jan Chaloupka jchaloup@redhat.com changed:
What |Removed |Added ---------------------------------------------------------------------------- CC| |stefw@redhat.com Flags| |needinfo?(stefw@redhat.com)
--- Comment #15 from Jan Chaloupka jchaloup@redhat.com --- Does it still occur?
https://bugzilla.redhat.com/show_bug.cgi?id=1210364
Andy Goldstein agoldste@redhat.com changed:
What |Removed |Added ---------------------------------------------------------------------------- CC| |agoldste@redhat.com
--- Comment #16 from Andy Goldstein agoldste@redhat.com --- Stef, could you please let us know if this is still an issue?
https://bugzilla.redhat.com/show_bug.cgi?id=1210364
Jan Chaloupka jchaloup@redhat.com changed:
What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |CLOSED Resolution|--- |CURRENTRELEASE Last Closed| |2016-06-29 06:14:58
--- Comment #17 from Jan Chaloupka jchaloup@redhat.com --- Currently, the f23 stable repository provides kubernetes-1.2.0-0.18.git4a3f9c5.fc23 build. Far from 0.17. At the same time the docker has been updated many times.
If the issue still persists, please reopen the bug. Closing for now.
golang@lists.fedoraproject.org
-
Red Hat Bugzilla