Re: [PATCH] ci: Add test cases for container fetching and loading

[Uladzimir Bely <ubely@ilbers.de>]

On Tue, 2024-08-06 at 12:46 +0200, Jan Kiszka wrote:
> On 06.08.24 11:48, Uladzimir Bely wrote:
> > On Tue, 2024-08-06 at 07:48 +0300, Uladzimir Bely wrote:
> > > On Mon, 2024-08-05 at 13:51 +0300, Uladzimir Bely wrote:
> > > > On Mon, 2024-08-05 at 12:43 +0200, Jan Kiszka wrote:
> > > > > On 05.08.24 11:40, Uladzimir Bely wrote:
> > > > > > On Mon, 2024-08-05 at 11:17 +0200, Jan Kiszka wrote:
> > > > > > > On 05.08.24 09:16, Uladzimir Bely wrote:
> > > > > > > > From: Jan Kiszka <jan.kiszka@siemens.com>
> > > > > > > > 
> > > > > > > > This plugs the two example recipes for loading
> > > > > > > > container
> > > > > > > > images
> > > > > > > > into
> > > > > > > > VM-based testing. The test consists of running 'true'
> > > > > > > > in
> > > > > > > > the
> > > > > > > > installed
> > > > > > > > alpine images.
> > > > > > > > 
> > > > > > > > Rather than enabling the ci user to do password-less
> > > > > > > > sudo,
> > > > > > > > this
> > > > > > > > uses su
> > > > > > > > with the piped-in password. Another trick needed is to
> > > > > > > > poll
> > > > > > > > for
> > > > > > > > the
> > > > > > > > images because loading is performed asynchronously.
> > > > > > > > 
> > > > > > > > Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
> > > > > > > > Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
> > > > > > > > ---
> > > > > > > >  .../recipes-core/images/isar-image-ci.bb      |  2 ++
> > > > > > > >  testsuite/citest.py                           | 24
> > > > > > > > +++++++++++++++++++
> > > > > > > >  2 files changed, 26 insertions(+)
> > > > > > > > 
> > > > > > > > This is a drop-in replacement of patch 4 from "[PATCH
> > > > > > > > v4
> > > > > > > > 0/5]
> > > > > > > > Introduce
> > > > > > > > container fetcher and pre-loader" series:
> > > > > > > > - Fixed syntax errors (incorrectly escaped '\$')
> > > > > > > 
> > > > > > > IIRC, we do need the escape inside the shell (sh -c
> > > > > > > '...').
> > > > > > > So,
> > > > > > > you
> > > > > > > likely rather need to escape the escape character.
> > > > > > > 
> > > > > > > Jan
> > > > > > > 
> > > > > > > > 
> > > > > > 
> > > > > > I just tried to make a simple check:
> > > > > > 
> > > > > > ```
> > > > > > $ su -c 'for i in $(seq 3); do echo $i; done'
> > > > > > Password: 
> > > > > > 1
> > > > > > 2
> > > > > > 3
> > > > > > 
> > > > > > $ su -c 'for i in \$(seq 3); do echo $i; done'
> > > > > > Password: 
> > > > > > bash: -c: line 1: syntax error near unexpected token `('
> > > > > > bash: -c: line 1: `for i in \$(seq 3); do echo $i; done'
> > > > > > 
> > > > > > $ su -c 'for i in \\$(seq 3); do echo $i; done'
> > > > > > Password: 
> > > > > > \1
> > > > > > 2
> > > > > > 3
> > > > > > ```
> > > > > > 
> > > > > > We are likely don't need escaping at all.
> > > > > 
> > > > > Interesting - anyway, if this sequence is not properly
> > > > > resolved,
> > > > > the
> > > > > test will fail. And I assume you had it running successfully,
> > > > > so
> > > > > we
> > > > > must
> > > > > be fine.
> > > > > 
> > > > > > 
> > > > > > Anyway, we could just convert the tests from
> > > > > > "cmd=<long_command"
> > > > > > to "script=test_prebuild_container.sh" and have test logic
> > > > > > in a
> > > > > > human-
> > > > > > readable form.
> > > > > > 
> > > > > 
> > > > > Also fine with me.
> > > > > 
> > > > > Jan
> > > > > 
> > > > 
> > > > OK, I've already prepared the script internally and will check
> > > > in
> > > > CI
> > > > with it.
> > > > 
> > > 
> > > ... and still having problems with running commands inside arm64
> > > container.
> > > 
> > > I manually run (with same command-line as CI does) qemuamd64 and
> > > qemuarm64 images.
> > > 
> > > Running prebuilt container in amd64 machine works well:
> > > 
> > > ```
> > > root@isar:~# docker images
> > > REPOSITORY              TAG       IMAGE ID       CREATED      
> > > SIZE
> > > quay.io/libpod/alpine   3.10.2    961769676411   4 years ago  
> > > 5.58MB
> > > root@isar:~# docker run --rm quay.io/libpod/alpine:3.10.2 true
> > > [   61.233873] docker0: port 1(veth1c2b6f9) entered blocking
> > > state
> > > [   61.234280] docker0: port 1(veth1c2b6f9) entered disabled
> > > state
> > > [   61.240243] device veth1c2b6f9 entered promiscuous mode
> > > [   62.650328] eth0: renamed from veth2aff680
> > > [   62.664713] IPv6: ADDRCONF(NETDEV_CHANGE): veth1c2b6f9: link
> > > becomes
> > > ready
> > > [   62.665407] docker0: port 1(veth1c2b6f9) entered blocking
> > > state
> > > [   62.665656] docker0: port 1(veth1c2b6f9) entered forwarding
> > > state
> > > [   62.666394] IPv6: ADDRCONF(NETDEV_CHANGE): docker0: link
> > > becomes
> > > ready
> > > [   63.220542] docker0: port 1(veth1c2b6f9) entered disabled
> > > state
> > > [   63.229530] veth2aff680: renamed from eth0
> > > [   63.308290] docker0: port 1(veth1c2b6f9) entered disabled
> > > state
> > > [   63.311282] device veth1c2b6f9 left promiscuous mode
> > > [   63.311507] docker0: port 1(veth1c2b6f9) entered disabled
> > > state
> > > root@isar:~# echo $?
> > > 0
> > > root@isar:~# podman images
> > > REPOSITORY             TAG         IMAGE ID      CREATED     
> > > SIZE
> > > quay.io/libpod/alpine  latest      961769676411  4 years ago 
> > > 5.85 MB
> > > root@isar:~# podman run --rm quay.io/libpod/alpine:latest true
> > > [   78.274955] cni-podman0: port 1(vethf6fde03e) entered blocking
> > > state
> > > [   78.275225] cni-podman0: port 1(vethf6fde03e) entered disabled
> > > state
> > > [   78.277667] device vethf6fde03e entered promiscuous mode
> > > [   78.626628] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes
> > > ready
> > > [   78.627038] IPv6: ADDRCONF(NETDEV_CHANGE): vethf6fde03e: link
> > > becomes ready
> > > [   78.627313] cni-podman0: port 1(vethf6fde03e) entered blocking
> > > state
> > > [   78.627513] cni-podman0: port 1(vethf6fde03e) entered
> > > forwarding
> > > state
> > > [   79.690462] audit: type=1400 audit(1722919083.116:6):
> > > apparmor="STATUS" operation="profile_load" profile="unconfined"
> > > name="containers-default-0.50.1" pid=750 comm="apparmor_parser"
> > > [   80.574314] cni-podman0: port 1(vethf6fde03e) entered disabled
> > > state
> > > [   80.575874] device vethf6fde03e left promiscuous mode
> > > [   80.576060] cni-podman0: port 1(vethf6fde03e) entered disabled
> > > state
> > > root@isar:~# echo $?
> > > 0
> > > ```
> > > 
> > > The same under arm64 fails:
> > > 
> > > ```
> > > root@isar:~# docker images
> > > REPOSITORY              TAG       IMAGE ID       CREATED      
> > > SIZE
> > > quay.io/libpod/alpine   3.10.2    915beeae4675   4 years ago  
> > > 5.33MB
> > > root@isar:~# docker run --rm quay.io/libpod/alpine:3.10.2 true
> > > [  407.689016] docker0: port 1(veth81a2857) entered blocking
> > > state
> > > [  407.689231] docker0: port 1(veth81a2857) entered disabled
> > > state
> > > [  407.698637] device veth81a2857 entered promiscuous mode
> > > [  410.003030] eth0: renamed from vethbe8a124
> > > [  410.026357] IPv6: ADDRCONF(NETDEV_CHANGE): veth81a2857: link
> > > becomes
> > > ready
> > > [  410.026727] docker0: port 1(veth81a2857) entered blocking
> > > state
> > > [  410.026872] docker0: port 1(veth81a2857) entered forwarding
> > > state
> > > [  410.767475] docker0: port 1(veth81a2857) entered disabled
> > > state
> > > [  410.788277] vethbe8a124: renamed from eth0
> > > [  410.941958] docker0: port 1(veth81a2857) entered disabled
> > > state
> > > [  410.944534] device veth81a2857 left promiscuous mode
> > > [  410.944676] docker0: port 1(veth81a2857) entered disabled
> > > state
> > > docker: Error response from daemon: failed to create shim task:
> > > OCI
> > > runtime create failed: runc create failed: unable to start
> > > container
> > > process: exec: "true": executable file not found in $PATH:
> > > unknown.
> > > root@isar:~# echo $?
> > > 127
> > > root@isar:~# podman images
> > > REPOSITORY             TAG         IMAGE ID      CREATED     
> > > SIZE
> > > quay.io/libpod/alpine  latest      915beeae4675  4 years ago 
> > > 5.59 MB
> > > root@isar:~# podman run --rm quay.io/libpod/alpine:latest true
> > > [  423.567388] cni-podman0: port 1(veth29135974) entered blocking
> > > state
> > > [  423.567593] cni-podman0: port 1(veth29135974) entered disabled
> > > state
> > > [  423.569719] device veth29135974 entered promiscuous mode
> > > [  423.754420] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes
> > > ready
> > > [  423.754765] IPv6: ADDRCONF(NETDEV_CHANGE): veth29135974: link
> > > becomes ready
> > > [  423.755036] cni-podman0: port 1(veth29135974) entered blocking
> > > state
> > > [  423.755183] cni-podman0: port 1(veth29135974) entered
> > > forwarding
> > > state
> > > [  426.090252] cni-podman0: port 1(veth29135974) entered disabled
> > > state
> > > [  426.098292] device veth29135974 left promiscuous mode
> > > [  426.098455] cni-podman0: port 1(veth29135974) entered disabled
> > > state
> > > Error: runc: runc create failed: unable to start container
> > > process:
> > > exec: "true": executable file not found in $PATH: OCI runtime
> > > attempted
> > > to invoke a command that was not found
> > > root@isar:~# echo $?
> > > 127
> > > ```
> > > 
> > > At first glance this looks like arm64 images are not functional.
> > > Continue debugging.
> > > 
> > 
> > After some debugging I can see that something makes docker prebuilt
> > image inside qemu broken. But removing it from and loading to
> > docker
> > engine again helps:
> > 
> > 
> > ```
> > root@isar:~# docker images
> > REPOSITORY              TAG       IMAGE ID       CREATED       SIZE
> > quay.io/libpod/alpine   3.10.2    915beeae4675   4 years ago  
> > 5.33MB
> > 
> > root@isar:~# docker run --rm quay.io/libpod/alpine:3.10.2 true
> > [  902.770874] docker0: port 1(veth8275b2c) entered blocking state
> > [  902.771066] docker0: port 1(veth8275b2c) entered disabled state
> > [  902.777051] device veth8275b2c entered promiscuous mode
> > [  904.813519] eth0: renamed from veth2f2256f
> > [  904.830269] IPv6: ADDRCONF(NETDEV_CHANGE): veth8275b2c: link
> > becomes
> > ready
> > [  904.830857] docker0: port 1(veth8275b2c) entered blocking state
> > [  904.830997] docker0: port 1(veth8275b2c) entered forwarding
> > state
> > [  904.831407] IPv6: ADDRCONF(NETDEV_CHANGE): docker0: link becomes
> > ready
> > [  905.372753] docker0: port 1(veth8275b2c) entered disabled state
> > [  905.385163] veth2f2256f: renamed from eth0
> > [  905.487707] docker0: port 1(veth8275b2c) entered disabled state
> > [  905.491396] device veth8275b2c left promiscuous mode
> > [  905.491533] docker0: port 1(veth8275b2c) entered disabled state
> > docker: Error response from daemon: failed to create shim task: OCI
> > runtime create failed: runc create failed: unable to start
> > container
> > process: exec: "true": executable file not found in $PATH: unknown.
> > ERRO[0003] error waiting for container: context canceled 
> > 
> > root@isar:~# echo $?
> > 127
> > 
> > root@isar:~# docker image rm 915beeae4675
> > Untagged: quay.io/libpod/alpine:3.10.2
> > Deleted:
> > sha256:915beeae46751fc564998c79e73a1026542e945ca4f73dc841d09ccc6c2c
> > 0672
> > Deleted:
> > sha256:5e0d8111135538b8a86ce5fc969849efce16c455fd016bb3dc53131bcedc
> > 4da5
> > 
> > root@isar:~# docker images
> > REPOSITORY   TAG       IMAGE ID   CREATED   SIZE
> > 
> > root@isar:~# pzstd -c -d /usr/share/prebuilt-docker-
> > img/images/quay.io.libpod.alpine\:3.10.2.zst | docker load
> > /usr/share/prebuilt-docker-
> > img/images/quay.io.libpod.alpine:3.10.2.zst:
> > 5598720 bytes 
> > 5e0d81111355: Loading layer   5.59MB/5.59MB
> > Loaded image: quay.io/libpod/alpine:3.10.2
> > 
> > root@isar:~# docker run --rm quay.io/libpod/alpine:3.10.2 true
> > [ 1023.800568] docker0: port 1(veth3eb45d3) entered blocking state
> > [ 1023.800790] docker0: port 1(veth3eb45d3) entered disabled state
> > [ 1023.805585] device veth3eb45d3 entered promiscuous mode
> > [ 1025.295999] eth0: renamed from veth7e4183e
> > [ 1025.310388] IPv6: ADDRCONF(NETDEV_CHANGE): veth3eb45d3: link
> > becomes
> > ready
> > [ 1025.310681] docker0: port 1(veth3eb45d3) entered blocking state
> > [ 1025.310801] docker0: port 1(veth3eb45d3) entered forwarding
> > state
> > [ 1025.979813] docker0: port 1(veth3eb45d3) entered disabled state
> > [ 1025.990858] veth7e4183e: renamed from eth0
> > [ 1026.087161] docker0: port 1(veth3eb45d3) entered disabled state
> > [ 1026.088367] device veth3eb45d3 left promiscuous mode
> > [ 1026.088471] docker0: port 1(veth3eb45d3) entered disabled state
> > 
> > root@isar:~# echo $?
> > 0
> > ```
> > 
> > This looks strange. Nothing changed (image hash is the same), but
> > the
> > second run works well. After rebooting qemu machine it still works.
> > 
> > Podman prebuilt image looks unaffected - it works from the
> > beginning.
> > 
> 
> Strange, all that used to work. You manually reproduced this as well,
> not only via the testsuite, right? Let me test again locally...
> 
> Jan
> 

For manual tests I used images taken from CI (that failed). As I could
see, the issue in my case was caused by zero-size "/bin/busybox"
somewhere in /var/lib/docker/overlay2/. The file was broken and
reinstalling the container fixed this.

But I guess this was caused by already "spoiled" image that was tested
in CI. When I just built (on a local machine) a new image and didn't
try to run qemu with it (e.g., didn't modify it), manual running docker
image in it worked well.. The busybox binary from alpine container was
OK in that case.

Continue debugging ...

-- 
Best regards,
Uladzimir.



-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/f2ed8eaf27f23533a0de6039f0124291edc4ffc6.camel%40ilbers.de.