[RFC 1/1] speedup localepurge task by emulating less

[RFC 1/1] speedup localepurge task by emulating less

['Felix Moessbauer' via isar-users]

The download and dpkg status tasks do not need to be emulated, as these
properly support running without a chroot and for a foreign architecture.

The install task itself does support running without a chroot, however
not for a foreign architecture. By that, we keep the emulation of the
install task for now.

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
---
 meta/classes/image-locales-extension.bbclass | 8 +++++---
 meta/classes/rootfs.bbclass                  | 1 +
 2 files changed, 6 insertions(+), 3 deletions(-)

diff --git a/meta/classes/image-locales-extension.bbclass b/meta/classes/image-locales-extension.bbclass
index 9149d643..787719cd 100644
--- a/meta/classes/image-locales-extension.bbclass
+++ b/meta/classes/image-locales-extension.bbclass
@@ -29,8 +29,10 @@ ROOTFS_INSTALL_COMMAND_BEFORE_EXPORT += "image_install_localepurge_download"
 image_install_localepurge_download[weight] = "40"
 image_install_localepurge_download[network] = "${TASK_USE_NETWORK_AND_SUDO}"
 image_install_localepurge_download() {
-    sudo -E chroot '${ROOTFSDIR}' \
-        /usr/bin/apt-get ${ROOTFS_APT_ARGS} --download-only localepurge
+    sudo -E apt-get \
+        ${ROOTFS_APT_ARGS} \
+        ${NO_CHROOT_APT_ARGS} \
+        --download-only localepurge
 }
 
 ROOTFS_INSTALL_COMMAND += "image_install_localepurge_install"
@@ -63,7 +65,7 @@ __EOF__
     sudo -E -s <<'EOSUDO'
         set -e
         localepurge_state='i'
-        if chroot '${ROOTFSDIR}' dpkg -s localepurge 2>/dev/null >&2
+        if dpkg --root '${ROOTFSDIR}' -s localepurge 2>/dev/null >&2
         then
             echo 'localepurge was installed (leaving it installed later)'
         else
diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass
index 4de684a4..1098a100 100644
--- a/meta/classes/rootfs.bbclass
+++ b/meta/classes/rootfs.bbclass
@@ -20,6 +20,7 @@ ROOTFS_BASE_DISTRO ?= "${BASE_DISTRO}"
 ROOTFS_FEATURES ?= ""
 
 ROOTFS_APT_ARGS="install --yes -o Debug::pkgProblemResolver=yes"
+NO_CHROOT_APT_ARGS ?= "-o APT::Architecture=${DISTRO_ARCH} -o Dir=${ROOTFSDIR}"
 
 ROOTFS_CLEAN_FILES="/etc/hostname /etc/resolv.conf"
 
-- 
2.50.1

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/20250821144519.298099-1-felix.moessbauer%40siemens.com.

Re: [RFC 1/1] speedup localepurge task by emulating less

['Jan Kiszka' via isar-users]

On 21.08.25 16:45, Felix Moessbauer wrote:
> The download and dpkg status tasks do not need to be emulated, as these
> properly support running without a chroot and for a foreign architecture.
> 
> The install task itself does support running without a chroot, however
> not for a foreign architecture. By that, we keep the emulation of the
> install task for now.

Can we always parse with older dpkg newer installations? Or does this
silently assume that builder-release == target-release then?

To avoid emulation, I would use the dpkg from the host-chroot instead.

Jan

> 
> Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
> ---
>  meta/classes/image-locales-extension.bbclass | 8 +++++---
>  meta/classes/rootfs.bbclass                  | 1 +
>  2 files changed, 6 insertions(+), 3 deletions(-)
> 
> diff --git a/meta/classes/image-locales-extension.bbclass b/meta/classes/image-locales-extension.bbclass
> index 9149d643..787719cd 100644
> --- a/meta/classes/image-locales-extension.bbclass
> +++ b/meta/classes/image-locales-extension.bbclass
> @@ -29,8 +29,10 @@ ROOTFS_INSTALL_COMMAND_BEFORE_EXPORT += "image_install_localepurge_download"
>  image_install_localepurge_download[weight] = "40"
>  image_install_localepurge_download[network] = "${TASK_USE_NETWORK_AND_SUDO}"
>  image_install_localepurge_download() {
> -    sudo -E chroot '${ROOTFSDIR}' \
> -        /usr/bin/apt-get ${ROOTFS_APT_ARGS} --download-only localepurge
> +    sudo -E apt-get \
> +        ${ROOTFS_APT_ARGS} \
> +        ${NO_CHROOT_APT_ARGS} \
> +        --download-only localepurge
>  }
>  
>  ROOTFS_INSTALL_COMMAND += "image_install_localepurge_install"
> @@ -63,7 +65,7 @@ __EOF__
>      sudo -E -s <<'EOSUDO'
>          set -e
>          localepurge_state='i'
> -        if chroot '${ROOTFSDIR}' dpkg -s localepurge 2>/dev/null >&2
> +        if dpkg --root '${ROOTFSDIR}' -s localepurge 2>/dev/null >&2
>          then
>              echo 'localepurge was installed (leaving it installed later)'
>          else
> diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass
> index 4de684a4..1098a100 100644
> --- a/meta/classes/rootfs.bbclass
> +++ b/meta/classes/rootfs.bbclass
> @@ -20,6 +20,7 @@ ROOTFS_BASE_DISTRO ?= "${BASE_DISTRO}"
>  ROOTFS_FEATURES ?= ""
>  
>  ROOTFS_APT_ARGS="install --yes -o Debug::pkgProblemResolver=yes"
> +NO_CHROOT_APT_ARGS ?= "-o APT::Architecture=${DISTRO_ARCH} -o Dir=${ROOTFSDIR}"
>  
>  ROOTFS_CLEAN_FILES="/etc/hostname /etc/resolv.conf"
>  

-- 
Siemens AG, Foundational Technologies
Linux Expert Center

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/b8e4485d-9f36-4b3e-9566-b2d7685ee5d7%40siemens.com.

Re: [RFC 1/1] speedup localepurge task by emulating less

['Jan Kiszka' via isar-users]

On 21.08.25 17:24, 'Jan Kiszka' via isar-users wrote:
> On 21.08.25 16:45, Felix Moessbauer wrote:
>> The download and dpkg status tasks do not need to be emulated, as these
>> properly support running without a chroot and for a foreign architecture.
>>
>> The install task itself does support running without a chroot, however
>> not for a foreign architecture. By that, we keep the emulation of the
>> install task for now.
> 
> Can we always parse with older dpkg newer installations? Or does this
> silently assume that builder-release == target-release then?
> 
> To avoid emulation, I would use the dpkg from the host-chroot instead.
> 

BTW, with this pattern adjusted, we could also apply it
rootfs_install_pkgs_download and deb-dl-dir.bbclass.

Jan

-- 
Siemens AG, Foundational Technologies
Linux Expert Center

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/02cd8640-569d-4eb6-aa8d-57c4f3212628%40siemens.com.

Re: [RFC 1/1] speedup localepurge task by emulating less

['MOESSBAUER, Felix' via isar-users]

On Mon, 2025-09-15 at 08:20 +0200, Jan Kiszka wrote:
> On 21.08.25 17:24, 'Jan Kiszka' via isar-users wrote:
> > On 21.08.25 16:45, Felix Moessbauer wrote:
> > > The download and dpkg status tasks do not need to be emulated, as these
> > > properly support running without a chroot and for a foreign architecture.
> > > 
> > > The install task itself does support running without a chroot, however
> > > not for a foreign architecture. By that, we keep the emulation of the
> > > install task for now.
> > 
> > Can we always parse with older dpkg newer installations? Or does this
> > silently assume that builder-release == target-release then?
> > 
> > To avoid emulation, I would use the dpkg from the host-chroot instead.
> > 
> 
> BTW, with this pattern adjusted, we could also apply it
> rootfs_install_pkgs_download and deb-dl-dir.bbclass.

This is already taken care of by the series "rootfs: introduce wrapper
to run native commands against a rootfs" by Cedric. I anyways propose
to merge his series first and then move on here.

On 20.08.2025 I already sent a ping, but did not get any response.

Felix

> 
> Jan
> 
> -- 
> Siemens AG, Foundational Technologies
> Linux Expert Center

-- 
Siemens AG
Linux Expert Center
Friedrich-Ludwig-Bauer-Str. 3
85748 Garching, Germany

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/f17dbd52987c0ea86a905065c044c070aba16e0f.camel%40siemens.com.

Re: [RFC 1/1] speedup localepurge task by emulating less

['Jan Kiszka' via isar-users]

On 15.09.25 10:23, Moessbauer, Felix (FT RPD CED OES-DE) wrote:
> On Mon, 2025-09-15 at 08:20 +0200, Jan Kiszka wrote:
>> On 21.08.25 17:24, 'Jan Kiszka' via isar-users wrote:
>>> On 21.08.25 16:45, Felix Moessbauer wrote:
>>>> The download and dpkg status tasks do not need to be emulated, as these
>>>> properly support running without a chroot and for a foreign architecture.
>>>>
>>>> The install task itself does support running without a chroot, however
>>>> not for a foreign architecture. By that, we keep the emulation of the
>>>> install task for now.
>>>
>>> Can we always parse with older dpkg newer installations? Or does this
>>> silently assume that builder-release == target-release then?
>>>
>>> To avoid emulation, I would use the dpkg from the host-chroot instead.
>>>
>>
>> BTW, with this pattern adjusted, we could also apply it
>> rootfs_install_pkgs_download and deb-dl-dir.bbclass.
> 
> This is already taken care of by the series "rootfs: introduce wrapper
> to run native commands against a rootfs" by Cedric. I anyways propose
> to merge his series first and then move on here.
> 

...for which the very same concern applies.

Jan

-- 
Siemens AG, Foundational Technologies
Linux Expert Center

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/7dcf3029-4be2-46fd-a303-48a88ea34298%40siemens.com.

Re: [RFC 1/1] speedup localepurge task by emulating less

[Zhihang Wei]

Hi,

this RFC patch caused "citest.py:DevTest.test_dev" to fail on fast CI.
"[PATCH v4 0/4] non-privileged commands in chroot" has already been applied
to next.

Log follows:

[stdlog] 2025-11-07 11:58:05,173 avocado.test cibuilder L0320 INFO | 
NOTE: Running task 69 of 84 
(mc:qemuarm-bookworm:/build/isar_wzh_3_fast/23/meta-isar/recipes-core/images/isar-image-base.bb:do_image_ext4)
[stdlog] 2025-11-07 11:58:05,180 avocado.test cibuilder L0320 INFO | 
NOTE: recipe isar-image-base-sdk-1.0-r0: task do_rootfs_install: Started
[stdlog] 2025-11-07 11:58:05,216 avocado.test cibuilder L0320 INFO | 
NOTE: recipe isar-image-base-1.0-r0: task do_image_ext4: Started
[stdlog] 2025-11-07 11:58:07,271 avocado.app cibuilder L0322 ERROR| 
ERROR: mc:qemuarm-bookworm:isar-image-base-sdk-1.0-r0 do_rootfs_install: 
ExecutionError('/build/isar_wzh_3_fast/23/build/tmp/work/debian-bookworm-armhf/isar-image-base-sdk-qemuarm/1.0-r0/temp/run.image_install_localepurge_download.2952562', 
100, None, None)
[stdlog] 2025-11-07 11:58:07,273 avocado.app cibuilder L0322 ERROR| 
ERROR: Logfile of failure stored in: 
/build/isar_wzh_3_fast/23/build/tmp/work/debian-bookworm-armhf/isar-image-base-sdk-qemuarm/1.0-r0/temp/log.do_rootfs_install.2952562
[stdlog] 2025-11-07 11:58:07,275 avocado.app cibuilder L0322 ERROR| 
ERROR: Task 
(mc:qemuarm-bookworm:virtual:sdk:/build/isar_wzh_3_fast/23/meta-isar/recipes-core/images/isar-image-base.bb:do_rootfs_install) 
failed with exit code '1'

log.do_rootfs_install.2952562:
......
The following packages have unmet dependencies:
  adduser : Depends: passwd but it is not going to be installed
  libfile-find-rule-perl : Depends: perl:any
  libpam-runtime : Depends: libpam-modules (>= 1.0.1-6) but it is not 
going to be installed
  localepurge : Depends: perl:any
                Depends: ucf but it is not going to be installed
                Depends: procps but it is not going to be installed
  locales : Depends: libc-bin (> 2.36) but it is not going to be installed
  perl-modules-5.36 : PreDepends: dpkg (>= 1.17.17) but it is not going 
to be installed
                      Depends: perl-base (>= 5.36.0-1) but it is not 
going to be installed
                      Recommends: perl (>= 5.36.0-1) but it is not going 
to be installed
  usrmerge : Depends: perl:any
E: Unmet dependencies. Try 'apt --fix-broken install' with no packages 
(or specify a solution).
WARNING: exit code 100 from a shell command.
DEBUG: Executing shell function rootfs_do_umounts
DEBUG: Shell function rootfs_do_umounts finished
DEBUG: Python function do_rootfs_install finished

To redo the test using avocado:
1. Have a clean clone of isar, checkout to branch next and apply your 
patches:
$ git clone -b next https://github.com/ilbers/isar.git
$ cd isar
$ git am /path-to/0001-my-contribution-to-isar.patch
2. Disable several unrelated targets to make error appears faster, by 
applying
the following diff to "testsuite/citest.py":

diff --git a/testsuite/citest.py b/testsuite/citest.py
index a1214e9c..1207ea28 100755
--- a/testsuite/citest.py
+++ b/testsuite/citest.py
@@ -66,10 +66,8 @@ class DevTest(CIBaseTest):

      def test_dev(self):
          targets = [
-            'mc:qemuamd64-bookworm:isar-image-ci',
              'mc:qemuarm-bookworm:isar-image-base',
  'mc:qemuarm-bookworm:isar-image-base:do_populate_sdk',
-            'mc:qemuarm64-bookworm:isar-image-base',
          ]

          self.init()

3.Run kas shell, setup CI prerequisites (avocado, qemu) and cleanup:
$ ./kas/kas-container shell kas/isar.yaml --command \
     "rm -rf /work/build/conf && /work/scripts/ci_setup.sh"
4.Run the failed test in fast:
$ cd /work/testsuite
$ avocado run citest.py:DevTest.test_dev$

Best regards,
Zhihang

On 8/21/25 16:45, 'Felix Moessbauer' via isar-users wrote:
> The download and dpkg status tasks do not need to be emulated, as these
> properly support running without a chroot and for a foreign architecture.
>
> The install task itself does support running without a chroot, however
> not for a foreign architecture. By that, we keep the emulation of the
> install task for now.
>
> Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
> ---
>   meta/classes/image-locales-extension.bbclass | 8 +++++---
>   meta/classes/rootfs.bbclass                  | 1 +
>   2 files changed, 6 insertions(+), 3 deletions(-)
>
> diff --git a/meta/classes/image-locales-extension.bbclass b/meta/classes/image-locales-extension.bbclass
> index 9149d643..787719cd 100644
> --- a/meta/classes/image-locales-extension.bbclass
> +++ b/meta/classes/image-locales-extension.bbclass
> @@ -29,8 +29,10 @@ ROOTFS_INSTALL_COMMAND_BEFORE_EXPORT += "image_install_localepurge_download"
>   image_install_localepurge_download[weight] = "40"
>   image_install_localepurge_download[network] = "${TASK_USE_NETWORK_AND_SUDO}"
>   image_install_localepurge_download() {
> -    sudo -E chroot '${ROOTFSDIR}' \
> -        /usr/bin/apt-get ${ROOTFS_APT_ARGS} --download-only localepurge
> +    sudo -E apt-get \
> +        ${ROOTFS_APT_ARGS} \
> +        ${NO_CHROOT_APT_ARGS} \
> +        --download-only localepurge
>   }
>   
>   ROOTFS_INSTALL_COMMAND += "image_install_localepurge_install"
> @@ -63,7 +65,7 @@ __EOF__
>       sudo -E -s <<'EOSUDO'
>           set -e
>           localepurge_state='i'
> -        if chroot '${ROOTFSDIR}' dpkg -s localepurge 2>/dev/null >&2
> +        if dpkg --root '${ROOTFSDIR}' -s localepurge 2>/dev/null >&2
>           then
>               echo 'localepurge was installed (leaving it installed later)'
>           else
> diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass
> index 4de684a4..1098a100 100644
> --- a/meta/classes/rootfs.bbclass
> +++ b/meta/classes/rootfs.bbclass
> @@ -20,6 +20,7 @@ ROOTFS_BASE_DISTRO ?= "${BASE_DISTRO}"
>   ROOTFS_FEATURES ?= ""
>   
>   ROOTFS_APT_ARGS="install --yes -o Debug::pkgProblemResolver=yes"
> +NO_CHROOT_APT_ARGS ?= "-o APT::Architecture=${DISTRO_ARCH} -o Dir=${ROOTFSDIR}"
>   
>   ROOTFS_CLEAN_FILES="/etc/hostname /etc/resolv.conf"
>   

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/821d6e41-34e4-4eb2-899a-416d634d944a%40ilbers.de.