[PATCH 0/5] More fixes and cleanups

[PATCH 0/5] More fixes and cleanups

[Jan Kiszka]

This fixes two (smaller) issues related to deb-dl-dir, makes
mk-build-dep ready for bullseye and sid, and comes with two cosmetic
cleanups.

With this series applied on top of [1] and kas-isar:next, I'm now able
to build bullseye again - except for mipsel in cross mode because it
lacks crossbuild-essential packages. Might work with sid, didn't test that.

Jan

[1] https://groups.google.com/d/msgid/isar-users/cover.1600788534.git.jan.kiszka%40siemens.com


CC: Henning Schild <henning.schild@siemens.com>

Jan Kiszka (5):
  buildchroot-host: Drop unneeded :native for riscv64 setup
  deb-dl-dir: Fix quoting
  Fix dependencies on isar-apt
  dpkg: Account for changes in mk-build-deps in bullseye
  isar-bootstrap: Align debootstrap calls

 meta/classes/deb-dl-dir.bbclass                   |  4 ++--
 meta/classes/dpkg-base.bbclass                    |  1 -
 meta/classes/rootfs.bbclass                       |  2 +-
 .../isar-bootstrap/isar-bootstrap.inc             | 15 +++++++--------
 .../buildchroot/buildchroot-host.bb               |  6 +++---
 meta/recipes-devtools/buildchroot/files/deps.sh   |  7 +++++--
 6 files changed, 18 insertions(+), 17 deletions(-)

-- 
2.26.2

[PATCH 1/5] buildchroot-host: Drop unneeded :native for riscv64 setup

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

This is not a control file, this is a list passed to apt, thus "native"
is implicit.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 meta/recipes-devtools/buildchroot/buildchroot-host.bb | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/meta/recipes-devtools/buildchroot/buildchroot-host.bb b/meta/recipes-devtools/buildchroot/buildchroot-host.bb
index 8e3a52cf..d8498a56 100644
--- a/meta/recipes-devtools/buildchroot/buildchroot-host.bb
+++ b/meta/recipes-devtools/buildchroot/buildchroot-host.bb
@@ -19,6 +19,6 @@ BUILDCHROOT_PREINSTALL ?= " \
 BUILDCHROOT_PREINSTALL_riscv64 ?= " \
     ${BUILDCHROOT_PREINSTALL_COMMON} \
     libc6:${DISTRO_ARCH} \
-    gcc-riscv64-linux-gnu:native \
-    g++-riscv64-linux-gnu:native \
-    dpkg-cross:native"
+    gcc-riscv64-linux-gnu \
+    g++-riscv64-linux-gnu \
+    dpkg-cross"
-- 
2.26.2

[PATCH 2/5] deb-dl-dir: Fix quoting

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

The outer flock call wraps this in '', so this can't be used inside the
block. The current code failed if some .deb file happened to lie around
in the build folder.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 meta/classes/deb-dl-dir.bbclass | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/meta/classes/deb-dl-dir.bbclass b/meta/classes/deb-dl-dir.bbclass
index e9967036..54c29947 100644
--- a/meta/classes/deb-dl-dir.bbclass
+++ b/meta/classes/deb-dl-dir.bbclass
@@ -14,8 +14,8 @@ deb_dl_dir_import() {
         set -e
         printenv | grep -q BB_VERBOSE_LOGS && set -x
 
-        sudo find "${pc}" -type f -iname '*\.deb' -exec \
-            cp -n --no-preserve=owner -t "${rootfs}"/var/cache/apt/archives/ '{}' +
+        sudo find "${pc}" -type f -iname "*\.deb" -exec \
+            cp -n --no-preserve=owner -t "${rootfs}"/var/cache/apt/archives/ {} +
     '
 }
 
-- 
2.26.2

[PATCH 3/5] Fix dependencies on isar-apt

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

Currently, we fail if only running isar-bootstrap and no other task that
needs isar-apt. That's because of deb_dl_dir_export expecting the repos
already being available (even if empty).

Fix by letting the isar-bootstrap task depend also on
isar-apt:do_cache_config. Drop redundant dependencies, also for
base-apt.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 meta/classes/dpkg-base.bbclass                      | 1 -
 meta/classes/rootfs.bbclass                         | 2 +-
 meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 2 +-
 3 files changed, 2 insertions(+), 3 deletions(-)

diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index 08880d7d..7e12ab0a 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -180,7 +180,6 @@ do_deploy_deb() {
 
 addtask deploy_deb after do_dpkg_build before do_build
 do_deploy_deb[lockfiles] = "${REPO_ISAR_DIR}/isar.lock"
-do_deploy_deb[depends] = "isar-apt:do_cache_config"
 do_deploy_deb[dirs] = "${S}"
 
 python do_devshell() {
diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass
index afec1cbc..00254ef5 100644
--- a/meta/classes/rootfs.bbclass
+++ b/meta/classes/rootfs.bbclass
@@ -151,7 +151,7 @@ rootfs_install_pkgs_install() {
 
 do_rootfs_install[root_cleandirs] = "${ROOTFSDIR}"
 do_rootfs_install[vardeps] += "${ROOTFS_CONFIGURE_COMMAND} ${ROOTFS_INSTALL_COMMAND}"
-do_rootfs_install[depends] = "isar-bootstrap-${@'target' if d.getVar('ROOTFS_ARCH') == d.getVar('DISTRO_ARCH') else 'host'}:do_build isar-apt:do_cache_config base-apt:do_cache"
+do_rootfs_install[depends] = "isar-bootstrap-${@'target' if d.getVar('ROOTFS_ARCH') == d.getVar('DISTRO_ARCH') else 'host'}:do_build"
 do_rootfs_install[deptask] = "do_deploy_deb"
 python do_rootfs_install() {
     configure_cmds = (d.getVar("ROOTFS_CONFIGURE_COMMAND", True) or "").split()
diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
index 1c5dba74..b7dc1ab7 100644
--- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
+++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
@@ -224,7 +224,7 @@ def get_host_release():
 
 do_bootstrap[vardeps] += "DISTRO_APT_PREMIRRORS ISAR_ENABLE_COMPAT_ARCH"
 do_bootstrap[dirs] = "${DEPLOY_DIR_BOOTSTRAP}"
-do_bootstrap[depends] = "base-apt:do_cache"
+do_bootstrap[depends] = "base-apt:do_cache isar-apt:do_cache_config"
 
 isar_bootstrap() {
     IS_HOST=""
-- 
2.26.2

[PATCH 4/5] dpkg: Account for changes in mk-build-deps in bullseye

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

In bullseye, mk-build-deps now emits two log files. When rebuilding
debian packages, those files end up in ${S} and will be recognized by
the package build as deviation from the source file. Avoid this failure
by stepping out of ${S} before calling mk-build-deps.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 meta/recipes-devtools/buildchroot/files/deps.sh | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/meta/recipes-devtools/buildchroot/files/deps.sh b/meta/recipes-devtools/buildchroot/files/deps.sh
index 93bc9cf5..1d617bc8 100644
--- a/meta/recipes-devtools/buildchroot/files/deps.sh
+++ b/meta/recipes-devtools/buildchroot/files/deps.sh
@@ -33,17 +33,20 @@ if ! grep "^Architecture:" debian/control | grep -qv "all"; then
     set_arch=""
 fi
 
+control_file=$(pwd)/debian/control
+cd ..
+
 # Install all build deps
 if [ "$3" = "--download-only" ]; then
     # this will not return 0 even when it worked
-    mk-build-deps $set_arch -t "${install_cmd}" -i -r debian/control &> \
+    mk-build-deps $set_arch -t "${install_cmd}" -i -r $control_file &> \
         mk-build-deps.output || true
     cat mk-build-deps.output
     # we assume success when we find this
     grep "mk-build-deps: Unable to install all build-dep packages" mk-build-deps.output
     rm -f mk-build-deps.output
 else
-    mk-build-deps $set_arch -t "${install_cmd}" -i -r debian/control
+    mk-build-deps $set_arch -t "${install_cmd}" -i -r $control_file
 
     # Upgrade any already installed packages in case we are partially rebuilding
     apt-get upgrade -y --allow-downgrades
-- 
2.26.2

[PATCH 5/5] isar-bootstrap: Align debootstrap calls

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

...both indention-wise as well as regarding quoting.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 13 ++++++-------
 1 file changed, 6 insertions(+), 7 deletions(-)

diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
index b7dc1ab7..0b3ad452 100644
--- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
+++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
@@ -265,14 +265,13 @@ isar_bootstrap() {
                                "${@get_distro_suite(d, True)}" \
                                "${ROOTFSDIR}" \
                                "${@get_distro_source(d, True)}"
-
             else
-                 "${DEBOOTSTRAP}" $debootstrap_args \
-                                  --arch="${DISTRO_ARCH}" \
-                                  ${@get_distro_components_argument(d, False)} \
-                                  "${@get_distro_suite(d, False)}" \
-                                  "${ROOTFSDIR}" \
-                                  "${@get_distro_source(d, False)}"
+                ${DEBOOTSTRAP} $debootstrap_args \
+                               --arch="${DISTRO_ARCH}" \
+                               ${@get_distro_components_argument(d, False)} \
+                               "${@get_distro_suite(d, False)}" \
+                               "${ROOTFSDIR}" \
+                               "${@get_distro_source(d, False)}"
             fi
 
             # Install apt config
-- 
2.26.2

Re: [PATCH 2/5] deb-dl-dir: Fix quoting

[Henning Schild]

Good catch!

On Thu, 24 Sep 2020 17:49:48 +0200
"[ext] Jan Kiszka" <jan.kiszka@siemens.com> wrote:

> From: Jan Kiszka <jan.kiszka@siemens.com>
> 
> The outer flock call wraps this in '', so this can't be used inside
> the block. The current code failed if some .deb file happened to lie
> around in the build folder.
> 
> Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
> ---
>  meta/classes/deb-dl-dir.bbclass | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
> 
> diff --git a/meta/classes/deb-dl-dir.bbclass
> b/meta/classes/deb-dl-dir.bbclass index e9967036..54c29947 100644
> --- a/meta/classes/deb-dl-dir.bbclass
> +++ b/meta/classes/deb-dl-dir.bbclass
> @@ -14,8 +14,8 @@ deb_dl_dir_import() {
>          set -e
>          printenv | grep -q BB_VERBOSE_LOGS && set -x
>  
> -        sudo find "${pc}" -type f -iname '*\.deb' -exec \
> -            cp -n --no-preserve=owner -t
> "${rootfs}"/var/cache/apt/archives/ '{}' +
> +        sudo find "${pc}" -type f -iname "*\.deb" -exec \
> +            cp -n --no-preserve=owner -t
> "${rootfs}"/var/cache/apt/archives/ {} + '
>  }
>  

Re: [PATCH 4/5] dpkg: Account for changes in mk-build-deps in bullseye

[Henning Schild]

On Thu, 24 Sep 2020 17:49:50 +0200
"[ext] Jan Kiszka" <jan.kiszka@siemens.com> wrote:

> From: Jan Kiszka <jan.kiszka@siemens.com>
> 
> In bullseye, mk-build-deps now emits two log files. When rebuilding
> debian packages, those files end up in ${S} and will be recognized by
> the package build as deviation from the source file. Avoid this
> failure by stepping out of ${S} before calling mk-build-deps.
> 
> Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
> ---
>  meta/recipes-devtools/buildchroot/files/deps.sh | 7 +++++--
>  1 file changed, 5 insertions(+), 2 deletions(-)
> 
> diff --git a/meta/recipes-devtools/buildchroot/files/deps.sh
> b/meta/recipes-devtools/buildchroot/files/deps.sh index
> 93bc9cf5..1d617bc8 100644 ---
> a/meta/recipes-devtools/buildchroot/files/deps.sh +++
> b/meta/recipes-devtools/buildchroot/files/deps.sh @@ -33,17 +33,20 @@
> if ! grep "^Architecture:" debian/control | grep -qv "all"; then
> set_arch="" fi
>  
> +control_file=$(pwd)/debian/control
> +cd ..

Would it be a good idea to make this absolute instead of relative? Or
maybe we can make mk-build-deps place those temporary files somewhere
else?

Changing the working directory somewhere inbetween can be pretty
confusing when extending later.

Henning

> +
>  # Install all build deps
>  if [ "$3" = "--download-only" ]; then
>      # this will not return 0 even when it worked
> -    mk-build-deps $set_arch -t "${install_cmd}" -i -r debian/control
> &> \
> +    mk-build-deps $set_arch -t "${install_cmd}" -i -r $control_file
> &> \ mk-build-deps.output || true
>      cat mk-build-deps.output
>      # we assume success when we find this
>      grep "mk-build-deps: Unable to install all build-dep packages"
> mk-build-deps.output rm -f mk-build-deps.output
>  else
> -    mk-build-deps $set_arch -t "${install_cmd}" -i -r debian/control
> +    mk-build-deps $set_arch -t "${install_cmd}" -i -r $control_file
>  
>      # Upgrade any already installed packages in case we are
> partially rebuilding apt-get upgrade -y --allow-downgrades

Re: [PATCH 4/5] dpkg: Account for changes in mk-build-deps in bullseye

[Jan Kiszka]

On 24.09.20 23:32, Henning Schild wrote:
> On Thu, 24 Sep 2020 17:49:50 +0200
> "[ext] Jan Kiszka" <jan.kiszka@siemens.com> wrote:
> 
>> From: Jan Kiszka <jan.kiszka@siemens.com>
>>
>> In bullseye, mk-build-deps now emits two log files. When rebuilding
>> debian packages, those files end up in ${S} and will be recognized by
>> the package build as deviation from the source file. Avoid this
>> failure by stepping out of ${S} before calling mk-build-deps.
>>
>> Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
>> ---
>>   meta/recipes-devtools/buildchroot/files/deps.sh | 7 +++++--
>>   1 file changed, 5 insertions(+), 2 deletions(-)
>>
>> diff --git a/meta/recipes-devtools/buildchroot/files/deps.sh
>> b/meta/recipes-devtools/buildchroot/files/deps.sh index
>> 93bc9cf5..1d617bc8 100644 ---
>> a/meta/recipes-devtools/buildchroot/files/deps.sh +++
>> b/meta/recipes-devtools/buildchroot/files/deps.sh @@ -33,17 +33,20 @@
>> if ! grep "^Architecture:" debian/control | grep -qv "all"; then
>> set_arch="" fi
>>   
>> +control_file=$(pwd)/debian/control
>> +cd ..
> 
> Would it be a good idea to make this absolute instead of relative? Or
> maybe we can make mk-build-deps place those temporary files somewhere
> else?

The latter was my first thought as well (or some switch to suppress it), 
but I found nothing in the mk-build-deps manual.

The former depends on a good suggestion for a path. Would possibly mean 
refactoring the interface of this script, passing in ${PP} and ${PPS} 
separately. Would that be better?

Jan

> 
> Changing the working directory somewhere inbetween can be pretty
> confusing when extending later.
> 
> Henning
> 
>> +
>>   # Install all build deps
>>   if [ "$3" = "--download-only" ]; then
>>       # this will not return 0 even when it worked
>> -    mk-build-deps $set_arch -t "${install_cmd}" -i -r debian/control
>> &> \
>> +    mk-build-deps $set_arch -t "${install_cmd}" -i -r $control_file
>> &> \ mk-build-deps.output || true
>>       cat mk-build-deps.output
>>       # we assume success when we find this
>>       grep "mk-build-deps: Unable to install all build-dep packages"
>> mk-build-deps.output rm -f mk-build-deps.output
>>   else
>> -    mk-build-deps $set_arch -t "${install_cmd}" -i -r debian/control
>> +    mk-build-deps $set_arch -t "${install_cmd}" -i -r $control_file
>>   
>>       # Upgrade any already installed packages in case we are
>> partially rebuilding apt-get upgrade -y --allow-downgrades
> 


-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

Re: [PATCH 4/5] dpkg: Account for changes in mk-build-deps in bullseye

[Henning Schild]

On Fri, 25 Sep 2020 08:21:25 +0200
Jan Kiszka <jan.kiszka@siemens.com> wrote:

> On 24.09.20 23:32, Henning Schild wrote:
> > On Thu, 24 Sep 2020 17:49:50 +0200
> > "[ext] Jan Kiszka" <jan.kiszka@siemens.com> wrote:
> >   
> >> From: Jan Kiszka <jan.kiszka@siemens.com>
> >>
> >> In bullseye, mk-build-deps now emits two log files. When rebuilding
> >> debian packages, those files end up in ${S} and will be recognized
> >> by the package build as deviation from the source file. Avoid this
> >> failure by stepping out of ${S} before calling mk-build-deps.
> >>
> >> Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
> >> ---
> >>   meta/recipes-devtools/buildchroot/files/deps.sh | 7 +++++--
> >>   1 file changed, 5 insertions(+), 2 deletions(-)
> >>
> >> diff --git a/meta/recipes-devtools/buildchroot/files/deps.sh
> >> b/meta/recipes-devtools/buildchroot/files/deps.sh index
> >> 93bc9cf5..1d617bc8 100644 ---
> >> a/meta/recipes-devtools/buildchroot/files/deps.sh +++
> >> b/meta/recipes-devtools/buildchroot/files/deps.sh @@ -33,17 +33,20
> >> @@ if ! grep "^Architecture:" debian/control | grep -qv "all"; then
> >> set_arch="" fi
> >>   
> >> +control_file=$(pwd)/debian/control
> >> +cd ..  
> > 
> > Would it be a good idea to make this absolute instead of relative?
> > Or maybe we can make mk-build-deps place those temporary files
> > somewhere else?  
> 
> The latter was my first thought as well (or some switch to suppress
> it), but I found nothing in the mk-build-deps manual.

Well whatever these logs say, we probably want them somewhere under
WORKDIR for debugging outside of the chroot.

> The former depends on a good suggestion for a path. Would possibly
> mean refactoring the interface of this script, passing in ${PP} and
> ${PPS} separately. Would that be better?

I do not feel strong about it and it can probably stay the way it is.

We are covered by conventions and from S (PPS) .. means WORKDIR (PP).

Henning

> Jan
> 
> > 
> > Changing the working directory somewhere inbetween can be pretty
> > confusing when extending later.
> > 
> > Henning
> >   
> >> +
> >>   # Install all build deps
> >>   if [ "$3" = "--download-only" ]; then
> >>       # this will not return 0 even when it worked
> >> -    mk-build-deps $set_arch -t "${install_cmd}" -i -r
> >> debian/control &> \  
> >> +    mk-build-deps $set_arch -t "${install_cmd}" -i -r
> >> $control_file &> \ mk-build-deps.output || true  
> >>       cat mk-build-deps.output
> >>       # we assume success when we find this
> >>       grep "mk-build-deps: Unable to install all build-dep
> >> packages" mk-build-deps.output rm -f mk-build-deps.output
> >>   else
> >> -    mk-build-deps $set_arch -t "${install_cmd}" -i -r
> >> debian/control
> >> +    mk-build-deps $set_arch -t "${install_cmd}" -i -r
> >> $control_file 
> >>       # Upgrade any already installed packages in case we are
> >> partially rebuilding apt-get upgrade -y --allow-downgrades  
> >   
> 
>