[PATCH v5 0/3] add support for debian build profiles

[PATCH v5 0/3] add support for debian build profiles

[Felix Moessbauer]

Changes since v4:

- fix devshell (glitch introduced in v3)

Changes since v3:

Set variables via export shell directive instead of setting via os.environ.
This change is required as os.environ sets the environment of the whole bitbake run, hence affects other tasks as well.
As we have to use export, there is no way for us to check if the variable is already exported.
Also checking the current env-vars in isar_deb_build_profiles does not help as the bb function is called prior to the invocation of the shell function that does the compiling.
I added a statement to the RECIPE-API-CHANGELOG about that.

Changes since v2:

- only provide DEB_BUILD_PROFILES variable (no _CROSS variable)
- add "cross" directly to env variable when cross-compiling
- add DEB_BUILD_OPTIONS bitbake variable
  (profiles like "nocheck" must also add "nocheck" to DEB_BUILD_OPTIONS)
- update API changelog
- use DEB_BUILD_OPTIONS in hello.bb example

Changes since v1:

- fix erronous code removal reported by Vijai
- only use value of DEB_BUILD_PROFILES_CROSS when cross-compiling (no combination of values)
- improve API changelog docs
- rebased onto next
- use DEB_BUILD_PROFILES infrastructure in linux-custom.inc

Felix Moessbauer (3):
  add support for debian build profiles and options
  refactor linux-custom.inc to use ISAR's DEB_BUILD_PROFILES support
  Use DEB_BUILD_OPTIONS bb variable in hello.bb example

 RECIPE-API-CHANGELOG.md                       |  8 ++++++++
 meta-isar/recipes-app/hello/hello.bb          |  8 ++------
 meta/classes/dpkg-base.bbclass                | 19 +++++++++++++++++++
 meta/classes/dpkg.bbclass                     |  7 ++++++-
 .../libubootenv/libubootenv_0.3-3.bb          |  5 +----
 meta/recipes-kernel/linux/linux-custom.inc    | 13 ++++++++-----
 6 files changed, 44 insertions(+), 16 deletions(-)

-- 
2.30.2

[PATCH v5 1/3] add support for debian build profiles and options

[Felix Moessbauer]

This patch adds the bitbake variables DEB_BUILD_PROFILES and DEB_BUILD_OPTIONS.
These are used to define the respective environment variables.

When cross-compiling, "cross" is added to the DEB_BUILD_PROFILES environment variable.

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
---
 RECIPE-API-CHANGELOG.md        |  8 ++++++++
 meta/classes/dpkg-base.bbclass | 19 +++++++++++++++++++
 meta/classes/dpkg.bbclass      |  7 ++++++-
 3 files changed, 33 insertions(+), 1 deletion(-)

diff --git a/RECIPE-API-CHANGELOG.md b/RECIPE-API-CHANGELOG.md
index 3bbb42a9..39a1616b 100644
--- a/RECIPE-API-CHANGELOG.md
+++ b/RECIPE-API-CHANGELOG.md
@@ -314,3 +314,11 @@ The "NAME" used to be rather static and the TAG was always "latest", now the val
 ### Renamed variable CONTAINER_FORMATS to CONTAINER_IMAGE_FORMATS
 
 The meaning remains the same, just the name changed.
+### Introduce debian build profiles
+
+All recipes that inherit from dpkg and dpkg-base can utilize the variables `DEB_BUILD_PROFILES` and `DEB_BUILD_OPTIONS`.
+The bitbake variable defines the respective environment variable which is available in `do_install_builddeps` and `do_dpkg_build`.
+When cross compiling, `cross` is added to the `DEB_BUILD_PROFILES` environment variable.
+Please note, that manually exported versions of the variables are overwritten.
+
+For a list of well-known Debian build profiles and common practices, we refer to Debian's BuildProfileSpec.
diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index cb5ce4a9..202cc115 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -13,6 +13,8 @@ inherit deb-dl-dir
 DEPENDS ?= ""
 
 DEPENDS_append_riscv64 = "${@' crossbuild-essential-riscv64' if d.getVar('ISAR_CROSS_COMPILE', True) == '1' and d.getVar('PN') != 'crossbuild-essential-riscv64' else ''}"
+DEB_BUILD_PROFILES ?= ""
+DEB_BUILD_OPTIONS ?= ""
 
 python do_adjust_git() {
     import subprocess
@@ -201,6 +203,22 @@ dpkg_runbuild() {
     die "This should never be called, overwrite it in your derived class"
 }
 
+def isar_deb_build_profiles(d):
+    deb_build_profiles = d.getVar('DEB_BUILD_PROFILES', True)
+    if d.getVar("ISAR_CROSS_COMPILE") == "1":
+        deb_build_profiles += ' cross'
+    return deb_build_profiles.strip()
+
+def isar_deb_build_options(d):
+    deb_build_options = d.getVar('DEB_BUILD_OPTIONS', True)
+    return deb_build_options.strip()
+
+# use with caution: might contaminate multiple tasks
+def isar_export_build_settings(d):
+    import os
+    os.environ['DEB_BUILD_OPTIONS']  = isar_deb_build_options(d)
+    os.environ['DEB_BUILD_PROFILES'] = isar_deb_build_profiles(d)
+
 python do_dpkg_build() {
     lock = bb.utils.lockfile(d.getVar("REPO_ISAR_DIR") + "/isar.lock",
                              shared=True)
@@ -279,6 +297,7 @@ python do_devshell() {
 
     isar_export_proxies(d)
     isar_export_ccache(d)
+    isar_export_build_settings(d)
 
     buildchroot = d.getVar('BUILDCHROOT_DIR')
     pp_pps = os.path.join(d.getVar('PP'), d.getVar('PPS'))
diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
index 27fe84f4..320102ba 100644
--- a/meta/classes/dpkg.bbclass
+++ b/meta/classes/dpkg.bbclass
@@ -9,10 +9,13 @@ PACKAGE_ARCH ?= "${DISTRO_ARCH}"
 do_install_builddeps() {
     dpkg_do_mounts
     E="${@ isar_export_proxies(d)}"
+    export DEB_BUILD_OPTIONS="${@ isar_deb_build_options(d)}"
+    export DEB_BUILD_PROFILES="${@ isar_deb_build_profiles(d)}"
     distro="${DISTRO}"
     if [ ${ISAR_CROSS_COMPILE} -eq 1 ]; then
-       distro="${HOST_DISTRO}"
+        distro="${HOST_DISTRO}"
     fi
+
     deb_dl_dir_import "${BUILDCHROOT_DIR}" "${distro}"
     sudo -E chroot ${BUILDCHROOT_DIR} /isar/deps.sh \
         ${PP}/${PPS} ${PACKAGE_ARCH} --download-only
@@ -33,6 +36,8 @@ addtask devshell after do_install_builddeps
 dpkg_runbuild() {
     E="${@ isar_export_proxies(d)}"
     E="${@ isar_export_ccache(d)}"
+    export DEB_BUILD_OPTIONS="${@ isar_deb_build_options(d)}"
+    export DEB_BUILD_PROFILES="${@ isar_deb_build_profiles(d)}"
     export PARALLEL_MAKE="${PARALLEL_MAKE}"
     sudo -E chroot --userspec=$( id -u ):$( id -g ) ${BUILDCHROOT_DIR} \
          /isar/build.sh ${PP}/${PPS} ${PACKAGE_ARCH}
-- 
2.30.2

[PATCH v5 2/3] refactor linux-custom.inc to use ISAR's DEB_BUILD_PROFILES support

[Felix Moessbauer]

This patch replaces the manual setup of the DEB_BUILD_PROFILES
environment variable in the linux-custom.inc recipe.
Instead, the recently introduced DEB_BUILD_PROFILES infrastructure
of ISAR is used.

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
---
 meta/recipes-kernel/linux/linux-custom.inc | 13 ++++++++-----
 1 file changed, 8 insertions(+), 5 deletions(-)

diff --git a/meta/recipes-kernel/linux/linux-custom.inc b/meta/recipes-kernel/linux/linux-custom.inc
index ed89aa09..59d42c84 100644
--- a/meta/recipes-kernel/linux/linux-custom.inc
+++ b/meta/recipes-kernel/linux/linux-custom.inc
@@ -117,6 +117,14 @@ def config_fragments(d):
             fragments.append(local)
     return fragments
 
+def get_additional_build_profiles(d):
+    profiles = d.getVar('BASE_DISTRO', True)
+    if d.getVar('KERNEL_LIBC_DEV_DEPLOY', True) != '1':
+        profiles += ' nolibcdev'
+    return profiles
+
+DEB_BUILD_PROFILES += "${@get_additional_build_profiles(d)}"
+
 do_prepare_build_prepend() {
 	# copy meta-data over to source tree
 	rm -rf ${S}/debian
@@ -176,10 +184,5 @@ dpkg_configure_kernel() {
 }
 
 dpkg_runbuild_prepend() {
-	profiles="${BASE_DISTRO}"
-	if [ "${KERNEL_LIBC_DEV_DEPLOY}" != "1" ]; then
-		profiles="${profiles} nolibcdev"
-	fi
-	export DEB_BUILD_PROFILES="${profiles}"
 	dpkg_configure_kernel
 }
-- 
2.30.2

[PATCH v5 3/3] Use DEB_BUILD_OPTIONS bb variable in hello.bb example

[Felix Moessbauer]

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
---
 meta-isar/recipes-app/hello/hello.bb              | 8 ++------
 meta/recipes-bsp/libubootenv/libubootenv_0.3-3.bb | 5 +----
 2 files changed, 3 insertions(+), 10 deletions(-)

diff --git a/meta-isar/recipes-app/hello/hello.bb b/meta-isar/recipes-app/hello/hello.bb
index d6bdf9bb..acf8ed73 100644
--- a/meta-isar/recipes-app/hello/hello.bb
+++ b/meta-isar/recipes-app/hello/hello.bb
@@ -15,6 +15,8 @@ SRC_URI = "apt://${PN}"
 MAINTAINER = "isar-users <isar-users@googlegroups.com>"
 CHANGELOG_V = "<orig-version>+isar"
 
+DEB_BUILD_OPTIONS += "${@ 'nocheck' if d.getVar('ISAR_CROSS_COMPILE') == '1' else '' }"
+
 do_prepare_build() {
 	deb_add_changelog
 	# this seems to be a build dep missing in the upstream control file
@@ -22,9 +24,3 @@ do_prepare_build() {
 		sed -i -e 's/Build-Depends:/Build-Depends: texinfo,/g' ${S}/debian/control
 	fi
 }
-
-dpkg_runbuild_prepend() {
-	if [ ${ISAR_CROSS_COMPILE} -eq 1 ]; then
-		export DEB_BUILD_OPTIONS="nocheck"
-	fi
-}
diff --git a/meta/recipes-bsp/libubootenv/libubootenv_0.3-3.bb b/meta/recipes-bsp/libubootenv/libubootenv_0.3-3.bb
index 68a55003..9af01aa7 100644
--- a/meta/recipes-bsp/libubootenv/libubootenv_0.3-3.bb
+++ b/meta/recipes-bsp/libubootenv/libubootenv_0.3-3.bb
@@ -19,7 +19,4 @@ SRCREV = "a1a3504e5cda1883928a8747a0bedc56afff6910"
 
 S = "${WORKDIR}/git"
 
-
-dpkg_runbuild_prepend() {
-	export DEB_BUILD_OPTIONS="nocheck"
-}
+DEB_BUILD_OPTIONS += "nocheck"
-- 
2.30.2

Re: [PATCH v5 2/3] refactor linux-custom.inc to use ISAR's DEB_BUILD_PROFILES support

[Jan Kiszka]

On 12.01.22 15:39, Felix Moessbauer wrote:
> This patch replaces the manual setup of the DEB_BUILD_PROFILES
> environment variable in the linux-custom.inc recipe.
> Instead, the recently introduced DEB_BUILD_PROFILES infrastructure
> of ISAR is used.
> 
> Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
> ---
>   meta/recipes-kernel/linux/linux-custom.inc | 13 ++++++++-----
>   1 file changed, 8 insertions(+), 5 deletions(-)
> 
> diff --git a/meta/recipes-kernel/linux/linux-custom.inc b/meta/recipes-kernel/linux/linux-custom.inc
> index ed89aa09..59d42c84 100644
> --- a/meta/recipes-kernel/linux/linux-custom.inc
> +++ b/meta/recipes-kernel/linux/linux-custom.inc
> @@ -117,6 +117,14 @@ def config_fragments(d):
>               fragments.append(local)
>       return fragments
>   
> +def get_additional_build_profiles(d):
> +    profiles = d.getVar('BASE_DISTRO', True)
> +    if d.getVar('KERNEL_LIBC_DEV_DEPLOY', True) != '1':
> +        profiles += ' nolibcdev'
> +    return profiles
> +
> +DEB_BUILD_PROFILES += "${@get_additional_build_profiles(d)}"
> +
>   do_prepare_build_prepend() {
>   	# copy meta-data over to source tree
>   	rm -rf ${S}/debian
> @@ -176,10 +184,5 @@ dpkg_configure_kernel() {
>   }
>   
>   dpkg_runbuild_prepend() {
> -	profiles="${BASE_DISTRO}"
> -	if [ "${KERNEL_LIBC_DEV_DEPLOY}" != "1" ]; then
> -		profiles="${profiles} nolibcdev"
> -	fi
> -	export DEB_BUILD_PROFILES="${profiles}"
>   	dpkg_configure_kernel
>   }

After local experiments, it looks like this patch must be folded into 
patch 1 in order to avoid that linux-libc-dev gets built accidentally 
when it shouldn't.

Jan

-- 
Siemens AG, Technology
Competence Center Embedded Linux

RE: [PATCH v5 2/3] refactor linux-custom.inc to use ISAR's DEB_BUILD_PROFILES support

[Moessbauer, Felix]

> -----Original Message-----
> From: Kiszka, Jan (T CED) <jan.kiszka@siemens.com>
> Sent: Thursday, January 27, 2022 10:29 AM
> To: Moessbauer, Felix (T CED SES-DE) <felix.moessbauer@siemens.com>; isar-
> users@googlegroups.com
> Cc: vijaikumar.kanagarajan@gmail.com
> Subject: Re: [PATCH v5 2/3] refactor linux-custom.inc to use ISAR's
> DEB_BUILD_PROFILES support
> 
> On 12.01.22 15:39, Felix Moessbauer wrote:
> > This patch replaces the manual setup of the DEB_BUILD_PROFILES
> > environment variable in the linux-custom.inc recipe.
> > Instead, the recently introduced DEB_BUILD_PROFILES infrastructure of
> > ISAR is used.
> >
> > Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
> > ---
> >   meta/recipes-kernel/linux/linux-custom.inc | 13 ++++++++-----
> >   1 file changed, 8 insertions(+), 5 deletions(-)
> >
> > diff --git a/meta/recipes-kernel/linux/linux-custom.inc
> > b/meta/recipes-kernel/linux/linux-custom.inc
> > index ed89aa09..59d42c84 100644
> > --- a/meta/recipes-kernel/linux/linux-custom.inc
> > +++ b/meta/recipes-kernel/linux/linux-custom.inc
> > @@ -117,6 +117,14 @@ def config_fragments(d):
> >               fragments.append(local)
> >       return fragments
> >
> > +def get_additional_build_profiles(d):
> > +    profiles = d.getVar('BASE_DISTRO', True)
> > +    if d.getVar('KERNEL_LIBC_DEV_DEPLOY', True) != '1':
> > +        profiles += ' nolibcdev'
> > +    return profiles
> > +
> > +DEB_BUILD_PROFILES += "${@get_additional_build_profiles(d)}"
> > +
> >   do_prepare_build_prepend() {
> >   	# copy meta-data over to source tree
> >   	rm -rf ${S}/debian
> > @@ -176,10 +184,5 @@ dpkg_configure_kernel() {
> >   }
> >
> >   dpkg_runbuild_prepend() {
> > -	profiles="${BASE_DISTRO}"
> > -	if [ "${KERNEL_LIBC_DEV_DEPLOY}" != "1" ]; then
> > -		profiles="${profiles} nolibcdev"
> > -	fi
> > -	export DEB_BUILD_PROFILES="${profiles}"
> >   	dpkg_configure_kernel
> >   }
> 
> After local experiments, it looks like this patch must be folded into patch 1 in
> order to avoid that linux-libc-dev gets built accidentally when it shouldn't.

I just sent out a v6 with all commits squashed.

While we are at it:
Without sbuilder, removing packages due to conflicting build dependencies is a recipe for disaster.
The do_install_builddeps and do_dpkg_build steps are not atomic, so another package might remove build deps in between the two tasks.

We should at least scan the do_install_builddeps log and search for "[1-9]+ to remove" and issue a warning if we find such a line.

But don't know when sbuilder will be ready, and if we should put time into this check.
Opinions?

Felix

> 
> Jan
> 
> --
> Siemens AG, Technology
> Competence Center Embedded Linux

Re: [PATCH v5 2/3] refactor linux-custom.inc to use ISAR's DEB_BUILD_PROFILES support

[Jan Kiszka]

On 27.01.22 16:51, Moessbauer, Felix (T CED SES-DE) wrote:
>> -----Original Message-----
>> From: Kiszka, Jan (T CED) <jan.kiszka@siemens.com>
>> Sent: Thursday, January 27, 2022 10:29 AM
>> To: Moessbauer, Felix (T CED SES-DE) <felix.moessbauer@siemens.com>; isar-
>> users@googlegroups.com
>> Cc: vijaikumar.kanagarajan@gmail.com
>> Subject: Re: [PATCH v5 2/3] refactor linux-custom.inc to use ISAR's
>> DEB_BUILD_PROFILES support
>>
>> On 12.01.22 15:39, Felix Moessbauer wrote:
>>> This patch replaces the manual setup of the DEB_BUILD_PROFILES
>>> environment variable in the linux-custom.inc recipe.
>>> Instead, the recently introduced DEB_BUILD_PROFILES infrastructure of
>>> ISAR is used.
>>>
>>> Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
>>> ---
>>>    meta/recipes-kernel/linux/linux-custom.inc | 13 ++++++++-----
>>>    1 file changed, 8 insertions(+), 5 deletions(-)
>>>
>>> diff --git a/meta/recipes-kernel/linux/linux-custom.inc
>>> b/meta/recipes-kernel/linux/linux-custom.inc
>>> index ed89aa09..59d42c84 100644
>>> --- a/meta/recipes-kernel/linux/linux-custom.inc
>>> +++ b/meta/recipes-kernel/linux/linux-custom.inc
>>> @@ -117,6 +117,14 @@ def config_fragments(d):
>>>                fragments.append(local)
>>>        return fragments
>>>
>>> +def get_additional_build_profiles(d):
>>> +    profiles = d.getVar('BASE_DISTRO', True)
>>> +    if d.getVar('KERNEL_LIBC_DEV_DEPLOY', True) != '1':
>>> +        profiles += ' nolibcdev'
>>> +    return profiles
>>> +
>>> +DEB_BUILD_PROFILES += "${@get_additional_build_profiles(d)}"
>>> +
>>>    do_prepare_build_prepend() {
>>>      # copy meta-data over to source tree
>>>      rm -rf ${S}/debian
>>> @@ -176,10 +184,5 @@ dpkg_configure_kernel() {
>>>    }
>>>
>>>    dpkg_runbuild_prepend() {
>>> -   profiles="${BASE_DISTRO}"
>>> -   if [ "${KERNEL_LIBC_DEV_DEPLOY}" != "1" ]; then
>>> -           profiles="${profiles} nolibcdev"
>>> -   fi
>>> -   export DEB_BUILD_PROFILES="${profiles}"
>>>      dpkg_configure_kernel
>>>    }
>>
>> After local experiments, it looks like this patch must be folded into patch 1 in
>> order to avoid that linux-libc-dev gets built accidentally when it shouldn't.
> 
> I just sent out a v6 with all commits squashed.
> 
> While we are at it:
> Without sbuilder, removing packages due to conflicting build dependencies is a recipe for disaster.
> The do_install_builddeps and do_dpkg_build steps are not atomic, so another package might remove build deps in between the two tasks.

They are atomic (do_install_builddeps[lockfiles] vs. read-lock in 
do_dpkg_build), don't worry about that.

But, yes, this can cause troubles if undesired packages are being pushed 
into the buildchroot and/or dependencies were not properly expressed.

> 
> We should at least scan the do_install_builddeps log and search for "[1-9]+ to remove" and issue a warning if we find such a line.

On partial rebuild, you do want to see those messages.

Jan

> 
> But don't know when sbuilder will be ready, and if we should put time into this check.
> Opinions?
> 
> Felix
> 
>>
>> Jan
>>
>> --
>> Siemens AG, Technology
>> Competence Center Embedded Linux

-- 
Siemens AG, Technology
Competence Center Embedded Linux