[PATCH 0/5] Separate prebuild activities

[PATCH 0/5] Separate prebuild activities

[Anton Mikanovich]

This patchset introduce three new pre-build tasks from the code was already
done during the build task:
 do_dpkg_source
 do_deploy_source
 do_local_isarapt
This allows to execute tasks more efficient. On simple targets like qemuamd64
the speed up of the build is around 10 second, but on complex downstream layers
this effect can be more visible.
We also introduce source packages storage in isar-apt repository which allows
obtaining package Build-Depends and Provides information in Debian way later.

Anton Mikanovich (5):
  repository: Add source package remove function
  dpkg: Limit dsc searching by workdir
  meta: Implement two stage build
  dpkg-base: Copy isar_apt to workdir in separate task
  RECIPE-API-CHANGELOG.md: Note tasks separation

 RECIPE-API-CHANGELOG.md                    | 14 +++++++++
 meta/classes/dpkg-base.bbclass             | 15 +++++-----
 meta/classes/dpkg-gbp.bbclass              |  2 +-
 meta/classes/dpkg-source.bbclass           | 33 ++++++++++++++++++++++
 meta/classes/dpkg.bbclass                  |  6 ++--
 meta/classes/repository.bbclass            | 14 +++++++++
 meta/classes/rootfs.bbclass                |  2 ++
 meta/recipes-kernel/linux/linux-custom.inc |  2 +-
 8 files changed, 74 insertions(+), 14 deletions(-)
 create mode 100644 meta/classes/dpkg-source.bbclass

-- 
2.34.1

[PATCH 1/5] repository: Add source package remove function

[Anton Mikanovich]

There was source package adding functionality in repository bbclass but
no way to remove this package after adding.
So implement also repo_del_srcpackage call.

Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
 meta/classes/repository.bbclass | 14 ++++++++++++++
 1 file changed, 14 insertions(+)

diff --git a/meta/classes/repository.bbclass b/meta/classes/repository.bbclass
index ac395dbe..a61e5dcd 100644
--- a/meta/classes/repository.bbclass
+++ b/meta/classes/repository.bbclass
@@ -61,6 +61,20 @@ repo_add_packages() {
         "$@"
 }
 
+repo_del_srcpackage() {
+    local dir="$1"
+    local dbdir="$2"
+    local codename="$3"
+    local packagename="$4"
+
+    if [ -n "${GNUPGHOME}" ]; then
+        export GNUPGHOME="${GNUPGHOME}"
+    fi
+    reprepro -b "${dir}" --dbdir "${dbdir}" \
+        removesrc "${codename}" \
+        "${packagename}"
+}
+
 repo_del_package() {
     local dir="$1"
     local dbdir="$2"
-- 
2.34.1

[PATCH 2/5] dpkg: Limit dsc searching by workdir

[Anton Mikanovich]

If local isar-apt copy has dsc file for current package it should be
ignored. This also prevent from any issues if source dir will have
matched files in it.

Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
 meta/classes/dpkg.bbclass | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
index c596adbf..90d717ca 100644
--- a/meta/classes/dpkg.bbclass
+++ b/meta/classes/dpkg.bbclass
@@ -97,9 +97,9 @@ dpkg_runbuild() {
 
     # Create a .dsc file from source directory to use it with sbuild
     DEB_SOURCE_NAME=$(dpkg-parsechangelog --show-field Source --file ${WORKDIR}/${PPS}/debian/changelog)
-    find ${WORKDIR} -name "${DEB_SOURCE_NAME}*.dsc" -delete
+    find ${WORKDIR} -name "${DEB_SOURCE_NAME}*.dsc" -maxdepth 1 -delete
     sh -c "cd ${WORKDIR}; dpkg-source -q -b ${PPS}"
-    DSC_FILE=$(find ${WORKDIR} -name "${DEB_SOURCE_NAME}*.dsc" -print)
+    DSC_FILE=$(find ${WORKDIR} -name "${DEB_SOURCE_NAME}*.dsc" -maxdepth 1 -print)
 
     sbuild -A -n -c ${SBUILD_CHROOT} --extra-repository="${ISAR_APT_REPO}" \
         --host=${PACKAGE_ARCH} --build=${BUILD_HOST_ARCH} ${profiles} \
-- 
2.34.1

[PATCH 3/5] meta: Implement two stage build

[Anton Mikanovich]

Split package build into two stages: building source package and
building binary package. This allows to obtain Debian package
dependencies before any build steps will actually start.

Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
 meta/classes/dpkg-gbp.bbclass              |  2 +-
 meta/classes/dpkg-source.bbclass           | 33 ++++++++++++++++++++++
 meta/classes/dpkg.bbclass                  |  4 +--
 meta/classes/rootfs.bbclass                |  2 ++
 meta/recipes-kernel/linux/linux-custom.inc |  2 +-
 5 files changed, 38 insertions(+), 5 deletions(-)
 create mode 100644 meta/classes/dpkg-source.bbclass

diff --git a/meta/classes/dpkg-gbp.bbclass b/meta/classes/dpkg-gbp.bbclass
index 2bd09eaa..a151ade2 100644
--- a/meta/classes/dpkg-gbp.bbclass
+++ b/meta/classes/dpkg-gbp.bbclass
@@ -11,7 +11,7 @@ PATCHTOOL ?= "git"
 
 GBP_EXTRA_OPTIONS ?= "--git-pristine-tar"
 
-dpkg_runbuild:prepend() {
+do_dpkg_source:prepend() {
     sh -c "
         cd ${WORKDIR}/${PPS}
         gbp buildpackage --git-ignore-new --git-builder=/bin/true ${GBP_EXTRA_OPTIONS}
diff --git a/meta/classes/dpkg-source.bbclass b/meta/classes/dpkg-source.bbclass
new file mode 100644
index 00000000..434f1752
--- /dev/null
+++ b/meta/classes/dpkg-source.bbclass
@@ -0,0 +1,33 @@
+# This software is a part of ISAR.
+# Copyright (C) 2023 ilbers GmbH
+#
+# SPDX-License-Identifier: MIT
+
+inherit dpkg-base
+
+do_dpkg_source() {
+    # Create a .dsc file from source directory to use it with sbuild
+    DEB_SOURCE_NAME=$(dpkg-parsechangelog --show-field Source --file ${WORKDIR}/${PPS}/debian/changelog)
+    find ${WORKDIR} -name "${DEB_SOURCE_NAME}*.dsc" -maxdepth 1 -delete
+    if [ ! -z "${SOURCE_DATE_EPOCH}" ]; then
+        export SOURCE_DATE_EPOCH="${SOURCE_DATE_EPOCH}"
+    fi
+    sh -c "cd ${WORKDIR}; dpkg-source -I -b ${PPS}"
+}
+addtask dpkg_source after do_prepare_build before do_dpkg_build
+
+do_deploy_source[depends] += "isar-apt:do_cache_config"
+do_deploy_source[lockfiles] = "${REPO_ISAR_DIR}/isar.lock"
+do_deploy_source[dirs] = "${S}"
+do_deploy_source() {
+    repo_del_srcpackage "${REPO_ISAR_DIR}"/"${DISTRO}" \
+        "${REPO_ISAR_DB_DIR}"/"${DISTRO}" "${DEBDISTRONAME}" "${BPN}"
+    find "${S}/../" -name '*\.dsc' -maxdepth 1 | while read package; do
+        repo_add_srcpackage "${REPO_ISAR_DIR}"/"${DISTRO}" \
+            "${REPO_ISAR_DB_DIR}"/"${DISTRO}" \
+            "${DEBDISTRONAME}" \
+            "${package}"
+    done
+}
+addtask deploy_source after do_dpkg_source before do_dpkg_build
+
diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
index 90d717ca..2bb167e4 100644
--- a/meta/classes/dpkg.bbclass
+++ b/meta/classes/dpkg.bbclass
@@ -2,6 +2,7 @@
 # Copyright (C) 2015-2018 ilbers GmbH
 
 inherit dpkg-base
+inherit dpkg-source
 
 PACKAGE_ARCH ?= "${DISTRO_ARCH}"
 
@@ -95,10 +96,7 @@ dpkg_runbuild() {
 
     echo '$apt_keep_downloaded_packages = 1;' >> ${SBUILD_CONFIG}
 
-    # Create a .dsc file from source directory to use it with sbuild
     DEB_SOURCE_NAME=$(dpkg-parsechangelog --show-field Source --file ${WORKDIR}/${PPS}/debian/changelog)
-    find ${WORKDIR} -name "${DEB_SOURCE_NAME}*.dsc" -maxdepth 1 -delete
-    sh -c "cd ${WORKDIR}; dpkg-source -q -b ${PPS}"
     DSC_FILE=$(find ${WORKDIR} -name "${DEB_SOURCE_NAME}*.dsc" -maxdepth 1 -print)
 
     sbuild -A -n -c ${SBUILD_CHROOT} --extra-repository="${ISAR_APT_REPO}" \
diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass
index 7dec7b36..65574a6c 100644
--- a/meta/classes/rootfs.bbclass
+++ b/meta/classes/rootfs.bbclass
@@ -90,6 +90,8 @@ rootfs_configure_isar_apt() {
     mkdir -p '${ROOTFSDIR}/etc/apt/sources.list.d'
     echo 'deb [trusted=yes] file:///isar-apt ${DEBDISTRONAME} main' > \
         '${ROOTFSDIR}/etc/apt/sources.list.d/isar-apt.list'
+    echo 'deb-src [trusted=yes] file:///isar-apt ${DEBDISTRONAME} main' >> \
+        '${ROOTFSDIR}/etc/apt/sources.list.d/isar-apt.list'
 
     mkdir -p '${ROOTFSDIR}/etc/apt/preferences.d'
     cat << EOF > '${ROOTFSDIR}/etc/apt/preferences.d/isar-apt'
diff --git a/meta/recipes-kernel/linux/linux-custom.inc b/meta/recipes-kernel/linux/linux-custom.inc
index 8b367a2f..fa5d957d 100644
--- a/meta/recipes-kernel/linux/linux-custom.inc
+++ b/meta/recipes-kernel/linux/linux-custom.inc
@@ -212,6 +212,6 @@ EOF
 	fi
 }
 
-dpkg_runbuild:prepend() {
+do_dpkg_source:prepend() {
 	dpkg_configure_kernel
 }
-- 
2.34.1

[PATCH 4/5] dpkg-base: Copy isar_apt to workdir in separate task

[Anton Mikanovich]

As only isar_apt copy functionality needs repo locking and dependencies
building it can be placed into separate task to simplify tasks
requirements.

Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
 meta/classes/dpkg-base.bbclass | 15 +++++++--------
 1 file changed, 7 insertions(+), 8 deletions(-)

diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index 7b02f378..f223fc16 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -186,10 +186,10 @@ do_prepare_build() {
 addtask prepare_build after do_patch do_transform_template before do_dpkg_build
 # If Isar recipes depend on each other, they typically need the package
 # deployed to isar-apt
-do_prepare_build[deptask] = "do_deploy_deb"
-do_prepare_build[depends] = "${SCHROOT_DEP}"
-
-do_prepare_build:append() {
+do_local_isarapt[depends] += "isar-apt:do_cache_config"
+do_local_isarapt[deptask] = "do_deploy_deb"
+do_local_isarapt[lockfiles] += "${REPO_ISAR_DIR}/isar.lock"
+do_local_isarapt() {
     # Make a local copy of isar-apt repo that is not affected by other parallel builds
     rm -rf "${WORKDIR}/isar-apt/${DISTRO}-${DISTRO_ARCH}/*"
     mkdir -p "${WORKDIR}/isar-apt/${DISTRO}-${DISTRO_ARCH}/apt/${DISTRO}"
@@ -198,8 +198,7 @@ do_prepare_build:append() {
         cp -Rlf "${REPO_ISAR_DIR}/${DISTRO}/pool" "${WORKDIR}/isar-apt/${DISTRO}-${DISTRO_ARCH}/apt/${DISTRO}/"
     fi
 }
-
-do_prepare_build[lockfiles] += "${REPO_ISAR_DIR}/isar.lock"
+addtask local_isarapt before do_dpkg_build
 
 # Placeholder for actual dpkg_runbuild() implementation
 dpkg_runbuild() {
@@ -324,7 +323,7 @@ python do_devshell() {
     bb.build.exec_func('schroot_delete_configs', d)
 }
 
-addtask devshell after do_prepare_build
+addtask devshell after do_local_isarapt
 DEVSHELL_STARTDIR ?= "${S}"
 do_devshell[dirs] = "${DEVSHELL_STARTDIR}"
 do_devshell[nostamp] = "1"
@@ -336,7 +335,7 @@ python do_devshell_nodeps() {
 
 # devshell may be placed after do_instell_builddeps in downstream classes.
 # devshell_nodeps will always stay right after do_prepare_build.
-addtask devshell_nodeps after do_prepare_build
+addtask devshell_nodeps after do_local_isarapt
 do_devshell_nodeps[dirs] = "${DEVSHELL_STARTDIR}"
 do_devshell_nodeps[nostamp] = "1"
 do_devshell_nodeps[network] = "${TASK_USE_SUDO}"
-- 
2.34.1

[PATCH 5/5] RECIPE-API-CHANGELOG.md: Note tasks separation

[Anton Mikanovich]

Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
 RECIPE-API-CHANGELOG.md | 14 ++++++++++++++
 1 file changed, 14 insertions(+)

diff --git a/RECIPE-API-CHANGELOG.md b/RECIPE-API-CHANGELOG.md
index 659e64c6..7d463473 100644
--- a/RECIPE-API-CHANGELOG.md
+++ b/RECIPE-API-CHANGELOG.md
@@ -525,3 +525,17 @@ If privileged execution is required `-u root` option can be added.
 ```
 SCHROOT_MOUNTS += "${OUT_PATH1}:${IN_PATH1} ${OUT_PATH2}"
 ```
+
+### Building source package tasks were separated
+
+Two new tasks are introduced: do_dpkg_source and do_deploy_source. The first
+one builds source package, second one adds this package to isar_apt.
+They are placed between do_prepare_build and do_dpkg_build.
+Now all source modifications should be done inside do_prepare_build task, in
+some cases dpkg_runbuild:prepend should be replaced by do_dpkg_source:prepend.
+
+### Local copy of isar-apt creation task was separated
+
+We need local copy of isar-apt to have build dependencies reachable. Now is
+prepared in separate task: do_local_isarapt.
+This task depends of do_deploy_deb of all build dependency recipes.
-- 
2.34.1

Re: [PATCH 0/5] Separate prebuild activities

[Anton Mikanovich]

15/09/2023 21:54, Anton Mikanovich wrote:
> This allows to execute tasks more efficient. On simple targets like qemuamd64
> the speed up of the build is around 10 second, but on complex downstream layers
> this effect can be more visible.

I've just checked isar-cip-core downstream and got 30s speedup from 3m30s
original overall building time.

Re: [PATCH 0/5] Separate prebuild activities

[Uladzimir Bely]

On Fri, 2023-09-15 at 21:54 +0300, Anton Mikanovich wrote:
> This patchset introduce three new pre-build tasks from the code was
> already
> done during the build task:
>  do_dpkg_source
>  do_deploy_source
>  do_local_isarapt
> This allows to execute tasks more efficient. On simple targets like
> qemuamd64
> the speed up of the build is around 10 second, but on complex
> downstream layers
> this effect can be more visible.
> We also introduce source packages storage in isar-apt repository
> which allows
> obtaining package Build-Depends and Provides information in Debian
> way later.
> 
> Anton Mikanovich (5):
>   repository: Add source package remove function
>   dpkg: Limit dsc searching by workdir
>   meta: Implement two stage build
>   dpkg-base: Copy isar_apt to workdir in separate task
>   RECIPE-API-CHANGELOG.md: Note tasks separation
> 
>  RECIPE-API-CHANGELOG.md                    | 14 +++++++++
>  meta/classes/dpkg-base.bbclass             | 15 +++++-----
>  meta/classes/dpkg-gbp.bbclass              |  2 +-
>  meta/classes/dpkg-source.bbclass           | 33
> ++++++++++++++++++++++
>  meta/classes/dpkg.bbclass                  |  6 ++--
>  meta/classes/repository.bbclass            | 14 +++++++++
>  meta/classes/rootfs.bbclass                |  2 ++
>  meta/recipes-kernel/linux/linux-custom.inc |  2 +-
>  8 files changed, 74 insertions(+), 14 deletions(-)
>  create mode 100644 meta/classes/dpkg-source.bbclass
> 
> -- 
> 2.34.1
> 

Applied to next.

Re: [PATCH 3/5] meta: Implement two stage build

[Jan Kiszka]

On 15.09.23 20:54, Anton Mikanovich wrote:
> Split package build into two stages: building source package and
> building binary package. This allows to obtain Debian package
> dependencies before any build steps will actually start.
> 
> Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
> ---
>  meta/classes/dpkg-gbp.bbclass              |  2 +-
>  meta/classes/dpkg-source.bbclass           | 33 ++++++++++++++++++++++
>  meta/classes/dpkg.bbclass                  |  4 +--
>  meta/classes/rootfs.bbclass                |  2 ++
>  meta/recipes-kernel/linux/linux-custom.inc |  2 +-
>  5 files changed, 38 insertions(+), 5 deletions(-)
>  create mode 100644 meta/classes/dpkg-source.bbclass
> 
> diff --git a/meta/classes/dpkg-gbp.bbclass b/meta/classes/dpkg-gbp.bbclass
> index 2bd09eaa..a151ade2 100644
> --- a/meta/classes/dpkg-gbp.bbclass
> +++ b/meta/classes/dpkg-gbp.bbclass
> @@ -11,7 +11,7 @@ PATCHTOOL ?= "git"
>  
>  GBP_EXTRA_OPTIONS ?= "--git-pristine-tar"
>  
> -dpkg_runbuild:prepend() {
> +do_dpkg_source:prepend() {
>      sh -c "
>          cd ${WORKDIR}/${PPS}
>          gbp buildpackage --git-ignore-new --git-builder=/bin/true ${GBP_EXTRA_OPTIONS}
> diff --git a/meta/classes/dpkg-source.bbclass b/meta/classes/dpkg-source.bbclass
> new file mode 100644
> index 00000000..434f1752
> --- /dev/null
> +++ b/meta/classes/dpkg-source.bbclass
> @@ -0,0 +1,33 @@
> +# This software is a part of ISAR.
> +# Copyright (C) 2023 ilbers GmbH
> +#
> +# SPDX-License-Identifier: MIT
> +
> +inherit dpkg-base
> +
> +do_dpkg_source() {
> +    # Create a .dsc file from source directory to use it with sbuild
> +    DEB_SOURCE_NAME=$(dpkg-parsechangelog --show-field Source --file ${WORKDIR}/${PPS}/debian/changelog)
> +    find ${WORKDIR} -name "${DEB_SOURCE_NAME}*.dsc" -maxdepth 1 -delete
> +    if [ ! -z "${SOURCE_DATE_EPOCH}" ]; then
> +        export SOURCE_DATE_EPOCH="${SOURCE_DATE_EPOCH}"
> +    fi
> +    sh -c "cd ${WORKDIR}; dpkg-source -I -b ${PPS}"
> +}
> +addtask dpkg_source after do_prepare_build before do_dpkg_build
> +
> +do_deploy_source[depends] += "isar-apt:do_cache_config"
> +do_deploy_source[lockfiles] = "${REPO_ISAR_DIR}/isar.lock"
> +do_deploy_source[dirs] = "${S}"
> +do_deploy_source() {
> +    repo_del_srcpackage "${REPO_ISAR_DIR}"/"${DISTRO}" \
> +        "${REPO_ISAR_DB_DIR}"/"${DISTRO}" "${DEBDISTRONAME}" "${BPN}"
> +    find "${S}/../" -name '*\.dsc' -maxdepth 1 | while read package; do
> +        repo_add_srcpackage "${REPO_ISAR_DIR}"/"${DISTRO}" \
> +            "${REPO_ISAR_DB_DIR}"/"${DISTRO}" \
> +            "${DEBDISTRONAME}" \
> +            "${package}"
> +    done
> +}
> +addtask deploy_source after do_dpkg_source before do_dpkg_build
> +
> diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
> index 90d717ca..2bb167e4 100644
> --- a/meta/classes/dpkg.bbclass
> +++ b/meta/classes/dpkg.bbclass
> @@ -2,6 +2,7 @@
>  # Copyright (C) 2015-2018 ilbers GmbH
>  
>  inherit dpkg-base
> +inherit dpkg-source
>  
>  PACKAGE_ARCH ?= "${DISTRO_ARCH}"
>  
> @@ -95,10 +96,7 @@ dpkg_runbuild() {
>  
>      echo '$apt_keep_downloaded_packages = 1;' >> ${SBUILD_CONFIG}
>  
> -    # Create a .dsc file from source directory to use it with sbuild
>      DEB_SOURCE_NAME=$(dpkg-parsechangelog --show-field Source --file ${WORKDIR}/${PPS}/debian/changelog)
> -    find ${WORKDIR} -name "${DEB_SOURCE_NAME}*.dsc" -maxdepth 1 -delete
> -    sh -c "cd ${WORKDIR}; dpkg-source -q -b ${PPS}"
>      DSC_FILE=$(find ${WORKDIR} -name "${DEB_SOURCE_NAME}*.dsc" -maxdepth 1 -print)
>  
>      sbuild -A -n -c ${SBUILD_CHROOT} --extra-repository="${ISAR_APT_REPO}" \
> diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass
> index 7dec7b36..65574a6c 100644
> --- a/meta/classes/rootfs.bbclass
> +++ b/meta/classes/rootfs.bbclass
> @@ -90,6 +90,8 @@ rootfs_configure_isar_apt() {
>      mkdir -p '${ROOTFSDIR}/etc/apt/sources.list.d'
>      echo 'deb [trusted=yes] file:///isar-apt ${DEBDISTRONAME} main' > \
>          '${ROOTFSDIR}/etc/apt/sources.list.d/isar-apt.list'
> +    echo 'deb-src [trusted=yes] file:///isar-apt ${DEBDISTRONAME} main' >> \
> +        '${ROOTFSDIR}/etc/apt/sources.list.d/isar-apt.list'
>  
>      mkdir -p '${ROOTFSDIR}/etc/apt/preferences.d'
>      cat << EOF > '${ROOTFSDIR}/etc/apt/preferences.d/isar-apt'
> diff --git a/meta/recipes-kernel/linux/linux-custom.inc b/meta/recipes-kernel/linux/linux-custom.inc
> index 8b367a2f..fa5d957d 100644
> --- a/meta/recipes-kernel/linux/linux-custom.inc
> +++ b/meta/recipes-kernel/linux/linux-custom.inc
> @@ -212,6 +212,6 @@ EOF
>  	fi
>  }
>  
> -dpkg_runbuild:prepend() {
> +do_dpkg_source:prepend() {
>  	dpkg_configure_kernel
>  }

This is not transparent to recipe as the very last hunk nicely
demonstrates. Where is the RECIPE-API-CHANGLOG entry?

I thought this change was already the reason, but I'm still searching
for a regression of this patch here on our downstream recipe
https://gitlab.com/cip-project/cip-core/isar-cip-core/-/tree/master/recipes-bsp/edk2
(can be watched under
https://github.com/siemens/meta-iot2050/actions/runs/6492616563/job/17631910317.
Any hints welcome.

Jan

-- 
Siemens AG, Technology
Linux Expert Center

Re: [PATCH 3/5] meta: Implement two stage build

[Anton Mikanovich]

12/10/2023 11:48, Jan Kiszka wrote:
> On 15.09.23 20:54, Anton Mikanovich wrote:
>> Split package build into two stages: building source package and
>> building binary package. This allows to obtain Debian package
>> dependencies before any build steps will actually start.
>>
>> Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
>> ---
>>   meta/classes/dpkg-gbp.bbclass              |  2 +-
>>   meta/classes/dpkg-source.bbclass           | 33 ++++++++++++++++++++++
>>   meta/classes/dpkg.bbclass                  |  4 +--
>>   meta/classes/rootfs.bbclass                |  2 ++
>>   meta/recipes-kernel/linux/linux-custom.inc |  2 +-
>>   5 files changed, 38 insertions(+), 5 deletions(-)
>>   create mode 100644 meta/classes/dpkg-source.bbclass
>>
>> diff --git a/meta/classes/dpkg-gbp.bbclass b/meta/classes/dpkg-gbp.bbclass
>> index 2bd09eaa..a151ade2 100644
>> --- a/meta/classes/dpkg-gbp.bbclass
>> +++ b/meta/classes/dpkg-gbp.bbclass
>> @@ -11,7 +11,7 @@ PATCHTOOL ?= "git"
>>   
>>   GBP_EXTRA_OPTIONS ?= "--git-pristine-tar"
>>   
>> -dpkg_runbuild:prepend() {
>> +do_dpkg_source:prepend() {
>>       sh -c "
>>           cd ${WORKDIR}/${PPS}
>>           gbp buildpackage --git-ignore-new --git-builder=/bin/true ${GBP_EXTRA_OPTIONS}
>> diff --git a/meta/classes/dpkg-source.bbclass b/meta/classes/dpkg-source.bbclass
>> new file mode 100644
>> index 00000000..434f1752
>> --- /dev/null
>> +++ b/meta/classes/dpkg-source.bbclass
>> @@ -0,0 +1,33 @@
>> +# This software is a part of ISAR.
>> +# Copyright (C) 2023 ilbers GmbH
>> +#
>> +# SPDX-License-Identifier: MIT
>> +
>> +inherit dpkg-base
>> +
>> +do_dpkg_source() {
>> +    # Create a .dsc file from source directory to use it with sbuild
>> +    DEB_SOURCE_NAME=$(dpkg-parsechangelog --show-field Source --file ${WORKDIR}/${PPS}/debian/changelog)
>> +    find ${WORKDIR} -name "${DEB_SOURCE_NAME}*.dsc" -maxdepth 1 -delete
>> +    if [ ! -z "${SOURCE_DATE_EPOCH}" ]; then
>> +        export SOURCE_DATE_EPOCH="${SOURCE_DATE_EPOCH}"
>> +    fi
>> +    sh -c "cd ${WORKDIR}; dpkg-source -I -b ${PPS}"
>> +}
>> +addtask dpkg_source after do_prepare_build before do_dpkg_build
>> +
>> +do_deploy_source[depends] += "isar-apt:do_cache_config"
>> +do_deploy_source[lockfiles] = "${REPO_ISAR_DIR}/isar.lock"
>> +do_deploy_source[dirs] = "${S}"
>> +do_deploy_source() {
>> +    repo_del_srcpackage "${REPO_ISAR_DIR}"/"${DISTRO}" \
>> +        "${REPO_ISAR_DB_DIR}"/"${DISTRO}" "${DEBDISTRONAME}" "${BPN}"
>> +    find "${S}/../" -name '*\.dsc' -maxdepth 1 | while read package; do
>> +        repo_add_srcpackage "${REPO_ISAR_DIR}"/"${DISTRO}" \
>> +            "${REPO_ISAR_DB_DIR}"/"${DISTRO}" \
>> +            "${DEBDISTRONAME}" \
>> +            "${package}"
>> +    done
>> +}
>> +addtask deploy_source after do_dpkg_source before do_dpkg_build
>> +
>> diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
>> index 90d717ca..2bb167e4 100644
>> --- a/meta/classes/dpkg.bbclass
>> +++ b/meta/classes/dpkg.bbclass
>> @@ -2,6 +2,7 @@
>>   # Copyright (C) 2015-2018 ilbers GmbH
>>   
>>   inherit dpkg-base
>> +inherit dpkg-source
>>   
>>   PACKAGE_ARCH ?= "${DISTRO_ARCH}"
>>   
>> @@ -95,10 +96,7 @@ dpkg_runbuild() {
>>   
>>       echo '$apt_keep_downloaded_packages = 1;' >> ${SBUILD_CONFIG}
>>   
>> -    # Create a .dsc file from source directory to use it with sbuild
>>       DEB_SOURCE_NAME=$(dpkg-parsechangelog --show-field Source --file ${WORKDIR}/${PPS}/debian/changelog)
>> -    find ${WORKDIR} -name "${DEB_SOURCE_NAME}*.dsc" -maxdepth 1 -delete
>> -    sh -c "cd ${WORKDIR}; dpkg-source -q -b ${PPS}"
>>       DSC_FILE=$(find ${WORKDIR} -name "${DEB_SOURCE_NAME}*.dsc" -maxdepth 1 -print)
>>   
>>       sbuild -A -n -c ${SBUILD_CHROOT} --extra-repository="${ISAR_APT_REPO}" \
>> diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass
>> index 7dec7b36..65574a6c 100644
>> --- a/meta/classes/rootfs.bbclass
>> +++ b/meta/classes/rootfs.bbclass
>> @@ -90,6 +90,8 @@ rootfs_configure_isar_apt() {
>>       mkdir -p '${ROOTFSDIR}/etc/apt/sources.list.d'
>>       echo 'deb [trusted=yes] file:///isar-apt ${DEBDISTRONAME} main' > \
>>           '${ROOTFSDIR}/etc/apt/sources.list.d/isar-apt.list'
>> +    echo 'deb-src [trusted=yes] file:///isar-apt ${DEBDISTRONAME} main' >> \
>> +        '${ROOTFSDIR}/etc/apt/sources.list.d/isar-apt.list'
>>   
>>       mkdir -p '${ROOTFSDIR}/etc/apt/preferences.d'
>>       cat << EOF > '${ROOTFSDIR}/etc/apt/preferences.d/isar-apt'
>> diff --git a/meta/recipes-kernel/linux/linux-custom.inc b/meta/recipes-kernel/linux/linux-custom.inc
>> index 8b367a2f..fa5d957d 100644
>> --- a/meta/recipes-kernel/linux/linux-custom.inc
>> +++ b/meta/recipes-kernel/linux/linux-custom.inc
>> @@ -212,6 +212,6 @@ EOF
>>   	fi
>>   }
>>   
>> -dpkg_runbuild:prepend() {
>> +do_dpkg_source:prepend() {
>>   	dpkg_configure_kernel
>>   }
> This is not transparent to recipe as the very last hunk nicely
> demonstrates. Where is the RECIPE-API-CHANGLOG entry?
In p5.
> I thought this change was already the reason, but I'm still searching
> for a regression of this patch here on our downstream recipe
> https://gitlab.com/cip-project/cip-core/isar-cip-core/-/tree/master/recipes-bsp/edk2
> (can be watched under
> https://github.com/siemens/meta-iot2050/actions/runs/6492616563/job/17631910317.
This change shouldn't have any influence on the linker for such a 
trivial recipe.
Are you sure this patchset is the only change between working and 
non-working hashes?
> Any hints welcome.
>
> Jan
>

Re: [PATCH 3/5] meta: Implement two stage build

[Jan Kiszka]

On 12.10.23 10:48, Jan Kiszka wrote:
> On 15.09.23 20:54, Anton Mikanovich wrote:
>> Split package build into two stages: building source package and
>> building binary package. This allows to obtain Debian package
>> dependencies before any build steps will actually start.
>>
>> Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
>> ---
>>  meta/classes/dpkg-gbp.bbclass              |  2 +-
>>  meta/classes/dpkg-source.bbclass           | 33 ++++++++++++++++++++++
>>  meta/classes/dpkg.bbclass                  |  4 +--
>>  meta/classes/rootfs.bbclass                |  2 ++
>>  meta/recipes-kernel/linux/linux-custom.inc |  2 +-
>>  5 files changed, 38 insertions(+), 5 deletions(-)
>>  create mode 100644 meta/classes/dpkg-source.bbclass
>>
>> diff --git a/meta/classes/dpkg-gbp.bbclass b/meta/classes/dpkg-gbp.bbclass
>> index 2bd09eaa..a151ade2 100644
>> --- a/meta/classes/dpkg-gbp.bbclass
>> +++ b/meta/classes/dpkg-gbp.bbclass
>> @@ -11,7 +11,7 @@ PATCHTOOL ?= "git"
>>  
>>  GBP_EXTRA_OPTIONS ?= "--git-pristine-tar"
>>  
>> -dpkg_runbuild:prepend() {
>> +do_dpkg_source:prepend() {
>>      sh -c "
>>          cd ${WORKDIR}/${PPS}
>>          gbp buildpackage --git-ignore-new --git-builder=/bin/true ${GBP_EXTRA_OPTIONS}
>> diff --git a/meta/classes/dpkg-source.bbclass b/meta/classes/dpkg-source.bbclass
>> new file mode 100644
>> index 00000000..434f1752
>> --- /dev/null
>> +++ b/meta/classes/dpkg-source.bbclass
>> @@ -0,0 +1,33 @@
>> +# This software is a part of ISAR.
>> +# Copyright (C) 2023 ilbers GmbH
>> +#
>> +# SPDX-License-Identifier: MIT
>> +
>> +inherit dpkg-base
>> +
>> +do_dpkg_source() {
>> +    # Create a .dsc file from source directory to use it with sbuild
>> +    DEB_SOURCE_NAME=$(dpkg-parsechangelog --show-field Source --file ${WORKDIR}/${PPS}/debian/changelog)
>> +    find ${WORKDIR} -name "${DEB_SOURCE_NAME}*.dsc" -maxdepth 1 -delete
>> +    if [ ! -z "${SOURCE_DATE_EPOCH}" ]; then
>> +        export SOURCE_DATE_EPOCH="${SOURCE_DATE_EPOCH}"
>> +    fi
>> +    sh -c "cd ${WORKDIR}; dpkg-source -I -b ${PPS}"
>> +}
>> +addtask dpkg_source after do_prepare_build before do_dpkg_build
>> +
>> +do_deploy_source[depends] += "isar-apt:do_cache_config"
>> +do_deploy_source[lockfiles] = "${REPO_ISAR_DIR}/isar.lock"
>> +do_deploy_source[dirs] = "${S}"
>> +do_deploy_source() {
>> +    repo_del_srcpackage "${REPO_ISAR_DIR}"/"${DISTRO}" \
>> +        "${REPO_ISAR_DB_DIR}"/"${DISTRO}" "${DEBDISTRONAME}" "${BPN}"
>> +    find "${S}/../" -name '*\.dsc' -maxdepth 1 | while read package; do
>> +        repo_add_srcpackage "${REPO_ISAR_DIR}"/"${DISTRO}" \
>> +            "${REPO_ISAR_DB_DIR}"/"${DISTRO}" \
>> +            "${DEBDISTRONAME}" \
>> +            "${package}"
>> +    done
>> +}
>> +addtask deploy_source after do_dpkg_source before do_dpkg_build
>> +
>> diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
>> index 90d717ca..2bb167e4 100644
>> --- a/meta/classes/dpkg.bbclass
>> +++ b/meta/classes/dpkg.bbclass
>> @@ -2,6 +2,7 @@
>>  # Copyright (C) 2015-2018 ilbers GmbH
>>  
>>  inherit dpkg-base
>> +inherit dpkg-source
>>  
>>  PACKAGE_ARCH ?= "${DISTRO_ARCH}"
>>  
>> @@ -95,10 +96,7 @@ dpkg_runbuild() {
>>  
>>      echo '$apt_keep_downloaded_packages = 1;' >> ${SBUILD_CONFIG}
>>  
>> -    # Create a .dsc file from source directory to use it with sbuild
>>      DEB_SOURCE_NAME=$(dpkg-parsechangelog --show-field Source --file ${WORKDIR}/${PPS}/debian/changelog)
>> -    find ${WORKDIR} -name "${DEB_SOURCE_NAME}*.dsc" -maxdepth 1 -delete
>> -    sh -c "cd ${WORKDIR}; dpkg-source -q -b ${PPS}"
>>      DSC_FILE=$(find ${WORKDIR} -name "${DEB_SOURCE_NAME}*.dsc" -maxdepth 1 -print)
>>  
>>      sbuild -A -n -c ${SBUILD_CHROOT} --extra-repository="${ISAR_APT_REPO}" \
>> diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass
>> index 7dec7b36..65574a6c 100644
>> --- a/meta/classes/rootfs.bbclass
>> +++ b/meta/classes/rootfs.bbclass
>> @@ -90,6 +90,8 @@ rootfs_configure_isar_apt() {
>>      mkdir -p '${ROOTFSDIR}/etc/apt/sources.list.d'
>>      echo 'deb [trusted=yes] file:///isar-apt ${DEBDISTRONAME} main' > \
>>          '${ROOTFSDIR}/etc/apt/sources.list.d/isar-apt.list'
>> +    echo 'deb-src [trusted=yes] file:///isar-apt ${DEBDISTRONAME} main' >> \
>> +        '${ROOTFSDIR}/etc/apt/sources.list.d/isar-apt.list'
>>  
>>      mkdir -p '${ROOTFSDIR}/etc/apt/preferences.d'
>>      cat << EOF > '${ROOTFSDIR}/etc/apt/preferences.d/isar-apt'
>> diff --git a/meta/recipes-kernel/linux/linux-custom.inc b/meta/recipes-kernel/linux/linux-custom.inc
>> index 8b367a2f..fa5d957d 100644
>> --- a/meta/recipes-kernel/linux/linux-custom.inc
>> +++ b/meta/recipes-kernel/linux/linux-custom.inc
>> @@ -212,6 +212,6 @@ EOF
>>  	fi
>>  }
>>  
>> -dpkg_runbuild:prepend() {
>> +do_dpkg_source:prepend() {
>>  	dpkg_configure_kernel
>>  }
> 
> This is not transparent to recipe as the very last hunk nicely
> demonstrates. Where is the RECIPE-API-CHANGLOG entry?
> 

OK, found it later on. But "in some cases" is not a helpful explanation.
I would have expected that all extensions of dpkg_runbuild:prepend
touched ${S} need to be moved, or so.

Jan

> I thought this change was already the reason, but I'm still searching
> for a regression of this patch here on our downstream recipe
> https://gitlab.com/cip-project/cip-core/isar-cip-core/-/tree/master/recipes-bsp/edk2
> (can be watched under
> https://github.com/siemens/meta-iot2050/actions/runs/6492616563/job/17631910317.
> Any hints welcome.
> 
> Jan
> 

-- 
Siemens AG, Technology
Linux Expert Center

Re: [PATCH 3/5] meta: Implement two stage build

[Jan Kiszka]

On 12.10.23 11:07, Anton Mikanovich wrote:
> 12/10/2023 11:48, Jan Kiszka wrote:
>> On 15.09.23 20:54, Anton Mikanovich wrote:
>>> Split package build into two stages: building source package and
>>> building binary package. This allows to obtain Debian package
>>> dependencies before any build steps will actually start.
>>>
>>> Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
>>> ---
>>>   meta/classes/dpkg-gbp.bbclass              |  2 +-
>>>   meta/classes/dpkg-source.bbclass           | 33 ++++++++++++++++++++++
>>>   meta/classes/dpkg.bbclass                  |  4 +--
>>>   meta/classes/rootfs.bbclass                |  2 ++
>>>   meta/recipes-kernel/linux/linux-custom.inc |  2 +-
>>>   5 files changed, 38 insertions(+), 5 deletions(-)
>>>   create mode 100644 meta/classes/dpkg-source.bbclass
>>>
>>> diff --git a/meta/classes/dpkg-gbp.bbclass
>>> b/meta/classes/dpkg-gbp.bbclass
>>> index 2bd09eaa..a151ade2 100644
>>> --- a/meta/classes/dpkg-gbp.bbclass
>>> +++ b/meta/classes/dpkg-gbp.bbclass
>>> @@ -11,7 +11,7 @@ PATCHTOOL ?= "git"
>>>     GBP_EXTRA_OPTIONS ?= "--git-pristine-tar"
>>>   -dpkg_runbuild:prepend() {
>>> +do_dpkg_source:prepend() {
>>>       sh -c "
>>>           cd ${WORKDIR}/${PPS}
>>>           gbp buildpackage --git-ignore-new --git-builder=/bin/true
>>> ${GBP_EXTRA_OPTIONS}
>>> diff --git a/meta/classes/dpkg-source.bbclass
>>> b/meta/classes/dpkg-source.bbclass
>>> new file mode 100644
>>> index 00000000..434f1752
>>> --- /dev/null
>>> +++ b/meta/classes/dpkg-source.bbclass
>>> @@ -0,0 +1,33 @@
>>> +# This software is a part of ISAR.
>>> +# Copyright (C) 2023 ilbers GmbH
>>> +#
>>> +# SPDX-License-Identifier: MIT
>>> +
>>> +inherit dpkg-base
>>> +
>>> +do_dpkg_source() {
>>> +    # Create a .dsc file from source directory to use it with sbuild
>>> +    DEB_SOURCE_NAME=$(dpkg-parsechangelog --show-field Source --file
>>> ${WORKDIR}/${PPS}/debian/changelog)
>>> +    find ${WORKDIR} -name "${DEB_SOURCE_NAME}*.dsc" -maxdepth 1 -delete
>>> +    if [ ! -z "${SOURCE_DATE_EPOCH}" ]; then
>>> +        export SOURCE_DATE_EPOCH="${SOURCE_DATE_EPOCH}"
>>> +    fi
>>> +    sh -c "cd ${WORKDIR}; dpkg-source -I -b ${PPS}"
>>> +}
>>> +addtask dpkg_source after do_prepare_build before do_dpkg_build
>>> +
>>> +do_deploy_source[depends] += "isar-apt:do_cache_config"
>>> +do_deploy_source[lockfiles] = "${REPO_ISAR_DIR}/isar.lock"
>>> +do_deploy_source[dirs] = "${S}"
>>> +do_deploy_source() {
>>> +    repo_del_srcpackage "${REPO_ISAR_DIR}"/"${DISTRO}" \
>>> +        "${REPO_ISAR_DB_DIR}"/"${DISTRO}" "${DEBDISTRONAME}" "${BPN}"
>>> +    find "${S}/../" -name '*\.dsc' -maxdepth 1 | while read package; do
>>> +        repo_add_srcpackage "${REPO_ISAR_DIR}"/"${DISTRO}" \
>>> +            "${REPO_ISAR_DB_DIR}"/"${DISTRO}" \
>>> +            "${DEBDISTRONAME}" \
>>> +            "${package}"
>>> +    done
>>> +}
>>> +addtask deploy_source after do_dpkg_source before do_dpkg_build
>>> +
>>> diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
>>> index 90d717ca..2bb167e4 100644
>>> --- a/meta/classes/dpkg.bbclass
>>> +++ b/meta/classes/dpkg.bbclass
>>> @@ -2,6 +2,7 @@
>>>   # Copyright (C) 2015-2018 ilbers GmbH
>>>     inherit dpkg-base
>>> +inherit dpkg-source
>>>     PACKAGE_ARCH ?= "${DISTRO_ARCH}"
>>>   @@ -95,10 +96,7 @@ dpkg_runbuild() {
>>>         echo '$apt_keep_downloaded_packages = 1;' >> ${SBUILD_CONFIG}
>>>   -    # Create a .dsc file from source directory to use it with sbuild
>>>       DEB_SOURCE_NAME=$(dpkg-parsechangelog --show-field Source
>>> --file ${WORKDIR}/${PPS}/debian/changelog)
>>> -    find ${WORKDIR} -name "${DEB_SOURCE_NAME}*.dsc" -maxdepth 1 -delete
>>> -    sh -c "cd ${WORKDIR}; dpkg-source -q -b ${PPS}"
>>>       DSC_FILE=$(find ${WORKDIR} -name "${DEB_SOURCE_NAME}*.dsc"
>>> -maxdepth 1 -print)
>>>         sbuild -A -n -c ${SBUILD_CHROOT}
>>> --extra-repository="${ISAR_APT_REPO}" \
>>> diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass
>>> index 7dec7b36..65574a6c 100644
>>> --- a/meta/classes/rootfs.bbclass
>>> +++ b/meta/classes/rootfs.bbclass
>>> @@ -90,6 +90,8 @@ rootfs_configure_isar_apt() {
>>>       mkdir -p '${ROOTFSDIR}/etc/apt/sources.list.d'
>>>       echo 'deb [trusted=yes] file:///isar-apt ${DEBDISTRONAME} main'
>>> > \
>>>           '${ROOTFSDIR}/etc/apt/sources.list.d/isar-apt.list'
>>> +    echo 'deb-src [trusted=yes] file:///isar-apt ${DEBDISTRONAME}
>>> main' >> \
>>> +        '${ROOTFSDIR}/etc/apt/sources.list.d/isar-apt.list'
>>>         mkdir -p '${ROOTFSDIR}/etc/apt/preferences.d'
>>>       cat << EOF > '${ROOTFSDIR}/etc/apt/preferences.d/isar-apt'
>>> diff --git a/meta/recipes-kernel/linux/linux-custom.inc
>>> b/meta/recipes-kernel/linux/linux-custom.inc
>>> index 8b367a2f..fa5d957d 100644
>>> --- a/meta/recipes-kernel/linux/linux-custom.inc
>>> +++ b/meta/recipes-kernel/linux/linux-custom.inc
>>> @@ -212,6 +212,6 @@ EOF
>>>       fi
>>>   }
>>>   -dpkg_runbuild:prepend() {
>>> +do_dpkg_source:prepend() {
>>>       dpkg_configure_kernel
>>>   }
>> This is not transparent to recipe as the very last hunk nicely
>> demonstrates. Where is the RECIPE-API-CHANGLOG entry?
> In p5.
>> I thought this change was already the reason, but I'm still searching
>> for a regression of this patch here on our downstream recipe
>> https://gitlab.com/cip-project/cip-core/isar-cip-core/-/tree/master/recipes-bsp/edk2
>> (can be watched under
>> https://github.com/siemens/meta-iot2050/actions/runs/6492616563/job/17631910317.
> This change shouldn't have any influence on the linker for such a
> trivial recipe.
> Are you sure this patchset is the only change between working and
> non-working hashes?

I found the reason: You added -I to dpkg-source, and that filters out
some questionable but needed prebuilt .a files in edk2. All fine again
when dropping it. I'll send a patch, we should not do that here.

Jan

-- 
Siemens AG, Technology
Linux Expert Center

Re: [PATCH 3/5] meta: Implement two stage build

[Baurzhan Ismagulov]

On 2023-10-12 13:12, 'Jan Kiszka' via isar-users wrote:
> I found the reason: You added -I to dpkg-source, and that filters out
> some questionable but needed prebuilt .a files in edk2. All fine again
> when dropping it. I'll send a patch, we should not do that here.

Thanks Jan for the analysis, this explains it.

That said, Debian doesn't want source packages to contain binary libraries or
.git files for a good reason (we were hit by the latter), so removing -I
altogether will break other stuff. We will quickly provide an option to drop -I
per recipe, which will solve this problem without affecting all recipes.

With kind regards,
Baurzhan

Re: [PATCH 3/5] meta: Implement two stage build

[Jan Kiszka]

On 12.10.23 16:39, Baurzhan Ismagulov wrote:
> On 2023-10-12 13:12, 'Jan Kiszka' via isar-users wrote:
>> I found the reason: You added -I to dpkg-source, and that filters out
>> some questionable but needed prebuilt .a files in edk2. All fine again
>> when dropping it. I'll send a patch, we should not do that here.
> 
> Thanks Jan for the analysis, this explains it.
> 
> That said, Debian doesn't want source packages to contain binary libraries or
> .git files for a good reason (we were hit by the latter), so removing -I
> altogether will break other stuff. We will quickly provide an option to drop -I
> per recipe, which will solve this problem without affecting all recipes.

Yes, that's why we need to solve things differently on the long run.
First, this is an Isar regression we need to fix: You changed behavior
in the middle of some harmless refactoring that breaks downstream
subtly. Not a good idea.

Jan

-- 
Siemens AG, Technology
Linux Expert Center

Re: [PATCH 3/5] meta: Implement two stage build

[Baurzhan Ismagulov]

On 2023-10-12 16:53, 'Jan Kiszka' via isar-users wrote:
> Yes, that's why we need to solve things differently on the long run.
> First, this is an Isar regression we need to fix: You changed behavior
> in the middle of some harmless refactoring that breaks downstream
> subtly. Not a good idea.

I see. -I being motivated by a real problem, I agree the patch landing in the
refactoring series is unfortunate. In the future, such changes should be sent
separately.

With kind regards,
Baurzhan

Re: [PATCH 0/5] Separate prebuild activities

[MOESSBAUER, Felix]

On Fri, 2023-09-15 at 21:54 +0300, Anton Mikanovich wrote:
> This patchset introduce three new pre-build tasks from the code was
> already
> done during the build task:
>  do_dpkg_source
>  do_deploy_source

We need to add sstate caching to the dpkg_source task as well.
Currently we see some significant slowdown on big source packages (like
the kernel) on otherwise fully cache-able builds.

Maybe Adriaan could add this, as he added the caching for dpkg_build as
well.

Best regards,
Felix

>  do_local_isarapt
> This allows to execute tasks more efficient. On simple targets like
> qemuamd64
> the speed up of the build is around 10 second, but on complex
> downstream layers
> this effect can be more visible.
> We also introduce source packages storage in isar-apt repository
> which allows
> obtaining package Build-Depends and Provides information in Debian
> way later.
> 
> Anton Mikanovich (5):
>   repository: Add source package remove function
>   dpkg: Limit dsc searching by workdir
>   meta: Implement two stage build
>   dpkg-base: Copy isar_apt to workdir in separate task
>   RECIPE-API-CHANGELOG.md: Note tasks separation
> 
>  RECIPE-API-CHANGELOG.md                    | 14 +++++++++
>  meta/classes/dpkg-base.bbclass             | 15 +++++-----
>  meta/classes/dpkg-gbp.bbclass              |  2 +-
>  meta/classes/dpkg-source.bbclass           | 33
> ++++++++++++++++++++++
>  meta/classes/dpkg.bbclass                  |  6 ++--
>  meta/classes/repository.bbclass            | 14 +++++++++
>  meta/classes/rootfs.bbclass                |  2 ++
>  meta/recipes-kernel/linux/linux-custom.inc |  2 +-
>  8 files changed, 74 insertions(+), 14 deletions(-)
>  create mode 100644 meta/classes/dpkg-source.bbclass
> 
> -- 
> 2.34.1
> 

RE: [PATCH 0/5] Separate prebuild activities

[Schmidt, Adriaan]

MOESSBAUER, Felix (T CED INW-CN), Monday, October 30, 2023 5:45 AM:
> On Fri, 2023-09-15 at 21:54 +0300, Anton Mikanovich wrote:
> > This patchset introduce three new pre-build tasks from the code was
> > already
> > done during the build task:
> >  do_dpkg_source
> >  do_deploy_source
> 
> We need to add sstate caching to the dpkg_source task as well.
> Currently we see some significant slowdown on big source packages (like
> the kernel) on otherwise fully cache-able builds.
> 
> Maybe Adriaan could add this, as he added the caching for dpkg_build as
> well.

Well, currently I expect that whenever we have a situation in which dpkg_build
would have to run (most likely a change in the recipe), that would mean
we need to rebuild the source as well. So I'm skeptical caching the source
would help much at this point.

BUT:
If we take the two-stage build one step further and actually have dpkg_build
work exclusively with the generated source package from isar-apt as input,
we could re-use one src package for all targets and variants (-compat/-native).

I'm currently working in this direction (prompted by bugs we see when installing
both "pkg" and "pkg-compat" in the same image), and so far I've found:

- We currently see a side-effect in the "remove source package" function:
  "reprepro removesrc" not only removes the src package, but also all related binary
  packages from the repository. So we need to make sure that only one
  task in a build deploys a src package (and that src is deployed before any binaries).
  Currently the source would be deployed by both "pkg" and "pkg-compat", with a
  resulting race condition. Already preparing a patch for that.
- The -compat/-native suffix sometimes "leaks" into the source package.
  Basically, there are still locations that should use "${BPN}" instead of "${PN}".
  Already preparing a patch for that.
- currently there are still tasks/dependencies of do_dpkg_build that rely on 
  the extracted source being present (e.g. finding out the name of the src
  package)
- we'd need to add a deb-src entry to the apt sources for sbuild
- to actually be able to share src packages across architectures, we'd need
  to make sure that the src packages are always identical, and don't depend
  on any architecture or -compat/-native variant. I guess in the Debian world
  that already is a rule, but in bitbake users can of course add any number
  of overrides (e.g. changing SRC_URI depending on PACKAGE_ARCH) to break
  this rule...

Adriaan

> Best regards,
> Felix
> 
> >  do_local_isarapt
> > This allows to execute tasks more efficient. On simple targets like
> > qemuamd64
> > the speed up of the build is around 10 second, but on complex
> > downstream layers
> > this effect can be more visible.
> > We also introduce source packages storage in isar-apt repository
> > which allows
> > obtaining package Build-Depends and Provides information in Debian
> > way later.
> >
> > Anton Mikanovich (5):
> >   repository: Add source package remove function
> >   dpkg: Limit dsc searching by workdir
> >   meta: Implement two stage build
> >   dpkg-base: Copy isar_apt to workdir in separate task
> >   RECIPE-API-CHANGELOG.md: Note tasks separation
> >
> >  RECIPE-API-CHANGELOG.md                    | 14 +++++++++
> >  meta/classes/dpkg-base.bbclass             | 15 +++++-----
> >  meta/classes/dpkg-gbp.bbclass              |  2 +-
> >  meta/classes/dpkg-source.bbclass           | 33
> > ++++++++++++++++++++++
> >  meta/classes/dpkg.bbclass                  |  6 ++--
> >  meta/classes/repository.bbclass            | 14 +++++++++
> >  meta/classes/rootfs.bbclass                |  2 ++
> >  meta/recipes-kernel/linux/linux-custom.inc |  2 +-
> >  8 files changed, 74 insertions(+), 14 deletions(-)
> >  create mode 100644 meta/classes/dpkg-source.bbclass
> >
> > --
> > 2.34.1
> >
>