[PATCH 1/2] dpkg-source: Fix source deployment

[PATCH 1/2] dpkg-source: Fix source deployment

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

This failed if S was not a direct subdir of WORKDIR. Align it with
do_dpkg_source.

Furthermore, move -maxdepth before -name to prevent warnings about a
global option specified after a positional one.

Fixes: 38b832ad8248 ("meta: Implement two stage build")
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 meta/classes/dpkg-source.bbclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/classes/dpkg-source.bbclass b/meta/classes/dpkg-source.bbclass
index 7fd5d2ed..005eafbe 100644
--- a/meta/classes/dpkg-source.bbclass
+++ b/meta/classes/dpkg-source.bbclass
@@ -21,7 +21,7 @@ do_deploy_source[dirs] = "${S}"
 do_deploy_source() {
     repo_del_srcpackage "${REPO_ISAR_DIR}"/"${DISTRO}" \
         "${REPO_ISAR_DB_DIR}"/"${DISTRO}" "${DEBDISTRONAME}" "${BPN}"
-    find "${S}/../" -name '*\.dsc' -maxdepth 1 | while read package; do
+    find "${WORKDIR}" -maxdepth 1 -name '*\.dsc' | while read package; do
         repo_add_srcpackage "${REPO_ISAR_DIR}"/"${DISTRO}" \
             "${REPO_ISAR_DB_DIR}"/"${DISTRO}" \
             "${DEBDISTRONAME}" \
-- 
2.35.3

[PATCH 2/2] dpkg-source: Build source package only once

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

Avoid building the source package multiple times, possibly even
inconsistently. This is achieved by delegating this task to to the base
package and installing the source package from isar-apt in the native
and compat package variants.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 meta/classes/dpkg-source.bbclass | 44 ++++++++++++++++++++++++++++++--
 1 file changed, 42 insertions(+), 2 deletions(-)

diff --git a/meta/classes/dpkg-source.bbclass b/meta/classes/dpkg-source.bbclass
index 005eafbe..7161641f 100644
--- a/meta/classes/dpkg-source.bbclass
+++ b/meta/classes/dpkg-source.bbclass
@@ -13,7 +13,7 @@ do_dpkg_source() {
     find ${WORKDIR} -name "${DEB_SOURCE_NAME}*.dsc" -maxdepth 1 -delete
     sh -c "cd ${WORKDIR}; dpkg-source ${DPKG_SOURCE_EXTRA_ARGS} -b ${PPS}"
 }
-addtask dpkg_source after do_prepare_build before do_dpkg_build
+addtask dpkg_source after do_prepare_build
 
 do_deploy_source[depends] += "isar-apt:do_cache_config"
 do_deploy_source[lockfiles] = "${REPO_ISAR_DIR}/isar.lock"
@@ -28,4 +28,44 @@ do_deploy_source() {
             "${package}"
     done
 }
-addtask deploy_source after do_dpkg_source before do_dpkg_build
+addtask deploy_source after do_dpkg_source
+
+do_dpkg_build[depends] += "${BPN}:do_deploy_source"
+
+SCHROOT_MOUNTS = "${WORKDIR}:/work ${REPO_ISAR_DIR}/${DISTRO}:/isar-apt"
+
+do_fetch_common_source[depends] += "${BPN}:do_deploy_source"
+do_fetch_common_source[network] = "${TASK_USE_SUDO}"
+do_fetch_common_source() {
+    schroot_create_configs
+    insert_mounts
+
+    session_id=$(schroot -q -b -c ${SBUILD_CHROOT})
+    echo "Started session: ${session_id}"
+
+    schroot_cleanup() {
+        schroot -q -f -e -c ${session_id} > /dev/null 2>&1
+        remove_mounts > /dev/null 2>&1
+        schroot_delete_configs
+    }
+    trap 'exit 1' INT HUP QUIT TERM ALRM USR1
+    trap 'schroot_cleanup' EXIT
+
+    schroot -r -c ${session_id} -d / -u root -- \
+        apt-get update -o Dir::Etc::SourceList="sources.list.d/isar-apt.list" -o Dir::Etc::SourceParts="-" -o APT::Get::List-Cleanup="0"
+    schroot -r -c ${session_id} -d / -- \
+        sh -c '
+            cd /work
+            apt-get -y --download-only --only-source -o Acquire::Source-Symlinks="false" source ${BPN}'
+
+    schroot -e -c ${session_id}
+    remove_mounts
+    schroot_delete_configs
+}
+addtask fetch_common_source after do_unpack
+
+def is_native_or_compat(d):
+    overrides = d.getVar('OVERRIDES').split(':')
+    return 'class-native' in overrides or 'class-compat' in overrides
+
+do_dpkg_build[depends] += "${@'${PN}:do_fetch_common_source' if is_native_or_compat(d) else ''}"
-- 
2.35.3

RE: [PATCH 2/2] dpkg-source: Build source package only once

[Schmidt, Adriaan]

Kiszka, Jan, Sonntag, 5. Mai 2024 22:33:
> Avoid building the source package multiple times, possibly even
> inconsistently. This is achieved by delegating this task to to the base
> package and installing the source package from isar-apt in the native
> and compat package variants.

Hi Jan,

I thought about changes along those lines a while back, and had two problems
that made this difficult for me:

- This assumes that the source packages are identical for different build
  variants. That is how it should be (there should be only one source, possibly
  even across distro releases and architectures), but Isar gives developers
  the tools to break that rule (SRC_URI:append:xy = "some.patch"), and
  they will use those tools! So we'd need at least documentation warning
  about this, or probably better some checks that restrict how SRC_URI
  can be constructed.
- In dpkg_runbuild (dpkg.bbclass), the code that finds DEB_SOURCE_NAME
  assumes extracted sources (with a debian/changelog). Looking at your
  patch, I suspect this only works for you because you still have a
  dependency on do_fetch, even for the compat/native case where
  you probably want to only depend on the fetch_common_source job.

To me it seems that we currently have an almost-but-not-100%-complete
decoupling of the build into source generation and binary package build.

I'd propose that we first complete that split (solving the two problems
I mentioned). This would then also allow sstate caching of the common
source packages, and it should make it possible for all build variants
to use the same code path, without the need for the is_native_or_compat
branch you have in your patch.

Adriaan

> 
> Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
> ---
>  meta/classes/dpkg-source.bbclass | 44 ++++++++++++++++++++++++++++++--
>  1 file changed, 42 insertions(+), 2 deletions(-)
> 
> diff --git a/meta/classes/dpkg-source.bbclass b/meta/classes/dpkg-
> source.bbclass
> index 005eafbe..7161641f 100644
> --- a/meta/classes/dpkg-source.bbclass
> +++ b/meta/classes/dpkg-source.bbclass
> @@ -13,7 +13,7 @@ do_dpkg_source() {
>      find ${WORKDIR} -name "${DEB_SOURCE_NAME}*.dsc" -maxdepth 1 -delete
>      sh -c "cd ${WORKDIR}; dpkg-source ${DPKG_SOURCE_EXTRA_ARGS} -b ${PPS}"
>  }
> -addtask dpkg_source after do_prepare_build before do_dpkg_build
> +addtask dpkg_source after do_prepare_build
> 
>  do_deploy_source[depends] += "isar-apt:do_cache_config"
>  do_deploy_source[lockfiles] = "${REPO_ISAR_DIR}/isar.lock"
> @@ -28,4 +28,44 @@ do_deploy_source() {
>              "${package}"
>      done
>  }
> -addtask deploy_source after do_dpkg_source before do_dpkg_build
> +addtask deploy_source after do_dpkg_source
> +
> +do_dpkg_build[depends] += "${BPN}:do_deploy_source"
> +
> +SCHROOT_MOUNTS = "${WORKDIR}:/work ${REPO_ISAR_DIR}/${DISTRO}:/isar-apt"
> +
> +do_fetch_common_source[depends] += "${BPN}:do_deploy_source"
> +do_fetch_common_source[network] = "${TASK_USE_SUDO}"
> +do_fetch_common_source() {
> +    schroot_create_configs
> +    insert_mounts
> +
> +    session_id=$(schroot -q -b -c ${SBUILD_CHROOT})
> +    echo "Started session: ${session_id}"
> +
> +    schroot_cleanup() {
> +        schroot -q -f -e -c ${session_id} > /dev/null 2>&1
> +        remove_mounts > /dev/null 2>&1
> +        schroot_delete_configs
> +    }
> +    trap 'exit 1' INT HUP QUIT TERM ALRM USR1
> +    trap 'schroot_cleanup' EXIT
> +
> +    schroot -r -c ${session_id} -d / -u root -- \
> +        apt-get update -o Dir::Etc::SourceList="sources.list.d/isar-
> apt.list" -o Dir::Etc::SourceParts="-" -o APT::Get::List-Cleanup="0"
> +    schroot -r -c ${session_id} -d / -- \
> +        sh -c '
> +            cd /work
> +            apt-get -y --download-only --only-source -o Acquire::Source-
> Symlinks="false" source ${BPN}'
> +
> +    schroot -e -c ${session_id}
> +    remove_mounts
> +    schroot_delete_configs
> +}
> +addtask fetch_common_source after do_unpack
> +
> +def is_native_or_compat(d):
> +    overrides = d.getVar('OVERRIDES').split(':')
> +    return 'class-native' in overrides or 'class-compat' in overrides
> +
> +do_dpkg_build[depends] += "${@'${PN}:do_fetch_common_source' if
> is_native_or_compat(d) else ''}"
> --
> 2.35.3

Re: [PATCH 2/2] dpkg-source: Build source package only once

[Jan Kiszka]

On 06.05.24 08:29, Schmidt, Adriaan (T CED EDC-DE) wrote:
> Kiszka, Jan, Sonntag, 5. Mai 2024 22:33:
>> Avoid building the source package multiple times, possibly even
>> inconsistently. This is achieved by delegating this task to to the base
>> package and installing the source package from isar-apt in the native
>> and compat package variants.
> 
> Hi Jan,
> 
> I thought about changes along those lines a while back, and had two problems
> that made this difficult for me:
> 
> - This assumes that the source packages are identical for different build
>   variants. That is how it should be (there should be only one source, possibly
>   even across distro releases and architectures), but Isar gives developers
>   the tools to break that rule (SRC_URI:append:xy = "some.patch"), and
>   they will use those tools! So we'd need at least documentation warning
>   about this, or probably better some checks that restrict how SRC_URI
>   can be constructed.

If developers do the wrong thing (like in this patch), they get what
they deserve. My patch prevents that this silently sneaks into production.

> - In dpkg_runbuild (dpkg.bbclass), the code that finds DEB_SOURCE_NAME
>   assumes extracted sources (with a debian/changelog). Looking at your
>   patch, I suspect this only works for you because you still have a
>   dependency on do_fetch, even for the compat/native case where
>   you probably want to only depend on the fetch_common_source job.

To my understanding, installing the source package ensures this.

Jan

-- 
Siemens AG, Technology
Linux Expert Center

Re: [PATCH 2/2] dpkg-source: Build source package only once

[Koch, Stefan]

On Sun, 2024-05-05 at 22:32 +0200, Jan Kiszka wrote:
> From: Jan Kiszka <jan.kiszka@siemens.com>
> 
> Avoid building the source package multiple times, possibly even
> inconsistently. This is achieved by delegating this task to to the
> base
> package and installing the source package from isar-apt in the native
> and compat package variants.
> 
> Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
> ---
>  meta/classes/dpkg-source.bbclass | 44
> ++++++++++++++++++++++++++++++--
>  1 file changed, 42 insertions(+), 2 deletions(-)
> 
> diff --git a/meta/classes/dpkg-source.bbclass b/meta/classes/dpkg-
> source.bbclass
> index 005eafbe..7161641f 100644
> --- a/meta/classes/dpkg-source.bbclass
> +++ b/meta/classes/dpkg-source.bbclass
> @@ -13,7 +13,7 @@ do_dpkg_source() {
>      find ${WORKDIR} -name "${DEB_SOURCE_NAME}*.dsc" -maxdepth 1 -
> delete
>      sh -c "cd ${WORKDIR}; dpkg-source ${DPKG_SOURCE_EXTRA_ARGS} -b
> ${PPS}"
>  }
> -addtask dpkg_source after do_prepare_build before do_dpkg_build
> +addtask dpkg_source after do_prepare_build
>  
>  do_deploy_source[depends] += "isar-apt:do_cache_config"
>  do_deploy_source[lockfiles] = "${REPO_ISAR_DIR}/isar.lock"
> @@ -28,4 +28,44 @@ do_deploy_source() {
>              "${package}"
>      done
>  }
> -addtask deploy_source after do_dpkg_source before do_dpkg_build
> +addtask deploy_source after do_dpkg_source
> +
> +do_dpkg_build[depends] += "${BPN}:do_deploy_source"
> +
> +SCHROOT_MOUNTS = "${WORKDIR}:/work ${REPO_ISAR_DIR}/${DISTRO}:/isar-
> apt"
> +
> +do_fetch_common_source[depends] += "${BPN}:do_deploy_source"
> +do_fetch_common_source[network] = "${TASK_USE_SUDO}"
> +do_fetch_common_source() {
> +    schroot_create_configs
> +    insert_mounts
> +
> +    session_id=$(schroot -q -b -c ${SBUILD_CHROOT})
> +    echo "Started session: ${session_id}"
> +
> +    schroot_cleanup() {
> +        schroot -q -f -e -c ${session_id} > /dev/null 2>&1
> +        remove_mounts > /dev/null 2>&1
> +        schroot_delete_configs
> +    }
> +    trap 'exit 1' INT HUP QUIT TERM ALRM USR1
> +    trap 'schroot_cleanup' EXIT
> +
> +    schroot -r -c ${session_id} -d / -u root -- \
> +        apt-get update -o Dir::Etc::SourceList="sources.list.d/isar-
> apt.list" -o Dir::Etc::SourceParts="-" -o APT::Get::List-Cleanup="0"
> +    schroot -r -c ${session_id} -d / -- \
> +        sh -c '
> +            cd /work
> +            apt-get -y --download-only --only-source -o
> Acquire::Source-Symlinks="false" source ${BPN}'
> +
> +    schroot -e -c ${session_id}
> +    remove_mounts
> +    schroot_delete_configs
> +}
> +addtask fetch_common_source after do_unpack
> +
> +def is_native_or_compat(d):
> +    overrides = d.getVar('OVERRIDES').split(':')
> +    return 'class-native' in overrides or 'class-compat' in
> overrides
> +
> +do_dpkg_build[depends] += "${@'${PN}:do_fetch_common_source' if
> is_native_or_compat(d) else ''}"
Thanks. For the linux-custom build the linux-....tar.gz sources are
created not only for linux-native but also for linux-kbuildtarget...

Stefan
-- 
Stefan Koch
Siemens AG
www.siemens.com

Re: [PATCH 2/2] dpkg-source: Build source package only once

[Jan Kiszka]

On 06.05.24 10:11, Koch, Stefan (DI PA DCP R&D 3) wrote:
> On Sun, 2024-05-05 at 22:32 +0200, Jan Kiszka wrote:
>> From: Jan Kiszka <jan.kiszka@siemens.com>
>>
>> Avoid building the source package multiple times, possibly even
>> inconsistently. This is achieved by delegating this task to to the
>> base
>> package and installing the source package from isar-apt in the native
>> and compat package variants.
>>
>> Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
>> ---
>>  meta/classes/dpkg-source.bbclass | 44
>> ++++++++++++++++++++++++++++++--
>>  1 file changed, 42 insertions(+), 2 deletions(-)
>>
>> diff --git a/meta/classes/dpkg-source.bbclass b/meta/classes/dpkg-
>> source.bbclass
>> index 005eafbe..7161641f 100644
>> --- a/meta/classes/dpkg-source.bbclass
>> +++ b/meta/classes/dpkg-source.bbclass
>> @@ -13,7 +13,7 @@ do_dpkg_source() {
>>      find ${WORKDIR} -name "${DEB_SOURCE_NAME}*.dsc" -maxdepth 1 -
>> delete
>>      sh -c "cd ${WORKDIR}; dpkg-source ${DPKG_SOURCE_EXTRA_ARGS} -b
>> ${PPS}"
>>  }
>> -addtask dpkg_source after do_prepare_build before do_dpkg_build
>> +addtask dpkg_source after do_prepare_build
>>  
>>  do_deploy_source[depends] += "isar-apt:do_cache_config"
>>  do_deploy_source[lockfiles] = "${REPO_ISAR_DIR}/isar.lock"
>> @@ -28,4 +28,44 @@ do_deploy_source() {
>>              "${package}"
>>      done
>>  }
>> -addtask deploy_source after do_dpkg_source before do_dpkg_build
>> +addtask deploy_source after do_dpkg_source
>> +
>> +do_dpkg_build[depends] += "${BPN}:do_deploy_source"
>> +
>> +SCHROOT_MOUNTS = "${WORKDIR}:/work ${REPO_ISAR_DIR}/${DISTRO}:/isar-
>> apt"
>> +
>> +do_fetch_common_source[depends] += "${BPN}:do_deploy_source"
>> +do_fetch_common_source[network] = "${TASK_USE_SUDO}"
>> +do_fetch_common_source() {
>> +    schroot_create_configs
>> +    insert_mounts
>> +
>> +    session_id=$(schroot -q -b -c ${SBUILD_CHROOT})
>> +    echo "Started session: ${session_id}"
>> +
>> +    schroot_cleanup() {
>> +        schroot -q -f -e -c ${session_id} > /dev/null 2>&1
>> +        remove_mounts > /dev/null 2>&1
>> +        schroot_delete_configs
>> +    }
>> +    trap 'exit 1' INT HUP QUIT TERM ALRM USR1
>> +    trap 'schroot_cleanup' EXIT
>> +
>> +    schroot -r -c ${session_id} -d / -u root -- \
>> +        apt-get update -o Dir::Etc::SourceList="sources.list.d/isar-
>> apt.list" -o Dir::Etc::SourceParts="-" -o APT::Get::List-Cleanup="0"
>> +    schroot -r -c ${session_id} -d / -- \
>> +        sh -c '
>> +            cd /work
>> +            apt-get -y --download-only --only-source -o
>> Acquire::Source-Symlinks="false" source ${BPN}'
>> +
>> +    schroot -e -c ${session_id}
>> +    remove_mounts
>> +    schroot_delete_configs
>> +}
>> +addtask fetch_common_source after do_unpack
>> +
>> +def is_native_or_compat(d):
>> +    overrides = d.getVar('OVERRIDES').split(':')
>> +    return 'class-native' in overrides or 'class-compat' in
>> overrides
>> +
>> +do_dpkg_build[depends] += "${@'${PN}:do_fetch_common_source' if
>> is_native_or_compat(d) else ''}"
> Thanks. For the linux-custom build the linux-....tar.gz sources are
> created not only for linux-native but also for linux-kbuildtarget...
> 

That target is not stressed in upstream, it looks like - recipe for
bitrotting...

Jan

-- 
Siemens AG, Technology
Linux Expert Center

Re: [PATCH 1/2] dpkg-source: Fix source deployment

[Jan Kiszka]

On 05.05.24 22:32, 'Jan Kiszka' via isar-users wrote:
> From: Jan Kiszka <jan.kiszka@siemens.com>
> 
> This failed if S was not a direct subdir of WORKDIR. Align it with
> do_dpkg_source.
> 
> Furthermore, move -maxdepth before -name to prevent warnings about a
> global option specified after a positional one.
> 
> Fixes: 38b832ad8248 ("meta: Implement two stage build")
> Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
> ---
>  meta/classes/dpkg-source.bbclass | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/meta/classes/dpkg-source.bbclass b/meta/classes/dpkg-source.bbclass
> index 7fd5d2ed..005eafbe 100644
> --- a/meta/classes/dpkg-source.bbclass
> +++ b/meta/classes/dpkg-source.bbclass
> @@ -21,7 +21,7 @@ do_deploy_source[dirs] = "${S}"
>  do_deploy_source() {
>      repo_del_srcpackage "${REPO_ISAR_DIR}"/"${DISTRO}" \
>          "${REPO_ISAR_DB_DIR}"/"${DISTRO}" "${DEBDISTRONAME}" "${BPN}"
> -    find "${S}/../" -name '*\.dsc' -maxdepth 1 | while read package; do
> +    find "${WORKDIR}" -maxdepth 1 -name '*\.dsc' | while read package; do
>          repo_add_srcpackage "${REPO_ISAR_DIR}"/"${DISTRO}" \
>              "${REPO_ISAR_DB_DIR}"/"${DISTRO}" \
>              "${DEBDISTRONAME}" \

The longer I look at this... Why is this a loop over potentially
multiple dsc files? We are expecting a single source package, and that
is assumed to be called ${BPN} (that assumption for cleaning may be
wrong with external source packages, though). So there should be also
only a single match for *_*.dsc, no?

I'm actually trying the clean the mess we have in dpkg_runbuild where we
are reading debian/changlog to get the dsc name. When not building the
source packages ourselves, we would have to unpack the sources in order
get the changelog in order to find out the dsc name that is needed to
unpack the source - this is nonsense.

Jan

-- 
Siemens AG, Technology
Linux Expert Center

Re: [PATCH 1/2] dpkg-source: Fix source deployment

[Jan Kiszka]

On 10.05.24 19:32, Jan Kiszka wrote:
> On 05.05.24 22:32, 'Jan Kiszka' via isar-users wrote:
>> From: Jan Kiszka <jan.kiszka@siemens.com>
>>
>> This failed if S was not a direct subdir of WORKDIR. Align it with
>> do_dpkg_source.
>>
>> Furthermore, move -maxdepth before -name to prevent warnings about a
>> global option specified after a positional one.
>>
>> Fixes: 38b832ad8248 ("meta: Implement two stage build")
>> Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
>> ---
>>  meta/classes/dpkg-source.bbclass | 2 +-
>>  1 file changed, 1 insertion(+), 1 deletion(-)
>>
>> diff --git a/meta/classes/dpkg-source.bbclass b/meta/classes/dpkg-source.bbclass
>> index 7fd5d2ed..005eafbe 100644
>> --- a/meta/classes/dpkg-source.bbclass
>> +++ b/meta/classes/dpkg-source.bbclass
>> @@ -21,7 +21,7 @@ do_deploy_source[dirs] = "${S}"
>>  do_deploy_source() {
>>      repo_del_srcpackage "${REPO_ISAR_DIR}"/"${DISTRO}" \
>>          "${REPO_ISAR_DB_DIR}"/"${DISTRO}" "${DEBDISTRONAME}" "${BPN}"
>> -    find "${S}/../" -name '*\.dsc' -maxdepth 1 | while read package; do
>> +    find "${WORKDIR}" -maxdepth 1 -name '*\.dsc' | while read package; do
>>          repo_add_srcpackage "${REPO_ISAR_DIR}"/"${DISTRO}" \
>>              "${REPO_ISAR_DB_DIR}"/"${DISTRO}" \
>>              "${DEBDISTRONAME}" \
> 
> The longer I look at this... Why is this a loop over potentially
> multiple dsc files? We are expecting a single source package, and that
> is assumed to be called ${BPN} (that assumption for cleaning may be
> wrong with external source packages, though). So there should be also
> only a single match for *_*.dsc, no?
> 

Ok, there is a case where there could be multiple dsc files: If someone
fetches apt://foo in order to build bar.deb. Then we will not only
deploy the sources of bar to isar-apt but also those of the dependency
foo - ouch.

We need to figure out the source name of the target package differently.
Not so easy...

Jan

> I'm actually trying the clean the mess we have in dpkg_runbuild where we
> are reading debian/changlog to get the dsc name. When not building the
> source packages ourselves, we would have to unpack the sources in order
> get the changelog in order to find out the dsc name that is needed to
> unpack the source - this is nonsense.
> 
> Jan
> 

-- 
Siemens AG, Technology
Linux Expert Center

Re: [PATCH 1/2] dpkg-source: Fix source deployment

[Jan Kiszka]

On 10.05.24 19:32, Jan Kiszka wrote:
> On 05.05.24 22:32, 'Jan Kiszka' via isar-users wrote:
>> From: Jan Kiszka <jan.kiszka@siemens.com>
>>
>> This failed if S was not a direct subdir of WORKDIR. Align it with
>> do_dpkg_source.
>>
>> Furthermore, move -maxdepth before -name to prevent warnings about a
>> global option specified after a positional one.
>>
>> Fixes: 38b832ad8248 ("meta: Implement two stage build")
>> Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
>> ---
>>  meta/classes/dpkg-source.bbclass | 2 +-
>>  1 file changed, 1 insertion(+), 1 deletion(-)
>>
>> diff --git a/meta/classes/dpkg-source.bbclass b/meta/classes/dpkg-source.bbclass
>> index 7fd5d2ed..005eafbe 100644
>> --- a/meta/classes/dpkg-source.bbclass
>> +++ b/meta/classes/dpkg-source.bbclass
>> @@ -21,7 +21,7 @@ do_deploy_source[dirs] = "${S}"
>>  do_deploy_source() {
>>      repo_del_srcpackage "${REPO_ISAR_DIR}"/"${DISTRO}" \
>>          "${REPO_ISAR_DB_DIR}"/"${DISTRO}" "${DEBDISTRONAME}" "${BPN}"
>> -    find "${S}/../" -name '*\.dsc' -maxdepth 1 | while read package; do
>> +    find "${WORKDIR}" -maxdepth 1 -name '*\.dsc' | while read package; do
>>          repo_add_srcpackage "${REPO_ISAR_DIR}"/"${DISTRO}" \
>>              "${REPO_ISAR_DB_DIR}"/"${DISTRO}" \
>>              "${DEBDISTRONAME}" \
> 
> The longer I look at this... Why is this a loop over potentially
> multiple dsc files? We are expecting a single source package, and that
> is assumed to be called ${BPN} (that assumption for cleaning may be
> wrong with external source packages, though). So there should be also
> only a single match for *_*.dsc, no?
> 
> I'm actually trying the clean the mess we have in dpkg_runbuild where we
> are reading debian/changlog to get the dsc name. When not building the
> source packages ourselves, we would have to unpack the sources in order
> get the changelog in order to find out the dsc name that is needed to
> unpack the source - this is nonsense.
> 

The following changes on top of the this patch is apparently working 
fine, and you can simply do

IMAGE_INSTALL += "linux-headers-${KERNEL_NAME}"

again.

diff --git a/meta/recipes-kernel/linux-module/module.inc b/meta/recipes-kernel/linux-module/module.inc
index 1cca9cfb..229e6a5c 100644
--- a/meta/recipes-kernel/linux-module/module.inc
+++ b/meta/recipes-kernel/linux-module/module.inc
@@ -17,8 +17,7 @@ PN .= "-${KERNEL_NAME}"
 
 KERNEL_IMAGE_PKG ??= "linux-image-${KERNEL_NAME}"
 KERNEL_HEADERS_PKG ??= "linux-headers-${KERNEL_NAME}"
-KERNEL_KBUILD_PKG ??= "linux-kbuild-${KERNEL_NAME}"
-DEPENDS += "${KERNEL_HEADERS_PKG} ${KERNEL_KBUILD_PKG}-native"
+DEPENDS += "${KERNEL_HEADERS_PKG}-native"
 DEBIAN_BUILD_DEPENDS = "${KERNEL_HEADERS_PKG}"
 
 SIGNATURE_KEYFILE ??= ""
diff --git a/meta/recipes-kernel/linux/linux-custom.inc b/meta/recipes-kernel/linux/linux-custom.inc
index af3504dd..00b169bc 100644
--- a/meta/recipes-kernel/linux/linux-custom.inc
+++ b/meta/recipes-kernel/linux/linux-custom.inc
@@ -113,11 +113,19 @@ BUILD_PROFILES:cross-profile = "kernel"
 
 # -native: kbuild package for host
 BUILD_PROFILES:class-native = "kbuild"
-RECIPE_PROVIDES:class-native = "linux-kbuild-${KERNEL_NAME_PROVIDED}-native"
+RECIPE_PROVIDES:class-native = " \
+    linux-headers-${KERNEL_NAME_PROVIDED}-native \
+    linux-kbuild-${KERNEL_NAME_PROVIDED}-native"
+# Use pseudo target. We cannot use ${BPN} because it will be auto-extended
+# with -native by multiarch.bbclass.
+RDEPENDS:class-native = "${BPN}-pseudo-native"
 
 # -kbuildtarget: kbuild package for target, enforcing non-cross-build
 BUILD_PROFILES:class-kbuildtarget = "kbuild"
-RECIPE_PROVIDES:class-kbuildtarget = "linux-kbuild-${KERNEL_NAME_PROVIDED}"
+RECIPE_PROVIDES:class-kbuildtarget = " \
+    linux-headers-${KERNEL_NAME_PROVIDED} \
+    linux-kbuild-${KERNEL_NAME_PROVIDED}"
+RDEPENDS:class-kbuildtarget = "${BPN}"
 ISAR_CROSS_COMPILE:class-kbuildtarget = "0"
 
 # Make bitbake know we will be producing linux-image and linux-headers packages
@@ -125,15 +133,21 @@ ISAR_CROSS_COMPILE:class-kbuildtarget = "0"
 RECIPE_PROVIDES = " \
     linux-image-${KERNEL_NAME_PROVIDED} \
     linux-headers-${KERNEL_NAME_PROVIDED} \
+    linux-headers-${KERNEL_NAME_PROVIDED}-native \
     linux-libc-dev \
     linux-libc-dev-${DISTRO_ARCH}-cross \
     linux-image-${KERNEL_NAME_PROVIDED}-dbg \
     linux-kbuild-${KERNEL_NAME_PROVIDED} \
+    ${BPN}-pseudo-native \
 "
 # When cross-profile is active:
-# kbuild package is provided by -native or -kbuildtarget variant
-# Otherwise it's provided by the default variant
-RECIPE_PROVIDES:remove:cross-profile = "linux-kbuild-${KERNEL_NAME_PROVIDED}"
+# kbuild package is provided by -native or -kbuildtarget variant. Also headers
+# provisioning moves over to ensure those variants are pulled, although the
+# package itself is still built by the base recipe.
+RECIPE_PROVIDES:remove:cross-profile = " \
+    linux-headers-${KERNEL_NAME_PROVIDED} \
+    linux-headers-${KERNEL_NAME_PROVIDED}-native \
+    linux-kbuild-${KERNEL_NAME_PROVIDED}"
 
 # Append headers depends
 HEADERS_DEPENDS = ", linux-kbuild-${KERNEL_NAME_PROVIDED} | linux-kbuild-${KERNEL_NAME_PROVIDED}-${DISTRO_ARCH}-cross"


Jan

-- 
Siemens AG, Technology
Linux Expert Center

Re: [PATCH 1/2] dpkg-source: Fix source deployment

[Jan Kiszka]

On 13.05.24 08:56, 'Jan Kiszka' via isar-users wrote:
> On 10.05.24 19:32, Jan Kiszka wrote:
>> On 05.05.24 22:32, 'Jan Kiszka' via isar-users wrote:
>>> From: Jan Kiszka <jan.kiszka@siemens.com>
>>>
>>> This failed if S was not a direct subdir of WORKDIR. Align it with
>>> do_dpkg_source.
>>>
>>> Furthermore, move -maxdepth before -name to prevent warnings about a
>>> global option specified after a positional one.
>>>
>>> Fixes: 38b832ad8248 ("meta: Implement two stage build")
>>> Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
>>> ---
>>>  meta/classes/dpkg-source.bbclass | 2 +-
>>>  1 file changed, 1 insertion(+), 1 deletion(-)
>>>
>>> diff --git a/meta/classes/dpkg-source.bbclass b/meta/classes/dpkg-source.bbclass
>>> index 7fd5d2ed..005eafbe 100644
>>> --- a/meta/classes/dpkg-source.bbclass
>>> +++ b/meta/classes/dpkg-source.bbclass
>>> @@ -21,7 +21,7 @@ do_deploy_source[dirs] = "${S}"
>>>  do_deploy_source() {
>>>      repo_del_srcpackage "${REPO_ISAR_DIR}"/"${DISTRO}" \
>>>          "${REPO_ISAR_DB_DIR}"/"${DISTRO}" "${DEBDISTRONAME}" "${BPN}"
>>> -    find "${S}/../" -name '*\.dsc' -maxdepth 1 | while read package; do
>>> +    find "${WORKDIR}" -maxdepth 1 -name '*\.dsc' | while read package; do
>>>          repo_add_srcpackage "${REPO_ISAR_DIR}"/"${DISTRO}" \
>>>              "${REPO_ISAR_DB_DIR}"/"${DISTRO}" \
>>>              "${DEBDISTRONAME}" \
>>
>> The longer I look at this... Why is this a loop over potentially
>> multiple dsc files? We are expecting a single source package, and that
>> is assumed to be called ${BPN} (that assumption for cleaning may be
>> wrong with external source packages, though). So there should be also
>> only a single match for *_*.dsc, no?
>>
>> I'm actually trying the clean the mess we have in dpkg_runbuild where we
>> are reading debian/changlog to get the dsc name. When not building the
>> source packages ourselves, we would have to unpack the sources in order
>> get the changelog in order to find out the dsc name that is needed to
>> unpack the source - this is nonsense.
>>
> 
> The following changes on top of the this patch is apparently working 
> fine, and you can simply do
> 
> IMAGE_INSTALL += "linux-headers-${KERNEL_NAME}"
> 
> again.
> 

[sorry, commented on the wrong thread]

Jan

-- 
Siemens AG, Technology
Linux Expert Center

Re: [PATCH 1/2] dpkg-source: Fix source deployment

[Jan Kiszka]

On 10.05.24 19:55, 'Jan Kiszka' via isar-users wrote:
> On 10.05.24 19:32, Jan Kiszka wrote:
>> On 05.05.24 22:32, 'Jan Kiszka' via isar-users wrote:
>>> From: Jan Kiszka <jan.kiszka@siemens.com>
>>>
>>> This failed if S was not a direct subdir of WORKDIR. Align it with
>>> do_dpkg_source.
>>>
>>> Furthermore, move -maxdepth before -name to prevent warnings about a
>>> global option specified after a positional one.
>>>
>>> Fixes: 38b832ad8248 ("meta: Implement two stage build")
>>> Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
>>> ---
>>>  meta/classes/dpkg-source.bbclass | 2 +-
>>>  1 file changed, 1 insertion(+), 1 deletion(-)
>>>
>>> diff --git a/meta/classes/dpkg-source.bbclass b/meta/classes/dpkg-source.bbclass
>>> index 7fd5d2ed..005eafbe 100644
>>> --- a/meta/classes/dpkg-source.bbclass
>>> +++ b/meta/classes/dpkg-source.bbclass
>>> @@ -21,7 +21,7 @@ do_deploy_source[dirs] = "${S}"
>>>  do_deploy_source() {
>>>      repo_del_srcpackage "${REPO_ISAR_DIR}"/"${DISTRO}" \
>>>          "${REPO_ISAR_DB_DIR}"/"${DISTRO}" "${DEBDISTRONAME}" "${BPN}"
>>> -    find "${S}/../" -name '*\.dsc' -maxdepth 1 | while read package; do
>>> +    find "${WORKDIR}" -maxdepth 1 -name '*\.dsc' | while read package; do
>>>          repo_add_srcpackage "${REPO_ISAR_DIR}"/"${DISTRO}" \
>>>              "${REPO_ISAR_DB_DIR}"/"${DISTRO}" \
>>>              "${DEBDISTRONAME}" \
>>
>> The longer I look at this... Why is this a loop over potentially
>> multiple dsc files? We are expecting a single source package, and that
>> is assumed to be called ${BPN} (that assumption for cleaning may be
>> wrong with external source packages, though). So there should be also
>> only a single match for *_*.dsc, no?
>>
> 
> Ok, there is a case where there could be multiple dsc files: If someone
> fetches apt://foo in order to build bar.deb. Then we will not only
> deploy the sources of bar to isar-apt but also those of the dependency
> foo - ouch.
> 
> We need to figure out the source name of the target package differently.
> Not so easy...
> 

I'm inclined to assume "Source: ${BPN}" and warn during dpkg_source in
case this assumption is violated.

First, this would allow to drop all those retrieval operations later on
using debian/changelog or worse. Second, I see no other way to get the
information to a recipe that fetches the sources from isar-apt. That
recipe would otherwise have to recreate debian/changelog beforehand,
just to fetch it along with the source package.

Thoughts?

Jan

-- 
Siemens AG, Technology
Linux Expert Center

Re: [PATCH 2/2] dpkg-source: Build source package only once

[Anton Mikanovich]

05/05/2024 23:32, 'Jan Kiszka' via isar-users wrote:
> From: Jan Kiszka <jan.kiszka@siemens.com>
>
> Avoid building the source package multiple times, possibly even
> inconsistently. This is achieved by delegating this task to to the base
> package and installing the source package from isar-apt in the native
> and compat package variants.
>
> Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
> ---
>   meta/classes/dpkg-source.bbclass | 44 ++++++++++++++++++++++++++++++--
>   1 file changed, 42 insertions(+), 2 deletions(-)
>
> diff --git a/meta/classes/dpkg-source.bbclass b/meta/classes/dpkg-source.bbclass
> index 005eafbe..7161641f 100644
> --- a/meta/classes/dpkg-source.bbclass
> +++ b/meta/classes/dpkg-source.bbclass
> @@ -13,7 +13,7 @@ do_dpkg_source() {
>       find ${WORKDIR} -name "${DEB_SOURCE_NAME}*.dsc" -maxdepth 1 -delete
>       sh -c "cd ${WORKDIR}; dpkg-source ${DPKG_SOURCE_EXTRA_ARGS} -b ${PPS}"
>   }
> -addtask dpkg_source after do_prepare_build before do_dpkg_build
> +addtask dpkg_source after do_prepare_build
>   
>   do_deploy_source[depends] += "isar-apt:do_cache_config"
>   do_deploy_source[lockfiles] = "${REPO_ISAR_DIR}/isar.lock"
> @@ -28,4 +28,44 @@ do_deploy_source() {
>               "${package}"
>       done
>   }
> -addtask deploy_source after do_dpkg_source before do_dpkg_build
> +addtask deploy_source after do_dpkg_source
> +
> +do_dpkg_build[depends] += "${BPN}:do_deploy_source"
> +
> +SCHROOT_MOUNTS = "${WORKDIR}:/work ${REPO_ISAR_DIR}/${DISTRO}:/isar-apt"
> +
> +do_fetch_common_source[depends] += "${BPN}:do_deploy_source"
As do_fetch_common_source is the first task which use schroot it should also
have ${SCHROOT_DEP} in depends flag.
Simple test case to reproduce the issue with missing schroot: build
mc:qemuarm64-bullseye:libhello-compat (with ISAR_ENABLE_COMPAT_ARCH enabled)
> +do_fetch_common_source[network] = "${TASK_USE_SUDO}"
> +do_fetch_common_source() {
> +    schroot_create_configs
> +    insert_mounts
> +
> +    session_id=$(schroot -q -b -c ${SBUILD_CHROOT})
> +    echo "Started session: ${session_id}"
> +
> +    schroot_cleanup() {
> +        schroot -q -f -e -c ${session_id} > /dev/null 2>&1
> +        remove_mounts > /dev/null 2>&1
> +        schroot_delete_configs
> +    }
> +    trap 'exit 1' INT HUP QUIT TERM ALRM USR1
> +    trap 'schroot_cleanup' EXIT
> +
> +    schroot -r -c ${session_id} -d / -u root -- \
> +        apt-get update -o Dir::Etc::SourceList="sources.list.d/isar-apt.list" -o Dir::Etc::SourceParts="-" -o APT::Get::List-Cleanup="0"
> +    schroot -r -c ${session_id} -d / -- \
> +        sh -c '
> +            cd /work
> +            apt-get -y --download-only --only-source -o Acquire::Source-Symlinks="false" source ${BPN}'
> +
> +    schroot -e -c ${session_id}
> +    remove_mounts
> +    schroot_delete_configs
> +}
> +addtask fetch_common_source after do_unpack
> +
> +def is_native_or_compat(d):
> +    overrides = d.getVar('OVERRIDES').split(':')
> +    return 'class-native' in overrides or 'class-compat' in overrides
> +
> +do_dpkg_build[depends] += "${@'${PN}:do_fetch_common_source' if is_native_or_compat(d) else ''}"

Re: [PATCH 2/2] dpkg-source: Build source package only once

[Jan Kiszka]

On 13.05.24 14:04, Anton Mikanovich wrote:
> 05/05/2024 23:32, 'Jan Kiszka' via isar-users wrote:
>> From: Jan Kiszka <jan.kiszka@siemens.com>
>>
>> Avoid building the source package multiple times, possibly even
>> inconsistently. This is achieved by delegating this task to to the base
>> package and installing the source package from isar-apt in the native
>> and compat package variants.
>>
>> Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
>> ---
>>   meta/classes/dpkg-source.bbclass | 44 ++++++++++++++++++++++++++++++--
>>   1 file changed, 42 insertions(+), 2 deletions(-)
>>
>> diff --git a/meta/classes/dpkg-source.bbclass
>> b/meta/classes/dpkg-source.bbclass
>> index 005eafbe..7161641f 100644
>> --- a/meta/classes/dpkg-source.bbclass
>> +++ b/meta/classes/dpkg-source.bbclass
>> @@ -13,7 +13,7 @@ do_dpkg_source() {
>>       find ${WORKDIR} -name "${DEB_SOURCE_NAME}*.dsc" -maxdepth 1 -delete
>>       sh -c "cd ${WORKDIR}; dpkg-source ${DPKG_SOURCE_EXTRA_ARGS} -b
>> ${PPS}"
>>   }
>> -addtask dpkg_source after do_prepare_build before do_dpkg_build
>> +addtask dpkg_source after do_prepare_build
>>     do_deploy_source[depends] += "isar-apt:do_cache_config"
>>   do_deploy_source[lockfiles] = "${REPO_ISAR_DIR}/isar.lock"
>> @@ -28,4 +28,44 @@ do_deploy_source() {
>>               "${package}"
>>       done
>>   }
>> -addtask deploy_source after do_dpkg_source before do_dpkg_build
>> +addtask deploy_source after do_dpkg_source
>> +
>> +do_dpkg_build[depends] += "${BPN}:do_deploy_source"
>> +
>> +SCHROOT_MOUNTS = "${WORKDIR}:/work ${REPO_ISAR_DIR}/${DISTRO}:/isar-apt"
>> +
>> +do_fetch_common_source[depends] += "${BPN}:do_deploy_source"
> As do_fetch_common_source is the first task which use schroot it should
> also
> have ${SCHROOT_DEP} in depends flag.

Yes, locally fixed already. There was another bug as well, forgot right
now. Will send a new series once Adriaan sent his v4 of the kbuild rework.

Jan

> Simple test case to reproduce the issue with missing schroot: build
> mc:qemuarm64-bullseye:libhello-compat (with ISAR_ENABLE_COMPAT_ARCH
> enabled)
>> +do_fetch_common_source[network] = "${TASK_USE_SUDO}"
>> +do_fetch_common_source() {
>> +    schroot_create_configs
>> +    insert_mounts
>> +
>> +    session_id=$(schroot -q -b -c ${SBUILD_CHROOT})
>> +    echo "Started session: ${session_id}"
>> +
>> +    schroot_cleanup() {
>> +        schroot -q -f -e -c ${session_id} > /dev/null 2>&1
>> +        remove_mounts > /dev/null 2>&1
>> +        schroot_delete_configs
>> +    }
>> +    trap 'exit 1' INT HUP QUIT TERM ALRM USR1
>> +    trap 'schroot_cleanup' EXIT
>> +
>> +    schroot -r -c ${session_id} -d / -u root -- \
>> +        apt-get update -o
>> Dir::Etc::SourceList="sources.list.d/isar-apt.list" -o
>> Dir::Etc::SourceParts="-" -o APT::Get::List-Cleanup="0"
>> +    schroot -r -c ${session_id} -d / -- \
>> +        sh -c '
>> +            cd /work
>> +            apt-get -y --download-only --only-source -o
>> Acquire::Source-Symlinks="false" source ${BPN}'
>> +
>> +    schroot -e -c ${session_id}
>> +    remove_mounts
>> +    schroot_delete_configs
>> +}
>> +addtask fetch_common_source after do_unpack
>> +
>> +def is_native_or_compat(d):
>> +    overrides = d.getVar('OVERRIDES').split(':')
>> +    return 'class-native' in overrides or 'class-compat' in overrides
>> +
>> +do_dpkg_build[depends] += "${@'${PN}:do_fetch_common_source' if
>> is_native_or_compat(d) else ''}"
> 
> 

-- 
Siemens AG, Technology
Linux Expert Center