* [PATCH 1/5] repository: Add source package remove function
2023-09-15 18:54 [PATCH 0/5] Separate prebuild activities Anton Mikanovich
@ 2023-09-15 18:54 ` Anton Mikanovich
2023-09-15 18:54 ` [PATCH 2/5] dpkg: Limit dsc searching by workdir Anton Mikanovich
` (6 subsequent siblings)
7 siblings, 0 replies; 17+ messages in thread
From: Anton Mikanovich @ 2023-09-15 18:54 UTC (permalink / raw)
To: isar-users; +Cc: Anton Mikanovich
There was source package adding functionality in repository bbclass but
no way to remove this package after adding.
So implement also repo_del_srcpackage call.
Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
meta/classes/repository.bbclass | 14 ++++++++++++++
1 file changed, 14 insertions(+)
diff --git a/meta/classes/repository.bbclass b/meta/classes/repository.bbclass
index ac395dbe..a61e5dcd 100644
--- a/meta/classes/repository.bbclass
+++ b/meta/classes/repository.bbclass
@@ -61,6 +61,20 @@ repo_add_packages() {
"$@"
}
+repo_del_srcpackage() {
+ local dir="$1"
+ local dbdir="$2"
+ local codename="$3"
+ local packagename="$4"
+
+ if [ -n "${GNUPGHOME}" ]; then
+ export GNUPGHOME="${GNUPGHOME}"
+ fi
+ reprepro -b "${dir}" --dbdir "${dbdir}" \
+ removesrc "${codename}" \
+ "${packagename}"
+}
+
repo_del_package() {
local dir="$1"
local dbdir="$2"
--
2.34.1
^ permalink raw reply [flat|nested] 17+ messages in thread* [PATCH 2/5] dpkg: Limit dsc searching by workdir
2023-09-15 18:54 [PATCH 0/5] Separate prebuild activities Anton Mikanovich
2023-09-15 18:54 ` [PATCH 1/5] repository: Add source package remove function Anton Mikanovich
@ 2023-09-15 18:54 ` Anton Mikanovich
2023-09-15 18:54 ` [PATCH 3/5] meta: Implement two stage build Anton Mikanovich
` (5 subsequent siblings)
7 siblings, 0 replies; 17+ messages in thread
From: Anton Mikanovich @ 2023-09-15 18:54 UTC (permalink / raw)
To: isar-users; +Cc: Anton Mikanovich
If local isar-apt copy has dsc file for current package it should be
ignored. This also prevent from any issues if source dir will have
matched files in it.
Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
meta/classes/dpkg.bbclass | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
index c596adbf..90d717ca 100644
--- a/meta/classes/dpkg.bbclass
+++ b/meta/classes/dpkg.bbclass
@@ -97,9 +97,9 @@ dpkg_runbuild() {
# Create a .dsc file from source directory to use it with sbuild
DEB_SOURCE_NAME=$(dpkg-parsechangelog --show-field Source --file ${WORKDIR}/${PPS}/debian/changelog)
- find ${WORKDIR} -name "${DEB_SOURCE_NAME}*.dsc" -delete
+ find ${WORKDIR} -name "${DEB_SOURCE_NAME}*.dsc" -maxdepth 1 -delete
sh -c "cd ${WORKDIR}; dpkg-source -q -b ${PPS}"
- DSC_FILE=$(find ${WORKDIR} -name "${DEB_SOURCE_NAME}*.dsc" -print)
+ DSC_FILE=$(find ${WORKDIR} -name "${DEB_SOURCE_NAME}*.dsc" -maxdepth 1 -print)
sbuild -A -n -c ${SBUILD_CHROOT} --extra-repository="${ISAR_APT_REPO}" \
--host=${PACKAGE_ARCH} --build=${BUILD_HOST_ARCH} ${profiles} \
--
2.34.1
^ permalink raw reply [flat|nested] 17+ messages in thread* [PATCH 3/5] meta: Implement two stage build
2023-09-15 18:54 [PATCH 0/5] Separate prebuild activities Anton Mikanovich
2023-09-15 18:54 ` [PATCH 1/5] repository: Add source package remove function Anton Mikanovich
2023-09-15 18:54 ` [PATCH 2/5] dpkg: Limit dsc searching by workdir Anton Mikanovich
@ 2023-09-15 18:54 ` Anton Mikanovich
2023-10-12 8:48 ` Jan Kiszka
2023-09-15 18:54 ` [PATCH 4/5] dpkg-base: Copy isar_apt to workdir in separate task Anton Mikanovich
` (4 subsequent siblings)
7 siblings, 1 reply; 17+ messages in thread
From: Anton Mikanovich @ 2023-09-15 18:54 UTC (permalink / raw)
To: isar-users; +Cc: Anton Mikanovich
Split package build into two stages: building source package and
building binary package. This allows to obtain Debian package
dependencies before any build steps will actually start.
Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
meta/classes/dpkg-gbp.bbclass | 2 +-
meta/classes/dpkg-source.bbclass | 33 ++++++++++++++++++++++
meta/classes/dpkg.bbclass | 4 +--
meta/classes/rootfs.bbclass | 2 ++
meta/recipes-kernel/linux/linux-custom.inc | 2 +-
5 files changed, 38 insertions(+), 5 deletions(-)
create mode 100644 meta/classes/dpkg-source.bbclass
diff --git a/meta/classes/dpkg-gbp.bbclass b/meta/classes/dpkg-gbp.bbclass
index 2bd09eaa..a151ade2 100644
--- a/meta/classes/dpkg-gbp.bbclass
+++ b/meta/classes/dpkg-gbp.bbclass
@@ -11,7 +11,7 @@ PATCHTOOL ?= "git"
GBP_EXTRA_OPTIONS ?= "--git-pristine-tar"
-dpkg_runbuild:prepend() {
+do_dpkg_source:prepend() {
sh -c "
cd ${WORKDIR}/${PPS}
gbp buildpackage --git-ignore-new --git-builder=/bin/true ${GBP_EXTRA_OPTIONS}
diff --git a/meta/classes/dpkg-source.bbclass b/meta/classes/dpkg-source.bbclass
new file mode 100644
index 00000000..434f1752
--- /dev/null
+++ b/meta/classes/dpkg-source.bbclass
@@ -0,0 +1,33 @@
+# This software is a part of ISAR.
+# Copyright (C) 2023 ilbers GmbH
+#
+# SPDX-License-Identifier: MIT
+
+inherit dpkg-base
+
+do_dpkg_source() {
+ # Create a .dsc file from source directory to use it with sbuild
+ DEB_SOURCE_NAME=$(dpkg-parsechangelog --show-field Source --file ${WORKDIR}/${PPS}/debian/changelog)
+ find ${WORKDIR} -name "${DEB_SOURCE_NAME}*.dsc" -maxdepth 1 -delete
+ if [ ! -z "${SOURCE_DATE_EPOCH}" ]; then
+ export SOURCE_DATE_EPOCH="${SOURCE_DATE_EPOCH}"
+ fi
+ sh -c "cd ${WORKDIR}; dpkg-source -I -b ${PPS}"
+}
+addtask dpkg_source after do_prepare_build before do_dpkg_build
+
+do_deploy_source[depends] += "isar-apt:do_cache_config"
+do_deploy_source[lockfiles] = "${REPO_ISAR_DIR}/isar.lock"
+do_deploy_source[dirs] = "${S}"
+do_deploy_source() {
+ repo_del_srcpackage "${REPO_ISAR_DIR}"/"${DISTRO}" \
+ "${REPO_ISAR_DB_DIR}"/"${DISTRO}" "${DEBDISTRONAME}" "${BPN}"
+ find "${S}/../" -name '*\.dsc' -maxdepth 1 | while read package; do
+ repo_add_srcpackage "${REPO_ISAR_DIR}"/"${DISTRO}" \
+ "${REPO_ISAR_DB_DIR}"/"${DISTRO}" \
+ "${DEBDISTRONAME}" \
+ "${package}"
+ done
+}
+addtask deploy_source after do_dpkg_source before do_dpkg_build
+
diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
index 90d717ca..2bb167e4 100644
--- a/meta/classes/dpkg.bbclass
+++ b/meta/classes/dpkg.bbclass
@@ -2,6 +2,7 @@
# Copyright (C) 2015-2018 ilbers GmbH
inherit dpkg-base
+inherit dpkg-source
PACKAGE_ARCH ?= "${DISTRO_ARCH}"
@@ -95,10 +96,7 @@ dpkg_runbuild() {
echo '$apt_keep_downloaded_packages = 1;' >> ${SBUILD_CONFIG}
- # Create a .dsc file from source directory to use it with sbuild
DEB_SOURCE_NAME=$(dpkg-parsechangelog --show-field Source --file ${WORKDIR}/${PPS}/debian/changelog)
- find ${WORKDIR} -name "${DEB_SOURCE_NAME}*.dsc" -maxdepth 1 -delete
- sh -c "cd ${WORKDIR}; dpkg-source -q -b ${PPS}"
DSC_FILE=$(find ${WORKDIR} -name "${DEB_SOURCE_NAME}*.dsc" -maxdepth 1 -print)
sbuild -A -n -c ${SBUILD_CHROOT} --extra-repository="${ISAR_APT_REPO}" \
diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass
index 7dec7b36..65574a6c 100644
--- a/meta/classes/rootfs.bbclass
+++ b/meta/classes/rootfs.bbclass
@@ -90,6 +90,8 @@ rootfs_configure_isar_apt() {
mkdir -p '${ROOTFSDIR}/etc/apt/sources.list.d'
echo 'deb [trusted=yes] file:///isar-apt ${DEBDISTRONAME} main' > \
'${ROOTFSDIR}/etc/apt/sources.list.d/isar-apt.list'
+ echo 'deb-src [trusted=yes] file:///isar-apt ${DEBDISTRONAME} main' >> \
+ '${ROOTFSDIR}/etc/apt/sources.list.d/isar-apt.list'
mkdir -p '${ROOTFSDIR}/etc/apt/preferences.d'
cat << EOF > '${ROOTFSDIR}/etc/apt/preferences.d/isar-apt'
diff --git a/meta/recipes-kernel/linux/linux-custom.inc b/meta/recipes-kernel/linux/linux-custom.inc
index 8b367a2f..fa5d957d 100644
--- a/meta/recipes-kernel/linux/linux-custom.inc
+++ b/meta/recipes-kernel/linux/linux-custom.inc
@@ -212,6 +212,6 @@ EOF
fi
}
-dpkg_runbuild:prepend() {
+do_dpkg_source:prepend() {
dpkg_configure_kernel
}
--
2.34.1
^ permalink raw reply [flat|nested] 17+ messages in thread* Re: [PATCH 3/5] meta: Implement two stage build
2023-09-15 18:54 ` [PATCH 3/5] meta: Implement two stage build Anton Mikanovich
@ 2023-10-12 8:48 ` Jan Kiszka
2023-10-12 9:07 ` Anton Mikanovich
2023-10-12 9:15 ` Jan Kiszka
0 siblings, 2 replies; 17+ messages in thread
From: Jan Kiszka @ 2023-10-12 8:48 UTC (permalink / raw)
To: Anton Mikanovich, isar-users
On 15.09.23 20:54, Anton Mikanovich wrote:
> Split package build into two stages: building source package and
> building binary package. This allows to obtain Debian package
> dependencies before any build steps will actually start.
>
> Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
> ---
> meta/classes/dpkg-gbp.bbclass | 2 +-
> meta/classes/dpkg-source.bbclass | 33 ++++++++++++++++++++++
> meta/classes/dpkg.bbclass | 4 +--
> meta/classes/rootfs.bbclass | 2 ++
> meta/recipes-kernel/linux/linux-custom.inc | 2 +-
> 5 files changed, 38 insertions(+), 5 deletions(-)
> create mode 100644 meta/classes/dpkg-source.bbclass
>
> diff --git a/meta/classes/dpkg-gbp.bbclass b/meta/classes/dpkg-gbp.bbclass
> index 2bd09eaa..a151ade2 100644
> --- a/meta/classes/dpkg-gbp.bbclass
> +++ b/meta/classes/dpkg-gbp.bbclass
> @@ -11,7 +11,7 @@ PATCHTOOL ?= "git"
>
> GBP_EXTRA_OPTIONS ?= "--git-pristine-tar"
>
> -dpkg_runbuild:prepend() {
> +do_dpkg_source:prepend() {
> sh -c "
> cd ${WORKDIR}/${PPS}
> gbp buildpackage --git-ignore-new --git-builder=/bin/true ${GBP_EXTRA_OPTIONS}
> diff --git a/meta/classes/dpkg-source.bbclass b/meta/classes/dpkg-source.bbclass
> new file mode 100644
> index 00000000..434f1752
> --- /dev/null
> +++ b/meta/classes/dpkg-source.bbclass
> @@ -0,0 +1,33 @@
> +# This software is a part of ISAR.
> +# Copyright (C) 2023 ilbers GmbH
> +#
> +# SPDX-License-Identifier: MIT
> +
> +inherit dpkg-base
> +
> +do_dpkg_source() {
> + # Create a .dsc file from source directory to use it with sbuild
> + DEB_SOURCE_NAME=$(dpkg-parsechangelog --show-field Source --file ${WORKDIR}/${PPS}/debian/changelog)
> + find ${WORKDIR} -name "${DEB_SOURCE_NAME}*.dsc" -maxdepth 1 -delete
> + if [ ! -z "${SOURCE_DATE_EPOCH}" ]; then
> + export SOURCE_DATE_EPOCH="${SOURCE_DATE_EPOCH}"
> + fi
> + sh -c "cd ${WORKDIR}; dpkg-source -I -b ${PPS}"
> +}
> +addtask dpkg_source after do_prepare_build before do_dpkg_build
> +
> +do_deploy_source[depends] += "isar-apt:do_cache_config"
> +do_deploy_source[lockfiles] = "${REPO_ISAR_DIR}/isar.lock"
> +do_deploy_source[dirs] = "${S}"
> +do_deploy_source() {
> + repo_del_srcpackage "${REPO_ISAR_DIR}"/"${DISTRO}" \
> + "${REPO_ISAR_DB_DIR}"/"${DISTRO}" "${DEBDISTRONAME}" "${BPN}"
> + find "${S}/../" -name '*\.dsc' -maxdepth 1 | while read package; do
> + repo_add_srcpackage "${REPO_ISAR_DIR}"/"${DISTRO}" \
> + "${REPO_ISAR_DB_DIR}"/"${DISTRO}" \
> + "${DEBDISTRONAME}" \
> + "${package}"
> + done
> +}
> +addtask deploy_source after do_dpkg_source before do_dpkg_build
> +
> diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
> index 90d717ca..2bb167e4 100644
> --- a/meta/classes/dpkg.bbclass
> +++ b/meta/classes/dpkg.bbclass
> @@ -2,6 +2,7 @@
> # Copyright (C) 2015-2018 ilbers GmbH
>
> inherit dpkg-base
> +inherit dpkg-source
>
> PACKAGE_ARCH ?= "${DISTRO_ARCH}"
>
> @@ -95,10 +96,7 @@ dpkg_runbuild() {
>
> echo '$apt_keep_downloaded_packages = 1;' >> ${SBUILD_CONFIG}
>
> - # Create a .dsc file from source directory to use it with sbuild
> DEB_SOURCE_NAME=$(dpkg-parsechangelog --show-field Source --file ${WORKDIR}/${PPS}/debian/changelog)
> - find ${WORKDIR} -name "${DEB_SOURCE_NAME}*.dsc" -maxdepth 1 -delete
> - sh -c "cd ${WORKDIR}; dpkg-source -q -b ${PPS}"
> DSC_FILE=$(find ${WORKDIR} -name "${DEB_SOURCE_NAME}*.dsc" -maxdepth 1 -print)
>
> sbuild -A -n -c ${SBUILD_CHROOT} --extra-repository="${ISAR_APT_REPO}" \
> diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass
> index 7dec7b36..65574a6c 100644
> --- a/meta/classes/rootfs.bbclass
> +++ b/meta/classes/rootfs.bbclass
> @@ -90,6 +90,8 @@ rootfs_configure_isar_apt() {
> mkdir -p '${ROOTFSDIR}/etc/apt/sources.list.d'
> echo 'deb [trusted=yes] file:///isar-apt ${DEBDISTRONAME} main' > \
> '${ROOTFSDIR}/etc/apt/sources.list.d/isar-apt.list'
> + echo 'deb-src [trusted=yes] file:///isar-apt ${DEBDISTRONAME} main' >> \
> + '${ROOTFSDIR}/etc/apt/sources.list.d/isar-apt.list'
>
> mkdir -p '${ROOTFSDIR}/etc/apt/preferences.d'
> cat << EOF > '${ROOTFSDIR}/etc/apt/preferences.d/isar-apt'
> diff --git a/meta/recipes-kernel/linux/linux-custom.inc b/meta/recipes-kernel/linux/linux-custom.inc
> index 8b367a2f..fa5d957d 100644
> --- a/meta/recipes-kernel/linux/linux-custom.inc
> +++ b/meta/recipes-kernel/linux/linux-custom.inc
> @@ -212,6 +212,6 @@ EOF
> fi
> }
>
> -dpkg_runbuild:prepend() {
> +do_dpkg_source:prepend() {
> dpkg_configure_kernel
> }
This is not transparent to recipe as the very last hunk nicely
demonstrates. Where is the RECIPE-API-CHANGLOG entry?
I thought this change was already the reason, but I'm still searching
for a regression of this patch here on our downstream recipe
https://gitlab.com/cip-project/cip-core/isar-cip-core/-/tree/master/recipes-bsp/edk2
(can be watched under
https://github.com/siemens/meta-iot2050/actions/runs/6492616563/job/17631910317.
Any hints welcome.
Jan
--
Siemens AG, Technology
Linux Expert Center
^ permalink raw reply [flat|nested] 17+ messages in thread* Re: [PATCH 3/5] meta: Implement two stage build
2023-10-12 8:48 ` Jan Kiszka
@ 2023-10-12 9:07 ` Anton Mikanovich
2023-10-12 11:12 ` Jan Kiszka
2023-10-12 9:15 ` Jan Kiszka
1 sibling, 1 reply; 17+ messages in thread
From: Anton Mikanovich @ 2023-10-12 9:07 UTC (permalink / raw)
To: Jan Kiszka, isar-users
12/10/2023 11:48, Jan Kiszka wrote:
> On 15.09.23 20:54, Anton Mikanovich wrote:
>> Split package build into two stages: building source package and
>> building binary package. This allows to obtain Debian package
>> dependencies before any build steps will actually start.
>>
>> Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
>> ---
>> meta/classes/dpkg-gbp.bbclass | 2 +-
>> meta/classes/dpkg-source.bbclass | 33 ++++++++++++++++++++++
>> meta/classes/dpkg.bbclass | 4 +--
>> meta/classes/rootfs.bbclass | 2 ++
>> meta/recipes-kernel/linux/linux-custom.inc | 2 +-
>> 5 files changed, 38 insertions(+), 5 deletions(-)
>> create mode 100644 meta/classes/dpkg-source.bbclass
>>
>> diff --git a/meta/classes/dpkg-gbp.bbclass b/meta/classes/dpkg-gbp.bbclass
>> index 2bd09eaa..a151ade2 100644
>> --- a/meta/classes/dpkg-gbp.bbclass
>> +++ b/meta/classes/dpkg-gbp.bbclass
>> @@ -11,7 +11,7 @@ PATCHTOOL ?= "git"
>>
>> GBP_EXTRA_OPTIONS ?= "--git-pristine-tar"
>>
>> -dpkg_runbuild:prepend() {
>> +do_dpkg_source:prepend() {
>> sh -c "
>> cd ${WORKDIR}/${PPS}
>> gbp buildpackage --git-ignore-new --git-builder=/bin/true ${GBP_EXTRA_OPTIONS}
>> diff --git a/meta/classes/dpkg-source.bbclass b/meta/classes/dpkg-source.bbclass
>> new file mode 100644
>> index 00000000..434f1752
>> --- /dev/null
>> +++ b/meta/classes/dpkg-source.bbclass
>> @@ -0,0 +1,33 @@
>> +# This software is a part of ISAR.
>> +# Copyright (C) 2023 ilbers GmbH
>> +#
>> +# SPDX-License-Identifier: MIT
>> +
>> +inherit dpkg-base
>> +
>> +do_dpkg_source() {
>> + # Create a .dsc file from source directory to use it with sbuild
>> + DEB_SOURCE_NAME=$(dpkg-parsechangelog --show-field Source --file ${WORKDIR}/${PPS}/debian/changelog)
>> + find ${WORKDIR} -name "${DEB_SOURCE_NAME}*.dsc" -maxdepth 1 -delete
>> + if [ ! -z "${SOURCE_DATE_EPOCH}" ]; then
>> + export SOURCE_DATE_EPOCH="${SOURCE_DATE_EPOCH}"
>> + fi
>> + sh -c "cd ${WORKDIR}; dpkg-source -I -b ${PPS}"
>> +}
>> +addtask dpkg_source after do_prepare_build before do_dpkg_build
>> +
>> +do_deploy_source[depends] += "isar-apt:do_cache_config"
>> +do_deploy_source[lockfiles] = "${REPO_ISAR_DIR}/isar.lock"
>> +do_deploy_source[dirs] = "${S}"
>> +do_deploy_source() {
>> + repo_del_srcpackage "${REPO_ISAR_DIR}"/"${DISTRO}" \
>> + "${REPO_ISAR_DB_DIR}"/"${DISTRO}" "${DEBDISTRONAME}" "${BPN}"
>> + find "${S}/../" -name '*\.dsc' -maxdepth 1 | while read package; do
>> + repo_add_srcpackage "${REPO_ISAR_DIR}"/"${DISTRO}" \
>> + "${REPO_ISAR_DB_DIR}"/"${DISTRO}" \
>> + "${DEBDISTRONAME}" \
>> + "${package}"
>> + done
>> +}
>> +addtask deploy_source after do_dpkg_source before do_dpkg_build
>> +
>> diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
>> index 90d717ca..2bb167e4 100644
>> --- a/meta/classes/dpkg.bbclass
>> +++ b/meta/classes/dpkg.bbclass
>> @@ -2,6 +2,7 @@
>> # Copyright (C) 2015-2018 ilbers GmbH
>>
>> inherit dpkg-base
>> +inherit dpkg-source
>>
>> PACKAGE_ARCH ?= "${DISTRO_ARCH}"
>>
>> @@ -95,10 +96,7 @@ dpkg_runbuild() {
>>
>> echo '$apt_keep_downloaded_packages = 1;' >> ${SBUILD_CONFIG}
>>
>> - # Create a .dsc file from source directory to use it with sbuild
>> DEB_SOURCE_NAME=$(dpkg-parsechangelog --show-field Source --file ${WORKDIR}/${PPS}/debian/changelog)
>> - find ${WORKDIR} -name "${DEB_SOURCE_NAME}*.dsc" -maxdepth 1 -delete
>> - sh -c "cd ${WORKDIR}; dpkg-source -q -b ${PPS}"
>> DSC_FILE=$(find ${WORKDIR} -name "${DEB_SOURCE_NAME}*.dsc" -maxdepth 1 -print)
>>
>> sbuild -A -n -c ${SBUILD_CHROOT} --extra-repository="${ISAR_APT_REPO}" \
>> diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass
>> index 7dec7b36..65574a6c 100644
>> --- a/meta/classes/rootfs.bbclass
>> +++ b/meta/classes/rootfs.bbclass
>> @@ -90,6 +90,8 @@ rootfs_configure_isar_apt() {
>> mkdir -p '${ROOTFSDIR}/etc/apt/sources.list.d'
>> echo 'deb [trusted=yes] file:///isar-apt ${DEBDISTRONAME} main' > \
>> '${ROOTFSDIR}/etc/apt/sources.list.d/isar-apt.list'
>> + echo 'deb-src [trusted=yes] file:///isar-apt ${DEBDISTRONAME} main' >> \
>> + '${ROOTFSDIR}/etc/apt/sources.list.d/isar-apt.list'
>>
>> mkdir -p '${ROOTFSDIR}/etc/apt/preferences.d'
>> cat << EOF > '${ROOTFSDIR}/etc/apt/preferences.d/isar-apt'
>> diff --git a/meta/recipes-kernel/linux/linux-custom.inc b/meta/recipes-kernel/linux/linux-custom.inc
>> index 8b367a2f..fa5d957d 100644
>> --- a/meta/recipes-kernel/linux/linux-custom.inc
>> +++ b/meta/recipes-kernel/linux/linux-custom.inc
>> @@ -212,6 +212,6 @@ EOF
>> fi
>> }
>>
>> -dpkg_runbuild:prepend() {
>> +do_dpkg_source:prepend() {
>> dpkg_configure_kernel
>> }
> This is not transparent to recipe as the very last hunk nicely
> demonstrates. Where is the RECIPE-API-CHANGLOG entry?
In p5.
> I thought this change was already the reason, but I'm still searching
> for a regression of this patch here on our downstream recipe
> https://gitlab.com/cip-project/cip-core/isar-cip-core/-/tree/master/recipes-bsp/edk2
> (can be watched under
> https://github.com/siemens/meta-iot2050/actions/runs/6492616563/job/17631910317.
This change shouldn't have any influence on the linker for such a
trivial recipe.
Are you sure this patchset is the only change between working and
non-working hashes?
> Any hints welcome.
>
> Jan
>
^ permalink raw reply [flat|nested] 17+ messages in thread* Re: [PATCH 3/5] meta: Implement two stage build
2023-10-12 9:07 ` Anton Mikanovich
@ 2023-10-12 11:12 ` Jan Kiszka
2023-10-12 14:39 ` Baurzhan Ismagulov
0 siblings, 1 reply; 17+ messages in thread
From: Jan Kiszka @ 2023-10-12 11:12 UTC (permalink / raw)
To: Anton Mikanovich, isar-users
On 12.10.23 11:07, Anton Mikanovich wrote:
> 12/10/2023 11:48, Jan Kiszka wrote:
>> On 15.09.23 20:54, Anton Mikanovich wrote:
>>> Split package build into two stages: building source package and
>>> building binary package. This allows to obtain Debian package
>>> dependencies before any build steps will actually start.
>>>
>>> Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
>>> ---
>>> meta/classes/dpkg-gbp.bbclass | 2 +-
>>> meta/classes/dpkg-source.bbclass | 33 ++++++++++++++++++++++
>>> meta/classes/dpkg.bbclass | 4 +--
>>> meta/classes/rootfs.bbclass | 2 ++
>>> meta/recipes-kernel/linux/linux-custom.inc | 2 +-
>>> 5 files changed, 38 insertions(+), 5 deletions(-)
>>> create mode 100644 meta/classes/dpkg-source.bbclass
>>>
>>> diff --git a/meta/classes/dpkg-gbp.bbclass
>>> b/meta/classes/dpkg-gbp.bbclass
>>> index 2bd09eaa..a151ade2 100644
>>> --- a/meta/classes/dpkg-gbp.bbclass
>>> +++ b/meta/classes/dpkg-gbp.bbclass
>>> @@ -11,7 +11,7 @@ PATCHTOOL ?= "git"
>>> GBP_EXTRA_OPTIONS ?= "--git-pristine-tar"
>>> -dpkg_runbuild:prepend() {
>>> +do_dpkg_source:prepend() {
>>> sh -c "
>>> cd ${WORKDIR}/${PPS}
>>> gbp buildpackage --git-ignore-new --git-builder=/bin/true
>>> ${GBP_EXTRA_OPTIONS}
>>> diff --git a/meta/classes/dpkg-source.bbclass
>>> b/meta/classes/dpkg-source.bbclass
>>> new file mode 100644
>>> index 00000000..434f1752
>>> --- /dev/null
>>> +++ b/meta/classes/dpkg-source.bbclass
>>> @@ -0,0 +1,33 @@
>>> +# This software is a part of ISAR.
>>> +# Copyright (C) 2023 ilbers GmbH
>>> +#
>>> +# SPDX-License-Identifier: MIT
>>> +
>>> +inherit dpkg-base
>>> +
>>> +do_dpkg_source() {
>>> + # Create a .dsc file from source directory to use it with sbuild
>>> + DEB_SOURCE_NAME=$(dpkg-parsechangelog --show-field Source --file
>>> ${WORKDIR}/${PPS}/debian/changelog)
>>> + find ${WORKDIR} -name "${DEB_SOURCE_NAME}*.dsc" -maxdepth 1 -delete
>>> + if [ ! -z "${SOURCE_DATE_EPOCH}" ]; then
>>> + export SOURCE_DATE_EPOCH="${SOURCE_DATE_EPOCH}"
>>> + fi
>>> + sh -c "cd ${WORKDIR}; dpkg-source -I -b ${PPS}"
>>> +}
>>> +addtask dpkg_source after do_prepare_build before do_dpkg_build
>>> +
>>> +do_deploy_source[depends] += "isar-apt:do_cache_config"
>>> +do_deploy_source[lockfiles] = "${REPO_ISAR_DIR}/isar.lock"
>>> +do_deploy_source[dirs] = "${S}"
>>> +do_deploy_source() {
>>> + repo_del_srcpackage "${REPO_ISAR_DIR}"/"${DISTRO}" \
>>> + "${REPO_ISAR_DB_DIR}"/"${DISTRO}" "${DEBDISTRONAME}" "${BPN}"
>>> + find "${S}/../" -name '*\.dsc' -maxdepth 1 | while read package; do
>>> + repo_add_srcpackage "${REPO_ISAR_DIR}"/"${DISTRO}" \
>>> + "${REPO_ISAR_DB_DIR}"/"${DISTRO}" \
>>> + "${DEBDISTRONAME}" \
>>> + "${package}"
>>> + done
>>> +}
>>> +addtask deploy_source after do_dpkg_source before do_dpkg_build
>>> +
>>> diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
>>> index 90d717ca..2bb167e4 100644
>>> --- a/meta/classes/dpkg.bbclass
>>> +++ b/meta/classes/dpkg.bbclass
>>> @@ -2,6 +2,7 @@
>>> # Copyright (C) 2015-2018 ilbers GmbH
>>> inherit dpkg-base
>>> +inherit dpkg-source
>>> PACKAGE_ARCH ?= "${DISTRO_ARCH}"
>>> @@ -95,10 +96,7 @@ dpkg_runbuild() {
>>> echo '$apt_keep_downloaded_packages = 1;' >> ${SBUILD_CONFIG}
>>> - # Create a .dsc file from source directory to use it with sbuild
>>> DEB_SOURCE_NAME=$(dpkg-parsechangelog --show-field Source
>>> --file ${WORKDIR}/${PPS}/debian/changelog)
>>> - find ${WORKDIR} -name "${DEB_SOURCE_NAME}*.dsc" -maxdepth 1 -delete
>>> - sh -c "cd ${WORKDIR}; dpkg-source -q -b ${PPS}"
>>> DSC_FILE=$(find ${WORKDIR} -name "${DEB_SOURCE_NAME}*.dsc"
>>> -maxdepth 1 -print)
>>> sbuild -A -n -c ${SBUILD_CHROOT}
>>> --extra-repository="${ISAR_APT_REPO}" \
>>> diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass
>>> index 7dec7b36..65574a6c 100644
>>> --- a/meta/classes/rootfs.bbclass
>>> +++ b/meta/classes/rootfs.bbclass
>>> @@ -90,6 +90,8 @@ rootfs_configure_isar_apt() {
>>> mkdir -p '${ROOTFSDIR}/etc/apt/sources.list.d'
>>> echo 'deb [trusted=yes] file:///isar-apt ${DEBDISTRONAME} main'
>>> > \
>>> '${ROOTFSDIR}/etc/apt/sources.list.d/isar-apt.list'
>>> + echo 'deb-src [trusted=yes] file:///isar-apt ${DEBDISTRONAME}
>>> main' >> \
>>> + '${ROOTFSDIR}/etc/apt/sources.list.d/isar-apt.list'
>>> mkdir -p '${ROOTFSDIR}/etc/apt/preferences.d'
>>> cat << EOF > '${ROOTFSDIR}/etc/apt/preferences.d/isar-apt'
>>> diff --git a/meta/recipes-kernel/linux/linux-custom.inc
>>> b/meta/recipes-kernel/linux/linux-custom.inc
>>> index 8b367a2f..fa5d957d 100644
>>> --- a/meta/recipes-kernel/linux/linux-custom.inc
>>> +++ b/meta/recipes-kernel/linux/linux-custom.inc
>>> @@ -212,6 +212,6 @@ EOF
>>> fi
>>> }
>>> -dpkg_runbuild:prepend() {
>>> +do_dpkg_source:prepend() {
>>> dpkg_configure_kernel
>>> }
>> This is not transparent to recipe as the very last hunk nicely
>> demonstrates. Where is the RECIPE-API-CHANGLOG entry?
> In p5.
>> I thought this change was already the reason, but I'm still searching
>> for a regression of this patch here on our downstream recipe
>> https://gitlab.com/cip-project/cip-core/isar-cip-core/-/tree/master/recipes-bsp/edk2
>> (can be watched under
>> https://github.com/siemens/meta-iot2050/actions/runs/6492616563/job/17631910317.
> This change shouldn't have any influence on the linker for such a
> trivial recipe.
> Are you sure this patchset is the only change between working and
> non-working hashes?
I found the reason: You added -I to dpkg-source, and that filters out
some questionable but needed prebuilt .a files in edk2. All fine again
when dropping it. I'll send a patch, we should not do that here.
Jan
--
Siemens AG, Technology
Linux Expert Center
^ permalink raw reply [flat|nested] 17+ messages in thread* Re: [PATCH 3/5] meta: Implement two stage build
2023-10-12 11:12 ` Jan Kiszka
@ 2023-10-12 14:39 ` Baurzhan Ismagulov
2023-10-12 14:53 ` Jan Kiszka
0 siblings, 1 reply; 17+ messages in thread
From: Baurzhan Ismagulov @ 2023-10-12 14:39 UTC (permalink / raw)
To: isar-users; +Cc: Jan Kiszka
On 2023-10-12 13:12, 'Jan Kiszka' via isar-users wrote:
> I found the reason: You added -I to dpkg-source, and that filters out
> some questionable but needed prebuilt .a files in edk2. All fine again
> when dropping it. I'll send a patch, we should not do that here.
Thanks Jan for the analysis, this explains it.
That said, Debian doesn't want source packages to contain binary libraries or
.git files for a good reason (we were hit by the latter), so removing -I
altogether will break other stuff. We will quickly provide an option to drop -I
per recipe, which will solve this problem without affecting all recipes.
With kind regards,
Baurzhan
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [PATCH 3/5] meta: Implement two stage build
2023-10-12 14:39 ` Baurzhan Ismagulov
@ 2023-10-12 14:53 ` Jan Kiszka
2023-10-12 15:20 ` Baurzhan Ismagulov
0 siblings, 1 reply; 17+ messages in thread
From: Jan Kiszka @ 2023-10-12 14:53 UTC (permalink / raw)
To: isar-users
On 12.10.23 16:39, Baurzhan Ismagulov wrote:
> On 2023-10-12 13:12, 'Jan Kiszka' via isar-users wrote:
>> I found the reason: You added -I to dpkg-source, and that filters out
>> some questionable but needed prebuilt .a files in edk2. All fine again
>> when dropping it. I'll send a patch, we should not do that here.
>
> Thanks Jan for the analysis, this explains it.
>
> That said, Debian doesn't want source packages to contain binary libraries or
> .git files for a good reason (we were hit by the latter), so removing -I
> altogether will break other stuff. We will quickly provide an option to drop -I
> per recipe, which will solve this problem without affecting all recipes.
Yes, that's why we need to solve things differently on the long run.
First, this is an Isar regression we need to fix: You changed behavior
in the middle of some harmless refactoring that breaks downstream
subtly. Not a good idea.
Jan
--
Siemens AG, Technology
Linux Expert Center
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [PATCH 3/5] meta: Implement two stage build
2023-10-12 14:53 ` Jan Kiszka
@ 2023-10-12 15:20 ` Baurzhan Ismagulov
0 siblings, 0 replies; 17+ messages in thread
From: Baurzhan Ismagulov @ 2023-10-12 15:20 UTC (permalink / raw)
To: isar-users
On 2023-10-12 16:53, 'Jan Kiszka' via isar-users wrote:
> Yes, that's why we need to solve things differently on the long run.
> First, this is an Isar regression we need to fix: You changed behavior
> in the middle of some harmless refactoring that breaks downstream
> subtly. Not a good idea.
I see. -I being motivated by a real problem, I agree the patch landing in the
refactoring series is unfortunate. In the future, such changes should be sent
separately.
With kind regards,
Baurzhan
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [PATCH 3/5] meta: Implement two stage build
2023-10-12 8:48 ` Jan Kiszka
2023-10-12 9:07 ` Anton Mikanovich
@ 2023-10-12 9:15 ` Jan Kiszka
1 sibling, 0 replies; 17+ messages in thread
From: Jan Kiszka @ 2023-10-12 9:15 UTC (permalink / raw)
To: Anton Mikanovich, isar-users
On 12.10.23 10:48, Jan Kiszka wrote:
> On 15.09.23 20:54, Anton Mikanovich wrote:
>> Split package build into two stages: building source package and
>> building binary package. This allows to obtain Debian package
>> dependencies before any build steps will actually start.
>>
>> Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
>> ---
>> meta/classes/dpkg-gbp.bbclass | 2 +-
>> meta/classes/dpkg-source.bbclass | 33 ++++++++++++++++++++++
>> meta/classes/dpkg.bbclass | 4 +--
>> meta/classes/rootfs.bbclass | 2 ++
>> meta/recipes-kernel/linux/linux-custom.inc | 2 +-
>> 5 files changed, 38 insertions(+), 5 deletions(-)
>> create mode 100644 meta/classes/dpkg-source.bbclass
>>
>> diff --git a/meta/classes/dpkg-gbp.bbclass b/meta/classes/dpkg-gbp.bbclass
>> index 2bd09eaa..a151ade2 100644
>> --- a/meta/classes/dpkg-gbp.bbclass
>> +++ b/meta/classes/dpkg-gbp.bbclass
>> @@ -11,7 +11,7 @@ PATCHTOOL ?= "git"
>>
>> GBP_EXTRA_OPTIONS ?= "--git-pristine-tar"
>>
>> -dpkg_runbuild:prepend() {
>> +do_dpkg_source:prepend() {
>> sh -c "
>> cd ${WORKDIR}/${PPS}
>> gbp buildpackage --git-ignore-new --git-builder=/bin/true ${GBP_EXTRA_OPTIONS}
>> diff --git a/meta/classes/dpkg-source.bbclass b/meta/classes/dpkg-source.bbclass
>> new file mode 100644
>> index 00000000..434f1752
>> --- /dev/null
>> +++ b/meta/classes/dpkg-source.bbclass
>> @@ -0,0 +1,33 @@
>> +# This software is a part of ISAR.
>> +# Copyright (C) 2023 ilbers GmbH
>> +#
>> +# SPDX-License-Identifier: MIT
>> +
>> +inherit dpkg-base
>> +
>> +do_dpkg_source() {
>> + # Create a .dsc file from source directory to use it with sbuild
>> + DEB_SOURCE_NAME=$(dpkg-parsechangelog --show-field Source --file ${WORKDIR}/${PPS}/debian/changelog)
>> + find ${WORKDIR} -name "${DEB_SOURCE_NAME}*.dsc" -maxdepth 1 -delete
>> + if [ ! -z "${SOURCE_DATE_EPOCH}" ]; then
>> + export SOURCE_DATE_EPOCH="${SOURCE_DATE_EPOCH}"
>> + fi
>> + sh -c "cd ${WORKDIR}; dpkg-source -I -b ${PPS}"
>> +}
>> +addtask dpkg_source after do_prepare_build before do_dpkg_build
>> +
>> +do_deploy_source[depends] += "isar-apt:do_cache_config"
>> +do_deploy_source[lockfiles] = "${REPO_ISAR_DIR}/isar.lock"
>> +do_deploy_source[dirs] = "${S}"
>> +do_deploy_source() {
>> + repo_del_srcpackage "${REPO_ISAR_DIR}"/"${DISTRO}" \
>> + "${REPO_ISAR_DB_DIR}"/"${DISTRO}" "${DEBDISTRONAME}" "${BPN}"
>> + find "${S}/../" -name '*\.dsc' -maxdepth 1 | while read package; do
>> + repo_add_srcpackage "${REPO_ISAR_DIR}"/"${DISTRO}" \
>> + "${REPO_ISAR_DB_DIR}"/"${DISTRO}" \
>> + "${DEBDISTRONAME}" \
>> + "${package}"
>> + done
>> +}
>> +addtask deploy_source after do_dpkg_source before do_dpkg_build
>> +
>> diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
>> index 90d717ca..2bb167e4 100644
>> --- a/meta/classes/dpkg.bbclass
>> +++ b/meta/classes/dpkg.bbclass
>> @@ -2,6 +2,7 @@
>> # Copyright (C) 2015-2018 ilbers GmbH
>>
>> inherit dpkg-base
>> +inherit dpkg-source
>>
>> PACKAGE_ARCH ?= "${DISTRO_ARCH}"
>>
>> @@ -95,10 +96,7 @@ dpkg_runbuild() {
>>
>> echo '$apt_keep_downloaded_packages = 1;' >> ${SBUILD_CONFIG}
>>
>> - # Create a .dsc file from source directory to use it with sbuild
>> DEB_SOURCE_NAME=$(dpkg-parsechangelog --show-field Source --file ${WORKDIR}/${PPS}/debian/changelog)
>> - find ${WORKDIR} -name "${DEB_SOURCE_NAME}*.dsc" -maxdepth 1 -delete
>> - sh -c "cd ${WORKDIR}; dpkg-source -q -b ${PPS}"
>> DSC_FILE=$(find ${WORKDIR} -name "${DEB_SOURCE_NAME}*.dsc" -maxdepth 1 -print)
>>
>> sbuild -A -n -c ${SBUILD_CHROOT} --extra-repository="${ISAR_APT_REPO}" \
>> diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass
>> index 7dec7b36..65574a6c 100644
>> --- a/meta/classes/rootfs.bbclass
>> +++ b/meta/classes/rootfs.bbclass
>> @@ -90,6 +90,8 @@ rootfs_configure_isar_apt() {
>> mkdir -p '${ROOTFSDIR}/etc/apt/sources.list.d'
>> echo 'deb [trusted=yes] file:///isar-apt ${DEBDISTRONAME} main' > \
>> '${ROOTFSDIR}/etc/apt/sources.list.d/isar-apt.list'
>> + echo 'deb-src [trusted=yes] file:///isar-apt ${DEBDISTRONAME} main' >> \
>> + '${ROOTFSDIR}/etc/apt/sources.list.d/isar-apt.list'
>>
>> mkdir -p '${ROOTFSDIR}/etc/apt/preferences.d'
>> cat << EOF > '${ROOTFSDIR}/etc/apt/preferences.d/isar-apt'
>> diff --git a/meta/recipes-kernel/linux/linux-custom.inc b/meta/recipes-kernel/linux/linux-custom.inc
>> index 8b367a2f..fa5d957d 100644
>> --- a/meta/recipes-kernel/linux/linux-custom.inc
>> +++ b/meta/recipes-kernel/linux/linux-custom.inc
>> @@ -212,6 +212,6 @@ EOF
>> fi
>> }
>>
>> -dpkg_runbuild:prepend() {
>> +do_dpkg_source:prepend() {
>> dpkg_configure_kernel
>> }
>
> This is not transparent to recipe as the very last hunk nicely
> demonstrates. Where is the RECIPE-API-CHANGLOG entry?
>
OK, found it later on. But "in some cases" is not a helpful explanation.
I would have expected that all extensions of dpkg_runbuild:prepend
touched ${S} need to be moved, or so.
Jan
> I thought this change was already the reason, but I'm still searching
> for a regression of this patch here on our downstream recipe
> https://gitlab.com/cip-project/cip-core/isar-cip-core/-/tree/master/recipes-bsp/edk2
> (can be watched under
> https://github.com/siemens/meta-iot2050/actions/runs/6492616563/job/17631910317.
> Any hints welcome.
>
> Jan
>
--
Siemens AG, Technology
Linux Expert Center
^ permalink raw reply [flat|nested] 17+ messages in thread
* [PATCH 4/5] dpkg-base: Copy isar_apt to workdir in separate task
2023-09-15 18:54 [PATCH 0/5] Separate prebuild activities Anton Mikanovich
` (2 preceding siblings ...)
2023-09-15 18:54 ` [PATCH 3/5] meta: Implement two stage build Anton Mikanovich
@ 2023-09-15 18:54 ` Anton Mikanovich
2023-09-15 18:54 ` [PATCH 5/5] RECIPE-API-CHANGELOG.md: Note tasks separation Anton Mikanovich
` (3 subsequent siblings)
7 siblings, 0 replies; 17+ messages in thread
From: Anton Mikanovich @ 2023-09-15 18:54 UTC (permalink / raw)
To: isar-users; +Cc: Anton Mikanovich
As only isar_apt copy functionality needs repo locking and dependencies
building it can be placed into separate task to simplify tasks
requirements.
Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
meta/classes/dpkg-base.bbclass | 15 +++++++--------
1 file changed, 7 insertions(+), 8 deletions(-)
diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index 7b02f378..f223fc16 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -186,10 +186,10 @@ do_prepare_build() {
addtask prepare_build after do_patch do_transform_template before do_dpkg_build
# If Isar recipes depend on each other, they typically need the package
# deployed to isar-apt
-do_prepare_build[deptask] = "do_deploy_deb"
-do_prepare_build[depends] = "${SCHROOT_DEP}"
-
-do_prepare_build:append() {
+do_local_isarapt[depends] += "isar-apt:do_cache_config"
+do_local_isarapt[deptask] = "do_deploy_deb"
+do_local_isarapt[lockfiles] += "${REPO_ISAR_DIR}/isar.lock"
+do_local_isarapt() {
# Make a local copy of isar-apt repo that is not affected by other parallel builds
rm -rf "${WORKDIR}/isar-apt/${DISTRO}-${DISTRO_ARCH}/*"
mkdir -p "${WORKDIR}/isar-apt/${DISTRO}-${DISTRO_ARCH}/apt/${DISTRO}"
@@ -198,8 +198,7 @@ do_prepare_build:append() {
cp -Rlf "${REPO_ISAR_DIR}/${DISTRO}/pool" "${WORKDIR}/isar-apt/${DISTRO}-${DISTRO_ARCH}/apt/${DISTRO}/"
fi
}
-
-do_prepare_build[lockfiles] += "${REPO_ISAR_DIR}/isar.lock"
+addtask local_isarapt before do_dpkg_build
# Placeholder for actual dpkg_runbuild() implementation
dpkg_runbuild() {
@@ -324,7 +323,7 @@ python do_devshell() {
bb.build.exec_func('schroot_delete_configs', d)
}
-addtask devshell after do_prepare_build
+addtask devshell after do_local_isarapt
DEVSHELL_STARTDIR ?= "${S}"
do_devshell[dirs] = "${DEVSHELL_STARTDIR}"
do_devshell[nostamp] = "1"
@@ -336,7 +335,7 @@ python do_devshell_nodeps() {
# devshell may be placed after do_instell_builddeps in downstream classes.
# devshell_nodeps will always stay right after do_prepare_build.
-addtask devshell_nodeps after do_prepare_build
+addtask devshell_nodeps after do_local_isarapt
do_devshell_nodeps[dirs] = "${DEVSHELL_STARTDIR}"
do_devshell_nodeps[nostamp] = "1"
do_devshell_nodeps[network] = "${TASK_USE_SUDO}"
--
2.34.1
^ permalink raw reply [flat|nested] 17+ messages in thread* [PATCH 5/5] RECIPE-API-CHANGELOG.md: Note tasks separation
2023-09-15 18:54 [PATCH 0/5] Separate prebuild activities Anton Mikanovich
` (3 preceding siblings ...)
2023-09-15 18:54 ` [PATCH 4/5] dpkg-base: Copy isar_apt to workdir in separate task Anton Mikanovich
@ 2023-09-15 18:54 ` Anton Mikanovich
2023-09-21 13:45 ` [PATCH 0/5] Separate prebuild activities Anton Mikanovich
` (2 subsequent siblings)
7 siblings, 0 replies; 17+ messages in thread
From: Anton Mikanovich @ 2023-09-15 18:54 UTC (permalink / raw)
To: isar-users; +Cc: Anton Mikanovich
Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
RECIPE-API-CHANGELOG.md | 14 ++++++++++++++
1 file changed, 14 insertions(+)
diff --git a/RECIPE-API-CHANGELOG.md b/RECIPE-API-CHANGELOG.md
index 659e64c6..7d463473 100644
--- a/RECIPE-API-CHANGELOG.md
+++ b/RECIPE-API-CHANGELOG.md
@@ -525,3 +525,17 @@ If privileged execution is required `-u root` option can be added.
```
SCHROOT_MOUNTS += "${OUT_PATH1}:${IN_PATH1} ${OUT_PATH2}"
```
+
+### Building source package tasks were separated
+
+Two new tasks are introduced: do_dpkg_source and do_deploy_source. The first
+one builds source package, second one adds this package to isar_apt.
+They are placed between do_prepare_build and do_dpkg_build.
+Now all source modifications should be done inside do_prepare_build task, in
+some cases dpkg_runbuild:prepend should be replaced by do_dpkg_source:prepend.
+
+### Local copy of isar-apt creation task was separated
+
+We need local copy of isar-apt to have build dependencies reachable. Now is
+prepared in separate task: do_local_isarapt.
+This task depends of do_deploy_deb of all build dependency recipes.
--
2.34.1
^ permalink raw reply [flat|nested] 17+ messages in thread* Re: [PATCH 0/5] Separate prebuild activities
2023-09-15 18:54 [PATCH 0/5] Separate prebuild activities Anton Mikanovich
` (4 preceding siblings ...)
2023-09-15 18:54 ` [PATCH 5/5] RECIPE-API-CHANGELOG.md: Note tasks separation Anton Mikanovich
@ 2023-09-21 13:45 ` Anton Mikanovich
2023-09-29 6:43 ` Uladzimir Bely
2023-10-30 4:44 ` MOESSBAUER, Felix
7 siblings, 0 replies; 17+ messages in thread
From: Anton Mikanovich @ 2023-09-21 13:45 UTC (permalink / raw)
To: isar-users
15/09/2023 21:54, Anton Mikanovich wrote:
> This allows to execute tasks more efficient. On simple targets like qemuamd64
> the speed up of the build is around 10 second, but on complex downstream layers
> this effect can be more visible.
I've just checked isar-cip-core downstream and got 30s speedup from 3m30s
original overall building time.
^ permalink raw reply [flat|nested] 17+ messages in thread* Re: [PATCH 0/5] Separate prebuild activities
2023-09-15 18:54 [PATCH 0/5] Separate prebuild activities Anton Mikanovich
` (5 preceding siblings ...)
2023-09-21 13:45 ` [PATCH 0/5] Separate prebuild activities Anton Mikanovich
@ 2023-09-29 6:43 ` Uladzimir Bely
2023-10-30 4:44 ` MOESSBAUER, Felix
7 siblings, 0 replies; 17+ messages in thread
From: Uladzimir Bely @ 2023-09-29 6:43 UTC (permalink / raw)
To: Anton Mikanovich, isar-users
On Fri, 2023-09-15 at 21:54 +0300, Anton Mikanovich wrote:
> This patchset introduce three new pre-build tasks from the code was
> already
> done during the build task:
> do_dpkg_source
> do_deploy_source
> do_local_isarapt
> This allows to execute tasks more efficient. On simple targets like
> qemuamd64
> the speed up of the build is around 10 second, but on complex
> downstream layers
> this effect can be more visible.
> We also introduce source packages storage in isar-apt repository
> which allows
> obtaining package Build-Depends and Provides information in Debian
> way later.
>
> Anton Mikanovich (5):
> repository: Add source package remove function
> dpkg: Limit dsc searching by workdir
> meta: Implement two stage build
> dpkg-base: Copy isar_apt to workdir in separate task
> RECIPE-API-CHANGELOG.md: Note tasks separation
>
> RECIPE-API-CHANGELOG.md | 14 +++++++++
> meta/classes/dpkg-base.bbclass | 15 +++++-----
> meta/classes/dpkg-gbp.bbclass | 2 +-
> meta/classes/dpkg-source.bbclass | 33
> ++++++++++++++++++++++
> meta/classes/dpkg.bbclass | 6 ++--
> meta/classes/repository.bbclass | 14 +++++++++
> meta/classes/rootfs.bbclass | 2 ++
> meta/recipes-kernel/linux/linux-custom.inc | 2 +-
> 8 files changed, 74 insertions(+), 14 deletions(-)
> create mode 100644 meta/classes/dpkg-source.bbclass
>
> --
> 2.34.1
>
Applied to next.
^ permalink raw reply [flat|nested] 17+ messages in thread* Re: [PATCH 0/5] Separate prebuild activities
2023-09-15 18:54 [PATCH 0/5] Separate prebuild activities Anton Mikanovich
` (6 preceding siblings ...)
2023-09-29 6:43 ` Uladzimir Bely
@ 2023-10-30 4:44 ` MOESSBAUER, Felix
2023-10-30 8:00 ` Schmidt, Adriaan
7 siblings, 1 reply; 17+ messages in thread
From: MOESSBAUER, Felix @ 2023-10-30 4:44 UTC (permalink / raw)
To: amikan, isar-users, Schmidt, Adriaan
On Fri, 2023-09-15 at 21:54 +0300, Anton Mikanovich wrote:
> This patchset introduce three new pre-build tasks from the code was
> already
> done during the build task:
> do_dpkg_source
> do_deploy_source
We need to add sstate caching to the dpkg_source task as well.
Currently we see some significant slowdown on big source packages (like
the kernel) on otherwise fully cache-able builds.
Maybe Adriaan could add this, as he added the caching for dpkg_build as
well.
Best regards,
Felix
> do_local_isarapt
> This allows to execute tasks more efficient. On simple targets like
> qemuamd64
> the speed up of the build is around 10 second, but on complex
> downstream layers
> this effect can be more visible.
> We also introduce source packages storage in isar-apt repository
> which allows
> obtaining package Build-Depends and Provides information in Debian
> way later.
>
> Anton Mikanovich (5):
> repository: Add source package remove function
> dpkg: Limit dsc searching by workdir
> meta: Implement two stage build
> dpkg-base: Copy isar_apt to workdir in separate task
> RECIPE-API-CHANGELOG.md: Note tasks separation
>
> RECIPE-API-CHANGELOG.md | 14 +++++++++
> meta/classes/dpkg-base.bbclass | 15 +++++-----
> meta/classes/dpkg-gbp.bbclass | 2 +-
> meta/classes/dpkg-source.bbclass | 33
> ++++++++++++++++++++++
> meta/classes/dpkg.bbclass | 6 ++--
> meta/classes/repository.bbclass | 14 +++++++++
> meta/classes/rootfs.bbclass | 2 ++
> meta/recipes-kernel/linux/linux-custom.inc | 2 +-
> 8 files changed, 74 insertions(+), 14 deletions(-)
> create mode 100644 meta/classes/dpkg-source.bbclass
>
> --
> 2.34.1
>
^ permalink raw reply [flat|nested] 17+ messages in thread* RE: [PATCH 0/5] Separate prebuild activities
2023-10-30 4:44 ` MOESSBAUER, Felix
@ 2023-10-30 8:00 ` Schmidt, Adriaan
0 siblings, 0 replies; 17+ messages in thread
From: Schmidt, Adriaan @ 2023-10-30 8:00 UTC (permalink / raw)
To: MOESSBAUER, FELIX JONATHAN, amikan, isar-users
MOESSBAUER, Felix (T CED INW-CN), Monday, October 30, 2023 5:45 AM:
> On Fri, 2023-09-15 at 21:54 +0300, Anton Mikanovich wrote:
> > This patchset introduce three new pre-build tasks from the code was
> > already
> > done during the build task:
> > do_dpkg_source
> > do_deploy_source
>
> We need to add sstate caching to the dpkg_source task as well.
> Currently we see some significant slowdown on big source packages (like
> the kernel) on otherwise fully cache-able builds.
>
> Maybe Adriaan could add this, as he added the caching for dpkg_build as
> well.
Well, currently I expect that whenever we have a situation in which dpkg_build
would have to run (most likely a change in the recipe), that would mean
we need to rebuild the source as well. So I'm skeptical caching the source
would help much at this point.
BUT:
If we take the two-stage build one step further and actually have dpkg_build
work exclusively with the generated source package from isar-apt as input,
we could re-use one src package for all targets and variants (-compat/-native).
I'm currently working in this direction (prompted by bugs we see when installing
both "pkg" and "pkg-compat" in the same image), and so far I've found:
- We currently see a side-effect in the "remove source package" function:
"reprepro removesrc" not only removes the src package, but also all related binary
packages from the repository. So we need to make sure that only one
task in a build deploys a src package (and that src is deployed before any binaries).
Currently the source would be deployed by both "pkg" and "pkg-compat", with a
resulting race condition. Already preparing a patch for that.
- The -compat/-native suffix sometimes "leaks" into the source package.
Basically, there are still locations that should use "${BPN}" instead of "${PN}".
Already preparing a patch for that.
- currently there are still tasks/dependencies of do_dpkg_build that rely on
the extracted source being present (e.g. finding out the name of the src
package)
- we'd need to add a deb-src entry to the apt sources for sbuild
- to actually be able to share src packages across architectures, we'd need
to make sure that the src packages are always identical, and don't depend
on any architecture or -compat/-native variant. I guess in the Debian world
that already is a rule, but in bitbake users can of course add any number
of overrides (e.g. changing SRC_URI depending on PACKAGE_ARCH) to break
this rule...
Adriaan
> Best regards,
> Felix
>
> > do_local_isarapt
> > This allows to execute tasks more efficient. On simple targets like
> > qemuamd64
> > the speed up of the build is around 10 second, but on complex
> > downstream layers
> > this effect can be more visible.
> > We also introduce source packages storage in isar-apt repository
> > which allows
> > obtaining package Build-Depends and Provides information in Debian
> > way later.
> >
> > Anton Mikanovich (5):
> > repository: Add source package remove function
> > dpkg: Limit dsc searching by workdir
> > meta: Implement two stage build
> > dpkg-base: Copy isar_apt to workdir in separate task
> > RECIPE-API-CHANGELOG.md: Note tasks separation
> >
> > RECIPE-API-CHANGELOG.md | 14 +++++++++
> > meta/classes/dpkg-base.bbclass | 15 +++++-----
> > meta/classes/dpkg-gbp.bbclass | 2 +-
> > meta/classes/dpkg-source.bbclass | 33
> > ++++++++++++++++++++++
> > meta/classes/dpkg.bbclass | 6 ++--
> > meta/classes/repository.bbclass | 14 +++++++++
> > meta/classes/rootfs.bbclass | 2 ++
> > meta/recipes-kernel/linux/linux-custom.inc | 2 +-
> > 8 files changed, 74 insertions(+), 14 deletions(-)
> > create mode 100644 meta/classes/dpkg-source.bbclass
> >
> > --
> > 2.34.1
> >
>
^ permalink raw reply [flat|nested] 17+ messages in thread