Re: [PATCH 04/26] Rework STAMP layout and cleaning

[Jan Kiszka <jan.kiszka@siemens.com>]

On 26.08.19 10:30, Claudius Heine wrote:
> Hi Jan,
> 
> On 26/08/2019 08.26, [ext] Jan Kiszka wrote:
>> From: Jan Kiszka <jan.kiszka@siemens.com>
>>
>> Follow OE more closely by aligning the stamps folder structure to
>> workdir. This may resolve false sharing of stamps between recipes of
>> common PF but different DISTRO/DISTRO_ARCH without the otherwise
>> required stamp-extra-info. That extra-info can now be removed.
>>
>> As long as recipes mess with PF, we can't use OE's pattern as-is which
>> is PN-based.
>>
>> Furthermore add proper STAMPCLEAN. This ensures that stamp files are
>> cleaned when a task is rebuilt. Not providing this pattern likely caused
>> a lot of failures when doing partial rebuilds.
>>
>> As we are at it, switch do_clean to STAMPCLEAN as well.
>>
>> Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
>> ---
>>   meta/classes/base.bbclass                  | 5 ++---
>>   meta/classes/dpkg-base.bbclass             | 6 ------
>>   meta/classes/dpkg-raw.bbclass              | 1 -
>>   meta/classes/dpkg.bbclass                  | 1 -
>>   meta/classes/patch.bbclass                 | 2 --
>>   meta/classes/template.bbclass              | 1 -
>>   meta/conf/bitbake.conf                     | 4 +++-
>>   meta/recipes-kernel/linux/linux-custom.inc | 1 -
>>   8 files changed, 5 insertions(+), 16 deletions(-)
>>
>> diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass
>> index ca80bf8..1d2852a 100644
>> --- a/meta/classes/base.bbclass
>> +++ b/meta/classes/base.bbclass
>> @@ -163,7 +163,6 @@ python do_fetch() {
>>   addtask fetch before do_build
>>   do_unpack[dirs] = "${WORKDIR}"
>> -do_unpack[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
>>   # Unpack package and put it into working directory
>>   python do_unpack() {
>> @@ -206,8 +205,8 @@ python do_clean() {
>>       workdir = d.expand("${WORKDIR}")
>>       subprocess.check_call(["sudo", "rm", "-rf", workdir])
>> -    stamppath = bb.data.expand(d.getVar('STAMP', False), d)
>> -    stampdirs = glob.glob(stamppath + ".*")
>> +    stampclean = bb.data.expand(d.getVar('STAMPCLEAN', False), d)
>> +    stampdirs = glob.glob(stampclean)
>>       subprocess.check_call(["sudo", "rm", "-rf"] + stampdirs)
>>   }
>> diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
>> index 39c8acb..2d26418 100644
>> --- a/meta/classes/dpkg-base.bbclass
>> +++ b/meta/classes/dpkg-base.bbclass
>> @@ -16,7 +16,6 @@ do_adjust_git() {
>>   }
>>   addtask adjust_git after do_unpack before do_patch
>> -do_adjust_git[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
>>   inherit patch
>>   addtask patch after do_adjust_git before do_build
>> @@ -42,7 +41,6 @@ do_apt_fetch() {
>>   addtask apt_fetch after do_unpack before do_patch
>>   do_apt_fetch[lockfiles] += "${REPO_ISAR_DIR}/isar.lock"
>> -do_apt_fetch[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
>>   def get_package_srcdir(d):
>>       s = os.path.abspath(d.getVar("S", True))
>> @@ -60,15 +58,12 @@ def get_package_srcdir(d):
>>   PP = "/home/builder/${PN}"
>>   PPS ?= "${@get_package_srcdir(d)}"
>> -do_build[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
>> -
>>   # Empty do_prepare_build() implementation, to be overwritten if needed
>>   do_prepare_build() {
>>       true
>>   }
>>   addtask prepare_build after do_patch do_transform_template before do_build
>> -do_prepare_build[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
>>   # If Isar recipes depend on each other, they typically need the package
>>   # deployed to isar-apt
>>   do_prepare_build[deptask] = "do_deploy_deb"
>> @@ -132,7 +127,6 @@ do_deploy_deb() {
>>   }
>>   addtask deploy_deb after do_build
>> -do_deploy_deb[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
>>   do_deploy_deb[lockfiles] = "${REPO_ISAR_DIR}/isar.lock"
>>   do_deploy_deb[depends] = "isar-apt:do_cache_config"
>>   do_deploy_deb[dirs] = "${S}"
>> diff --git a/meta/classes/dpkg-raw.bbclass b/meta/classes/dpkg-raw.bbclass
>> index ea03ea4..f8dfb2d 100644
>> --- a/meta/classes/dpkg-raw.bbclass
>> +++ b/meta/classes/dpkg-raw.bbclass
>> @@ -16,7 +16,6 @@ do_install() {
>>   }
>>   do_install[cleandirs] = "${D}"
>> -do_install[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
>>   addtask install after do_unpack before do_prepare_build
>>   do_prepare_build[cleandirs] += "${D}/debian"
>> diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
>> index 997f2f3..dd123c3 100644
>> --- a/meta/classes/dpkg.bbclass
>> +++ b/meta/classes/dpkg.bbclass
>> @@ -14,7 +14,6 @@ do_install_builddeps() {
>>   addtask install_builddeps after do_prepare_build before do_build
>>   # apt and reprepro may not run in parallel, acquire the Isar lock
>>   do_install_builddeps[lockfiles] += "${REPO_ISAR_DIR}/isar.lock"
>> -do_install_builddeps[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
>>   # Build package from sources using build script
>>   dpkg_runbuild() {
>> diff --git a/meta/classes/patch.bbclass b/meta/classes/patch.bbclass
>> index 2662702..c5ba463 100644
>> --- a/meta/classes/patch.bbclass
>> +++ b/meta/classes/patch.bbclass
>> @@ -78,5 +78,3 @@ python do_patch() {
>>           except bb.fetch2.BBFetchException as e:
>>               raise bb.build.FuncFailed(e)
>>   }
>> -
>> -do_patch[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
>> diff --git a/meta/classes/template.bbclass b/meta/classes/template.bbclass
>> index f3b5f80..fb9d118 100644
>> --- a/meta/classes/template.bbclass
>> +++ b/meta/classes/template.bbclass
>> @@ -7,7 +7,6 @@ TEMPLATE_FILES ?= ""
>>   TEMPLATE_VARS ?= "PN PV DESCRIPTION HOMEPAGE MAINTAINER DISTRO_ARCH"
>>   do_transform_template[vardeps] = "TEMPLATE_FILES ${TEMPLATE_VARS}"
>> -do_transform_template[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
>>   python do_transform_template() {
>>       import subprocess, contextlib
>> diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
>> index 3d18223..c41f294 100644
>> --- a/meta/conf/bitbake.conf
>> +++ b/meta/conf/bitbake.conf
>> @@ -35,7 +35,9 @@ PV = "${@bb.parse.BBHandler.vars_from_file(d.getVar('FILE', 
>> False),d)[1] or '1.0
>>   S = "${WORKDIR}/${P}"
>>   AUTOREV = "${@bb.fetch2.get_autorev(d)}"
>>   SRC_URI = "file://${FILE}"
>> -STAMP = "${TMPDIR}/stamps/${PF}"
>> +STAMPS_DIR ?= "${TMPDIR}/stamps"
>> +STAMP = "${STAMPS_DIR}/${DISTRO}-${DISTRO_ARCH}/${PF}/${PV}-${PR}"
> 
> Can you explain the issue of the PN/PF usage a bit more. I did not understand:
> 
>> As long as recipes mess with PF, we can't use OE's pattern as-is which
>> is PN-based.
> 
> The recipes that mess with PF are: buildchroot-*.bb sdkchroot.bb 
> isar-bootstrap-*.bb and image.bbclass. All of which do not really have a version.
> 
> So I would think that:
> 
> STAMP = "${STAMPS_DIR}/${DISTRO}-${DISTRO_ARCH}/${PN}/${PF}"
> STAMPCLEAN = "${STAMPS_DIR}/${DISTRO}-${DISTRO_ARCH}/${PN}/*-*"
> 
> should work. Or am I missing something?

PF is a effectively read-only. This is fixed later on this series. Then there is 
no need to deviate from OE any more.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux