From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6661145085876371456 X-Received: by 2002:a1c:cb8d:: with SMTP id b135mr1130148wmg.22.1551690721523; Mon, 04 Mar 2019 01:12:01 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a1c:b755:: with SMTP id h82ls1448656wmf.9.canary-gmail; Mon, 04 Mar 2019 01:12:01 -0800 (PST) X-Google-Smtp-Source: APXvYqxKOovVYeKQgoC+Avw5z/gghlNAZLfDyZZuK8D4O7HuxMZBo9x7UF/+vJCMvSoVQxKWTiku X-Received: by 2002:a05:600c:21d4:: with SMTP id x20mr1102586wmj.25.1551690720987; Mon, 04 Mar 2019 01:12:00 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1551690720; cv=none; d=google.com; s=arc-20160816; b=EE+NMYFJmMylbgqLLVe+FD4rlw4ZBOi1I0HBqw6Bz4+4ewiTu8SZeVTebld/UuQeiM WxL0q7FKSiM4JBWB3UEUVPg6PSKTi6zRJ0kwcoXnJHLF4gTXrPAoSDmccnEAVWUdoefV VuQASpjdI4pBvtErIkXX9Pc067znkdRLAHvulC1cVn/ivLsr+yYJf9n5vefDYB6gtSqO zWbK/ametIi3xZWMDXoKfpJgS6MG8ojalI+JLzpxMqggSV7P6BVHdw5HLwxVOis76bJS +/9r2Eb5YicQ5h+G4BPkpdUGGzgbwQlRx9dpFH5L/k4iikF2QfMAquVHvbcIDbMFTe5S QNaQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:content-language:in-reply-to:mime-version :user-agent:date:message-id:from:references:cc:to:subject; bh=5jeQD6WAk0K/CPc3+iRtY/BwBShxOZRX2VZh80K1i8g=; b=ysr+mCALqZyKZlIf31CITGavZ6CoNTgPHxJL2/gbV53kO1XYhenrdDrKsQTEhb+Yb5 5rkhD2VfxfICm2ECvAQZbyT3fNrXQmz3rF3gui2tpaBZwRRRQULIpne2aSNDRd2z+2UN zKsxTi4HdcHYmhH3sAX0DW7AHyjH+KOBH6r3+l9FoyVB+dLkrBhAl1mkRsjgRjDhDTwx jI5upl/7Dt5qaf2xvmpYwXgqGREgGehbSkxHFzvhBra0v3BIq3EzI7V+FX84I+vI+XX5 JPMM2BK+x6TToskNJK441Ahc9Pvt/QJ8Wz6HCuJQ4dS8WAc8RP7PtxQcAtX7NzONgsiL Dwrw== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of jan.kiszka@siemens.com designates 194.138.37.39 as permitted sender) smtp.mailfrom=jan.kiszka@siemens.com Return-Path: Received: from lizzard.sbs.de (lizzard.sbs.de. [194.138.37.39]) by gmr-mx.google.com with ESMTPS id b8si283544wru.1.2019.03.04.01.12.00 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 04 Mar 2019 01:12:00 -0800 (PST) Received-SPF: pass (google.com: domain of jan.kiszka@siemens.com designates 194.138.37.39 as permitted sender) client-ip=194.138.37.39; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of jan.kiszka@siemens.com designates 194.138.37.39 as permitted sender) smtp.mailfrom=jan.kiszka@siemens.com Received: from mail2.sbs.de (mail2.sbs.de [192.129.41.66]) by lizzard.sbs.de (8.15.2/8.15.2) with ESMTPS id x249C0rv008583 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK) for ; Mon, 4 Mar 2019 10:12:00 +0100 Received: from [167.87.45.191] ([167.87.45.191]) by mail2.sbs.de (8.15.2/8.15.2) with ESMTP id x249Bxqd018201; Mon, 4 Mar 2019 10:12:00 +0100 Subject: Re: [PATCH] Merge patch and unpack tasks To: "[ext] Claudius Heine" , isar-users Cc: Henning Schild References: <9da9415f-28f4-d67d-e1d4-910215591362@siemens.com> From: Jan Kiszka Message-ID: Date: Mon, 4 Mar 2019 10:11:59 +0100 User-Agent: Mozilla/5.0 (X11; U; Linux i686 (x86_64); de; rv:1.8.1.12) Gecko/20080226 SUSE/2.0.0.12-1.1 Thunderbird/2.0.0.12 Mnenhy/0.7.5.666 MIME-Version: 1.0 In-Reply-To: <9da9415f-28f4-d67d-e1d4-910215591362@siemens.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 8bit X-TUID: za8WWWJT0tWr On 04.03.19 09:51, [ext] Claudius Heine wrote: > Hi Jan, > > On 23/02/2019 11.44, Jan Kiszka wrote: >> From: Jan Kiszka >> >> Our patch implementations does not support rolling back previously >> applied patches, thus fails on patch updates without a rerun of unpack. >> Avoid this by moving both steps into the same task. > > IMO merging multiple conceptual different tasks into one is a step back, > especially with such a fundamental task as do_patch. > > If our implementation lacks behind the one in OE, why not copy stuff from there > instead? Because it's OE and not Isar. I would have copied things already if they were well isolated on the OE side. But there seem to be more complex dependencies that need to be resolved - or we need a conceptually equivalent solution. We likely need to go that path as we need to account for the debian source package use case as well. Jan > > If that is just impossible for whatever reason, then I would rather bind those > tasks together with [postfuncs]. That would allow each function to be customized > from other layers if necessary. > > regards, > Claudius > >> >> Signed-off-by: Jan Kiszka >> --- >>   meta/classes/base.bbclass                          | 47 ++++++++++++++++++---- >>   meta/classes/dpkg-base.bbclass                     |  9 ++--- >>   meta/classes/dpkg-raw.bbclass                      |  2 +- >>   meta/classes/fit-img.bbclass                       |  2 +- >>   meta/classes/image.bbclass                         |  2 +- >>   meta/classes/patch.bbclass                         | 33 --------------- >>   meta/classes/template.bbclass                      |  2 +- >>   meta/classes/ubi-img.bbclass                       |  2 +- >>   .../isar-bootstrap/isar-bootstrap-host.bb          |  2 +- >>   .../isar-bootstrap/isar-bootstrap-target.bb        |  2 +- >>   .../recipes-core/isar-bootstrap/isar-bootstrap.inc |  2 +- >>   meta/recipes-devtools/isar-apt/isar-apt.bb         |  2 +- >>   .../isar-cfg-localepurge/isar-cfg-localepurge.bb   |  2 +- >>   13 files changed, 52 insertions(+), 57 deletions(-) >>   delete mode 100644 meta/classes/patch.bbclass >> >> diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass >> index 4279a68..7bdaf37 100644 >> --- a/meta/classes/base.bbclass >> +++ b/meta/classes/base.bbclass >> @@ -1,4 +1,5 @@ >>   # Copyright (C) 2003  Chris Larson >> +# Copyright (c) Siemens AG, 2018 >>   # >>   # Permission is hereby granted, free of charge, to any person obtaining a >>   # copy of this software and associated documentation files (the "Software"), >> @@ -17,6 +18,8 @@ >>   # OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, >>   # ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR >>   # OTHER DEALINGS IN THE SOFTWARE. >> +# >> +# SPDX-License-Identifier: MIT >> >>   THISDIR = "${@os.path.dirname(d.getVar('FILE', True))}" >> >> @@ -135,25 +138,53 @@ python do_fetch() { >> >>   addtask fetch before do_build >> >> -do_unpack[dirs] = "${WORKDIR}" >> -do_unpack[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}" >> +do_unpack_and_patch[dirs] = "${WORKDIR}" >> +do_unpack_and_patch[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}" >> >> -# Unpack package and put it into working directory >> -python do_unpack() { >> -    src_uri = (d.getVar('SRC_URI', True) or "").split() >> -    if len(src_uri) == 0: >> +# Unpack package, put it into working directory, and apply potential patches >> +python do_unpack_and_patch() { >> +    import subprocess >> + >> +    src_uris = (d.getVar('SRC_URI', True) or "").split() >> +    if len(src_uris) == 0: >>           return >> >>       rootdir = d.getVar('WORKDIR', True) >> +    src_dir = d.getVar("S", True) >> >> +    # unpack src_uris >>       try: >> -        fetcher = bb.fetch2.Fetch(src_uri, d) >> +        fetcher = bb.fetch2.Fetch(src_uris, d) >>           fetcher.unpack(rootdir) >>       except bb.fetch2.BBFetchException as e: >>           raise bb.build.FuncFailed(e) >> + >> +    # apply patches >> +    for src_uri in src_uris: >> +        try: >> +            fetcher = bb.fetch2.Fetch([src_uri], d) >> + >> +            apply = fetcher.ud[src_uri].parm.get("apply") >> +            if apply == "no": >> +                continue >> + >> +            basename = fetcher.ud[src_uri].basename or "" >> +            if not (basename.endswith(".patch") or apply == "yes"): >> +                continue >> + >> +            striplevel = fetcher.ud[src_uri].parm.get("striplevel") or "1" >> + >> +            cmd = "patch --no-backup-if-mismatch -p " + striplevel + \ >> +                  " --directory " + src_dir + \ >> +                  " --input " + rootdir + '/' + basename >> +            bb.note(cmd) >> +            if subprocess.call(cmd, shell=True) != 0: >> +                bb.fatal("patching failed") >> +        except bb.fetch2.BBFetchException as e: >> +            raise bb.build.FuncFailed(e) >>   } >> >> -addtask unpack after do_fetch before do_build >> +addtask unpack_and_patch after do_fetch before do_build >> >>   addtask build >>   do_build[dirs] = "${TOPDIR}" >> diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass >> index 742b8ad..8ef1f06 100644 >> --- a/meta/classes/dpkg-base.bbclass >> +++ b/meta/classes/dpkg-base.bbclass >> @@ -15,12 +15,9 @@ do_adjust_git() { >>       fi >>   } >> >> -addtask adjust_git after do_unpack before do_patch >> +addtask adjust_git after do_unpack_and_patch before do_prepare_build >>   do_adjust_git[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}" >> >> -inherit patch >> -addtask patch after do_adjust_git before do_build >> - >>   SRC_APT ?= "" >> >>   do_apt_fetch[depends] = "buildchroot-target:do_build" >> @@ -40,7 +37,7 @@ do_apt_fetch() { >>       dpkg_undo_mounts >>   } >> >> -addtask apt_fetch after do_unpack before do_patch >> +addtask apt_fetch before do_unpack_and_patch >>   do_apt_fetch[lockfiles] += "${REPO_ISAR_DIR}/isar.lock" >>   do_apt_fetch[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}" >> >> @@ -64,7 +61,7 @@ do_prepare_build() { >>       true >>   } >> >> -addtask prepare_build after do_patch do_transform_template before do_build >> +addtask prepare_build after do_unpack_and_patch do_transform_template before >> do_build >>   do_prepare_build[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}" >>   # If Isar recipes depend on each other, they typically need the package >>   # deployed to isar-apt >> diff --git a/meta/classes/dpkg-raw.bbclass b/meta/classes/dpkg-raw.bbclass >> index ea03ea4..e63ba1d 100644 >> --- a/meta/classes/dpkg-raw.bbclass >> +++ b/meta/classes/dpkg-raw.bbclass >> @@ -17,7 +17,7 @@ do_install() { >> >>   do_install[cleandirs] = "${D}" >>   do_install[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}" >> -addtask install after do_unpack before do_prepare_build >> +addtask install after do_unpack_and_patch before do_prepare_build >> >>   do_prepare_build[cleandirs] += "${D}/debian" >>   do_prepare_build() { >> diff --git a/meta/classes/fit-img.bbclass b/meta/classes/fit-img.bbclass >> index edca09f..97f1d1a 100644 >> --- a/meta/classes/fit-img.bbclass >> +++ b/meta/classes/fit-img.bbclass >> @@ -29,4 +29,4 @@ do_fit_image() { >>       sudo chroot ${BUILDCHROOT_DIR} /usr/bin/mkimage ${MKIMAGE_ARGS} \ >>                   -f '${PP_WORK}/${FIT_IMAGE_SOURCE}' >> '${PP_DEPLOY}/${FIT_IMAGE_FILE}' >>   } >> -addtask fit_image before do_build after do_copy_boot_files >> do_install_imager_deps do_unpack do_transform_template >> +addtask fit_image before do_build after do_copy_boot_files >> do_install_imager_deps do_unpack_and_patch do_transform_template >> diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass >> index 574fb46..a5952eb 100644 >> --- a/meta/classes/image.bbclass >> +++ b/meta/classes/image.bbclass >> @@ -92,7 +92,7 @@ do_rootfs() { >>       die "No root filesystem function defined, please implement in your recipe" >>   } >> >> -addtask rootfs before do_build after do_unpack >> +addtask rootfs before do_build after do_unpack_and_patch >>   do_rootfs[deptask] = "do_deploy_deb" >> >>   do_mark_rootfs() { >> diff --git a/meta/classes/patch.bbclass b/meta/classes/patch.bbclass >> deleted file mode 100644 >> index 0bc449f..0000000 >> --- a/meta/classes/patch.bbclass >> +++ /dev/null >> @@ -1,33 +0,0 @@ >> -# This software is a part of ISAR. >> -# Copyright (c) Siemens AG, 2018 >> - >> -python do_patch() { >> -    import subprocess >> - >> -    workdir = d.getVar("WORKDIR", True) + "/" >> -    src_dir = d.getVar("S", True) >> - >> -    for src_uri in (d.getVar("SRC_URI", True) or "").split(): >> -        try: >> -            fetcher = bb.fetch2.Fetch([src_uri], d) >> - >> -            apply = fetcher.ud[src_uri].parm.get("apply") >> -            if apply == "no": >> -                continue >> - >> -            basename = fetcher.ud[src_uri].basename or "" >> -            if not (basename.endswith(".patch") or apply == "yes"): >> -                continue >> - >> -            striplevel = fetcher.ud[src_uri].parm.get("striplevel") or "1" >> - >> -            cmd = "patch --no-backup-if-mismatch -p " + striplevel + \ >> -                  " --directory " + src_dir + " --input " + workdir + basename >> -            bb.note(cmd) >> -            if subprocess.call(cmd, shell=True) != 0: >> -                bb.fatal("patching failed") >> -        except bb.fetch2.BBFetchException as e: >> -            raise bb.build.FuncFailed(e) >> -} >> - >> -do_patch[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}" >> diff --git a/meta/classes/template.bbclass b/meta/classes/template.bbclass >> index 324511a..8846ab0 100644 >> --- a/meta/classes/template.bbclass >> +++ b/meta/classes/template.bbclass >> @@ -59,4 +59,4 @@ python do_transform_template() { >>               if process.wait() != 0: >>                   bb.fatal("processing of template failed") >>   } >> -addtask do_transform_template after do_unpack >> +addtask do_transform_template after do_unpack_and_patch >> diff --git a/meta/classes/ubi-img.bbclass b/meta/classes/ubi-img.bbclass >> index f61a940..f06ab04 100644 >> --- a/meta/classes/ubi-img.bbclass >> +++ b/meta/classes/ubi-img.bbclass >> @@ -32,4 +32,4 @@ do_ubi_image() { >>       sudo chroot ${BUILDCHROOT_DIR} /usr/sbin/ubinize ${UBINIZE_ARGS} \ >>                   -o '${PP_DEPLOY}/${UBI_IMAGE_FILE}' '${PP_WORK}/${UBINIZE_CFG}' >>   } >> -addtask ubi_image before do_build after do_copy_boot_files >> do_install_imager_deps do_unpack do_transform_template >> +addtask ubi_image before do_build after do_copy_boot_files >> do_install_imager_deps do_unpack_and_patch do_transform_template >> diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb >> b/meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb >> index a793585..da6d5e1 100644 >> --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb >> +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb >> @@ -40,7 +40,7 @@ python do_apt_config_prepare() { >>           aggregate_files(d, apt_sources_list, apt_sources_init_out) >>           aggregate_aptsources_list(d, apt_sources_list, apt_sources_out) >>   } >> -addtask apt_config_prepare before do_bootstrap after do_unpack >> +addtask apt_config_prepare before do_bootstrap after do_unpack_and_patch >> >>   OVERRIDES_append = ":${@get_distro_needs_https_support(d, True)}" >> >> diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap-target.bb >> b/meta/recipes-core/isar-bootstrap/isar-bootstrap-target.bb >> index bec6fa8..b338adf 100644 >> --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap-target.bb >> +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap-target.bb >> @@ -39,7 +39,7 @@ python do_apt_config_prepare() { >>           aggregate_files(d, apt_sources_list, apt_sources_init_out) >>           aggregate_aptsources_list(d, apt_sources_list, apt_sources_out) >>   } >> -addtask apt_config_prepare before do_bootstrap after do_unpack >> +addtask apt_config_prepare before do_bootstrap after do_unpack_and_patch >> >>   OVERRIDES_append = ":${@get_distro_needs_https_support(d, False)}" >> >> diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc >> b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc >> index 234d339..15995d4 100644 >> --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc >> +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc >> @@ -184,7 +184,7 @@ do_generate_keyring() { >>           done >>       fi >>   } >> -addtask generate_keyring before do_build after do_unpack >> +addtask generate_keyring before do_build after do_unpack_and_patch >> >> >> >> diff --git a/meta/recipes-devtools/isar-apt/isar-apt.bb >> b/meta/recipes-devtools/isar-apt/isar-apt.bb >> index a959691..aea7ff7 100644 >> --- a/meta/recipes-devtools/isar-apt/isar-apt.bb >> +++ b/meta/recipes-devtools/isar-apt/isar-apt.bb >> @@ -26,4 +26,4 @@ do_cache_config() { >>       fi >>   } >> >> -addtask cache_config after do_unpack before do_build >> +addtask cache_config after do_unpack_and_patch before do_build >> diff --git a/meta/recipes-support/isar-cfg-localepurge/isar-cfg-localepurge.bb >> b/meta/recipes-support/isar-cfg-localepurge/isar-cfg-localepurge.bb >> index 62b4b2d..b231e1b 100644 >> --- a/meta/recipes-support/isar-cfg-localepurge/isar-cfg-localepurge.bb >> +++ b/meta/recipes-support/isar-cfg-localepurge/isar-cfg-localepurge.bb >> @@ -55,7 +55,7 @@ do_gen_config() { >>           ${@get_nopurge(d)} >>       __EOF__ >>   } >> -addtask gen_config after do_unpack before do_install >> +addtask gen_config after do_unpack_and_patch before do_install >> >>   do_install() { >>       install -v -d ${D}/usr/lib/${PN} >> -- >> 2.16.4 >> -- Siemens AG, Corporate Technology, CT RDA IOT SES-DE Corporate Competence Center Embedded Linux