Re: [PATCH v2] Convert apt source fetcher into native bitbake variant

["'Jan Kiszka' via isar-users" <isar-users@googlegroups.com>]

On 28.11.24 14:23, Uladzimir Bely wrote:
> On Thu, 2024-11-28 at 09:03 +0300, Uladzimir Bely wrote:
>> On Thu, 2024-11-28 at 12:55 +0800, Jan Kiszka wrote:
>>> On 27.11.24 22:07, Uladzimir Bely wrote:
>>>> On Fri, 2024-11-15 at 17:40 +0100, Jan Kiszka wrote:
>>>>> From: Jan Kiszka <jan.kiszka@siemens.com>
>>>>>
>>>>> There is no major functional difference, but we no longer have
>>>>> to
>>>>> manipulate SRC_URI by registering an official fetcher for
>>>>> apt://.
>>>>>
>>>>> As the fetching no longer takes place in separate tasks,
>>>>> do_fetch
>>>>> and
>>>>> do_unpack need to gain the extra flags that were so far
>>>>> assigned
>>>>> to
>>>>> apt_fetch and apt_unpack. That happens conditionally, i.e. only
>>>>> if
>>>>> SRC_URI actually contains an apt URL.
>>>>>
>>>>> One difference to the original version is the possibility -
>>>>> even
>>>>> if
>>>>> practically of minor relevance - to unpack multiple apt sources
>>>>> into
>>>>> S.
>>>>> The old version contained a loop but was directing dpkg-source
>>>>> to
>>>>> a
>>>>> pre-existing dir which would have failed on the second
>>>>> iteration.
>>>>> The
>>>>> new version now folds the results together after each step.
>>>>>
>>>>> Another minor difference is that unversioned fetches put their
>>>>> results
>>>>> into the same subfolder in DL_DIR, also when specifying a
>>>>> distro
>>>>> codename. Only versioned fetches get dedicated folders (and
>>>>> .done
>>>>> stamps).
>>>>>
>>>>> There is no progress report realized because dpkg-source
>>>>> unfortunately
>>>>> does not provide information upfront to make this predictable,
>>>>> thus
>>>>> expressible in form of percentage.
>>>>>
>>>>> Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
>>>>> ---
>>>>>
>>>>> Changes in v2:
>>>>>  - rebased, including the removal of isar-apt sources in
>>>>> apt_unpack
>>>>>
>>>>> I'm carefully optimistic that this change also resolves the
>>>>> previously 
>>>>> seen issue in CI.
>>>>>
>>>>>  meta/classes/dpkg-base.bbclass | 104 ++++---------------------
>>>>> --
>>>>> ----
>>>>> --
>>>>>  meta/lib/aptsrc_fetcher.py     |  93
>>>>> +++++++++++++++++++++++++++++
>>>>>  2 files changed, 104 insertions(+), 93 deletions(-)
>>>>>  create mode 100644 meta/lib/aptsrc_fetcher.py
>>>>>
>>>>> diff --git a/meta/classes/dpkg-base.bbclass
>>>>> b/meta/classes/dpkg-
>>>>> base.bbclass
>>>>> index b4ea8e17..c02c07a8 100644
>>>>> --- a/meta/classes/dpkg-base.bbclass
>>>>> +++ b/meta/classes/dpkg-base.bbclass
>>>>> @@ -79,110 +79,28 @@ do_adjust_git[lockfiles] +=
>>>>> "${DL_DIR}/git/isar.lock"
>>>>>  inherit patch
>>>>>  addtask patch after do_adjust_git
>>>>>  
>>>>> -SRC_APT ?= ""
>>>>> -
>>>>> -# filter out all "apt://" URIs out of SRC_URI and stick them
>>>>> into
>>>>> SRC_APT
>>>>>  python() {
>>>>> -    src_uri = (d.getVar('SRC_URI', False) or "").split()
>>>>> +    from bb.fetch2 import methods
>>>>>  
>>>>> -    prefix = "apt://"
>>>>> -    src_apt = []
>>>>> -    for u in src_uri:
>>>>> -        if u.startswith(prefix):
>>>>> -            src_apt.append(u[len(prefix) :])
>>>>> -            d.setVar('SRC_URI:remove', u)
>>>>> +    # apt-src fetcher
>>>>> +    import aptsrc_fetcher
>>>>> +    methods.append(aptsrc_fetcher.AptSrc())
>>>>>  
>>>>> -    d.prependVar('SRC_APT', ' '.join(src_apt))
>>>>> +    src_uri = (d.getVar('SRC_URI', False) or "").split()
>>>>> +    for u in src_uri:
>>>>> +        if u.startswith("apt://"):
>>>>> +            d.appendVarFlag('do_fetch', 'depends',
>>>>> d.getVar('SCHROOT_DEP'))
>>>>>  
>>>>> -    if len(d.getVar('SRC_APT').strip()) > 0:
>>>>> -        bb.build.addtask('apt_unpack', 'do_patch', '', d)
>>>>> -        bb.build.addtask('cleanall_apt', 'do_cleanall', '', d)
>>>>> +            d.appendVarFlag('do_unpack', 'cleandirs',
>>>>> d.getVar('S'))
>>>>> +            d.setVarFlag('do_unpack', 'network',
>>>>> d.getVar('TASK_USE_SUDO'))
>>>>> +            break
>>>>>  
>>>>>      # container docker fetcher
>>>>>      import container_fetcher
>>>>> -    from bb.fetch2 import methods
>>>>>  
>>>>>      methods.append(container_fetcher.Container())
>>>>>  }
>>>>>  
>>>>> -do_apt_fetch() {
>>>>> -    E="${@ isar_export_proxies(d)}"
>>>>> -    schroot_create_configs
>>>>> -
>>>>> -    session_id=$(schroot -q -b -c ${SBUILD_CHROOT})
>>>>> -    echo "Started session: ${session_id}"
>>>>> -
>>>>> -    schroot_cleanup() {
>>>>> -        schroot -q -f -e -c ${session_id} > /dev/null 2>&1
>>>>> -        schroot_delete_configs
>>>>> -    }
>>>>> -    trap 'exit 1' INT HUP QUIT TERM ALRM USR1
>>>>> -    trap 'schroot_cleanup' EXIT
>>>>> -
>>>>> -    schroot -r -c ${session_id} -d / -u root -- \
>>>>> -        rm /etc/apt/sources.list.d/isar-apt.list
>>>>> /etc/apt/preferences.d/isar-apt
>>>>> -    schroot -r -c ${session_id} -d / -- \
>>>>> -        sh -c '
>>>>> -            set -e
>>>>> -            for uri in $2; do
>>>>> -                mkdir -p /downloads/deb-src/"$1"/${uri}
>>>>> -                cd /downloads/deb-src/"$1"/${uri}
>>>>> -                apt-get -y --download-only --only-source
>>>>> source
>>>>> ${uri}
>>>>> -            done' \
>>>>> -                my_script "${BASE_DISTRO}-
>>>>> ${BASE_DISTRO_CODENAME}"
>>>>> "${SRC_APT}"
>>>>> -
>>>>> -    schroot -e -c ${session_id}
>>>>> -    schroot_delete_configs
>>>>> -}
>>>>> -
>>>>> -addtask apt_fetch
>>>>> -do_apt_fetch[lockfiles] += "${REPO_ISAR_DIR}/isar.lock"
>>>>> -do_apt_fetch[network] = "${TASK_USE_NETWORK_AND_SUDO}"
>>>>> -
>>>>> -# Add dependency from the correct schroot: host or target
>>>>> -do_apt_fetch[depends] += "${SCHROOT_DEP}"
>>>>> -
>>>>> -do_apt_unpack() {
>>>>> -    rm -rf ${S}
>>>>> -    schroot_create_configs
>>>>> -
>>>>> -    session_id=$(schroot -q -b -c ${SBUILD_CHROOT})
>>>>> -    echo "Started session: ${session_id}"
>>>>> -
>>>>> -    schroot_cleanup() {
>>>>> -        schroot -q -f -e -c ${session_id} > /dev/null 2>&1
>>>>> -        schroot_delete_configs
>>>>> -    }
>>>>> -    trap 'exit 1' INT HUP QUIT TERM ALRM USR1
>>>>> -    trap 'schroot_cleanup' EXIT
>>>>> -
>>>>> -    schroot -r -c ${session_id} -d / -u root -- \
>>>>> -        rm /etc/apt/sources.list.d/isar-apt.list
>>>>> /etc/apt/preferences.d/isar-apt
>>>>> -    schroot -r -c ${session_id} -d / -- \
>>>>> -        sh -c '
>>>>> -            set -e
>>>>> -            for uri in $2; do
>>>>> -                dscfile="$(apt-get -y -qq --print-uris --only-
>>>>> source
>>>>> source $uri | cut -d " " -f2 | grep -E "*.dsc")"
>>>>> -                cd ${PP}
>>>>> -                cp /downloads/deb-src/"${1}"/${uri}/* ${PP}
>>>>> -                dpkg-source -x "${dscfile}" "${PPS}"
>>>>> -            done' \
>>>>> -                my_script "${BASE_DISTRO}-
>>>>> ${BASE_DISTRO_CODENAME}"
>>>>> "${SRC_APT}"
>>>>> -
>>>>> -    schroot -e -c ${session_id}
>>>>> -    schroot_delete_configs
>>>>> -}
>>>>> -do_apt_unpack[network] = "${TASK_USE_SUDO}"
>>>>> -
>>>>> -addtask apt_unpack after do_apt_fetch
>>>>> -
>>>>> -do_cleanall_apt[nostamp] = "1"
>>>>> -do_cleanall_apt() {
>>>>> -    for uri in "${SRC_APT}"; do
>>>>> -        rm -rf "${DEBSRCDIR}/${BASE_DISTRO}-
>>>>> ${BASE_DISTRO_CODENAME}/$uri"
>>>>> -    done
>>>>> -}
>>>>> -
>>>>>  def get_package_srcdir(d):
>>>>>      s = os.path.abspath(d.getVar("S"))
>>>>>      workdir = os.path.abspath(d.getVar("WORKDIR"))
>>>>> diff --git a/meta/lib/aptsrc_fetcher.py
>>>>> b/meta/lib/aptsrc_fetcher.py
>>>>> new file mode 100644
>>>>> index 00000000..ee726202
>>>>> --- /dev/null
>>>>> +++ b/meta/lib/aptsrc_fetcher.py
>>>>> @@ -0,0 +1,93 @@
>>>>> +# This software is a part of ISAR.
>>>>> +# Copyright (c) Siemens AG, 2024
>>>>> +#
>>>>> +# SPDX-License-Identifier: MIT
>>>>> +
>>>>> +from bb.fetch2 import FetchError
>>>>> +from bb.fetch2 import FetchMethod
>>>>> +from bb.fetch2 import logger
>>>>> +from bb.fetch2 import runfetchcmd
>>>>> +
>>>>> +class AptSrc(FetchMethod):
>>>>> +    def supports(self, ud, d):
>>>>> +        return ud.type in ['apt']
>>>>> +
>>>>> +    def urldata_init(self, ud, d):
>>>>> +        ud.src_package = ud.url[len('apt://'):]
>>>>> +        ud.host = ud.host.replace('=', '_')
>>>>> +
>>>>> +        base_distro = d.getVar('BASE_DISTRO')
>>>>> +
>>>>> +        # For these distros we know that the same version
>>>>> means
>>>>> the
>>>>> same
>>>>> +        # source package, also across distro releases.
>>>>> +        distro_suffix = '' if base_distro in ['debian',
>>>>> 'ubuntu']
>>>>> else \
>>>>> +            '-' + d.getVar('BASE_DISTRO_CODENAME')
> 
> I think, to avoid the issue I mentioned, we should continue using
> ${BASE_DISTRO}-${BASE_DISTRO_CODENAME} here without exceptions.
> 

Yeah, the main issue is that we cannot predict the actual version that
will be requested and that we therefore end up with a single,
unversioned .done file for the fetching tasks. We would have to retrieve
the version on every run in order to decide if to re-run the fetch - not
simple and likely not worth it. Will change back to the existing path.

> Also, cache_deb_src() function in rootfs.bbclass still uses this
> location.

Right, that would be easier fixable, though.

Thanks,
Jan

-- 
Siemens AG, Technology
Linux Expert Center

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/3312df35-35fd-4874-aa42-4b9c56cb5b4c%40siemens.com.