From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from shymkent.ilbers.de ([unix socket]) by shymkent (Cyrus 2.5.10-Debian-2.5.10-3+deb9u2) with LMTPA; Thu, 25 Jul 2024 17:10:25 +0200 X-Sieve: CMU Sieve 2.4 Received: from mail-lj1-f190.google.com (mail-lj1-f190.google.com [209.85.208.190]) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPS id 46PFAOG7007135 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Thu, 25 Jul 2024 17:10:24 +0200 Received: by mail-lj1-f190.google.com with SMTP id 38308e7fff4ca-2ef2e57fb7csf2927501fa.1 for ; Thu, 25 Jul 2024 08:10:24 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1721920218; cv=pass; d=google.com; s=arc-20160816; b=HcQMaJkSeXnNhYze27Yb+84CnEzj/dal0fn981HRTNNoC5U2aVDMcqQQpokY5JZcI9 wM9WFuqBCDngOLo0AtXj06+oubChwDWXkphO4Kh7FZyx8CDPEOB6+doAZhCiTEYN79UE 3/utbkREAvKVkUT/PehjOH0FmIx59esA3V50lA3eBH7jBGTsQtlxY7LnGHq7+kZRYPAr CqLlgN/d3zHUWe9kNGOKNOF3Ob95lJMlGpo45Cqoeb6nSvxd61Y7Qj3eDEUbN/Jy56Q5 A8bM3V7T4FDEp495leMwLeCBOMf5gVmslK8V1yzB7IYKTxltYU9J61xV0cn9rG1Dfpc+ 6KNg== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:mime-version:references:in-reply-to :message-id:date:subject:to:from:sender:dkim-signature; bh=H312RUkhXbQ+C2T2MXZ1Y+e/a+b8dgyoR9x6njq3gmk=; fh=TjEn2xQ0rSp+BJW6L/ZXL5XzKCjfDr9SVmXRDSFB3yQ=; b=no1y++dNnoLvcSEEf0BSfHDfqwU1q+XxDM/tzngkyDcv7yPQFKhuEftCd//zJL/kSM tug6E3N2mWEzhWDkRqgnZIvOnN8K5mhkldlryzA+HGvroTt26NHpFQn5wE8ju9TkEfEY yURaa3LmV/nFAAa4ypRb7KJZAF7lVsgLDqkvddPmV1W9J2XgtpBcdmMK49VFFuDfhoTE vOTK6ZrbHlDKHCwfxCNIG4Y5biqtNB9OivEH+IzcTbc5jOFzj2dNJJZLq20RURFa2gUu Sk7NvbjWM1RQGXehF8QdJ23a17n/mVr2VmGO+l+bbqTLbiqHEt0xJME7GbvjAqty4SX7 aL4A==; darn=isar-build.org ARC-Authentication-Results: i=2; gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20230601; t=1721920218; x=1722525018; darn=isar-build.org; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:x-original-authentication-results :x-original-sender:mime-version:references:in-reply-to:message-id :date:subject:to:from:sender:from:to:cc:subject:date:message-id :reply-to; bh=H312RUkhXbQ+C2T2MXZ1Y+e/a+b8dgyoR9x6njq3gmk=; b=oetHX2rqnzJMWo3iNBHweaHg4zii5ItaZQ61Ysy+diHy5IpCbIbogyo2VLm6VWHo2+ poyMPH+ewZAmYjtPAjZPhiQRuf0ytHxa4dgtDpVcywByNemhIkBnSqeRgCrrAzF1DPkm 3hwTz/zToT2xanb9jB1Sriwtcabtnq6hQDF/laxa1BzaFhy0elId5JX3GuvJGZmM4bUJ Vnwfp4jqMYSoLAJugi1WU9loc3/MDP5YCvMM08bmKtXqr+88Qq3xDSjGpp3tOxuEFlXX 71fUU6oiwECs85DaBlMcaCTbvxbXZVgpq9m7UGih0HwPUpgtQ5XIJYlpNVCMGXZEQqqk JlXA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721920218; x=1722525018; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :x-spam-checked-in-group:list-id:mailing-list:precedence :x-original-authentication-results:x-original-sender:mime-version :references:in-reply-to:message-id:date:subject:to:from:x-beenthere :x-gm-message-state:sender:from:to:cc:subject:date:message-id :reply-to; bh=H312RUkhXbQ+C2T2MXZ1Y+e/a+b8dgyoR9x6njq3gmk=; b=m2WYSv2J1kHce2C5j76mpPPIsplelEbajyaHC5CDF1UskiyJVRuKy93c63dPSsjfni sOdr2OcBOus9giS8GUxnBM0e3m+J4OIAxXQ5VRAYYeFA+r4ackxM7ezpQXBee2V/iBTY DgZWbVzVjK+NJcjNY6lZMeOilg9e88e/8ZfJUNFbMO+6vMMs5aVmdiReMSKlg2T0U5vA 2NCZBCxVdOhf+u668mzr6UV2cRaLSqGd6MpGHxHrQJgd0UQidNbyTqBE2YAPXs1UCZHB Ts6DyhU/+uZMkcVKiauEs3QLMZKe3btxxq2+qYi/WecSPcuqjXP93F79l/mCXP/Wa3Wu f1NQ== Sender: isar-users@googlegroups.com X-Forwarded-Encrypted: i=2; AJvYcCVw/IdDPjOvQTrKQrngAkonCJp11oMetob/WTf5J8jHTdVMnZGVYgoaCyBTnDQ+FSlXY0EMNcJj9cm0teMtd2ok7mY= X-Gm-Message-State: AOJu0YwGAMLp/QOhJuDvsLqXm3pID8uDNyN3CW4Rd7xK+gC48toV8qwV kXqKFqq3PNaTjQiV6xvBqJwQynZbpECSwEr4Wekvb0G3f5MaYH4i X-Google-Smtp-Source: AGHT+IEc4aVQAJ3kESGdudhwKWKnpyaruU+5PC4xLCxVhlnZhMuqADBYRx1fkZBzW+RrnuHYrjtsJA== X-Received: by 2002:a2e:8094:0:b0:2f0:25dc:1886 with SMTP id 38308e7fff4ca-2f039c4e89dmr22880161fa.10.1721920217418; Thu, 25 Jul 2024 08:10:17 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a05:651c:b12:b0:2ee:de96:7d96 with SMTP id 38308e7fff4ca-2f03aa5d97els4383221fa.1.-pod-prod-01-eu; Thu, 25 Jul 2024 08:10:15 -0700 (PDT) X-Received: by 2002:a05:651c:108:b0:2ef:2a60:c1c1 with SMTP id 38308e7fff4ca-2f039ca273bmr22591061fa.21.1721920214928; Thu, 25 Jul 2024 08:10:14 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1721920214; cv=none; d=google.com; s=arc-20160816; b=H9lnopjNCz6GnFlKDgddbBqK6dIyrFyYwZmQVYYaCbjaXHoIiCavjXz6lEnxWrsiZ7 rSrvLTH4kJPP4YMXWQIorHUAzR+6CvkyiesmQSf7HtYuZcrQIyFhub+xjCUsmgJTWBnh c7U7CZZ1qcpxzLrGKLWMLJcGjfZvCVxcpI574ZsLeJwtV4TRjTTDT6HlQkc+ZsTV5bo9 tB7yamTcml2kBADAFTU6Fvelra7nr7zFnUKEAA7sRtjzkFnh9vbwmWYC+DnAGGGmvN/r LX+VlAViMsvxVnUpX7EX9FG3sG9W+VtutWnPDraFNvjFf0BI/02R46Xln9eEnMx2WgBT xS1w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from; bh=okoNhLFL9jjP8xknS1E0x96pTNm+QJ3hxTPfjLrM95U=; fh=7tclEdh7YbwSQowgJ6LNq720O7H5HTEaqj22NJWRE2E=; b=X8VGI1BjgOWHQAIiHU2CzAArtas+6QRJtDgXNe0SlOGe2LIAfxAvzLvgxEFQHTxmJ4 Ne9HfPf/vNbkDgyHItOhL9RcTlmgILOSqIF3k/QTkf97fxZf/GbvgYqePiVfAD8/UBUu WHkBg7tn/JpUtQ1n0aF8XGA91IxblfDv5XdBW5kcJE90tHgWL0/BnBWQqVOmAyhi7fAm N3+/os41VCizb3ShMedfV43yW+djQeuN5iouxBMTQPZHqtAOZ7QQSK8yTZbFjtxnn6Qj 0uaf1/yf1NSU7zH145E74Bny+aT5GDQ+Lx8xRyl746yWfva73WfEUEKMcrTGnNdSw2M9 kiYQ==; dara=google.com ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de Received: from shymkent.ilbers.de (shymkent.ilbers.de. [85.214.156.166]) by gmr-mx.google.com with ESMTPS id 4fb4d7f45d1cf-5ac6673bbcbsi45941a12.5.2024.07.25.08.10.14 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Thu, 25 Jul 2024 08:10:14 -0700 (PDT) Received-SPF: pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) client-ip=85.214.156.166; Received: from localhost.localdomain (44-208-124-178-static.mgts.by [178.124.208.44] (may be forged)) (authenticated bits=0) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPSA id 46PFACKY007063 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Thu, 25 Jul 2024 17:10:13 +0200 From: Uladzimir Bely To: isar-users@googlegroups.com Subject: [PATCH v7 02/10] meta: Add debrepo bbclass handling base-apt prefetching Date: Thu, 25 Jul 2024 18:07:34 +0300 Message-ID: <20240725151006.2129-3-ubely@ilbers.de> X-Mailer: git-send-email 2.44.2 In-Reply-To: <20240725151006.2129-1-ubely@ilbers.de> References: <20240725151006.2129-1-ubely@ilbers.de> MIME-Version: 1.0 X-Spam-Status: No, score=-4.6 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_EF,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,RCVD_IN_MSPIKE_H2,RCVD_IN_RP_CERTIFIED, RCVD_IN_RP_RNBL,RCVD_IN_RP_SAFE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-Original-Sender: ubely@ilbers.de X-Original-Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de Content-Type: text/plain; charset="UTF-8" Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-TUID: jzX765sWHS5z This class uses 'scripts/debrepo' python script to prefetch given packages or sources to local base-apt repository. Signed-off-by: Uladzimir Bely --- RECIPE-API-CHANGELOG.md | 10 ++++ meta/classes/debrepo.bbclass | 90 ++++++++++++++++++++++++++++++++++++ meta/conf/bitbake.conf | 5 ++ 3 files changed, 105 insertions(+) create mode 100644 meta/classes/debrepo.bbclass diff --git a/RECIPE-API-CHANGELOG.md b/RECIPE-API-CHANGELOG.md index 12ea93ec..68204a96 100644 --- a/RECIPE-API-CHANGELOG.md +++ b/RECIPE-API-CHANGELOG.md @@ -629,3 +629,13 @@ into kernel kbuild package. Only the "host" specific package is built automatically at cross builds. * Support emulated module build with cross-compiled kernel for linux-module + +### "Prefetch" mode for base-apt + +Originally, `base-apt` repo is created only during second build when variable +ISAR_USE_CACHED_BASE_REPO is set. The repo is populated with every package that +took part in the first build and was cached in DL_DIR. + +New ISAR_PREFETCH_BASE_APT variable changes the way `base-apt` is populated. +Packages added to the repo before running any task that need them. Separate +`debrepo` script is used for populating base-apt repo. diff --git a/meta/classes/debrepo.bbclass b/meta/classes/debrepo.bbclass new file mode 100644 index 00000000..003c4e03 --- /dev/null +++ b/meta/classes/debrepo.bbclass @@ -0,0 +1,90 @@ +# This software is a part of Isar. +# Copyright (C) 2024 ilbers GmbH +# +# SPDX-License-Identifier: MIT + +# Prefetch to base-apt repo by default +ISAR_PREFETCH_BASE_APT ??= "1" + +DEBREPO_WORKDIR ??= "${DEBREPO_TARGET_DIR}" + +debrepo_update_apt_source_list() { + [ "${ISAR_PREFETCH_BASE_APT}" != "1" ] && return + + chroot_dir=${1} + apt_list=${2} + + flock -x "${REPO_BASE_DIR}/repo.lock" -c " + sudo -E chroot ${chroot_dir} /usr/bin/apt-get update \ + -o Dir::Etc::SourceList=\"sources.list.d/${apt_list}.list\" \ + -o Dir::Etc::SourceParts=\"-\" \ + -o APT::Get::List-Cleanup=\"0\" + " +} + +debrepo_add_packages() { + [ "${ISAR_PREFETCH_BASE_APT}" != "1" ] && return + [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ] && return + + args="" + if [ "${1}" = "--srcmode" ]; then + args="${args} --srcmode" + shift + fi + + if [ "${1}" = "--isarapt" ]; then + args="${args} --extrarepo=${REPO_ISAR_DIR}/${DISTRO}" + shift + fi + + workdir="${1}" + args="${args} ${2}" + + if [ -n "${GNUPGHOME}" ]; then + export GNUPGHOME="${GNUPGHOME}" + else + export GNUPGHOME="${WORKDIR}/gpghome" + fi + + ${SCRIPTSDIR}/debrepo \ + --workdir="${workdir}" \ + ${args} +} + +debrepo_parse_dscfile() { + [ "${ISAR_PREFETCH_BASE_APT}" != "1" ] && return + [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ] && return + + dscfile="${1}" + args="" + + build_arch=${DISTRO_ARCH} + if [ "${ISAR_CROSS_COMPILE}" = "1" ]; then + build_arch=${HOST_ARCH} + fi + if [ "${PACKAGE_ARCH}" != "${build_arch}" ]; then + args="--crossbuild \ + crossbuild-essential-${PACKAGE_ARCH}:${build_arch} \ + dose-distcheck:${build_arch} \ + libc-dev:${PACKAGE_ARCH} \ + libstdc++-dev:${PACKAGE_ARCH} \ + " + fi + + args="${args} --extrarepo=${WORKDIR}/isar-apt/${DISTRO}-${DISTRO_ARCH}/apt/${DISTRO}" + + if [ -n "${GNUPGHOME}" ]; then + export GNUPGHOME="${GNUPGHOME}" + else + export GNUPGHOME="${WORKDIR}/gpghome" + fi + + if [ -n "${DEB_BUILD_PROFILES}" ]; then + export DEB_BUILD_PROFILES="${DEB_BUILD_PROFILES}" + fi + + ${SCRIPTSDIR}/debrepo \ + --workdir="${DEBREPO_WORKDIR}" \ + --dscfile="${dscfile}" \ + ${args} +} diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf index 4cfa8b10..b0e33477 100644 --- a/meta/conf/bitbake.conf +++ b/meta/conf/bitbake.conf @@ -69,6 +69,11 @@ KERNEL_FILE:mipsel ?= "vmlinux" KERNEL_FILE:riscv64 ?= "vmlinux" KERNEL_FILE:arm64 ?= "vmlinux" +# debrepo config +DEBREPO_DIR = "${TOPDIR}/debrepo" +DEBREPO_HOST_DIR = "${DEBREPO_DIR}/${HOST_DISTRO}-${HOST_ARCH}_${DISTRO}-${DISTRO_ARCH}" +DEBREPO_TARGET_DIR = "${DEBREPO_DIR}/${DISTRO}-${DISTRO_ARCH}" + MACHINEOVERRIDES ?= "${MACHINE}" DISTROOVERRIDES ?= "${DISTRO}" OVERRIDES = "${PACKAGE_ARCH}:${MACHINEOVERRIDES}:${DISTROOVERRIDES}:${BASE_DISTRO_CODENAME}:forcevariable" -- 2.44.2 -- You received this message because you are subscribed to the Google Groups "isar-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/20240725151006.2129-3-ubely%40ilbers.de.