From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 7380320960081035264 X-Received: by 2002:a2e:6e17:0:b0:2ea:df2e:428c with SMTP id 38308e7fff4ca-2ec0e5abf40mr17978051fa.49.1718364880773; Fri, 14 Jun 2024 04:34:40 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a05:651c:313:b0:2eb:ff5d:15d with SMTP id 38308e7fff4ca-2ec02936c1dls8894431fa.2.-pod-prod-03-eu; Fri, 14 Jun 2024 04:34:38 -0700 (PDT) X-Google-Smtp-Source: AGHT+IE59hCaAAi8p5Fm/jveKvgUugu0ZqvHIkdBo+5reHp9M6+li1t+KM3k5OJMewjEhgAdbKq+ X-Received: by 2002:a2e:99d8:0:b0:2ea:eb96:b952 with SMTP id 38308e7fff4ca-2ec0e46ec08mr17675351fa.22.1718364878440; Fri, 14 Jun 2024 04:34:38 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1718364878; cv=none; d=google.com; s=arc-20160816; b=UW8HyP+ISxJnhsEl6eWjkv7PkXGg8qg7f9k2huisxgQq4rhdAzqvNo++tzCh2S69no KZN7zNMmNizn3+9vf80K7oDXyYjYmdtLDwOmKlCt4DVM/lIEzMvCaZgZi3HJn1CsUvPO hJVyDZlkghKetTiIM7P13mbWcd87s5xF1+0FpW1XlkJkfdRAq2PUOliUIQf0T5iVTUpf yf1/jPDXuSvYKrzWxxEH08JsgvEfArhgO+xKrBGhJJbZdijaLwcBWgiMCEKT5UEFWfhe rIcQaRbtCKm/zomXCSnCCv0dSB1yLKbvtqJMfReO0UuhijtP2LJoBn0LxQDV7XTBXloT LzmQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from; bh=XGoSey23MmGfWxb+QhdSEo89ZAoBGDs+5t0yTJoYugQ=; fh=/h9QQkzJ8EboVkWg45aWwpaUro6WMavIVd2OhN45RtE=; b=Gzo0jHOXFTz9j6ZdYb9NSWpaB8nvUbIYw3pA6aOA7y3uSXJGvyrt/yZAYYAciCKWaf BfEP4cFRna1t8kgtpI/w6eRufYsMTgc6G4aZqI3iX8VR3GPEzrFy6vHH83xAR0oMdlZ5 Ff3huNddrjjNQe8reJ3FcYl2amuoshxF+b17Jmv9V2sBzntO+viZq3tjIKWtbB9CDxXx Gn8tm28XCjrLRXKtcir4KTwoEeEYuTkI2o/ccY/H7byiybSXXfT8JaPT5NL51AiEJLhO g8LSXc5y/AY2vmSEfxYqPmjlqRDwPwbZQFXmIoNPW+zkWObv8OTJcj1zK9cxDJ8Xduqi SOVg==; dara=google.com ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of amikan@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=amikan@ilbers.de Return-Path: Received: from shymkent.ilbers.de (shymkent.ilbers.de. [85.214.156.166]) by gmr-mx.google.com with ESMTPS id 38308e7fff4ca-2ec05cc82efsi789541fa.7.2024.06.14.04.34.38 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Fri, 14 Jun 2024 04:34:38 -0700 (PDT) Received-SPF: pass (google.com: domain of amikan@ilbers.de designates 85.214.156.166 as permitted sender) client-ip=85.214.156.166; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of amikan@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=amikan@ilbers.de Received: from localhost.localdomain (r154-240-38-77-broadband.btv.lv [77.38.240.154] (may be forged)) (authenticated bits=0) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPSA id 45EBYZOx032043 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 14 Jun 2024 13:34:37 +0200 From: Anton Mikanovich To: isar-users@googlegroups.com Cc: Anton Mikanovich Subject: [PATCH v9 2/8] meta: Add mmdebstrap recipe Date: Fri, 14 Jun 2024 14:34:19 +0300 Message-Id: <20240614113425.122722-3-amikan@ilbers.de> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240614113425.122722-1-amikan@ilbers.de> References: <20240614113425.122722-1-amikan@ilbers.de> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-TUID: Vt86l9qnjDMa It can be used as debootstrap alternative for rootfs prepare. Internally, it uses apt and allows to bootstrap the distro from multiple repositories. chroot-setup.sh and locale are copied from debootstrap recipe. Signed-off-by: Anton Mikanovich --- meta/classes/bootstrap.bbclass | 1 + .../isar-mmdebstrap/isar-mmdebstrap-host.bb | 17 ++ .../isar-mmdebstrap/isar-mmdebstrap-target.bb | 12 + .../isar-mmdebstrap/isar-mmdebstrap.inc | 230 ++++++++++++++++++ 4 files changed, 260 insertions(+) create mode 100644 meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap-host.bb create mode 100644 meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap-target.bb create mode 100644 meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc diff --git a/meta/classes/bootstrap.bbclass b/meta/classes/bootstrap.bbclass index b392677a..deb00b23 100644 --- a/meta/classes/bootstrap.bbclass +++ b/meta/classes/bootstrap.bbclass @@ -26,6 +26,7 @@ DISTRO_BOOTSTRAP_BASE_PACKAGES ??= "" DISTRO_VARS_PREFIX ?= "${@'HOST_' if bb.utils.to_boolean(d.getVar('BOOTSTRAP_FOR_HOST')) else ''}" BOOTSTRAP_DISTRO = "${@d.getVar('HOST_DISTRO' if bb.utils.to_boolean(d.getVar('BOOTSTRAP_FOR_HOST')) else 'DISTRO')}" BOOTSTRAP_BASE_DISTRO = "${@d.getVar('HOST_BASE_DISTRO' if bb.utils.to_boolean(d.getVar('BOOTSTRAP_FOR_HOST')) else 'BASE_DISTRO')}" +BOOTSTRAP_DISTRO_ARCH = "${@d.getVar('HOST_ARCH' if bb.utils.to_boolean(d.getVar('BOOTSTRAP_FOR_HOST')) else 'DISTRO_ARCH')}" # reproducible builds, only enabled if ISAR_USE_APT_SNAPSHOT ISAR_APT_SNAPSHOT_MIRROR ??= "" diff --git a/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap-host.bb b/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap-host.bb new file mode 100644 index 00000000..66c8d11e --- /dev/null +++ b/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap-host.bb @@ -0,0 +1,17 @@ +# Minimal host Debian root file system +# +# This software is a part of Isar. +# Copyright (C) 2024 ilbers GmbH +# +# SPDX-License-Identifier: MIT + +Description = "Minimal host Debian root file system" + +DEPLOY_ISAR_BOOTSTRAP = "${DEPLOY_DIR_BOOTSTRAP}/${HOST_DISTRO}-host_${DISTRO}-${DISTRO_ARCH}" + +BOOTSTRAP_FOR_HOST = "1" + +require isar-mmdebstrap.inc + +HOST_DISTRO_BOOTSTRAP_KEYS ?= "" +DISTRO_BOOTSTRAP_KEYS = "${HOST_DISTRO_BOOTSTRAP_KEYS}" diff --git a/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap-target.bb b/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap-target.bb new file mode 100644 index 00000000..84a89ff1 --- /dev/null +++ b/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap-target.bb @@ -0,0 +1,12 @@ +# Minimal target Debian root file system +# +# This software is a part of Isar. +# Copyright (C) 2024 ilbers GmbH +# +# SPDX-License-Identifier: MIT + +Description = "Minimal target Debian root file system" + +DEPLOY_ISAR_BOOTSTRAP = "${DEPLOY_DIR_BOOTSTRAP}/${DISTRO}-${DISTRO_ARCH}" + +require isar-mmdebstrap.inc diff --git a/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc b/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc new file mode 100644 index 00000000..72096ae2 --- /dev/null +++ b/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc @@ -0,0 +1,230 @@ +# Minimal debian root file system +# +# This software is a part of Isar. +# Copyright (C) 2024 ilbers GmbH +# +# SPDX-License-Identifier: MIT + +inherit bootstrap +inherit compat +inherit deb-dl-dir + +FILESEXTRAPATHS:append = ":${LAYERDIR_core}/recipes-core/isar-bootstrap/files" + +ROOTFSDIR = "${WORKDIR}/rootfs" +DISTRO_BOOTSTRAP_BASE_PACKAGES = "locales,apt,usrmerge" +DISTRO_BOOTSTRAP_BASE_PACKAGES:append:gnupg = ",gnupg" +DISTRO_BOOTSTRAP_BASE_PACKAGES:append:https-support = ",ca-certificates" +BOOTSTRAP_TMPDIR = "${WORKDIR}/tempdir" + +def get_distro_primary_source_entry(d): + for source in generate_distro_sources(d): + if source[0] == "deb": + return source[2:] + bb.fatal('Invalid apt sources list') + +def get_distro_have_https_source(d): + return any(source[2].startswith("https://") for source in generate_distro_sources(d)) + +def get_distro_needs_https_support(d): + if get_distro_have_https_source(d): + return "https-support" + else: + return "" + +OVERRIDES:append = ":${@get_distro_needs_https_support(d)}" + +def get_distro_needs_gpg_support(d): + if d.getVar("DISTRO_BOOTSTRAP_KEYS") or \ + d.getVar("THIRD_PARTY_APT_KEYS") or \ + d.getVar("BASE_REPO_KEY"): + return "gnupg" + else: + return "" + +OVERRIDES:append = ":${@get_distro_needs_gpg_support(d)}" + +APT_KEYS_DIR = "${WORKDIR}/aptkeys" +DISTRO_BOOTSTRAP_KEYRING = "${WORKDIR}/distro-keyring.gpg" + +do_generate_keyrings[cleandirs] = "${APT_KEYS_DIR}" +do_generate_keyrings[dirs] = "${DL_DIR}" +do_generate_keyrings[vardeps] += "DISTRO_BOOTSTRAP_KEYS THIRD_PARTY_APT_KEYS" +do_generate_keyrings[network] = "${TASK_USE_SUDO}" +do_generate_keyrings() { + if [ -n "${@d.getVar("THIRD_PARTY_APT_KEYFILES") or ""}" ]; then + chmod 777 "${APT_KEYS_DIR}" + for keyfile in ${@d.getVar("THIRD_PARTY_APT_KEYFILES")}; do + cp "$keyfile" "${APT_KEYS_DIR}"/"$(basename "$keyfile")" + done + fi + if [ -n "${@d.getVar("DISTRO_BOOTSTRAP_KEYFILES") or ""}" ]; then + for keyfile in ${@d.getVar("DISTRO_BOOTSTRAP_KEYFILES")}; do + sudo apt-key --keyring "${DISTRO_BOOTSTRAP_KEYRING}" add $keyfile + cp "$keyfile" "${APT_KEYS_DIR}"/"$(basename "$keyfile")" + done + fi +} +addtask generate_keyrings before do_build after do_unpack + +do_bootstrap[vardeps] += " \ + DISTRO_APT_PREMIRRORS \ + ISAR_ENABLE_COMPAT_ARCH \ + ${DISTRO_VARS_PREFIX}DISTRO_APT_SOURCES \ + " +do_bootstrap[dirs] = "${DEPLOY_DIR_BOOTSTRAP} ${BOOTSTRAP_TMPDIR}" +do_bootstrap[depends] = "base-apt:do_cache isar-apt:do_cache_config" +do_bootstrap[network] = "${TASK_USE_NETWORK_AND_SUDO}" + +do_bootstrap() { + if [ "${ISAR_ENABLE_COMPAT_ARCH}" = "1" ]; then + if [ -z "${COMPAT_DISTRO_ARCH}" ]; then + bbfatal "${DISTRO_ARCH} does not have a compat arch" + fi + fi + bootstrap_args="--verbose --variant=minbase --include=${DISTRO_BOOTSTRAP_BASE_PACKAGES}" + if [ -f "${DISTRO_BOOTSTRAP_KEYRING}" ]; then + bootstrap_args="$bootstrap_args --keyring=${DISTRO_BOOTSTRAP_KEYRING}" + fi + E="${@ isar_export_proxies(d)}" + export BOOTSTRAP_FOR_HOST + + deb_dl_dir_import "${ROOTFSDIR}" "${BOOTSTRAP_BASE_DISTRO}-${BASE_DISTRO_CODENAME}" + sudo rm -rf --one-file-system "${ROOTFSDIR}" + mkdir -p "${ROOTFSDIR}" + + arch_param="--arch=${BOOTSTRAP_DISTRO_ARCH},${DISTRO_ARCH}" + + sudo TMPDIR="${BOOTSTRAP_TMPDIR}" mmdebstrap $bootstrap_args \ + $arch_param \ + --mode=unshare \ + ${@get_distro_components_argument(d)} \ + "${@get_distro_suite(d)}" \ + "${WORKDIR}/rootfs.tar.zst" \ + "${@get_distro_source(d)}" + + sudo -E -s <<'EOSUDO' + set -e + + tar -xf "${WORKDIR}/rootfs.tar.zst" -C "${ROOTFSDIR}" --exclude="./dev/console" + + # Install apt config + mkdir -p "${ROOTFSDIR}/etc/apt/preferences.d" + install -v -m644 "${APTPREFS}" \ + "${ROOTFSDIR}/etc/apt/preferences.d/bootstrap" + mkdir -p "${ROOTFSDIR}/etc/apt/sources.list.d" + if [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ]; then + line="file:///base-apt/${BOOTSTRAP_BASE_DISTRO} ${BASE_DISTRO_CODENAME} main" + if [ -z "${BASE_REPO_KEY}" ]; then + line="[trusted=yes] ${line}" + fi + echo "deb ${line}" > "${ROOTFSDIR}/etc/apt/sources.list.d/base-apt.list" + line="file:///base-apt/${BASE_DISTRO} ${BASE_DISTRO_CODENAME} main" + if [ -z "${BASE_REPO_KEY}" ]; then + line="[trusted=yes] ${line}" + fi + echo "deb-src ${line}" >> "${ROOTFSDIR}/etc/apt/sources.list.d/base-apt.list" + + mkdir -p ${ROOTFSDIR}/base-apt + mount --bind ${REPO_BASE_DIR} ${ROOTFSDIR}/base-apt + else + install -v -m644 "${APTSRCS}" \ + "${ROOTFSDIR}/etc/apt/sources.list.d/bootstrap.list" + fi + install -v -m644 "${APTSRCS_INIT}" "${ROOTFSDIR}/etc/apt/sources-list" + rm -f "${ROOTFSDIR}/etc/apt/sources.list" + rm -rf "${ROOTFSDIR}/var/lib/apt/lists/"* + find ${APT_KEYS_DIR}/ -type f | while read keyfile + do + MY_GPGHOME="$(chroot "${ROOTFSDIR}" mktemp -d /tmp/gpghomeXXXXXXXXXX)" + echo "Created temporary directory ${MY_GPGHOME} for gpg-agent" + export GNUPGHOME="${MY_GPGHOME}" + APT_KEY_APPEND="--homedir ${MY_GPGHOME}" + + kfn="$(basename $keyfile)" + cp $keyfile "${ROOTFSDIR}/tmp/$kfn" + chroot "${ROOTFSDIR}" /usr/bin/gpg-agent --daemon -- /usr/bin/apt-key \ + --keyring ${THIRD_PARTY_APT_KEYRING} ${APT_KEY_APPEND} add "/tmp/$kfn" + rm "${ROOTFSDIR}/tmp/$kfn" + + echo "Removing ${MY_GPGHOME}" + rm -rf "${ROOTFSDIR}${MY_GPGHOME}" + done + + # Set locale + install -v -m644 "${WORKDIR}/locale" "${ROOTFSDIR}/etc/locale" + + sed -i '/en_US.UTF-8 UTF-8/s/^#//g' "${ROOTFSDIR}/etc/locale.gen" + chroot "${ROOTFSDIR}" /usr/sbin/locale-gen + + # setup chroot + install -v -m755 "${WORKDIR}/chroot-setup.sh" "${ROOTFSDIR}/chroot-setup.sh" + "${ROOTFSDIR}/chroot-setup.sh" "setup" "${ROOTFSDIR}" + + # update APT + mount -o bind,private /dev ${ROOTFSDIR}/dev + mount --bind /dev/pts ${ROOTFSDIR}/dev/pts + mount -t tmpfs none "${ROOTFSDIR}/dev/shm" + mount -t proc none ${ROOTFSDIR}/proc + mount --rbind /sys ${ROOTFSDIR}/sys + mount --make-rslave ${ROOTFSDIR}/sys + + export DEBIAN_FRONTEND=noninteractive + + if [ "${BOOTSTRAP_FOR_HOST}" = "1" ]; then + chroot "${ROOTFSDIR}" /usr/bin/dpkg --add-architecture ${DISTRO_ARCH} + fi + + if [ "${ISAR_ENABLE_COMPAT_ARCH}" = "1" ]; then + chroot "${ROOTFSDIR}" /usr/bin/dpkg --add-architecture ${COMPAT_DISTRO_ARCH} + fi + + chroot "${ROOTFSDIR}" /usr/bin/apt-get update -y \ + -o APT::Update::Error-Mode=any + chroot "${ROOTFSDIR}" /usr/bin/apt-get install -y -f + chroot "${ROOTFSDIR}" /usr/bin/apt-get dist-upgrade -y \ + -o Debug::pkgProblemResolver=yes + + umount -l "${ROOTFSDIR}/dev/shm" + umount -l "${ROOTFSDIR}/dev/pts" + umount -l "${ROOTFSDIR}/dev" + umount -l "${ROOTFSDIR}/proc" + umount -l "${ROOTFSDIR}/sys" + umount -l "${ROOTFSDIR}/base-apt" || true + + # Finalize bootstrap by setting the link in deploy + ln -Tfsr "${ROOTFSDIR}" "${DEPLOY_ISAR_BOOTSTRAP}" +EOSUDO + deb_dl_dir_export "${ROOTFSDIR}" "${BOOTSTRAP_BASE_DISTRO}-${BASE_DISTRO_CODENAME}" + + # Cleanup apt cache + sudo -Es chroot "${ROOTFSDIR}" /usr/bin/apt-get -y clean +} +addtask bootstrap before do_build after do_generate_keyrings + +SSTATETASKS += "do_bootstrap" +SSTATECREATEFUNCS += "bootstrap_sstate_prepare" +SSTATEPOSTINSTFUNCS += "bootstrap_sstate_finalize" + +bootstrap_sstate_prepare() { + # this runs in SSTATE_BUILDDIR, which will be deleted automatically + sudo cp -a "$(dirname "${ROOTFSDIR}")/rootfs.tar.zst" ./bootstrap.tar.zst + sudo chown $(id -u):$(id -g) bootstrap.tar.zst +} + +bootstrap_sstate_finalize() { + # this runs in SSTATE_INSTDIR + # we should restore symlinks after using tar + if [ -f bootstrap.tar.zst ]; then + mv bootstrap.tar.zst "$(dirname "${ROOTFSDIR}")/rootfs.tar.zst" + sudo ln -Tfsr "$(dirname "${ROOTFSDIR}")/rootfs.tar.zst" \ + "${DEPLOY_ISAR_BOOTSTRAP}.tar.zst" + fi +} + +python do_bootstrap_setscene() { + sstate_setscene(d) +} + +addtask do_bootstrap_setscene +do_bootstrap_setscene[dirs] = "${DEPLOY_DIR_BOOTSTRAP}" -- 2.34.1