From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6635926855683670016 X-Received: by 2002:a05:6000:114e:: with SMTP id d14mr58107wrx.5.1545047122973; Mon, 17 Dec 2018 03:45:22 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a1c:280a:: with SMTP id o10ls446wmo.11.gmail; Mon, 17 Dec 2018 03:45:22 -0800 (PST) X-Google-Smtp-Source: AFSGD/VtjFpKFOW809hvxxNDcHvDA81kJ7SX+ONkWLE6OBKsmLyEARC/8MJyEIhejwgWeuKBvWKL X-Received: by 2002:a1c:ac87:: with SMTP id v129mr1544162wme.20.1545047122114; Mon, 17 Dec 2018 03:45:22 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1545047122; cv=none; d=google.com; s=arc-20160816; b=ZsXZKV6oQdYud04vEnTTH98mPO09sGEIe6a+JtJn9gVPLRTIj2cc0yWuRJKZxaMAky vJ4p44RkHu0xAs9PeFKIi4qVvHlzxSXYth4eUkumfJv6kWgytzL3JA6Zx/vNVdMUfmZ1 78L6NsVkMoTKJbcWZsR5gA2e9zmkfeX51pZ4niMFQjxSAUvhwy4miYHzR+HIj5g9QqS/ a8PAfaut3jt//iR4tqnfsq7tfidjNCksJMdhV0vbf1OwDs6zK9hZMvE9+Q/6I9TZDjXW SkC0X78bS5W9652JfBujvRI27nHfBGWy3PQAHyM72LIdiEVtOw5vCbWQDsH5Kd0Ktvb5 DFAQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:dkim-signature; bh=8aanXl8j9y41XQTB+A1xKc0R/evSgCRLAFVuSFPB+kQ=; b=R3l1Ko3U2qk0I+UQp6Xpgv5imJhihDIhAr25BrT9oZIQUgeRoDRHgXAx5oDbUJrrDk tJzmZlrzz4PvGXAR3ppCrW5VohJOP7S5ZVAfZywsAnCrsC6C1Fb/8gwQ/PJLGrfFK/z4 zA8IrxE+RRGgDoSzjdSjxu/3nNLi//NlwbqvyBFkGw/kG65TJ28lq8fTLmWAJLneeWa/ S6opCVOl8+tN0x+ctR2ey7Q0s728t8FAGYKVrIPTuIKXy+NXGmx/Dei0k/mTSbnCHN4T ntU5crtpgMZYKc8DS9VM6kvmXjyq8hqR7WLArKcbm5LlqqB3woLqNV91G7Y+5JaW2g4S eNZA== ARC-Authentication-Results: i=1; gmr-mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=KMV7v0Cg; spf=pass (google.com: domain of tlghosgor@gmail.com designates 2a00:1450:4864:20::344 as permitted sender) smtp.mailfrom=tlghosgor@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from mail-wm1-x344.google.com (mail-wm1-x344.google.com. [2a00:1450:4864:20::344]) by gmr-mx.google.com with ESMTPS id h6si1065983wmc.2.2018.12.17.03.45.22 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 17 Dec 2018 03:45:22 -0800 (PST) Received-SPF: pass (google.com: domain of tlghosgor@gmail.com designates 2a00:1450:4864:20::344 as permitted sender) client-ip=2a00:1450:4864:20::344; Authentication-Results: gmr-mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=KMV7v0Cg; spf=pass (google.com: domain of tlghosgor@gmail.com designates 2a00:1450:4864:20::344 as permitted sender) smtp.mailfrom=tlghosgor@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: by mail-wm1-x344.google.com with SMTP id f81so12229456wmd.4 for ; Mon, 17 Dec 2018 03:45:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=8aanXl8j9y41XQTB+A1xKc0R/evSgCRLAFVuSFPB+kQ=; b=KMV7v0CgtzM3TQZ2zdMHSv7mv76ud2WW+AOMj/ZzURzMYedeJ/ssibt4dcYQkc8ewF VKkT9WyFTWgaVJ2JVlDKmCNsF+XEc4r/Q3lvDr1VoS2Xd4cY1jPaaxPDy0fz0AvYoE0Z s0Bvc3sVDncrGJa+S2yjd3QWCgDUvr6EmJPJ+mPuhW6H1e6n/RV27pUG9MBOa9nk35DX EhGV4UUlQcejx41kJQ3KeVoOHNu1jHF4e8h/bmv27PXeLV82sTMd/wbewdYmWZ3PRJd/ QEYIdb1kyo9k/dO5FfyoI3wMfNqHWiwMF3LukuDYyuDg/ENMmXYjgFftT6NfanQUPBGD aQMA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=8aanXl8j9y41XQTB+A1xKc0R/evSgCRLAFVuSFPB+kQ=; b=K6dll6uMoNnkPXRVq5SwvIWzzt1h3Ar0pzA5SfPlQN1JeLtP/Bnf63TjQ0aA9OTzTJ Frjv6o/M3Th/gtxoVMY+B0n3Unboa2NRy39Dym5MJ3SCaD1emxl2QgCwP/nsb8/19Cxa HNrvhxSzlFbS7zU3I+usIhAedHJwi8Do/lZcALslSuNeV3XonZNc9E9PqXWH8/VUOlXF 3U4l4x+vny0031M9bBjCSnnIJadnaKJwUl+raLqP69l0zJaQrFe8TKpW03PZ1FRrWAIf ZMbr3ihotLGZm+rD80XhG94hMuA0COaJMY4sk00XPzz0rGpWJR9usxxTGWBLlCP/zLlD V8zg== X-Gm-Message-State: AA+aEWau97b1fdo7HXLSeHQ7zsx9FaAWYO1vUYBvdLcyfTdaZKpmLcA6 r2xaPIoQvQ1x6WRNEYcKm02wzOGRiy19QA== X-Received: by 2002:a1c:a3c3:: with SMTP id m186mr11261951wme.16.1545047121325; Mon, 17 Dec 2018 03:45:21 -0800 (PST) Return-Path: Received: from localhost ([165.225.72.104]) by smtp.gmail.com with ESMTPSA id r64sm13093352wmg.5.2018.12.17.03.45.20 for (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Mon, 17 Dec 2018 03:45:20 -0800 (PST) From: "Hosgor, Tolga (CT RDA DS EU TR MTS)" X-Google-Original-From: "Hosgor, Tolga (CT RDA DS EU TR MTS)" To: isar-users@googlegroups.com Subject: [PATCH v2 1/2] isar-bootstrap: debootstrap https support Date: Mon, 17 Dec 2018 14:45:17 +0300 Message-Id: <20181217114518.17995-2-tolga.hosgor@siemens.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20181217114518.17995-1-tolga.hosgor@siemens.com> References: <20181217114518.17995-1-tolga.hosgor@siemens.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-TUID: SmuDxernRxzH Building 'isar-bootstrap-target' fails when there are HTTPS URIs in the distro APT sources and the first URI is not HTTPS. The first URI in the APT sources is passed to 'debootstrap' and the distro APT sources file is written to the isar bootstrap rootfs. Then, following 'apt-get update' fails due to apt-transport-https, ca-certificates being missing. This patch allows a distro to specify 'DISTRO_BOOTSTRAP_BASE_PACKAGES' and introduces 'https-support' concept using bitbake OVERRIDES. An example usage is specifying the necessary packages for http support via 'DISTRO_BOOTSTRAP_BASE_PACKAGES_append_https-support' in distro configuration. Signed-off-by: Hosgor, Tolga (CT RDA DS EU TR MTS) --- meta-isar/conf/distro/debian-buster.list | 1 + meta-isar/conf/distro/debian-jessie.list | 1 + .../conf/multiconfig/qemuamd64-buster.conf | 2 ++ .../conf/multiconfig/qemuamd64-jessie.conf | 2 ++ .../isar-bootstrap/isar-bootstrap-host.bb | 2 ++ .../isar-bootstrap/isar-bootstrap-target.bb | 2 ++ .../isar-bootstrap/isar-bootstrap.inc | 29 +++++++++++++++++-- 7 files changed, 36 insertions(+), 3 deletions(-) diff --git a/meta-isar/conf/distro/debian-buster.list b/meta-isar/conf/distro/debian-buster.list index 18311d8..92d20e1 100644 --- a/meta-isar/conf/distro/debian-buster.list +++ b/meta-isar/conf/distro/debian-buster.list @@ -1,3 +1,4 @@ deb http://ftp.de.debian.org/debian buster main contrib non-free +deb https://debian.inf.tu-dresden.de/debian buster main contrib non-free deb http://ftp.de.debian.org/debian buster-updates main contrib non-free deb http://security.debian.org buster/updates main contrib non-free diff --git a/meta-isar/conf/distro/debian-jessie.list b/meta-isar/conf/distro/debian-jessie.list index be46a57..2471402 100644 --- a/meta-isar/conf/distro/debian-jessie.list +++ b/meta-isar/conf/distro/debian-jessie.list @@ -1,3 +1,4 @@ +deb https://debian.inf.tu-dresden.de/debian jessie main contrib non-free deb http://ftp.de.debian.org/debian jessie main contrib non-free deb http://ftp.de.debian.org/debian jessie-updates main contrib non-free deb http://security.debian.org jessie/updates main contrib non-free diff --git a/meta-isar/conf/multiconfig/qemuamd64-buster.conf b/meta-isar/conf/multiconfig/qemuamd64-buster.conf index 059ea00..394d76b 100644 --- a/meta-isar/conf/multiconfig/qemuamd64-buster.conf +++ b/meta-isar/conf/multiconfig/qemuamd64-buster.conf @@ -15,3 +15,5 @@ QEMU_ARCH ?= "x86_64" QEMU_MACHINE ?= "q35" QEMU_CPU ?= "" QEMU_DISK_ARGS ?= "-hda ##ROOTFS_IMAGE## -bios /usr/local/share/ovmf/OVMF.fd" + +DISTRO_BOOTSTRAP_BASE_PACKAGES_append_https-support = " apt-transport-https ca-certificates" diff --git a/meta-isar/conf/multiconfig/qemuamd64-jessie.conf b/meta-isar/conf/multiconfig/qemuamd64-jessie.conf index ca00e15..d1335ff 100644 --- a/meta-isar/conf/multiconfig/qemuamd64-jessie.conf +++ b/meta-isar/conf/multiconfig/qemuamd64-jessie.conf @@ -14,3 +14,5 @@ QEMU_ARCH ?= "x86_64" QEMU_MACHINE ?= "pc" QEMU_CPU ?= "" QEMU_DISK_ARGS ?= "-hda ##ROOTFS_IMAGE##" + +DISTRO_BOOTSTRAP_BASE_PACKAGES_append_https-support = " apt-transport-https ca-certificates" diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb b/meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb index 55696ea..47cff63 100644 --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb @@ -40,6 +40,8 @@ python do_apt_config_prepare() { } addtask apt_config_prepare before do_bootstrap after do_unpack +OVERRIDES_append = ":${@get_distro_needs_https_support(d, True)}" + do_bootstrap[stamp-extra-info] = "${HOST_DISTRO}-${HOST_ARCH}" do_bootstrap[vardeps] += "HOST_DISTRO_APT_SOURCES" do_bootstrap() { diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap-target.bb b/meta/recipes-core/isar-bootstrap/isar-bootstrap-target.bb index 5752b14..57b607f 100644 --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap-target.bb +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap-target.bb @@ -39,6 +39,8 @@ python do_apt_config_prepare() { } addtask apt_config_prepare before do_bootstrap after do_unpack +OVERRIDES_append = ":${@get_distro_needs_https_support(d, False)}" + do_bootstrap[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}" do_bootstrap[vardeps] += "DISTRO_APT_SOURCES" do_bootstrap() { diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc index cfad136..d868bb6 100644 --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc @@ -23,6 +23,7 @@ APTKEYFILES = "" APTKEYRING = "${WORKDIR}/apt-keyring.gpg" DEBOOTSTRAP_KEYRING = "" DEPLOY_ISAR_BOOTSTRAP ?= "" +DISTRO_BOOTSTRAP_BASE_PACKAGES = "locales" python () { from urllib.parse import urlparse @@ -109,11 +110,15 @@ def aggregate_aptsources_list(d, file_list, file_out): out_fd.write("\n".encode()) out_fd.write("\n".encode()) -def get_distro_primary_source_entry(d, is_host=False): +def get_aptsources_list(d, is_host=False): if is_host: apt_sources_list = (d.getVar("HOST_DISTRO_APT_SOURCES", True) or "").split() else: apt_sources_list = (d.getVar("DISTRO_APT_SOURCES", True) or "").split() + return apt_sources_list + +def get_distro_primary_source_entry(d, is_host=False): + apt_sources_list = get_aptsources_list(d, is_host) for entry in apt_sources_list: entry_real = bb.parse.resolve_file(entry, d) with open(entry_real, "r") as in_fd: @@ -125,6 +130,24 @@ def get_distro_primary_source_entry(d, is_host=False): return parsed[2:] return ["", "", ""] +def get_distro_have_https_source(d, is_host=False): + for entry in get_aptsources_list(d, is_host): + entry_real = bb.parse.resolve_file(entry, d) + with open(entry_real, "r") as in_fd: + for line in in_fd: + parsed = parse_aptsources_list_line(line) + if parsed: + parsed = get_apt_source_mirror(d, parsed) + if parsed[2].startswith("https://"): + return True + return False + +def get_distro_needs_https_support(d, is_host=False): + if get_distro_have_https_source(d, is_host): + return "https-support" + else: + return "" + def get_distro_source(d, is_host): return get_distro_primary_source_entry(d, is_host)[0] @@ -183,7 +206,7 @@ isar_bootstrap() { if [ ${IS_HOST} ]; then ${DEBOOTSTRAP} --verbose \ --variant=minbase \ - --include=locales \ + --include='${DISTRO_BOOTSTRAP_BASE_PACKAGES}' \ ${@get_distro_components_argument(d, True)} \ ${DEBOOTSTRAP_KEYRING} \ "${@get_distro_suite(d, True)}" \ @@ -194,7 +217,7 @@ isar_bootstrap() { "${DEBOOTSTRAP}" --verbose \ --variant=minbase \ --arch="${DISTRO_ARCH}" \ - --include=locales \ + --include='${DISTRO_BOOTSTRAP_BASE_PACKAGES}' \ ${@get_distro_components_argument(d, False)} \ ${DEBOOTSTRAP_KEYRING} \ "${@get_distro_suite(d, False)}" \ -- 2.19.2