From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6631131420328722432 X-Received: by 2002:adf:b717:: with SMTP id l23mr2940550wre.11.1544012856559; Wed, 05 Dec 2018 04:27:36 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a1c:3581:: with SMTP id c123ls3981573wma.9.gmail; Wed, 05 Dec 2018 04:27:36 -0800 (PST) X-Google-Smtp-Source: AFSGD/UWvto4VrWG0rkNTQQ+npr588TbDlhT13hvvpoay3QsObYdguhXyvkeDVPGsAB5D9hy93Ry X-Received: by 2002:a1c:e703:: with SMTP id e3mr2649874wmh.1.1544012856157; Wed, 05 Dec 2018 04:27:36 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1544012856; cv=none; d=google.com; s=arc-20160816; b=gEKJ1kFLq4PbcfAcPkNuDhSrxSrm1HQNgPk0HXwGY+2IQpN1fHVDbp+AHN1q46irY+ GA5LuqnTa7LXq0I0GnqoHFpLMZpgGollTie3ouDH2YkNuZYhIuVWT/FGBH0gdSjYtbq6 dNCJ2zKSOErRUxJSK85vuygCrOKqs8zeedpg0LW4OKDxCwXHP+74axQEbh1VBg1fr327 i36XCKsKW/jheArjuxOjCy7CCvqMUaACQHAzjSjoqqXsPcz9bukZjQ4V3/U6p7xK4l5v v+3+He7YS2gxYfYGHshL/gXnX6zRCrMz+hu0uAJkbgdXWyKfPofYz87Uhy0YEoYXw/MF IiEA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:content-language:in-reply-to:mime-version :user-agent:date:message-id:from:references:cc:to:subject; bh=2xmcesNNrk5ysggPv6FbtrsauaV3F6vW07meo96XKCw=; b=zU/EqRpZsaiVA8AaT7WCV/xvecrHa39bPMxMdUnkPEpHRrhNYQmNyfb3LZNdMI/Hgi yG1LQokDG0F/kTMRutU7chmtYuiar2I2KVU/qv05H5t0U6SNg/VXPskh+3/PvB0xvlnl XupW/y8hKbOHinf5vE52+lu8a7Qw+g13JHlHHwKyGWYqhePRyDf/a/t6+Z1Gl5q26Ti7 utqRDKpk691hft2pfLSsXGrfPozYs9bZRjtKUtVYhysH1nOcJzNrKG+DATS40VXkf9Wa SZVbWzSC7011RuCqM5dzzytDJm1QpXyDOQTV3/b4l2fZ2jB4cviEOgKsb9d6HqlI/Zrf o2fw== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of claudius.heine.ext@siemens.com designates 192.35.17.28 as permitted sender) smtp.mailfrom=claudius.heine.ext@siemens.com Return-Path: Received: from goliath.siemens.de (goliath.siemens.de. [192.35.17.28]) by gmr-mx.google.com with ESMTPS id j45si529407wre.0.2018.12.05.04.27.36 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 05 Dec 2018 04:27:36 -0800 (PST) Received-SPF: pass (google.com: domain of claudius.heine.ext@siemens.com designates 192.35.17.28 as permitted sender) client-ip=192.35.17.28; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of claudius.heine.ext@siemens.com designates 192.35.17.28 as permitted sender) smtp.mailfrom=claudius.heine.ext@siemens.com Received: from mail1.sbs.de (mail1.sbs.de [192.129.41.35]) by goliath.siemens.de (8.15.2/8.15.2) with ESMTPS id wB5CRYKb028378 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 5 Dec 2018 13:27:35 +0100 Received: from [139.25.69.181] (linux-ses-ext02.ppmd.siemens.net [139.25.69.181]) by mail1.sbs.de (8.15.2/8.15.2) with ESMTP id wB5CRYxc011348; Wed, 5 Dec 2018 13:27:34 +0100 Subject: Re: [PATCH] isar-bootstrap: debootstrap --include flag to support https:// sources To: =?UTF-8?B?VG9sZ2EgSG/Fn2fDtnI=?= Cc: jan.kiszka@siemens.com, isar-users@googlegroups.com References: <20181204133544.50621-1-tolga.hosgor@siemens.com> <45323f2d-7dac-9b9d-6382-013b9ae1242c@siemens.com> From: Claudius Heine Message-ID: Date: Wed, 5 Dec 2018 13:27:34 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.3.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 8bit X-TUID: QdkMeYclzUpz Hi Tolga, On 05/12/2018 11.28, Tolga Hoşgör wrote: > On Tue, Dec 4, 2018 at 5:46 PM Claudius Heine > wrote: >> >> Hi Jan, >> >> On 04/12/2018 15.28, Jan Kiszka wrote: >>> On 04.12.18 15:23, [ext] Claudius Heine wrote: >>>>> @@ -138,6 +154,12 @@ def get_distro_components_argument(d, is_host): >>>>> else: >>>>> return "" >>>>> +def get_debootstrap_includes(d, is_host=False): >>>>> + if get_distro_have_https_source(d, is_host): >>>>> + return "locales,apt-transport-https,ca-certificates" >>>> >>>> Those package names should be configured in the distro config. > > I partly agree. It makes sense that it is the responsibility of distro > config but "debootstrap" does this automatically [1]. This > intervention by "debootstrap" creates some erratic behavior. debootstrap has subscripts for each distribution, most of them link to the same script, but that doesn't change the fact that they already have a mechanism in place to implement different logic depending on the distribution. isar-bootstrap has to work with every possible debootstrap implementation, so in order to do that, it shouldn't contain distro specific stuff. > > Scenario 1: > - No special bootstrap config on distro. > - Distro sources have an "https" and "http" in that order. > > Scenario 2: > - No special bootstrap config on distro. > - Distro sources have an "http" and "https" in that order. > > While we were happily building on scenario 1, with a minor change now > the build magically fails on scenario 2. > > An acceptable solution for consistency would be to disable this > behavior [1] but "debootstrap" does not seem to have an option to > disable this magic-package-adding behavior. How about explicitly > `--exclude` the packages for consistency? No explicit excluding of packages please. That would hard code our implementation to internal debootstrap stuff. But if we just add the required packages to debootstrap to handle https when we detect the requirement, then this should work in what ever order the repository sources come. > "debootstrap" also did this in its distro-specific scripts [1], like > your approach. But now it is on a common script "debian-common" [2]. > So now they think this is more or less a standard package I guess. debootstrap should do what ever debootstrap does. What it does in detail shouldn't concern isar. (at least as long as everything works) > Actually what is needed is something the output of: > "DEBOOTSTRAP_DIR=/usr/share/debootstrap source > /usr/share/debootstrap/scripts/; MIRRORS= > work_out_debs && echo $base" but this is not publicly exposed in any > way. Could call `debootstrap --print-debs ` for each mirror > and sort/uniq the result but it would be ugly. Don't go that way. That will just lead us implementing our own debootstrap and that is not currently the scope of isar. >>> Means you would suggest to introduce two variables, e.g. >>> DEBOOTSTRAP_INCLUDE and DEBOOTSTRAP_INCLUDE_HTTPS to encode those >>> packages? Those could then be set in distro confs and simply used here. >> > > Now I am completely lost. I suppose there is no need to create the > logic to to differentiate HTTP and HTTPS in ISAR if distro conf will > set the packages. Distro configuration knows whether if it is using > HTTP or HTTPS (or in theory something else) and it should be enough to > define/extend _one_ variable accordingly. I would suggest calling those variables like this: DISTRO_BOOTSTRAP_BASE_PACKAGES = "locales" DISTRO_BOOTSTRAP_BASE_PACKAGES_append_https-support = " apt-transport-https ca-certificates" # OVERRIDE = "...:https-support:..." # when you detect that https is needed. or similar. And then use those variables in isar-bootstrap with your logic. > Do you agree on adding `--exclude=apt-transport-https,ca-certificates` > and introducing a `BOOTSTRAP_PACKAGES` variable to be configured in > distro? > > [1] https://salsa.debian.org/installer-team/debootstrap/blob/08cfced48bdf7bc1240efb63f69cde7ed385038b/scripts/sid#L36-40 > [2] https://salsa.debian.org/installer-team/debootstrap/blob/4a4e7186de7237c58eae4f0b2de3ad670ad7436f/scripts/debian-common#L30-34 NACK for exclude ACK for BOOTSTRAP_PACKAGES variables (or similar named, see above) kind regards, Claudius -- DENX Software Engineering GmbH, Managing Director: Wolfgang Denk HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de