Re: [PATCH] isar-bootstrap: debootstrap --include flag to support https:// sources

[Henning Schild <henning.schild@siemens.com>]

Very good idea and thanks for looking into that. Please make sure
to add a CI test. Would be simple as changing one of our .list files to
have http in the first line and at least one https in another. And
maybe comment on the weird mix inside the file. I would suggest the
buster one.

Henning

Am Tue, 4 Dec 2018 16:35:44 +0300
schrieb "Hosgor, Tolga (CT RDA DS EU TR MTS)" <tlghosgor@gmail.com>:

> Building 'isar-bootstrap-target' fails when there are HTTPS URIs in
> the distro APT sources and the first URI is not HTTPS.
> The first URI in the APT sources is passed to 'debootstrap' and
> the distro APT sources file is written to the isar bootstrap rootfs.
> Then, following 'apt-get update' fails due to apt-transport-https,
> ca-certificates being missing. In a normal workflow, 'debootstrap'
> automatically includes them if the MIRROR is HTTPS.
> 
> Signed-off-by: Hosgor, Tolga (CT RDA DS EU TR MTS)
> <tolga.hosgor@siemens.com> ---
>  .../isar-bootstrap/isar-bootstrap.inc         | 28
> +++++++++++++++++-- 1 file changed, 25 insertions(+), 3 deletions(-)
> 
> diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
> b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc index
> cfad136..33ad8e8 100644 ---
> a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc +++
> b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc @@ -109,11
> +109,15 @@ def aggregate_aptsources_list(d, file_list, file_out):
> out_fd.write("\n".encode()) out_fd.write("\n".encode())
>  
> -def get_distro_primary_source_entry(d, is_host=False):
> +def get_aptsources_list(d, is_host=False):
>      if is_host:
>          apt_sources_list = (d.getVar("HOST_DISTRO_APT_SOURCES",
> True) or "").split() else:
>          apt_sources_list = (d.getVar("DISTRO_APT_SOURCES", True) or
> "").split()
> +    return apt_sources_list
> +
> +def get_distro_primary_source_entry(d, is_host=False):
> +    apt_sources_list = get_aptsources_list(d, is_host)
>      for entry in apt_sources_list:
>          entry_real = bb.parse.resolve_file(entry, d)
>          with open(entry_real, "r") as in_fd:
> @@ -125,6 +129,18 @@ def get_distro_primary_source_entry(d,
> is_host=False): return parsed[2:]
>      return ["", "", ""]
>  
> +def get_distro_have_https_source(d, is_host=False):
> +    for entry in get_aptsources_list(d, is_host):
> +        entry_real = bb.parse.resolve_file(entry, d)
> +        with open(entry_real, "r") as in_fd:
> +            for line in in_fd:
> +                parsed = parse_aptsources_list_line(line)
> +                if parsed:
> +                    parsed = get_apt_source_mirror(d, parsed)
> +                    if parsed[2].startswith("https://"):
> +                        return True
> +    return False
> +
>  def get_distro_source(d, is_host):
>      return get_distro_primary_source_entry(d, is_host)[0]
>  
> @@ -138,6 +154,12 @@ def get_distro_components_argument(d, is_host):
>      else:
>          return ""
>  
> +def get_debootstrap_includes(d, is_host=False):
> +    if get_distro_have_https_source(d, is_host):
> +        return "locales,apt-transport-https,ca-certificates"
> +    else:
> +        return "locales"
> +
>  do_generate_keyring[dirs] = "${DL_DIR}"
>  do_generate_keyring[vardeps] += "DISTRO_APT_KEYS"
>  do_generate_keyring() {
> @@ -183,7 +205,7 @@ isar_bootstrap() {
>              if [ ${IS_HOST} ]; then
>                  ${DEBOOTSTRAP} --verbose \
>                                 --variant=minbase \
> -                               --include=locales \
> +
> --include="${@get_debootstrap_includes(d, True)}" \
> ${@get_distro_components_argument(d, True)} \ ${DEBOOTSTRAP_KEYRING} \
>                                 "${@get_distro_suite(d, True)}" \
> @@ -194,7 +216,7 @@ isar_bootstrap() {
>                   "${DEBOOTSTRAP}" --verbose \
>                                    --variant=minbase \
>                                    --arch="${DISTRO_ARCH}" \
> -                                  --include=locales \
> +
> --include="${@get_debootstrap_includes(d, False)}" \
> ${@get_distro_components_argument(d, False)} \ ${DEBOOTSTRAP_KEYRING}
> \ "${@get_distro_suite(d, False)}" \