From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6520117824876183552 X-Received: by 10.28.192.26 with SMTP id q26mr83887wmf.29.1518089209970; Thu, 08 Feb 2018 03:26:49 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 10.28.26.132 with SMTP id a126ls939387wma.1.canary-gmail; Thu, 08 Feb 2018 03:26:49 -0800 (PST) X-Google-Smtp-Source: AH8x226O40OX1Yb97ulgVfTXelh5TFTuBqiSiUKV1qrgExrhLFJhT6RF0LETbvsKSh1OE2gqKKmK X-Received: by 10.28.184.88 with SMTP id i85mr83965wmf.22.1518089209344; Thu, 08 Feb 2018 03:26:49 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1518089209; cv=none; d=google.com; s=arc-20160816; b=yJjqq1QuJQzcYzgJfWQJMwbWc1RmKeVdfweR96799+XntRQzSPZSYhk8+48vUudVov vCH562QiGLwe80wpGI+53SR2GTYr/zJNKrH277re3D/6XRSd/5GKSmTK1flclwrszBZ5 3zCH1dXaVyJeuKLSH/OWzCTFKRTPPrRAazl5qYH26C0u31d3u66btT2rMZIpj5mOowSa BMS2kL5Ke/RKQDZ/jWHI9FQ0DVJPRwFKz60ALW1AEUI+bGYoyXhrfyLmhGXBw/9yiwrX PO2FdGzxhveRJjmDbRs5p4OkxTIB+037fF+oKWbTJ+0mDfwCccTvGNPh+7YysASeB6SB dgGg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:content-language:in-reply-to:mime-version :user-agent:date:message-id:from:references:to:subject :arc-authentication-results; bh=fcK7KEI4AqLIwkBIXTSgZzwGYc9X+mLk8x8ETKoWL54=; b=ApSoKIcUdq8s1i3P4K5u3P49SMczPjzsd43m7TUKBA7fGJBDkqzGc12R8v44V/ZRbo z3yZ+NNmLFaBwryefU6M/bAHEocx3cnKNgWwf5nKJgenxUIqYOEJ6a/EOWOG3d2BTb3T IOFVqkgNo5EgDHIMCIoOpNPHpJHFkgi2d1iCU6vvK8CLT4zP1vrnkhsbEHlGTeS5mobR KDgxqw4O1LnWERsMYMAJuJEN4FCrqUkgVqKwidWQadynadeoj7nyci5TasxQu8HUrenl w8LrdTK5dXrd4EFGOglNrEWyzD0XcriyBevNYyXEHGS2xxdHTu9YAhKJnRIo+f8VQKSV 4uuQ== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of jan.kiszka@siemens.com designates 192.35.17.14 as permitted sender) smtp.mailfrom=jan.kiszka@siemens.com Return-Path: Received: from david.siemens.de (david.siemens.de. [192.35.17.14]) by gmr-mx.google.com with ESMTPS id m23si258084wrb.4.2018.02.08.03.26.49 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 08 Feb 2018 03:26:49 -0800 (PST) Received-SPF: pass (google.com: domain of jan.kiszka@siemens.com designates 192.35.17.14 as permitted sender) client-ip=192.35.17.14; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of jan.kiszka@siemens.com designates 192.35.17.14 as permitted sender) smtp.mailfrom=jan.kiszka@siemens.com Received: from mail2.siemens.de (mail2.siemens.de [139.25.208.11]) by david.siemens.de (8.15.2/8.15.2) with ESMTPS id w18BQmxm011040 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 8 Feb 2018 12:26:49 +0100 Received: from [139.25.68.37] (md1q0hnc.ad001.siemens.net [139.25.68.37] (may be forged)) by mail2.siemens.de (8.15.2/8.15.2) with ESMTP id w18BQmrk005895; Thu, 8 Feb 2018 12:26:48 +0100 Subject: Re: [RFC PATCH 1/1] RFC-PATCH: partial implementation for support of repos from other layers To: Benedikt Niedermayr , isar-users@googlegroups.com References: <20180208094653.1819-1-claudius.heine.ext@siemens.com> <20180208094653.1819-2-claudius.heine.ext@siemens.com> From: Jan Kiszka Message-ID: Date: Thu, 8 Feb 2018 12:26:48 +0100 User-Agent: Mozilla/5.0 (X11; U; Linux i686 (x86_64); de; rv:1.8.1.12) Gecko/20080226 SUSE/2.0.0.12-1.1 Thunderbird/2.0.0.12 Mnenhy/0.7.5.666 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 8bit X-TUID: /13fUOrr/Cum On 2018-02-08 12:09, 'Benedikt Niedermayr' via isar-users wrote: > Am 08.02.2018 um 10:46 schrieb claudius.heine.ext@siemens.com: >> From: Claudius Heine >> >> This patch changes the implementation of isar-image-base to support >> the definition of additional repositories from other layers, by addding >> apt source list compatible files to the 'DEBIAN_APT_SOURCES' variable. >> >> This also implement the generation of the multistrap.conf file from >> within the recipe. >> >> Signed-off-by: Claudius Heine >> --- >>   meta-isar/conf/distro/debian-stretch.conf          |   6 +- >>   meta-isar/conf/distro/stretch.list                 |   3 + >>   .../recipes-core/images/files/multistrap.conf.in   |  38 ------ >>   meta-isar/recipes-core/images/isar-image-base.bb   | 134 >> +++++++++++++++++---- >>   4 files changed, 117 insertions(+), 64 deletions(-) >>   create mode 100644 meta-isar/conf/distro/stretch.list >>   delete mode 100644 >> meta-isar/recipes-core/images/files/multistrap.conf.in >> >> diff --git a/meta-isar/conf/distro/debian-stretch.conf >> b/meta-isar/conf/distro/debian-stretch.conf >> index 74177fd..083eb86 100644 >> --- a/meta-isar/conf/distro/debian-stretch.conf >> +++ b/meta-isar/conf/distro/debian-stretch.conf >> @@ -1,10 +1,14 @@ >>   # This software is a part of ISAR. >>   # Copyright (C) 2017 ilbers GmbH >>   +DEBIAN_APT_SOURCES += "conf/distro/stretch.list" >> +DEBIAN_APT_PREFERENCES ??= "" >> +DISTRO_CONFIG_SCRIPT ??= "debian-configscript.sh" >> + >> +# No longer needed, after buildchroot is converted: >>   DISTRO_SUITE ?= "stretch" >>   DISTRO_COMPONENTS ?= "main contrib non-free" >>   DISTRO_APT_SOURCE ?= "http://ftp.debian.org/debian" >>   DISTRO_APT_SOURCE_SEC ?= "http://security.debian.org/debian-security" >> -DISTRO_CONFIG_SCRIPT ?= "debian-configscript.sh" >>   DISTRO_MULTICONF_BOOTSTRAP ?= "base updates security" >>   DISTRO_MULTICONF_APTSOURCES ?= "${DISTRO_MULTICONF_BOOTSTRAP}" >> diff --git a/meta-isar/conf/distro/stretch.list >> b/meta-isar/conf/distro/stretch.list >> new file mode 100644 >> index 0000000..a5ed5db >> --- /dev/null >> +++ b/meta-isar/conf/distro/stretch.list >> @@ -0,0 +1,3 @@ >> +deb    http://ftp.de.debian.org/debian    stretch        main contrib >> non-free >> +deb    http://ftp.de.debian.org/debian stretch-updates    main >> contrib non-free >> +deb    http://security.debian.org    stretch/updates    main contrib >> non-free >> diff --git a/meta-isar/recipes-core/images/files/multistrap.conf.in >> b/meta-isar/recipes-core/images/files/multistrap.conf.in >> deleted file mode 100644 >> index 432b6af..0000000 >> --- a/meta-isar/recipes-core/images/files/multistrap.conf.in >> +++ /dev/null >> @@ -1,38 +0,0 @@ >> -# This software is a part of ISAR. >> -# Copyright (C) 2015-2017 ilbers GmbH >> - >> -[General] >> -noauth=true >> -unpack=true >> -ignorenativearch=true >> -bootstrap=##DISTRO_MULTICONF_BOOTSTRAP## Isar >> -aptsources=##DISTRO_MULTICONF_APTSOURCES## >> -configscript=##CONFIG_SCRIPT## >> -setupscript=##SETUP_SCRIPT## >> -hookdir=##DIR_HOOKS## >> - >> -[base] >> -source=##DISTRO_APT_SOURCE## >> -suite=##DISTRO_SUITE## >> -components=##DISTRO_COMPONENTS## >> -packages=##IMAGE_PREINSTALL## >> -omitdebsrc=true >> - >> -[updates] >> -source=##DISTRO_APT_SOURCE## >> -suite=##DISTRO_SUITE##-updates >> -components=##DISTRO_COMPONENTS## >> -omitdebsrc=true >> - >> -[security] >> -source=##DISTRO_APT_SOURCE_SEC## >> -suite=##DISTRO_SUITE##/updates >> -components=##DISTRO_COMPONENTS## >> -omitdebsrc=true >> - >> -[Isar] >> -packages=##IMAGE_INSTALL## >> -source=##DEPLOY_DIR_APT## >> -suite=##ISAR_DISTRO_SUITE## >> -components=main >> -omitdebsrc=true >> diff --git a/meta-isar/recipes-core/images/isar-image-base.bb >> b/meta-isar/recipes-core/images/isar-image-base.bb >> index e359ac3..655dbb2 100644 >> --- a/meta-isar/recipes-core/images/isar-image-base.bb >> +++ b/meta-isar/recipes-core/images/isar-image-base.bb >> @@ -9,8 +9,7 @@ LICENSE = "gpl-2.0" >>   LIC_FILES_CHKSUM = >> "file://${LAYERDIR_isar}/licenses/COPYING.GPLv2;md5=751419260aa954499f7abaabaa882bbe" >> >>     FILESPATH =. "${LAYERDIR_isar}/recipes-core/images/files:" >> -SRC_URI = "file://multistrap.conf.in \ >> -       file://${DISTRO_CONFIG_SCRIPT} \ >> +SRC_URI = "file://${DISTRO_CONFIG_SCRIPT} \ >>          file://setup.sh \ >>          file://download_dev-random" >>   @@ -26,34 +25,119 @@ IMAGE_PREINSTALL += "apt \ >>     WORKDIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/${PN}" >>   -do_rootfs[dirs] = "${WORKDIR}/hooks_multistrap" >> - >> -do_rootfs() { >> -    E="${@ bb.utils.export_proxies(d)}" >> +do_multistrap_prepare[dirs] = "${WORKDIR}/hooks_multistrap" >>   +do_multistrap_prepare() { >>       chmod +x "${WORKDIR}/${DISTRO_CONFIG_SCRIPT}" >>       chmod +x "${WORKDIR}/setup.sh" >>       install -m 755 "${WORKDIR}/download_dev-random" >> "${WORKDIR}/hooks_multistrap/" >> +} >> +addtask multistrap_prepare before do_multistrap_config after do_unpack >> + >> +python do_multistrap_config() { >> +    import shutil >> +    import configparser as cp >> +    import textwrap >> + >> +    workdir = d.getVar("WORKDIR", True) >> +    workdir_rel = os.path.relpath(workdir) >> + >> +    multistrap_conf_file = os.path.join(workdir, "multistrap.conf") >> + >> +    apt_sources_list = d.getVar("DEBIAN_APT_SOURCES", True).split() >> + >> +    apt_preferences_list = d.getVar("DEBIAN_APT_PREFERENCES", >> True).split() >> +    apt_preferences_out = os.path.join(workdir_rel, "apt-preferences") >> + >> +    image_preinstall = d.getVar("IMAGE_PREINSTALL", True).split() >> +    image_install = d.getVar("IMAGE_INSTALL", True).split() >> + >> +    repos = [] >> + >> +    msconf = cp.RawConfigParser() >> + >> +    # Set general multistrap configuration: >> +    msconf.add_section("General") >> +    msconf.set("General", "noauth", "true") >> +    msconf.set("General", "unpack", "true") >> +    msconf.set("General", "ignorenativearch", "true") >> +    msconf.set("General", "aptpreferences", "./" + apt_preferences_out) >> +    msconf.set("General", "configscript", "./" + >> +               os.path.join(workdir_rel, >> +                            d.getVar("DISTRO_CONFIG_SCRIPT", True))) >> +    msconf.set("General", "setupscript", "./" + >> +               os.path.join(workdir_rel, >> +                            "setup.sh")) >> +    msconf.set("General", "hookdir", "./" + >> +               os.path.join(workdir_rel, >> +                            "hooks_multistrap")) >> + >> +    # Set repository in multistrap configuration: >> +    repo_counter = 0 >> +    for aptsrc in apt_sources_list: >> +        aptsrc_real = bb.parse.resolve_file(aptsrc, d) >> +        with open(aptsrc_real, "r") as in_fd: >> +            for line in in_fd: >> +                entry = line.split() >> + >> +                # Only care about 'deb' repos for now: >> +                if entry[0] != "deb": >> +                    continue >> + >> +                # Generate on unique name for the section >> +                repo_name = "repo{}".format(repo_counter) >> +                repos.append(repo_name) >> + >> +                source = entry[1] >> +                suite = entry[2] >> +                components = entry[3:] >> + >> +                msconf.add_section(repo_name) >> +                msconf.set(repo_name, "source", source) >> +                msconf.set(repo_name, "suite", suite) >> +                msconf.set(repo_name, "components", " >> ".join(components)) >> +                msconf.set(repo_name, "omitdebsrc", "true") >> + >> +                # The first repo gets assigned all IMAGE_PREINSTALL >> +                # packages: >> +                if image_preinstall: >> +                   msconf.set(repo_name, "packages", >> +                              " ".join(image_preinstall)) >> +                   image_preinstall = [] >> + >> +                repo_counter += 1 >> + >> +    msconf.add_section("isar-generated") >> +    msconf.set("isar-generated", "packages", " ".join(image_install)) >> +    msconf.set("isar-generated", "source", "copy:///{}/{}".format( >> +               d.getVar("DEPLOY_DIR_APT", True), d.getVar("DISTRO", >> True))) >> +    msconf.set("isar-generated", "suite", d.getVar("DEBDISTRONAME", >> True)) >> +    msconf.set("isar-generated", "components", "main") >> +    msconf.set("isar-generated", "omitdebsrc", "true") >> + >> +    # Generate apt preference file: >> +    with open(apt_preferences_out, "wb") as out_fd: >> +        for aptpref in apt_preferences_list: >> +            artpref_real = bb.parse.resolve_file(aptpref, d) >> +            with open(aptpref_real, "rb") as in_fd: >> +                 shutil.copyfileobj(in_fd, out_fd, 1024*1024*10) >> +            out_fd.write("\n".encode()) >> +        out_fd.write(textwrap.dedent("""\ >> +                     Package: * >> +                     Pin: release n={} >> +                     Pin-Priority: 1000 >> +                     """.format(d.getVar("DEBDISTRONAME", >> True))).encode()) >> + >> +    msconf.set("General", "bootstrap", " ".join(repos)) >> +    msconf.set("General", "aptsources", " ".join(repos + >> ["isar-generated"])) >> + >> +    with open(multistrap_conf_file, "w") as out_fd: >> +        msconf.write(out_fd) >> +} >> +addtask multistrap_config after do_multistrap_prepare before do_rootfs >>   -    # Multistrap accepts only relative path in configuration files, >> so get it: >> -    cd ${TOPDIR} >> -    WORKDIR_REL=${@ os.path.relpath(d.getVar("WORKDIR", True))} >> - >> -    # Adjust multistrap config >> -    sed -e 's|##IMAGE_PREINSTALL##|${IMAGE_PREINSTALL}|g' \ >> -        -e >> 's|##DISTRO_MULTICONF_BOOTSTRAP##|${DISTRO_MULTICONF_BOOTSTRAP}|g' \ >> -        -e >> 's|##DISTRO_MULTICONF_APTSOURCES##|${DISTRO_MULTICONF_APTSOURCES}|g' \ >> -        -e 's|##DISTRO_APT_SOURCE##|${DISTRO_APT_SOURCE}|g' \ >> -        -e 's|##DISTRO_APT_SOURCE_SEC##|${DISTRO_APT_SOURCE_SEC}|g' \ >> -        -e 's|##DISTRO_SUITE##|${DISTRO_SUITE}|g' \ >> -        -e 's|##DISTRO_COMPONENTS##|${DISTRO_COMPONENTS}|g' \ >> -        -e >> 's|##CONFIG_SCRIPT##|./'"$WORKDIR_REL"'/${DISTRO_CONFIG_SCRIPT}|g' \ >> -        -e 's|##SETUP_SCRIPT##|./'"$WORKDIR_REL"'/setup.sh|g' \ >> -        -e 's|##DIR_HOOKS##|./'"$WORKDIR_REL"'/hooks_multistrap|g' \ >> -        -e 's|##IMAGE_INSTALL##|${IMAGE_INSTALL}|g' \ >> -        -e >> 's|##DEPLOY_DIR_APT##|copy:///${DEPLOY_DIR_APT}/${DISTRO}|g' \ >> -        -e 's|##ISAR_DISTRO_SUITE##|${DEBDISTRONAME}|g' \ >> -           "${WORKDIR}/multistrap.conf.in" > >> "${WORKDIR}/multistrap.conf" >> +do_rootfs() { >> +    E="${@ bb.utils.export_proxies(d)}" >>         [ ! -d ${IMAGE_ROOTFS}/proc ] && sudo install -d -o 0 -g 0 -m >> 555 ${IMAGE_ROOTFS}/proc >>       sudo mount -t proc none ${IMAGE_ROOTFS}/proc > Hi, > > thanks.... I think it  goes into the right direction. > > My proposal: > > - Saving all Parts of  conf/distro/stretch.list in bitbake variables, so > we can reuse them in other parts of Isar. >   Maybe something like >   DISTRO_CONFIG_LIST = "foo" >   DISTRO_CONFIG_LIST[type] = "deb" >   DISTRO_CONFIG_LIST[url] = "http://ftp.de.debian.org/debian " >   DISTRO_CONFIG_LIST[distro] = "stretch" >   DISTRO_CONFIG_LIST[components] = "main contrib non-free" > >   Those vars should be stored in global context which in turn would > presuppose >   running  do_multistrap_config() in a global context, in order to > getting access to those vars from everywhere. The variables can come from normal config files (distro.conf, multiconf etc.) which are generally available. However, I think that distributing the per-repo information in such a way over multiple bitbake variables will make them ugly to parse and highly fragile. E.g., when someone adds a repo but forgets to specify one of the 4 parameters, the array will become unbalanced and the association will be subtly wrong. Therefore the proposal to list everything in sources.list format in fragment files. Jan -- Siemens AG, Corporate Technology, CT RDA IOT SES-DE Corporate Competence Center Embedded Linux