From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6522397978051739648 X-Received: by 10.28.153.194 with SMTP id b185mr1096693wme.17.1518773478983; Fri, 16 Feb 2018 01:31:18 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 10.28.202.2 with SMTP id a2ls200245wmg.4.gmail; Fri, 16 Feb 2018 01:31:18 -0800 (PST) X-Google-Smtp-Source: AH8x225bhvcZNTsKEjnZtxjp+yxM2wLk/JpcZZ6vwiDlI4KQzSiYqjVsn/uA4Lv9Dy/TPrLmGYSj X-Received: by 10.28.91.147 with SMTP id p141mr1209633wmb.31.1518773478330; Fri, 16 Feb 2018 01:31:18 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1518773478; cv=none; d=google.com; s=arc-20160816; b=C7KSxm0GUnLNDAXV4TTLwNwqHo9+B9JyvbsFsg3jtZ2f1L5SPPQMvx9aWiNm5f7WD4 OvQVXF2ZlQDJlzvT9vvGPH4B27GS2ZI7ZYW3qEWmgo3AmBipc53LUcUza/JX7UxRM8D5 EWc7becIW1302oYADhuC5v1cl506/4V6DFqsoidSFmwe+BMP+c748SL3gSJcYHdhbOCD TKOa9iKrvPrQV6I96RSGexdrzpYDOkRhVktJZ6MPUGDvuitEDYh1dKbHg9PGeLzpt0La H2sKuW2MjBPiquIpfh/fOv5Pdx++yZ/W7TjLCd23AX6y0/eqTKkQYwtoR7OwYzphkXbx O+eg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=in-reply-to:mime-version:user-agent:date:message-id:openpgp:from :references:to:subject:arc-authentication-results; bh=TKJk0dVrSvqUNtEkhjwH55ZmZ2/v67UG8ClpuXvxj40=; b=YgjU6/LQs2uF+Q7ICFmb15i+bPbvoZBvefL0MGi80ASKSBoOEXl5L5KC51rvP04wmU KGdVyHejCWXGJvOBsaUvH6brEf6lEPOmfsa/R19q/jkI40CcuwjD6AaV1aPD12CbGSsG Rz1dGXZOaQ5ls9S5S9eAAR/ulfzFu0MSNud4sQ8zvkymTFS26i+E+9Ho/a4y1hx8OsO0 xn9USPfDz1P7bS102rc+9Ng7hWPnKSd1iJ7TXsQ4IvtGRgJNMIjy+5BLaFzgI1GRnCBA mi37nDC3DTOheSD5sRvUGtOMj/emi1kvebJy1foCFfB/huy3VyApGbOzmYjkKJTs8QF7 9Bfg== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=neutral (google.com: 212.18.0.10 is neither permitted nor denied by best guess record for domain of ch@denx.de) smtp.mailfrom=ch@denx.de Return-Path: Received: from mail-out.m-online.net (mail-out.m-online.net. [212.18.0.10]) by gmr-mx.google.com with ESMTPS id e14si119609wmg.0.2018.02.16.01.31.18 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 16 Feb 2018 01:31:18 -0800 (PST) Received-SPF: neutral (google.com: 212.18.0.10 is neither permitted nor denied by best guess record for domain of ch@denx.de) client-ip=212.18.0.10; Authentication-Results: gmr-mx.google.com; spf=neutral (google.com: 212.18.0.10 is neither permitted nor denied by best guess record for domain of ch@denx.de) smtp.mailfrom=ch@denx.de Received: from frontend01.mail.m-online.net (unknown [192.168.8.182]) by mail-out.m-online.net (Postfix) with ESMTP id 3zjSZF6zxNz1qvP2; Fri, 16 Feb 2018 10:31:17 +0100 (CET) Received: from localhost (dynscan1.mnet-online.de [192.168.6.70]) by mail.m-online.net (Postfix) with ESMTP id 3zjSZF6Sw8z1sQwp; Fri, 16 Feb 2018 10:31:17 +0100 (CET) X-Virus-Scanned: amavisd-new at mnet-online.de Received: from mail.mnet-online.de ([192.168.8.182]) by localhost (dynscan1.mail.m-online.net [192.168.6.70]) (amavisd-new, port 10024) with ESMTP id YSwkET6KRqD2; Fri, 16 Feb 2018 10:31:15 +0100 (CET) X-Auth-Info: d+/qQyr+gHPgfDlcFagUOnI048yO4Hxsl6qiSQ3yLaM= Received: from deneb.denx.de (p578adb1c.dip0.t-ipconnect.de [87.138.219.28]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.mnet-online.de (Postfix) with ESMTPSA; Fri, 16 Feb 2018 10:31:15 +0100 (CET) Subject: Re: [PATCH 2/2] centralize multistrap configuration generation To: Alexander Smirnov , Claudius Heine , isar-users@googlegroups.com, Baurzhan Ismagulov References: <20180214131509.16361-1-claudius.heine.ext@siemens.com> <20180214131509.16361-3-claudius.heine.ext@siemens.com> <20180215114214.GC5374@yssyq.radix50.net> <1619976eb38.27ac.034a6b0541ed39b7fb4e17f4ac219eaa@ilbers.de> <22e22642-dc3d-4a08-19ac-69ac37423d86@ilbers.de> From: Claudius Heine Openpgp: id=6FF2E59F00C6BC2831D864C11173CB199808B153; url=http://pool.sks-keyservers.net/pks/lookup?op=get&search=0x1173CB199808B153 Message-ID: Date: Fri, 16 Feb 2018 10:31:11 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.5.2 MIME-Version: 1.0 In-Reply-To: <22e22642-dc3d-4a08-19ac-69ac37423d86@ilbers.de> Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="zNetEK2LWUKp90Z8iIAKUPU20e2aSLKmk" X-TUID: pBdL5zZ2rG/j This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --zNetEK2LWUKp90Z8iIAKUPU20e2aSLKmk Content-Type: multipart/mixed; boundary="T0WlsrqS3aY9lyj3lvc17t4Y0m0X30U2R"; protected-headers="v1" From: Claudius Heine To: Alexander Smirnov , Claudius Heine , isar-users@googlegroups.com, Baurzhan Ismagulov Message-ID: Subject: Re: [PATCH 2/2] centralize multistrap configuration generation References: <20180214131509.16361-1-claudius.heine.ext@siemens.com> <20180214131509.16361-3-claudius.heine.ext@siemens.com> <20180215114214.GC5374@yssyq.radix50.net> <1619976eb38.27ac.034a6b0541ed39b7fb4e17f4ac219eaa@ilbers.de> <22e22642-dc3d-4a08-19ac-69ac37423d86@ilbers.de> In-Reply-To: <22e22642-dc3d-4a08-19ac-69ac37423d86@ilbers.de> --T0WlsrqS3aY9lyj3lvc17t4Y0m0X30U2R Content-Type: text/plain; charset=utf-8 Content-Language: en-MW Content-Transfer-Encoding: quoted-printable On 02/15/2018 02:39 PM, Alexander Smirnov wrote: >=20 >=20 > On 02/15/2018 04:20 PM, Claudius Heine wrote: >> On 02/15/2018 01:37 PM, Alexander Smirnov wrote: >>> >>> Claudius Heine 15 =D1=84=D0=B5=D0=B2= =D1=80=D0=B0=D0=BB=D1=8F 2018 =D0=B3. >>> 15:08:50 =D0=BD=D0=B0=D0=BF=D0=B8=D1=81=D0=B0=D0=BB: >>> >>>> Hi Baurzhan, >>>> >>>> On 02/15/2018 12:42 PM, Baurzhan Ismagulov wrote: >>>>> On Wed, Feb 14, 2018 at 02:15:09PM +0100, >>>>> claudius.heine.ext@siemens.com wrote: >>>>>> From: Claudius Heine >>>>>> >>>>>> Previous there was redundant code between the buildchroot and >>>>>> isar-image-base recipe. >>>>>> >>>>>> This patch moves the generation of the multistrap configuration to= a >>>>>> central python script that can be parametrized by both recipes. >>>>>> >>>>>> Signed-off-by: Claudius Heine >>>>>> --- >>>>>> =C2=A0 .../recipes-core/images/files/multistrap.conf.in=C2=A0=C2=A0= | 38 ---------- >>>>>> =C2=A0 meta/lib/isar/multistrap.py=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 | 80 >>>>>> ++++++++++++++++++++++ >>>>>> =C2=A0 .../buildchroot/files/multistrap.conf.in=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 | 37 ---------- >>>>> >>>>> Could you please post two final multistrap.confs you'd like to get >>>>> with your >>>>> changes? >>>> >>>> 'like to get'? I don't want to get any other multistrap.conf than >>>> before. This is not a change of the multistrap.conf files, but just = how >>>> they are generated. The python configparser created ini-file format >>>> only >>>> puts spaces around the '=3D' characters, but the resulting deseriali= zed >>>> objects is the same. >>> >>> The difference that I see IMHO: now you need to feed multistrap by >>> community-standardized input, you propose to add custom tool that >>> eats proprietary input. >> >> I don't get this point. What do you mean by that? >=20 > You've introduced new language, like: >=20 > +MULTISTRAP_ISAR_SOURCE =3D "copy:///${DEPLOY_DIR_APT}/${DISTRO}" > +MULTISTRAP_ISAR_INSTALL =3D "${IMAGE_INSTALL}" > +MULTISTRAP_ISAR_SUITE =3D "${DEBDISTRONAME}" >=20 > So to have working multistrap config I *have* to write pre-config in > this new language, what it's the simplification here, one set of option= s > is replaced by another... ??? Thats not a 'new language' that just bitbake variables. I think I don't get what you are saying. Thats the same stuff that previous to this change happend in this big wall of text sed command. IMO that this is much more to the point and therefore easier to handle than this big search and replace list. >> >>> >>>> >>>> Ok, that is not 100% true, I changed the name of the isar repository= >>>> section. Now its called 'isar' in the image (previously 'Isar') and >>>> buildchroot (previously 'isar-apt') multiconfig. But that is it. It = has >>>> no different effect outside of multistrap. >>>> >>>>> I'd like to understand the reasons for moving from templates to a >>>>> magic >>>>> generator. >>>>> >>>>> In my experience, magic is usually poorly documented (if at all) >>>>> and very >>>>> ununderstandable for users. Simple change of a line requires >>>>> locating the >>>>> generator, understanding what it does (they tend to be hairy with >>>>> tons of >>>>> conditions) and patching the upstream [Isar] code. Templates have >>>>> their >>>>> disadvantages, but are straightforward for users. In general, I'd >>>>> like to keep >>>>> the current UX and stick with templates, which is possible most of >>>>> the time. >>>> >>>> You prefer duplicated code? Two slightly different templates for two= >>>> buildchroot and image recipe? >>>> >>> >>> AFAIK the key idea is to have the only single multistrap that works >>> with upstream repos. Other multistraps fetch packages from local >>> cache. So after this is done, the problem with duplicacations in >>> configs will go. >>> >>>> Also this patch is only an intermediate step of the way for multi-re= po >>>> support. IMO it has value on its own, since it centralized the >>>> generation of the multistrap configuration. >>>> The current kind of 'template' does not support creating repository >>>> entries on the fly. >>> >>> What is the use case to have them "on the fly"? >> >> See: Also this patch is only an intermediate step of the way for >> multi-repo support. >> >> Use-case for multi-repo: docker-ce repository available. >=20 > Sorry, don't know what is docker-ce. Could you please describe whole > repositories layout, just to understand the topic? I have the need to install docker community edition from the developers repository in my image. Currently I do it by downloading the 'deb' file from the repo manually and installing it, here I have to take care of updating the version to the latest one. What would be much better if I can just add this repository to the multistrap process and install the most current version like that. For that I need the ability to add additional repositories to the multistrap configuration from other layers= =2E I don't understand what you mean by 'whole repositories layout'. You mean like a tree, but from what repository? >=20 >> >>> >>> Alex >>> >>> It might be possible to extent the multiconfig with >>>> shell scripting, but I prefer using python for such advanced logic. >>> >>> I don't care about python or bash > >> At least Baurzhan seems to care, because he doesn't like debootstrap >> because of it being in shell. >=20 > Aghr, sent the mail on interrupted line, sorry. :-( >=20 > I meant I don't care whether python or bash is used, my concern what is= > the input for they. If you prefer to parse multistrap config with pytho= n > instead of bash, I could understand this. I don't understand you again. I don't parse the multistrap config. I just generate one. > But here you want to introduce another language for doing the same thin= gs. Again, what other language? I don't remember writing or using any language parse in this patch. > So at least for me, it would be much easier to take some template for > multistrap config (which syntax is well known and available in man page= ) > and create the custom layout that I need. If you prefer the template approach, then we might need to introduce a new templating language, because only search and replace via sed doesn't cut it for the multi-repo feature. Since I know jinja [1] we could do it with this. But it still might result in something more hacky. [1] http://jinja.pocoo.org/ Also since Jan, already decreed the switch to debootstrap I will now continue working on a patchset for this. Cheers, Claudius >=20 > Alex >=20 >> >> Cheers, >> Claudius >> >>> >>>> >>>> regards, >>>> Claudius >>>> >>>> --=C2=A0 >>>> DENX Software Engineering GmbH,=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Managi= ng Director: Wolfgang Denk >>>> HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, German= y >>>> Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.d= e >>>> >>>> --=C2=A0 >>>> You received this message because you are subscribed to the Google >>>> Groups "isar-users" group. >>>> To unsubscribe from this group and stop receiving emails from it, >>>> send an email to isar-users+unsubscribe@googlegroups.com. >>>> To post to this group, send email to isar-users@googlegroups.com. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/isar-users/d58343b9-c284-81e7-f743= -2672c1f5f8ef%40siemens.com. >>>> >>>> For more options, visit https://groups.google.com/d/optout. >>> >>> >>> >> >=20 --=20 DENX Software Engineering GmbH, Managing Director: Wolfgang Denk HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de PGP key: 6FF2 E59F 00C6 BC28 31D8 64C1 1173 CB19 9808 B153 Keyserver: hkp://pool.sks-keyservers.net --T0WlsrqS3aY9lyj3lvc17t4Y0m0X30U2R-- --zNetEK2LWUKp90Z8iIAKUPU20e2aSLKmk Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEEb/LlnwDGvCgx2GTBEXPLGZgIsVMFAlqGpN8ACgkQEXPLGZgI sVMUtw/+PEGEpgQXX/3sv7ye1vspfiCGcCMXcvibx/PgnH2WSo1t7hjuPTZ9Vqr5 WEuWnpDne2e4AgDKF3k2aarsRuSo2TTMV34/uym/eEq2rLzgo5pyxhnw4REM9qz0 N/v7lS9H50yqygvwhRxTDBPiygmhXmX3Jyy2U4h+FXdTMioDOrOzVLejQW4Pkady p6fFJHuinjkppLx9525AzGhUaVjbVXOfwCbjxJpcjmY2jOz5IB9S9eXU3VvKI5vJ 6dTFeKnf3M2ZeBbzmiGzfzMg70UmPPcolB0ubfFjb0CGErozp3jm5ZV/o+Y3dNY9 epY9W5KB0KDK5yxoSGE0zZNgxz64e0X9bN0gCgdXIsME7jcSfggfvEqNz5JW1cin pmq35X+SiLfX+Bz2outQhEGGSR07WhKMpbP4mxIWKh25MihWGzjnBwWszbS4k2jI qqBlXFRgc/S3kR6L2sgTpPZ4TfwdAuW9AzTC3VXBx6+nN+6W0zbsxnK7+g9iEV2f s81yxYoJE1lVORcLk6IqRn0Cy6KhTXm1cL9drS7Nte57WSQoZBBsIV2mlIItfPGD 31Xv5d0ZFf6v+M184pJEZrEwL5yeZTQIVstwDZx3qL0lyNrncaEi87GTFlwysGey TYGRD9weEOblQgFxrDgFf7OYW1K6S+pEfVU9Lsyq7vHfWzmJ9lo= =YVcX -----END PGP SIGNATURE----- --zNetEK2LWUKp90Z8iIAKUPU20e2aSLKmk--