From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6740510031426355200 X-Received: by 2002:a19:2207:: with SMTP id i7mr1808476lfi.185.1569495745542; Thu, 26 Sep 2019 04:02:25 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a2e:8803:: with SMTP id x3ls216299ljh.10.gmail; Thu, 26 Sep 2019 04:02:24 -0700 (PDT) X-Google-Smtp-Source: APXvYqzL4BBUO2+tR7SdNHcXGtOBLbOMUrrCSa6ic9RG/42tzvaFCF3+ml/62HxErRv+v9+VrDWl X-Received: by 2002:a2e:8054:: with SMTP id p20mr2206759ljg.36.1569495744790; Thu, 26 Sep 2019 04:02:24 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1569495744; cv=none; d=google.com; s=arc-20160816; b=gbD/O8Tf2eGvWNuS2wq8NtO+lv4oLl0zd1VuD0NEbmIlQ+rUXYSu+famjpQoAvIBbJ VX/G9tqatiiomTqjWbYTX9jg72iL1WUkL0Jax+zt9RmEYCeusvMOxOf+l0zg427uRXTb B0RGqAM3jL3d9vhYv2Mxo+Iys8u7eYiE5FtdOnwmAr3HKrdreNhiPMi/u6izRQYWS+vX pDn571CaMsbIEBNQWn6/lC+iDQQ28EnHVxDm02G+wcylqOzZBEZjAVH+kCqN5vTi+4vP rHZ70GHle2H4VbkHjx1USXH0Zu+l7Y02znXG9OsXsJ42S90BjrJYs5JQPAyMPc4707Ha 6EEw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=user-agent:in-reply-to:content-disposition:mime-version:references :message-id:subject:cc:to:date:from:dkim-signature; bh=7O3zrN0xFKkAumY2MLM/XbCKyRPtgnmopKBEMfiunJA=; b=GXzxvpVbE0kwrDr/X/JZxSFKC856N75RhKs/U3psklvQbglTwFR25ZZuT6tVf2iIT3 VlMK7r6C7hW6Bwji9er/iOeJcf7xriHH8w3C845eAG3Cmx42Kwwrrhl9ISrFtTjlVwN0 HOZfTyH9mj9UnuSSphbmTT0Jziqc31fbtlLJcno4lEhbLHMTY9k+fVDiJwLCjiQNRYy3 6FIT4BjyrGy3je2qUFVgI7caAb7Qn+XtA163kCGGj529UYuk8yqENiPVn2pvTpq/uqgc X7slJ0nuEazPuncK1R6ULRfQ89MB9DDVMSdkT2AIurx8QJDmLg/xMXRyGIVwDEj9IGzg Gwmg== ARC-Authentication-Results: i=1; gmr-mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=AHgz5nO+; spf=pass (google.com: domain of vijaikumar.kanagarajan@gmail.com designates 2a00:1450:4864:20::344 as permitted sender) smtp.mailfrom=vijaikumar.kanagarajan@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from mail-wm1-x344.google.com (mail-wm1-x344.google.com. [2a00:1450:4864:20::344]) by gmr-mx.google.com with ESMTPS id c25si128714lji.2.2019.09.26.04.02.24 for (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Thu, 26 Sep 2019 04:02:24 -0700 (PDT) Received-SPF: pass (google.com: domain of vijaikumar.kanagarajan@gmail.com designates 2a00:1450:4864:20::344 as permitted sender) client-ip=2a00:1450:4864:20::344; Authentication-Results: gmr-mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=AHgz5nO+; spf=pass (google.com: domain of vijaikumar.kanagarajan@gmail.com designates 2a00:1450:4864:20::344 as permitted sender) smtp.mailfrom=vijaikumar.kanagarajan@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: by mail-wm1-x344.google.com with SMTP id m18so2115841wmc.1 for ; Thu, 26 Sep 2019 04:02:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:date:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to:user-agent; bh=7O3zrN0xFKkAumY2MLM/XbCKyRPtgnmopKBEMfiunJA=; b=AHgz5nO+sZ3rmxOVhWQOEPShtnWgcAnHxfkRAc4rJs73c1n31HKQkiyOQl7vME9NLZ xKhTYqRk/S6DOQCNHlDoYSqtWGZdR0Z7F122OAxoS480yDgCXAWQO7mTSeTZjS9dPvYO P3iprhVWJBAyptkoNzgNErPXBPIHPXHv9a4JcUZZKU8+esHZ2Dy2Hf5q6VHqJSNrCxSk 1ItTjvDoIdLZqB4DzWtn6Xrk8hb14y7tlR+jjY7qbLDOH0PZhDrw+ZtwsAaDht+ah1Z/ kwNiRs0LZptNEygDGkqRijXpbjwro9q9FDoZc++WWnCjCSt9aEc+Y1LBabMqQqXdTicj DPgg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:date:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to:user-agent; bh=7O3zrN0xFKkAumY2MLM/XbCKyRPtgnmopKBEMfiunJA=; b=uc4Y+mMMQMPKGgtyfopJE6I6Qwu1Imyqps7l01xZ3062hs8uuzbtAIwq/CJVsXKLdE oct/d6w//cO6e8oQLNBB7Y3XtrBLSdEEZQXlNBk6qUlgcFNcRUeOh6seXvjp6pHUu3Wc TGOaW14iXSQ/qr+4rmcRlpdYPyDO+of38F/ZcGfeAeYbb9NTZ211VcoBc6Qgogbivmrb U1TOyBfvc6oWpXLUDhwCrhpoTuOUOv2Uyc/AjMZhVskc1rejXE8aTgNWAX8g0z37KTdu 4dCa0Nx1NpAby7uSjO+KZeSOHrhAAgL5Fz2eUTTw/23HckRcbsdFd+iYiE1eHVZt8Hrs HEEA== X-Gm-Message-State: APjAAAWbOsQ0ed7IAWe2wbbyxdKxbNQV6ZOaf5ywQJStX2ayDa6DV+VO Ys5LjZlI86emJ2Urkskf1mk= X-Received: by 2002:a7b:cbcf:: with SMTP id n15mr2449573wmi.106.1569495744038; Thu, 26 Sep 2019 04:02:24 -0700 (PDT) Return-Path: Received: from lightning (nat-sch.mentorg.com. [139.181.36.34]) by smtp.gmail.com with ESMTPSA id n22sm2077806wmk.19.2019.09.26.04.02.21 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Thu, 26 Sep 2019 04:02:23 -0700 (PDT) From: Vijai Kumar K X-Google-Original-From: Vijai Kumar K Date: Thu, 26 Sep 2019 16:32:16 +0530 To: Henning Schild Cc: Vijai Kumar K , isar-users@googlegroups.com, claudius.heine.ext@siemens.com, jan.kiszka@siemens.com, ibr@radix50.net Subject: Re: Discussion: Base-apt features Message-ID: <20190926110216.GA18169@lightning> References: <20190925074122.GA12490@lightning> <20190926114026.182600d1@md1za8fc.ad001.siemens.net> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20190926114026.182600d1@md1za8fc.ad001.siemens.net> User-Agent: Mutt/1.9.4 (2018-02-28) X-TUID: qHVLKVwzACFd On Thu, Sep 26, 2019 at 11:40:26AM +0200, Henning Schild wrote: > Hi Vijai Kumar K, > > that is a lot of points at once, we should probably go step by step > since they are all more complicated than just one bullet-point on a > TODO-list. > > Let me start off with a pragmatic "solution" that is easy to implement > and likely to solve a lot or problems at once. > > Instead of relying on Isar to get you a partial debian mirror, you you > just mirror upstream and freeze it for a product. You would end up with > a consistent mirror that will contain all you might ever need, all > signed with sources etc. A little big ... yes, but easy. > If you decide to trust in the availability of > https://snapshot.debian.org/ that might also be a path to address some > problems you might be seeing today. We could do that but would be an overhead considering we have only close to 200 packages which we need. Also there might be cases where in we patch a package locally for a CVE(like openssl) and include it in repo. Maintaining changes like those becomes complicated. > > But yes, we probably still want a partial mirroring mechanism inside > Isar. > > Am Wed, 25 Sep 2019 13:11:22 +0530 > schrieb Vijai Kumar K : > > > Hi All, > > > > Starting this thread to discuss the base-apt features and limitations. > > > > Here I am listing down some of the issues/features and possibly the > > need for them. > > > > > > 1. Support for adding source packages. > > > > Currently we have support only for binaries. The corresponding source > > files could also be added. > > It goes even further. You probably want the sources for every binary > you install. > + all the other binaries coming from those sources > + all build deps, again sources and all binaries > + recursion of the former > > We would have to see how big that gets ;). Agreed. The repo size might get compounded. How about we keep it as an optional feature so in case some one needs it, they could bring in all the sources and their dependencies. > > > 2. Support for using password protected keys. > > > > It is a good practice to have the gpg key protected to have an > > additional level of security. Right now ISAR does not have provisions > > to use password protected keys. > > Let us put this aside for now, is is technically just something small > on top of signing. Agreed. We have bigger things to do and this could wait till we piece in the rest of the things. > > > 3. Support for specifying the signing key. > > > > Right now, the signing mechanism uses the default gpg key of the > > system. This is problematic in many ways. Especially for CI. In the > > current implementation, eventhough we specify the key, we are not > > really using it. > > Did you not have a patch for that "=yes" bug? Anyways signing is not > trivial and we need revisit that anyways. Yes. It was proposed by Amy Fong. However I was into the idea of using KeyID instead of key file. Later decided instead of doing a incremental change to fix it, will wait for to freeze the new design if any. > > There are package-level signatures and repo signatures. For the partial > debian-mirror we just need to take care of the repo-level. > Yes. We could just have the metadata signed. > All customizations are currently coming out of isar-apt, which is > totally unsigned. > > > 4. Support for adding packages only to base-apt. > > > > Sometimes, we might need a package to be present in base-apt but not > > in the target yet. Things like dev & dbg packages. It would be good > > if we have something like BASE_APT_INSTALL which contains the list > > which would be populated only in base-apt. > > I would suggest to have another image to do that. You can do that > today already. > > bulky-image_0.1.bb: > require my-image > BASE_APT_INSTALL = "strace gdb" > IMAGE_INSTALL += BASE_APT_INSTALL > > Now you build that and ship my-image. I assume that my-image will be > able to apt-get from base-apt later. Thanks for the pointer. There is a catch though. Not all images can be installed simultaneously to the image. Some may conflict. For example I would not be able to install libssl1.0-dev and libssl-dev together. But one might need them to co-exist in the repo like how they do now in upstream. So what I thought was that BASE_APT_INSTALL shall be processed by a task which calls apt with --download-only to download these and include it in the local partial repo. > > If that pattern works we can probably add an example, support-code and > test cases to make that the "supported pattern". > > > 5. Refactoring code to consolidate reprepro calls. > > > > Right now, reprepro calls are spread across the build system. Its > > dependencies are spread across too(Handling envs like GNUPGHOME, > > distributions file etc). My first thought is to have a seperate > > module implemented to handle these calls. > > I think that is not too related. The calls for isar-apt are spread and > will probably stay that way, but they should currently not deal with > anything around signing. > > For base-apt i envision a post-process task (like implemented today) as > a first implementation. If we decide to build it up on the fly, those > reprepro calls, downloads, signing will shift forward in the buildchain > and will spread like we have it for isar-apt. Even if isar-apt and base-apt have their own differences, the tools used to generate them is same. There might be some differences in options. Bringing those tools(right now reprepro) under one roof would make the code easier to maintain. Otherwise things like GNUPGHOME patch[1] would show up and lead to more duplication and difficult maintainence. [1] https://github.com/ilbers/isar/commit/b8a2c6020ccb1b85cc4d0f6b75bc3ac090c24dc7 Thanks, Vijai Kumar K > > Next steps: > - try the idea in 4. > - implement 1 with revisiting [1] > > [1] > https://groups.google.com/forum/#!searchin/isar-users/base-apt$20caching$20improvements|sort:date/isar-users/_dqKYWUtTa0/velGfVg0BgAJ > > Henning > > > Like how > > https://github.com/openembedded/openembedded-core/blob/master/meta/lib/oe/gpg_sign.py > > is used for all signing purpose. > > > > Please add more if you have some features/limitations which needs to > > be addressed. > > > > Thanks, > > Vijai Kumar K >