From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6467463440282681344 X-Received: by 10.25.16.224 with SMTP id 93mr453967lfq.8.1506336572876; Mon, 25 Sep 2017 03:49:32 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 10.46.93.193 with SMTP id v62ls2301684lje.7.gmail; Mon, 25 Sep 2017 03:49:32 -0700 (PDT) X-Google-Smtp-Source: AOwi7QCu/jwNHTorZiO9T1jINSc3MEVk6ALaotx4dDwImTQSPyPvQgT+fziDdF3J18aDka6o0VpH X-Received: by 10.25.38.140 with SMTP id m134mr474312lfm.2.1506336572488; Mon, 25 Sep 2017 03:49:32 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1506336572; cv=none; d=google.com; s=arc-20160816; b=qNjzeXFc2oYacHDBjIW2iLOtwn+KaNIfAZHYQJ624nz97XTEnFHKFuAmSSWAycIjUX olR/evmenwRDGtx7MEGQx0f62eaZngGWDehdFQDRx1NZpjEl31yxRQBTqlQ4Xg3zXaPC AhaNs9I1/W86l/b3WOtuvHMKpFrvEJkKGyz3oG54S/2+0DmEBGeyY0JA3ZUR8chQX5EZ vtcW1IuLiQFS6TQjRhhMwOyQjh2WNfaYkAXB+BRpEwadxK5yr/eopJtaZ1gXmnxat8dH etNqGxB4KwORE9yFC26fjEaQyDMubfaGPZAesbaXoWaMbMKB7k+xixYkIyZC6fMm4lV9 cJkA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:content-language:in-reply-to:mime-version :user-agent:date:message-id:from:references:to:subject :arc-authentication-results; bh=bHam9CHDk3JGxMa89W2EVlszu0T4MsNnppJzk5W/mxA=; b=ekzt7w6u/tJc/qNRKRI2iRzv+iyw41r82YAXMrH2ytHwwcgJmq3+dEYRdV9Z2Vqsej ZVOygByc0GRoi2cEJmdiTy6fyuLCPpONdhynOM+hx9udsrS46mb2ELMxMG+OEFbx0FmC WxLSMYSGIG0uon511r/wRa3JgztqQ3YO7Ppo8AkB7+Pk1Wv2Ns22kwq9DMAvT0+wBPT9 1cV2Hp6/bm1wrxfDpQHTlbH4kjC++j8+BChkL1+NOJESRReWHltl7P1ZibfEyqTGio4V 5ELOmld8YUdIkwJHG3REYezBS4r+Km4EGqXqX8Y15RarDTyUO5L+o7zdYLhHMzDUZrwk rUbQ== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=neutral (google.com: 194.138.37.40 is neither permitted nor denied by domain of claudius.heine.ext@siemens.com) smtp.mailfrom=claudius.heine.ext@siemens.com Return-Path: Received: from gecko.sbs.de (gecko.sbs.de. [194.138.37.40]) by gmr-mx.google.com with ESMTPS id j4si550502wmi.2.2017.09.25.03.49.32 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 25 Sep 2017 03:49:32 -0700 (PDT) Received-SPF: neutral (google.com: 194.138.37.40 is neither permitted nor denied by domain of claudius.heine.ext@siemens.com) client-ip=194.138.37.40; Authentication-Results: gmr-mx.google.com; spf=neutral (google.com: 194.138.37.40 is neither permitted nor denied by domain of claudius.heine.ext@siemens.com) smtp.mailfrom=claudius.heine.ext@siemens.com Received: from mail1.sbs.de (mail1.sbs.de [192.129.41.35]) by gecko.sbs.de (8.15.2/8.15.2) with ESMTPS id v8PAnWYZ023151 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK) for ; Mon, 25 Sep 2017 12:49:32 +0200 Received: from [139.25.68.223] (linux-ses-ext02.ppmd.siemens.net [139.25.68.223]) by mail1.sbs.de (8.15.2/8.15.2) with ESMTP id v8PAnVZ9005545 for ; Mon, 25 Sep 2017 12:49:32 +0200 Subject: Re: [PATCH v2 2/4] meta-isar-bin: Generate cache repos To: isar-users@googlegroups.com References: <20170919122052.28688-1-asmirnov@ilbers.de> <20170919122052.28688-3-asmirnov@ilbers.de> <20170920101159.5b086d0c@md1em3qc> <69b055e2-e711-8a94-3e1b-0f066566d7d1@ilbers.de> <20170921085535.GA27874@iiotirae> <20170922105638.GB4240@yssyq.radix50.net> From: Claudius Heine Message-ID: Date: Mon, 25 Sep 2017 12:49:31 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.3.0 MIME-Version: 1.0 In-Reply-To: <20170922105638.GB4240@yssyq.radix50.net> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-TUID: dyfh96Ln1f4Q Hi Baurzhan, On 09/22/2017 12:56 PM, Baurzhan Ismagulov wrote: > Hello Andreas and Claudius, > > On Thu, Sep 21, 2017 at 10:55:37AM +0200, Andreas Reichel wrote: >> This looks like a misdesign, because meta-layers should not be populated >> by the build process and furthermore, should contain recipes and not a >> cache itself. It would be like the sstate-cache lying inside meta-oe... >> >> Why not define a variable like 'DEB_CACHE_DIR' or something alike and >> use this directory. Then you also have more flexibility and can handle >> multiple different caches, i.e. the caches become selectable and >> independent of the layer itself... >> >> Furthermore, you could set the DEB_CACHE_DIR variable in the local.conf >> snippents for multiconf and then you can have separate caches for >> different target architectures and don't mix everything up in one pool. >> >> This way you can easily drop a pool by architecture without sorting >> files out. > > I share your concerns and don't like that myself. That said, it's a demo > implementation we ended up considering the following project requirements: > > R1. Product's binary repo is persistent across builds on the same host and > across the hosts of different developers. Why does the binary repo needs to be be 'persistent' across different hosts? This repo is only about the packages that are created by the recipes and those should already build reproducible packages so they already are 'persistent'. So why do you need a cache that can be shared with other hosts? Creating this sharable cache just raises the problem that some packages aren't build all the time. So its possible that if they break at some later point, it might happen unnoticed, because all devs and maybe even the CI is using some binary cache. Not ideal. You have to think about what is the real point of this cache is. Normally a build cache is there to make succeeding builds faster. To detect if some item in the cache needs to be deleted, because the source has updated is not a trivial problem. To do that with multiple hosts involved it becomes much more difficult. If you put the cache into a version control system, you have to bind the cache to the exact revisions of the project repositories. Just to have some knowledge about how those packages are created. But those revisions hashes might be unique, but they are not persistent. The can disappear with a rebase, etc. And now we get into the situations where we have a cache that references some non existing repo revisions and nobody knows how the packages are build, but since they do exactly what is needed, nobody touches the sources anymore, but instead start to modify the cached packages directly if they want small changes. That is the nightmare scenario for reproducible builds. The longer I think about this, the more bad scenarios I can think of. And I am sure the reality might teach us all some more in time. Please don't give the developers a repository with binary packages, they really should not be exposed to this kind of seduction. > R2. Building isar-image-base should be kept simple. New users should not be > overwhelmed with repo setup, external tools, or manual configuration (as > much as possible). > > R3. Build directory must be removable at any time without affecting cloned or > newly built binary cache. > > We've translated that to the following design points: > > D1. Apt repo is versioned, e.g. in git. > > D2. Config files for this feature are put under isar/... > > D3. Given D1 and D2, binary packages go under isar/... :( . > > The idea is that in a project, the cache is a separate repo at the same level > as Isar: > > product (originally cloned from isar) > product-bin > > So, for production use, a tool like kas should be used to clone both at right > revisions. We didn't want to do that for the demo due to R2. Just like > meta-isar is currently a template for real products, meta-isar-bin is an > example that should be moved out of the product source repo for a real product. Since you mention the 'meta' and 'meta-isar' layers. I would be in favor of moving everything from the 'meta' layer into the 'meta-isar' layer. We don't really need two layers IMO. We could put examples into 'meta-isar-examples'. Maybe we should discuss this separately. > > Technically, meta-isar-bin is a layer that contains configuration files that > have to be put somewhere. As package installation via apt is a requirement > (dpkg doesn't install package dependencies), meta-isar-bin should become the > default and would thus be required in the current implementation. > > We had considered cloning meta-isar-bin into the build directory, but that > would mean manual configuration for new users, and (possibly modified) configs > could be deleted if the build directory is deleted -- an unpleasant surprise > for users familiar with OE. > > If it's a separate directory, the config files should go there, since they are > per-repo, and there could be several ones: > > product > company-bin > department-bin > product-bin > > For a source distribution, having binary repos as layers sounds perverse. But > given Isar's focus on binary packages, it's layering in the sense that more > specific repos could override more general ones. > > In the future, we'd like to move more stuff into the core Isar (either moving > files from meta-isar to meta, or merging meta into meta-isar and introducing > meta-template). Maybe we could introduce kas in a simple way, which would solve > this problem. > > So, ATM I'd suggest to document the steps regarding meta-isar-bin for creating > real products. > > If anyone has an elegant solution for the issues above, I would be glad to hear > it. I agree with a separate directory approach, but suggest to postpone it till > we find a good way to introduce it to new users. I am still not convinced about the need to put binary repositories into layers. And introducing just a simple binary package repository as a package cache in the build directory does not need any special configuration files or overwriting existing configuration AFAIK. So introducing it to 'new users' would just work transparent. Of course my solution would be a host local cache. But as I said, I don't really see any reason for having some kind of project global cache that is shared over multiple hosts. Ok you have a first first build, but at the same time you don't even test if those packages can still be created from the sources/recipes. So thats not really an argument for me. Cheers, Claudius -- DENX Software Engineering GmbH, Managing Director: Wolfgang Denk HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de