From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6467463440282681344 X-Received: by 10.46.84.69 with SMTP id y5mr446396ljd.37.1506340682042; Mon, 25 Sep 2017 04:58:02 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 10.28.174.129 with SMTP id x123ls1177607wme.3.gmail; Mon, 25 Sep 2017 04:58:01 -0700 (PDT) X-Google-Smtp-Source: AOwi7QCVcO4hdJzEKeNrFe+BZX0wPJmtIJdDMSktpTzJhr2F54kamhfutOmrki9d5oOKUQr2XIpG X-Received: by 10.28.9.12 with SMTP id 12mr18907wmj.7.1506340681825; Mon, 25 Sep 2017 04:58:01 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1506340681; cv=none; d=google.com; s=arc-20160816; b=cXlh00+AdfOsdWRz71D2lcbqUlD7J1I3T7f2dk3zJofk/RalA6G6TaIC6o7440ZMp4 Yuy3dwgoYwh+1+HClPmiMEdUB/CQ3nqwz+2VTecagBV6GkNz+DpNXBSiB/OUi9aLiIOR TXUoosg5D1GQ4DnKxcCYe2aTJJBdfEMD+O2pynRJeaASS7xr9+9bZOfb/Ao1kTI4s+w5 Dr1ul7rZFAKI9luUuGrM0BgpbxO8cGWdkzY5i6WIYK1ErGiQRR7n//sEpVxAAzcRftBS pzUqAw89TQ9WRQhzv/lDINdYlrLtnKK6uXz6c2q2tW/LTES32XCyazM+ZCM9O5vpuK9C 0rFg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:content-language:in-reply-to:mime-version :user-agent:date:message-id:from:references:to:subject :arc-authentication-results; bh=vf+xSJh+pntYJxEWHdpQ8NVtqxSCFhDDJePXEFrtmf4=; b=bsCHnbS6t3zNLtGPvuyOL8+dwcholvOuhd+TPYeNbfa0o2s4LKqi/rUmMdlWFDmuGd iNPNXkwG881T7t6UXNsYCSNH8JK82pDtCiv/MXd7g4lmp5oA+6n/3uUPYrBeXj+B5deF kg/u0F060nKBaG1hHTxY6fmv4BdOBF6ziLHY+dOd0gJN9HlX38vscIf2rKwt7KbibVVR vzGKomC6kDC3NioXPt7S6n2vCOcsxRiMXvu4+Bj6Q9z7l3sc4ijUEIR/5QQ2uv2TAEYs teWPM4Tut+JWRvvpSX6ruAwlXiq6lrBGb+X4jVBSju80asw+ztiAuac1vCd7AYlNficd bjRw== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: best guess record for domain of asmirnov@ilbers.de designates 85.214.62.211 as permitted sender) smtp.mailfrom=asmirnov@ilbers.de Return-Path: Received: from aqmola.ilbers.de (aqmola.ilbers.de. [85.214.62.211]) by gmr-mx.google.com with ESMTPS id 200si699769wmj.0.2017.09.25.04.58.01 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 25 Sep 2017 04:58:01 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of asmirnov@ilbers.de designates 85.214.62.211 as permitted sender) client-ip=85.214.62.211; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: best guess record for domain of asmirnov@ilbers.de designates 85.214.62.211 as permitted sender) smtp.mailfrom=asmirnov@ilbers.de Received: from [10.0.2.15] ([188.227.110.165]) (authenticated bits=0) by aqmola.ilbers.de (8.14.4/8.14.4/Debian-4+deb7u1) with ESMTP id v8PBvwEw002388 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES128-SHA bits=128 verify=NOT); Mon, 25 Sep 2017 13:58:00 +0200 Subject: Re: [PATCH v2 2/4] meta-isar-bin: Generate cache repos To: Claudius Heine , isar-users@googlegroups.com References: <20170919122052.28688-1-asmirnov@ilbers.de> <20170919122052.28688-3-asmirnov@ilbers.de> <20170920101159.5b086d0c@md1em3qc> <69b055e2-e711-8a94-3e1b-0f066566d7d1@ilbers.de> <20170921085535.GA27874@iiotirae> <20170922105638.GB4240@yssyq.radix50.net> From: Alexander Smirnov Message-ID: <3464c875-4c7b-b953-c6fc-ed52088a3744@ilbers.de> Date: Mon, 25 Sep 2017 14:57:53 +0300 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.2.1 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-TUID: 3BKrqNmnbp1J Hi, On 09/25/2017 01:49 PM, Claudius Heine wrote: > Hi Baurzhan, > > On 09/22/2017 12:56 PM, Baurzhan Ismagulov wrote: >> Hello Andreas and Claudius, >> >> On Thu, Sep 21, 2017 at 10:55:37AM +0200, Andreas Reichel wrote: >>> This looks like a misdesign, because meta-layers should not be populated >>> by the build process and furthermore, should contain recipes and not a >>> cache itself. It would be like the sstate-cache lying inside meta-oe... >>> >>> Why not define a variable like 'DEB_CACHE_DIR' or something alike and >>> use this directory. Then you also have more flexibility and can handle >>> multiple different caches, i.e. the caches become selectable and >>> independent of the layer itself... >>> >>> Furthermore, you could set the DEB_CACHE_DIR variable in the local.conf >>> snippents for multiconf and then you can have separate caches for >>> different target architectures and don't mix everything up in one pool. >>> >>> This way you can easily drop a pool by architecture without sorting >>> files out. >> >> I share your concerns and don't like that myself. That said, it's a demo >> implementation we ended up considering the following project >> requirements: >> >> R1. Product's binary repo is persistent across builds on the same host >> and >> across the hosts of different developers. > > Why does the binary repo needs to be be 'persistent' across different > hosts? In general Isar designed to work with binary repositories, that's the key feature. Nobody forces you to use this cache persistent across different hosts, you are able to build everything from sources at any time. But at the same time I want to have the possibility to cache some stable Isar part for my needs and do not rebuild it everytime. > > This repo is only about the packages that are created by the recipes and > those should already build reproducible packages so they already are > 'persistent'. So why do you need a cache that can be shared with other > hosts? This topic is not about reproducible builds. Each software product has sources, so it can be built from the scratch at anytime. But why I need to do this, when I have binary? Installing Debian, user usually prefers binary apt instead of building the system from sources. It's just more comfortable and takes much less time. Also having Isar in binary repo in general I don't need the Isar at all (including setup development host). I'm able to generate target filesystem by using multistrap only. > > Creating this sharable cache just raises the problem that some packages > aren't build all the time. So its possible that if they break at some > later point, it might happen unnoticed, because all devs and maybe even > the CI is using some binary cache. Not ideal. This problem doesn't relate to binary cache. If the software package contains a bug, it doesn't matter if the package is binary or it's built from sources every time. If after bug-fix the respective binary package wasn't updated - it's an issue of engineer. If you don't want to export the cache - you don't need to think about this problem. > > You have to think about what is the real point of this cache is. > > Normally a build cache is there to make succeeding builds faster. To > detect if some item in the cache needs to be deleted, because the source > has updated is not a trivial problem. To do that with multiple hosts > involved it becomes much more difficult. Actually it doesn't. The cache is built using reprepro, which provides possibility to query various information for specific packages. So, for example, it's quite easy to implement the following steps in anonymous function: - get binary package version - compare with the version in recipe - if it differs - force Isar to update the cache for this package > > If you put the cache into a version control system, you have to bind the > cache to the exact revisions of the project repositories. Just to have > some knowledge about how those packages are created. But those revisions > hashes might be unique, but they are not persistent. The can disappear > with a rebase, etc. And now we get into the situations where we have a > cache that references some non existing repo revisions and nobody knows > how the packages are build, but since they do exactly what is needed, > nobody touches the sources anymore, but instead start to modify the > cached packages directly if they want small changes. That is the > nightmare scenario for reproducible builds. > > The longer I think about this, the more bad scenarios I can think of. > And I am sure the reality might teach us all some more in time. > Please don't give the developers a repository with binary packages, they > really should not be exposed to this kind of seduction. I don't agree here, it's a matter of taste. Again, nobody forces you to export the cache. For me - I don't want to spend hours building everything from sources, when I know that there are stable binaries. > >> R2. Building isar-image-base should be kept simple. New users should >> not be >> overwhelmed with repo setup, external tools, or manual >> configuration (as >> much as possible). >> >> R3. Build directory must be removable at any time without affecting >> cloned or >> newly built binary cache. >> >> We've translated that to the following design points: >> >> D1. Apt repo is versioned, e.g. in git. >> >> D2. Config files for this feature are put under isar/... >> >> D3. Given D1 and D2, binary packages go under isar/... :( . >> >> The idea is that in a project, the cache is a separate repo at the >> same level >> as Isar: >> >> product (originally cloned from isar) >> product-bin >> >> So, for production use, a tool like kas should be used to clone both >> at right >> revisions. We didn't want to do that for the demo due to R2. Just like >> meta-isar is currently a template for real products, meta-isar-bin is an >> example that should be moved out of the product source repo for a real >> product. > > Since you mention the 'meta' and 'meta-isar' layers. I would be in favor > of moving everything from the 'meta' layer into the 'meta-isar' layer. > We don't really need two layers IMO. We could put examples into > 'meta-isar-examples'. > > Maybe we should discuss this separately. Yes, I'd suggest to discuss this out of this thread. > >> >> Technically, meta-isar-bin is a layer that contains configuration >> files that >> have to be put somewhere. As package installation via apt is a >> requirement >> (dpkg doesn't install package dependencies), meta-isar-bin should >> become the >> default and would thus be required in the current implementation. >> >> We had considered cloning meta-isar-bin into the build directory, but >> that >> would mean manual configuration for new users, and (possibly modified) >> configs >> could be deleted if the build directory is deleted -- an unpleasant >> surprise >> for users familiar with OE. >> >> If it's a separate directory, the config files should go there, since >> they are >> per-repo, and there could be several ones: >> >> product >> company-bin >> department-bin >> product-bin >> >> For a source distribution, having binary repos as layers sounds >> perverse. But >> given Isar's focus on binary packages, it's layering in the sense that >> more >> specific repos could override more general ones. >> >> In the future, we'd like to move more stuff into the core Isar (either >> moving >> files from meta-isar to meta, or merging meta into meta-isar and >> introducing >> meta-template). Maybe we could introduce kas in a simple way, which >> would solve >> this problem. >> >> So, ATM I'd suggest to document the steps regarding meta-isar-bin for >> creating >> real products. >> >> If anyone has an elegant solution for the issues above, I would be >> glad to hear >> it. I agree with a separate directory approach, but suggest to >> postpone it till >> we find a good way to introduce it to new users. > > I am still not convinced about the need to put binary repositories into > layers. And introducing just a simple binary package repository as a > package cache in the build directory does not need any special > configuration files or overwriting existing configuration AFAIK. So > introducing it to 'new users' would just work transparent. reprepro requires configuration file. > > Of course my solution would be a host local cache. But as I said, I > don't really see any reason for having some kind of project global cache > that is shared over multiple hosts. Ok you have a first first build, but > at the same time you don't even test if those packages can still be > created from the sources/recipes. So thats not really an argument for me. > But Debian users prefers to use ftp://ftp.debian.org/debian/ instead of building from tarballs. -- With best regards, Alexander Smirnov