From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6449958519840964608 X-Received: by 10.223.202.15 with SMTP id o15mr1417681wrh.5.1510676545081; Tue, 14 Nov 2017 08:22:25 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 10.28.227.197 with SMTP id a188ls1589634wmh.7.gmail; Tue, 14 Nov 2017 08:22:24 -0800 (PST) X-Google-Smtp-Source: AGs4zMavzqwHgOCGdHXLFPALtCQ5sf/1lOmekeAB9WL3CFJQtXaUGN3AoTYgDaXd5MWk9QSw6XLA X-Received: by 10.28.127.10 with SMTP id a10mr1546798wmd.0.1510676544708; Tue, 14 Nov 2017 08:22:24 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1510676544; cv=none; d=google.com; s=arc-20160816; b=AKCEuwmcR8JIHnW2RUd4TaXU9VR6Z0Goee6e7fb/xm1gq0tSF23Dw6nOvqJ7EoU9wE xMmeUMQ/cYHcIGAHMFrqDZtAxf5NM+xmFi+vytGudjGqufYC87jvsa6dIiblHugq4q/c 4Z9PEA/30VbYPj8A/a+BIQtDW3X58R9fedyCJVEzyvW1QDjZasm7XGT3Mu0JE0vHMW5a V/EGyedwccsOYqOMwiPLd4fzzMnO0aNeMCncSEDz8NJmJ03YDq9YxgNgIYWbkJQ7jB+n V/Qomx67mwnJzT9zfGimKoOQ3UesRrs/tPk2dtAsPr0FvYKmgzKg8TwEgmMl/jE5zw7i HWiw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:content-language:in-reply-to:mime-version :user-agent:date:message-id:from:cc:references:to:subject :arc-authentication-results; bh=nbeTi+UvD/WTE1Gsli3FzrqSvO0bp6IlPP3K6ugi14Y=; b=iOhempVgDNp/Ba7yB2ixu3E5OA1KdAGmWvl0RehQPBSjfwOilTysfF8nqoYUbJgdjO bQvIWvUTSPdSCucyhCndPnC3ofv7b2sWiC+cEKglT+agg02BkqHlwZED2JwnvnHjR0Pw 5DC7SuZf+RezDh9u3s4kP4vqNMAwa09220xbsuFit33+baablJ07ShZzZ4CdwVuccyyK 4RmUEnGaM8lUO3aRw9yJTJd4MMLb+cO5WeutRdrv+hJ/PASheJwOeb5SjUsNAgPmko3b QGi2xqyldd4HvP+s17nbO62TpMx9Sv7lPGD9D4w86DcH47+TH3ILpZfbRrhp8HlyRPV0 BTYA== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of claudius.heine.ext@siemens.com designates 192.35.17.2 as permitted sender) smtp.mailfrom=claudius.heine.ext@siemens.com Return-Path: Received: from thoth.sbs.de (thoth.sbs.de. [192.35.17.2]) by gmr-mx.google.com with ESMTPS id f8si1462173wrf.0.2017.11.14.08.22.24 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 14 Nov 2017 08:22:24 -0800 (PST) Received-SPF: pass (google.com: domain of claudius.heine.ext@siemens.com designates 192.35.17.2 as permitted sender) client-ip=192.35.17.2; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of claudius.heine.ext@siemens.com designates 192.35.17.2 as permitted sender) smtp.mailfrom=claudius.heine.ext@siemens.com Received: from mail2.sbs.de (mail2.sbs.de [192.129.41.66]) by thoth.sbs.de (8.15.2/8.15.2) with ESMTPS id vAEGMOmC021103 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK) for ; Tue, 14 Nov 2017 17:22:24 +0100 Received: from [139.25.68.223] (linux-ses-ext02.ppmd.siemens.net [139.25.68.223]) by mail2.sbs.de (8.15.2/8.15.2) with ESMTP id vAEGMNlb026254; Tue, 14 Nov 2017 17:22:24 +0100 Subject: Re: Reproducibility of builds To: "[ext] Christian Storm" References: <20171114160427.xzkzqwc322ih5265@MD1KR9XC.ww002.siemens.net> Cc: isar-users@googlegroups.com From: Claudius Heine Message-ID: <6196aaa3-763a-7d5a-9971-d864d0943970@siemens.com> Date: Tue, 14 Nov 2017 17:22:23 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.4.0 MIME-Version: 1.0 In-Reply-To: <20171114160427.xzkzqwc322ih5265@MD1KR9XC.ww002.siemens.net> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-TUID: GBWZVu0/J1b+ Hi Christian, On 11/14/2017 05:04 PM, [ext] Christian Storm wrote: > Hi, > > since I'm very interested in this feature, I'd like to resume this > discussion and to eventually come to an agreed upon proposal on how > to implement it. So, without further ado, here are my thoughts on > the subject: > > Regardless of the concrete technical implementation, I guess we can > agree on the need for a local cache/repository/store in which the Debian > binary packages plus their sources have to be stored since one may not > rely on the availability of those files online for eternity. > > These files in this cache/repository/store are the union of the Debian > binary packages installed in the resulting image plus their sources as > well as those installed in the buildchroot plus their sources. > The latter is required to be able to rebuild Debian packages built from > source with the same compiler version, libraries, -dev packages, etc. pp. > > Having the cache/repository/store at hand, there should be a mechanism > to prime Isar with it, i.e., Isar should only and exclusively use Debian > binary packages and sources from this cache/repository/store. > This is again, irrespective of the technical implementation, be it via > a repository cache or other means like git, a proxy server or whatsoever. > > Granted, if one changes, e.g, IMAGE_INSTALL_append, the build fails but > does so rightfully as the set of packages is modified, resulting in a > new version/epoch (=set of Debian packages plus their sources). So, > there should be a convenient "interface" provided by Isar to maintain > the cache/repository/store. For example, one may want to have different > versions/epochs that may correspond to particular versions (git sha) of > the Isar layer. Or one wants to later add a Debian package plus its > source (which is automatically fetched), resulting in a new > version/epoch etc. > > The remaining question is how to fill the cache/repository/store. In > order to have a consistent version/epoch (=set of Debian packages plus > their sources), there should not be duplicate packages in it, i.e., the > same Debian package but with different versions. > This could currently happen because there is a "window of vulnerability": > multistrap is run twice, once for isar-image-base.bb and once for > buildchroot.bb. In between those two runs, the Debian mirror used could > get updated, resulting in a different version of the Debian package > being installed in buildchroot than in the resulting image. > This is an inherent problem of relying on the Debian way of distributing > packages as one cannot a priori control what particular package versions > one gets: In contrast to, e.g., Yocto where the particular package > versions are specified in the recipes, this does not hold for Isar as > the particular package versions are defined by the Debian mirror used, > hence, one gets "injected" the particular package versions. > So, what's required to reduce the "window of vulnerability" and to have > a consistent cache/repository/store for a particular version/epoch is to > make a snapshot-type download of the required packages. For this, of > course, one needs to know the concrete set of packages. This list could > be delivered by a "package trace" Isar run since not only multistrap > does install packages but sprinkled apt-get install commands do as well. > Thereafter, knowing the list, the snapshot-type download can happen, > hopefully resulting in a consistent cache/repository/store. > > > So, what do you think? I agree with your formulation of the problem here. Simple tracing of installed packages will have the problem you described, that its possible that different versions of a package are installed into buildchroot and image. So this trace needs to be cleaned up and then based on that the whole process has to be started again to create a consistent package list between buildchroot and image. This doubles the build time in the trivial implementation. With my suggestion of using a caching proxy, this could be solved without any additional overhead. I do have other ideas to do this, but that would restructure most of isar. Cheers, Claudius -- DENX Software Engineering GmbH, Managing Director: Wolfgang Denk HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de