From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6520131612833742848 X-Received: by 10.28.63.6 with SMTP id m6mr63852wma.3.1518086440742; Thu, 08 Feb 2018 02:40:40 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 10.223.154.77 with SMTP id z71ls1282199wrb.7.gmail; Thu, 08 Feb 2018 02:40:40 -0800 (PST) X-Google-Smtp-Source: AH8x22527MPz1kdXYiLOMl5BlZ7KGVNeorOzo8LSjStwcXGFGJkMlROG2LeIP73dR/yCiunfGR/A X-Received: by 10.223.133.198 with SMTP id 6mr25689wru.31.1518086440254; Thu, 08 Feb 2018 02:40:40 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1518086440; cv=none; d=google.com; s=arc-20160816; b=GTn1UwbYro4xKMkThUz5lPdEy/3jGo7EfnJTNXG5U/s0BidzOW3i4b1ibYQWcHTt2H 9QyqeAKmAWiAr73igQfa532hd9JeCyZlz5S47sNqtn6HUHxsrvvhJJOR/kT4fnpuluyT oJ2SzDGAeImJtGJDweXeDA3KDLoZrJ6gFTooqGI+/mSqgl73K1b7rrxm+NueOr4YBEXy 1tgPkEStqO8VzNBZFB/C9aweyW8+JBuuHGBDB/ykSVdRwRZzmhI0aT8cYXHrhUGl+oqh 8I1l9wuAwrkE4JA2+QX9blq/UPSQbgeUk/KaK51DrNqZ2lr/j0iBuag1HF58xSxMZkBD TG+w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:dkim-signature:arc-authentication-results; bh=Y1bUHljFXqXxZSvGEEFphNea7sA8rdzky7vjpM9xdgo=; b=mG34FcvdMz3+ac4Yn1sIK43rzKqSRdyBEIduGhq0KuTsAeswLVMEKctBSp9/G5biHJ 6Ek31/VGzVwp6YZ5p/K9frWNk1O4ujM+tHAE2SYso+AJu7kGARnVF/MA6wioMGyOKW6i Z+zVEPiGw1DBaC8uXBSv7Q5SxszRk48PLZE34cyjHIOvJrxhymlaUrvnf8YgZEz5gJx3 FMNLwLayQSzbj7LYULCKkC7QoHYsm99HegFHq2r/hJaj7RYVx+dHKsT/BlmgoIe6yu+D pLGqqW851Bdi7tbRF2OCGlEKRVStme6MxXSGh0T+2QhVIs7CDuuKI/nvZQpOj3YEo7Rv f9GQ== ARC-Authentication-Results: i=1; gmr-mx.google.com; dkim=pass header.i=@googlemail.com header.s=20161025 header.b=anqNr8/g; spf=pass (google.com: domain of benbrenson89@googlemail.com designates 2a00:1450:400c:c0c::231 as permitted sender) smtp.mailfrom=benbrenson89@googlemail.com; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=googlemail.com Return-Path: Received: from mail-wr0-x231.google.com (mail-wr0-x231.google.com. [2a00:1450:400c:c0c::231]) by gmr-mx.google.com with ESMTPS id v8si203205wrg.2.2018.02.08.02.40.40 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 08 Feb 2018 02:40:40 -0800 (PST) Received-SPF: pass (google.com: domain of benbrenson89@googlemail.com designates 2a00:1450:400c:c0c::231 as permitted sender) client-ip=2a00:1450:400c:c0c::231; Authentication-Results: gmr-mx.google.com; dkim=pass header.i=@googlemail.com header.s=20161025 header.b=anqNr8/g; spf=pass (google.com: domain of benbrenson89@googlemail.com designates 2a00:1450:400c:c0c::231 as permitted sender) smtp.mailfrom=benbrenson89@googlemail.com; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=googlemail.com Received: by mail-wr0-x231.google.com with SMTP id a43so4147116wrc.4 for ; Thu, 08 Feb 2018 02:40:40 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlemail.com; s=20161025; h=from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=Y1bUHljFXqXxZSvGEEFphNea7sA8rdzky7vjpM9xdgo=; b=anqNr8/gISQ1h8yW1qn/3aNL5YxteAubJiEoYEMXlgLIOvVOK48MRM85eUrteQW9OO s8xP3w02blzFgACyQayKt1LBNb/QFAnrEd4McpjaSN7MDNleYxOjwLLg5LCv0ddzCqga qd/Rblw54tmzSzrNGX1Cii5VSn72deJjFxKbHCxXHNInJkIOs5FjVC0tPmfyje4mOSMV ImZatCPlxDQyhelj76YR64cYV4TI15s2q+vvJLHQZ+JdKUgX1DAdg7FsBU7s4bV7i/YB tnzJwNdEBFjcv0COSIdZW1TY3gTMdgBkAAPTWVL7skFS5rkrBf6NUQEGt+AICWzR1PM9 dfmw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=Y1bUHljFXqXxZSvGEEFphNea7sA8rdzky7vjpM9xdgo=; b=sfQEhM3bPA83y/tmbhw24/GPImppegcRY8ZeNiYCo386IqZnnuc5IBII+9VbjcWrZ6 VHKS8VVxjAwkYZnJm56dQ6dqU4ql/fOKUrVgQ6C32cYdFN2NQ5xttH/F+nUeRIUuLQYP ZqwhEHCZVaY58Q9ETjl4KHPc5vER41tSnkF8BBcPU/rxJ0r4AgsN/uQkBmudUK1xuH7L Ps/jlsTY9t4sIKit93kBa8yYrz9k3pOnsz0W2p1aFGi2JEXVqTI/SzXA1y/7AymNL6Tc 71aT5F7oC3cfB8+9y74WHniEXiPtzIOsoAGVQ8Ix8d1mDEjBQFqSDYCUAVZN7gS03wM3 NdSQ== X-Gm-Message-State: APf1xPDFupCJRN/Tqea0PxQK4+eOn3WgC8M5XSaaO/mdiUuZj5nj3Lw9 rdw3U1iNbycdcZpaS5XHhbwL/A== X-Received: by 10.223.158.193 with SMTP id b1mr303287wrf.156.1518086439687; Thu, 08 Feb 2018 02:40:39 -0800 (PST) Return-Path: Received: from localhost.localdomain (ip-109-41-193-215.web.vodafone.de. [109.41.193.215]) by smtp.gmail.com with ESMTPSA id k74sm764689wrc.2.2018.02.08.02.40.38 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Thu, 08 Feb 2018 02:40:38 -0800 (PST) From: Benedikt Niedermayr X-Google-Original-From: Benedikt Niedermayr To: isar-users@googlegroups.com Cc: Benedikt Niedermayr Subject: [PATCH v2 0/7] Local apt cache with aptly V2 Date: Thu, 8 Feb 2018 11:40:05 +0100 Message-Id: <1518086412-12567-1-git-send-email-Benedikt.Niedermayr@mixed-mode.de> X-Mailer: git-send-email 2.7.4 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-TUID: vtgQWPRgRx/p Hi all, this series implements a PoC for running reproducible builds. The core is based on aptly, which is a nice mirroring tool for debian repositories. The mirroring solution does the following: - Aptly generates a mirrors for upstream repositories as well as snapshots and also a local repository for isar packages. - Get a list of all packages for mirroring before rootfs and buildchroot is beeing created. This is done by running an own bitbake cooker for parsing the recipes by taking append files, configs, datastore and also selected layers(bblayers) into account. This solves the problem when upstream repos getting updates between rootfs and buildchroot creation. - Aptly will recieve the list of packages and calculates all further required dependencies for the mirror. - Rootfs and buildchroot will never pull from upstream repos but instead from the local mirrors. - The following dependencies will be collected: * Depends * Build-Depends * Build-Depends-Indep Open Questions: - Think about a more dynamic multistrap.conf generation. Ŕunning sed to multiconfig templates is not really flexible. - Get a better naming convention for aptly mirrors, snapshots and repos. This will help to get multiconfig compatible. Where are pros and cons here? - Where to store the cache? TMPDIR might not be the best location. - Finding Build-Depends for packages build with dpkg-raw is not supported, yet. Howto test: Prefetching of Build-Depends requires all debian control files to be available, therefore two things have to be done: - Run unpackall task: bitbake isar-image-base:do_unpackall - Provide a debian/control file at ${WORKDIR}/${SRC_DIR}/debian/control - Overload the do_unpack_debian hook to make this possible(only for non debianized packages). I know not everbody will like this more complicated way, and first I was with them. But now I don't see any possibility to run real deterministic builds. Benedikt Niedermayr (7): Implement support for setting up the local apt mirror and isar repository with aptly. Added API class for apt cache. Added apt-cache functionality for buildchroot. Added apt-cache functionality for image rootfs. Added do_finalize_image task. Added support for installing isar packages to local isar repository. Introducing fetchall, unpackall and unpack_debian tasks. .../recipes-core/images/files/multistrap.conf.in | 8 +- meta-isar/recipes-core/images/isar-image-base.bb | 25 ++- meta/classes/apt-cache.bbclass | 237 +++++++++++++++++++++ meta/classes/base.bbclass | 20 ++ meta/classes/dpkg-base.bbclass | 15 +- meta/classes/image.bbclass | 23 +- meta/recipes-devtools/buildchroot/buildchroot.bb | 36 +++- .../buildchroot/files/configscript.sh | 1 + .../buildchroot/files/multistrap.conf.in | 17 +- meta/recipes-devtools/isar-apt/files/aptly.conf.in | 17 ++ meta/recipes-devtools/isar-apt/isar-apt.bb | 174 +++++++++++++-- 11 files changed, 524 insertions(+), 49 deletions(-) create mode 100644 meta/classes/apt-cache.bbclass create mode 100644 meta/recipes-devtools/isar-apt/files/aptly.conf.in -- 2.7.4