From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6519381492065370112 X-Received: by 10.28.30.68 with SMTP id e65mr143374wme.17.1517911789948; Tue, 06 Feb 2018 02:09:49 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 10.28.199.10 with SMTP id x10ls2063794wmf.9.gmail; Tue, 06 Feb 2018 02:09:49 -0800 (PST) X-Google-Smtp-Source: AH8x225QiQXRTQkg6CuDGNQhsbM0aoMQ2756asp4MjVyl39v/MgBlUc5l/PHarQTi7ZfwJRxEGXc X-Received: by 10.28.5.206 with SMTP id 197mr152691wmf.24.1517911789506; Tue, 06 Feb 2018 02:09:49 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1517911789; cv=none; d=google.com; s=arc-20160816; b=zcjLsvQXl1ooAgGpToq0YJtmBcgoNMxuWZnIesXqY3E92ZRuaUHcVVPhOEzz5QKiCE RezGPqw0w4UX0s06LDVzA347YSeQ2pxhE8E7EzbI0lvPWegkAgYI39s0jt4rAvJ+EFpy LBHeC0uSBMfIQgdWR8R0UjLQgh+bfwLcNGMSRLB+a/7TdwHxZTkmT9qowxsveS7Aa8Mt ZCVKQelF/XN5rHvrza5ffRTgQqhUiX1txonDyCP44uSQ7prRk1vJqobiRgWmr7PgvMG2 mWd61B21M8Xb1sv/EaQYTbO4UtXJTmTz0D5M44ofCmbHlyJE5p+LwTWl6mkB7bxeIqal RB5Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:dkim-signature:arc-authentication-results; bh=yY3J+KgnyiPOjt7KDApUj4QtV1e4VvmITTDKNFsY4Sk=; b=QwFc2UNQPVkQkuOaEAn3FkxwEWNFK4PwOM/lH+BO0kt7skGg2WlPxuUkGxnz5fDHMX SuBjDLLytsn6tj6V+H/nXMJ1LRrpWHkXdADqbOxHOtUQOzzzrqCEIfcp8uF1kGvGnv6k Lc2OyJ6e2iZefbctEX6diIgpoAqROkOh5/O9SFAsXmaVS2vnBenf5PG8pcA0OnlDwivi 1kWMGCmRSRQXc9lE3HDk7XjGRB0IHTLGROnPrGAPqFeppdJarsefjuhmdTto9K09N6wr qh3tmer9Y2ikP6fTciL8nR6pdLPoA9lNkYBN/wybDphU4O0KLli8vsBX2TNzaTIVPu2Q PvAQ== ARC-Authentication-Results: i=1; gmr-mx.google.com; dkim=pass header.i=@googlemail.com header.s=20161025 header.b=baFyLICR; spf=pass (google.com: domain of benbrenson89@googlemail.com designates 2a00:1450:400c:c09::230 as permitted sender) smtp.mailfrom=benbrenson89@googlemail.com; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=googlemail.com Return-Path: Received: from mail-wm0-x230.google.com (mail-wm0-x230.google.com. [2a00:1450:400c:c09::230]) by gmr-mx.google.com with ESMTPS id b10si603868wmc.0.2018.02.06.02.09.49 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 06 Feb 2018 02:09:49 -0800 (PST) Received-SPF: pass (google.com: domain of benbrenson89@googlemail.com designates 2a00:1450:400c:c09::230 as permitted sender) client-ip=2a00:1450:400c:c09::230; Authentication-Results: gmr-mx.google.com; dkim=pass header.i=@googlemail.com header.s=20161025 header.b=baFyLICR; spf=pass (google.com: domain of benbrenson89@googlemail.com designates 2a00:1450:400c:c09::230 as permitted sender) smtp.mailfrom=benbrenson89@googlemail.com; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=googlemail.com Received: by mail-wm0-x230.google.com with SMTP id x4-v6so17082943wmc.0 for ; Tue, 06 Feb 2018 02:09:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlemail.com; s=20161025; h=from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=yY3J+KgnyiPOjt7KDApUj4QtV1e4VvmITTDKNFsY4Sk=; b=baFyLICRAsvADChUAKrnRUF8yQ0lZ5ovoca/zMyeL9GoHoOUZF3iD8xacHfig+EY0r 8KFbbYgWdKTKmgNGs/pB2TAslU//x6k4d7Sc2v4PiCkMC8pvyGOFPX0/9j1+BVkoeBpd TG9fzGCmEx8gUbMAXYUM/Miv6fw1CVjDvrAeQiZNgDjc7McjGtmUvayL4XoqZZ93mNrA IuiW8WJvkYFqmeIkmYhdd8bbiXA9WB/DrQXgKVTWD/AvTufG8whXKfI92vIG57FXjQvq Gomk5le8YbWD2Buo7e4brpmvQVz5mnh6vEp5xvXHHaX/4UYaHMlUP8qOm9TvWhhAw7X4 6TDw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=yY3J+KgnyiPOjt7KDApUj4QtV1e4VvmITTDKNFsY4Sk=; b=AZhCvP7tesIawfqgshtPPmvAGhsnGryJVE4+pA8d0DP3cUN8SiEl8QTjl2KesreDsM mqAa8Wyb0IOnL0rcwYAv0dNIOWtyVNF8tB+8l8vvv6TsOKhIhadyhK8afDuHm5YuvHvF ezKBO/Exc4zp2AvOF3EGGlTuHqdzXJWeriLQ34oETZZoyoE0iU5rW/7pfFn6p2t0TU38 h+cV6ABerYhkMrFghbDw8yd+GTkFDj6a8k2VJxXjW56nthLL7wtCTKfTM0PIfOEvs1Rk esqLIx1BttxzIT4CiBmwB7EbvQ2b4cmWWfVb1x0jpq41wapgaWU3+9yDuDbriFeLXR0f /lrg== X-Gm-Message-State: APf1xPCmEgOYHzMG9BwWG8kW2VXYDW98gWpsfa8u9UTTOyKTZxn2NV67 BnpC62PMv4oYM6vMKDWg/MgpLg== X-Received: by 10.28.127.73 with SMTP id a70mr1206528wmd.43.1517911789005; Tue, 06 Feb 2018 02:09:49 -0800 (PST) Return-Path: Received: from localhost.localdomain (ip-109-41-195-133.web.vodafone.de. [109.41.195.133]) by smtp.gmail.com with ESMTPSA id r5sm13466342wmg.30.2018.02.06.02.09.47 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Tue, 06 Feb 2018 02:09:48 -0800 (PST) From: Benedikt Niedermayr X-Google-Original-From: Benedikt Niedermayr To: isar-users@googlegroups.com Cc: Benedikt Niedermayr Subject: [PATCH 0/6] Local apt cache with aptly Date: Tue, 6 Feb 2018 11:09:33 +0100 Message-Id: <1517911779-30507-1-git-send-email-Benedikt.Niedermayr@mixed-mode.de> X-Mailer: git-send-email 2.7.4 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-TUID: Ypx0JlDM1QqU Hi all, this series implements a PoC for running reproducible builds. The core is based on aptly, which is a nice mirroring tool for debian repositories. The mirroring solution does the following: - Aptly generates a mirrors for upstream repositories as well as snapshots and also a local repository for isar packages. - Get a list of all packages for mirroring before rootfs and buildchroot is beeing created. This is done by running an own bitbake cooker for parsing the recipes by taking append files, configs, datastore and also selected layers(bblayers) into account. This solves the problem when upstream repos getting updates between rootfs and buildchroot creation. - Aptly will recieve the list of packages and calculates all further required dependencies for the mirror. - Rootfs and buildchroot will never pull from upstream repos but instead from the local mirrors. Open Questions: - Think about a more dynamic multistrap.conf generation. Ŕunning sed to multiconfig templates is not really flexible. - Get a better naming convention for aptly mirrors, snapshots and repos. This will help to get multiconfig compatible. Where are pros and cons here? - Where to store the cache? TMPDIR might not be the best location. This series is based on commit 384750d of next branch. Running with multiconfig was not tested, yet. But maybe you can do it? Benedikt Niedermayr (6): Implement support for setting up the local apt mirror and isar repository with aptly. Added API class for apt cache. Added apt-cache functionality for buildchroot. Added apt-cache functionality for image rootfs. Added do_finalize_image task. Added support for installing isar packages to local isar repository. .../recipes-core/images/files/multistrap.conf.in | 8 +- meta-isar/recipes-core/images/isar-image-base.bb | 22 +- meta/classes/apt-cache.bbclass | 235 +++++++++++++++++++++ meta/classes/dpkg-base.bbclass | 15 +- meta/classes/image.bbclass | 23 +- meta/recipes-devtools/buildchroot/buildchroot.bb | 35 ++- .../buildchroot/files/multistrap.conf.in | 17 +- meta/recipes-devtools/isar-apt/files/aptly.conf.in | 17 ++ meta/recipes-devtools/isar-apt/isar-apt.bb | 131 ++++++++++-- 9 files changed, 454 insertions(+), 49 deletions(-) create mode 100644 meta/classes/apt-cache.bbclass create mode 100644 meta/recipes-devtools/isar-apt/files/aptly.conf.in -- 2.7.4