From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6752900179667451904 X-Received: by 2002:aa7:cd01:: with SMTP id b1mr20708060edw.122.1572282095410; Mon, 28 Oct 2019 10:01:35 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:aa7:d80b:: with SMTP id v11ls577641edq.0.gmail; Mon, 28 Oct 2019 10:01:34 -0700 (PDT) X-Google-Smtp-Source: APXvYqxr7XVdjBt58pr+2lwg7ypE+q/toiBH7ZDtTrhNaYVDUB/3nobkLU4Bpr/Mg6VXnlkZ5fAU X-Received: by 2002:a50:9b43:: with SMTP id a3mr17731313edj.73.1572282094713; Mon, 28 Oct 2019 10:01:34 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1572282094; cv=none; d=google.com; s=arc-20160816; b=AbeWN2VzaMx5EO9AFyme+LLWMcDkggtdGUElgQ/NmrTLJUsu1Y/J+0THUB9BCoRyZc 2YhWzy9jUb18lcGgElMTGT4lQk3eZxD+LhmiApeKWKWMlc/7HKZ1RBA8YG9PYU6p2Jcr 73Y/mI8R/0ePbPZP7j0SLB0aCTKsfWVxgGIFtydJRtFiv2+sCxcOBc6bBvur+nooZ2G6 xcqXSWpHiMJlmkvGfe6ttyzewHhZAOCax+7OqGWaG+Oz7YG1i3l4m2atO7GN5zCKjRrw QNnBaYmAUxwb83txWl7+0hlFiYcEQMgc3oXU86hgPwFRtzgStnew41VrrFSolacGjxrZ Nl5g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from; bh=jku7+nztG9XWbqqibyEfKjayoyohLV1z9xj5OSLfA0Y=; b=IRWBFPGWPt0k3bw67gEFm6DfWN4+fhLIcSnJ82v2bytjf1lJ5brwCTpYcQ+7KtRzWV R/1N/lV777MX700gb9Q2X1A2EXFAFmRXaQYScAdnBBo3PIhnYcg0tUx4KvxwBR4L1dC1 VabW7JvrquXJr/xcW+9LsqiWHMxXrQEIiqKX9asAxYUBH17wx6Yrz3S6Hbzp4g3ZJk4L KhrfuTwUF7/EIbfgUW2+wiW5Sf6zqeTzg6pCXYn5V8UVCy3KqqhaOJrNPAJ2OLdJ6Kru oqas76Qc7AYePyhc2QmHVX8WWfo0d2tl7TsaUkDrbpPwo7z3mH61VOoOWHiGmzeq/V9t JsAg== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of henning.schild@siemens.com designates 194.138.37.39 as permitted sender) smtp.mailfrom=henning.schild@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Return-Path: Received: from lizzard.sbs.de (lizzard.sbs.de. [194.138.37.39]) by gmr-mx.google.com with ESMTPS id y21si376468ejp.1.2019.10.28.10.01.34 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 28 Oct 2019 10:01:34 -0700 (PDT) Received-SPF: pass (google.com: domain of henning.schild@siemens.com designates 194.138.37.39 as permitted sender) client-ip=194.138.37.39; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of henning.schild@siemens.com designates 194.138.37.39 as permitted sender) smtp.mailfrom=henning.schild@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Received: from mail1.sbs.de (mail1.sbs.de [192.129.41.35]) by lizzard.sbs.de (8.15.2/8.15.2) with ESMTPS id x9SH1YIF032232 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK) for ; Mon, 28 Oct 2019 18:01:34 +0100 Received: from md1za8fc.ad001.siemens.net ([139.25.69.244]) by mail1.sbs.de (8.15.2/8.15.2) with ESMTP id x9SH1YP6025390; Mon, 28 Oct 2019 18:01:34 +0100 From: Henning Schild To: isar-users@googlegroups.com Cc: Henning Schild Subject: [PATCH 00/17] base-apt rework Date: Mon, 28 Oct 2019 18:01:15 +0100 Message-Id: <20191028170132.28072-1-henning.schild@siemens.com> X-Mailer: git-send-email 2.23.0 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-TUID: bt2Z/2zZkBPG From: Henning Schild This series build upon "[PATCHv2 0/9] a few cleanups a bug and a feature". base-apt has two main issues that this series fixes: - it relied on the rootfs /var/cache/apt to contain all *.debs - this is only true of nobody cleaned up, could happen with hooks like used in "isar-disable-apt-cache" or similar mechanisms - it did not handle "dsc"s (source packages) It had other minor issues that are described in the individual patches. The series fixes those issues and extends CI to fully test base-apt. The general mechanism used in the end is: - split all apt-gets into a download and an execution phase - copy out all download inbetween the two phases - execution phase can now go ahead and possibly delete cached files from the rootfs - actually copy in everything "old" before the download phase - that means the download phase potentially becomes a noop and we speed up by using already downloaded files (new feature) There are a few open points that i will mention later. Henning Schild (17): repository: new class to deal with repos dpkg-base: add download caching of apt:// downloads base-apt: change the sources.list to also offer deb-src base-apt: add deb-src packages as well base-apt: do not skip gpg check when it is signed ci: conf: add "hello" to the sample config and every build meta: split all apt-get invocations into download and execution meta: create DL_DIR support for all apt-get downloaded .debs meta: import DL_DIR debs before apt-get download steps base-apt: populate from DEBDIR as well base-apt: drop the "apt_cache" feature base-apt: do not copy debs directly out of rootfss anymore base-apt: rework base-apt population base-apt: move class "base-apt-helper" into only user CI: include "isar-disable-apt-cache" into all CI images CI: include "cowsay" into default build to test dpkg-gbp CI: set BB_NO_NETWORK for cached rebuild doc/user_manual.md | 1 + meta-isar/conf/local.conf.sample | 10 +-- meta/classes/base-apt-helper.bbclass | 57 -------------- meta/classes/deb-dl-dir.bbclass | 24 ++++++ meta/classes/dpkg-base.bbclass | 45 ++++++----- meta/classes/dpkg-gbp.bbclass | 5 ++ meta/classes/dpkg.bbclass | 7 +- meta/classes/image-cache-extension.bbclass | 67 +++++++++++++--- meta/classes/image-locales-extension.bbclass | 2 +- meta/classes/image-tools-extension.bbclass | 7 ++ meta/classes/image.bbclass | 2 +- meta/classes/repository.bbclass | 78 +++++++++++++++++++ meta/classes/rootfs.bbclass | 28 ++++--- meta/conf/bitbake.conf | 2 + .../isar-bootstrap/files/base-apt-sources | 1 - .../isar-bootstrap/isar-bootstrap.inc | 16 ++-- meta/recipes-devtools/base-apt/base-apt.bb | 31 ++------ .../base-apt/files/distributions.in | 3 - .../buildchroot/files/deps.sh | 37 +++++---- .../isar-apt/files/distributions.in | 3 - meta/recipes-devtools/isar-apt/isar-apt.bb | 26 ++----- scripts/ci_build.sh | 7 +- 22 files changed, 272 insertions(+), 187 deletions(-) delete mode 100644 meta/classes/base-apt-helper.bbclass create mode 100644 meta/classes/deb-dl-dir.bbclass create mode 100644 meta/classes/repository.bbclass delete mode 100644 meta/recipes-core/isar-bootstrap/files/base-apt-sources delete mode 100644 meta/recipes-devtools/base-apt/files/distributions.in delete mode 100644 meta/recipes-devtools/isar-apt/files/distributions.in -- 2.23.0