From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6788114222392803328 X-Received: by 2002:a1c:9d08:: with SMTP id g8mr11879815wme.141.1580481003221; Fri, 31 Jan 2020 06:30:03 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 2002:adf:c3d1:: with SMTP id d17ls4291341wrg.10.gmail; Fri, 31 Jan 2020 06:30:02 -0800 (PST) X-Google-Smtp-Source: APXvYqz6cIP/WaMQl0k/xJ/lTeY1ThMuSjITh+2/GMrmcV/XNYl+VxxJqgTarAvJNaHwXY1Vw0jH X-Received: by 2002:a5d:61cb:: with SMTP id q11mr730470wrv.71.1580481002457; Fri, 31 Jan 2020 06:30:02 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1580481002; cv=none; d=google.com; s=arc-20160816; b=zYXTgcZ9jah9piEb6vBjKLKo+0aNF9O0Lrqq+5rXaztjf23lwGyC2fmcCxS+5riQ/c ZsFO6ke8+uSzXf+ye5+EG/KrqT/6Q3BqkAfP8wlrSXfeuFL3T8FFnjxdEKjWkI+o8QqO nSccwueAH8jql5+wF2TqoFnW2L8Zw8tK+Z6IMQGpXAfHY1uRU9tHH62Em9I+4dJvP9Ap JnMTqvCOKnhD9yH5Vz0GwEmcrg+mnYEdLTplnFLHBxnwT3fnS8P9s0Ww6Gpker+tg1RU GesLd0C3ds09veXJ2xGbIj2A8Omy9Ug0CHwDCrXW7gm41/4l4NXntnDY/6qeHLocBR9e k/+A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from; bh=KU20zgUAt3TyzrKp6RqSgc3ZpE2A7Htg88pv4/KnU7k=; b=P9i6P7P+6kO7hi8l76akBMUQx1rFHIEtjMBXFXY9VzS5MDYzkkhZDMZAhiX5+vowRc 9CNpFWNDBJfmRHtVtYdkJY0DOLk8M3h6kZbHBrWApxuowHzcnOwPi6xcy/K3tq5X4YxD vD1PmIamwTT3AEZMEf8ujEic39eLbCUpmDI70bm4R4s6J0eWvFnLiCMIPfgOGEMuN0+s njwMjvF43JgCpfuL6SMI3wZOJSb/0ubZmcThqItmr0R02somdPS7z0zvYmL1/LjPscIH jL+CI7r6y/RxTlCojAgz9hwRACNry97cdslacvxovEeDKtjpKzGx9kpNbzi9C3ezIoOv 1pZw== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of henning.schild@siemens.com designates 192.35.17.2 as permitted sender) smtp.mailfrom=henning.schild@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Return-Path: Received: from thoth.sbs.de (thoth.sbs.de. [192.35.17.2]) by gmr-mx.google.com with ESMTPS id e21si607276wme.1.2020.01.31.06.30.02 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 31 Jan 2020 06:30:02 -0800 (PST) Received-SPF: pass (google.com: domain of henning.schild@siemens.com designates 192.35.17.2 as permitted sender) client-ip=192.35.17.2; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of henning.schild@siemens.com designates 192.35.17.2 as permitted sender) smtp.mailfrom=henning.schild@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Received: from mail1.sbs.de (mail1.sbs.de [192.129.41.35]) by thoth.sbs.de (8.15.2/8.15.2) with ESMTPS id 00VEU1vn022690 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 31 Jan 2020 15:30:01 +0100 Received: from md1za8fc.ad001.siemens.net ([139.25.68.229]) by mail1.sbs.de (8.15.2/8.15.2) with ESMTP id 00VEU1RE030231; Fri, 31 Jan 2020 15:30:01 +0100 From: Henning Schild To: isar-users@googlegroups.com Cc: Vijai Kumar K , Henning Schild Subject: [PATCHv4 00/26] base-apt-rework Date: Fri, 31 Jan 2020 15:29:34 +0100 Message-Id: <20200131143000.14873-1-henning.schild@siemens.com> X-Mailer: git-send-email 2.24.1 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-TUID: ANFqhZ5BO6p1 From: Henning Schild changes since v2: Rebased/merged onto current next and added p26. The custom kernel caching that was missing in v2 is not required on next anymore. This one was sent as v3 by Vijai Kumar K . That just contained a fix in case the patches would not be rebased, can be ignored. --- That one had to wait for some time. I finally managed to have a look at it again and came up with a v2. Sorry that one is really big and a lot of things happened since v1. The most significant change to last time it p23, which switches the way the offline cache works. Unfortunately this is turned out pretty big, splitting it into readably and working chunks was not reasonable. Everything up to p19 was already sent once, at least roughly ... I hope a v3 will come faster and we will have better tracking of changes between 2 and 3. changes since v1: - pull the base-apt building to the front of the build chain - performance optimze all that copying - add import/export of bootstrap step - rebased on current next - exclude isar-apt in deb_dl_dir_export - use find exec + for deb_dl_dir_import - fixed "==" operator in deps.sh in p8 - fixed typo in p2 message - fixed variable scope to local in repository functions Henning Schild (26): repository: new class to deal with repos dpkg-base: add download caching of apt:// downloads meta: dpkg-base: convert "exit" into "return 0" base-apt: change the sources.list to also offer deb-src base-apt: add deb-src packages as well base-apt: do not skip gpg check when it is signed ci: conf: add "hello" to the sample config and every build meta: split all apt-get invocations into download and execution meta: create DL_DIR support for all apt-get downloaded .debs meta: import DL_DIR debs before apt-get download steps meta: include DL_DIR deb import/export into bootstrap base-apt: populate from DEBDIR as well base-apt: drop the "apt_cache" feature base-apt: do not copy debs directly out of rootfss anymore base-apt: rework base-apt population base-apt: move class "base-apt-helper" into only user CI: include "isar-disable-apt-cache" into all CI images CI: include "cowsay" into default build to test dpkg-gbp CI: set BB_NO_NETWORK for cached rebuild meta: repository: implement repo_contains_package and use it in base_apt meta: repository: simplify the matching of packages meta: deb-dl-dir: do not cache debs from isar-apt base-apt: pull base-apt population to the front of the build chain meta: base-apt: remove isar-apt check from population meta: deb-dl-dir: only export newly downloaded files CI: test a custom kernel build in the base-apt offline rebuild doc/user_manual.md | 9 +- meta-isar/conf/local.conf.sample | 10 +- meta/classes/base-apt-helper.bbclass | 57 ---------- meta/classes/deb-dl-dir.bbclass | 38 +++++++ meta/classes/dpkg-base.bbclass | 47 ++++---- meta/classes/dpkg-gbp.bbclass | 5 + meta/classes/dpkg.bbclass | 7 +- meta/classes/image-cache-extension.bbclass | 36 ------- meta/classes/image-locales-extension.bbclass | 2 +- meta/classes/image-tools-extension.bbclass | 7 ++ meta/classes/image.bbclass | 3 +- meta/classes/repository.bbclass | 100 ++++++++++++++++++ meta/classes/rootfs.bbclass | 30 +++--- meta/conf/bitbake.conf | 6 +- .../isar-bootstrap/files/base-apt-sources | 1 - .../isar-bootstrap/isar-bootstrap.inc | 21 ++-- meta/recipes-devtools/base-apt/base-apt.bb | 92 ++++++++++------ .../base-apt/files/distributions.in | 3 - .../buildchroot/files/deps.sh | 37 ++++--- .../isar-apt/files/distributions.in | 3 - meta/recipes-devtools/isar-apt/isar-apt.bb | 26 ++--- scripts/ci_build.sh | 15 ++- 22 files changed, 322 insertions(+), 233 deletions(-) delete mode 100644 meta/classes/base-apt-helper.bbclass create mode 100644 meta/classes/deb-dl-dir.bbclass delete mode 100644 meta/classes/image-cache-extension.bbclass create mode 100644 meta/classes/repository.bbclass delete mode 100644 meta/recipes-core/isar-bootstrap/files/base-apt-sources delete mode 100644 meta/recipes-devtools/base-apt/files/distributions.in delete mode 100644 meta/recipes-devtools/isar-apt/files/distributions.in -- 2.24.1