From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6750653680817537024 X-Received: by 2002:a5d:63ca:: with SMTP id c10mr4349836wrw.79.1571759041982; Tue, 22 Oct 2019 08:44:01 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a5d:690e:: with SMTP id t14ls9478843wru.2.gmail; Tue, 22 Oct 2019 08:44:01 -0700 (PDT) X-Google-Smtp-Source: APXvYqxKOp6iZDC971cSQnNwAbuaVh2IJVCMhrBkrjXgxjDt6vW3NpLO00SlKyXKqiMHvZhD/YIW X-Received: by 2002:a5d:4748:: with SMTP id o8mr4249337wrs.239.1571759041446; Tue, 22 Oct 2019 08:44:01 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1571759041; cv=none; d=google.com; s=arc-20160816; b=wy/bw5JyPgvmFHzZ4rg05iXBEAgGhvZytRAAf44GfQJuPYttl3m8KT76fokNDUjkbs KymXzMHBml77ypYQ/f0MG/6Ij4GDzPD4PtozrcRdQBx1PtkHO7JyP+2LiXK9WQQI/3es C6lGd/LOWD4dR2tWo1rXEY/8oXw/c1Ef8+J4iMLPM2F4F4A5tmTRVlScEsF/xrbl7Ie5 NHPtvEibRLIrYkka/wmrIJsarZgqjlWUlMz1z+TH8XrBhp5jAaJ4MZj8tv7dWxp4n+nU iivK9CL1S+V5ue6htEw5/Tly+xZNAwwCjnY8Iz48wczbnCcizJE4Nsz0usCkuGMiDa6a irBA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:message-id:subject:to:from :date; bh=/ZX6i4+8mQZWW8BTpVk1ROPP6b+tE04pdL9H/sg2CF0=; b=QHS/zLAHoP0vFHPdDfWuHq0ecASy2cia/C036wObUEiUDT/rLW7j0cMnpRGAkcp6JN vC3bbmyHsM/8hFBFcOH91D+txafKLzXAXoGLC2e/jWsvCpHrNUvaBoV5fE/lGXZG/iZS n1gL8/WvQpnlbPQY3jTzKz7uCsIkA9yQoiZsVxIfkzOk7Vqgi1VlhpPGai99/l7AoJ75 Zc3feIb6pTdtmaPIx5NnlR5a0ryIQAxlqiNDdg4UlWyAosALl69xNeoDB/hjkqpASpZs CwatRMSLsAIhHhVmFHBwElJ1rvkQDR4BiNO84tiTNhjYc3zthdEXln3I60VH22ewl4cT PDwQ== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of henning.schild@siemens.com designates 192.35.17.2 as permitted sender) smtp.mailfrom=henning.schild@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Return-Path: Received: from thoth.sbs.de (thoth.sbs.de. [192.35.17.2]) by gmr-mx.google.com with ESMTPS id l23si688125wmg.2.2019.10.22.08.44.01 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 22 Oct 2019 08:44:01 -0700 (PDT) Received-SPF: pass (google.com: domain of henning.schild@siemens.com designates 192.35.17.2 as permitted sender) client-ip=192.35.17.2; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of henning.schild@siemens.com designates 192.35.17.2 as permitted sender) smtp.mailfrom=henning.schild@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Received: from mail1.sbs.de (mail1.sbs.de [192.129.41.35]) by thoth.sbs.de (8.15.2/8.15.2) with ESMTPS id x9MFi0dk026980 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 22 Oct 2019 17:44:00 +0200 Received: from md1za8fc.ad001.siemens.net ([139.25.69.254]) by mail1.sbs.de (8.15.2/8.15.2) with ESMTP id x9MFi0Id014133; Tue, 22 Oct 2019 17:44:00 +0200 Date: Tue, 22 Oct 2019 17:43:59 +0200 From: Henning Schild To: , Claudius Heine , "Kiszka, Jan (CT RDA IOT SES-DE)" Subject: base-apt work in progress, RFC Message-ID: <20191022174359.40fb4045@md1za8fc.ad001.siemens.net> X-Mailer: Claws Mail 3.17.3 (GTK+ 2.24.32; x86_64-pc-linux-gnu) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-TUID: jdb4x/heSm+w Hi all, i now have a working version that changed a few things around the implementation of base-apt, to make sure it will also work when some rootfs clear their package cache etc. major changes: - do not rely on rootfs package cache anymore - support .dsc source packages - include full debootstrap via a common mechanism - full download caching support in DL_DIR, .deb + .dsc + tar - base-apt building now is the first task in the build chain, not the last - many cleanups planned changes on top: - deep tree download - sibbling packages - source packages - build deps - while more goto -3 But now i hit a major issue that i do not know how to best deal with, it is connected to my best friend feature ... multiconfig. But in fact it could also show in single config builds, because they have up to 4 rootfs as well. So far base-apt mixes together packages from several rootfss. If these rootfss had different versions of a package the latest version of that package would win. See meta/classes/image-cache-extension.bbclass, we are mixing the image with the two buildchroots. The preferences and sources in these three rootfs can lead to such mixing. When this base-apt gets used for a rebuild such newer packages can leak to rootfs where they were not in the first run. I see two general directions to solve that: 1. align the sources and preferences across the 4-tuple image, buildchroots, sdk ... and have a base-apt for every image-4-tuple 2. have a base-apt for every rootfs-PN-PV 3. if reprepro has deduplication accross distros/suites possibly have all "base-apts" in one repo, and make every instance a "suite" so instead of "buster" think "isar-image-base-MACHINE-DISTRO-DISTRO_ARCH_1.0" Now if they in the end are all buster and use the same file, we only need to store it once and can still store differences. Same for sources, where this even works cross-arch. I think 1 is not a good idea, isar currently allows sources and preferences to differ and it is probably a wanted feature. 2 is "clear" but we have a lot of those repos, which might be hard to handle, if we think people pull updates from there 3 is currently me preferred version if deduplication is in place, i think even without it I am not sure whether the solution to that problem should be part of the first patch-series. It is not a new problem ... In fact i will probably sent my current state without the "deep tree download", which would also be something on top. Henning