From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6662603828652670976 X-Received: by 2002:a50:8ed8:: with SMTP id x24mr228800edx.10.1551258337939; Wed, 27 Feb 2019 01:05:37 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a50:8b8a:: with SMTP id m10ls287930edm.7.gmail; Wed, 27 Feb 2019 01:05:37 -0800 (PST) X-Google-Smtp-Source: AHgI3Iae+dnm3T9ZdqCgf0eMMtwWe7bP4IvS7eWChMX42o+DOPfhntfkp1ayNyd4kZclQ1vepucy X-Received: by 2002:a50:8881:: with SMTP id d1mr235029edd.0.1551258337558; Wed, 27 Feb 2019 01:05:37 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1551258337; cv=none; d=google.com; s=arc-20160816; b=0fXAz8nCrVF4dSGZF9iSDtlE7FaXne7JYOJCXe4GWaInJ4ESOobH2BQ/KRVYT2q5DJ qYfhR8gSzeyclet8O9eMnVwoYvDGO7l9qb9oFxnn4OrHrzGrI8YHU7Rcy/0n4IHP2k8Y bS12H28DuGMnIQ7kEuHFLpevjQCZ37he61bwCWkxVP9VQVlrVfTpbZr+m9KcE4oYG/hR qmZy7lJ8oc0BJnusvJ4AqacfvWfCxhj/li4dUG48BW+u0iGpBG9+pF3TLkz43h24aU28 HB96vOoIU5WCWNdDDxvirtoNvVAlFjtvzT7cYOf2wgaDwSR7gDi48T85LHlZqZcU3iyL dB+A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:message-id:subject:to:from :date; bh=PMmuNUbUotSGbufl59uxlMMYIacMqgVIY0d7fTrJWsc=; b=aT5I/reyPVvJNBLDpNLxvm1/sl4lGWPpmlOTssregRcUgxfRYvVPX13MTrLKzt3SJM oUYb0RV27RYDOBBFDu4VCbOnz1It0gSS6ypfYDxZIV0AQVy23SDD/0lNBMWEBxhPJry0 MPtDfXkd+iwHa432LvxGF6IMsOp72KXiCOf2wcYrGUPdyqL11PM5bjQY6BO6N8JznMoK vQ4oTiEnMCQqRYE1CoTQteAHBIvWJzL92ekNtqMVySQv70STZ5rePhyzQv8cKkcVxcFe +H1RQ0UVMlBsZfESM4TOInG52QMYqNTFVGCY7mOnvv9w621eijVpIV54lQIiUID6LyHa 7Krw== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of henning.schild@siemens.com designates 192.35.17.28 as permitted sender) smtp.mailfrom=henning.schild@siemens.com Return-Path: Received: from goliath.siemens.de (goliath.siemens.de. [192.35.17.28]) by gmr-mx.google.com with ESMTPS id c7si330558ejx.1.2019.02.27.01.05.37 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 27 Feb 2019 01:05:37 -0800 (PST) Received-SPF: pass (google.com: domain of henning.schild@siemens.com designates 192.35.17.28 as permitted sender) client-ip=192.35.17.28; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of henning.schild@siemens.com designates 192.35.17.28 as permitted sender) smtp.mailfrom=henning.schild@siemens.com Received: from mail1.sbs.de (mail1.sbs.de [192.129.41.35]) by goliath.siemens.de (8.15.2/8.15.2) with ESMTPS id x1R95aQx012237 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 27 Feb 2019 10:05:37 +0100 Received: from md1za8fc.ad001.siemens.net ([139.25.69.171]) by mail1.sbs.de (8.15.2/8.15.2) with ESMTP id x1R95avC010541; Wed, 27 Feb 2019 10:05:36 +0100 Date: Wed, 27 Feb 2019 10:05:36 +0100 From: Henning Schild To: , "Maxim Yu. Osipov" , Alexander Smirnov , "Kiszka, Jan (CT RDA IOT SES-DE)" Subject: RFC: base-apt caching improvements Message-ID: <20190227100536.10caef1d@md1za8fc.ad001.siemens.net> X-Mailer: Claws Mail 3.17.3 (GTK+ 2.24.32; x86_64-pc-linux-gnu) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-TUID: B5naGxK+j4dq Hi, i did not really like the current approach to how we cache, because i thought we can and should do that somehow transparent and route all "apt" downloads through a proxy that will reprepro all files as the are requested. I did not find a proxy that can do that, maybe i did not look hard enough. So i came back to the currently implemented model. Collect the files from the rootfs-es after we are all done. The way that is currently done is getting them from the apt cache. Unfortunately that is not guaranteed to work. Any package we install could mess with apt-config and therefore the caching of the rootfs-es. (i.e. something like /etc/apt/apt.conf.d/docker-clean you will find a container images) So that cache can not be trusted. And that cache does not work for source packages and can not fulfill our needs with regards to full caching/archiving. So instead we should go and download all debs and sources of the currently installed packages of all rootfs-es (target and its buildchroots) explicitly and reprepro from there. Here is what this could look like: rm -rf /tmp/foo cd /tmp/foo dpkg -l | grep "^ii" | awk '{print $2"="$3}' | xargs apt-get -y download ### reprepro *.deb rm -rf /tmp/foo-src cd /tmp/foo-src dpkg -l | grep "^ii" | awk '{print $2"="$3}' | xargs apt-get -y source --download-only ### reprepro *.dsc This still lacks filtering out all the packages from isar-apt but could be an improvement to our current way. We do not need to trust those fragile caches because we are explicit. We can do sources ... and should probably just do all of them anyways. For selective sources we could "apt-get source" into a central dldir and copy the sources to the recipe workdir. Would be a matter of mounting that shared dir and changing do_apt_fetch to use that staging dir to leave a copy. In fact we (Siemens) want all sources, and it might be a sane default for others as well. Henning