From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6662603828652670976 X-Received: by 2002:a50:b361:: with SMTP id r30mr1037378edd.8.1551346845144; Thu, 28 Feb 2019 01:40:45 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a17:906:2a9b:: with SMTP id l27ls1002293eje.10.gmail; Thu, 28 Feb 2019 01:40:44 -0800 (PST) X-Google-Smtp-Source: AHgI3Ia9+/cniGjDSfD73r/sR3xEHLEYA2shKI4SuS1lMtplBXBoMaU/8e0O471J/8dsvnl7ehJg X-Received: by 2002:a17:906:3744:: with SMTP id e4mr994063ejc.3.1551346844676; Thu, 28 Feb 2019 01:40:44 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1551346844; cv=none; d=google.com; s=arc-20160816; b=aKKcXKDin7szykuiVcOyOOZ9NnwS7/n15ttFrGRkU5VQle+AxzB7ziKCXULxXhj7mH J2zZyZXgf4HvANBRFXeOQ6TW5IjJX7icIjt56SCHymQPQMGVSoU4Y69z3Qn+aG/hW7xA aMPUnl2YWYj+5mifgP7rKy1/m72f8woCYiU3WZiy/PMj/Rjs0VEngG7Zv3IZTKl95zAp gTJJ+wTQghk/gg2jjxXY/sGi7GDZGyU7pJqTSWl+wgC98pROQB8SacFvY8hufb0hXJrJ EOhIlN46Qkev/wY62jdNeIsQA0NP04dDniWHsO9FCwqgqJQh4tsB8SKAElXuB/ZAkmpH YCaA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:to:from:date; bh=+idTJA2OmT7xqCQVxfd4K3NCCmXOSr0b841PE+ntKwo=; b=m1k1soA49WB26P80KbTTFkF0CNpNKBaHaFOZX6ZYjBx0IMVUpvKUu5G2dj3WDSWvfO gpDAy7qRQ0x60Y/aaVEXOOJPkO8mlTMouEBG5ZoCVi53KiGfnqNXjV+VhLQNvXceB23+ p1FmsBmX46dXmwjdRyhz6izgBSjRb1CYcR9J5TseQAGWIN6zJC9ptkjiNo/cbssSBpbN 79jl9MYgDmTrxBMi+2fsfHqal/O4Lb2IwY1xIAMJD9JtMTiODPtfpjhgAnKTACROlaqV V7AlOpbl6ortt2VOQU/q17nVX3nqCGQL2Br0A3bU9cdJENM3KL7+XmMmgt9icanAjqip puyA== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of henning.schild@siemens.com designates 194.138.37.39 as permitted sender) smtp.mailfrom=henning.schild@siemens.com Return-Path: Received: from lizzard.sbs.de (lizzard.sbs.de. [194.138.37.39]) by gmr-mx.google.com with ESMTPS id r37si554277edd.2.2019.02.28.01.40.44 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 28 Feb 2019 01:40:44 -0800 (PST) Received-SPF: pass (google.com: domain of henning.schild@siemens.com designates 194.138.37.39 as permitted sender) client-ip=194.138.37.39; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of henning.schild@siemens.com designates 194.138.37.39 as permitted sender) smtp.mailfrom=henning.schild@siemens.com Received: from mail1.sbs.de (mail1.sbs.de [192.129.41.35]) by lizzard.sbs.de (8.15.2/8.15.2) with ESMTPS id x1S9eipN002886 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 28 Feb 2019 10:40:44 +0100 Received: from md1za8fc.ad001.siemens.net ([139.25.69.171]) by mail1.sbs.de (8.15.2/8.15.2) with ESMTP id x1S9ehMT007247; Thu, 28 Feb 2019 10:40:43 +0100 Date: Thu, 28 Feb 2019 10:40:42 +0100 From: Henning Schild To: , "Maxim Yu. Osipov" , Alexander Smirnov , "Kiszka, Jan (CT RDA IOT SES-DE)" Subject: Re: RFC: base-apt caching improvements Message-ID: <20190228104042.08bd64ac@md1za8fc.ad001.siemens.net> In-Reply-To: <20190227100536.10caef1d@md1za8fc.ad001.siemens.net> References: <20190227100536.10caef1d@md1za8fc.ad001.siemens.net> X-Mailer: Claws Mail 3.17.3 (GTK+ 2.24.32; x86_64-pc-linux-gnu) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-TUID: hzGudJIln4VU Am Wed, 27 Feb 2019 10:05:36 +0100 schrieb "[ext] Henning Schild" : > Hi, > > i did not really like the current approach to how we cache, because i > thought we can and should do that somehow transparent and route all > "apt" downloads through a proxy that will reprepro all files as the > are requested. > I did not find a proxy that can do that, maybe i did not look hard > enough. > > So i came back to the currently implemented model. Collect the files > from the rootfs-es after we are all done. The way that is currently > done is getting them from the apt cache. Unfortunately that is not > guaranteed to work. Any package we install could mess with apt-config > and therefore the caching of the rootfs-es. (i.e. something > like /etc/apt/apt.conf.d/docker-clean you will find a container > images) > > So that cache can not be trusted. And that cache does not work for > source packages and can not fulfill our needs with regards to full > caching/archiving. > > So instead we should go and download all debs and sources of the > currently installed packages of all rootfs-es (target and its > buildchroots) explicitly and reprepro from there. > > Here is what this could look like: > > rm -rf /tmp/foo > cd /tmp/foo > dpkg -l | grep "^ii" | awk '{print $2"="$3}' | xargs apt-get -y > download ### reprepro *.deb > rm -rf /tmp/foo-src > cd /tmp/foo-src > dpkg -l | grep "^ii" | awk '{print $2"="$3}' | xargs apt-get -y > source --download-only ### reprepro *.dsc > > This still lacks filtering out all the packages from isar-apt but Here are some snipplets to filter out isar-apt, does seem non-trivial and i ended up parsing. Might actually require some perl to dig into the guts of apt. apt-cache policy sed | grep -A 1 "\*\*\* $( dpkg-query --show --showformat '${Version}' sed )" | grep -q "file:/isar-apt" or grep -e "^Package: hello$" /var/lib/apt/lists/_isar-apt_dists_isar_main*_Packages -A 1 | grep "Version: $( dpkg-query --show --showformat '{Version}' hello ) The way we currently filter out isar-apt is pretty fragile and should be improved. We find packages by matching file names. Instead we should read the .debs with dpkg-deb and read the repo with reprepro. Henning > could be an improvement to our current way. We do not need to trust > those fragile caches because we are explicit. We can do sources ... > and should probably just do all of them anyways. > For selective sources we could "apt-get source" into a central dldir > and copy the sources to the recipe workdir. Would be a matter of > mounting that shared dir and changing do_apt_fetch to use that > staging dir to leave a copy. In fact we (Siemens) want all sources, > and it might be a sane default for others as well. > > Henning >