From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 7182796543154454528 X-Received: by 2002:a25:c185:0:b0:7b4:11fb:dc1e with SMTP id r127-20020a25c185000000b007b411fbdc1emr4019629ybf.206.1675073124899; Mon, 30 Jan 2023 02:05:24 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a0d:df54:0:b0:506:36bb:59a4 with SMTP id i81-20020a0ddf54000000b0050636bb59a4ls7713637ywe.3.-pod-prod-gmail; Mon, 30 Jan 2023 02:05:24 -0800 (PST) X-Google-Smtp-Source: AK7set/T8aIJ4/3SDo9QWH6sm5aUjEz2GJhvq4AcVkgthJ7vNjK137fZQssBHpgRL5Ea25cgPr4h X-Received: by 2002:a81:6085:0:b0:506:3e44:7c3 with SMTP id u127-20020a816085000000b005063e4407c3mr6728043ywb.34.1675073124092; Mon, 30 Jan 2023 02:05:24 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1675073124; cv=none; d=google.com; s=arc-20160816; b=teaxatn0qOMEB3EPDHQyadyVoHgRgfw4l9qUK9St8lqU7FP1KLU0v1LU2TND9g2F8P M+dOmwsDTnjDoL5MvoYeMjMKmSnX/GB6O9xaGVsbuzbYw4Q98FlncVzHbXHPpkVFQaQd t3Q0Z06ecBd5JKMRb74JpjeUH5EyR3N4fnp3d2qjUqavsHApqyxf1zYms9hLXDgUGCQ7 puc7NNuMRbI5FdNMfGAlpy7vSNI2JQLlS4CTWg/DUMgUgrLRPhLITFG22MlhWYkJUjph 678xUm60cZhwREO2rp3YYf7p1M/FB+OrioW87UOrgaOEtZLo8XHkRoYXUqukW6jTLkH5 DPOQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:dkim-signature; bh=E3NTemlc1jph+TyQc1Rj/ZtDeTjGrm+NA1JQ3UZSjqA=; b=MgLsizC9cBGczj9hag0oeuFG8PuvzSVSEOtc0hMx5vrxw0I905OxkYXYj5XwuHfJCH ShRkgEYq+JIAV8PjSeCU49BksAueoK5M6pWBqJPbtLrPUebcsIHSSUfk89MZG1YqT5a/ yubLvV/CygVpxkd8q1ofraKhtqx9v+IftBGHWrZs8GNvuvJF0ihJRoJvNwgyMqWDU85W x6RCitNg6ry5p8b/zVejMKhK5upfC4n3x52BevZywHZeK5ao99qClKAdslU7SBnaQVMb vdwJ2N1htk0q4M793vjiercfWuKz+Kmt3LfvPQh6DwZ0wf4kzSXJCxNcNLfdDzi7WHTh 8x5g== ARC-Authentication-Results: i=1; gmr-mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b=qZ34UNI6; spf=pass (google.com: domain of roberto.foglietta@gmail.com designates 2607:f8b0:4864:20::f33 as permitted sender) smtp.mailfrom=roberto.foglietta@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from mail-qv1-xf33.google.com (mail-qv1-xf33.google.com. [2607:f8b0:4864:20::f33]) by gmr-mx.google.com with ESMTPS id bc29-20020a05690c001d00b004fa49c05aa9si2140165ywb.0.2023.01.30.02.05.24 for (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 30 Jan 2023 02:05:24 -0800 (PST) Received-SPF: pass (google.com: domain of roberto.foglietta@gmail.com designates 2607:f8b0:4864:20::f33 as permitted sender) client-ip=2607:f8b0:4864:20::f33; Authentication-Results: gmr-mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b=qZ34UNI6; spf=pass (google.com: domain of roberto.foglietta@gmail.com designates 2607:f8b0:4864:20::f33 as permitted sender) smtp.mailfrom=roberto.foglietta@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: by mail-qv1-xf33.google.com with SMTP id k2so8502444qvd.12 for ; Mon, 30 Jan 2023 02:05:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=E3NTemlc1jph+TyQc1Rj/ZtDeTjGrm+NA1JQ3UZSjqA=; b=qZ34UNI6A8yQmsX8BQAlWVaxY3j+bn3WgcNN+LT6f3BfyL0oSQ+2a/+dRxi/K0gr7h dXYeGv85YtSb+E7ixOlpsZp9GmxjGNlwGH4xXI8bCDQsbgsvJVlNnqI4pxEv1f86187d Ov6AS1E53MUUv0v6uzjiWHOWdzN/BozHGD4DFUdQn3QP79auPcZdq214Nz829tro+S6T sZgiJ2kGqzy1vCEKwuixjKApWzblZYLAiigVRqnd7aitAZ/LdbKsgwug2opXqQ4OdwXL ApGvZt5576ueOCEJSyciSvS4y4ImpKIFirje9+hq+Mv/gvnCUE4f7s3+N9xm13JFHvOu I6LA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=E3NTemlc1jph+TyQc1Rj/ZtDeTjGrm+NA1JQ3UZSjqA=; b=p7v5EtENBP0NSODBegAVen3Vx7tjpgHhIsFgeGY7WB7Suj7nyACsjomDZ6uzmcNJWe atociTtc/d+0Re7TWZOqT8HZkP+4l9p8+c5hKymkBktv2mF3AaZ7FP8FStY1y8n4bcjT S+zFx3xee+AYu7H95YOUbcOYG5Aoa4uyJEh3ZTjJWtgLb0WisyHJ98MiMioVVZ2WeHgq +McwB0L3lY3f/ogwYzwxls1mFsa6hUPLchXu2PSIktkkcZtPwGkdMLgFltKKXpnCXKiN Rq4zB/LbWZbVwvUqk1eIMztRCcOws4J52MdoFJaqBj8oRrFMip/n90Xbmy3f92drqGrS Sp8g== X-Gm-Message-State: AO0yUKXoMYgklbSfyeT5pshZ4dt2gxOu7MxA1vhXzjqobFztu0cVBFvC prZl+rXRXyxhepTpDi+HSOINapBk7VKx0poHBira3p20SA== X-Received: by 2002:ad4:580a:0:b0:539:57f0:fa5d with SMTP id dd10-20020ad4580a000000b0053957f0fa5dmr513752qvb.59.1675073123696; Mon, 30 Jan 2023 02:05:23 -0800 (PST) MIME-Version: 1.0 References: <20230120073106.5453-1-ubely@ilbers.de> In-Reply-To: From: "Roberto A. Foglietta" Date: Mon, 30 Jan 2023 11:04:47 +0100 Message-ID: Subject: Re: [PATCH v3 0/5] Improving apt cache To: "Moessbauer, Felix" Cc: "ubely@ilbers.de" , "isar-users@googlegroups.com" , "Kiszka, Jan" , "Schild, Henning" Content-Type: text/plain; charset="UTF-8" X-TUID: VapCABplNVcf On Mon, 30 Jan 2023 at 10:56, Roberto A. Foglietta wrote: > > On Mon, 30 Jan 2023 at 09:45, Moessbauer, Felix > wrote: > > > > On Fri, 2023-01-20 at 08:31 +0100, Uladzimir Bely wrote: > > > Currently, apt cache (e.g. `var/cache/apt/archives`) import and > > > export > > > functions are not optimal. Multiple files are copied from global > > > DL_DIR to package WORKDIR, increasing disk IO and space needed. > > > > > > Also, various chroots (bootstrap, buildchroot, sbuild chroot) include > > > their apt caches to sstate cache files. > > > > > > This patchset switches to hardlinks instead of copies and removes apt > > > cache from bootstrapped images ans sstate caches. > > > > I just saw that this pattern does NOT work in case the cache is on a > > different filesystem. This unfortunately is the case for all CI systems > > with locally mounted caches, as well as for kas-container builds with > > DL_DIR outside the KAS_WORK_DIR. > > The entire cache system needs to be reworked and this was pretty > clear. However, every step (hard link included) brings us nearer to > this conclusion and moreover it helps to reduce the building time (now > 11m29s on complete image) that allows us to perform more tests in less > and lesser time. I am currently on this and I have a good feeling that > I will reach a general solution soon. As you can imagine the cash > system is not an easy piece to rework due to its implications in many > different starting points it can have. So, build and rebuild are just > two basic cases but variations of the top or ISAR layers that cause a > partial rebuilding and breaks in building are many other cases. > Obviously the main idea is to reach a reasonable goal AND then break > it in several small steps that can be integrated upstream because a > single large patch would be equivalent to fork another 'next' branch > which is possible but IHMO not acceptable for a long time. After all, > also the GPLv3 on composition is not a long-term way to go. So, in the > best case it makes sense that we reach the common goal to rationalise > and speed-up the cache subsystem. > apt-cache and sstate-cache are two different subsystems: yes they are but they are not separated much enough to allow a general straightforward solution for apt-cache to work but just in dpkg.class. Best regards, R-