From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 7067480740990025728 X-Received: by 2002:a5d:5985:0:b0:1f0:46a7:bb74 with SMTP id n5-20020a5d5985000000b001f046a7bb74mr8115590wri.699.1646388184006; Fri, 04 Mar 2022 02:03:04 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 2002:adf:d0c5:0:b0:1f0:7735:e337 with SMTP id z5-20020adfd0c5000000b001f07735e337ls50254wrh.1.gmail; Fri, 04 Mar 2022 02:03:03 -0800 (PST) X-Google-Smtp-Source: ABdhPJzShSZgEztpDpgcQUETq5R5MX7DtqC4t3R6zVqDA6MGsl1d3N6P2aUgJkbvhwfoSWdWk5UZ X-Received: by 2002:adf:bc09:0:b0:1f0:2483:48b with SMTP id s9-20020adfbc09000000b001f02483048bmr10192453wrg.118.1646388183055; Fri, 04 Mar 2022 02:03:03 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1646388183; cv=none; d=google.com; s=arc-20160816; b=FzamC9znnEoGJY7Ni7GzU/T1l/2cDtWKfZ1Ou/6N7VCP/aRmgXEyPSTTUgTHNRSHbQ XtLhHKMWlMZTLJRrpOP+PPn04xUSDQCNeTMPPUvxY4ENz8UO4C79bDO4Mqs4YSeu/yim CZ1G3B4lEnVoQigNXclVsRYt9c88+AUkhxKwcwlWvc0I9TSJ0T84WsJ52QNHj9Mg+mqd A8vXqrNkrp/rn59ZYzFP86dqcRYaTpV3QmLzyred/AtHudeSSlDJG4+nYVT6yEPoAuJA /2Pkz8VAYnbfE30Y0xCPQJtvowFojn/xqNW7DNtRGCJrWthPEqYL1PtEgjdnWtPkDLU1 LSGQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=in-reply-to:content-disposition:mime-version:references :mail-followup-to:message-id:subject:to:from:date; bh=5KhNG84LW+lQHfx6bxKEORTySnMdv2N2x2ABlyGDNcU=; b=DBxoAkZRcssKIiKfq699oYJ7o1du3ghpu8WlglJXw3D/PpFRYdXWwTV4h5ZzGFMqgf zgRR7dpbOpFvJLAn9mRjGP2ZvpoChsHK+Jrvugud+ZeueWKRZmt+k0LEsm3/swHID2Bx +B6MMb4PknIMu55g7zn/dQfPWhBp5J0lhyPcpJJv2ifewhRMF4yliR5X0zXPrlC+Dbdb VfNUE9wHAv2pos9hOcNDYAfc9K/BhfT/KEI98bDfase6zw188+y+7PQJB/nwtxjS2+bj JPWTbUOhS5OtSPxg8ZjF6FWooP31GM2dCTHZX2kNVChQQi8r+DUu+WqIfuODA2WZNdHw GxLQ== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=neutral (google.com: 85.214.156.166 is neither permitted nor denied by best guess record for domain of ibr@radix50.net) smtp.mailfrom=ibr@radix50.net Return-Path: Received: from shymkent.ilbers.de (shymkent.ilbers.de. [85.214.156.166]) by gmr-mx.google.com with ESMTPS id 189-20020a1c02c6000000b00387e237f125si118506wmc.3.2022.03.04.02.03.02 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Fri, 04 Mar 2022 02:03:02 -0800 (PST) Received-SPF: neutral (google.com: 85.214.156.166 is neither permitted nor denied by best guess record for domain of ibr@radix50.net) client-ip=85.214.156.166; Authentication-Results: gmr-mx.google.com; spf=neutral (google.com: 85.214.156.166 is neither permitted nor denied by best guess record for domain of ibr@radix50.net) smtp.mailfrom=ibr@radix50.net Received: from ilbers.de (host-80-81-17-52.static.customer.m-online.net [80.81.17.52]) (authenticated bits=0) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8) with ESMTPSA id 224A30Ss032456 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Fri, 4 Mar 2022 11:03:02 +0100 Date: Fri, 4 Mar 2022 11:03:00 +0100 From: Baurzhan Ismagulov To: isar-users Subject: Re: [Discussion]: Metadata to consolidate and rebuild base-apt from distributed CI builds Message-ID: Mail-Followup-To: isar-users References: <20220222153136.08432cb3@md1za8fc.ad001.siemens.net> <20220224164244.6e4bb002@md1za8fc.ad001.siemens.net> <45e6e9ba-8a05-e617-d5ae-949efb53c35b@siemens.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-TUID: Tx96ou4aeO9K On Thu, Mar 03, 2022 at 07:15:40PM +0530, vijai kumar wrote: > If we are in agreement then we can think about how to achieve this. > There are changes coming in soon, so the implementation should take > that into consideration. > > I am not sure if the caching part is reworked. If so having an idea on > the design would definitely help; Thanks Vijai for the discussion. In short, we've already started further base-apt improvement due to a number of reasons, e.g.: * Strict usage of base-apt for debootstrap and build-dep to ensure base-apt correctness in any build. * Pluggability of debootstrap, which is necessary for multistrapping, sudo removal, and maintainability. * We need to know which PN-PV is satisfiable from which location (base-apt, isar-apt, bitbake) in order to use Debian Build-Depends in bitbake. python-apt provides the necessary functionality. After we have the above, more necessary use cases become possible. E.g., storing and reusing built packages in per-layer apt repos. We also want to have parallel building. For us, it comes more from the CI side, as we have 3 h for fast and 10 h for full testsuite on the latest inexpensive hardware. The first step would be to parallelize the testcases with storing of intermediate results in a shared location. The second step would be extending that to individual bitbake tasks. Maybe icecc would be good enough to cover either or both, we have to test. Regarding your implementation proposal, I think that could be done. However, I'd like to better understand the motivation first. Is it e.g. creating a canonical repo for a given project? That would be easier to implement on top of the above. Regarding downloading time -- we had tested full local Debian mirrors and didn't see any performance improvement of CI jobs. We haven't dug deeper, maybe we have some parallelization killers in Isar. Regarding the central repo for remote building sites -- in my experience, it is very slow, our customers end up installing local replication servers. We aim at full Debian support, be it packages, repos, or images. Debian, being a binary server / desktop distribution and not a source-based development kit, has a number of inflexibilities such as sudo, versioning, rules, etc.; we would like to work towards more developer friendliness here. Bitbake and Yocto contribute much here, and we would like to find a good working solution. That is why we welcome this use case and would like to work on that after understanding the details. Jan told me you already had some implementations for this. You also mention time and costs. Could you please share the concept behind the work so far, and which time and costs you mean? Then we could proceed step by step while having the big picture in mind. With kind regards, Baurzhan.