From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 7067480740990025728 X-Received: by 2002:a5d:5746:0:b0:1ea:9643:92ac with SMTP id q6-20020a5d5746000000b001ea964392acmr19870304wrw.597.1647344724427; Tue, 15 Mar 2022 04:45:24 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a1c:2747:0:b0:381:80e8:be59 with SMTP id n68-20020a1c2747000000b0038180e8be59ls1159571wmn.1.gmail; Tue, 15 Mar 2022 04:45:23 -0700 (PDT) X-Google-Smtp-Source: ABdhPJwQ1V7T8tM0CF93KKY18oTY2fNh6+Ewg8s/tN3Jkh49hXXXOyIlt2FcXfZeVD9nAXrzBsHh X-Received: by 2002:a05:600c:26c8:b0:389:a542:c20b with SMTP id 8-20020a05600c26c800b00389a542c20bmr3024055wmv.46.1647344723487; Tue, 15 Mar 2022 04:45:23 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1647344723; cv=none; d=google.com; s=arc-20160816; b=sXJRbu8C30UNYh556dnmieF91coQcJlcREExpScG8fTlN+NV7eCy1w2l39wmGLmVUh PwKOUgoy2ZtjzYwX2rmIMySIn+cLnyvrBqitBj0BsPZUpytWvGbLnj2XukWihUL6q0lU roGNokLYzU76Slw5eTMdlmvtZwp7alJdwQ/716FSox+WurEqIgRSQDpsbnbYaT0BDzR/ xtfwK5syOtaFus81qlOayUVTaTYaw3muvMQ/EhEbbrdP69Q6hUZCGeYtwu9bjy/qWp1O YsHLvoTAPRql1quTCkIpmuKiDwOK21mri2CDroUq6dLo9lhf1f6kRoklA+m4RTGl4s3Z 4BUQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=in-reply-to:content-disposition:mime-version:references :mail-followup-to:message-id:subject:to:from:date; bh=fUnSeX00fLM+0rpgWktYpb8k6Uzb2JsGKkRQrzztCmY=; b=HgKTtShctnqsz9gR5pf7Zpc4r9yPOk7hB6XA0NBZbk13+nwLIqpi2EpQ1nTXqcJ1sW /LhqcBjoVYkJOTd/w0XahcWW+iuk/dw9R6MKskJ+dN0LKEsjRNg4o2o7ubNNBMQESbsp HnPnbenbLCAodtZoJPgY73fAOGk4l1ariO+zkzjrbSDK8n7TJs0qpikOoWezC4tlnMsh qM9R0AqcKRyw/IlSqeBES3ojIzUkPLJqeV4PtUdRU2Iop0CnFs2j0w9cnzgZVl92hgL5 6Br807ArveYMntlYe4dzC28GgYdE9ohJ/ku9wHLqCy4AQY+ITw6H0FsbrssBGEn6qDb3 vRpg== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=neutral (google.com: 85.214.156.166 is neither permitted nor denied by best guess record for domain of ibr@radix50.net) smtp.mailfrom=ibr@radix50.net Return-Path: Received: from shymkent.ilbers.de (shymkent.ilbers.de. [85.214.156.166]) by gmr-mx.google.com with ESMTPS id az24-20020a05600c601800b00389a1a68b71si127150wmb.0.2022.03.15.04.45.23 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Tue, 15 Mar 2022 04:45:23 -0700 (PDT) Received-SPF: neutral (google.com: 85.214.156.166 is neither permitted nor denied by best guess record for domain of ibr@radix50.net) client-ip=85.214.156.166; Authentication-Results: gmr-mx.google.com; spf=neutral (google.com: 85.214.156.166 is neither permitted nor denied by best guess record for domain of ibr@radix50.net) smtp.mailfrom=ibr@radix50.net Received: from ilbers.de (host-80-81-17-52.static.customer.m-online.net [80.81.17.52]) (authenticated bits=0) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8) with ESMTPSA id 22FBjLoT017451 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Tue, 15 Mar 2022 12:45:22 +0100 Date: Tue, 15 Mar 2022 12:45:21 +0100 From: Baurzhan Ismagulov To: isar-users@googlegroups.com Subject: Re: [Discussion]: Metadata to consolidate and rebuild base-apt from distributed CI builds Message-ID: Mail-Followup-To: isar-users@googlegroups.com References: <20220222153136.08432cb3@md1za8fc.ad001.siemens.net> <20220224164244.6e4bb002@md1za8fc.ad001.siemens.net> <45e6e9ba-8a05-e617-d5ae-949efb53c35b@siemens.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-TUID: z2tFEoVcv9MW On Mon, Mar 07, 2022 at 12:53:26PM +0530, vijai kumar wrote: > If there is already a branch for this activity with some initial > implementations, can you please point to it? v2: https://groups.google.com/g/isar-users/c/65lRtw4EU_8/m/_O2hIRPBAgAJ v3 WIP: https://github.com/ilbers/isar/tree/baseapt_v3/10 > It is for recreating a repo for a given project from some kind of > manifest. This way we could > avoid pushing the repos between multiple CI runners. > > The current goal is to create a single repo from multiple projects. We > might have multiple projects running parallel in different CI runners, > the idea is to create a single repo from all those builds without the > need to push data around. So, some kind of project manifest. > > These manifests then can be used to create a single repo. Instead of > copying over all the debs to a single location and trigger creation of > base-apt. Ok, so it's about creating One Canonical Base-Apt (for a project / department / business unit / company). > We thought about a few options, > 1. Gather the download urls for all the packages we download. This > could be our metadata. > 2. Club all manifests at the end of build. (Buildchroot, > isar-bootstrap & image rootfs) to recreate a master list of packages > used in the build. > > 2 has some disadvantages. We have to probe buildchroot after image > build to get a complete package list. Even then it doesnot capture the > packages that are removed. > 1 seems like a solution, It would have to be injected as part of the > build, like how we injected downloading debs. The advantage it brings > is that we don't necessarily need the apt sources information to > recreate the repo. A simple wget would do. There is also a risk of > urls becoming obsolete. > > There could be better solutions, maybe our new way of creating > base-apt might help in creating metadata in a cleaner way. I agree that post-build collection has some limitations -- that was the motivation for us to make a small step towards a more Debian-like repo management. The patchset implements the approach #1. We use python-apt to determine what we need upfront. The debootstrap part works in v2. Build-deps part is about to be finished in v3. We download immediately, but updating to output only is easy -- that is in fact one of our requirements. We don't address package removal and recursive fetching (rebuilding the whole base-apt exclusively from local files) in this step. Implementing https://wiki.debian.org/HelmutGrohne/rebootstrap with Isar would be cool (and we need at least parts of that logic for certain use cases), but we have to see whether we need other stuff like Build-Depends support before that. With kind regards, Baurzhan.