From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 7067480740990025728 X-Received: by 2002:a2e:9206:0:b0:244:c5e5:977c with SMTP id k6-20020a2e9206000000b00244c5e5977cmr1796561ljg.339.1645708862698; Thu, 24 Feb 2022 05:21:02 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a05:6512:3b8e:b0:443:9610:6a0c with SMTP id g14-20020a0565123b8e00b0044396106a0cls304950lfv.1.gmail; Thu, 24 Feb 2022 05:21:01 -0800 (PST) X-Google-Smtp-Source: ABdhPJwhH3oUS/NRAccOl8AZeF90ssQbMw6/Ji3HvXanYDje8RpzLIsZXx1yjKKtNX048hg+zvNp X-Received: by 2002:ac2:5313:0:b0:443:99c1:7e89 with SMTP id c19-20020ac25313000000b0044399c17e89mr1695830lfh.531.1645708861530; Thu, 24 Feb 2022 05:21:01 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1645708861; cv=none; d=google.com; s=arc-20160816; b=tRHQ6ePgt0q7uMdWnPn4CDnZOh1wzTVpaq6SNt276YG10VgTdt0oTenfuvC8uxfUkS kRdgopYZtDqQDLbfzGFaO3EFtUqIyUjOnLrdWyhl67wlehtO0KBZpzj+6QTAtUfWXpO8 1lksUb/bL6J6lDsPgmvCJhyCjDRKIbX08ZnlI/aXeQCd3zEK8MTrEsGbv0oaVOKF9bjN uKd5H7mIFPYg0GPz77HwQs/uV3nuLjWJsRovMc97jla9SA7Itm9q3uXUOojCV+j2eK27 PfsL8z/CGCNQayeVs22RYruGod3GDcqsl/xm52H4WjmQRE4ea3jicsKpry6sphylgdsk 5Wog== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:dkim-signature; bh=DrveiD5kY+WwN1UVTnNn4lgbkCH6yMywcQ0kygeCiqQ=; b=B5YzG+QMJyVzVZf8e4STj5F07O4lo9oyE6UYKfFeImPbOW0CXpOitLKL1Zxc1IgTya +IHNwUicN+BKkHy1vWEuctdRaIt1g6g8LkB7FN2hY7JZMAReuiYvVY2r5N5Jbzbi+1se YJQfi5mvTqhI1t+1RdgUR8+qYo7iTDKSzSoZoziT1fPIweWUKV4almD1rnkBOfiThbJv bCg8MbVXnE2D9KDwBo01/wPd8xX4Od2NVeOsd6tcrbazhoSuXMtLJZVKACBYRjyoHHvp IAC4tdcOo3DSgF9lqaWil2Rp8LK1W8/mAqwUbv1xFImWCV81ZGP7s0XtCiceF4Pn0JEg dHjg== ARC-Authentication-Results: i=1; gmr-mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b=PnQCDHXl; spf=pass (google.com: domain of vijaikumar.kanagarajan@gmail.com designates 2a00:1450:4864:20::432 as permitted sender) smtp.mailfrom=vijaikumar.kanagarajan@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from mail-wr1-x432.google.com (mail-wr1-x432.google.com. [2a00:1450:4864:20::432]) by gmr-mx.google.com with ESMTPS id d35si94482lfv.5.2022.02.24.05.21.01 for (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Thu, 24 Feb 2022 05:21:01 -0800 (PST) Received-SPF: pass (google.com: domain of vijaikumar.kanagarajan@gmail.com designates 2a00:1450:4864:20::432 as permitted sender) client-ip=2a00:1450:4864:20::432; Authentication-Results: gmr-mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b=PnQCDHXl; spf=pass (google.com: domain of vijaikumar.kanagarajan@gmail.com designates 2a00:1450:4864:20::432 as permitted sender) smtp.mailfrom=vijaikumar.kanagarajan@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: by mail-wr1-x432.google.com with SMTP id j22so2968088wrb.13 for ; Thu, 24 Feb 2022 05:21:01 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=DrveiD5kY+WwN1UVTnNn4lgbkCH6yMywcQ0kygeCiqQ=; b=PnQCDHXlh6Yemp5q21UQ8zauSBTeJnI6hEwLds5hE7P/3FV1kpdjXPPdsD7kG9+22X 2DhPS3NHdeUH9klpGzQpWPq7Lgp7rDqQcQtHbPdDGQhlNhMPkigLDlAFAENUnI9g9Bfn TvDoiElR9KW62sloGSqkiybYfpn9j3Mj7RvbKEPpESPr9WPEXNl15N62dmlR/kYkvXEP FEonGCCJd0o6g3iKaSMOWmiGiV6w2W9dVo+WA433WRcniRqgN3QuhRrW4oLtwwZYijrz vgri2k+qLL5XvTXKKi8ZdlZgW5QE7APg1HOytZP4G5XVegiyKYbXQaVPQnu0CIbBfl6u ajTw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=DrveiD5kY+WwN1UVTnNn4lgbkCH6yMywcQ0kygeCiqQ=; b=eoCeIVkfeltISuIRNKvLDvbQJ5F/NdAqChRP8taCLkGm1RdE5NlC84zDtweiZNm7Op L8cLpbdLiJ0sRjZcZqkIWsrFqCc+hBew3EzLbgpPtkEEoD07lCQBYu2UbtuDT7Uq7I3e z1XJJ/IGHtdAjSiABZjfWvh1EAMCPWl/D5P5Xlq5yY34fikZFszbPFV9Nb++nNf4cdlb tgRu4X7aiAF95dSM59l2PNm3zJT4g5VJwxmCr0c6jqocPM3QRc0yCUwGTXdqep/B6CyD YvufN6ME0/Nd4+ma1PE04UFYmEy9hva1v4fZp1dHmPA/ubvZQsBLUGUMGSdvjhLmHmrB Jblw== X-Gm-Message-State: AOAM530BglgcLRhrF6q7Bk6xLan0S0fpcAw+xmdkxIVYUYhjTipEoHxh NR8tr/ErAuKjmbpnTKpdq5potKXD+h+IRfK5tdnp3lJof00= X-Received: by 2002:adf:9103:0:b0:1e5:8cc9:5aa1 with SMTP id j3-20020adf9103000000b001e58cc95aa1mr2138953wrj.17.1645708860902; Thu, 24 Feb 2022 05:21:00 -0800 (PST) MIME-Version: 1.0 References: <20220222153136.08432cb3@md1za8fc.ad001.siemens.net> In-Reply-To: <20220222153136.08432cb3@md1za8fc.ad001.siemens.net> From: vijai kumar Date: Thu, 24 Feb 2022 18:50:50 +0530 Message-ID: Subject: Re: [Discussion]: Metadata to consolidate and rebuild base-apt from distributed CI builds To: Henning Schild Cc: isar-users , Baurzhan Ismagulov , Jan Kiszka Content-Type: text/plain; charset="UTF-8" X-TUID: Ef+n3Xx4NigS Hi Henning, On Tue, Feb 22, 2022 at 8:01 PM Henning Schild wrote: > > Hey Vijai, > > Am Tue, 22 Feb 2022 16:04:36 +0530 > schrieb vijai kumar : > > > Problem: > > -------- > > We could have several CI jobs that are running in parallel in > > different nodes. One might want to consolidate and build a base-apt > > from the debs/deb-srcs of all these builds. > > Can you go into more detail. I do not yet get the problem. runner 1(Germany) -> Building de0 nano runner 2(India) -> Building qemuarm runner 3(US) -> Building qemuamd64 All these builds are running in different servers. If we wanted to create a single base-apt from all these servers, then we need to copy over their deb/debsrcs/base-apt to a common server and then create a consolidated repo. This involves moving around this data. The problem can be avoided if we have a single metadata produced by all these builds which would have details of all the packages the build used. Basically a manifest of the build. This manifest can be later used to recreate the repo which can be hosted later on for these jobs. Having metadata and recreating repo is one way. There might be other ways as well. That is where we thought about the --print-uris option of apt. It basically gives you the complete URL to the package which we can download using wget. A manifest containing all the packages ever used by the build with its complete url. It could easily be used for several purposes, like as clearing input, repo regeneration etc. I don't think sstate can help here. I might be wrong though. Thanks, Vijai Kumar K > > It seems like you want to save compute time by sharing pre-built > artifacts via some common storage. The sstate can do that very well, we > are using shared folders for on-prem runners, s3 for AWS and sstate > mirrors for population of "new empty runners" and "partial result > delivery" of failed jobs and to sync on-prem with s3. > > isar is a tool to build images, not distros or repos or packages. While > it can do all of that using it for such things can get tricky and isar > was not designed for such cases. Meaning "base-apt" is not meant to be > your cache to build many images from ... it is meant to be the cache > for exactly one ... and sharing can cause problems. > > sstate would detect false sharing, say a package recipe for some reason > uses a machine-conf variable. multiconfig or base-apt sharing would > make you run into that bug, while sstate would likely not. > > So if it is about build time i suggest you have a look at sstate and the > not yet upstreamed python helper scripts for sharing/eviction i can > point you to in case you do not find it yourself. > > Henning > > > What's possible: > > --------------- > > With the current state of ISAR, the below is possible. > > > > 1. Run all the jobs in parallel in separate CI runners > > 2. Collect all the debs and deb-srcs from those builds and push to a > > common file server. > > 3. Download the debs and deb-srcs and create a repo out of it in the > > final CI step, > > 4. Upload the base-apt to the server. > > > > This has some disadvantages, we need to move all those > > data(deb/debsrcs), this increases time and cost. > > > > What's needed: > > -------------- > > The idea is to have a simple meta-data that can be used by repo > > generation tools to recreate the repo. > > > > Why manifest cannot be used: > > ---------------------------- > > Manifest does not serve this particular need. Below are the > > shortcomings of image manifest, > > 1. Does not have details about removed packages(eg localepurge) > > 2. Manifest of buildchroot would not have details about the package > > dependencies/imager installs at the time of generation(i.e. > > postprocess) > > > > Some ideas: > > ----------- > > There were a couple of ideas, > > 1. To use an external script to create a manifest of the > > downloads/{deb, debsrc} folder and try to download the packages using > > that manifest and appropriate sourceslist in the final runner. > > 2. To use "apt --print-uris" + "debootstrap --keep-debootstrap-dir" to > > create a metadata with complete url to the package. Later wget can be > > used to download those from the web. > > > > We are wondering if we could discuss and derive a solution for this > > here in ISAR itself instead of opting for some local scripts in > > downstream layers. > > > > Thanks, > > Vijai Kumar K > > -- > You received this message because you are subscribed to the Google Groups "isar-users" group. > To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com. > To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/20220222153136.08432cb3%40md1za8fc.ad001.siemens.net.