From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 7067480740990025728 X-Received: by 2002:a05:600c:4307:b0:389:4f8f:f189 with SMTP id p7-20020a05600c430700b003894f8ff189mr942559wme.29.1646315152539; Thu, 03 Mar 2022 05:45:52 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a1c:2bc2:0:b0:386:8897:8f1 with SMTP id r185-20020a1c2bc2000000b00386889708f1ls1727342wmr.2.canary-gmail; Thu, 03 Mar 2022 05:45:51 -0800 (PST) X-Google-Smtp-Source: ABdhPJzU71118jhURzKEoOzxH6uEp8t8rEpxAlQinKYtfOvGnmLVCFfNVsvmSE9QrQewq2f2YOD8 X-Received: by 2002:a7b:ca57:0:b0:381:570c:ed95 with SMTP id m23-20020a7bca57000000b00381570ced95mr566954wml.169.1646315151505; Thu, 03 Mar 2022 05:45:51 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1646315151; cv=none; d=google.com; s=arc-20160816; b=XpCusK2vPd5CFDbx9bkTIEIC4rqxTW90fSzjStmD7FHilYqbNxoZJD/Wd4oWsGYb+k rHtyqXDHwV0DiVjx4AQ6etIAp01q+0Dr3niWQgr6BVzNIfGoPOebEDyuI9EUYeTPbvWG OWSHUhSAB4/SWuwhFTReaqFNJkOjmvgInNMVVDFZFsUpOvY714ScKP1T0dTzZ33ywuz7 K1uMzgCB3broBYGmu/IkqoeFF3MsYmU3GGTqx6BRP6BxFEOp2CFIglKM4L5K+/rjhUZM qFfz+/sk+leWd1lX/mMBeoTrHhzWFN8gMfnifw+AMf3Tyx63cAVlBFLazWLoBgA+jw6h lsfQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:dkim-signature; bh=NwaoqhbhHBtCYM9bbdFnA11VVimAlvhA2DCmaiOsYfc=; b=k4aUVUoSdPxg/gI5G2ulR+v/A0ZWBBx3kyV4xFBtGH1QIVL/JrSKSHUlpu39CxdhHw iiseUsh+xnQfVHfP6GDjuilEWwWEf1ApK1HrJLEjSQrnXk+zlMqP0F4dMwXnK1ikrNZq nYe4XX7ZfJkwDmv/jVM+j9OYpLUvi0YYLdWFCm7L8ZMOT2D2PsAhIo7vkAuCSvaA7Q4d dI9XVSgGVhGAunzHtUWtkyDhFSvE/oYNgh0ajfhtj/0etMgjg3ryqyMrXCiRWaLuevqp ayagFNTXbazHl6kyb1DreIhM7cDDyiEfBn5gDo8jAHCOS69RAv816/Pwtvqxohj/hXVm 2HJA== ARC-Authentication-Results: i=1; gmr-mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b=W92A+JhZ; spf=pass (google.com: domain of vijaikumar.kanagarajan@gmail.com designates 2a00:1450:4864:20::435 as permitted sender) smtp.mailfrom=vijaikumar.kanagarajan@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from mail-wr1-x435.google.com (mail-wr1-x435.google.com. [2a00:1450:4864:20::435]) by gmr-mx.google.com with ESMTPS id a13-20020adfed0d000000b001f0626bf294si57705wro.5.2022.03.03.05.45.51 for (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Thu, 03 Mar 2022 05:45:51 -0800 (PST) Received-SPF: pass (google.com: domain of vijaikumar.kanagarajan@gmail.com designates 2a00:1450:4864:20::435 as permitted sender) client-ip=2a00:1450:4864:20::435; Authentication-Results: gmr-mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b=W92A+JhZ; spf=pass (google.com: domain of vijaikumar.kanagarajan@gmail.com designates 2a00:1450:4864:20::435 as permitted sender) smtp.mailfrom=vijaikumar.kanagarajan@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: by mail-wr1-x435.google.com with SMTP id u1so7898879wrg.11 for ; Thu, 03 Mar 2022 05:45:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=NwaoqhbhHBtCYM9bbdFnA11VVimAlvhA2DCmaiOsYfc=; b=W92A+JhZfYYwlw1oT4LtKClOxWdKmIQOJTGebZyXfzjzwG41TLdauvnD9vbaPM1wol 15y9jmhm7TDh0iqqWBhvk6vMqUEDt/lPBWmWQyj3wT8pOuhJQXbtPyyKuFoEGAvS1/cW FNctzUVC+WQqWCEaZmi7jzUGkJGQwnDGovsyEkitOpL9gQ6xML0zRcZ5Vy8QVWZd7M6P c2wzRGfzVHijGiUUix8OH5P4O2HF927sD7EwXPsAnMfMKEBHCGB0xYhMqbp0dlmWpj7s HmKJNExv3oJWb/lFRBl+aVrihImAd4oOWJ9P4vsOTomdj/J4sck9mJrk6DgK88/LVJGZ D/Cg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=NwaoqhbhHBtCYM9bbdFnA11VVimAlvhA2DCmaiOsYfc=; b=i+QvpVUwJnOy5OoF5p0aEFJlWkmncteRYTDO5B5OVarELBNgBqMPZrZdaWk84YAtaE TyvYoj/JAGQN6Lq9wcbxOE/NH2m469TMGRypA9OXMZ2kSqfQIpOx5LNZRXRrodqSU92V r/FflFAKp0AGtBIbGiaRZpvpxqrC2pS/sG7ow2mZfdPpb2IP7Rck1MwZqotXNVFBp5fq 7zBUfen68vcLYcwMJfPFq05NcYCZMVjGbglCfb8JjcKVafvo0Xblg53YR+5JTnh++2yJ LP/tCvXP4dg75KvjyGcMgjfTS33eQbMwxQt5eX0XsdUVRcAsBEC3ap3agI9rtoHDeWMg buCQ== X-Gm-Message-State: AOAM531TOxL7lYZ+LQR+9KDkuwP/orqEON612PsZslWBc6RUKnJ9W/f7 +09jhCG3WTVpS3IVmlL3JkGhtA0+IdNLTsDudWo= X-Received: by 2002:a05:6000:184c:b0:1ef:f8a2:3283 with SMTP id c12-20020a056000184c00b001eff8a23283mr12617207wri.467.1646315150906; Thu, 03 Mar 2022 05:45:50 -0800 (PST) MIME-Version: 1.0 References: <20220222153136.08432cb3@md1za8fc.ad001.siemens.net> <20220224164244.6e4bb002@md1za8fc.ad001.siemens.net> <45e6e9ba-8a05-e617-d5ae-949efb53c35b@siemens.com> In-Reply-To: <45e6e9ba-8a05-e617-d5ae-949efb53c35b@siemens.com> From: vijai kumar Date: Thu, 3 Mar 2022 19:15:40 +0530 Message-ID: Subject: Re: [Discussion]: Metadata to consolidate and rebuild base-apt from distributed CI builds To: Jan Kiszka Cc: Henning Schild , isar-users , Baurzhan Ismagulov Content-Type: text/plain; charset="UTF-8" X-TUID: +HVLZ2vHpJXL On Fri, Feb 25, 2022 at 10:57 PM Jan Kiszka wrote: > > On 24.02.22 16:42, Henning Schild wrote: > > Am Thu, 24 Feb 2022 18:50:50 +0530 > > schrieb vijai kumar : > > > >> Hi Henning, > >> > >> On Tue, Feb 22, 2022 at 8:01 PM Henning Schild > >> wrote: > >>> > >>> Hey Vijai, > >>> > >>> Am Tue, 22 Feb 2022 16:04:36 +0530 > >>> schrieb vijai kumar : > >>> > >>>> Problem: > >>>> -------- > >>>> We could have several CI jobs that are running in parallel in > >>>> different nodes. One might want to consolidate and build a > >>>> base-apt from the debs/deb-srcs of all these builds. > >>> > >>> Can you go into more detail. I do not yet get the problem. > >> > >> runner 1(Germany) -> Building de0 nano > >> runner 2(India) -> Building qemuarm > >> runner 3(US) -> Building qemuamd64 > >> > >> > >> All these builds are running in different servers. > >> If we wanted to create a single base-apt from all these servers, then > >> we need to copy over their deb/debsrcs/base-apt to a common server and > >> then > >> create a consolidated repo. > > > > But why would you want to do that? I mean i get why you would want to > > store all in the same location, but not why it should be one repo. > > Maybe to save some space on sources and arch all .. but hey there are > > ways of deduplcating on filesystem or block level. > > You are just risking a weird local "all" package not being so "all" > > after all ... false sharing. > > We want to auto-build a single, "offline" capable repo from the BoM > accumulated from those builds of all possible targets. And that in a way > that does not require pushing large artifacts between the build stages, > ideally only those BoM lists. If we are in agreement then we can think about how to achieve this. There are changes coming in soon, so the implementation should take that into consideration. I am not sure if the caching part is reworked. If so having an idea on the design would definitely help; Maybe ISAR maintainers can clarify on this. Thanks, Vijai Kumar K > > > > >> This involves moving around this data. > > > > Yes, if it one central storage place. No matter if it is one "repo" or > > many "repos" in i.e. folders. > > > >> The problem can be avoided if we have a single metadata produced by > >> all these builds which would have details of all the packages the > >> build used. > >> Basically a manifest of the build. This manifest can be later used to > >> recreate the repo which can be hosted later on for these jobs. > > > > We have a manifest for "image content" which already is fed into > > clearing, it is a bill of materials an nothing else, it can not > > be used to rebuild. > > Even if you had all metadata you need to store sources and binaries > > somewhere reliable, whether that is central or distributed is another > > story. > > Pointers to anything on the internet (including all debian repos) will > > at some point stop working. So if "exact rebuilding" in a "far away > > future" is what you want, mirroring is what you will need. > > Exactly, this mirror is supposed to be generated, and that shortly after > the individual builds succeeded (in a common pipeline stage). That can > fail as any build can fail if a referenced version picked up during > bootstrap got dropped while building an image. > > > Partial mirroring based on base-apt even with sources will be shaky and > > you will find yourself digging in snapshots again. But it will work. > > Yes, it works for us (you should know ;)). > > Jan > > -- > Siemens AG, Technology > Competence Center Embedded Linux