From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 7067480740990025728 X-Received: by 2002:a05:6512:23a5:b0:443:15cc:d7d4 with SMTP id c37-20020a05651223a500b0044315ccd7d4mr16734727lfv.562.1645526089258; Tue, 22 Feb 2022 02:34:49 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a05:651c:1a10:b0:246:4772:295c with SMTP id by16-20020a05651c1a1000b002464772295cls575663ljb.10.gmail; Tue, 22 Feb 2022 02:34:48 -0800 (PST) X-Google-Smtp-Source: ABdhPJx9vyw6ZFf0OX5aJFgHWsrgM/R8K79GPgChLqjPnV5UhDPlOPwiowKn/J01yMwfdk8fAly2 X-Received: by 2002:a2e:2a44:0:b0:243:6b73:1c0 with SMTP id q65-20020a2e2a44000000b002436b7301c0mr17332324ljq.376.1645526088194; Tue, 22 Feb 2022 02:34:48 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1645526088; cv=none; d=google.com; s=arc-20160816; b=pm8jxXUdL0OBf9ldJJa2OXmQwpKwl5y6IEHbqy8v4RRRfMcmFYMf453jeL5NASDfYJ IqzsPaxzY6z0aL1peyO1sHry9vmE347D4vp4IltNko1WG1MJUMcQesFg0YDNSkoeZwD/ Wt/TTXy3JVDDZhpNQolcDWkcuycqh8ydwTug9lbeq6xbY53/w/ow96fYv705macEfo5H xXC5sZDZb5zh5B51pRRV7OKfFSU5GOroCNr4D1GCVEh+WYZ2BQTQygYIw/tZ5FK+2Utq BSVQWIRpuHoYLHhRX04+0NKeAglDCD08g47s2pL3vZHrADQ2d98P1oquG7HGPge0E8iQ kY0Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=cc:to:subject:message-id:date:from:mime-version:dkim-signature; bh=aFrZCrVosvuUcXnGez2UkNofl47rIvd6rbrCl/vJsnY=; b=hj6LaTirXh093DdPwl9+wvftWrt//c/0AG8geUsBkW7XeyRk3Bfw4omJwgMIO/Qsr1 Cf3f3INQzFw2B2ngy0+JTiYQm8EWY6srRD8tL7YdDEm/D4EicTZ+5fmObjC8CncOSGTj 1OHnK9OkqdSTC8D/AY6Bv8cuXaEbYrXCZG1JT+iITFnhAAOMCQtYzPtaGEzwreu3L85v PIFHJVDZ3PkTsP/Bj870ZWGN1gHSlZXjPEcGBMe9iQGTYma1PrNYvDogXxQ+gM9g6O9N Uimql2OiiwGWU+be/JzBMqDUpoLZno+x7qGeQv7FF1vPUCkTtYRVk5WpoBmAdCyM01Gl iTzw== ARC-Authentication-Results: i=1; gmr-mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b=WgYhhlUI; spf=pass (google.com: domain of vijaikumar.kanagarajan@gmail.com designates 2a00:1450:4864:20::42e as permitted sender) smtp.mailfrom=vijaikumar.kanagarajan@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from mail-wr1-x42e.google.com (mail-wr1-x42e.google.com. [2a00:1450:4864:20::42e]) by gmr-mx.google.com with ESMTPS id n5si604010lfq.6.2022.02.22.02.34.48 for (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 22 Feb 2022 02:34:48 -0800 (PST) Received-SPF: pass (google.com: domain of vijaikumar.kanagarajan@gmail.com designates 2a00:1450:4864:20::42e as permitted sender) client-ip=2a00:1450:4864:20::42e; Authentication-Results: gmr-mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b=WgYhhlUI; spf=pass (google.com: domain of vijaikumar.kanagarajan@gmail.com designates 2a00:1450:4864:20::42e as permitted sender) smtp.mailfrom=vijaikumar.kanagarajan@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: by mail-wr1-x42e.google.com with SMTP id j22so6144983wrb.13 for ; Tue, 22 Feb 2022 02:34:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=mime-version:from:date:message-id:subject:to:cc; bh=aFrZCrVosvuUcXnGez2UkNofl47rIvd6rbrCl/vJsnY=; b=WgYhhlUIqibuj3gJWmyp8fAvpeCV567a8/CiKW1dJqawGZ7L2tRiC32e0NQ1zYyEG+ /Ha2yYLFmwUXlDW3nQMuA/e3D9p108CKHzYYfkR44ql6o9FO2gIGrZnSL5LplZEPy1ri MUGkcGYrY3zelX/o/rs5V36QlS0iXecHIHttGhOVsPXSYHc6yfoRjYcA9jefIO3NV1vu QaJnLCAd525eSxye0ELo7Ay/EhN6ksmskYvNYVQhevuFixC1K6Z1nEhDYjFPH/uSJFOP TyuQswTSjqI8lnxa6BbYSPZSexLxbXG7OlYp9hFC72jP2R4mGLzO6Qg5vc+121oYB/Lj 3aDQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:from:date:message-id:subject:to:cc; bh=aFrZCrVosvuUcXnGez2UkNofl47rIvd6rbrCl/vJsnY=; b=lfy/NT4ZX/T2vN4zld9welmfIq2299JERm9DkVLCJLVV7dOlZVznTSyQ6YWDiu/5oW RvbZ0u/U2v94Qa0uwBsR6Of4gEFRoAG28eVHnLwSBIhMnFINYF84UPVISOqpYkAjzMwm iwjCPeBOCO9ef84I9mfUJwVktuDfJBy72Xd5S8ZAG0gnA5LfjvXHyz5KD3RTTz+xzTcD gNKc97tb7hmDUpuYjqaMsr2Xje7vrrQliHyY8t1wpsfJsubphiM9fRMrnDlVXPMRgDVW qXvVXv1/GydnQnlLbnURNM53V/U0Bc9oA6YCB9ULjDy7VHlrNuFYfo9qnf3VZF362Fd6 GKuA== X-Gm-Message-State: AOAM531yLzTnS1rRO6xq96vjIbknqI5USSJINem0F5Jh/A2iL2kFc10L bgitXbiQzp7Qr2A1PvJeVvhVJWK0NsCq1+T2Wc5gTvIIkOQ= X-Received: by 2002:adf:e951:0:b0:1e6:85e6:65d9 with SMTP id m17-20020adfe951000000b001e685e665d9mr18505291wrn.412.1645526087209; Tue, 22 Feb 2022 02:34:47 -0800 (PST) MIME-Version: 1.0 From: vijai kumar Date: Tue, 22 Feb 2022 16:04:36 +0530 Message-ID: Subject: [Discussion]: Metadata to consolidate and rebuild base-apt from distributed CI builds To: isar-users , Henning Schild , Baurzhan Ismagulov Cc: Jan Kiszka Content-Type: text/plain; charset="UTF-8" X-TUID: p6jNVby3yOpM Problem: -------- We could have several CI jobs that are running in parallel in different nodes. One might want to consolidate and build a base-apt from the debs/deb-srcs of all these builds. What's possible: --------------- With the current state of ISAR, the below is possible. 1. Run all the jobs in parallel in separate CI runners 2. Collect all the debs and deb-srcs from those builds and push to a common file server. 3. Download the debs and deb-srcs and create a repo out of it in the final CI step, 4. Upload the base-apt to the server. This has some disadvantages, we need to move all those data(deb/debsrcs), this increases time and cost. What's needed: -------------- The idea is to have a simple meta-data that can be used by repo generation tools to recreate the repo. Why manifest cannot be used: ---------------------------- Manifest does not serve this particular need. Below are the shortcomings of image manifest, 1. Does not have details about removed packages(eg localepurge) 2. Manifest of buildchroot would not have details about the package dependencies/imager installs at the time of generation(i.e. postprocess) Some ideas: ----------- There were a couple of ideas, 1. To use an external script to create a manifest of the downloads/{deb, debsrc} folder and try to download the packages using that manifest and appropriate sourceslist in the final runner. 2. To use "apt --print-uris" + "debootstrap --keep-debootstrap-dir" to create a metadata with complete url to the package. Later wget can be used to download those from the web. We are wondering if we could discuss and derive a solution for this here in ISAR itself instead of opting for some local scripts in downstream layers. Thanks, Vijai Kumar K