From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 7067480740990025728 X-Received: by 2002:a02:c733:0:b0:2fe:ab40:d2cc with SMTP id h19-20020a02c733000000b002feab40d2ccmr2679167jao.126.1645717372317; Thu, 24 Feb 2022 07:42:52 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a05:6638:16d2:b0:306:25ec:11a0 with SMTP id g18-20020a05663816d200b0030625ec11a0ls715669jat.6.gmail; Thu, 24 Feb 2022 07:42:51 -0800 (PST) X-Google-Smtp-Source: ABdhPJxpbrMOAqkhtb4EzAvBwwyKRi8LEL7wQyW4RHzzsxcQQQgc17XfC4s1Sf8fZ5jZyQcb2CUL X-Received: by 2002:a02:a398:0:b0:314:ad84:75fd with SMTP id y24-20020a02a398000000b00314ad8475fdmr2532815jak.20.1645717371729; Thu, 24 Feb 2022 07:42:51 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1645717371; cv=pass; d=google.com; s=arc-20160816; b=yWNa37y4b2lOe5Uijr9t7HuZvNuw9uM2dz5GRLg1A7tr3d1pGpQvtx1dfPkUfVs4ii D4mOWxw5QOsAUmxfY5iH0wK/X05cjve0gl1UOrCZYtcNrGComjaF1y2uXpWLbHk+wdhl F9Z70oP0XN8haGm4TIw1OA4qW/uRcMGEbI6TjEodj75y1iHY9UErvRmFwYl6uAEENc3T l5KbEfVX2/gc9PBCQNfJzXzKBnm65PWgEWa7DpU2/iyU7hWvfwmKIzuzyPsr+BCePGvC irecOTl/F9k/DSLyOrhl4t7QzmVnrHdnXaIVhpfAEpHxxMPNSEMyBn0deEyVb6rpoPEY ki8Q== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:content-transfer-encoding:references:in-reply-to :message-id:subject:cc:to:from:date:dkim-signature; bh=zcRxT4ps2UpcB7juTZlxImgc1Ct6yJLP9+oqnf3cqXQ=; b=cL+Ld/GSTSSfhurWM1rrHXB83cSg1KoTpYHYXekOC4rdToYwoKPZEaJw0NB6amhtKG 9VHJkeDqrepDxqsTTQX1TywWVsZGmkYHJp7mUVHt3bYQngozwueArc3lHx1wvY7BIQkc 7FaHbkUZlmC+8xTJQk42d4bweCODRZR2z/NR0i+lbv+sZrWIYRk9d9rRbRNAXrhkiZtm B+zvOAcaPvtUynrvG1DCX8kkzSvlAMTId5B5KrINsu5oG605Sil5Ox+4eqPjFpw1FGXC sLWwKNMlWS8o25EBCqVgu4LurXRXi5nqIv9i0K4z/4R11bIjAnwD51XoI5keM5Oy86EH H5Cw== ARC-Authentication-Results: i=2; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=selector2 header.b=JDgTGzr1; arc=pass (i=1 spf=pass spfdomain=siemens.com dkim=pass dkdomain=siemens.com dmarc=pass fromdomain=siemens.com); spf=pass (google.com: domain of henning.schild@siemens.com designates 2a01:111:f400:fe0d::626 as permitted sender) smtp.mailfrom=henning.schild@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Return-Path: Received: from EUR04-HE1-obe.outbound.protection.outlook.com (mail-he1eur04on0626.outbound.protection.outlook.com. [2a01:111:f400:fe0d::626]) by gmr-mx.google.com with ESMTPS id i7si178607iov.0.2022.02.24.07.42.51 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 24 Feb 2022 07:42:51 -0800 (PST) Received-SPF: pass (google.com: domain of henning.schild@siemens.com designates 2a01:111:f400:fe0d::626 as permitted sender) client-ip=2a01:111:f400:fe0d::626; Authentication-Results: gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=selector2 header.b=JDgTGzr1; arc=pass (i=1 spf=pass spfdomain=siemens.com dkim=pass dkdomain=siemens.com dmarc=pass fromdomain=siemens.com); spf=pass (google.com: domain of henning.schild@siemens.com designates 2a01:111:f400:fe0d::626 as permitted sender) smtp.mailfrom=henning.schild@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=fEVFFqZqaW9JO31kp4w53sGFHrWVxZ3BGAgDnavIjvW1roohd7kZff6Ywdy+WH6rrfwZY+h4/b9mnfqi1cjL4wFDIbWFFkFKAsmxWorNxWBR8BW05XWwDUGElKpAZX9R3iuBnVsmKa5T6M5ASLKTRJO+8MsKEhKEwwAD/TpRnrD6nKa+Xf/yL4JmOazWro4yK97toJvd/dhmrhQtgVcuGDaPdyECIzGhedyLS8P+2QFU9bKnXkvq3lbKFyx+9jj0MNhXzc42VgvmSTLSEytGyub90wh6UiQBJlWhzGl9ozn+gXoWAo1tw0gg/Q0sBn0pR4wlUR4LVSje7LTNqtX2cg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=zcRxT4ps2UpcB7juTZlxImgc1Ct6yJLP9+oqnf3cqXQ=; b=fxCpI/fvg2s3CcjzynWdfFQwPA4Tjc7ySEkrp9G7FuoTfqMM5qHXHuFOsyy3iHDeChx/T1hH7TnRBxBfy9xTi71cy/P5FOImTf/gOdP+17b0GdQbOdOFigB7o/06Eghl986Jwd65E0PfpoqAekXehKwYD4/CL2q7qLalaFu89pDxPgtuN1DPqH7Ob9+0FDkD+GWj9HPSz+fKFwR9rUGq+1fujBd1Y5RwSWdw+U3MwOL533Rxt/FWi7w7fzG1mEnFZj8YaEzxvdDiJaRjqaARleDJHLbSgl1VCFVUHcwZ8qwEY1Yl57Jvxys0NI3ZHs93nCOF8ehpcRCgbsa7NFD7YA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=siemens.com; dmarc=pass action=none header.from=siemens.com; dkim=pass header.d=siemens.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=siemens.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=zcRxT4ps2UpcB7juTZlxImgc1Ct6yJLP9+oqnf3cqXQ=; b=JDgTGzr1/nYZ+aLEgEP8hMlMwZ817VYcRdLrWs4bg9750JwjJXMykRKnGBWE6G9HkeVVTXgzAIhhunsJpEtOFxfVGJKtYOraIr8ymVCEOz9tn2GH5blAg7DAbOGtvkXnALq9VNyjdqJTPgbtjNRvpSwyEQPDcieh37iD8fGsTn5/a4cnAVHdA3whKthNLVLRbqHEHgwMWVPFcZEgJkuoEmvf8ZVz7wuVmp+JXE7KLYvXqAXVXTgjNQXhSG9h7DwAd9dediPA43uGfWaPjgVWTddO74QSO0uYd+Bpbh8uxE54CohRwJZ0e/S0P4YMkQIPj7saIMLev1549TNjtkI5iA== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=siemens.com; Received: from AM7PR10MB3462.EURPRD10.PROD.OUTLOOK.COM (2603:10a6:20b:13d::17) by AM7PR10MB3778.EURPRD10.PROD.OUTLOOK.COM (2603:10a6:20b:179::11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5017.22; Thu, 24 Feb 2022 15:42:48 +0000 Received: from AM7PR10MB3462.EURPRD10.PROD.OUTLOOK.COM ([fe80::a116:222:eb1b:efdf]) by AM7PR10MB3462.EURPRD10.PROD.OUTLOOK.COM ([fe80::a116:222:eb1b:efdf%4]) with mapi id 15.20.5017.022; Thu, 24 Feb 2022 15:42:48 +0000 Date: Thu, 24 Feb 2022 16:42:44 +0100 From: Henning Schild To: vijai kumar Cc: isar-users , Baurzhan Ismagulov , Jan Kiszka Subject: Re: [Discussion]: Metadata to consolidate and rebuild base-apt from distributed CI builds Message-ID: <20220224164244.6e4bb002@md1za8fc.ad001.siemens.net> In-Reply-To: References: <20220222153136.08432cb3@md1za8fc.ad001.siemens.net> X-Mailer: Claws Mail 3.18.0 (GTK+ 2.24.33; x86_64-pc-linux-gnu) Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-ClientProxiedBy: AM6PR0502CA0048.eurprd05.prod.outlook.com (2603:10a6:20b:56::25) To AM7PR10MB3462.EURPRD10.PROD.OUTLOOK.COM (2603:10a6:20b:13d::17) Return-Path: henning.schild@siemens.com MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 20cb913c-3fb9-4687-4aaf-08d9f7ac4ed3 X-MS-TrafficTypeDiagnostic: AM7PR10MB3778:EE_ X-LD-Processed: 38ae3bcd-9579-4fd4-adda-b42e1495d55a,ExtAddr X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: Nww/EcIyaahI1SZHajETfRAqeysr8PP2nWpKtKXxDl+jCvCq73LyS4rpvTctxFcuPgKc6XmfE/+MU2Gw5G0mkeVmIJm70dhfU6j5JyuoNmVyM+ZeliKPuK/tWRyXiupoIl/8rmUiGnd9CBfumxWwo8GzNiwzHN2G2wKFqpZknnr92uCPIQIRRYB/yMECcUnDlgG60NMMux7aFYKQ7IvYLNy3Vv/xkBDwqh6w+L6AhM8nPMXPIg4kMmKrx9FOqWT/UPpLrFBHVq2J+CDjNqndWa8+6WcE8kkQS1MoWR67apmWwMRBrKYgVEItsIE9MPkWrkBXROhWhzBY3/mQfSbcKW1Cos6A92G1gmtIOzI/+enOh8QpUN1VgjL2e99jcCQWbLTQGSz4lhetJev+qqL2o0w2BRWwKCxqZSIkC+uA0L6tEqnx6vbJuPUFnRA3ct4SylY2a+Tr3R32/HJ9nUj1RGOq+HcikUlJrkZEH3UzoFBUIP0n8CxYGyEHnv6oc5D6rABql1Rqo2efOKtn00hOQUg4mCcWbbG+EbUYqdmEBnsy8cU3/mig5faN/u2EWx/Fu4MJwhmWWmuwDkcZjadHBZeBmhvCaXsbA2oGJfSodyAu0YuJZWLmpisSBhUfdKy0umJdaTbvjWucJ9Rb97NplzfOtj27i8mGSrcAUXNPx9yTKD829ncAzVZOboJ/xWsWYemk/v3taGSQk/YfcahBZjj08dZUOCQWggp+x3lA5BG1urZUFLNkTP8WhDQZaUEsQ1Z8UnD3p6GHmkQHQlX/Hw== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:AM7PR10MB3462.EURPRD10.PROD.OUTLOOK.COM;PTR:;CAT:NONE;SFS:(13230001)(4636009)(366004)(54906003)(316002)(6506007)(66556008)(66476007)(6666004)(6916009)(6512007)(66946007)(53546011)(38100700002)(9686003)(86362001)(4326008)(66574015)(83380400001)(82960400001)(107886003)(508600001)(8676002)(8936002)(1076003)(5660300002)(966005)(6486002)(186003)(44832011)(2906002);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?ZZ0JMZymcV5JvYmA89CS2F/YaPNrcoR+DVGqiEVIYDTfN3r1E+p8cvLr/7Og?= =?us-ascii?Q?qW+WrUtkHEo5OCXvFCun6NswfRJrrPGRUoCEuxF9Q44MhM3OHJ0wKhUH7ibk?= =?us-ascii?Q?8lE3nq5Lofgz/+jkuMlgE/pDe30QRy4KmuvNxIfG2Zc21qBSOSNGlczyssPu?= =?us-ascii?Q?7+5N/ugX1rPZIzdjOSRXUI1eTCwHQeQaupGoJI46zDjPsbcmdQlCOeD3m+k0?= =?us-ascii?Q?upXpAbLNvOgc/nRW8oAy9zOfvJoz1N9mp5lrr/XOYZs1qc3TD9koV+/N3e3X?= =?us-ascii?Q?MAyFzvZpSh324mRH93dVOkik3gezC/3oPiOTJsTgufXzIVoSKeaIOL2VEwl4?= =?us-ascii?Q?SywXPLej10xEax9nyWaryfOFKr/vBHF8WGloF/nW9BKtmbcnbweT8S/smrVi?= =?us-ascii?Q?XVDNCxWkyUvHvxYweBHGtmj3k7MLHHG8tq1baAhi6EsYj21SJBvdCE/k0Jns?= =?us-ascii?Q?r/8jj4UOvxgcpf53c4Ui8xEhGwvChSloN8XQbcykAHl/Eu8L1SjG+D+tvd3b?= =?us-ascii?Q?UZDg1IDKv4EMhtC5HLKA79DXUyZsMjgZqexSoNo3yR0Xdm1vwbwqWVthtkPx?= =?us-ascii?Q?bQI02eY6oepA8fyqXCxJnw8SmoQsUCrVVtSEl7PhlWxTxASSkYOwDJ2LGdrY?= =?us-ascii?Q?9uZkLFDdKHatEku5wB4KhUKCkWLOIxoCcCQ9aFuU2nlDrpNe+kFrBnIbANeN?= =?us-ascii?Q?OzCzMjlhc+XeMtrQO2umc3ZGTPVMn+gN9WHNu89+Pcj+F9k+pG3/mPxoc/Bj?= =?us-ascii?Q?96bzQz7i/+p7uUI8DOu/EzFZ39ltygFeFypiGL5PxRvFtk1cfxmE3cieM8/O?= =?us-ascii?Q?vNgA/SZMbX1L0wpIvGtuAQ2ex6/EwNcBO7uxIbKNkknWQK97WD/dWwWcL9sZ?= =?us-ascii?Q?Bg6Qp77lJ4RYPZE4iVZ4HF3rE+a96uos3ylJ+MJ8Ktj1YjOGZ2/Q6qqYq4k+?= =?us-ascii?Q?Yu7aw9AB27IshOBaznOgPn5kRgc0ReSgruhSViQY7v2ap6jDeYBxtW9iUYcG?= =?us-ascii?Q?jk20HdB/q5jPz+4i2XVoMKUM9OtH8/OC/TknpmO8kn2EPpAmBJm+wZxB1K4C?= =?us-ascii?Q?wiTMGprc3nFTlENUj50F6a0Htp/G3nVQmCNpRR36dDJmGB3Np+z83bhoVwaA?= =?us-ascii?Q?h+VbgMxcnD8U4vKLbsOUSGOleFmVGsAQetnaRftMEivjdkOHhUqwuMK+JNIV?= =?us-ascii?Q?Qj5yrUFJcQGCoi6c6BCpxUH+ue+1ATvx9do5YvUa6lFlKT9mZIU+BsuXQVO6?= =?us-ascii?Q?w8qcihrOBwWGLGkNxHDJv91CecdFNP0CIZTueUfrrTYbG3GfeTVv0DsGyKHQ?= =?us-ascii?Q?NKdVhtZ0CZwcjgLmDxegmennoSXk7P28WSgxEnFjefzZbIoloR5fzpFImfAq?= =?us-ascii?Q?4toFa18DnldJ0WD3gv1RN0O28UoD9cvgDhD9v7dQeVqYB9Sn0XdTWAgLiqxs?= =?us-ascii?Q?dx/h98fiovLXbDEnAoHD1NOWuVHlSkbx32U3EPjyX1X4eu+uG8lNKtjpWI3E?= =?us-ascii?Q?SIIGIeFeWS+ZXnhjuZo4xmByo4xhP+u+zTIqyrTypQBOrW3wz8tWnh4jbWN1?= =?us-ascii?Q?hwOwi6/sDxEgdinIYhxpr6CtrgZmwLV14ew6m+MtMiTitfhqyIdfjLaWyzRp?= =?us-ascii?Q?ZA2yiG4y2/rZ+XlXUQaGh1a94r2B1Cl9qW7LujqCXRb6oKjxqznVoZ7BmYkw?= =?us-ascii?Q?JuH1Ush1gfsUyIBAaS67HYr5WKHU7PVFmiMTzbtlMbZsbBKlqEOJwBHzXmLg?= =?us-ascii?Q?9ZPV540MIs+hYmSZocXHJprhhWmJFBY=3D?= X-OriginatorOrg: siemens.com X-MS-Exchange-CrossTenant-Network-Message-Id: 20cb913c-3fb9-4687-4aaf-08d9f7ac4ed3 X-MS-Exchange-CrossTenant-AuthSource: AM7PR10MB3462.EURPRD10.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 24 Feb 2022 15:42:48.0103 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 38ae3bcd-9579-4fd4-adda-b42e1495d55a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: fpB15DXk0l++/6pefEujosJ6b51IweS2acZoiFt3uGZypv6I/aZWs+vuCXIcugNG2JV4YcQDhnfdEH+AoKH0dksw1PYNe+KZi652fjr66kQ= X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM7PR10MB3778 X-TUID: RavS276zLiSn Am Thu, 24 Feb 2022 18:50:50 +0530 schrieb vijai kumar : > Hi Henning, > > On Tue, Feb 22, 2022 at 8:01 PM Henning Schild > wrote: > > > > Hey Vijai, > > > > Am Tue, 22 Feb 2022 16:04:36 +0530 > > schrieb vijai kumar : > > > > > Problem: > > > -------- > > > We could have several CI jobs that are running in parallel in > > > different nodes. One might want to consolidate and build a > > > base-apt from the debs/deb-srcs of all these builds. > > > > Can you go into more detail. I do not yet get the problem. > > runner 1(Germany) -> Building de0 nano > runner 2(India) -> Building qemuarm > runner 3(US) -> Building qemuamd64 > > > All these builds are running in different servers. > If we wanted to create a single base-apt from all these servers, then > we need to copy over their deb/debsrcs/base-apt to a common server and > then > create a consolidated repo. But why would you want to do that? I mean i get why you would want to store all in the same location, but not why it should be one repo. Maybe to save some space on sources and arch all .. but hey there are ways of deduplcating on filesystem or block level. You are just risking a weird local "all" package not being so "all" after all ... false sharing. > This involves moving around this data. Yes, if it one central storage place. No matter if it is one "repo" or many "repos" in i.e. folders. > The problem can be avoided if we have a single metadata produced by > all these builds which would have details of all the packages the > build used. > Basically a manifest of the build. This manifest can be later used to > recreate the repo which can be hosted later on for these jobs. We have a manifest for "image content" which already is fed into clearing, it is a bill of materials an nothing else, it can not be used to rebuild. Even if you had all metadata you need to store sources and binaries somewhere reliable, whether that is central or distributed is another story. Pointers to anything on the internet (including all debian repos) will at some point stop working. So if "exact rebuilding" in a "far away future" is what you want, mirroring is what you will need. Partial mirroring based on base-apt even with sources will be shaky and you will find yourself digging in snapshots again. But it will work. In the worst case you will not want "exact rebuild" but "fix backported rebuild", which means you will need all build-deps mirrored ... rescursively. In fact any "package relationship" maybe even a Conflicts might become rebuild relevant. A partial mirror will not cut it, rather take a full one, so you do not need to care of which bits to ignore and do not risk forgetting anything. The ideal way would be to eventually liberate snapshots of its throttling, the short term way is to spend some bucks on some buckets (S3). > Having metadata and recreating repo is one way. There might be other > ways as well. I am afraid you likely can not recreate if you do not keep everything yourself or a place you trust (snapshots?). There have been several threads on that topic already, including how one could help make snapshot work for debootstrap and co. Coming from reproducible builds and qubes-os [1] [2]. If you dig deeper you will find many people offering help and funding but for some reason things seem still "stuck". On top we could maybe see if we can establish something like snapshots in Siemens. But i guess outside and open to anyone will be much better. [1] https://groups.google.com/g/isar-users/c/X9B5chyEWpc/m/nVXwZuIRBAAJ [2] https://www.qubes-os.org/news/2021/10/08/reproducible-builds-for-debian-a-big-step-forward/ > That is where we thought about the --print-uris option of apt. It > basically gives you the complete URL to the package which we can > download using wget. > A manifest containing all the packages ever used by the build with its > complete url. It could easily be used for several purposes, like as > clearing input, > repo regeneration etc. Maybe we can find valid reasons to extend the manifests. But URLs to packages seem almost redundant, knowing the package names and versions and all sources.list entries one can generate these URLs for any mirror, picking just one of many mirrors would be limiting. And maybe there are valid reasons to having manifests even for buildchroots. But the problem here is that they change all the time while we still use one buildchoot. We see packages being added as build deps all the time, but also removed when build deps conflict. > I don't think sstate can help here. I might be wrong though. I guess sstate will not help. It is even more storage needs and more storage sync needs between runners if you want to share. Henning > Thanks, > Vijai Kumar K > > > > > It seems like you want to save compute time by sharing pre-built > > artifacts via some common storage. The sstate can do that very > > well, we are using shared folders for on-prem runners, s3 for AWS > > and sstate mirrors for population of "new empty runners" and > > "partial result delivery" of failed jobs and to sync on-prem with > > s3. > > > > isar is a tool to build images, not distros or repos or packages. > > While it can do all of that using it for such things can get tricky > > and isar was not designed for such cases. Meaning "base-apt" is not > > meant to be your cache to build many images from ... it is meant to > > be the cache for exactly one ... and sharing can cause problems. > > > > sstate would detect false sharing, say a package recipe for some > > reason uses a machine-conf variable. multiconfig or base-apt > > sharing would make you run into that bug, while sstate would likely > > not. > > > > So if it is about build time i suggest you have a look at sstate > > and the not yet upstreamed python helper scripts for > > sharing/eviction i can point you to in case you do not find it > > yourself. > > > > Henning > > > > > What's possible: > > > --------------- > > > With the current state of ISAR, the below is possible. > > > > > > 1. Run all the jobs in parallel in separate CI runners > > > 2. Collect all the debs and deb-srcs from those builds and push > > > to a common file server. > > > 3. Download the debs and deb-srcs and create a repo out of it in > > > the final CI step, > > > 4. Upload the base-apt to the server. > > > > > > This has some disadvantages, we need to move all those > > > data(deb/debsrcs), this increases time and cost. > > > > > > What's needed: > > > -------------- > > > The idea is to have a simple meta-data that can be used by repo > > > generation tools to recreate the repo. > > > > > > Why manifest cannot be used: > > > ---------------------------- > > > Manifest does not serve this particular need. Below are the > > > shortcomings of image manifest, > > > 1. Does not have details about removed packages(eg localepurge) > > > 2. Manifest of buildchroot would not have details about the > > > package dependencies/imager installs at the time of > > > generation(i.e. postprocess) > > > > > > Some ideas: > > > ----------- > > > There were a couple of ideas, > > > 1. To use an external script to create a manifest of the > > > downloads/{deb, debsrc} folder and try to download the packages > > > using that manifest and appropriate sourceslist in the final > > > runner. 2. To use "apt --print-uris" + "debootstrap > > > --keep-debootstrap-dir" to create a metadata with complete url to > > > the package. Later wget can be used to download those from the > > > web. > > > > > > We are wondering if we could discuss and derive a solution for > > > this here in ISAR itself instead of opting for some local scripts > > > in downstream layers. > > > > > > Thanks, > > > Vijai Kumar K > > > > -- > > You received this message because you are subscribed to the Google > > Groups "isar-users" group. To unsubscribe from this group and stop > > receiving emails from it, send an email to > > isar-users+unsubscribe@googlegroups.com. To view this discussion on > > the web visit > > https://groups.google.com/d/msgid/isar-users/20220222153136.08432cb3%40md1za8fc.ad001.siemens.net. > >