From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 7067480740990025728 X-Received: by 2002:a92:6402:0:b0:2ba:a9a1:95d0 with SMTP id y2-20020a926402000000b002baa9a195d0mr21375745ilb.303.1645540305084; Tue, 22 Feb 2022 06:31:45 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a92:d444:0:b0:2c2:6d0c:9366 with SMTP id r4-20020a92d444000000b002c26d0c9366ls177267ilm.7.gmail; Tue, 22 Feb 2022 06:31:44 -0800 (PST) X-Google-Smtp-Source: ABdhPJzR4Pv3OsnZmD4OiCx5BMgX0vt5eUh/CN0fvXBEhqBhxz5vaDoLLSXNPkR+esRgPoDHRtYk X-Received: by 2002:a92:6406:0:b0:2bb:f1de:e13e with SMTP id y6-20020a926406000000b002bbf1dee13emr20464215ilb.305.1645540304608; Tue, 22 Feb 2022 06:31:44 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1645540304; cv=pass; d=google.com; s=arc-20160816; b=1DQuir8CgvuxJazwuaRZggo0Wbqr7bWCz9KCQqExFb6+sabij6/QTjlIV7BPbg6miM UJ8lIbjLkl5E2FiNNC1or3CZKuUNn2gfi+pd+CJwMsPNRHRIhxbLStASISYlwB88Xv4v lyLn2oXXnkoGXTiQ0Pqy7qniWFbgAzmZeStcFz4zHQ6ICmrE7OSGK3LepV0gR0bSPXZQ rmziQ1Xm4n7BYyeKlut5zINj6I3k5F7wWjI9gADNeCtX8ekHGqYATRh50AZffG27AXQX m+eLObKIvj3X/ar5V2l9vzJjYdD5WUogX+kxpKvXla16drceueZtgJVOAVc8KlQmmQxG 5ieQ== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:content-transfer-encoding:references:in-reply-to :message-id:subject:cc:to:from:date:dkim-signature; bh=wYGRLSqVlzoOdR8RzrNK1xrJekJd/qKKD0ZluHzpDnw=; b=WiBGepg6/JKmj/DqpIHTJ7ak3hHWHkx2tQOja1zjVUMw9JgQph4cftDHWbEXfWgoWb ouiilJgIsJvOmdrMGsh4SnZI/gaZApjktM/akGIyyTOM27EfE6wztOACDMP6odWX4p7f GwLdCNxZ+ghdZFWr0h37BgRU4omfE/EgT9DG+bI/9l0RpPiA2Vn5CvB2gwF0+bj6J2sB Zgwp9yR5sqwsivkSAwwOGnbqOyE0l0+EU6PXA3r/qMigvhL4lTghKTnebbS9GrVyHNFV zgWSbjxaID+YyujdfVdOiJIWyf611y330Rxoy/i8iQ/un3aHJMA0arT4TFGJjCUN3JaB zsfw== ARC-Authentication-Results: i=2; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=selector2 header.b=K39FLkT8; arc=pass (i=1 spf=pass spfdomain=siemens.com dkim=pass dkdomain=siemens.com dmarc=pass fromdomain=siemens.com); spf=pass (google.com: domain of henning.schild@siemens.com designates 2a01:111:f400:fe08::60e as permitted sender) smtp.mailfrom=henning.schild@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Return-Path: Received: from EUR03-AM5-obe.outbound.protection.outlook.com (mail-am5eur03on060e.outbound.protection.outlook.com. [2a01:111:f400:fe08::60e]) by gmr-mx.google.com with ESMTPS id q20si1080725jaj.3.2022.02.22.06.31.44 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 22 Feb 2022 06:31:44 -0800 (PST) Received-SPF: pass (google.com: domain of henning.schild@siemens.com designates 2a01:111:f400:fe08::60e as permitted sender) client-ip=2a01:111:f400:fe08::60e; Authentication-Results: gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=selector2 header.b=K39FLkT8; arc=pass (i=1 spf=pass spfdomain=siemens.com dkim=pass dkdomain=siemens.com dmarc=pass fromdomain=siemens.com); spf=pass (google.com: domain of henning.schild@siemens.com designates 2a01:111:f400:fe08::60e as permitted sender) smtp.mailfrom=henning.schild@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=baNEXo/kghQjZ79nj+rhq958V6IQtd2++7myXG+WTTSdi1ayNNDFyFK65Zz8VOK7wUCvgazX7K3s7McLcXgrFV/cHwtF9KPuB1YiNkpbKPsur5gZSv/s5gMnnfv83vFhLfOgyh8T4TFaKbQWnUoiqZFwvs8yw76HO7F6FoaAhLzV4AjzPfWc3G2sgzyM/0qypuXdz5BJYFhiSq6dhYqutgSjC0uFas4LNaTl3zhZoF7fRbfznOebJarz1QwwlHhvU0tFhhqlZeVRHJCA2M6m5Td3fZ4A8j+juCBB8eLa/HlMJRMKM3oY0uOIbUaHGCNAEHY3d496weqUmQiaDA6wsg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=wYGRLSqVlzoOdR8RzrNK1xrJekJd/qKKD0ZluHzpDnw=; b=klp4msOp0O/y1b1mTPvTHWa1qeDGgvgUldcjya0E9JtsC1OhJD9s5LE582JpOC13atNvHpQ7SMGB6x+CY4dCoQXk7BYmXEWC86WU1M1lT8WY1/Y+oZNwSnBZrkIeNtqzJkBQQvRAc3uQMEXsaLnZe64e6YLHTPPFS8TjeAX4hf+PtoM7j0qg8WvkyS/EBgnMS9umKAUJss0QbsiBHRPKYIrl/K8NKAvGhHnunrOkv8n3scGBT6fhrwx6bBXru1B8u0Y8qhKXg0ge1/aplfk/kW4uslJ0TBkrjS8A7LRNXc/mkC470dbI/UdtWfhmvQXt2r/TRz8ib4EaukVRHq9sQw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=siemens.com; dmarc=pass action=none header.from=siemens.com; dkim=pass header.d=siemens.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=siemens.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=wYGRLSqVlzoOdR8RzrNK1xrJekJd/qKKD0ZluHzpDnw=; b=K39FLkT85Hy4kC1ULf/v5vBpdwlkdyT3c7r9trS7pbhoMsM4g9bj/xoizsPDJM/VPW8bMWGEywb4yuSb0RSi9dGproW0YsiGcEgWduXSQfkqGM2E61t1iBbwbfW4nvNYrQL/7/P2YR0HeFgUUfII9G19uERxRhKaNULnBlyZEkbyL6cAZ8yZHKqGTb6w07BbFRZZMEtc0mNj2anXHCIagEuxksknOmCd1O49B+ZiVKNZjvWCLoGJaYWbaz0M0MyWxkD/K3MLTelLgviMzgkTIMQxdOa6UT3WiUZ9VBHVjGB9zXtQgSRFOsBzLZwiFUMVByz0miboU4K0ootdomxhaw== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=siemens.com; Received: from AM0PR10MB3459.EURPRD10.PROD.OUTLOOK.COM (2603:10a6:20b:155::20) by AM0PR10MB3970.EURPRD10.PROD.OUTLOOK.COM (2603:10a6:20b:155::23) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4995.16; Tue, 22 Feb 2022 14:31:42 +0000 Received: from AM0PR10MB3459.EURPRD10.PROD.OUTLOOK.COM ([fe80::d433:79c4:958b:1551]) by AM0PR10MB3459.EURPRD10.PROD.OUTLOOK.COM ([fe80::d433:79c4:958b:1551%7]) with mapi id 15.20.4995.027; Tue, 22 Feb 2022 14:31:42 +0000 Date: Tue, 22 Feb 2022 15:31:36 +0100 From: Henning Schild To: vijai kumar Cc: isar-users , Baurzhan Ismagulov , Jan Kiszka Subject: Re: [Discussion]: Metadata to consolidate and rebuild base-apt from distributed CI builds Message-ID: <20220222153136.08432cb3@md1za8fc.ad001.siemens.net> In-Reply-To: References: X-Mailer: Claws Mail 3.18.0 (GTK+ 2.24.33; x86_64-pc-linux-gnu) Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-ClientProxiedBy: MN2PR19CA0035.namprd19.prod.outlook.com (2603:10b6:208:178::48) To AM0PR10MB3459.EURPRD10.PROD.OUTLOOK.COM (2603:10a6:20b:155::20) Return-Path: henning.schild@siemens.com MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: f84da179-c4d7-47af-7a8b-08d9f6100b74 X-MS-TrafficTypeDiagnostic: AM0PR10MB3970:EE_ X-LD-Processed: 38ae3bcd-9579-4fd4-adda-b42e1495d55a,ExtAddr X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: ww9jgn5ClzFOoS7EA0ni1c9Gs9wtuHMek/lLIWKAIkYhGaEA1vqoiT73Pnr+C7smPl3CW0xA4sO3BFjo5bg85dksYE9WnkJaaTl6PtffTklLDP5II+BsjXj007MBT0gbSy5E+BkpdsSAlYcvSeOh28GRgeeLjLPpRQB4Ux+1uA3x/CqLbs8RiSpV5CMlD9G0AqHf0ONgs+3mtoaNSqWkHkZsbEf/BD5sstluYBTt+/eNOFbnkRBcPBujLKg8NfuKCVsb7wztJKO0/pICzHkHY2yJ1oTYhEnFh37wWey0ialnXvwvbfi1kFM4c/5ah5WZrNaduVyPQHQbeqyHqlieP7RNvQA5ssKeJ7Ikpbe31zrpw7FNero4VdR4x52UfboOk37DjiWVJRamHKFMxkExO5nqm7nZETP2x6z20P4AdzpKKvI1sJjjdZbwQCKu1bCucbRgYsvzCZI6qSLZZwkAXXSOHsKLPX3cIGVZ1x88MIC3orlBH+H1mSACLaXt+NMMJ9yLoF00MKGGThwLW/L1ga6rkaGlpQXqfDWwnxtNUgWawvZPdE1rx7ygV/AMisA1vp5aR8hPlvti2IAt/t7Hv7zEKYeytCVUcBf34uBoqN3w0LAGbvp57coq92Hn1U6TPELIi8ysEfe4eKBWxXEtEQ== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:AM0PR10MB3459.EURPRD10.PROD.OUTLOOK.COM;PTR:;CAT:NONE;SFS:(13230001)(4636009)(366004)(8936002)(2906002)(6512007)(9686003)(82960400001)(498600001)(1076003)(6486002)(107886003)(86362001)(54906003)(83380400001)(44832011)(6916009)(5660300002)(186003)(38100700002)(8676002)(6506007)(4326008)(6666004)(66946007)(66556008)(66476007);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?cFI0Keo7zZTBM0V65VTtRUX+QuQ+2sOPeSoAWZyx3NXAja3iFeG0abPjZbTX?= =?us-ascii?Q?pZMFcs3a3hOvRQgrBP85ANB1WvYPVxYlhbmfIeI5srcXf5La08K7mdeLenuM?= =?us-ascii?Q?QgZbXYOU12Cq2A37FujqqOTlM8wo/cVGvJoZoEs1jwFzydC9SyhfqPrDe2Nf?= =?us-ascii?Q?ImVimn5N4/fP0gCPiYhvXe3ZH80VJLc+iyGWzFDmmbiLOhlhHI95dNiC9Fbl?= =?us-ascii?Q?/vr6uPVsDBh6sB6htyr8yBF3VB43XpYkR2ugEmuCmG9MGfZOUyNrRXF0zyRY?= =?us-ascii?Q?7YUE1p5+SXqsczx6Cuu+y8aM7pdbxZUqgDew5C6vPmEeCKH7/l/5KqTy7ccB?= =?us-ascii?Q?yyBK7wan2h8FSFE+/hNggCffktQvW3bqJ9iyl8zPX/h9F1j3glpVVqpTmTLw?= =?us-ascii?Q?QxD9JntHjQ8mf01QZ4RewEbfhHjb/fArx4vNePmP8hHpcRV5VkbljaFOE6hL?= =?us-ascii?Q?LGlfWRjnqp3gcGiPpi1mOX+4Qx+8D/brpk5I5y0H8i5rlTyenlPkMeny4VSE?= =?us-ascii?Q?rj/gvsIOnY9RGJ+3v5pRmLrNCRRt2XHjmlb0lpQPkYDfgL1+LgBz52yRoumo?= =?us-ascii?Q?I68TBsOCmt+wRVJ5OUnc6KBmFbV/4l3zreSZhwOUrkoPMTcr8KG1ycmqQKew?= =?us-ascii?Q?r3byDVj9c33XAcBxccz+rZOAK1SMJ+0KdRYSdbHF8kG4zmHITVh24C51jA+f?= =?us-ascii?Q?VtCr+AwVkO40EuNHxeYrG1eOfatBsx9fFXHUeZDAau40RpKi8Ffy542qQ0zz?= =?us-ascii?Q?A8VfQ7p0TvFI80vkvgcq3rX/QfMh6e/mTnnZjtbfWKS8a1yiZewPZ9UOi9PW?= =?us-ascii?Q?HZlgFOjRFNjndxw1F8Bk7LjqpyYToIUc4ctzxUD0mFE5ji5weaau2ARMTKJA?= =?us-ascii?Q?ExyzTGvlEGIvgpUUW+ucZaJ57p/SzUFBVreqoUWdnUGUo8bcp/yYfk9Q3dLq?= =?us-ascii?Q?sNzjPTh93IXQ2FFV05HK97hrtRGKT8WEWDdpdQsp5M3VHlBluL25lGexqx27?= =?us-ascii?Q?1f5n/fWixWc3vW75dtR+D5tHvAh6xsQZ1nyf9rvLEd6CvodF5w93RKNtyXNb?= =?us-ascii?Q?I9upddjIYHaxmZOWw+LXRGwmGo1PmJRuPHVvT41hNkmUuDoK9V435MlJgauI?= =?us-ascii?Q?c+r5Y+FQ3r60An4bKeQaCxadeOwjR9tSU1FVCffbsY03tWci+FQZR6P1T3Kw?= =?us-ascii?Q?uWdNZy2kDl5GP9u6KBPL6lJTKgFbR1TkItgo+AR5ySF2/bz5q2UCeb2MP+i8?= =?us-ascii?Q?OtQ3xPKFIcS4t8q6Yb/d9pcYI4D3HArgK/vhkh2vaLzGbnE1PWdyg2yAyz6I?= =?us-ascii?Q?1jwN/lQmOqjhMkb3vJMrIdludZYBGNrRzyhFfoVsKhA5Pn+OhpFLHQer58FH?= =?us-ascii?Q?iFVs9ir1uS4kUnVOh2D0hl2PqvreD1oOSj/qFqHJ+NuwEiZe9LwzQEEyXnTm?= =?us-ascii?Q?Wtj01B/pEUSPF2H96mlHX34IGmevkUX7EIHgbp1XSpDyApihVg+/2EAmhPrO?= =?us-ascii?Q?cwGmgmTMNBVu6t7GE8WWnuWA1XQLUTHWziXgjj5JRh/y8FxHISU1ULno41uY?= =?us-ascii?Q?grqxkcoxNuQNLBvBmtaKZ4r2vTeQv+QznZctFHwIFza/nmls2r2t9k9QLQDB?= =?us-ascii?Q?dvLjF8i4FzC89iuCmVAcjx4kQi4+MI18/JWJKPBXqAB/AWFP5tY6f2HZOi1U?= =?us-ascii?Q?ngjcKweXwz+rToj5zdcjBs+1X4S3nlCn5kKq1ErH0zdPQQhsrg91jdMFr5Rh?= =?us-ascii?Q?3rdBz0ubDHGkJdP2kieMjOCV9Px2Pck=3D?= X-OriginatorOrg: siemens.com X-MS-Exchange-CrossTenant-Network-Message-Id: f84da179-c4d7-47af-7a8b-08d9f6100b74 X-MS-Exchange-CrossTenant-AuthSource: AM0PR10MB3459.EURPRD10.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Feb 2022 14:31:42.4696 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 38ae3bcd-9579-4fd4-adda-b42e1495d55a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: A4dyBMc2FmP3Tm8BKIQ6F/K2Xd5pVfGBCZ0Hhn72UfYWoC1rdZpl0oLmjlPentFM6aBtV/COIDKLIaaZmM1HrRIoQdCRiL9bTrGFZktw6D0= X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM0PR10MB3970 X-TUID: quISeJr8pCI6 Hey Vijai, Am Tue, 22 Feb 2022 16:04:36 +0530 schrieb vijai kumar : > Problem: > -------- > We could have several CI jobs that are running in parallel in > different nodes. One might want to consolidate and build a base-apt > from the debs/deb-srcs of all these builds. Can you go into more detail. I do not yet get the problem. It seems like you want to save compute time by sharing pre-built artifacts via some common storage. The sstate can do that very well, we are using shared folders for on-prem runners, s3 for AWS and sstate mirrors for population of "new empty runners" and "partial result delivery" of failed jobs and to sync on-prem with s3. isar is a tool to build images, not distros or repos or packages. While it can do all of that using it for such things can get tricky and isar was not designed for such cases. Meaning "base-apt" is not meant to be your cache to build many images from ... it is meant to be the cache for exactly one ... and sharing can cause problems. sstate would detect false sharing, say a package recipe for some reason uses a machine-conf variable. multiconfig or base-apt sharing would make you run into that bug, while sstate would likely not. So if it is about build time i suggest you have a look at sstate and the not yet upstreamed python helper scripts for sharing/eviction i can point you to in case you do not find it yourself. Henning > What's possible: > --------------- > With the current state of ISAR, the below is possible. > > 1. Run all the jobs in parallel in separate CI runners > 2. Collect all the debs and deb-srcs from those builds and push to a > common file server. > 3. Download the debs and deb-srcs and create a repo out of it in the > final CI step, > 4. Upload the base-apt to the server. > > This has some disadvantages, we need to move all those > data(deb/debsrcs), this increases time and cost. > > What's needed: > -------------- > The idea is to have a simple meta-data that can be used by repo > generation tools to recreate the repo. > > Why manifest cannot be used: > ---------------------------- > Manifest does not serve this particular need. Below are the > shortcomings of image manifest, > 1. Does not have details about removed packages(eg localepurge) > 2. Manifest of buildchroot would not have details about the package > dependencies/imager installs at the time of generation(i.e. > postprocess) > > Some ideas: > ----------- > There were a couple of ideas, > 1. To use an external script to create a manifest of the > downloads/{deb, debsrc} folder and try to download the packages using > that manifest and appropriate sourceslist in the final runner. > 2. To use "apt --print-uris" + "debootstrap --keep-debootstrap-dir" to > create a metadata with complete url to the package. Later wget can be > used to download those from the web. > > We are wondering if we could discuss and derive a solution for this > here in ISAR itself instead of opting for some local scripts in > downstream layers. > > Thanks, > Vijai Kumar K