From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 7067480740990025728 X-Received: by 2002:a02:c48d:0:b0:314:5d51:7c97 with SMTP id t13-20020a02c48d000000b003145d517c97mr6679956jam.161.1645810063116; Fri, 25 Feb 2022 09:27:43 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a05:6e02:1bee:b0:2c1:18bc:5faa with SMTP id y14-20020a056e021bee00b002c118bc5faals1321798ilv.1.gmail; Fri, 25 Feb 2022 09:27:42 -0800 (PST) X-Google-Smtp-Source: ABdhPJzQ3E0HEV3quOcjUsL888dyyOpAkpP9sFRDgaFDszpEJDP0OtMNx37+ZIzjg7duz1SfqYmn X-Received: by 2002:a92:c88d:0:b0:2bf:6a06:8e9f with SMTP id w13-20020a92c88d000000b002bf6a068e9fmr7190296ilo.52.1645810062619; Fri, 25 Feb 2022 09:27:42 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1645810062; cv=pass; d=google.com; s=arc-20160816; b=zy8kpGKO60icuFx0ReodTD4Zb7K7Pz5gt8aoZWdrY5ScWcFVpY7fCkeJ95NkfABaos rnXV85FGcamygz0bZpFXEVn7hk0Zo4ZffCSRs+OlyKC5rSte/lcC7VYJWI2DaIYIQQ2j u3pVXRlnCNMfKfbdPrdSQHqPutHUhE69ZV0u6VsIqxcKLtOy+g7q7mhuF20K9hiXaY0X HUboUCbxdWh2xLIcGjhtvv5tCQeWurJWJLyGuyNhP1hxwG2bGTFQ7ORcd654ez7z6V3e 86FFir7o9YF4E7GETdBMNfoiA/dPEOZXhoNBuKVi8xys2U71mDhofkaPpLwg3W4xDlIW 5JUg== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:in-reply-to:from:references:cc:to :content-language:subject:user-agent:mime-version:date:message-id :dkim-signature; bh=JwfYy+yWWtxHVH27gg5pvvgm86H7HT6Hyx0DNttnz4M=; b=GeJDbNGlRORfArlsKz5CpZsqcaKmxuAmwoLbdqClmJiziz0kf/Ca86QuriWpk5PVbD z0I+qCmH82+bTYJQyYau7Y3aaVKev4kZcOkCTsvoEQa/Zuwr8mUm/Pupy1WKo4LS9IdG uwfVCSegRlDagm3NFhvlwQcRQlIo9mI+8e15eYjbnltSJ+MnKqngbaDNUUOQFEjIKkji cBDrPDzT8GnhGuIPXmsqRhSg5GDhR0vomeYnIgdtvTBwxZxOh9MxVxk1v1R9dhf9HXqy DID8ueg9Y4a83hVsZgk7xoXfudx/onSSoWFhoKY3J/FQAzJLFyvVBE79yZy111iTOkFU xwvA== ARC-Authentication-Results: i=2; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=selector2 header.b=y6xFlpiG; arc=pass (i=1 spf=pass spfdomain=siemens.com dmarc=pass fromdomain=siemens.com); spf=pass (google.com: domain of jan.kiszka@siemens.com designates 2a01:111:f400:7d00::617 as permitted sender) smtp.mailfrom=jan.kiszka@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Return-Path: Received: from EUR05-VI1-obe.outbound.protection.outlook.com (mail-vi1eur05on20617.outbound.protection.outlook.com. [2a01:111:f400:7d00::617]) by gmr-mx.google.com with ESMTPS id x8-20020a029088000000b0031529dcc182si212322jaf.7.2022.02.25.09.27.42 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Fri, 25 Feb 2022 09:27:42 -0800 (PST) Received-SPF: pass (google.com: domain of jan.kiszka@siemens.com designates 2a01:111:f400:7d00::617 as permitted sender) client-ip=2a01:111:f400:7d00::617; Authentication-Results: gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=selector2 header.b=y6xFlpiG; arc=pass (i=1 spf=pass spfdomain=siemens.com dmarc=pass fromdomain=siemens.com); spf=pass (google.com: domain of jan.kiszka@siemens.com designates 2a01:111:f400:7d00::617 as permitted sender) smtp.mailfrom=jan.kiszka@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=JoCMMKQAmHjReStj7RU44BaKlC13kxqMfdyLvIhRzsv7cE/HX/bEG8fJWOiUcaQ73DTsig4J7GR2/+zm4ufLMclg/nJGyxT2F7JCFZtt+Y1wnCL6mg2OJOvf2KHGfELccwLz7xJzMpJBB0mr/FIcZuNTv0u2fg837Y02iK7aJJHyoZSMYh+ujYHbotHw/tyh+Qnty/5JH1j5z5Ou+fOmhpgTYAuk/+p1hvR6dpa6OOCZ7M9kep0o56QzzKHDC0VsJyFEFox+/79AvXeAscGsddsB6oS/xtK4hCqeoSSSZflTolMCLnqt6+96NTX5sDcPh1fSIKFICKpbcR2VvjdOWg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=JwfYy+yWWtxHVH27gg5pvvgm86H7HT6Hyx0DNttnz4M=; b=gCJSujbHAIic+ukvfvsSu3gklhM37rK9/sFOxdBOdc96JZHUFo67WasO2MbHsXhbhatSjumALD5NcrBU/pvHSvKDnHZHjWJiOmWHMj2q6Jnkl0ZVP81wgDmDtOIMm9+OXwxwevks6lk8eWcpccJ7vkwv4tUW6+J/LJveQfhtQUTt8TgOQ6MwkYub4gu1ZPb0rlTo9GrZlzMdrH5jzgXJoWEUxW2FNqeirbFjpz2gUUupGIiE3mcE0ZfPftnsUSuSsloaGWQjVja7UNH91jvoDzchaGe+T/vZpAHF3tlz/3me+3FtL6mfy9ls61O4TvuJg39UOcIzYNVOSJg+3e+QEg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 194.138.21.71) smtp.rcpttodomain=gmail.com smtp.mailfrom=siemens.com; dmarc=pass (p=none sp=none pct=100) action=none header.from=siemens.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=siemens.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=JwfYy+yWWtxHVH27gg5pvvgm86H7HT6Hyx0DNttnz4M=; b=y6xFlpiG0jMYXaAd2Xaxq/Vcym1S6pUQgzFnIBcKz0nUYZtVAmq2GQMRaHhU80uhnv+MqyBSR7D0kpwiF1DcOa0J6vzZG3uFnfddlr/JHxUVMexyfpdvJAGU9fHmpS/0o14mXngeP5opLi9vsBOd9mQcpC/sAyjqXzkClV4U3LYe4+c2uyqpg9pZpmbD4rFrESoK/K9KxA1s3CO+1gEOtcGaY8/f4SsbDkRGcGz0M1f6fqgcMSrvQZqryQcK1v2MeeXNMxJIQlnC7qrKyiGi5d2OJqlbc0mTvsPqizv6d8josNiuXa+kSgjMF/rLMG0ly61PNW3xsBDFRQPl956CKg== Received: from DB6PR0201CA0014.eurprd02.prod.outlook.com (2603:10a6:4:3f::24) by DB9PR10MB4650.EURPRD10.PROD.OUTLOOK.COM (2603:10a6:10:23e::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5017.25; Fri, 25 Feb 2022 17:27:40 +0000 Received: from DB5EUR01FT031.eop-EUR01.prod.protection.outlook.com (2603:10a6:4:3f:cafe::ea) by DB6PR0201CA0014.outlook.office365.com (2603:10a6:4:3f::24) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4975.11 via Frontend Transport; Fri, 25 Feb 2022 17:27:40 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 194.138.21.71) smtp.mailfrom=siemens.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=siemens.com; Received-SPF: Pass (protection.outlook.com: domain of siemens.com designates 194.138.21.71 as permitted sender) receiver=protection.outlook.com; client-ip=194.138.21.71; helo=hybrid.siemens.com; Received: from hybrid.siemens.com (194.138.21.71) by DB5EUR01FT031.mail.protection.outlook.com (10.152.4.253) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.5017.22 via Frontend Transport; Fri, 25 Feb 2022 17:27:39 +0000 Received: from DEMCHDC8A0A.ad011.siemens.net (139.25.226.106) by DEMCHDC9SKA.ad011.siemens.net (194.138.21.71) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.18; Fri, 25 Feb 2022 18:27:39 +0100 Received: from [139.22.139.122] (139.22.139.122) by DEMCHDC8A0A.ad011.siemens.net (139.25.226.106) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.18; Fri, 25 Feb 2022 18:27:39 +0100 Message-ID: <45e6e9ba-8a05-e617-d5ae-949efb53c35b@siemens.com> Date: Fri, 25 Feb 2022 18:27:38 +0100 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101 Thunderbird/91.6.1 Subject: Re: [Discussion]: Metadata to consolidate and rebuild base-apt from distributed CI builds Content-Language: en-US To: Henning Schild , vijai kumar CC: isar-users , Baurzhan Ismagulov References: <20220222153136.08432cb3@md1za8fc.ad001.siemens.net> <20220224164244.6e4bb002@md1za8fc.ad001.siemens.net> From: Jan Kiszka In-Reply-To: <20220224164244.6e4bb002@md1za8fc.ad001.siemens.net> Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit Return-Path: jan.kiszka@siemens.com X-Originating-IP: [139.22.139.122] X-ClientProxiedBy: DEMCHDC8A1A.ad011.siemens.net (139.25.226.107) To DEMCHDC8A0A.ad011.siemens.net (139.25.226.106) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: c77228c5-412a-42e1-06f7-08d9f8841fa9 X-MS-TrafficTypeDiagnostic: DB9PR10MB4650:EE_ X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: RCTOpCY2XZV1XCRRdCcoxPazD3E3Zh97YwaaHMiK7fX/74UOzVWYrum+sbFcgefuwAvYUKB54cwmOJTJcQGh+K0LvSVl2h3wRPXz1/7bOn5S44r3zB/nGBKU4QNLp2A6q2iFMYX0VnxOzohn9yM7YXOBFz344bd5kfHHs0H7NZSkoPAyxbc2TYXxd86Q84FfhLXq08PTp0qfEyqIvongkyFfsyPjr8vys1qMbuPlrwZ0brp/KYqZxKwp/uq0LzkOg2a4hCjsAsBvk9D9+KpgKDBKHy+Kjn1WECERnApOs54xqglyepM7UjeRhLxdL3+7xz/bQhkUJQPe1NK3slY8BFSJSfqsC4Tt/IcM0d35qHfCfRfAoK7PE8sFo642zaddbWk0TypjBQKR0oq3DUr9mnKXYf8bKHR2idj2DdSTM3NXHj2L41RkyqWIUUayZrlH59lcWdRBpmZOPt/FQ8XXf2n2Q/WDBnIvYsMqG7eDkr1Jc0997TZokKAalYIeiq4ihfZOrcE6GzXViiyCuLytQFkBCEE5cU8k8M1M+1b0z/6NAKgVaPVrimIMETzPnsHo+/lkN/AQ8RdThzFCMSwVh6JIQn6lz452BPGUORJ8Mevt8L7+1aiCgRPJ37OnNSuYGTAWQF/HyK6Bu70tClxg+v9wGmdTKGouNv0J9VNsk/xYv4j3pEtq9njW+aAfuG4LpX9l8P3rP3kkrtTRg/iwERSIkhxO6/j9aSjSAyIk3SsDS0wR5twWC1x1Hd5CcFIZjG3tC6RqrWW5FTN3EbtepA== X-Forefront-Antispam-Report: CIP:194.138.21.71;CTRY:DE;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:hybrid.siemens.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230001)(4636009)(40470700004)(46966006)(36840700001)(36756003)(70206006)(44832011)(81166007)(82960400001)(31686004)(2906002)(36860700001)(66574015)(5660300002)(8936002)(40460700003)(47076005)(86362001)(53546011)(83380400001)(4326008)(8676002)(16526019)(336012)(16576012)(316002)(110136005)(6706004)(54906003)(26005)(186003)(82310400004)(31696002)(356005)(508600001)(956004)(2616005)(70586007)(3940600001)(36900700001)(43740500002);DIR:OUT;SFP:1101; X-OriginatorOrg: siemens.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 25 Feb 2022 17:27:39.9553 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: c77228c5-412a-42e1-06f7-08d9f8841fa9 X-MS-Exchange-CrossTenant-Id: 38ae3bcd-9579-4fd4-adda-b42e1495d55a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=38ae3bcd-9579-4fd4-adda-b42e1495d55a;Ip=[194.138.21.71];Helo=[hybrid.siemens.com] X-MS-Exchange-CrossTenant-AuthSource: DB5EUR01FT031.eop-EUR01.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DB9PR10MB4650 X-TUID: eAMs5UqP5zGK On 24.02.22 16:42, Henning Schild wrote: > Am Thu, 24 Feb 2022 18:50:50 +0530 > schrieb vijai kumar : > >> Hi Henning, >> >> On Tue, Feb 22, 2022 at 8:01 PM Henning Schild >> wrote: >>> >>> Hey Vijai, >>> >>> Am Tue, 22 Feb 2022 16:04:36 +0530 >>> schrieb vijai kumar : >>> >>>> Problem: >>>> -------- >>>> We could have several CI jobs that are running in parallel in >>>> different nodes. One might want to consolidate and build a >>>> base-apt from the debs/deb-srcs of all these builds. >>> >>> Can you go into more detail. I do not yet get the problem. >> >> runner 1(Germany) -> Building de0 nano >> runner 2(India) -> Building qemuarm >> runner 3(US) -> Building qemuamd64 >> >> >> All these builds are running in different servers. >> If we wanted to create a single base-apt from all these servers, then >> we need to copy over their deb/debsrcs/base-apt to a common server and >> then >> create a consolidated repo. > > But why would you want to do that? I mean i get why you would want to > store all in the same location, but not why it should be one repo. > Maybe to save some space on sources and arch all .. but hey there are > ways of deduplcating on filesystem or block level. > You are just risking a weird local "all" package not being so "all" > after all ... false sharing. We want to auto-build a single, "offline" capable repo from the BoM accumulated from those builds of all possible targets. And that in a way that does not require pushing large artifacts between the build stages, ideally only those BoM lists. > >> This involves moving around this data. > > Yes, if it one central storage place. No matter if it is one "repo" or > many "repos" in i.e. folders. > >> The problem can be avoided if we have a single metadata produced by >> all these builds which would have details of all the packages the >> build used. >> Basically a manifest of the build. This manifest can be later used to >> recreate the repo which can be hosted later on for these jobs. > > We have a manifest for "image content" which already is fed into > clearing, it is a bill of materials an nothing else, it can not > be used to rebuild. > Even if you had all metadata you need to store sources and binaries > somewhere reliable, whether that is central or distributed is another > story. > Pointers to anything on the internet (including all debian repos) will > at some point stop working. So if "exact rebuilding" in a "far away > future" is what you want, mirroring is what you will need. Exactly, this mirror is supposed to be generated, and that shortly after the individual builds succeeded (in a common pipeline stage). That can fail as any build can fail if a referenced version picked up during bootstrap got dropped while building an image. > Partial mirroring based on base-apt even with sources will be shaky and > you will find yourself digging in snapshots again. But it will work. Yes, it works for us (you should know ;)). Jan -- Siemens AG, Technology Competence Center Embedded Linux