From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 7067480740990025728 X-Received: by 2002:a7b:c38f:0:b0:385:e56c:8624 with SMTP id s15-20020a7bc38f000000b00385e56c8624mr17543161wmj.19.1646637819482; Sun, 06 Mar 2022 23:23:39 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a05:6000:1d94:b0:1f1:dc97:d9c4 with SMTP id bk20-20020a0560001d9400b001f1dc97d9c4ls110349wrb.3.gmail; Sun, 06 Mar 2022 23:23:38 -0800 (PST) X-Google-Smtp-Source: ABdhPJwM5/sJxTaP479Qr5/MV2cUEcFq1b6RAKBrCMRzagbqOttUEI5ihmh8gfGyrxQWgRJHxHGe X-Received: by 2002:a5d:648c:0:b0:1ed:b04d:300 with SMTP id o12-20020a5d648c000000b001edb04d0300mr4595006wri.347.1646637818513; Sun, 06 Mar 2022 23:23:38 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1646637818; cv=none; d=google.com; s=arc-20160816; b=HOsMFyV53G8ZGa8uFrjYLstgQW8C3rkPVUmx9X1pJ/thmYR9cWDSy93bVS6kYLPB3h eDSdm3LlNz88nou0AXDGB1nmH+bAYHU55UxyWE37QZNRTZKSPANWaUjlMddPaIoeqxvc /OvC6LBI1xQ8IKs6JV4SBlQnre+7Wbqolg963o0OY+jsfyGEnyTVroIpjdgZygdeJOmO igq8GwxtDC7fwoeLvg6D+pY+NNhz+Epmufn7VHYbnJ+mG+H+QtzPHLNW4sY1wQxawDrY SwGvIy179TopD90n9wothdHp+Yvt8AZz0X0s1JAZhaBHDJR26Se3CRpLpfQE0NnABljf nzSw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=to:subject:message-id:date:from:in-reply-to:references:mime-version :dkim-signature; bh=X92W16EW+1FnW7n3iv6RdP3GPbhVlHIyLI1k8d1hMms=; b=YokVfbn+nS0/q/KlzjWKz0UaOIMs5BtOO7a6S3jlB+7dGWNeq0D4CrQcUWRg+Rv4VJ zDJEZr5q9hWonnVR91tfKkJh8oL6LczrXRmmrGuMqGSqbd9E11WcKJkz/1DpR4CNtruL uNGJfCk0EeIDXe6DAKcYvc2+ny/wx+g7qPffz1qx/YrfYBtGf/68Zve+LeFUL+zabE68 g1Po54/qXGwFIGk0Vooj49NV14MRxUlHOwrkRgbr5IA5x/k8SIcdcfm3Sx9+kvTamqFu JjtoF4P2IysCPckChab2j/de9gMH8WJFtkhytabNjQ2VWoE6D2ysmn2D2qdtX0CPSZB4 Hisg== ARC-Authentication-Results: i=1; gmr-mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b=PyfswDUV; spf=pass (google.com: domain of vijaikumar.kanagarajan@gmail.com designates 2a00:1450:4864:20::429 as permitted sender) smtp.mailfrom=vijaikumar.kanagarajan@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from mail-wr1-x429.google.com (mail-wr1-x429.google.com. [2a00:1450:4864:20::429]) by gmr-mx.google.com with ESMTPS id 189-20020a1c02c6000000b00387e237f125si597067wmc.3.2022.03.06.23.23.38 for (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Sun, 06 Mar 2022 23:23:38 -0800 (PST) Received-SPF: pass (google.com: domain of vijaikumar.kanagarajan@gmail.com designates 2a00:1450:4864:20::429 as permitted sender) client-ip=2a00:1450:4864:20::429; Authentication-Results: gmr-mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b=PyfswDUV; spf=pass (google.com: domain of vijaikumar.kanagarajan@gmail.com designates 2a00:1450:4864:20::429 as permitted sender) smtp.mailfrom=vijaikumar.kanagarajan@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: by mail-wr1-x429.google.com with SMTP id p9so21656759wra.12 for ; Sun, 06 Mar 2022 23:23:38 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=X92W16EW+1FnW7n3iv6RdP3GPbhVlHIyLI1k8d1hMms=; b=PyfswDUVpv+SksmAv7smfaqk3LajtycVnA0p/ixvkbmqPmB9UkoXPaoF8jArBFp8fe d1LfvbCJUwbeXF7LTOq6+2hv4YMFz7TgS6plkOCM0QELgoIJwTmQ9/uRmhimrfaCUwb7 ooTv5+SNHtnEgYkoWq23u7guVuZ38eG0CDhO1fVLG9iss40gpSw/ByUakIO+qlMxmAmd oMsh/b0ZRCydh8Yv0amgxQrno9YyZot0AJWG4izh8lAfAzKg8QtqG9+me2A4ixET/t+J 52jfvmXU4hFMqapv5ng5lmX5IfuuNrT/lrfewVIC7w+kVjCxP6EvavEyEoxBAPKA4QT9 y80A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=X92W16EW+1FnW7n3iv6RdP3GPbhVlHIyLI1k8d1hMms=; b=A+vZiWiTeIcpzf1pUmfNB3ERgViU+LwIHWAohSFZKtMgxecEpu7JYBq1PFNwzF1hSj KtE3n9fO9j4za3pJklqYTu4VC0U/DsmxBjwd2ujJcHm3msyyXH8c3EpWggzUQOD14fLY MIJCtcGCNOzOjUS2y/rkZqVOb/Qiu5VqHAorrphGH4/kUAV7mjmzwDkAQ55SseLcjme3 TxGZfUBSVljyYxQgQiE04LBS0DfMdqx8zbXB4rHWaz82G3apO8aHdXIiHNkugk1k5Civ Vd8/Of6Z75uBYKEjTlhUWg2v8tahGSi5jIkEjfwQNf54THlLbj0z9WEWqkVU3SZhGZq8 KPeA== X-Gm-Message-State: AOAM533uMeNPpL9QjImsTq5sJuP59exQenki8O1nrat+YqDx7ncHykiw YBuhIMzMNkOPyrs20w17hr7EffKRgZQ8c6YIZH9e7t6L42E= X-Received: by 2002:a5d:628d:0:b0:1f1:d81d:d6da with SMTP id k13-20020a5d628d000000b001f1d81dd6damr6932068wru.20.1646637817905; Sun, 06 Mar 2022 23:23:37 -0800 (PST) MIME-Version: 1.0 References: <20220222153136.08432cb3@md1za8fc.ad001.siemens.net> <20220224164244.6e4bb002@md1za8fc.ad001.siemens.net> <45e6e9ba-8a05-e617-d5ae-949efb53c35b@siemens.com> In-Reply-To: From: vijai kumar Date: Mon, 7 Mar 2022 12:53:26 +0530 Message-ID: Subject: Re: [Discussion]: Metadata to consolidate and rebuild base-apt from distributed CI builds To: isar-users Content-Type: text/plain; charset="UTF-8" X-TUID: qZLASZ9EXtk2 On Fri, Mar 4, 2022 at 3:33 PM Baurzhan Ismagulov wrote: > > On Thu, Mar 03, 2022 at 07:15:40PM +0530, vijai kumar wrote: > > If we are in agreement then we can think about how to achieve this. > > There are changes coming in soon, so the implementation should take > > that into consideration. > > > > I am not sure if the caching part is reworked. If so having an idea on > > the design would definitely help; > > Thanks Vijai for the discussion. In short, we've already started further > base-apt improvement due to a number of reasons, e.g.: If there is already a branch for this activity with some initial implementations, can you please point to it? > > * Strict usage of base-apt for debootstrap and build-dep to ensure base-apt > correctness in any build. > > * Pluggability of debootstrap, which is necessary for multistrapping, sudo > removal, and maintainability. > > * We need to know which PN-PV is satisfiable from which location (base-apt, > isar-apt, bitbake) in order to use Debian Build-Depends in bitbake. > > python-apt provides the necessary functionality. After we have the above, more > necessary use cases become possible. E.g., storing and reusing built packages > in per-layer apt repos. > > We also want to have parallel building. For us, it comes more from the CI side, > as we have 3 h for fast and 10 h for full testsuite on the latest inexpensive > hardware. The first step would be to parallelize the testcases with storing of > intermediate results in a shared location. The second step would be extending > that to individual bitbake tasks. Maybe icecc would be good enough to cover > either or both, we have to test. > > Regarding your implementation proposal, I think that could be done. However, > I'd like to better understand the motivation first. Is it e.g. creating a > canonical repo for a given project? That would be easier to implement on top of > the above. It is for recreating a repo for a given project from some kind of manifest. This way we could avoid pushing the repos between multiple CI runners. The current goal is to create a single repo from multiple projects. We might have multiple projects running parallel in different CI runners, the idea is to create a single repo from all those builds without the need to push data around. So, some kind of project manifest. These manifests then can be used to create a single repo. Instead of copying over all the debs to a single location and trigger creation of base-apt. > > Regarding downloading time -- we had tested full local Debian mirrors and > didn't see any performance improvement of CI jobs. We haven't dug deeper, maybe > we have some parallelization killers in Isar. > > Regarding the central repo for remote building sites -- in my experience, it is > very slow, our customers end up installing local replication servers. > > We aim at full Debian support, be it packages, repos, or images. Debian, being > a binary server / desktop distribution and not a source-based development kit, > has a number of inflexibilities such as sudo, versioning, rules, etc.; we would > like to work towards more developer friendliness here. Bitbake and Yocto > contribute much here, and we would like to find a good working solution. > > That is why we welcome this use case and would like to work on that after > understanding the details. Jan told me you already had some implementations for > this. You also mention time and costs. Could you please share the concept > behind the work so far, and which time and costs you mean? Then we could > proceed step by step while having the big picture in mind. We thought about a few options, 1. Gather the download urls for all the packages we download. This could be our metadata. 2. Club all manifests at the end of build. (Buildchroot, isar-bootstrap & image rootfs) to recreate a master list of packages used in the build. 2 has some disadvantages. We have to probe buildchroot after image build to get a complete package list. Even then it doesnot capture the packages that are removed. 1 seems like a solution, It would have to be injected as part of the build, like how we injected downloading debs. The advantage it brings is that we don't necessarily need the apt sources information to recreate the repo. A simple wget would do. There is also a risk of urls becoming obsolete. There could be better solutions, maybe our new way of creating base-apt might help in creating metadata in a cleaner way. Thanks, Vijai Kumar K > > With kind regards, > Baurzhan. > > -- > You received this message because you are subscribed to the Google Groups "isar-users" group. > To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com. > To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/YiHj1KTffbhLxPl5%40ilbers.de.