Re: [PATCH 3/4] build.sh: Update apt sources

[Jan Kiszka <jan.kiszka@siemens.com>]

On 2018-02-01 14:43, Alexander Smirnov wrote:
> On 02/01/2018 04:13 PM, Jan Kiszka wrote:
>> On 2018-02-01 12:48, Alexander Smirnov wrote:
>>> On 02/01/2018 02:40 PM, Jan Kiszka wrote:
>>>> On 2018-02-01 12:29, Alexander Smirnov wrote:
>>>>> Trigger apt sources update before building the package to be sure that
>>>>> we are in sync with isar-apt content.
>>>>>
>>>>> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
>>>>> ---
>>>>>    meta/recipes-devtools/buildchroot/files/build.sh | 3 +++
>>>>>    1 file changed, 3 insertions(+)
>>>>>
>>>>> diff --git a/meta/recipes-devtools/buildchroot/files/build.sh
>>>>> b/meta/recipes-devtools/buildchroot/files/build.sh
>>>>> index 77e0fdd..5c59193 100644
>>>>> --- a/meta/recipes-devtools/buildchroot/files/build.sh
>>>>> +++ b/meta/recipes-devtools/buildchroot/files/build.sh
>>>>> @@ -3,6 +3,9 @@
>>>>>    # This software is a part of ISAR.
>>>>>    # Copyright (C) 2015-2017 ilbers GmbH
>>>>>    +# Make sure that we have latest isar-apt content
>>>>> +apt-get update
>>>>> +
>>>>
>>>> Can we confine this to the isar-apt repo? Or is there no risk that an
>>>> upstream repo update will result in a buildchroot change that is
>>>> inconsistent with an earlier build stage?
>>>
>>> That's a good point, I've already checked it, it seems to be possible
>>> like:
>>>
>>> sudo apt-get update -o Dir::Etc::sourcelist="sources.list.d/${source}" \
>>> -o Dir::Etc::sourceparts="-" -o APT::Get::List-Cleanup="0"
>>>
>>> [https://askubuntu.com/questions/65245/apt-get-update-only-for-a-specific-repository]
>>>
>>>
>>>
>>> But I decided that it makes no sense, because target image will be
>>> generated later and use upstream apt. So if something has been changed
>>> during buildchroot operating, this will affect target image anyway.
>>
>> buildchroot shall not deviate from target rootfs. We will eventually
>> (rather sooner than later) have to provide it for SDK use, and than you
>> definitely don't want surprises, but actually already earlier.
>>
> 
> I mean that the issue is not in buildchroot, I'll fix this line to
> update isar-apt only, but this doesn't solve the issue of atomic
> upstream fetch. You can't generate target image using only buildchroot.
> So for example:
> 
>  - You've generated buildchroot SDK in Feb.
>  - You want to use this SDK and build target image in March.
> 
> No warranty that this will work until you prefetch packages for *both*
> target image and buildchroot.

That... is true. :)

> 
>>>
>>> Again I think this should be handled outside buildchroot by caching
>>> upstream apts.
>>
>> Well, do you plan to /always/ cache them? Last time I asked you said we
>> need both cases, cached and "live" ;). But even when live, we must keep
>> things atomic over a single build at least.
> 
> No, I meant a bit different thing: if build_rep assumes to cache
> *everything* exists in Isar tree, I'd like to have an option to disable
> it, I don't want to waste the time by downloading tons of unneeded
> packages.
> 
> In my opinion reproducibility should be always enabled and should cache
> everything needed for *the current* bitbake target, i.e.:
> 
> 1. $ bitbake example-hello
> Isar prefetches everything needed to build example-hello (buildchroot +
> build deps) and use local cache only.
> 
> 2. $ bitabke isar-image-base
> Isar prefetches everything needed to build target image (buildchroot +
> build deps + target image packages) and use local cache only.
> 
> 
> Without this Isar reproducibility and build correctness will be based on
> user's hopes and prays :-)

We don't disagree.

> 
>>
>> So, if you can avoid pulling from upstream here, please add that. We can
>> still drop it again when the external infrastructure can always
>> guarantee atomicity in the repo state.
>>
> 
> Ok.

Let's consider it a "mitigation" measure for "improving" consistency,
not able to guarantee it. As you correctly said, as long as we do not
cache and freeze our fetches, we are lost with the two rootfs'es we
create asynchronously.

Thanks,
Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux