local apt repo cache not working as documented

local apt repo cache not working as documented

[Henning Schild]

Hi,

i tried building an image from a local cache, just like the docs
describe the process. On the second try debootstrap always complains
about gpg issues. It does not work on current next and also did not
work the day the feature was merged "2997d073dc1a".

Yet another feature that got merged without a test-case ...

@Andreas you are working with that, did you apply any local patches
that should be sent upstream?

Henning

Re: local apt repo cache not working as documented

[Jan Kiszka]

On 26.11.18 10:55, [ext] Henning Schild wrote:
> Hi,
> 
> i tried building an image from a local cache, just like the docs
> describe the process. On the second try debootstrap always complains
> about gpg issues. It does not work on current next and also did not
> work the day the feature was merged "2997d073dc1a".
> 
> Yet another feature that got merged without a test-case ...
> 
> @Andreas you are working with that, did you apply any local patches
> that should be sent upstream?
> 

Cedric, sounds like the issue you were discussing on Friday off-list.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

Re: local apt repo cache not working as documented

[Hombourger, Cedric]

Hi Jan

It does sound like. I am adding Amy to the thread. Amy, please “git send-email” your patch to isar-users@googlegroups.com (traveling to eastern regions today but should be online in the evening if you need me)

Thanks
Cedric

Sent from a mobile

> On 26 Nov 2018, at 10:57, Jan Kiszka <jan.kiszka@siemens.com> wrote:
> 
>> On 26.11.18 10:55, [ext] Henning Schild wrote:
>> Hi,
>> i tried building an image from a local cache, just like the docs
>> describe the process. On the second try debootstrap always complains
>> about gpg issues. It does not work on current next and also did not
>> work the day the feature was merged "2997d073dc1a".
>> Yet another feature that got merged without a test-case ...
>> @Andreas you are working with that, did you apply any local patches
>> that should be sent upstream?
> 
> Cedric, sounds like the issue you were discussing on Friday off-list.
> 
> Jan
> 
> -- 
> Siemens AG, Corporate Technology, CT RDA IOT SES-DE
> Corporate Competence Center Embedded Linux

Re: local apt repo cache not working as documented

[Andreas Reichel]

On Mon, Nov 26, 2018 at 10:55:20AM +0100, Henning Schild wrote:
> Hi,
> 
> i tried building an image from a local cache, just like the docs
> describe the process. On the second try debootstrap always complains
> about gpg issues. It does not work on current next and also did not
> work the day the feature was merged "2997d073dc1a".
> 
> Yet another feature that got merged without a test-case ...
> 
> @Andreas you are working with that, did you apply any local patches
> that should be sent upstream?
> 

Interesting. I used commit SHA 60db83cdca, Jan Kiszka, Mon Nov 5
directly on the next branch.

It worked as described in the user_manual.md

However: I only built the following 4 images:

multiconfig:qemuarm-stretch:isar-image-base
multiconfig:qemuarm64-stretch:isar-image-base
multiconfig:qemuamd64-stretch:isar-image-base
multiconfig:bananapi-stretch:isar-image-base

I could rebuild these offline without problems.

I did not use any merged master branch or commits after that.

Andreas

> Henning

-- 
Andreas Reichel
Dipl.-Phys. (Univ.)
Software Consultant

Andreas.Reichel@tngtech.com, +49-174-3180074
TNG Technology Consulting GmbH, Betastr. 13a, 85774 Unterfoehring
Geschaeftsfuehrer: Henrik Klagges, Dr. Robert Dahlke, Gerhard Mueller
Sitz: Unterfoehring * Amtsgericht Muenchen * HRB 135082

Re: local apt repo cache not working as documented

[Henning Schild]

Am Mon, 26 Nov 2018 11:10:27 +0100
schrieb Andreas Reichel <andreas.reichel.ext@siemens.com>:

> On Mon, Nov 26, 2018 at 10:55:20AM +0100, Henning Schild wrote:
> > Hi,
> > 
> > i tried building an image from a local cache, just like the docs
> > describe the process. On the second try debootstrap always complains
> > about gpg issues. It does not work on current next and also did not
> > work the day the feature was merged "2997d073dc1a".
> > 
> > Yet another feature that got merged without a test-case ...
> > 
> > @Andreas you are working with that, did you apply any local patches
> > that should be sent upstream?
> >   
> 
> Interesting. I used commit SHA 60db83cdca, Jan Kiszka, Mon Nov 5
> directly on the next branch.

I will try with this. Shortly after that we have "d40a9ac093c24", which
probably causes the problem we see in current next.
I guess in case we use the cache we should call debootstrap with the
arg to ignore gpg.

Henning

> It worked as described in the user_manual.md
> 
> However: I only built the following 4 images:
> 
> multiconfig:qemuarm-stretch:isar-image-base
> multiconfig:qemuarm64-stretch:isar-image-base
> multiconfig:qemuamd64-stretch:isar-image-base
> multiconfig:bananapi-stretch:isar-image-base
> 
> I could rebuild these offline without problems.
> 
> I did not use any merged master branch or commits after that.
> 
> Andreas
> 
> > Henning  
> 

Re: local apt repo cache not working as documented

[Maxim Yu. Osipov]

On 11/26/18 12:55 PM, Henning Schild wrote:
> Hi,
> 
> i tried building an image from a local cache, just like the docs
> describe the process. On the second try debootstrap always complains
> about gpg issues. It does not work on current next and also did not
> work the day the feature was merged "2997d073dc1a".

I've checked out 2997d073dc1a, followed instructions in 
doc/user-manual.md for multiconfig:qemuarm-stretch:isar-image-base - 
feature works.

As for the current 'next' - I confirm that feature doesn't work. As for 
a quick fix one may revert d40a9ac0, but proper fix would be adding 
signing of local repo I'm currently busy with.

Maxim.

> Yet another feature that got merged without a test-case ...
> 
> @Andreas you are working with that, did you apply any local patches
> that should be sent upstream?
> 
> Henning
> 


-- 
Maxim Osipov
ilbers GmbH
Maria-Merian-Str. 8
85521 Ottobrunn
Germany
+49 (151) 6517 6917
mosipov@ilbers.de
http://ilbers.de/
Commercial register Munich, HRB 214197
General Manager: Baurzhan Ismagulov

Re: local apt repo cache not working as documented

[Henning Schild]

Am Mon, 26 Nov 2018 14:01:35 +0300
schrieb "Maxim Yu. Osipov" <mosipov@ilbers.de>:

> On 11/26/18 12:55 PM, Henning Schild wrote:
> > Hi,
> > 
> > i tried building an image from a local cache, just like the docs
> > describe the process. On the second try debootstrap always complains
> > about gpg issues. It does not work on current next and also did not
> > work the day the feature was merged "2997d073dc1a".  
> 
> I've checked out 2997d073dc1a, followed instructions in 
> doc/user-manual.md for multiconfig:qemuarm-stretch:isar-image-base - 
> feature works.
> 
> As for the current 'next' - I confirm that feature doesn't work. As
> for a quick fix one may revert d40a9ac0, but proper fix would be
> adding signing of local repo I'm currently busy with.

And a CI test case. Did you disconnect from all networks when you
succeeded with 2997d?

Henning

> Maxim.
> 
> > Yet another feature that got merged without a test-case ...
> > 
> > @Andreas you are working with that, did you apply any local patches
> > that should be sent upstream?
> > 
> > Henning
> >   
> 
> 

Re: local apt repo cache not working as documented

[Maxim Yu. Osipov]

On 11/26/18 2:12 PM, Henning Schild wrote:
> Am Mon, 26 Nov 2018 14:01:35 +0300
> schrieb "Maxim Yu. Osipov" <mosipov@ilbers.de>:
> 
>> On 11/26/18 12:55 PM, Henning Schild wrote:
>>> Hi,
>>>
>>> i tried building an image from a local cache, just like the docs
>>> describe the process. On the second try debootstrap always complains
>>> about gpg issues. It does not work on current next and also did not
>>> work the day the feature was merged "2997d073dc1a".
>>
>> I've checked out 2997d073dc1a, followed instructions in
>> doc/user-manual.md for multiconfig:qemuarm-stretch:isar-image-base -
>> feature works.
>>
>> As for the current 'next' - I confirm that feature doesn't work. As
>> for a quick fix one may revert d40a9ac0, but proper fix would be
>> adding signing of local repo I'm currently busy with.
> 
> And a CI test case. Did you disconnect from all networks when you
> succeeded with 2997d?

Yes, I disabled networking.

Maxim.

> Henning
> 
>> Maxim.
>>
>>> Yet another feature that got merged without a test-case ...
>>>
>>> @Andreas you are working with that, did you apply any local patches
>>> that should be sent upstream?
>>>
>>> Henning
>>>    
>>
>>
> 


-- 
Maxim Osipov
ilbers GmbH
Maria-Merian-Str. 8
85521 Ottobrunn
Germany
+49 (151) 6517 6917
mosipov@ilbers.de
http://ilbers.de/
Commercial register Munich, HRB 214197
General Manager: Baurzhan Ismagulov

Re: local apt repo cache not working as documented

[Henning Schild]

Hi,

there is one more issue with that feature. The image you get will
contain an invalid sources.list and will not be "apt-get update"-able,
unlike the one we just "rebuild".

Henning

Am Mon, 26 Nov 2018 10:55:20 +0100
schrieb "[ext] Henning Schild" <henning.schild@siemens.com>:

> Hi,
> 
> i tried building an image from a local cache, just like the docs
> describe the process. On the second try debootstrap always complains
> about gpg issues. It does not work on current next and also did not
> work the day the feature was merged "2997d073dc1a".
> 
> Yet another feature that got merged without a test-case ...
> 
> @Andreas you are working with that, did you apply any local patches
> that should be sent upstream?
> 
> Henning
> 

Re: local apt repo cache not working as documented

[Baurzhan Ismagulov]

Hello Henning,

On Tue, Nov 27, 2018 at 10:45:02AM +0100, Henning Schild wrote:
> there is one more issue with that feature. The image you get will
> contain an invalid sources.list and will not be "apt-get update"-able,
> unlike the one we just "rebuild".

Do I understand correctly, you suggest to replace the local URI we bootstrapped
from with the upstream URI from meta-isar/conf/distro/<something>.list?

I agree that the local URI isn't of much use on the target (unless one
NFS-mounts the local repo), but I think that either variant is an arbitrary
product policy that won't fit every product. If they target reproducibility, it
isn't going to be a Debian URI anyway (if apt is to be used at all).

That said, for meta-isar -- which is a playground -- that would be convenient
for the user. Feel free to submit a patch.

With kind regards,
Baurzhan.

Re: local apt repo cache not working as documented

[Jan Kiszka]

On 28.11.18 10:00, Baurzhan Ismagulov wrote:
> Hello Henning,
> 
> On Tue, Nov 27, 2018 at 10:45:02AM +0100, Henning Schild wrote:
>> there is one more issue with that feature. The image you get will
>> contain an invalid sources.list and will not be "apt-get update"-able,
>> unlike the one we just "rebuild".
> 
> Do I understand correctly, you suggest to replace the local URI we bootstrapped
> from with the upstream URI from meta-isar/conf/distro/<something>.list?
> 
> I agree that the local URI isn't of much use on the target (unless one
> NFS-mounts the local repo), but I think that either variant is an arbitrary
> product policy that won't fit every product. If they target reproducibility, it
> isn't going to be a Debian URI anyway (if apt is to be used at all).
> 
> That said, for meta-isar -- which is a playground -- that would be convenient
> for the user. Feel free to submit a patch.

It's a default feature for images based on isar-image.bbclass that apt-get 
install works if the target has online connectivity. That should not be 
destroyed. It's actually exploited downstream (jailhouse-images, xenomai-images, 
and even at least one upcoming product).

Allowing to overwrite which repos should be used or even to remove packaging 
from the target completely are additional features on top.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

Re: local apt repo cache not working as documented

[Henning Schild]

Am Wed, 28 Nov 2018 10:04:36 +0100
schrieb "[ext] Jan Kiszka" <jan.kiszka@siemens.com>:

> On 28.11.18 10:00, Baurzhan Ismagulov wrote:
> > Hello Henning,
> > 
> > On Tue, Nov 27, 2018 at 10:45:02AM +0100, Henning Schild wrote:  
> >> there is one more issue with that feature. The image you get will
> >> contain an invalid sources.list and will not be "apt-get
> >> update"-able, unlike the one we just "rebuild".  
> > 
> > Do I understand correctly, you suggest to replace the local URI we
> > bootstrapped from with the upstream URI from
> > meta-isar/conf/distro/<something>.list?
> > 
> > I agree that the local URI isn't of much use on the target (unless
> > one NFS-mounts the local repo), but I think that either variant is
> > an arbitrary product policy that won't fit every product. If they
> > target reproducibility, it isn't going to be a Debian URI anyway
> > (if apt is to be used at all).
> > 
> > That said, for meta-isar -- which is a playground -- that would be
> > convenient for the user. Feel free to submit a patch.  
> 
> It's a default feature for images based on isar-image.bbclass that
> apt-get install works if the target has online connectivity. That
> should not be destroyed. It's actually exploited downstream
> (jailhouse-images, xenomai-images, and even at least one upcoming
> product).
> 
> Allowing to overwrite which repos should be used or even to remove
> packaging from the target completely are additional features on top.

That is all already solved in Isar. You have DISTRO_APT_SOURCES where
you select what should be used for installation and for delivery. The
caching mechanism needs to change "installation" but there is no excuse
to break "delivery".

So yes, you need to undo the patching before do_image and get the user
back to what they specified in DISTRO_APT_SOURCES.

Henning

> Jan
> 

Re: local apt repo cache not working as documented

[Baurzhan Ismagulov]

On Wed, Nov 28, 2018 at 04:13:59PM +0100, Henning Schild wrote:
> That is all already solved in Isar. You have DISTRO_APT_SOURCES where
> you select what should be used for installation and for delivery. The
> caching mechanism needs to change "installation" but there is no excuse
> to break "delivery".

The definition of "breaking" depends on whether upstream sources are required /
allowed in a particular product. But as discussed offline, it's convenient for
users, so we'll look at that.


> So yes, you need to undo the patching before do_image and get the user
> back to what they specified in DISTRO_APT_SOURCES.

Technically, it's actively patching the upstream URI into sources.list, since
it's created with the local URI by debootstrap.


With kind regards,
Baurzhan.

Re: local apt repo cache not working as documented

[Henning Schild]

Hi,

i was working on a more elegant solution than just the revert.
Basically something that would act on [trusted=yes] in the sources.list
and derive the debootstrap arg from that.
But turns out the [trusted=yes] enters the rootfs in another way than
debootstrap is tricked into using the other mirror.

Long story short, please apply the revert, the better solution will
have to wait.

Henning

Am Mon, 26 Nov 2018 10:55:20 +0100
schrieb "[ext] Henning Schild" <henning.schild@siemens.com>:

> Hi,
> 
> i tried building an image from a local cache, just like the docs
> describe the process. On the second try debootstrap always complains
> about gpg issues. It does not work on current next and also did not
> work the day the feature was merged "2997d073dc1a".
> 
> Yet another feature that got merged without a test-case ...
> 
> @Andreas you are working with that, did you apply any local patches
> that should be sent upstream?
> 
> Henning
>