Re: [PATCH] isar-bootstrap: Clear local apt state information

[Vijai Kumar K <vijaikumar.kanagarajan@gmail.com>]

On Thu, Nov 21, 2019 at 12:40:40PM +0100, Henning Schild wrote:
> Am Thu, 21 Nov 2019 15:24:01 +0530
> schrieb Vijai Kumar K <vijaikumar.kanagarajan@gmail.com>:
> 
> > On Thu, Nov 21, 2019 at 10:22:18AM +0100, Henning Schild wrote:
> > > Hi,
> > > 
> > > i am not sure i fully get the problem. With the first stage you
> > > probably mean "debootstrap" and the second stage the isar bootstrap
> > > on top.  
> > 
> > In case of crossdebootstrap, the bootstrapping runs in two stages.
> > Basically qemu-debootstrap takes care of that and calls debootstrap
> > --second-stage when it identifies the build as crossbuild.
> > 
> > The apt url passed in is not used in the second stage. Instead, it
> > uses deb.debian.org/debian and reuses the state information like
> > below(log from debootstrap --second-stage)
> > 
> > + mv debootstrap.invalid_dists_stretch_InRelease
> > deb.debian.org_debian_dists_stretch_InRelease
> > + mv debootstrap.invalid_dists_stretch_Release
> > deb.debian.org_debian_dists_stretch_Release
> > + mv debootstrap.invalid_dists_stretch_Release.gpg
> > deb.debian.org_debian_dists_stretch_Release.gpg
> > 
> > The debootstrap.invalid_* has the InRelease contents of my custom
> > repo(a minimal debian repo) passed in as a argument to
> > qemu-debootstrap. Post second stage the apt-state information is
> > messed up. Has the name of deb.debian.org but contents of my local
> > apt mirror. So, if apt-get update is called with this state, it would
> > fail when I have both deb.debian.org and the custom repo mentioned in
> > my DEBIAN_APT_SOURCES file.
> > 
> > > 
> > > The first stage should be using the primary entry from the
> > > sources.list, so as far as i remember we are just adding more
> > > entries. Meaning the state information should stay valid. Maybe in
> > > the offline "base-apt" case we have a flow that actually switches
> > > the upstream mirror.
> > > 
> > > Maybe you can go into detail how to reproduce the issue at hand,
> > > possibly add a test-case for it.  
> > 
> > It is one such corner case. Adding a test case for such a unique
> > situation is difficult since it involves custom repos.
> 
> I still do not fully get it.
> 
> For the test coverage of custom repos in general, and signed ones on
> top, we need to find a solution anyways.
> 
> I would prefer trying to treat the real ones as "custom". i.e. by
> removing the default gpg trust from debootstrap and adding it as custom
> trust. For this one this trick would not work.
> 
> The second obvious way, which requires some more coding is to use Isar
> itself to produce the custom repo. That is most likely the source of
> your custom one anyways, it is the source of the one Jan had his custom
> gpg keys for.

Ah, reading back my comment looks like I am totally against test case
for custom repo. I sounded wrong on that. What I was trying to say is we
would need to remove /var/lib/apt/lists/* since its a residue left over
by qemu-debootstrap. Not removing it has caused me an issue due to
the way how debootstrap tool handled the state information. Yes, we
should definitely do a test case for custom repo. But I dont think this
patch has any effect on that. It is just a proper cleanup before adding
our sources list.

Thanks,
Vijai Kumar K

> 
> Henning
> 
> > > 
> > > If the files should be deleted, my guess is that the deletion is
> > > related to the deletion of sources.list in line 282. So it might be
> > > a good idea to add your deletion right after or before that.  
> > 
> > Yes. I can move it over there.
> > 
> > 
> > Thanks,
> > Vijai Kumar K
> > 
> > > 
> > > Henning
> > > 
> > > Am Wed, 20 Nov 2019 16:23:40 +0530
> > > schrieb <vijaikumar.kanagarajan@gmail.com>:
> > >   
> > > > From: Vijai Kumar K <Vijaikumar_Kanagarajan@mentor.com>
> > > > 
> > > > Debootstrap's second stage switches to using the default debian
> > > > mirror. But the apt state information is renamed and used from the
> > > > first stage. This might cause apt-get update to fail with the
> > > > below error when the sources list contain both custom and
> > > > upstream repos.
> > > > 
> > > > | Reading package lists...
> > > > | E: Failed to fetch
> > > > http://deb.debian.org/debian/dists/stretch/InRelease \ |
> > > > Unable to find expected entry 'contrib/source/Sources' in Release
> > > > file \ |		(Wrong sources.list entry or malformed
> > > > file)
> > > > 
> > > > Clear the apt state information before apt-get update.
> > > > 
> > > > Signed-off-by: Vijai Kumar K <Vijaikumar_Kanagarajan@mentor.com>
> > > > ---
> > > >  meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 1 +
> > > >  1 file changed, 1 insertion(+)
> > > > 
> > > > diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
> > > > b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc index
> > > > d614d9a..5a379f7 100644 ---
> > > > a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc +++
> > > > b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc @@ -320,6
> > > > +320,7 @@ isar_bootstrap() { chroot "${ROOTFSDIR}" /usr/bin/dpkg
> > > > --add-architecture ${DISTRO_ARCH} fi
> > > >  
> > > > +            rm -rf "${ROOTFSDIR}/var/lib/apt/lists/"*
> > > >              chroot "${ROOTFSDIR}" /usr/bin/apt-get update -y
> > > >              chroot "${ROOTFSDIR}" /usr/bin/apt-get install -y -f
> > > >              chroot "${ROOTFSDIR}" /usr/bin/apt-get dist-upgrade
> > > > -y \  
> > >   
>