* [PATCH] isar-bootstrap: Clear local apt state information @ 2019-11-20 10:53 vijaikumar.kanagarajan 2019-11-21 9:22 ` Henning Schild 2019-11-21 10:01 ` Jan Kiszka 0 siblings, 2 replies; 12+ messages in thread From: vijaikumar.kanagarajan @ 2019-11-20 10:53 UTC (permalink / raw) To: isar-users Cc: henning.schild, ibr, claudius.heine.ext, jan.kiszka, Vijai Kumar K From: Vijai Kumar K <Vijaikumar_Kanagarajan@mentor.com> Debootstrap's second stage switches to using the default debian mirror. But the apt state information is renamed and used from the first stage. This might cause apt-get update to fail with the below error when the sources list contain both custom and upstream repos. | Reading package lists... | E: Failed to fetch http://deb.debian.org/debian/dists/stretch/InRelease \ | Unable to find expected entry 'contrib/source/Sources' in Release file \ | (Wrong sources.list entry or malformed file) Clear the apt state information before apt-get update. Signed-off-by: Vijai Kumar K <Vijaikumar_Kanagarajan@mentor.com> --- meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 1 + 1 file changed, 1 insertion(+) diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc index d614d9a..5a379f7 100644 --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc @@ -320,6 +320,7 @@ isar_bootstrap() { chroot "${ROOTFSDIR}" /usr/bin/dpkg --add-architecture ${DISTRO_ARCH} fi + rm -rf "${ROOTFSDIR}/var/lib/apt/lists/"* chroot "${ROOTFSDIR}" /usr/bin/apt-get update -y chroot "${ROOTFSDIR}" /usr/bin/apt-get install -y -f chroot "${ROOTFSDIR}" /usr/bin/apt-get dist-upgrade -y \ -- 2.17.1 ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH] isar-bootstrap: Clear local apt state information 2019-11-20 10:53 [PATCH] isar-bootstrap: Clear local apt state information vijaikumar.kanagarajan @ 2019-11-21 9:22 ` Henning Schild 2019-11-21 9:54 ` Vijai Kumar K 2019-11-21 10:01 ` Jan Kiszka 1 sibling, 1 reply; 12+ messages in thread From: Henning Schild @ 2019-11-21 9:22 UTC (permalink / raw) To: vijaikumar.kanagarajan; +Cc: isar-users, Vijai Kumar K Hi, i am not sure i fully get the problem. With the first stage you probably mean "debootstrap" and the second stage the isar bootstrap on top. The first stage should be using the primary entry from the sources.list, so as far as i remember we are just adding more entries. Meaning the state information should stay valid. Maybe in the offline "base-apt" case we have a flow that actually switches the upstream mirror. Maybe you can go into detail how to reproduce the issue at hand, possibly add a test-case for it. If the files should be deleted, my guess is that the deletion is related to the deletion of sources.list in line 282. So it might be a good idea to add your deletion right after or before that. Henning Am Wed, 20 Nov 2019 16:23:40 +0530 schrieb <vijaikumar.kanagarajan@gmail.com>: > From: Vijai Kumar K <Vijaikumar_Kanagarajan@mentor.com> > > Debootstrap's second stage switches to using the default debian > mirror. But the apt state information is renamed and used from the > first stage. This might cause apt-get update to fail with the below > error when the sources list contain both custom and upstream repos. > > | Reading package lists... > | E: Failed to fetch > http://deb.debian.org/debian/dists/stretch/InRelease \ | > Unable to find expected entry 'contrib/source/Sources' in Release > file \ | (Wrong sources.list entry or malformed file) > > Clear the apt state information before apt-get update. > > Signed-off-by: Vijai Kumar K <Vijaikumar_Kanagarajan@mentor.com> > --- > meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 1 + > 1 file changed, 1 insertion(+) > > diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc > b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc index > d614d9a..5a379f7 100644 --- > a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc +++ > b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc @@ -320,6 > +320,7 @@ isar_bootstrap() { chroot "${ROOTFSDIR}" /usr/bin/dpkg > --add-architecture ${DISTRO_ARCH} fi > > + rm -rf "${ROOTFSDIR}/var/lib/apt/lists/"* > chroot "${ROOTFSDIR}" /usr/bin/apt-get update -y > chroot "${ROOTFSDIR}" /usr/bin/apt-get install -y -f > chroot "${ROOTFSDIR}" /usr/bin/apt-get dist-upgrade -y \ ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH] isar-bootstrap: Clear local apt state information 2019-11-21 9:22 ` Henning Schild @ 2019-11-21 9:54 ` Vijai Kumar K 2019-11-21 10:05 ` [PATCH v2] " vijaikumar.kanagarajan 2019-11-21 11:40 ` [PATCH] " Henning Schild 0 siblings, 2 replies; 12+ messages in thread From: Vijai Kumar K @ 2019-11-21 9:54 UTC (permalink / raw) To: Henning Schild; +Cc: vijaikumar.kanagarajan, isar-users On Thu, Nov 21, 2019 at 10:22:18AM +0100, Henning Schild wrote: > Hi, > > i am not sure i fully get the problem. With the first stage you > probably mean "debootstrap" and the second stage the isar bootstrap on > top. In case of crossdebootstrap, the bootstrapping runs in two stages. Basically qemu-debootstrap takes care of that and calls debootstrap --second-stage when it identifies the build as crossbuild. The apt url passed in is not used in the second stage. Instead, it uses deb.debian.org/debian and reuses the state information like below(log from debootstrap --second-stage) + mv debootstrap.invalid_dists_stretch_InRelease deb.debian.org_debian_dists_stretch_InRelease + mv debootstrap.invalid_dists_stretch_Release deb.debian.org_debian_dists_stretch_Release + mv debootstrap.invalid_dists_stretch_Release.gpg deb.debian.org_debian_dists_stretch_Release.gpg The debootstrap.invalid_* has the InRelease contents of my custom repo(a minimal debian repo) passed in as a argument to qemu-debootstrap. Post second stage the apt-state information is messed up. Has the name of deb.debian.org but contents of my local apt mirror. So, if apt-get update is called with this state, it would fail when I have both deb.debian.org and the custom repo mentioned in my DEBIAN_APT_SOURCES file. > > The first stage should be using the primary entry from the > sources.list, so as far as i remember we are just adding more entries. > Meaning the state information should stay valid. Maybe in the offline > "base-apt" case we have a flow that actually switches the upstream > mirror. > > Maybe you can go into detail how to reproduce the issue at hand, > possibly add a test-case for it. It is one such corner case. Adding a test case for such a unique situation is difficult since it involves custom repos. > > If the files should be deleted, my guess is that the deletion is > related to the deletion of sources.list in line 282. So it might be a > good idea to add your deletion right after or before that. Yes. I can move it over there. Thanks, Vijai Kumar K > > Henning > > Am Wed, 20 Nov 2019 16:23:40 +0530 > schrieb <vijaikumar.kanagarajan@gmail.com>: > > > From: Vijai Kumar K <Vijaikumar_Kanagarajan@mentor.com> > > > > Debootstrap's second stage switches to using the default debian > > mirror. But the apt state information is renamed and used from the > > first stage. This might cause apt-get update to fail with the below > > error when the sources list contain both custom and upstream repos. > > > > | Reading package lists... > > | E: Failed to fetch > > http://deb.debian.org/debian/dists/stretch/InRelease \ | > > Unable to find expected entry 'contrib/source/Sources' in Release > > file \ | (Wrong sources.list entry or malformed file) > > > > Clear the apt state information before apt-get update. > > > > Signed-off-by: Vijai Kumar K <Vijaikumar_Kanagarajan@mentor.com> > > --- > > meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 1 + > > 1 file changed, 1 insertion(+) > > > > diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc > > b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc index > > d614d9a..5a379f7 100644 --- > > a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc +++ > > b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc @@ -320,6 > > +320,7 @@ isar_bootstrap() { chroot "${ROOTFSDIR}" /usr/bin/dpkg > > --add-architecture ${DISTRO_ARCH} fi > > > > + rm -rf "${ROOTFSDIR}/var/lib/apt/lists/"* > > chroot "${ROOTFSDIR}" /usr/bin/apt-get update -y > > chroot "${ROOTFSDIR}" /usr/bin/apt-get install -y -f > > chroot "${ROOTFSDIR}" /usr/bin/apt-get dist-upgrade -y \ > ^ permalink raw reply [flat|nested] 12+ messages in thread
* [PATCH v2] isar-bootstrap: Clear local apt state information 2019-11-21 9:54 ` Vijai Kumar K @ 2019-11-21 10:05 ` vijaikumar.kanagarajan 2019-11-25 6:38 ` Jan Kiszka 2019-11-28 15:21 ` Baurzhan Ismagulov 2019-11-21 11:40 ` [PATCH] " Henning Schild 1 sibling, 2 replies; 12+ messages in thread From: vijaikumar.kanagarajan @ 2019-11-21 10:05 UTC (permalink / raw) To: isar-users Cc: henning.schild, ibr, claudius.heine.ext, jan.kiszka, Vijai Kumar K From: Vijai Kumar K <Vijaikumar_Kanagarajan@mentor.com> Debootstrap's second stage switches to using the default debian mirror. But the apt state information is renamed and used from the first stage. This might cause apt-get update to fail with the below error when the sources list contain both custom and upstream repos. | Reading package lists... | E: Failed to fetch http://deb.debian.org/debian/dists/stretch/InRelease \ | Unable to find expected entry 'contrib/source/Sources' in Release file \ | (Wrong sources.list entry or malformed file) Clear the apt state information before apt-get update. Signed-off-by: Vijai Kumar K <Vijaikumar_Kanagarajan@mentor.com> --- Changes in v2: - Addressed review comments from Henning. meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 1 + 1 file changed, 1 insertion(+) diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc index d614d9a..9096775 100644 --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc @@ -280,6 +280,7 @@ isar_bootstrap() { install -v -m644 "${APTSRCS_INIT}" \ "${ROOTFSDIR}/etc/apt/sources-list" rm -f "${ROOTFSDIR}/etc/apt/sources.list" + rm -rf "${ROOTFSDIR}/var/lib/apt/lists/"* mkdir -p "${ROOTFSDIR}/etc/apt/apt.conf.d" install -v -m644 "${WORKDIR}/isar-apt.conf" \ "${ROOTFSDIR}/etc/apt/apt.conf.d/50isar.conf" -- 2.17.1 ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH v2] isar-bootstrap: Clear local apt state information 2019-11-21 10:05 ` [PATCH v2] " vijaikumar.kanagarajan @ 2019-11-25 6:38 ` Jan Kiszka 2019-11-25 6:46 ` Vijai Kumar K 2019-11-28 15:21 ` Baurzhan Ismagulov 1 sibling, 1 reply; 12+ messages in thread From: Jan Kiszka @ 2019-11-25 6:38 UTC (permalink / raw) To: vijaikumar.kanagarajan, isar-users; +Cc: henning.schild, ibr, Vijai Kumar K On 21.11.19 11:05, vijaikumar.kanagarajan@gmail.com wrote: > From: Vijai Kumar K <Vijaikumar_Kanagarajan@mentor.com> > > Debootstrap's second stage switches to using the default debian mirror. > But the apt state information is renamed and used from the first stage. > This might cause apt-get update to fail with the below error when the > sources list contain both custom and upstream repos. > > | Reading package lists... > | E: Failed to fetch http://deb.debian.org/debian/dists/stretch/InRelease \ > | Unable to find expected entry 'contrib/source/Sources' in Release file \ > | (Wrong sources.list entry or malformed file) > > Clear the apt state information before apt-get update. > > Signed-off-by: Vijai Kumar K <Vijaikumar_Kanagarajan@mentor.com> > --- > Changes in v2: > - Addressed review comments from Henning. > > meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 1 + > 1 file changed, 1 insertion(+) > > diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc > index d614d9a..9096775 100644 > --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc > +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc > @@ -280,6 +280,7 @@ isar_bootstrap() { > install -v -m644 "${APTSRCS_INIT}" \ > "${ROOTFSDIR}/etc/apt/sources-list" > rm -f "${ROOTFSDIR}/etc/apt/sources.list" > + rm -rf "${ROOTFSDIR}/var/lib/apt/lists/"* Normally, you would put the wildcard inside the quotes. Jan > mkdir -p "${ROOTFSDIR}/etc/apt/apt.conf.d" > install -v -m644 "${WORKDIR}/isar-apt.conf" \ > "${ROOTFSDIR}/etc/apt/apt.conf.d/50isar.conf" > -- Siemens AG, Corporate Technology, CT RDA IOT SES-DE Corporate Competence Center Embedded Linux ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH v2] isar-bootstrap: Clear local apt state information 2019-11-25 6:38 ` Jan Kiszka @ 2019-11-25 6:46 ` Vijai Kumar K 2019-11-25 7:00 ` Jan Kiszka 0 siblings, 1 reply; 12+ messages in thread From: Vijai Kumar K @ 2019-11-25 6:46 UTC (permalink / raw) To: Jan Kiszka; +Cc: vijaikumar.kanagarajan, isar-users, henning.schild, ibr On Mon, Nov 25, 2019 at 07:38:18AM +0100, Jan Kiszka wrote: > On 21.11.19 11:05, vijaikumar.kanagarajan@gmail.com wrote: > > From: Vijai Kumar K <Vijaikumar_Kanagarajan@mentor.com> > > > > Debootstrap's second stage switches to using the default debian mirror. > > But the apt state information is renamed and used from the first stage. > > This might cause apt-get update to fail with the below error when the > > sources list contain both custom and upstream repos. > > > > | Reading package lists... > > | E: Failed to fetch http://deb.debian.org/debian/dists/stretch/InRelease \ > > | Unable to find expected entry 'contrib/source/Sources' in Release file \ > > | (Wrong sources.list entry or malformed file) > > > > Clear the apt state information before apt-get update. > > > > Signed-off-by: Vijai Kumar K <Vijaikumar_Kanagarajan@mentor.com> > > --- > > Changes in v2: > > - Addressed review comments from Henning. > > > > meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 1 + > > 1 file changed, 1 insertion(+) > > > > diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc > > index d614d9a..9096775 100644 > > --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc > > +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc > > @@ -280,6 +280,7 @@ isar_bootstrap() { > > install -v -m644 "${APTSRCS_INIT}" \ > > "${ROOTFSDIR}/etc/apt/sources-list" > > rm -f "${ROOTFSDIR}/etc/apt/sources.list" > > + rm -rf "${ROOTFSDIR}/var/lib/apt/lists/"* > > Normally, you would put the wildcard inside the quotes. > > Jan Unfortunately file globbing doesnot work under quotes. Another alternative is to use quotes only around the variable as "${ROOTFSDIR}"/var/lib/apt/lists/* Thanks, Vijai Kumar K > > > mkdir -p "${ROOTFSDIR}/etc/apt/apt.conf.d" > > install -v -m644 "${WORKDIR}/isar-apt.conf" \ > > "${ROOTFSDIR}/etc/apt/apt.conf.d/50isar.conf" > > > > -- > Siemens AG, Corporate Technology, CT RDA IOT SES-DE > Corporate Competence Center Embedded Linux ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH v2] isar-bootstrap: Clear local apt state information 2019-11-25 6:46 ` Vijai Kumar K @ 2019-11-25 7:00 ` Jan Kiszka 0 siblings, 0 replies; 12+ messages in thread From: Jan Kiszka @ 2019-11-25 7:00 UTC (permalink / raw) To: Vijai Kumar K; +Cc: isar-users, henning.schild, ibr On 25.11.19 07:46, Vijai Kumar K wrote: > On Mon, Nov 25, 2019 at 07:38:18AM +0100, Jan Kiszka wrote: >> On 21.11.19 11:05, vijaikumar.kanagarajan@gmail.com wrote: >>> From: Vijai Kumar K <Vijaikumar_Kanagarajan@mentor.com> >>> >>> Debootstrap's second stage switches to using the default debian mirror. >>> But the apt state information is renamed and used from the first stage. >>> This might cause apt-get update to fail with the below error when the >>> sources list contain both custom and upstream repos. >>> >>> | Reading package lists... >>> | E: Failed to fetch http://deb.debian.org/debian/dists/stretch/InRelease \ >>> | Unable to find expected entry 'contrib/source/Sources' in Release file \ >>> | (Wrong sources.list entry or malformed file) >>> >>> Clear the apt state information before apt-get update. >>> >>> Signed-off-by: Vijai Kumar K <Vijaikumar_Kanagarajan@mentor.com> >>> --- >>> Changes in v2: >>> - Addressed review comments from Henning. >>> >>> meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 1 + >>> 1 file changed, 1 insertion(+) >>> >>> diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc >>> index d614d9a..9096775 100644 >>> --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc >>> +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc >>> @@ -280,6 +280,7 @@ isar_bootstrap() { >>> install -v -m644 "${APTSRCS_INIT}" \ >>> "${ROOTFSDIR}/etc/apt/sources-list" >>> rm -f "${ROOTFSDIR}/etc/apt/sources.list" >>> + rm -rf "${ROOTFSDIR}/var/lib/apt/lists/"* >> >> Normally, you would put the wildcard inside the quotes. >> >> Jan > > Unfortunately file globbing doesnot work under quotes. Another alternative is to use quotes only around > the variable as "${ROOTFSDIR}"/var/lib/apt/lists/* > Ah, yes, that was nonsense. Jan > Thanks, > Vijai Kumar K > >> >>> mkdir -p "${ROOTFSDIR}/etc/apt/apt.conf.d" >>> install -v -m644 "${WORKDIR}/isar-apt.conf" \ >>> "${ROOTFSDIR}/etc/apt/apt.conf.d/50isar.conf" >>> >> >> -- >> Siemens AG, Corporate Technology, CT RDA IOT SES-DE >> Corporate Competence Center Embedded Linux -- Siemens AG, Corporate Technology, CT RDA IOT SES-DE Corporate Competence Center Embedded Linux ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH v2] isar-bootstrap: Clear local apt state information 2019-11-21 10:05 ` [PATCH v2] " vijaikumar.kanagarajan 2019-11-25 6:38 ` Jan Kiszka @ 2019-11-28 15:21 ` Baurzhan Ismagulov 1 sibling, 0 replies; 12+ messages in thread From: Baurzhan Ismagulov @ 2019-11-28 15:21 UTC (permalink / raw) To: isar-users On Thu, Nov 21, 2019 at 03:35:32PM +0530, vijaikumar.kanagarajan@gmail.com wrote: > Debootstrap's second stage switches to using the default debian mirror. > But the apt state information is renamed and used from the first stage. > This might cause apt-get update to fail with the below error when the > sources list contain both custom and upstream repos. > > | Reading package lists... > | E: Failed to fetch http://deb.debian.org/debian/dists/stretch/InRelease \ > | Unable to find expected entry 'contrib/source/Sources' in Release file \ > | (Wrong sources.list entry or malformed file) > > Clear the apt state information before apt-get update. Applied to next, thanks. With kind regards, Baurzhan. ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH] isar-bootstrap: Clear local apt state information 2019-11-21 9:54 ` Vijai Kumar K 2019-11-21 10:05 ` [PATCH v2] " vijaikumar.kanagarajan @ 2019-11-21 11:40 ` Henning Schild 2019-11-21 12:09 ` Vijai Kumar K 1 sibling, 1 reply; 12+ messages in thread From: Henning Schild @ 2019-11-21 11:40 UTC (permalink / raw) To: Vijai Kumar K; +Cc: isar-users Am Thu, 21 Nov 2019 15:24:01 +0530 schrieb Vijai Kumar K <vijaikumar.kanagarajan@gmail.com>: > On Thu, Nov 21, 2019 at 10:22:18AM +0100, Henning Schild wrote: > > Hi, > > > > i am not sure i fully get the problem. With the first stage you > > probably mean "debootstrap" and the second stage the isar bootstrap > > on top. > > In case of crossdebootstrap, the bootstrapping runs in two stages. > Basically qemu-debootstrap takes care of that and calls debootstrap > --second-stage when it identifies the build as crossbuild. > > The apt url passed in is not used in the second stage. Instead, it > uses deb.debian.org/debian and reuses the state information like > below(log from debootstrap --second-stage) > > + mv debootstrap.invalid_dists_stretch_InRelease > deb.debian.org_debian_dists_stretch_InRelease > + mv debootstrap.invalid_dists_stretch_Release > deb.debian.org_debian_dists_stretch_Release > + mv debootstrap.invalid_dists_stretch_Release.gpg > deb.debian.org_debian_dists_stretch_Release.gpg > > The debootstrap.invalid_* has the InRelease contents of my custom > repo(a minimal debian repo) passed in as a argument to > qemu-debootstrap. Post second stage the apt-state information is > messed up. Has the name of deb.debian.org but contents of my local > apt mirror. So, if apt-get update is called with this state, it would > fail when I have both deb.debian.org and the custom repo mentioned in > my DEBIAN_APT_SOURCES file. > > > > > The first stage should be using the primary entry from the > > sources.list, so as far as i remember we are just adding more > > entries. Meaning the state information should stay valid. Maybe in > > the offline "base-apt" case we have a flow that actually switches > > the upstream mirror. > > > > Maybe you can go into detail how to reproduce the issue at hand, > > possibly add a test-case for it. > > It is one such corner case. Adding a test case for such a unique > situation is difficult since it involves custom repos. I still do not fully get it. For the test coverage of custom repos in general, and signed ones on top, we need to find a solution anyways. I would prefer trying to treat the real ones as "custom". i.e. by removing the default gpg trust from debootstrap and adding it as custom trust. For this one this trick would not work. The second obvious way, which requires some more coding is to use Isar itself to produce the custom repo. That is most likely the source of your custom one anyways, it is the source of the one Jan had his custom gpg keys for. Henning > > > > If the files should be deleted, my guess is that the deletion is > > related to the deletion of sources.list in line 282. So it might be > > a good idea to add your deletion right after or before that. > > Yes. I can move it over there. > > > Thanks, > Vijai Kumar K > > > > > Henning > > > > Am Wed, 20 Nov 2019 16:23:40 +0530 > > schrieb <vijaikumar.kanagarajan@gmail.com>: > > > > > From: Vijai Kumar K <Vijaikumar_Kanagarajan@mentor.com> > > > > > > Debootstrap's second stage switches to using the default debian > > > mirror. But the apt state information is renamed and used from the > > > first stage. This might cause apt-get update to fail with the > > > below error when the sources list contain both custom and > > > upstream repos. > > > > > > | Reading package lists... > > > | E: Failed to fetch > > > http://deb.debian.org/debian/dists/stretch/InRelease \ | > > > Unable to find expected entry 'contrib/source/Sources' in Release > > > file \ | (Wrong sources.list entry or malformed > > > file) > > > > > > Clear the apt state information before apt-get update. > > > > > > Signed-off-by: Vijai Kumar K <Vijaikumar_Kanagarajan@mentor.com> > > > --- > > > meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 1 + > > > 1 file changed, 1 insertion(+) > > > > > > diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc > > > b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc index > > > d614d9a..5a379f7 100644 --- > > > a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc +++ > > > b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc @@ -320,6 > > > +320,7 @@ isar_bootstrap() { chroot "${ROOTFSDIR}" /usr/bin/dpkg > > > --add-architecture ${DISTRO_ARCH} fi > > > > > > + rm -rf "${ROOTFSDIR}/var/lib/apt/lists/"* > > > chroot "${ROOTFSDIR}" /usr/bin/apt-get update -y > > > chroot "${ROOTFSDIR}" /usr/bin/apt-get install -y -f > > > chroot "${ROOTFSDIR}" /usr/bin/apt-get dist-upgrade > > > -y \ > > ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH] isar-bootstrap: Clear local apt state information 2019-11-21 11:40 ` [PATCH] " Henning Schild @ 2019-11-21 12:09 ` Vijai Kumar K 0 siblings, 0 replies; 12+ messages in thread From: Vijai Kumar K @ 2019-11-21 12:09 UTC (permalink / raw) To: Henning Schild; +Cc: Vijai Kumar K, isar-users On Thu, Nov 21, 2019 at 12:40:40PM +0100, Henning Schild wrote: > Am Thu, 21 Nov 2019 15:24:01 +0530 > schrieb Vijai Kumar K <vijaikumar.kanagarajan@gmail.com>: > > > On Thu, Nov 21, 2019 at 10:22:18AM +0100, Henning Schild wrote: > > > Hi, > > > > > > i am not sure i fully get the problem. With the first stage you > > > probably mean "debootstrap" and the second stage the isar bootstrap > > > on top. > > > > In case of crossdebootstrap, the bootstrapping runs in two stages. > > Basically qemu-debootstrap takes care of that and calls debootstrap > > --second-stage when it identifies the build as crossbuild. > > > > The apt url passed in is not used in the second stage. Instead, it > > uses deb.debian.org/debian and reuses the state information like > > below(log from debootstrap --second-stage) > > > > + mv debootstrap.invalid_dists_stretch_InRelease > > deb.debian.org_debian_dists_stretch_InRelease > > + mv debootstrap.invalid_dists_stretch_Release > > deb.debian.org_debian_dists_stretch_Release > > + mv debootstrap.invalid_dists_stretch_Release.gpg > > deb.debian.org_debian_dists_stretch_Release.gpg > > > > The debootstrap.invalid_* has the InRelease contents of my custom > > repo(a minimal debian repo) passed in as a argument to > > qemu-debootstrap. Post second stage the apt-state information is > > messed up. Has the name of deb.debian.org but contents of my local > > apt mirror. So, if apt-get update is called with this state, it would > > fail when I have both deb.debian.org and the custom repo mentioned in > > my DEBIAN_APT_SOURCES file. > > > > > > > > The first stage should be using the primary entry from the > > > sources.list, so as far as i remember we are just adding more > > > entries. Meaning the state information should stay valid. Maybe in > > > the offline "base-apt" case we have a flow that actually switches > > > the upstream mirror. > > > > > > Maybe you can go into detail how to reproduce the issue at hand, > > > possibly add a test-case for it. > > > > It is one such corner case. Adding a test case for such a unique > > situation is difficult since it involves custom repos. > > I still do not fully get it. > > For the test coverage of custom repos in general, and signed ones on > top, we need to find a solution anyways. > > I would prefer trying to treat the real ones as "custom". i.e. by > removing the default gpg trust from debootstrap and adding it as custom > trust. For this one this trick would not work. > > The second obvious way, which requires some more coding is to use Isar > itself to produce the custom repo. That is most likely the source of > your custom one anyways, it is the source of the one Jan had his custom > gpg keys for. Ah, reading back my comment looks like I am totally against test case for custom repo. I sounded wrong on that. What I was trying to say is we would need to remove /var/lib/apt/lists/* since its a residue left over by qemu-debootstrap. Not removing it has caused me an issue due to the way how debootstrap tool handled the state information. Yes, we should definitely do a test case for custom repo. But I dont think this patch has any effect on that. It is just a proper cleanup before adding our sources list. Thanks, Vijai Kumar K > > Henning > > > > > > > If the files should be deleted, my guess is that the deletion is > > > related to the deletion of sources.list in line 282. So it might be > > > a good idea to add your deletion right after or before that. > > > > Yes. I can move it over there. > > > > > > Thanks, > > Vijai Kumar K > > > > > > > > Henning > > > > > > Am Wed, 20 Nov 2019 16:23:40 +0530 > > > schrieb <vijaikumar.kanagarajan@gmail.com>: > > > > > > > From: Vijai Kumar K <Vijaikumar_Kanagarajan@mentor.com> > > > > > > > > Debootstrap's second stage switches to using the default debian > > > > mirror. But the apt state information is renamed and used from the > > > > first stage. This might cause apt-get update to fail with the > > > > below error when the sources list contain both custom and > > > > upstream repos. > > > > > > > > | Reading package lists... > > > > | E: Failed to fetch > > > > http://deb.debian.org/debian/dists/stretch/InRelease \ | > > > > Unable to find expected entry 'contrib/source/Sources' in Release > > > > file \ | (Wrong sources.list entry or malformed > > > > file) > > > > > > > > Clear the apt state information before apt-get update. > > > > > > > > Signed-off-by: Vijai Kumar K <Vijaikumar_Kanagarajan@mentor.com> > > > > --- > > > > meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 1 + > > > > 1 file changed, 1 insertion(+) > > > > > > > > diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc > > > > b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc index > > > > d614d9a..5a379f7 100644 --- > > > > a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc +++ > > > > b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc @@ -320,6 > > > > +320,7 @@ isar_bootstrap() { chroot "${ROOTFSDIR}" /usr/bin/dpkg > > > > --add-architecture ${DISTRO_ARCH} fi > > > > > > > > + rm -rf "${ROOTFSDIR}/var/lib/apt/lists/"* > > > > chroot "${ROOTFSDIR}" /usr/bin/apt-get update -y > > > > chroot "${ROOTFSDIR}" /usr/bin/apt-get install -y -f > > > > chroot "${ROOTFSDIR}" /usr/bin/apt-get dist-upgrade > > > > -y \ > > > > ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH] isar-bootstrap: Clear local apt state information 2019-11-20 10:53 [PATCH] isar-bootstrap: Clear local apt state information vijaikumar.kanagarajan 2019-11-21 9:22 ` Henning Schild @ 2019-11-21 10:01 ` Jan Kiszka 2019-11-21 10:14 ` Vijai Kumar K 1 sibling, 1 reply; 12+ messages in thread From: Jan Kiszka @ 2019-11-21 10:01 UTC (permalink / raw) To: vijaikumar.kanagarajan, isar-users; +Cc: henning.schild, ibr, Vijai Kumar K On 20.11.19 11:53, vijaikumar.kanagarajan@gmail.com wrote: > From: Vijai Kumar K <Vijaikumar_Kanagarajan@mentor.com> > > Debootstrap's second stage switches to using the default debian mirror. > But the apt state information is renamed and used from the first stage. > This might cause apt-get update to fail with the below error when the > sources list contain both custom and upstream repos. > > | Reading package lists... > | E: Failed to fetch http://deb.debian.org/debian/dists/stretch/InRelease \ > | Unable to find expected entry 'contrib/source/Sources' in Release file \ > | (Wrong sources.list entry or malformed file) > > Clear the apt state information before apt-get update. I had problems the other day with bootstrapping behind our proxy when there was both a local and the upstream repo involved: Reading package lists... E: Repository 'http://cdn-fastly.deb.debian.org/debian buster InRelease' changed its 'Origin' value from '' to 'Debian' E: Repository 'http://cdn-fastly.deb.debian.org/debian buster InRelease' changed its 'Label' value from '' to 'Debian' E: Repository 'http://cdn-fastly.deb.debian.org/debian buster InRelease' changed its 'Suite' value from '' to 'stable' Would that be resolved by your patch as well? Can reproduce ATM, that's why I'm asking. Jan > > Signed-off-by: Vijai Kumar K <Vijaikumar_Kanagarajan@mentor.com> > --- > meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 1 + > 1 file changed, 1 insertion(+) > > diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc > index d614d9a..5a379f7 100644 > --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc > +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc > @@ -320,6 +320,7 @@ isar_bootstrap() { > chroot "${ROOTFSDIR}" /usr/bin/dpkg --add-architecture ${DISTRO_ARCH} > fi > > + rm -rf "${ROOTFSDIR}/var/lib/apt/lists/"* > chroot "${ROOTFSDIR}" /usr/bin/apt-get update -y > chroot "${ROOTFSDIR}" /usr/bin/apt-get install -y -f > chroot "${ROOTFSDIR}" /usr/bin/apt-get dist-upgrade -y \ > -- Siemens AG, Corporate Technology, CT RDA IOT SES-DE Corporate Competence Center Embedded Linux ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH] isar-bootstrap: Clear local apt state information 2019-11-21 10:01 ` Jan Kiszka @ 2019-11-21 10:14 ` Vijai Kumar K 0 siblings, 0 replies; 12+ messages in thread From: Vijai Kumar K @ 2019-11-21 10:14 UTC (permalink / raw) To: Jan Kiszka; +Cc: vijaikumar.kanagarajan, isar-users, henning.schild, ibr On Thu, Nov 21, 2019 at 11:01:31AM +0100, Jan Kiszka wrote: > On 20.11.19 11:53, vijaikumar.kanagarajan@gmail.com wrote: > > From: Vijai Kumar K <Vijaikumar_Kanagarajan@mentor.com> > > > > Debootstrap's second stage switches to using the default debian mirror. > > But the apt state information is renamed and used from the first stage. > > This might cause apt-get update to fail with the below error when the > > sources list contain both custom and upstream repos. > > > > | Reading package lists... > > | E: Failed to fetch http://deb.debian.org/debian/dists/stretch/InRelease \ > > | Unable to find expected entry 'contrib/source/Sources' in Release file \ > > | (Wrong sources.list entry or malformed file) > > > > Clear the apt state information before apt-get update. > > I had problems the other day with bootstrapping behind our proxy when there > was both a local and the upstream repo involved: > > Reading package lists... > > > > E: Repository 'http://cdn-fastly.deb.debian.org/debian buster InRelease' > changed its 'Origin' value from '' to 'Debian' > > > E: Repository 'http://cdn-fastly.deb.debian.org/debian buster InRelease' > changed its 'Label' value from '' to 'Debian' > > > E: Repository 'http://cdn-fastly.deb.debian.org/debian buster InRelease' > changed its 'Suite' value from '' to 'stable' > > > > Would that be resolved by your patch as well? Can reproduce ATM, that's why > I'm asking. Possibily. But, I am not sure. You could give it a try. Thanks, Vijai Kumar K > > Jan > > > > > Signed-off-by: Vijai Kumar K <Vijaikumar_Kanagarajan@mentor.com> > > --- > > meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 1 + > > 1 file changed, 1 insertion(+) > > > > diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc > > index d614d9a..5a379f7 100644 > > --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc > > +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc > > @@ -320,6 +320,7 @@ isar_bootstrap() { > > chroot "${ROOTFSDIR}" /usr/bin/dpkg --add-architecture ${DISTRO_ARCH} > > fi > > + rm -rf "${ROOTFSDIR}/var/lib/apt/lists/"* > > chroot "${ROOTFSDIR}" /usr/bin/apt-get update -y > > chroot "${ROOTFSDIR}" /usr/bin/apt-get install -y -f > > chroot "${ROOTFSDIR}" /usr/bin/apt-get dist-upgrade -y \ > > > > > -- > Siemens AG, Corporate Technology, CT RDA IOT SES-DE > Corporate Competence Center Embedded Linux > > -- > You received this message because you are subscribed to the Google Groups "isar-users" group. > To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com. > To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/ed6a0a83-7189-d260-2df1-e7b0378ccd76%40siemens.com. ^ permalink raw reply [flat|nested] 12+ messages in thread
end of thread, other threads:[~2019-11-28 15:21 UTC | newest] Thread overview: 12+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2019-11-20 10:53 [PATCH] isar-bootstrap: Clear local apt state information vijaikumar.kanagarajan 2019-11-21 9:22 ` Henning Schild 2019-11-21 9:54 ` Vijai Kumar K 2019-11-21 10:05 ` [PATCH v2] " vijaikumar.kanagarajan 2019-11-25 6:38 ` Jan Kiszka 2019-11-25 6:46 ` Vijai Kumar K 2019-11-25 7:00 ` Jan Kiszka 2019-11-28 15:21 ` Baurzhan Ismagulov 2019-11-21 11:40 ` [PATCH] " Henning Schild 2019-11-21 12:09 ` Vijai Kumar K 2019-11-21 10:01 ` Jan Kiszka 2019-11-21 10:14 ` Vijai Kumar K
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox