* Re: [PATCH] dpkg-base: apt-get "update" before "source"
2019-02-11 12:53 ` Jan Kiszka
@ 2019-02-11 13:33 ` chombourger
2019-02-18 14:16 ` Maxim Yu. Osipov
2019-02-11 18:08 ` Henning Schild
2019-02-26 15:00 ` Henning Schild
2 siblings, 1 reply; 11+ messages in thread
From: chombourger @ 2019-02-11 13:33 UTC (permalink / raw)
To: isar-users
[-- Attachment #1.1: Type: text/plain, Size: 2645 bytes --]
Hi Jan,
Will need to recheck things here but there should be no .list files
referencing public repos when building from the local apt cache (and
therefore when you are wanting a reproducible build)
My local patch essentially does a full apt-get update but I now need to
check if I did not alter the expected behavior :)
Cedric
On Monday, February 11, 2019 at 1:53:09 PM UTC+1, Jan Kiszka wrote:
>
> On 11.02.19 10:35, [ext] Henning Schild wrote:
> > Here i see an often repeating pattern. That "apt-get update" is now
> > present in many entry points to the buildchroot.
> > I guess we should factor it out and put it into a central place. And
> > the rule of thumb probably is ... whenever you use anything apt,
> > apt-get update before you do ...
>
> If you are only talking about updating our locally maintained repo (like
> below),
> that is fine to factor out and reuse. However, we must not update against
> public
> repos after the initial pulling, in order to ensure we have a consistent
> package
> set along the whole build.
>
> Jan
>
> >
> > Henning
> >
> > Am Mon, 11 Feb 2019 10:33:24 +0100
> > schrieb Henning Schild <henning...@siemens.com <javascript:>>:
> >
> >> From: Henning Schild <henning...@siemens.com <javascript:>>
> >>
> >> When rebuilding we can run into an inconsistent view where buildchroot
> >> assumes it could download the sources of a modified upstream package.
> >> After a "reprepro ... remove" we always need to "apt-get update" to
> >> not operate on an old version of the metadata.
> >>
> >> Signed-off-by: Henning Schild <henning...@siemens.com <javascript:>>
> >> ---
> >> meta/classes/dpkg-base.bbclass | 4 ++++
> >> 1 file changed, 4 insertions(+)
> >>
> >> diff --git a/meta/classes/dpkg-base.bbclass
> >> b/meta/classes/dpkg-base.bbclass index 175dc80..5425df8 100644
> >> --- a/meta/classes/dpkg-base.bbclass
> >> +++ b/meta/classes/dpkg-base.bbclass
> >> @@ -31,6 +31,10 @@ do_apt_fetch() {
> >> fi
> >> dpkg_do_mounts
> >> E="${@ bb.utils.export_proxies(d)}"
> >> + sudo -E chroot ${BUILDCHROOT_DIR} /usr/bin/apt-get update \
> >> + -o
> >> Dir::Etc::sourcelist="sources.list.d/isar-apt.list" \
> >> + -o Dir::Etc::sourceparts="-" \
> >> + -o APT::Get::List-Cleanup="0"
> >> sudo -E chroot --userspec=$( id -u ):$( id -g )
> >> ${BUILDCHROOT_DIR} \ sh -c 'cd ${PP} && apt-get -y source ${SRC_APT}'
> >> dpkg_undo_mounts
> >
>
> --
> Siemens AG, Corporate Technology, CT RDA IOT SES-DE
> Corporate Competence Center Embedded Linux
>
[-- Attachment #1.2: Type: text/html, Size: 3982 bytes --]
^ permalink raw reply [flat|nested] 11+ messages in thread* Re: [PATCH] dpkg-base: apt-get "update" before "source"
2019-02-11 13:33 ` chombourger
@ 2019-02-18 14:16 ` Maxim Yu. Osipov
0 siblings, 0 replies; 11+ messages in thread
From: Maxim Yu. Osipov @ 2019-02-18 14:16 UTC (permalink / raw)
To: chombourger, isar-users
Hi Cedric,
Any updates on this topic?
Thanks,
Maxim.
On 2/11/19 2:33 PM, chombourger@gmail.com wrote:
> Hi Jan,
>
> Will need to recheck things here but there should be no .list files
> referencing public repos when building from the local apt cache (and
> therefore when you are wanting a reproducible build)
> My local patch essentially does a full apt-get update but I now need to
> check if I did not alter the expected behavior :)
>
> Cedric
>
> On Monday, February 11, 2019 at 1:53:09 PM UTC+1, Jan Kiszka wrote:
>
> On 11.02.19 10:35, [ext] Henning Schild wrote:
> > Here i see an often repeating pattern. That "apt-get update" is now
> > present in many entry points to the buildchroot.
> > I guess we should factor it out and put it into a central place. And
> > the rule of thumb probably is ... whenever you use anything apt,
> > apt-get update before you do ...
>
> If you are only talking about updating our locally maintained repo
> (like below),
> that is fine to factor out and reuse. However, we must not update
> against public
> repos after the initial pulling, in order to ensure we have a
> consistent package
> set along the whole build.
>
> Jan
>
> >
> > Henning
> >
> > Am Mon, 11 Feb 2019 10:33:24 +0100
> > schrieb Henning Schild <henning...@siemens.com <javascript:>>:
> >
> >> From: Henning Schild <henning...@siemens.com <javascript:>>
> >>
> >> When rebuilding we can run into an inconsistent view where
> buildchroot
> >> assumes it could download the sources of a modified upstream
> package.
> >> After a "reprepro ... remove" we always need to "apt-get update" to
> >> not operate on an old version of the metadata.
> >>
> >> Signed-off-by: Henning Schild <henning...@siemens.com
> <javascript:>>
> >> ---
> >> meta/classes/dpkg-base.bbclass | 4 ++++
> >> 1 file changed, 4 insertions(+)
> >>
> >> diff --git a/meta/classes/dpkg-base.bbclass
> >> b/meta/classes/dpkg-base.bbclass index 175dc80..5425df8 100644
> >> --- a/meta/classes/dpkg-base.bbclass
> >> +++ b/meta/classes/dpkg-base.bbclass
> >> @@ -31,6 +31,10 @@ do_apt_fetch() {
> >> fi
> >> dpkg_do_mounts
> >> E="${@ bb.utils.export_proxies(d)}"
> >> + sudo -E chroot ${BUILDCHROOT_DIR} /usr/bin/apt-get
> update \
> >> + -o
> >> Dir::Etc::sourcelist="sources.list.d/isar-apt.list" \
> >> + -o Dir::Etc::sourceparts="-" \
> >> + -o APT::Get::List-Cleanup="0"
> >> sudo -E chroot --userspec=$( id -u ):$( id -g )
> >> ${BUILDCHROOT_DIR} \ sh -c 'cd ${PP} && apt-get -y source
> ${SRC_APT}'
> >> dpkg_undo_mounts
> >
>
> --
> Siemens AG, Corporate Technology, CT RDA IOT SES-DE
> Corporate Competence Center Embedded Linux
>
> --
> You received this message because you are subscribed to the Google
> Groups "isar-users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to isar-users+unsubscribe@googlegroups.com
> <mailto:isar-users+unsubscribe@googlegroups.com>.
> To post to this group, send email to isar-users@googlegroups.com
> <mailto:isar-users@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/isar-users/b9ca5913-f1be-441d-9e2c-3304bc6c5868%40googlegroups.com
> <https://groups.google.com/d/msgid/isar-users/b9ca5913-f1be-441d-9e2c-3304bc6c5868%40googlegroups.com?utm_medium=email&utm_source=footer>.
> For more options, visit https://groups.google.com/d/optout.
--
Maxim Osipov
ilbers GmbH
Maria-Merian-Str. 8
85521 Ottobrunn
Germany
+49 (151) 6517 6917
mosipov@ilbers.de
http://ilbers.de/
Commercial register Munich, HRB 214197
General Manager: Baurzhan Ismagulov
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [PATCH] dpkg-base: apt-get "update" before "source"
2019-02-11 12:53 ` Jan Kiszka
2019-02-11 13:33 ` chombourger
@ 2019-02-11 18:08 ` Henning Schild
2019-02-26 15:00 ` Henning Schild
2 siblings, 0 replies; 11+ messages in thread
From: Henning Schild @ 2019-02-11 18:08 UTC (permalink / raw)
To: Jan Kiszka; +Cc: isar-users
Am Mon, 11 Feb 2019 13:53:07 +0100
schrieb Jan Kiszka <jan.kiszka@siemens.com>:
> On 11.02.19 10:35, [ext] Henning Schild wrote:
> > Here i see an often repeating pattern. That "apt-get update" is now
> > present in many entry points to the buildchroot.
> > I guess we should factor it out and put it into a central place. And
> > the rule of thumb probably is ... whenever you use anything apt,
> > apt-get update before you do ...
>
> If you are only talking about updating our locally maintained repo
> (like below), that is fine to factor out and reuse. However, we must
> not update against public repos after the initial pulling, in order
> to ensure we have a consistent package set along the whole build.
Sure. In fact all the ones we reprepro with between partial builds. At
the moment just the one, but i guess there is more to come.
In which case it might be a good idea to switch to using
Dir::Etc::SourceParts with a value of i.e.
"sources.list.isar.d", or have all the repos with that feature on
one .list file.
Henning
> Jan
>
> >
> > Henning
> >
> > Am Mon, 11 Feb 2019 10:33:24 +0100
> > schrieb Henning Schild <henning.schild@siemens.com>:
> >
> >> From: Henning Schild <henning.schild@siemens.com>
> >>
> >> When rebuilding we can run into an inconsistent view where
> >> buildchroot assumes it could download the sources of a modified
> >> upstream package. After a "reprepro ... remove" we always need to
> >> "apt-get update" to not operate on an old version of the metadata.
> >>
> >> Signed-off-by: Henning Schild <henning.schild@siemens.com>
> >> ---
> >> meta/classes/dpkg-base.bbclass | 4 ++++
> >> 1 file changed, 4 insertions(+)
> >>
> >> diff --git a/meta/classes/dpkg-base.bbclass
> >> b/meta/classes/dpkg-base.bbclass index 175dc80..5425df8 100644
> >> --- a/meta/classes/dpkg-base.bbclass
> >> +++ b/meta/classes/dpkg-base.bbclass
> >> @@ -31,6 +31,10 @@ do_apt_fetch() {
> >> fi
> >> dpkg_do_mounts
> >> E="${@ bb.utils.export_proxies(d)}"
> >> + sudo -E chroot ${BUILDCHROOT_DIR} /usr/bin/apt-get update
> >> \
> >> + -o
> >> Dir::Etc::sourcelist="sources.list.d/isar-apt.list" \
> >> + -o Dir::Etc::sourceparts="-" \
> >> + -o APT::Get::List-Cleanup="0"
> >> sudo -E chroot --userspec=$( id -u ):$( id -g )
> >> ${BUILDCHROOT_DIR} \ sh -c 'cd ${PP} && apt-get -y source
> >> ${SRC_APT}' dpkg_undo_mounts
> >
>
^ permalink raw reply [flat|nested] 11+ messages in thread* Re: [PATCH] dpkg-base: apt-get "update" before "source"
2019-02-11 12:53 ` Jan Kiszka
2019-02-11 13:33 ` chombourger
2019-02-11 18:08 ` Henning Schild
@ 2019-02-26 15:00 ` Henning Schild
2019-02-26 15:03 ` Jan Kiszka
2 siblings, 1 reply; 11+ messages in thread
From: Henning Schild @ 2019-02-26 15:00 UTC (permalink / raw)
To: Jan Kiszka; +Cc: isar-users
Am Mon, 11 Feb 2019 13:53:07 +0100
schrieb Jan Kiszka <jan.kiszka@siemens.com>:
> On 11.02.19 10:35, [ext] Henning Schild wrote:
> > Here i see an often repeating pattern. That "apt-get update" is now
> > present in many entry points to the buildchroot.
> > I guess we should factor it out and put it into a central place. And
> > the rule of thumb probably is ... whenever you use anything apt,
> > apt-get update before you do ...
>
> If you are only talking about updating our locally maintained repo
> (like below), that is fine to factor out and reuse. However, we must
> not update against public repos after the initial pulling, in order
> to ensure we have a consistent package set along the whole build.
I think that is actually not true. The whole process is an "eventual
consistency" topic. Say one of your package compiles for 2 weeks ...
you would never be able to build a buster image. And in fact, no matter
how long the time windows is. You can always hit the point where a file
that should be there according to your view of the repo, can not be
downloaded anymore .... in which case you need to "update/upgrade".
So i tend to believe that we should always "apt-get update" without
restrictions. For people that truly care about having a reproducible
build with only one consistent state, they should create images from
there partial debian mirror / cache. (where an "apt-get update" is a
noop)
Henning
> Jan
>
> >
> > Henning
> >
> > Am Mon, 11 Feb 2019 10:33:24 +0100
> > schrieb Henning Schild <henning.schild@siemens.com>:
> >
> >> From: Henning Schild <henning.schild@siemens.com>
> >>
> >> When rebuilding we can run into an inconsistent view where
> >> buildchroot assumes it could download the sources of a modified
> >> upstream package. After a "reprepro ... remove" we always need to
> >> "apt-get update" to not operate on an old version of the metadata.
> >>
> >> Signed-off-by: Henning Schild <henning.schild@siemens.com>
> >> ---
> >> meta/classes/dpkg-base.bbclass | 4 ++++
> >> 1 file changed, 4 insertions(+)
> >>
> >> diff --git a/meta/classes/dpkg-base.bbclass
> >> b/meta/classes/dpkg-base.bbclass index 175dc80..5425df8 100644
> >> --- a/meta/classes/dpkg-base.bbclass
> >> +++ b/meta/classes/dpkg-base.bbclass
> >> @@ -31,6 +31,10 @@ do_apt_fetch() {
> >> fi
> >> dpkg_do_mounts
> >> E="${@ bb.utils.export_proxies(d)}"
> >> + sudo -E chroot ${BUILDCHROOT_DIR} /usr/bin/apt-get update
> >> \
> >> + -o
> >> Dir::Etc::sourcelist="sources.list.d/isar-apt.list" \
> >> + -o Dir::Etc::sourceparts="-" \
> >> + -o APT::Get::List-Cleanup="0"
> >> sudo -E chroot --userspec=$( id -u ):$( id -g )
> >> ${BUILDCHROOT_DIR} \ sh -c 'cd ${PP} && apt-get -y source
> >> ${SRC_APT}' dpkg_undo_mounts
> >
>
^ permalink raw reply [flat|nested] 11+ messages in thread* Re: [PATCH] dpkg-base: apt-get "update" before "source"
2019-02-26 15:00 ` Henning Schild
@ 2019-02-26 15:03 ` Jan Kiszka
2019-03-05 10:11 ` Henning Schild
0 siblings, 1 reply; 11+ messages in thread
From: Jan Kiszka @ 2019-02-26 15:03 UTC (permalink / raw)
To: Henning Schild; +Cc: isar-users
On 26.02.19 16:00, Henning Schild wrote:
> Am Mon, 11 Feb 2019 13:53:07 +0100
> schrieb Jan Kiszka <jan.kiszka@siemens.com>:
>
>> On 11.02.19 10:35, [ext] Henning Schild wrote:
>>> Here i see an often repeating pattern. That "apt-get update" is now
>>> present in many entry points to the buildchroot.
>>> I guess we should factor it out and put it into a central place. And
>>> the rule of thumb probably is ... whenever you use anything apt,
>>> apt-get update before you do ...
>>
>> If you are only talking about updating our locally maintained repo
>> (like below), that is fine to factor out and reuse. However, we must
>> not update against public repos after the initial pulling, in order
>> to ensure we have a consistent package set along the whole build.
>
> I think that is actually not true. The whole process is an "eventual
> consistency" topic. Say one of your package compiles for 2 weeks ...
Then you will not be able to work. This is no realistic scenario.
> you would never be able to build a buster image. And in fact, no matter
> how long the time windows is. You can always hit the point where a file
> that should be there according to your view of the repo, can not be
> downloaded anymore .... in which case you need to "update/upgrade".
>
> So i tend to believe that we should always "apt-get update" without
> restrictions. For people that truly care about having a reproducible
Nope, we won't do that - unless we have a stable (versioned and frozen) repo.
This does not apply to upstream Debian, so the current pattern is the way to go:
single update for a single build.
Jan
--
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [PATCH] dpkg-base: apt-get "update" before "source"
2019-02-26 15:03 ` Jan Kiszka
@ 2019-03-05 10:11 ` Henning Schild
0 siblings, 0 replies; 11+ messages in thread
From: Henning Schild @ 2019-03-05 10:11 UTC (permalink / raw)
To: Jan Kiszka; +Cc: isar-users
Am Tue, 26 Feb 2019 16:03:42 +0100
schrieb Jan Kiszka <jan.kiszka@siemens.com>:
> On 26.02.19 16:00, Henning Schild wrote:
> > Am Mon, 11 Feb 2019 13:53:07 +0100
> > schrieb Jan Kiszka <jan.kiszka@siemens.com>:
> >
> >> On 11.02.19 10:35, [ext] Henning Schild wrote:
> >>> Here i see an often repeating pattern. That "apt-get update" is
> >>> now present in many entry points to the buildchroot.
> >>> I guess we should factor it out and put it into a central place.
> >>> And the rule of thumb probably is ... whenever you use anything
> >>> apt, apt-get update before you do ...
> >>
> >> If you are only talking about updating our locally maintained repo
> >> (like below), that is fine to factor out and reuse. However, we
> >> must not update against public repos after the initial pulling, in
> >> order to ensure we have a consistent package set along the whole
> >> build.
> >
> > I think that is actually not true. The whole process is an "eventual
> > consistency" topic. Say one of your package compiles for 2
> > weeks ...
>
> Then you will not be able to work. This is no realistic scenario.
Sure, just trying to stress the point that your local view of your
mirror can expire any time. ( 2 weeks or 2 minutes ... )
> > you would never be able to build a buster image. And in fact, no
> > matter how long the time windows is. You can always hit the point
> > where a file that should be there according to your view of the
> > repo, can not be downloaded anymore .... in which case you need to
> > "update/upgrade".
> >
> > So i tend to believe that we should always "apt-get update" without
> > restrictions. For people that truly care about having a
> > reproducible
>
> Nope, we won't do that - unless we have a stable (versioned and
> frozen) repo. This does not apply to upstream Debian, so the current
> pattern is the way to go: single update for a single build.
Several updates would most likely be a noop anyways but ensure that
every "apt-get install" will succeed. We do have the cache and everyone
can have a versioned and frozen repo.
I just started this topic again because Andreas introduced a full
"apt-get update" to deal with the gpg topic.
Henning
> Jan
>
^ permalink raw reply [flat|nested] 11+ messages in thread