* Re: [PATCH] image: copy .list files from SRC_URI to the image
2024-09-23 14:27 [PATCH] image: copy .list files from SRC_URI to the image 'Cedric Hombourger' via isar-users
@ 2024-09-25 16:04 ` 'Jan Kiszka' via isar-users
2024-09-25 19:22 ` 'cedric.hombourger@siemens.com' via isar-users
2024-10-04 11:36 ` Uladzimir Bely
2024-10-08 5:12 ` Uladzimir Bely
2 siblings, 1 reply; 5+ messages in thread
From: 'Jan Kiszka' via isar-users @ 2024-09-25 16:04 UTC (permalink / raw)
To: Cedric Hombourger, isar-users, Moessbauer, Felix (T CED SES-DE)
On 23.09.24 16:27, 'Cedric Hombourger' via isar-users wrote:
> There are use-cases where the .list files used by Isar / apt during the
> build would not be the same as the initial .list files presented to the
> end-user of the image (e.g. build from base-apt or from a local mirror).
> If any .list files are listed in the SRC_URI of an image recipe, they
> will be copied to /etc/apt/sources.list.d/ verbatim and in lieu of the
> bootstrap sources that were used by Isar.
How well does this interact with the list swappings we do already, e.g.
for using snapshots during build but then upstream on the target?
Jan
>
> Signed-off-by: Cedric Hombourger <cedric.hombourger@siemens.com>
> ---
> doc/user_manual.md | 14 ++++++++++++++
> meta/classes/image.bbclass | 18 ++++++++++++++++++
> 2 files changed, 32 insertions(+)
>
> diff --git a/doc/user_manual.md b/doc/user_manual.md
> index 847f4b07..89c3a293 100644
> --- a/doc/user_manual.md
> +++ b/doc/user_manual.md
> @@ -769,6 +769,20 @@ inherit image
>
> ```
>
> +If the resulting image should not ship apt sources used during the build but custom ones (e.g. for end-users to point
> +to an external or simply different server when they "apt-get update", custom list files may be listed in `SRC_URI`:
> +Isar will copy them to `/etc/apt/sources.list.d/` and omit bootstrap sources. Possible use-cases:
> +
> + * image built from base-apt (which is by definition local to the build host)
> +
> + * image built from an internal mirror, not reachable by devices running the produced image
> +
> + * ship template list files for the end-user to edit (e.g. letting him uncomment `deb` or `deb-src` entries)
> +
> +It should be noted that Isar will not validate or even load supplied list files: they are simply copied verbatim to
> +the root file-system just before creating an image out of it (loading sources from the network would make the build
> +non-reproducible).
> +
> ### Additional Notes
>
> The distribution selected via the `DISTRO` variable may need to run a post-configuration script after the root file-system
> diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
> index c29d9e26..0c162aa3 100644
> --- a/meta/classes/image.bbclass
> +++ b/meta/classes/image.bbclass
> @@ -392,6 +392,18 @@ python do_deploy() {
> }
> addtask deploy before do_build after do_image
>
> +def apt_list_files(d):
> + lists = []
> + sources = d.getVar("SRC_URI").split()
> + for s in sources:
> + _, _, local, _, _, parm = bb.fetch.decodeurl(s)
> + base, ext = os.path.splitext(os.path.basename(local))
> + if ext and ext in (".list"):
> + lists.append(local)
> + return lists
> +
> +IMAGE_LISTS = "${@ ' '.join(apt_list_files(d)) }"
> +
> do_rootfs_finalize() {
> sudo -s <<'EOSUDO'
> set -e
> @@ -429,6 +441,12 @@ do_rootfs_finalize() {
> mv "${ROOTFSDIR}/etc/apt/sources-list" \
> "${ROOTFSDIR}/etc/apt/sources.list.d/bootstrap.list"
> fi
> + if [ -n "${IMAGE_LISTS}" ]; then
> + rm -f "${ROOTFSDIR}/etc/apt/sources.list.d/bootstrap.list"
> + for l in ${IMAGE_LISTS}; do
> + cp "${WORKDIR}"/${l} "${ROOTFSDIR}/etc/apt/sources.list.d/"
> + done
> + fi
>
> rm -f "${ROOTFSDIR}/run/blkid/blkid.tab"
> rm -f "${ROOTFSDIR}/run/blkid/blkid.tab.old"
--
Siemens AG, Technology
Linux Expert Center
--
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/d6062f14-fea7-418e-a7f5-cccb00d76512%40siemens.com.
^ permalink raw reply [flat|nested] 5+ messages in thread* Re: [PATCH] image: copy .list files from SRC_URI to the image
2024-09-25 16:04 ` 'Jan Kiszka' via isar-users
@ 2024-09-25 19:22 ` 'cedric.hombourger@siemens.com' via isar-users
0 siblings, 0 replies; 5+ messages in thread
From: 'cedric.hombourger@siemens.com' via isar-users @ 2024-09-25 19:22 UTC (permalink / raw)
To: isar-users, Kiszka, Jan, MOESSBAUER, Felix
On Wed, 2024-09-25 at 18:04 +0200, Jan Kiszka wrote:
> On 23.09.24 16:27, 'Cedric Hombourger' via isar-users wrote:
> > There are use-cases where the .list files used by Isar / apt during
> > the
> > build would not be the same as the initial .list files presented to
> > the
> > end-user of the image (e.g. build from base-apt or from a local
> > mirror).
> > If any .list files are listed in the SRC_URI of an image recipe,
> > they
> > will be copied to /etc/apt/sources.list.d/ verbatim and in lieu of
> > the
> > bootstrap sources that were used by Isar.
>
> How well does this interact with the list swappings we do already,
> e.g.
> for using snapshots during build but then upstream on the target?
Rechecked to be sure. Lines above my changes are responsible for the
swap (/etc/apt/sources-list => /etc/apt/sources.list.d/bootstrap.list)
head of /etc/apt/sources.list.d/bootstrap.list before
do_rootfs_finalize()
deb [check-valid-until=no]
http://snapshot.debian.org/archive/debian/20240925T180156Z/ bookworm
main contrib non-free-firmware non-free
head of /etc/apt/sources-list before do_rootfs_finalize()
deb http://deb.debian.org/debian bookworm main contrib non-free-
firmware non-free
After do_rootfs_finalize(), sources-list is gone and bootstrap.list is
no lomger pointing to the selected snapshot
With my changes and with user-supplied .list files in SRC_URI
bootstrap.list is removed and only user-supplied .list files are
present in /etc/apt/sources.list.d/
Hope this clarifies
Cedric
>
> Jan
>
> >
> > Signed-off-by: Cedric Hombourger <cedric.hombourger@siemens.com>
> > ---
> > doc/user_manual.md | 14 ++++++++++++++
> > meta/classes/image.bbclass | 18 ++++++++++++++++++
> > 2 files changed, 32 insertions(+)
> >
> > diff --git a/doc/user_manual.md b/doc/user_manual.md
> > index 847f4b07..89c3a293 100644
> > --- a/doc/user_manual.md
> > +++ b/doc/user_manual.md
> > @@ -769,6 +769,20 @@ inherit image
> >
> > ```
> >
> > +If the resulting image should not ship apt sources used during the
> > build but custom ones (e.g. for end-users to point
> > +to an external or simply different server when they "apt-get
> > update", custom list files may be listed in `SRC_URI`:
> > +Isar will copy them to `/etc/apt/sources.list.d/` and omit
> > bootstrap sources. Possible use-cases:
> > +
> > + * image built from base-apt (which is by definition local to the
> > build host)
> > +
> > + * image built from an internal mirror, not reachable by devices
> > running the produced image
> > +
> > + * ship template list files for the end-user to edit (e.g. letting
> > him uncomment `deb` or `deb-src` entries)
> > +
> > +It should be noted that Isar will not validate or even load
> > supplied list files: they are simply copied verbatim to
> > +the root file-system just before creating an image out of it
> > (loading sources from the network would make the build
> > +non-reproducible).
> > +
> > ### Additional Notes
> >
> > The distribution selected via the `DISTRO` variable may need to
> > run a post-configuration script after the root file-system
> > diff --git a/meta/classes/image.bbclass
> > b/meta/classes/image.bbclass
> > index c29d9e26..0c162aa3 100644
> > --- a/meta/classes/image.bbclass
> > +++ b/meta/classes/image.bbclass
> > @@ -392,6 +392,18 @@ python do_deploy() {
> > }
> > addtask deploy before do_build after do_image
> >
> > +def apt_list_files(d):
> > + lists = []
> > + sources = d.getVar("SRC_URI").split()
> > + for s in sources:
> > + _, _, local, _, _, parm = bb.fetch.decodeurl(s)
> > + base, ext = os.path.splitext(os.path.basename(local))
> > + if ext and ext in (".list"):
> > + lists.append(local)
> > + return lists
> > +
> > +IMAGE_LISTS = "${@ ' '.join(apt_list_files(d)) }"
> > +
> > do_rootfs_finalize() {
> > sudo -s <<'EOSUDO'
> > set -e
> > @@ -429,6 +441,12 @@ do_rootfs_finalize() {
> > mv "${ROOTFSDIR}/etc/apt/sources-list" \
> >
> > "${ROOTFSDIR}/etc/apt/sources.list.d/bootstrap.list"
> > fi
> > + if [ -n "${IMAGE_LISTS}" ]; then
> > + rm -f
> > "${ROOTFSDIR}/etc/apt/sources.list.d/bootstrap.list"
> > + for l in ${IMAGE_LISTS}; do
> > + cp "${WORKDIR}"/${l}
> > "${ROOTFSDIR}/etc/apt/sources.list.d/"
> > + done
> > + fi
> >
> > rm -f "${ROOTFSDIR}/run/blkid/blkid.tab"
> > rm -f "${ROOTFSDIR}/run/blkid/blkid.tab.old"
>
--
Cedric Hombourger
Siemens AG
www.siemens.com
--
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/3647c8924a945ecf84ead4d889db25e03ab26b8a.camel%40siemens.com.
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [PATCH] image: copy .list files from SRC_URI to the image
2024-09-23 14:27 [PATCH] image: copy .list files from SRC_URI to the image 'Cedric Hombourger' via isar-users
2024-09-25 16:04 ` 'Jan Kiszka' via isar-users
@ 2024-10-04 11:36 ` Uladzimir Bely
2024-10-08 5:12 ` Uladzimir Bely
2 siblings, 0 replies; 5+ messages in thread
From: Uladzimir Bely @ 2024-10-04 11:36 UTC (permalink / raw)
To: Cedric Hombourger, isar-users
On Mon, 2024-09-23 at 16:27 +0200, 'Cedric Hombourger' via isar-users
wrote:
> There are use-cases where the .list files used by Isar / apt during
> the
> build would not be the same as the initial .list files presented to
> the
> end-user of the image (e.g. build from base-apt or from a local
> mirror).
> If any .list files are listed in the SRC_URI of an image recipe, they
> will be copied to /etc/apt/sources.list.d/ verbatim and in lieu of
> the
> bootstrap sources that were used by Isar.
>
> Signed-off-by: Cedric Hombourger <cedric.hombourger@siemens.com>
> ---
> doc/user_manual.md | 14 ++++++++++++++
> meta/classes/image.bbclass | 18 ++++++++++++++++++
> 2 files changed, 32 insertions(+)
>
> diff --git a/doc/user_manual.md b/doc/user_manual.md
> index 847f4b07..89c3a293 100644
> --- a/doc/user_manual.md
> +++ b/doc/user_manual.md
> @@ -769,6 +769,20 @@ inherit image
>
> ```
>
> +If the resulting image should not ship apt sources used during the
> build but custom ones (e.g. for end-users to point
> +to an external or simply different server when they "apt-get
> update", custom list files may be listed in `SRC_URI`:
> +Isar will copy them to `/etc/apt/sources.list.d/` and omit bootstrap
> sources. Possible use-cases:
> +
> + * image built from base-apt (which is by definition local to the
> build host)
> +
> + * image built from an internal mirror, not reachable by devices
> running the produced image
> +
> + * ship template list files for the end-user to edit (e.g. letting
> him uncomment `deb` or `deb-src` entries)
> +
> +It should be noted that Isar will not validate or even load supplied
> list files: they are simply copied verbatim to
> +the root file-system just before creating an image out of it
> (loading sources from the network would make the build
> +non-reproducible).
> +
> ### Additional Notes
>
> The distribution selected via the `DISTRO` variable may need to run
> a post-configuration script after the root file-system
> diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
> index c29d9e26..0c162aa3 100644
> --- a/meta/classes/image.bbclass
> +++ b/meta/classes/image.bbclass
> @@ -392,6 +392,18 @@ python do_deploy() {
> }
> addtask deploy before do_build after do_image
>
> +def apt_list_files(d):
> + lists = []
> + sources = d.getVar("SRC_URI").split()
> + for s in sources:
> + _, _, local, _, _, parm = bb.fetch.decodeurl(s)
> + base, ext = os.path.splitext(os.path.basename(local))
> + if ext and ext in (".list"):
> + lists.append(local)
> + return lists
> +
> +IMAGE_LISTS = "${@ ' '.join(apt_list_files(d)) }"
> +
> do_rootfs_finalize() {
> sudo -s <<'EOSUDO'
> set -e
> @@ -429,6 +441,12 @@ do_rootfs_finalize() {
> mv "${ROOTFSDIR}/etc/apt/sources-list" \
> "${ROOTFSDIR}/etc/apt/sources.list.d/bootstrap.list"
> fi
> + if [ -n "${IMAGE_LISTS}" ]; then
> + rm -f
> "${ROOTFSDIR}/etc/apt/sources.list.d/bootstrap.list"
> + for l in ${IMAGE_LISTS}; do
> + cp "${WORKDIR}"/${l}
> "${ROOTFSDIR}/etc/apt/sources.list.d/"
> + done
> + fi
>
> rm -f "${ROOTFSDIR}/run/blkid/blkid.tab"
> rm -f "${ROOTFSDIR}/run/blkid/blkid.tab.old"
> --
> 2.39.2
>
Hello all.
If there are no objection and futher discussion, we are ready to merge
the patch.
--
Best regards,
Uladzimir.
--
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/d97b287af26340c2ae39dbf26f2492ff785b4b2c.camel%40ilbers.de.
^ permalink raw reply [flat|nested] 5+ messages in thread* Re: [PATCH] image: copy .list files from SRC_URI to the image
2024-09-23 14:27 [PATCH] image: copy .list files from SRC_URI to the image 'Cedric Hombourger' via isar-users
2024-09-25 16:04 ` 'Jan Kiszka' via isar-users
2024-10-04 11:36 ` Uladzimir Bely
@ 2024-10-08 5:12 ` Uladzimir Bely
2 siblings, 0 replies; 5+ messages in thread
From: Uladzimir Bely @ 2024-10-08 5:12 UTC (permalink / raw)
To: Cedric Hombourger, isar-users
On Mon, 2024-09-23 at 16:27 +0200, 'Cedric Hombourger' via isar-users
wrote:
> There are use-cases where the .list files used by Isar / apt during
> the
> build would not be the same as the initial .list files presented to
> the
> end-user of the image (e.g. build from base-apt or from a local
> mirror).
> If any .list files are listed in the SRC_URI of an image recipe, they
> will be copied to /etc/apt/sources.list.d/ verbatim and in lieu of
> the
> bootstrap sources that were used by Isar.
>
> Signed-off-by: Cedric Hombourger <cedric.hombourger@siemens.com>
> ---
> doc/user_manual.md | 14 ++++++++++++++
> meta/classes/image.bbclass | 18 ++++++++++++++++++
> 2 files changed, 32 insertions(+)
>
> diff --git a/doc/user_manual.md b/doc/user_manual.md
> index 847f4b07..89c3a293 100644
> --- a/doc/user_manual.md
> +++ b/doc/user_manual.md
> @@ -769,6 +769,20 @@ inherit image
>
> ```
>
> +If the resulting image should not ship apt sources used during the
> build but custom ones (e.g. for end-users to point
> +to an external or simply different server when they "apt-get
> update", custom list files may be listed in `SRC_URI`:
> +Isar will copy them to `/etc/apt/sources.list.d/` and omit bootstrap
> sources. Possible use-cases:
> +
> + * image built from base-apt (which is by definition local to the
> build host)
> +
> + * image built from an internal mirror, not reachable by devices
> running the produced image
> +
> + * ship template list files for the end-user to edit (e.g. letting
> him uncomment `deb` or `deb-src` entries)
> +
> +It should be noted that Isar will not validate or even load supplied
> list files: they are simply copied verbatim to
> +the root file-system just before creating an image out of it
> (loading sources from the network would make the build
> +non-reproducible).
> +
> ### Additional Notes
>
> The distribution selected via the `DISTRO` variable may need to run
> a post-configuration script after the root file-system
> diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
> index c29d9e26..0c162aa3 100644
> --- a/meta/classes/image.bbclass
> +++ b/meta/classes/image.bbclass
> @@ -392,6 +392,18 @@ python do_deploy() {
> }
> addtask deploy before do_build after do_image
>
> +def apt_list_files(d):
> + lists = []
> + sources = d.getVar("SRC_URI").split()
> + for s in sources:
> + _, _, local, _, _, parm = bb.fetch.decodeurl(s)
> + base, ext = os.path.splitext(os.path.basename(local))
> + if ext and ext in (".list"):
> + lists.append(local)
> + return lists
> +
> +IMAGE_LISTS = "${@ ' '.join(apt_list_files(d)) }"
> +
> do_rootfs_finalize() {
> sudo -s <<'EOSUDO'
> set -e
> @@ -429,6 +441,12 @@ do_rootfs_finalize() {
> mv "${ROOTFSDIR}/etc/apt/sources-list" \
> "${ROOTFSDIR}/etc/apt/sources.list.d/bootstrap.list"
> fi
> + if [ -n "${IMAGE_LISTS}" ]; then
> + rm -f
> "${ROOTFSDIR}/etc/apt/sources.list.d/bootstrap.list"
> + for l in ${IMAGE_LISTS}; do
> + cp "${WORKDIR}"/${l}
> "${ROOTFSDIR}/etc/apt/sources.list.d/"
> + done
> + fi
>
> rm -f "${ROOTFSDIR}/run/blkid/blkid.tab"
> rm -f "${ROOTFSDIR}/run/blkid/blkid.tab.old"
> --
> 2.39.2
>
Applied to next, thanks.
--
Best regards,
Uladzimir.
--
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/82afb3c7f686dd50a05fa11ae9a2309112dcbe73.camel%40ilbers.de.
^ permalink raw reply [flat|nested] 5+ messages in thread