[PATCH] image: copy .list files from SRC_URI to the image

[PATCH] image: copy .list files from SRC_URI to the image

['Cedric Hombourger' via isar-users]

There are use-cases where the .list files used by Isar / apt during the
build would not be the same as the initial .list files presented to the
end-user of the image (e.g. build from base-apt or from a local mirror).
If any .list files are listed in the SRC_URI of an image recipe, they
will be copied to /etc/apt/sources.list.d/ verbatim and in lieu of the
bootstrap sources that were used by Isar.

Signed-off-by: Cedric Hombourger <cedric.hombourger@siemens.com>
---
 doc/user_manual.md         | 14 ++++++++++++++
 meta/classes/image.bbclass | 18 ++++++++++++++++++
 2 files changed, 32 insertions(+)

diff --git a/doc/user_manual.md b/doc/user_manual.md
index 847f4b07..89c3a293 100644
--- a/doc/user_manual.md
+++ b/doc/user_manual.md
@@ -769,6 +769,20 @@ inherit image
 
 ```
 
+If the resulting image should not ship apt sources used during the build but custom ones (e.g. for end-users to point
+to an external or simply different server when they "apt-get update", custom list files may be listed in `SRC_URI`:
+Isar will copy them to `/etc/apt/sources.list.d/` and omit bootstrap sources. Possible use-cases:
+
+ * image built from base-apt (which is by definition local to the build host)
+
+ * image built from an internal mirror, not reachable by devices running the produced image
+
+ * ship template list files for the end-user to edit (e.g. letting him uncomment `deb` or `deb-src` entries)
+
+It should be noted that Isar will not validate or even load supplied list files: they are simply copied verbatim to
+the root file-system just before creating an image out of it (loading sources from the network would make the build
+non-reproducible).
+
 ### Additional Notes
 
 The distribution selected via the `DISTRO` variable may need to run a post-configuration script after the root file-system
diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index c29d9e26..0c162aa3 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -392,6 +392,18 @@ python do_deploy() {
 }
 addtask deploy before do_build after do_image
 
+def apt_list_files(d):
+    lists = []
+    sources = d.getVar("SRC_URI").split()
+    for s in sources:
+        _, _, local, _, _, parm = bb.fetch.decodeurl(s)
+        base, ext = os.path.splitext(os.path.basename(local))
+        if ext and ext in (".list"):
+            lists.append(local)
+    return lists
+
+IMAGE_LISTS = "${@ ' '.join(apt_list_files(d)) }"
+
 do_rootfs_finalize() {
     sudo -s <<'EOSUDO'
         set -e
@@ -429,6 +441,12 @@ do_rootfs_finalize() {
             mv "${ROOTFSDIR}/etc/apt/sources-list" \
                 "${ROOTFSDIR}/etc/apt/sources.list.d/bootstrap.list"
         fi
+        if [ -n "${IMAGE_LISTS}" ]; then
+            rm -f "${ROOTFSDIR}/etc/apt/sources.list.d/bootstrap.list"
+            for l in ${IMAGE_LISTS}; do
+                cp "${WORKDIR}"/${l} "${ROOTFSDIR}/etc/apt/sources.list.d/"
+            done
+        fi
 
         rm -f "${ROOTFSDIR}/run/blkid/blkid.tab"
         rm -f "${ROOTFSDIR}/run/blkid/blkid.tab.old"
-- 
2.39.2

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/20240923142704.389438-1-cedric.hombourger%40siemens.com.

Re: [PATCH] image: copy .list files from SRC_URI to the image

['Jan Kiszka' via isar-users]

On 23.09.24 16:27, 'Cedric Hombourger' via isar-users wrote:
> There are use-cases where the .list files used by Isar / apt during the
> build would not be the same as the initial .list files presented to the
> end-user of the image (e.g. build from base-apt or from a local mirror).
> If any .list files are listed in the SRC_URI of an image recipe, they
> will be copied to /etc/apt/sources.list.d/ verbatim and in lieu of the
> bootstrap sources that were used by Isar.

How well does this interact with the list swappings we do already, e.g.
for using snapshots during build but then upstream on the target?

Jan

> 
> Signed-off-by: Cedric Hombourger <cedric.hombourger@siemens.com>
> ---
>  doc/user_manual.md         | 14 ++++++++++++++
>  meta/classes/image.bbclass | 18 ++++++++++++++++++
>  2 files changed, 32 insertions(+)
> 
> diff --git a/doc/user_manual.md b/doc/user_manual.md
> index 847f4b07..89c3a293 100644
> --- a/doc/user_manual.md
> +++ b/doc/user_manual.md
> @@ -769,6 +769,20 @@ inherit image
>  
>  ```
>  
> +If the resulting image should not ship apt sources used during the build but custom ones (e.g. for end-users to point
> +to an external or simply different server when they "apt-get update", custom list files may be listed in `SRC_URI`:
> +Isar will copy them to `/etc/apt/sources.list.d/` and omit bootstrap sources. Possible use-cases:
> +
> + * image built from base-apt (which is by definition local to the build host)
> +
> + * image built from an internal mirror, not reachable by devices running the produced image
> +
> + * ship template list files for the end-user to edit (e.g. letting him uncomment `deb` or `deb-src` entries)
> +
> +It should be noted that Isar will not validate or even load supplied list files: they are simply copied verbatim to
> +the root file-system just before creating an image out of it (loading sources from the network would make the build
> +non-reproducible).
> +
>  ### Additional Notes
>  
>  The distribution selected via the `DISTRO` variable may need to run a post-configuration script after the root file-system
> diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
> index c29d9e26..0c162aa3 100644
> --- a/meta/classes/image.bbclass
> +++ b/meta/classes/image.bbclass
> @@ -392,6 +392,18 @@ python do_deploy() {
>  }
>  addtask deploy before do_build after do_image
>  
> +def apt_list_files(d):
> +    lists = []
> +    sources = d.getVar("SRC_URI").split()
> +    for s in sources:
> +        _, _, local, _, _, parm = bb.fetch.decodeurl(s)
> +        base, ext = os.path.splitext(os.path.basename(local))
> +        if ext and ext in (".list"):
> +            lists.append(local)
> +    return lists
> +
> +IMAGE_LISTS = "${@ ' '.join(apt_list_files(d)) }"
> +
>  do_rootfs_finalize() {
>      sudo -s <<'EOSUDO'
>          set -e
> @@ -429,6 +441,12 @@ do_rootfs_finalize() {
>              mv "${ROOTFSDIR}/etc/apt/sources-list" \
>                  "${ROOTFSDIR}/etc/apt/sources.list.d/bootstrap.list"
>          fi
> +        if [ -n "${IMAGE_LISTS}" ]; then
> +            rm -f "${ROOTFSDIR}/etc/apt/sources.list.d/bootstrap.list"
> +            for l in ${IMAGE_LISTS}; do
> +                cp "${WORKDIR}"/${l} "${ROOTFSDIR}/etc/apt/sources.list.d/"
> +            done
> +        fi
>  
>          rm -f "${ROOTFSDIR}/run/blkid/blkid.tab"
>          rm -f "${ROOTFSDIR}/run/blkid/blkid.tab.old"

-- 
Siemens AG, Technology
Linux Expert Center

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/d6062f14-fea7-418e-a7f5-cccb00d76512%40siemens.com.

Re: [PATCH] image: copy .list files from SRC_URI to the image

['cedric.hombourger@siemens.com' via isar-users]

On Wed, 2024-09-25 at 18:04 +0200, Jan Kiszka wrote:
> On 23.09.24 16:27, 'Cedric Hombourger' via isar-users wrote:
> > There are use-cases where the .list files used by Isar / apt during
> > the
> > build would not be the same as the initial .list files presented to
> > the
> > end-user of the image (e.g. build from base-apt or from a local
> > mirror).
> > If any .list files are listed in the SRC_URI of an image recipe,
> > they
> > will be copied to /etc/apt/sources.list.d/ verbatim and in lieu of
> > the
> > bootstrap sources that were used by Isar.
> 
> How well does this interact with the list swappings we do already,
> e.g.
> for using snapshots during build but then upstream on the target?

Rechecked to be sure. Lines above my changes are responsible for the
swap (/etc/apt/sources-list => /etc/apt/sources.list.d/bootstrap.list)

head of /etc/apt/sources.list.d/bootstrap.list before
do_rootfs_finalize()

deb [check-valid-until=no]
http://snapshot.debian.org/archive/debian/20240925T180156Z/ bookworm
main contrib non-free-firmware non-free

head of /etc/apt/sources-list before do_rootfs_finalize()

deb     http://deb.debian.org/debian bookworm main contrib non-free-
firmware non-free

After do_rootfs_finalize(), sources-list is gone and bootstrap.list is
no lomger pointing to the selected snapshot

With my changes and with user-supplied .list files in SRC_URI
bootstrap.list is removed and only user-supplied .list files are
present in /etc/apt/sources.list.d/

Hope this clarifies
Cedric

> 
> Jan
> 
> > 
> > Signed-off-by: Cedric Hombourger <cedric.hombourger@siemens.com>
> > ---
> >  doc/user_manual.md         | 14 ++++++++++++++
> >  meta/classes/image.bbclass | 18 ++++++++++++++++++
> >  2 files changed, 32 insertions(+)
> > 
> > diff --git a/doc/user_manual.md b/doc/user_manual.md
> > index 847f4b07..89c3a293 100644
> > --- a/doc/user_manual.md
> > +++ b/doc/user_manual.md
> > @@ -769,6 +769,20 @@ inherit image
> >  
> >  ```
> >  
> > +If the resulting image should not ship apt sources used during the
> > build but custom ones (e.g. for end-users to point
> > +to an external or simply different server when they "apt-get
> > update", custom list files may be listed in `SRC_URI`:
> > +Isar will copy them to `/etc/apt/sources.list.d/` and omit
> > bootstrap sources. Possible use-cases:
> > +
> > + * image built from base-apt (which is by definition local to the
> > build host)
> > +
> > + * image built from an internal mirror, not reachable by devices
> > running the produced image
> > +
> > + * ship template list files for the end-user to edit (e.g. letting
> > him uncomment `deb` or `deb-src` entries)
> > +
> > +It should be noted that Isar will not validate or even load
> > supplied list files: they are simply copied verbatim to
> > +the root file-system just before creating an image out of it
> > (loading sources from the network would make the build
> > +non-reproducible).
> > +
> >  ### Additional Notes
> >  
> >  The distribution selected via the `DISTRO` variable may need to
> > run a post-configuration script after the root file-system
> > diff --git a/meta/classes/image.bbclass
> > b/meta/classes/image.bbclass
> > index c29d9e26..0c162aa3 100644
> > --- a/meta/classes/image.bbclass
> > +++ b/meta/classes/image.bbclass
> > @@ -392,6 +392,18 @@ python do_deploy() {
> >  }
> >  addtask deploy before do_build after do_image
> >  
> > +def apt_list_files(d):
> > +    lists = []
> > +    sources = d.getVar("SRC_URI").split()
> > +    for s in sources:
> > +        _, _, local, _, _, parm = bb.fetch.decodeurl(s)
> > +        base, ext = os.path.splitext(os.path.basename(local))
> > +        if ext and ext in (".list"):
> > +            lists.append(local)
> > +    return lists
> > +
> > +IMAGE_LISTS = "${@ ' '.join(apt_list_files(d)) }"
> > +
> >  do_rootfs_finalize() {
> >      sudo -s <<'EOSUDO'
> >          set -e
> > @@ -429,6 +441,12 @@ do_rootfs_finalize() {
> >              mv "${ROOTFSDIR}/etc/apt/sources-list" \
> >                 
> > "${ROOTFSDIR}/etc/apt/sources.list.d/bootstrap.list"
> >          fi
> > +        if [ -n "${IMAGE_LISTS}" ]; then
> > +            rm -f
> > "${ROOTFSDIR}/etc/apt/sources.list.d/bootstrap.list"
> > +            for l in ${IMAGE_LISTS}; do
> > +                cp "${WORKDIR}"/${l}
> > "${ROOTFSDIR}/etc/apt/sources.list.d/"
> > +            done
> > +        fi
> >  
> >          rm -f "${ROOTFSDIR}/run/blkid/blkid.tab"
> >          rm -f "${ROOTFSDIR}/run/blkid/blkid.tab.old"
> 

-- 
Cedric Hombourger
Siemens AG
www.siemens.com

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/3647c8924a945ecf84ead4d889db25e03ab26b8a.camel%40siemens.com.

Re: [PATCH] image: copy .list files from SRC_URI to the image

[Uladzimir Bely]

On Mon, 2024-09-23 at 16:27 +0200, 'Cedric Hombourger' via isar-users
wrote:
> There are use-cases where the .list files used by Isar / apt during
> the
> build would not be the same as the initial .list files presented to
> the
> end-user of the image (e.g. build from base-apt or from a local
> mirror).
> If any .list files are listed in the SRC_URI of an image recipe, they
> will be copied to /etc/apt/sources.list.d/ verbatim and in lieu of
> the
> bootstrap sources that were used by Isar.
> 
> Signed-off-by: Cedric Hombourger <cedric.hombourger@siemens.com>
> ---
>  doc/user_manual.md         | 14 ++++++++++++++
>  meta/classes/image.bbclass | 18 ++++++++++++++++++
>  2 files changed, 32 insertions(+)
> 
> diff --git a/doc/user_manual.md b/doc/user_manual.md
> index 847f4b07..89c3a293 100644
> --- a/doc/user_manual.md
> +++ b/doc/user_manual.md
> @@ -769,6 +769,20 @@ inherit image
>  
>  ```
>  
> +If the resulting image should not ship apt sources used during the
> build but custom ones (e.g. for end-users to point
> +to an external or simply different server when they "apt-get
> update", custom list files may be listed in `SRC_URI`:
> +Isar will copy them to `/etc/apt/sources.list.d/` and omit bootstrap
> sources. Possible use-cases:
> +
> + * image built from base-apt (which is by definition local to the
> build host)
> +
> + * image built from an internal mirror, not reachable by devices
> running the produced image
> +
> + * ship template list files for the end-user to edit (e.g. letting
> him uncomment `deb` or `deb-src` entries)
> +
> +It should be noted that Isar will not validate or even load supplied
> list files: they are simply copied verbatim to
> +the root file-system just before creating an image out of it
> (loading sources from the network would make the build
> +non-reproducible).
> +
>  ### Additional Notes
>  
>  The distribution selected via the `DISTRO` variable may need to run
> a post-configuration script after the root file-system
> diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
> index c29d9e26..0c162aa3 100644
> --- a/meta/classes/image.bbclass
> +++ b/meta/classes/image.bbclass
> @@ -392,6 +392,18 @@ python do_deploy() {
>  }
>  addtask deploy before do_build after do_image
>  
> +def apt_list_files(d):
> +    lists = []
> +    sources = d.getVar("SRC_URI").split()
> +    for s in sources:
> +        _, _, local, _, _, parm = bb.fetch.decodeurl(s)
> +        base, ext = os.path.splitext(os.path.basename(local))
> +        if ext and ext in (".list"):
> +            lists.append(local)
> +    return lists
> +
> +IMAGE_LISTS = "${@ ' '.join(apt_list_files(d)) }"
> +
>  do_rootfs_finalize() {
>      sudo -s <<'EOSUDO'
>          set -e
> @@ -429,6 +441,12 @@ do_rootfs_finalize() {
>              mv "${ROOTFSDIR}/etc/apt/sources-list" \
>                  "${ROOTFSDIR}/etc/apt/sources.list.d/bootstrap.list"
>          fi
> +        if [ -n "${IMAGE_LISTS}" ]; then
> +            rm -f
> "${ROOTFSDIR}/etc/apt/sources.list.d/bootstrap.list"
> +            for l in ${IMAGE_LISTS}; do
> +                cp "${WORKDIR}"/${l}
> "${ROOTFSDIR}/etc/apt/sources.list.d/"
> +            done
> +        fi
>  
>          rm -f "${ROOTFSDIR}/run/blkid/blkid.tab"
>          rm -f "${ROOTFSDIR}/run/blkid/blkid.tab.old"
> -- 
> 2.39.2
> 

Hello all.

If there are no objection and futher discussion, we are ready to merge
the patch.

-- 
Best regards,
Uladzimir.



-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/d97b287af26340c2ae39dbf26f2492ff785b4b2c.camel%40ilbers.de.

Re: [PATCH] image: copy .list files from SRC_URI to the image

[Uladzimir Bely]

On Mon, 2024-09-23 at 16:27 +0200, 'Cedric Hombourger' via isar-users
wrote:
> There are use-cases where the .list files used by Isar / apt during
> the
> build would not be the same as the initial .list files presented to
> the
> end-user of the image (e.g. build from base-apt or from a local
> mirror).
> If any .list files are listed in the SRC_URI of an image recipe, they
> will be copied to /etc/apt/sources.list.d/ verbatim and in lieu of
> the
> bootstrap sources that were used by Isar.
> 
> Signed-off-by: Cedric Hombourger <cedric.hombourger@siemens.com>
> ---
>  doc/user_manual.md         | 14 ++++++++++++++
>  meta/classes/image.bbclass | 18 ++++++++++++++++++
>  2 files changed, 32 insertions(+)
> 
> diff --git a/doc/user_manual.md b/doc/user_manual.md
> index 847f4b07..89c3a293 100644
> --- a/doc/user_manual.md
> +++ b/doc/user_manual.md
> @@ -769,6 +769,20 @@ inherit image
>  
>  ```
>  
> +If the resulting image should not ship apt sources used during the
> build but custom ones (e.g. for end-users to point
> +to an external or simply different server when they "apt-get
> update", custom list files may be listed in `SRC_URI`:
> +Isar will copy them to `/etc/apt/sources.list.d/` and omit bootstrap
> sources. Possible use-cases:
> +
> + * image built from base-apt (which is by definition local to the
> build host)
> +
> + * image built from an internal mirror, not reachable by devices
> running the produced image
> +
> + * ship template list files for the end-user to edit (e.g. letting
> him uncomment `deb` or `deb-src` entries)
> +
> +It should be noted that Isar will not validate or even load supplied
> list files: they are simply copied verbatim to
> +the root file-system just before creating an image out of it
> (loading sources from the network would make the build
> +non-reproducible).
> +
>  ### Additional Notes
>  
>  The distribution selected via the `DISTRO` variable may need to run
> a post-configuration script after the root file-system
> diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
> index c29d9e26..0c162aa3 100644
> --- a/meta/classes/image.bbclass
> +++ b/meta/classes/image.bbclass
> @@ -392,6 +392,18 @@ python do_deploy() {
>  }
>  addtask deploy before do_build after do_image
>  
> +def apt_list_files(d):
> +    lists = []
> +    sources = d.getVar("SRC_URI").split()
> +    for s in sources:
> +        _, _, local, _, _, parm = bb.fetch.decodeurl(s)
> +        base, ext = os.path.splitext(os.path.basename(local))
> +        if ext and ext in (".list"):
> +            lists.append(local)
> +    return lists
> +
> +IMAGE_LISTS = "${@ ' '.join(apt_list_files(d)) }"
> +
>  do_rootfs_finalize() {
>      sudo -s <<'EOSUDO'
>          set -e
> @@ -429,6 +441,12 @@ do_rootfs_finalize() {
>              mv "${ROOTFSDIR}/etc/apt/sources-list" \
>                  "${ROOTFSDIR}/etc/apt/sources.list.d/bootstrap.list"
>          fi
> +        if [ -n "${IMAGE_LISTS}" ]; then
> +            rm -f
> "${ROOTFSDIR}/etc/apt/sources.list.d/bootstrap.list"
> +            for l in ${IMAGE_LISTS}; do
> +                cp "${WORKDIR}"/${l}
> "${ROOTFSDIR}/etc/apt/sources.list.d/"
> +            done
> +        fi
>  
>          rm -f "${ROOTFSDIR}/run/blkid/blkid.tab"
>          rm -f "${ROOTFSDIR}/run/blkid/blkid.tab.old"
> -- 
> 2.39.2
> 

Applied to next, thanks.

-- 
Best regards,
Uladzimir.



-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/82afb3c7f686dd50a05fa11ae9a2309112dcbe73.camel%40ilbers.de.