[PATCH 0/3] Sparse tar generation

public inbox for isar-users@googlegroups.com
 help / color / mirror / Atom feed

* [PATCH 0/3] Sparse tar generation
@ 2022-03-05 16:34 Jan Kiszka
  2022-03-05 16:34 ` [PATCH 1/3] sstate: Generate image packages using sparse Jan Kiszka
                   ` (3 more replies)
  0 siblings, 4 replies; 14+ messages in thread
From: Jan Kiszka @ 2022-03-05 16:34 UTC (permalink / raw)
  To: isar-users

Just some small optimization: Consistently use tar -S when generating
archives to avoid expanding sparse files.

Jan

Jan Kiszka (3):
  sstate: Generate image packages using sparse
  targz-img: Handle sparse files efficiently
  vm-img: Handle sparse files efficiently

 meta/classes/rootfs.bbclass                         | 2 +-
 meta/classes/targz-img.bbclass                      | 2 +-
 meta/classes/vm-img.bbclass                         | 6 +++---
 meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 2 +-
 4 files changed, 6 insertions(+), 6 deletions(-)

-- 
2.34.1


^ permalink raw reply	[flat|nested] 14+ messages in thread

* [PATCH 1/3] sstate: Generate image packages using sparse
  2022-03-05 16:34 [PATCH 0/3] Sparse tar generation Jan Kiszka
@ 2022-03-05 16:34 ` Jan Kiszka
  2022-05-24 19:03   ` Jan Kiszka
  2022-03-05 16:34 ` [PATCH 2/3] targz-img: Handle sparse files efficiently Jan Kiszka
                   ` (2 subsequent siblings)
  3 siblings, 1 reply; 14+ messages in thread
From: Jan Kiszka @ 2022-03-05 16:34 UTC (permalink / raw)
  To: isar-users

From: Jan Kiszka <jan.kiszka@siemens.com>

Avoids needless expansion of sparse files while filling the sstate
cache.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 meta/classes/rootfs.bbclass                         | 2 +-
 meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass
index ba86c60a..927af13f 100644
--- a/meta/classes/rootfs.bbclass
+++ b/meta/classes/rootfs.bbclass
@@ -299,7 +299,7 @@ rootfs_install_sstate_prepare() {
     # so we use some mount magic to prevent that
     mkdir -p ${WORKDIR}/mnt/rootfs
     sudo mount --bind ${WORKDIR}/rootfs ${WORKDIR}/mnt/rootfs -o ro
-    sudo tar -C ${WORKDIR}/mnt -cpf rootfs.tar --one-file-system rootfs
+    sudo tar -C ${WORKDIR}/mnt -cpSf rootfs.tar --one-file-system rootfs
     sudo umount ${WORKDIR}/mnt/rootfs
     sudo chown $(id -u):$(id -g) rootfs.tar
 }
diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
index d6f90f63..1b16f874 100644
--- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
+++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
@@ -400,7 +400,7 @@ SSTATEPOSTINSTFUNCS += "bootstrap_sstate_finalize"
 
 bootstrap_sstate_prepare() {
     # this runs in SSTATE_BUILDDIR, which will be deleted automatically
-    sudo tar -C $(dirname "${ROOTFSDIR}") -cpf bootstrap.tar --one-file-system $(basename "${ROOTFSDIR}")
+    sudo tar -C $(dirname "${ROOTFSDIR}") -cpSf bootstrap.tar --one-file-system $(basename "${ROOTFSDIR}")
     sudo chown $(id -u):$(id -g) bootstrap.tar
 }
 
-- 
2.34.1


^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: [PATCH 1/3] sstate: Generate image packages using sparse
  2022-03-05 16:34 ` [PATCH 1/3] sstate: Generate image packages using sparse Jan Kiszka
@ 2022-05-24 19:03   ` Jan Kiszka
  0 siblings, 0 replies; 14+ messages in thread
From: Jan Kiszka @ 2022-05-24 19:03 UTC (permalink / raw)
  To: isar-users, Schmidt, Adriaan; +Cc: Anton Mikanovich

On 05.03.22 17:34, Jan Kiszka wrote:
> From: Jan Kiszka <jan.kiszka@siemens.com>
> 
> Avoids needless expansion of sparse files while filling the sstate
> cache.
> 
> Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
> ---
>  meta/classes/rootfs.bbclass                         | 2 +-
>  meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 2 +-
>  2 files changed, 2 insertions(+), 2 deletions(-)
> 
> diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass
> index ba86c60a..927af13f 100644
> --- a/meta/classes/rootfs.bbclass
> +++ b/meta/classes/rootfs.bbclass
> @@ -299,7 +299,7 @@ rootfs_install_sstate_prepare() {
>      # so we use some mount magic to prevent that
>      mkdir -p ${WORKDIR}/mnt/rootfs
>      sudo mount --bind ${WORKDIR}/rootfs ${WORKDIR}/mnt/rootfs -o ro
> -    sudo tar -C ${WORKDIR}/mnt -cpf rootfs.tar --one-file-system rootfs
> +    sudo tar -C ${WORKDIR}/mnt -cpSf rootfs.tar --one-file-system rootfs
>      sudo umount ${WORKDIR}/mnt/rootfs
>      sudo chown $(id -u):$(id -g) rootfs.tar
>  }
> diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
> index d6f90f63..1b16f874 100644
> --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
> +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
> @@ -400,7 +400,7 @@ SSTATEPOSTINSTFUNCS += "bootstrap_sstate_finalize"
>  
>  bootstrap_sstate_prepare() {
>      # this runs in SSTATE_BUILDDIR, which will be deleted automatically
> -    sudo tar -C $(dirname "${ROOTFSDIR}") -cpf bootstrap.tar --one-file-system $(basename "${ROOTFSDIR}")
> +    sudo tar -C $(dirname "${ROOTFSDIR}") -cpSf bootstrap.tar --one-file-system $(basename "${ROOTFSDIR}")
>      sudo chown $(id -u):$(id -g) bootstrap.tar
>  }
>  

Ping again - still applies and seems reasonable to me.

Jan

-- 
Siemens AG, Technology
Competence Center Embedded Linux

^ permalink raw reply	[flat|nested] 14+ messages in thread

* [PATCH 2/3] targz-img: Handle sparse files efficiently
  2022-03-05 16:34 [PATCH 0/3] Sparse tar generation Jan Kiszka
  2022-03-05 16:34 ` [PATCH 1/3] sstate: Generate image packages using sparse Jan Kiszka
@ 2022-03-05 16:34 ` Jan Kiszka
  2022-03-07  9:02   ` Henning Schild
  2022-03-05 16:34 ` [PATCH 3/3] vm-img: " Jan Kiszka
  2022-04-21 16:05 ` [PATCH 0/3] Sparse tar generation Jan Kiszka
  3 siblings, 1 reply; 14+ messages in thread
From: Jan Kiszka @ 2022-03-05 16:34 UTC (permalink / raw)
  To: isar-users

From: Jan Kiszka <jan.kiszka@siemens.com>

Saves space in case the image contains any sparse files.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 meta/classes/targz-img.bbclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/classes/targz-img.bbclass b/meta/classes/targz-img.bbclass
index bf94af02..638e7767 100644
--- a/meta/classes/targz-img.bbclass
+++ b/meta/classes/targz-img.bbclass
@@ -7,7 +7,7 @@ TARGZ_IMAGE_FILE = "${DEPLOY_DIR_IMAGE}/${IMAGE_FULLNAME}.tar.gz"
 
 do_targz_image() {
     rm -f ${TARGZ_IMAGE_FILE}
-    sudo tar -cvzf ${TARGZ_IMAGE_FILE} --one-file-system -C ${IMAGE_ROOTFS} .
+    sudo tar -cvzSf ${TARGZ_IMAGE_FILE} --one-file-system -C ${IMAGE_ROOTFS} .
     sudo chown $(id -u):$(id -g) ${TARGZ_IMAGE_FILE}
 }
 
-- 
2.34.1


^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: [PATCH 2/3] targz-img: Handle sparse files efficiently
  2022-03-05 16:34 ` [PATCH 2/3] targz-img: Handle sparse files efficiently Jan Kiszka
@ 2022-03-07  9:02   ` Henning Schild
  2022-03-07  9:43     ` Jan Kiszka
  0 siblings, 1 reply; 14+ messages in thread
From: Henning Schild @ 2022-03-07  9:02 UTC (permalink / raw)
  To: Jan Kiszka; +Cc: isar-users

Am Sat, 5 Mar 2022 17:34:49 +0100
schrieb Jan Kiszka <jan.kiszka@siemens.com>:

> From: Jan Kiszka <jan.kiszka@siemens.com>
> 
> Saves space in case the image contains any sparse files.
> 
> Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
> ---
>  meta/classes/targz-img.bbclass | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/meta/classes/targz-img.bbclass
> b/meta/classes/targz-img.bbclass index bf94af02..638e7767 100644
> --- a/meta/classes/targz-img.bbclass
> +++ b/meta/classes/targz-img.bbclass
> @@ -7,7 +7,7 @@ TARGZ_IMAGE_FILE =
> "${DEPLOY_DIR_IMAGE}/${IMAGE_FULLNAME}.tar.gz" 
>  do_targz_image() {
>      rm -f ${TARGZ_IMAGE_FILE}
> -    sudo tar -cvzf ${TARGZ_IMAGE_FILE} --one-file-system -C
> ${IMAGE_ROOTFS} .
> +    sudo tar -cvzSf ${TARGZ_IMAGE_FILE} --one-file-system -C
> ${IMAGE_ROOTFS} . sudo chown $(id -u):$(id -g) ${TARGZ_IMAGE_FILE}

I assume that can be unpacked with or without -S now, so no interface
change. right?

Henning

>  }
>  


^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: [PATCH 2/3] targz-img: Handle sparse files efficiently
  2022-03-07  9:02   ` Henning Schild
@ 2022-03-07  9:43     ` Jan Kiszka
  0 siblings, 0 replies; 14+ messages in thread
From: Jan Kiszka @ 2022-03-07  9:43 UTC (permalink / raw)
  To: Henning Schild; +Cc: isar-users

On 07.03.22 10:02, Henning Schild wrote:
> Am Sat, 5 Mar 2022 17:34:49 +0100
> schrieb Jan Kiszka <jan.kiszka@siemens.com>:
> 
>> From: Jan Kiszka <jan.kiszka@siemens.com>
>>
>> Saves space in case the image contains any sparse files.
>>
>> Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
>> ---
>>  meta/classes/targz-img.bbclass | 2 +-
>>  1 file changed, 1 insertion(+), 1 deletion(-)
>>
>> diff --git a/meta/classes/targz-img.bbclass
>> b/meta/classes/targz-img.bbclass index bf94af02..638e7767 100644
>> --- a/meta/classes/targz-img.bbclass
>> +++ b/meta/classes/targz-img.bbclass
>> @@ -7,7 +7,7 @@ TARGZ_IMAGE_FILE =
>> "${DEPLOY_DIR_IMAGE}/${IMAGE_FULLNAME}.tar.gz" 
>>  do_targz_image() {
>>      rm -f ${TARGZ_IMAGE_FILE}
>> -    sudo tar -cvzf ${TARGZ_IMAGE_FILE} --one-file-system -C
>> ${IMAGE_ROOTFS} .
>> +    sudo tar -cvzSf ${TARGZ_IMAGE_FILE} --one-file-system -C
>> ${IMAGE_ROOTFS} . sudo chown $(id -u):$(id -g) ${TARGZ_IMAGE_FILE}
> 
> I assume that can be unpacked with or without -S now, so no interface
> change. right?
> 

Yes, see also "man tar".

Jan

-- 
Siemens AG, Technology
Competence Center Embedded Linux

^ permalink raw reply	[flat|nested] 14+ messages in thread

* [PATCH 3/3] vm-img: Handle sparse files efficiently
  2022-03-05 16:34 [PATCH 0/3] Sparse tar generation Jan Kiszka
  2022-03-05 16:34 ` [PATCH 1/3] sstate: Generate image packages using sparse Jan Kiszka
  2022-03-05 16:34 ` [PATCH 2/3] targz-img: Handle sparse files efficiently Jan Kiszka
@ 2022-03-05 16:34 ` Jan Kiszka
  2022-03-07  9:04   ` Henning Schild
  2022-04-21 16:05 ` [PATCH 0/3] Sparse tar generation Jan Kiszka
  3 siblings, 1 reply; 14+ messages in thread
From: Jan Kiszka @ 2022-03-05 16:34 UTC (permalink / raw)
  To: isar-users

From: Jan Kiszka <jan.kiszka@siemens.com>

Saves space in case the image contains any sparse files.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 meta/classes/vm-img.bbclass | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/meta/classes/vm-img.bbclass b/meta/classes/vm-img.bbclass
index 4bc977b9..27aedf12 100644
--- a/meta/classes/vm-img.bbclass
+++ b/meta/classes/vm-img.bbclass
@@ -93,15 +93,15 @@ do_create_ova() {
         export VM_UUID=$(uuidgen)
         # create ovf
         cat ${PP_WORK}/${OVF_TEMPLATE_STAGE2} | envsubst > ${PP_DEPLOY}/${OVA_NAME}.ovf
-        tar -cvf ${PP_DEPLOY}/${OVA_NAME}.ova -C ${PP_DEPLOY} ${OVA_NAME}.ovf
+        tar -cvSf ${PP_DEPLOY}/${OVA_NAME}.ova -C ${PP_DEPLOY} ${OVA_NAME}.ovf
 
         # VirtualBox needs here a manifest file. VMware does accept that format.
         if [ "${VMDK_SUBFORMAT}" = "monolithicSparse" ]; then
             echo "SHA${OVA_SHA_ALG}(${VIRTUAL_MACHINE_IMAGE_FILE})=$(sha${OVA_SHA_ALG}sum ${PP_DEPLOY}/${VIRTUAL_MACHINE_IMAGE_FILE} | cut -d' ' -f1)" >> ${PP_DEPLOY}/${OVA_NAME}.mf
             echo "SHA${OVA_SHA_ALG}(${OVA_NAME}.ovf)=$(sha${OVA_SHA_ALG}sum ${PP_DEPLOY}/${OVA_NAME}.ovf | cut -d' ' -f1)" >> ${PP_DEPLOY}/${OVA_NAME}.mf
-            tar -uvf ${PP_DEPLOY}/${OVA_NAME}.ova -C ${PP_DEPLOY} ${OVA_NAME}.mf
+            tar -uvSf ${PP_DEPLOY}/${OVA_NAME}.ova -C ${PP_DEPLOY} ${OVA_NAME}.mf
         fi
-        tar -uvf ${PP_DEPLOY}/${OVA_NAME}.ova -C ${PP_DEPLOY} ${VIRTUAL_MACHINE_IMAGE_FILE}
+        tar -uvSf ${PP_DEPLOY}/${OVA_NAME}.ova -C ${PP_DEPLOY} ${VIRTUAL_MACHINE_IMAGE_FILE}
 EOSUDO
 }
 
-- 
2.34.1


^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: [PATCH 3/3] vm-img: Handle sparse files efficiently
  2022-03-05 16:34 ` [PATCH 3/3] vm-img: " Jan Kiszka
@ 2022-03-07  9:04   ` Henning Schild
  2022-03-07  9:46     ` Jan Kiszka
  0 siblings, 1 reply; 14+ messages in thread
From: Henning Schild @ 2022-03-07  9:04 UTC (permalink / raw)
  To: Jan Kiszka; +Cc: isar-users

Am Sat, 5 Mar 2022 17:34:50 +0100
schrieb Jan Kiszka <jan.kiszka@siemens.com>:

> From: Jan Kiszka <jan.kiszka@siemens.com>
> 
> Saves space in case the image contains any sparse files.
> 
> Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
> ---
>  meta/classes/vm-img.bbclass | 6 +++---
>  1 file changed, 3 insertions(+), 3 deletions(-)
> 
> diff --git a/meta/classes/vm-img.bbclass b/meta/classes/vm-img.bbclass
> index 4bc977b9..27aedf12 100644
> --- a/meta/classes/vm-img.bbclass
> +++ b/meta/classes/vm-img.bbclass
> @@ -93,15 +93,15 @@ do_create_ova() {
>          export VM_UUID=$(uuidgen)
>          # create ovf
>          cat ${PP_WORK}/${OVF_TEMPLATE_STAGE2} | envsubst >
> ${PP_DEPLOY}/${OVA_NAME}.ovf
> -        tar -cvf ${PP_DEPLOY}/${OVA_NAME}.ova -C ${PP_DEPLOY}
> ${OVA_NAME}.ovf
> +        tar -cvSf ${PP_DEPLOY}/${OVA_NAME}.ova -C ${PP_DEPLOY}
> ${OVA_NAME}.ovf 
>          # VirtualBox needs here a manifest file. VMware does accept
> that format. if [ "${VMDK_SUBFORMAT}" = "monolithicSparse" ]; then

Not sure that already suggested it but we did not do it.

>              echo
> "SHA${OVA_SHA_ALG}(${VIRTUAL_MACHINE_IMAGE_FILE})=$(sha${OVA_SHA_ALG}sum
> ${PP_DEPLOY}/${VIRTUAL_MACHINE_IMAGE_FILE} | cut -d' ' -f1)" >>
> ${PP_DEPLOY}/${OVA_NAME}.mf echo
> "SHA${OVA_SHA_ALG}(${OVA_NAME}.ovf)=$(sha${OVA_SHA_ALG}sum
> ${PP_DEPLOY}/${OVA_NAME}.ovf | cut -d' ' -f1)" >>
> ${PP_DEPLOY}/${OVA_NAME}.mf
> -            tar -uvf ${PP_DEPLOY}/${OVA_NAME}.ova -C ${PP_DEPLOY}
> ${OVA_NAME}.mf
> +            tar -uvSf ${PP_DEPLOY}/${OVA_NAME}.ova -C ${PP_DEPLOY}
> ${OVA_NAME}.mf fi
> -        tar -uvf ${PP_DEPLOY}/${OVA_NAME}.ova -C ${PP_DEPLOY}
> ${VIRTUAL_MACHINE_IMAGE_FILE}
> +        tar -uvSf ${PP_DEPLOY}/${OVA_NAME}.ova -C ${PP_DEPLOY}
> ${VIRTUAL_MACHINE_IMAGE_FILE} EOSUDO

Did you get around to testing this on vmware and vbox? Do you think it
would need to be tested? I could to that.

Henning

>  }
>  


^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: [PATCH 3/3] vm-img: Handle sparse files efficiently
  2022-03-07  9:04   ` Henning Schild
@ 2022-03-07  9:46     ` Jan Kiszka
  2022-03-07 14:09       ` Henning Schild
  0 siblings, 1 reply; 14+ messages in thread
From: Jan Kiszka @ 2022-03-07  9:46 UTC (permalink / raw)
  To: Henning Schild; +Cc: isar-users

On 07.03.22 10:04, Henning Schild wrote:
> Am Sat, 5 Mar 2022 17:34:50 +0100
> schrieb Jan Kiszka <jan.kiszka@siemens.com>:
> 
>> From: Jan Kiszka <jan.kiszka@siemens.com>
>>
>> Saves space in case the image contains any sparse files.
>>
>> Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
>> ---
>>  meta/classes/vm-img.bbclass | 6 +++---
>>  1 file changed, 3 insertions(+), 3 deletions(-)
>>
>> diff --git a/meta/classes/vm-img.bbclass b/meta/classes/vm-img.bbclass
>> index 4bc977b9..27aedf12 100644
>> --- a/meta/classes/vm-img.bbclass
>> +++ b/meta/classes/vm-img.bbclass
>> @@ -93,15 +93,15 @@ do_create_ova() {
>>          export VM_UUID=$(uuidgen)
>>          # create ovf
>>          cat ${PP_WORK}/${OVF_TEMPLATE_STAGE2} | envsubst >
>> ${PP_DEPLOY}/${OVA_NAME}.ovf
>> -        tar -cvf ${PP_DEPLOY}/${OVA_NAME}.ova -C ${PP_DEPLOY}
>> ${OVA_NAME}.ovf
>> +        tar -cvSf ${PP_DEPLOY}/${OVA_NAME}.ova -C ${PP_DEPLOY}
>> ${OVA_NAME}.ovf 
>>          # VirtualBox needs here a manifest file. VMware does accept
>> that format. if [ "${VMDK_SUBFORMAT}" = "monolithicSparse" ]; then
> 
> Not sure that already suggested it but we did not do it.
> 
>>              echo
>> "SHA${OVA_SHA_ALG}(${VIRTUAL_MACHINE_IMAGE_FILE})=$(sha${OVA_SHA_ALG}sum
>> ${PP_DEPLOY}/${VIRTUAL_MACHINE_IMAGE_FILE} | cut -d' ' -f1)" >>
>> ${PP_DEPLOY}/${OVA_NAME}.mf echo
>> "SHA${OVA_SHA_ALG}(${OVA_NAME}.ovf)=$(sha${OVA_SHA_ALG}sum
>> ${PP_DEPLOY}/${OVA_NAME}.ovf | cut -d' ' -f1)" >>
>> ${PP_DEPLOY}/${OVA_NAME}.mf
>> -            tar -uvf ${PP_DEPLOY}/${OVA_NAME}.ova -C ${PP_DEPLOY}
>> ${OVA_NAME}.mf
>> +            tar -uvSf ${PP_DEPLOY}/${OVA_NAME}.ova -C ${PP_DEPLOY}
>> ${OVA_NAME}.mf fi
>> -        tar -uvf ${PP_DEPLOY}/${OVA_NAME}.ova -C ${PP_DEPLOY}
>> ${VIRTUAL_MACHINE_IMAGE_FILE}
>> +        tar -uvSf ${PP_DEPLOY}/${OVA_NAME}.ova -C ${PP_DEPLOY}
>> ${VIRTUAL_MACHINE_IMAGE_FILE} EOSUDO
> 
> Did you get around to testing this on vmware and vbox? Do you think it
> would need to be tested? I could to that.
> 

That would be valuable. Just add a sparse file to the image (dd
if=/dev/zero of=sparse.file bs=1G seek=1 count=0) and check if that
bloats the files. And, of course, check if things still boot.

Thanks,
Jan

-- 
Siemens AG, Technology
Competence Center Embedded Linux

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: [PATCH 3/3] vm-img: Handle sparse files efficiently
  2022-03-07  9:46     ` Jan Kiszka
@ 2022-03-07 14:09       ` Henning Schild
  2022-03-07 14:41         ` Jan Kiszka
  0 siblings, 1 reply; 14+ messages in thread
From: Henning Schild @ 2022-03-07 14:09 UTC (permalink / raw)
  To: Jan Kiszka; +Cc: isar-users

Am Mon, 7 Mar 2022 10:46:34 +0100
schrieb Jan Kiszka <jan.kiszka@siemens.com>:

> On 07.03.22 10:04, Henning Schild wrote:
> > Am Sat, 5 Mar 2022 17:34:50 +0100
> > schrieb Jan Kiszka <jan.kiszka@siemens.com>:
> >   
> >> From: Jan Kiszka <jan.kiszka@siemens.com>
> >>
> >> Saves space in case the image contains any sparse files.
> >>
> >> Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
> >> ---
> >>  meta/classes/vm-img.bbclass | 6 +++---
> >>  1 file changed, 3 insertions(+), 3 deletions(-)
> >>
> >> diff --git a/meta/classes/vm-img.bbclass
> >> b/meta/classes/vm-img.bbclass index 4bc977b9..27aedf12 100644
> >> --- a/meta/classes/vm-img.bbclass
> >> +++ b/meta/classes/vm-img.bbclass
> >> @@ -93,15 +93,15 @@ do_create_ova() {
> >>          export VM_UUID=$(uuidgen)
> >>          # create ovf
> >>          cat ${PP_WORK}/${OVF_TEMPLATE_STAGE2} | envsubst >
> >> ${PP_DEPLOY}/${OVA_NAME}.ovf
> >> -        tar -cvf ${PP_DEPLOY}/${OVA_NAME}.ova -C ${PP_DEPLOY}
> >> ${OVA_NAME}.ovf
> >> +        tar -cvSf ${PP_DEPLOY}/${OVA_NAME}.ova -C ${PP_DEPLOY}
> >> ${OVA_NAME}.ovf 
> >>          # VirtualBox needs here a manifest file. VMware does
> >> accept that format. if [ "${VMDK_SUBFORMAT}" = "monolithicSparse"
> >> ]; then  
> > 
> > Not sure that already suggested it but we did not do it.
> >   
> >>              echo
> >> "SHA${OVA_SHA_ALG}(${VIRTUAL_MACHINE_IMAGE_FILE})=$(sha${OVA_SHA_ALG}sum
> >> ${PP_DEPLOY}/${VIRTUAL_MACHINE_IMAGE_FILE} | cut -d' ' -f1)" >>
> >> ${PP_DEPLOY}/${OVA_NAME}.mf echo
> >> "SHA${OVA_SHA_ALG}(${OVA_NAME}.ovf)=$(sha${OVA_SHA_ALG}sum
> >> ${PP_DEPLOY}/${OVA_NAME}.ovf | cut -d' ' -f1)" >>
> >> ${PP_DEPLOY}/${OVA_NAME}.mf
> >> -            tar -uvf ${PP_DEPLOY}/${OVA_NAME}.ova -C ${PP_DEPLOY}
> >> ${OVA_NAME}.mf
> >> +            tar -uvSf ${PP_DEPLOY}/${OVA_NAME}.ova -C ${PP_DEPLOY}
> >> ${OVA_NAME}.mf fi
> >> -        tar -uvf ${PP_DEPLOY}/${OVA_NAME}.ova -C ${PP_DEPLOY}
> >> ${VIRTUAL_MACHINE_IMAGE_FILE}
> >> +        tar -uvSf ${PP_DEPLOY}/${OVA_NAME}.ova -C ${PP_DEPLOY}
> >> ${VIRTUAL_MACHINE_IMAGE_FILE} EOSUDO  
> > 
> > Did you get around to testing this on vmware and vbox? Do you think
> > it would need to be tested? I could to that.
> >   
> 
> That would be valuable. Just add a sparse file to the image (dd
> if=/dev/zero of=sparse.file bs=1G seek=1 count=0) and check if that
> bloats the files. And, of course, check if things still boot.

NACK, breaks ova import into both vmware and virtualbox. I compared and
it is really this patch which breaks both.

Did not check the vmware logs but vbox says
VERR_TAR_UNSUPPORTED_GNU_HDR_TYPE

So this tarball does not support it because its consumers both do not.

Henning

> Thanks,
> Jan
> 


^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: [PATCH 3/3] vm-img: Handle sparse files efficiently
  2022-03-07 14:09       ` Henning Schild
@ 2022-03-07 14:41         ` Jan Kiszka
  2022-03-07 14:53           ` Henning Schild
  0 siblings, 1 reply; 14+ messages in thread
From: Jan Kiszka @ 2022-03-07 14:41 UTC (permalink / raw)
  To: Henning Schild; +Cc: isar-users

On 07.03.22 15:09, Henning Schild wrote:
> Am Mon, 7 Mar 2022 10:46:34 +0100
> schrieb Jan Kiszka <jan.kiszka@siemens.com>:
> 
>> On 07.03.22 10:04, Henning Schild wrote:
>>> Am Sat, 5 Mar 2022 17:34:50 +0100
>>> schrieb Jan Kiszka <jan.kiszka@siemens.com>:
>>>   
>>>> From: Jan Kiszka <jan.kiszka@siemens.com>
>>>>
>>>> Saves space in case the image contains any sparse files.
>>>>
>>>> Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
>>>> ---
>>>>  meta/classes/vm-img.bbclass | 6 +++---
>>>>  1 file changed, 3 insertions(+), 3 deletions(-)
>>>>
>>>> diff --git a/meta/classes/vm-img.bbclass
>>>> b/meta/classes/vm-img.bbclass index 4bc977b9..27aedf12 100644
>>>> --- a/meta/classes/vm-img.bbclass
>>>> +++ b/meta/classes/vm-img.bbclass
>>>> @@ -93,15 +93,15 @@ do_create_ova() {
>>>>          export VM_UUID=$(uuidgen)
>>>>          # create ovf
>>>>          cat ${PP_WORK}/${OVF_TEMPLATE_STAGE2} | envsubst >
>>>> ${PP_DEPLOY}/${OVA_NAME}.ovf
>>>> -        tar -cvf ${PP_DEPLOY}/${OVA_NAME}.ova -C ${PP_DEPLOY}
>>>> ${OVA_NAME}.ovf
>>>> +        tar -cvSf ${PP_DEPLOY}/${OVA_NAME}.ova -C ${PP_DEPLOY}
>>>> ${OVA_NAME}.ovf 
>>>>          # VirtualBox needs here a manifest file. VMware does
>>>> accept that format. if [ "${VMDK_SUBFORMAT}" = "monolithicSparse"
>>>> ]; then  
>>>
>>> Not sure that already suggested it but we did not do it.
>>>   
>>>>              echo
>>>> "SHA${OVA_SHA_ALG}(${VIRTUAL_MACHINE_IMAGE_FILE})=$(sha${OVA_SHA_ALG}sum
>>>> ${PP_DEPLOY}/${VIRTUAL_MACHINE_IMAGE_FILE} | cut -d' ' -f1)" >>
>>>> ${PP_DEPLOY}/${OVA_NAME}.mf echo
>>>> "SHA${OVA_SHA_ALG}(${OVA_NAME}.ovf)=$(sha${OVA_SHA_ALG}sum
>>>> ${PP_DEPLOY}/${OVA_NAME}.ovf | cut -d' ' -f1)" >>
>>>> ${PP_DEPLOY}/${OVA_NAME}.mf
>>>> -            tar -uvf ${PP_DEPLOY}/${OVA_NAME}.ova -C ${PP_DEPLOY}
>>>> ${OVA_NAME}.mf
>>>> +            tar -uvSf ${PP_DEPLOY}/${OVA_NAME}.ova -C ${PP_DEPLOY}
>>>> ${OVA_NAME}.mf fi
>>>> -        tar -uvf ${PP_DEPLOY}/${OVA_NAME}.ova -C ${PP_DEPLOY}
>>>> ${VIRTUAL_MACHINE_IMAGE_FILE}
>>>> +        tar -uvSf ${PP_DEPLOY}/${OVA_NAME}.ova -C ${PP_DEPLOY}
>>>> ${VIRTUAL_MACHINE_IMAGE_FILE} EOSUDO  
>>>
>>> Did you get around to testing this on vmware and vbox? Do you think
>>> it would need to be tested? I could to that.
>>>   
>>
>> That would be valuable. Just add a sparse file to the image (dd
>> if=/dev/zero of=sparse.file bs=1G seek=1 count=0) and check if that
>> bloats the files. And, of course, check if things still boot.
> 
> NACK, breaks ova import into both vmware and virtualbox. I compared and
> it is really this patch which breaks both.
> 
> Did not check the vmware logs but vbox says
> VERR_TAR_UNSUPPORTED_GNU_HDR_TYPE

Amazingly primitive format, this OVO...

> 
> So this tarball does not support it because its consumers both do not.
> 

OK, thanks for testing. We may then consider whether to take patch 2 as
well as it may surprise constrained consumers. Patch 1 is fine as it
only targets Isar itself.

Jan

-- 
Siemens AG, Technology
Competence Center Embedded Linux

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: [PATCH 3/3] vm-img: Handle sparse files efficiently
  2022-03-07 14:41         ` Jan Kiszka
@ 2022-03-07 14:53           ` Henning Schild
  0 siblings, 0 replies; 14+ messages in thread
From: Henning Schild @ 2022-03-07 14:53 UTC (permalink / raw)
  To: Jan Kiszka; +Cc: isar-users

Am Mon, 7 Mar 2022 15:41:04 +0100
schrieb Jan Kiszka <jan.kiszka@siemens.com>:

> On 07.03.22 15:09, Henning Schild wrote:
> > Am Mon, 7 Mar 2022 10:46:34 +0100
> > schrieb Jan Kiszka <jan.kiszka@siemens.com>:
> >   
> >> On 07.03.22 10:04, Henning Schild wrote:  
> >>> Am Sat, 5 Mar 2022 17:34:50 +0100
> >>> schrieb Jan Kiszka <jan.kiszka@siemens.com>:
> >>>     
> >>>> From: Jan Kiszka <jan.kiszka@siemens.com>
> >>>>
> >>>> Saves space in case the image contains any sparse files.
> >>>>
> >>>> Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
> >>>> ---
> >>>>  meta/classes/vm-img.bbclass | 6 +++---
> >>>>  1 file changed, 3 insertions(+), 3 deletions(-)
> >>>>
> >>>> diff --git a/meta/classes/vm-img.bbclass
> >>>> b/meta/classes/vm-img.bbclass index 4bc977b9..27aedf12 100644
> >>>> --- a/meta/classes/vm-img.bbclass
> >>>> +++ b/meta/classes/vm-img.bbclass
> >>>> @@ -93,15 +93,15 @@ do_create_ova() {
> >>>>          export VM_UUID=$(uuidgen)
> >>>>          # create ovf
> >>>>          cat ${PP_WORK}/${OVF_TEMPLATE_STAGE2} | envsubst >
> >>>> ${PP_DEPLOY}/${OVA_NAME}.ovf
> >>>> -        tar -cvf ${PP_DEPLOY}/${OVA_NAME}.ova -C ${PP_DEPLOY}
> >>>> ${OVA_NAME}.ovf
> >>>> +        tar -cvSf ${PP_DEPLOY}/${OVA_NAME}.ova -C ${PP_DEPLOY}
> >>>> ${OVA_NAME}.ovf 
> >>>>          # VirtualBox needs here a manifest file. VMware does
> >>>> accept that format. if [ "${VMDK_SUBFORMAT}" = "monolithicSparse"
> >>>> ]; then    
> >>>
> >>> Not sure that already suggested it but we did not do it.
> >>>     
> >>>>              echo
> >>>> "SHA${OVA_SHA_ALG}(${VIRTUAL_MACHINE_IMAGE_FILE})=$(sha${OVA_SHA_ALG}sum
> >>>> ${PP_DEPLOY}/${VIRTUAL_MACHINE_IMAGE_FILE} | cut -d' ' -f1)" >>
> >>>> ${PP_DEPLOY}/${OVA_NAME}.mf echo
> >>>> "SHA${OVA_SHA_ALG}(${OVA_NAME}.ovf)=$(sha${OVA_SHA_ALG}sum
> >>>> ${PP_DEPLOY}/${OVA_NAME}.ovf | cut -d' ' -f1)" >>
> >>>> ${PP_DEPLOY}/${OVA_NAME}.mf
> >>>> -            tar -uvf ${PP_DEPLOY}/${OVA_NAME}.ova -C
> >>>> ${PP_DEPLOY} ${OVA_NAME}.mf
> >>>> +            tar -uvSf ${PP_DEPLOY}/${OVA_NAME}.ova -C
> >>>> ${PP_DEPLOY} ${OVA_NAME}.mf fi
> >>>> -        tar -uvf ${PP_DEPLOY}/${OVA_NAME}.ova -C ${PP_DEPLOY}
> >>>> ${VIRTUAL_MACHINE_IMAGE_FILE}
> >>>> +        tar -uvSf ${PP_DEPLOY}/${OVA_NAME}.ova -C ${PP_DEPLOY}
> >>>> ${VIRTUAL_MACHINE_IMAGE_FILE} EOSUDO    
> >>>
> >>> Did you get around to testing this on vmware and vbox? Do you
> >>> think it would need to be tested? I could to that.
> >>>     
> >>
> >> That would be valuable. Just add a sparse file to the image (dd
> >> if=/dev/zero of=sparse.file bs=1G seek=1 count=0) and check if that
> >> bloats the files. And, of course, check if things still boot.  
> > 
> > NACK, breaks ova import into both vmware and virtualbox. I compared
> > and it is really this patch which breaks both.
> > 
> > Did not check the vmware logs but vbox says
> > VERR_TAR_UNSUPPORTED_GNU_HDR_TYPE  
> 
> Amazingly primitive format, this OVO...

Not sure, the ova is just a tar of the disk image and an xml. That disk
image itself can itself be sparse.

see VMDK_SUBFORMAT

> > 
> > So this tarball does not support it because its consumers both do
> > not. 
> 
> OK, thanks for testing. We may then consider whether to take patch 2
> as well as it may surprise constrained consumers. Patch 1 is fine as
> it only targets Isar itself.

I think p2 can default to -S and maybe make the args configurable.

But i think Adriaan might have some generic remarks on the targz-img
anyways.

I usually use it for nfs so for me the S would add value. If i remember
to specify it on unpack ;).

Henning
 
> Jan
> 


^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: [PATCH 0/3] Sparse tar generation
  2022-03-05 16:34 [PATCH 0/3] Sparse tar generation Jan Kiszka
                   ` (2 preceding siblings ...)
  2022-03-05 16:34 ` [PATCH 3/3] vm-img: " Jan Kiszka
@ 2022-04-21 16:05 ` Jan Kiszka
  2022-04-25  6:04   ` Anton Mikanovich
  3 siblings, 1 reply; 14+ messages in thread
From: Jan Kiszka @ 2022-04-21 16:05 UTC (permalink / raw)
  To: isar-users, Schmidt, Adriaan

On 05.03.22 17:34, Jan Kiszka wrote:
> Just some small optimization: Consistently use tar -S when generating
> archives to avoid expanding sparse files.
> 
> Jan
> 
> Jan Kiszka (3):
>   sstate: Generate image packages using sparse
>   targz-img: Handle sparse files efficiently
>   vm-img: Handle sparse files efficiently
> 
>  meta/classes/rootfs.bbclass                         | 2 +-
>  meta/classes/targz-img.bbclass                      | 2 +-
>  meta/classes/vm-img.bbclass                         | 6 +++---
>  meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 2 +-
>  4 files changed, 6 insertions(+), 6 deletions(-)
> 

Ping for patch 1 and 2 (3 is dropped due to breaking compatibility),
specifically towards Adriaan.

Jan

-- 
Siemens AG, Technology
Competence Center Embedded Linux

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: [PATCH 0/3] Sparse tar generation
  2022-04-21 16:05 ` [PATCH 0/3] Sparse tar generation Jan Kiszka
@ 2022-04-25  6:04   ` Anton Mikanovich
  0 siblings, 0 replies; 14+ messages in thread
From: Anton Mikanovich @ 2022-04-25  6:04 UTC (permalink / raw)
  To: Jan Kiszka, isar-users, Schmidt, Adriaan

21.04.2022 19:05, Jan Kiszka wrote:
> On 05.03.22 17:34, Jan Kiszka wrote:
>> Just some small optimization: Consistently use tar -S when generating
>> archives to avoid expanding sparse files.
>>
>> Jan
>>
>> Jan Kiszka (3):
>>    sstate: Generate image packages using sparse
>>    targz-img: Handle sparse files efficiently
>>    vm-img: Handle sparse files efficiently
>>
>>   meta/classes/rootfs.bbclass                         | 2 +-
>>   meta/classes/targz-img.bbclass                      | 2 +-
>>   meta/classes/vm-img.bbclass                         | 6 +++---
>>   meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 2 +-
>>   4 files changed, 6 insertions(+), 6 deletions(-)
>>
> Ping for patch 1 and 2 (3 is dropped due to breaking compatibility),
> specifically towards Adriaan.
>
> Jan

We are ready to merge if there will be no v2 and we'll have an approve.


^ permalink raw reply	[flat|nested] 14+ messages in thread

end of thread, other threads:[~2022-05-24 19:03 UTC | newest]

Thread overview: 14+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-03-05 16:34 [PATCH 0/3] Sparse tar generation Jan Kiszka
2022-03-05 16:34 ` [PATCH 1/3] sstate: Generate image packages using sparse Jan Kiszka
2022-05-24 19:03   ` Jan Kiszka
2022-03-05 16:34 ` [PATCH 2/3] targz-img: Handle sparse files efficiently Jan Kiszka
2022-03-07  9:02   ` Henning Schild
2022-03-07  9:43     ` Jan Kiszka
2022-03-05 16:34 ` [PATCH 3/3] vm-img: " Jan Kiszka
2022-03-07  9:04   ` Henning Schild
2022-03-07  9:46     ` Jan Kiszka
2022-03-07 14:09       ` Henning Schild
2022-03-07 14:41         ` Jan Kiszka
2022-03-07 14:53           ` Henning Schild
2022-04-21 16:05 ` [PATCH 0/3] Sparse tar generation Jan Kiszka
2022-04-25  6:04   ` Anton Mikanovich

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox