* [PATCH 0/3] Sparse tar generation
@ 2022-03-05 16:34 Jan Kiszka
2022-03-05 16:34 ` [PATCH 1/3] sstate: Generate image packages using sparse Jan Kiszka
` (3 more replies)
0 siblings, 4 replies; 14+ messages in thread
From: Jan Kiszka @ 2022-03-05 16:34 UTC (permalink / raw)
To: isar-users
Just some small optimization: Consistently use tar -S when generating
archives to avoid expanding sparse files.
Jan
Jan Kiszka (3):
sstate: Generate image packages using sparse
targz-img: Handle sparse files efficiently
vm-img: Handle sparse files efficiently
meta/classes/rootfs.bbclass | 2 +-
meta/classes/targz-img.bbclass | 2 +-
meta/classes/vm-img.bbclass | 6 +++---
meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 2 +-
4 files changed, 6 insertions(+), 6 deletions(-)
--
2.34.1
^ permalink raw reply [flat|nested] 14+ messages in thread
* [PATCH 1/3] sstate: Generate image packages using sparse
2022-03-05 16:34 [PATCH 0/3] Sparse tar generation Jan Kiszka
@ 2022-03-05 16:34 ` Jan Kiszka
2022-05-24 19:03 ` Jan Kiszka
2022-03-05 16:34 ` [PATCH 2/3] targz-img: Handle sparse files efficiently Jan Kiszka
` (2 subsequent siblings)
3 siblings, 1 reply; 14+ messages in thread
From: Jan Kiszka @ 2022-03-05 16:34 UTC (permalink / raw)
To: isar-users
From: Jan Kiszka <jan.kiszka@siemens.com>
Avoids needless expansion of sparse files while filling the sstate
cache.
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
meta/classes/rootfs.bbclass | 2 +-
meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass
index ba86c60a..927af13f 100644
--- a/meta/classes/rootfs.bbclass
+++ b/meta/classes/rootfs.bbclass
@@ -299,7 +299,7 @@ rootfs_install_sstate_prepare() {
# so we use some mount magic to prevent that
mkdir -p ${WORKDIR}/mnt/rootfs
sudo mount --bind ${WORKDIR}/rootfs ${WORKDIR}/mnt/rootfs -o ro
- sudo tar -C ${WORKDIR}/mnt -cpf rootfs.tar --one-file-system rootfs
+ sudo tar -C ${WORKDIR}/mnt -cpSf rootfs.tar --one-file-system rootfs
sudo umount ${WORKDIR}/mnt/rootfs
sudo chown $(id -u):$(id -g) rootfs.tar
}
diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
index d6f90f63..1b16f874 100644
--- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
+++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
@@ -400,7 +400,7 @@ SSTATEPOSTINSTFUNCS += "bootstrap_sstate_finalize"
bootstrap_sstate_prepare() {
# this runs in SSTATE_BUILDDIR, which will be deleted automatically
- sudo tar -C $(dirname "${ROOTFSDIR}") -cpf bootstrap.tar --one-file-system $(basename "${ROOTFSDIR}")
+ sudo tar -C $(dirname "${ROOTFSDIR}") -cpSf bootstrap.tar --one-file-system $(basename "${ROOTFSDIR}")
sudo chown $(id -u):$(id -g) bootstrap.tar
}
--
2.34.1
^ permalink raw reply [flat|nested] 14+ messages in thread
* [PATCH 2/3] targz-img: Handle sparse files efficiently
2022-03-05 16:34 [PATCH 0/3] Sparse tar generation Jan Kiszka
2022-03-05 16:34 ` [PATCH 1/3] sstate: Generate image packages using sparse Jan Kiszka
@ 2022-03-05 16:34 ` Jan Kiszka
2022-03-07 9:02 ` Henning Schild
2022-03-05 16:34 ` [PATCH 3/3] vm-img: " Jan Kiszka
2022-04-21 16:05 ` [PATCH 0/3] Sparse tar generation Jan Kiszka
3 siblings, 1 reply; 14+ messages in thread
From: Jan Kiszka @ 2022-03-05 16:34 UTC (permalink / raw)
To: isar-users
From: Jan Kiszka <jan.kiszka@siemens.com>
Saves space in case the image contains any sparse files.
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
meta/classes/targz-img.bbclass | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/meta/classes/targz-img.bbclass b/meta/classes/targz-img.bbclass
index bf94af02..638e7767 100644
--- a/meta/classes/targz-img.bbclass
+++ b/meta/classes/targz-img.bbclass
@@ -7,7 +7,7 @@ TARGZ_IMAGE_FILE = "${DEPLOY_DIR_IMAGE}/${IMAGE_FULLNAME}.tar.gz"
do_targz_image() {
rm -f ${TARGZ_IMAGE_FILE}
- sudo tar -cvzf ${TARGZ_IMAGE_FILE} --one-file-system -C ${IMAGE_ROOTFS} .
+ sudo tar -cvzSf ${TARGZ_IMAGE_FILE} --one-file-system -C ${IMAGE_ROOTFS} .
sudo chown $(id -u):$(id -g) ${TARGZ_IMAGE_FILE}
}
--
2.34.1
^ permalink raw reply [flat|nested] 14+ messages in thread
* [PATCH 3/3] vm-img: Handle sparse files efficiently
2022-03-05 16:34 [PATCH 0/3] Sparse tar generation Jan Kiszka
2022-03-05 16:34 ` [PATCH 1/3] sstate: Generate image packages using sparse Jan Kiszka
2022-03-05 16:34 ` [PATCH 2/3] targz-img: Handle sparse files efficiently Jan Kiszka
@ 2022-03-05 16:34 ` Jan Kiszka
2022-03-07 9:04 ` Henning Schild
2022-04-21 16:05 ` [PATCH 0/3] Sparse tar generation Jan Kiszka
3 siblings, 1 reply; 14+ messages in thread
From: Jan Kiszka @ 2022-03-05 16:34 UTC (permalink / raw)
To: isar-users
From: Jan Kiszka <jan.kiszka@siemens.com>
Saves space in case the image contains any sparse files.
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
meta/classes/vm-img.bbclass | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/meta/classes/vm-img.bbclass b/meta/classes/vm-img.bbclass
index 4bc977b9..27aedf12 100644
--- a/meta/classes/vm-img.bbclass
+++ b/meta/classes/vm-img.bbclass
@@ -93,15 +93,15 @@ do_create_ova() {
export VM_UUID=$(uuidgen)
# create ovf
cat ${PP_WORK}/${OVF_TEMPLATE_STAGE2} | envsubst > ${PP_DEPLOY}/${OVA_NAME}.ovf
- tar -cvf ${PP_DEPLOY}/${OVA_NAME}.ova -C ${PP_DEPLOY} ${OVA_NAME}.ovf
+ tar -cvSf ${PP_DEPLOY}/${OVA_NAME}.ova -C ${PP_DEPLOY} ${OVA_NAME}.ovf
# VirtualBox needs here a manifest file. VMware does accept that format.
if [ "${VMDK_SUBFORMAT}" = "monolithicSparse" ]; then
echo "SHA${OVA_SHA_ALG}(${VIRTUAL_MACHINE_IMAGE_FILE})=$(sha${OVA_SHA_ALG}sum ${PP_DEPLOY}/${VIRTUAL_MACHINE_IMAGE_FILE} | cut -d' ' -f1)" >> ${PP_DEPLOY}/${OVA_NAME}.mf
echo "SHA${OVA_SHA_ALG}(${OVA_NAME}.ovf)=$(sha${OVA_SHA_ALG}sum ${PP_DEPLOY}/${OVA_NAME}.ovf | cut -d' ' -f1)" >> ${PP_DEPLOY}/${OVA_NAME}.mf
- tar -uvf ${PP_DEPLOY}/${OVA_NAME}.ova -C ${PP_DEPLOY} ${OVA_NAME}.mf
+ tar -uvSf ${PP_DEPLOY}/${OVA_NAME}.ova -C ${PP_DEPLOY} ${OVA_NAME}.mf
fi
- tar -uvf ${PP_DEPLOY}/${OVA_NAME}.ova -C ${PP_DEPLOY} ${VIRTUAL_MACHINE_IMAGE_FILE}
+ tar -uvSf ${PP_DEPLOY}/${OVA_NAME}.ova -C ${PP_DEPLOY} ${VIRTUAL_MACHINE_IMAGE_FILE}
EOSUDO
}
--
2.34.1
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH 2/3] targz-img: Handle sparse files efficiently
2022-03-05 16:34 ` [PATCH 2/3] targz-img: Handle sparse files efficiently Jan Kiszka
@ 2022-03-07 9:02 ` Henning Schild
2022-03-07 9:43 ` Jan Kiszka
0 siblings, 1 reply; 14+ messages in thread
From: Henning Schild @ 2022-03-07 9:02 UTC (permalink / raw)
To: Jan Kiszka; +Cc: isar-users
Am Sat, 5 Mar 2022 17:34:49 +0100
schrieb Jan Kiszka <jan.kiszka@siemens.com>:
> From: Jan Kiszka <jan.kiszka@siemens.com>
>
> Saves space in case the image contains any sparse files.
>
> Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
> ---
> meta/classes/targz-img.bbclass | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/meta/classes/targz-img.bbclass
> b/meta/classes/targz-img.bbclass index bf94af02..638e7767 100644
> --- a/meta/classes/targz-img.bbclass
> +++ b/meta/classes/targz-img.bbclass
> @@ -7,7 +7,7 @@ TARGZ_IMAGE_FILE =
> "${DEPLOY_DIR_IMAGE}/${IMAGE_FULLNAME}.tar.gz"
> do_targz_image() {
> rm -f ${TARGZ_IMAGE_FILE}
> - sudo tar -cvzf ${TARGZ_IMAGE_FILE} --one-file-system -C
> ${IMAGE_ROOTFS} .
> + sudo tar -cvzSf ${TARGZ_IMAGE_FILE} --one-file-system -C
> ${IMAGE_ROOTFS} . sudo chown $(id -u):$(id -g) ${TARGZ_IMAGE_FILE}
I assume that can be unpacked with or without -S now, so no interface
change. right?
Henning
> }
>
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH 3/3] vm-img: Handle sparse files efficiently
2022-03-05 16:34 ` [PATCH 3/3] vm-img: " Jan Kiszka
@ 2022-03-07 9:04 ` Henning Schild
2022-03-07 9:46 ` Jan Kiszka
0 siblings, 1 reply; 14+ messages in thread
From: Henning Schild @ 2022-03-07 9:04 UTC (permalink / raw)
To: Jan Kiszka; +Cc: isar-users
Am Sat, 5 Mar 2022 17:34:50 +0100
schrieb Jan Kiszka <jan.kiszka@siemens.com>:
> From: Jan Kiszka <jan.kiszka@siemens.com>
>
> Saves space in case the image contains any sparse files.
>
> Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
> ---
> meta/classes/vm-img.bbclass | 6 +++---
> 1 file changed, 3 insertions(+), 3 deletions(-)
>
> diff --git a/meta/classes/vm-img.bbclass b/meta/classes/vm-img.bbclass
> index 4bc977b9..27aedf12 100644
> --- a/meta/classes/vm-img.bbclass
> +++ b/meta/classes/vm-img.bbclass
> @@ -93,15 +93,15 @@ do_create_ova() {
> export VM_UUID=$(uuidgen)
> # create ovf
> cat ${PP_WORK}/${OVF_TEMPLATE_STAGE2} | envsubst >
> ${PP_DEPLOY}/${OVA_NAME}.ovf
> - tar -cvf ${PP_DEPLOY}/${OVA_NAME}.ova -C ${PP_DEPLOY}
> ${OVA_NAME}.ovf
> + tar -cvSf ${PP_DEPLOY}/${OVA_NAME}.ova -C ${PP_DEPLOY}
> ${OVA_NAME}.ovf
> # VirtualBox needs here a manifest file. VMware does accept
> that format. if [ "${VMDK_SUBFORMAT}" = "monolithicSparse" ]; then
Not sure that already suggested it but we did not do it.
> echo
> "SHA${OVA_SHA_ALG}(${VIRTUAL_MACHINE_IMAGE_FILE})=$(sha${OVA_SHA_ALG}sum
> ${PP_DEPLOY}/${VIRTUAL_MACHINE_IMAGE_FILE} | cut -d' ' -f1)" >>
> ${PP_DEPLOY}/${OVA_NAME}.mf echo
> "SHA${OVA_SHA_ALG}(${OVA_NAME}.ovf)=$(sha${OVA_SHA_ALG}sum
> ${PP_DEPLOY}/${OVA_NAME}.ovf | cut -d' ' -f1)" >>
> ${PP_DEPLOY}/${OVA_NAME}.mf
> - tar -uvf ${PP_DEPLOY}/${OVA_NAME}.ova -C ${PP_DEPLOY}
> ${OVA_NAME}.mf
> + tar -uvSf ${PP_DEPLOY}/${OVA_NAME}.ova -C ${PP_DEPLOY}
> ${OVA_NAME}.mf fi
> - tar -uvf ${PP_DEPLOY}/${OVA_NAME}.ova -C ${PP_DEPLOY}
> ${VIRTUAL_MACHINE_IMAGE_FILE}
> + tar -uvSf ${PP_DEPLOY}/${OVA_NAME}.ova -C ${PP_DEPLOY}
> ${VIRTUAL_MACHINE_IMAGE_FILE} EOSUDO
Did you get around to testing this on vmware and vbox? Do you think it
would need to be tested? I could to that.
Henning
> }
>
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH 2/3] targz-img: Handle sparse files efficiently
2022-03-07 9:02 ` Henning Schild
@ 2022-03-07 9:43 ` Jan Kiszka
0 siblings, 0 replies; 14+ messages in thread
From: Jan Kiszka @ 2022-03-07 9:43 UTC (permalink / raw)
To: Henning Schild; +Cc: isar-users
On 07.03.22 10:02, Henning Schild wrote:
> Am Sat, 5 Mar 2022 17:34:49 +0100
> schrieb Jan Kiszka <jan.kiszka@siemens.com>:
>
>> From: Jan Kiszka <jan.kiszka@siemens.com>
>>
>> Saves space in case the image contains any sparse files.
>>
>> Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
>> ---
>> meta/classes/targz-img.bbclass | 2 +-
>> 1 file changed, 1 insertion(+), 1 deletion(-)
>>
>> diff --git a/meta/classes/targz-img.bbclass
>> b/meta/classes/targz-img.bbclass index bf94af02..638e7767 100644
>> --- a/meta/classes/targz-img.bbclass
>> +++ b/meta/classes/targz-img.bbclass
>> @@ -7,7 +7,7 @@ TARGZ_IMAGE_FILE =
>> "${DEPLOY_DIR_IMAGE}/${IMAGE_FULLNAME}.tar.gz"
>> do_targz_image() {
>> rm -f ${TARGZ_IMAGE_FILE}
>> - sudo tar -cvzf ${TARGZ_IMAGE_FILE} --one-file-system -C
>> ${IMAGE_ROOTFS} .
>> + sudo tar -cvzSf ${TARGZ_IMAGE_FILE} --one-file-system -C
>> ${IMAGE_ROOTFS} . sudo chown $(id -u):$(id -g) ${TARGZ_IMAGE_FILE}
>
> I assume that can be unpacked with or without -S now, so no interface
> change. right?
>
Yes, see also "man tar".
Jan
--
Siemens AG, Technology
Competence Center Embedded Linux
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH 3/3] vm-img: Handle sparse files efficiently
2022-03-07 9:04 ` Henning Schild
@ 2022-03-07 9:46 ` Jan Kiszka
2022-03-07 14:09 ` Henning Schild
0 siblings, 1 reply; 14+ messages in thread
From: Jan Kiszka @ 2022-03-07 9:46 UTC (permalink / raw)
To: Henning Schild; +Cc: isar-users
On 07.03.22 10:04, Henning Schild wrote:
> Am Sat, 5 Mar 2022 17:34:50 +0100
> schrieb Jan Kiszka <jan.kiszka@siemens.com>:
>
>> From: Jan Kiszka <jan.kiszka@siemens.com>
>>
>> Saves space in case the image contains any sparse files.
>>
>> Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
>> ---
>> meta/classes/vm-img.bbclass | 6 +++---
>> 1 file changed, 3 insertions(+), 3 deletions(-)
>>
>> diff --git a/meta/classes/vm-img.bbclass b/meta/classes/vm-img.bbclass
>> index 4bc977b9..27aedf12 100644
>> --- a/meta/classes/vm-img.bbclass
>> +++ b/meta/classes/vm-img.bbclass
>> @@ -93,15 +93,15 @@ do_create_ova() {
>> export VM_UUID=$(uuidgen)
>> # create ovf
>> cat ${PP_WORK}/${OVF_TEMPLATE_STAGE2} | envsubst >
>> ${PP_DEPLOY}/${OVA_NAME}.ovf
>> - tar -cvf ${PP_DEPLOY}/${OVA_NAME}.ova -C ${PP_DEPLOY}
>> ${OVA_NAME}.ovf
>> + tar -cvSf ${PP_DEPLOY}/${OVA_NAME}.ova -C ${PP_DEPLOY}
>> ${OVA_NAME}.ovf
>> # VirtualBox needs here a manifest file. VMware does accept
>> that format. if [ "${VMDK_SUBFORMAT}" = "monolithicSparse" ]; then
>
> Not sure that already suggested it but we did not do it.
>
>> echo
>> "SHA${OVA_SHA_ALG}(${VIRTUAL_MACHINE_IMAGE_FILE})=$(sha${OVA_SHA_ALG}sum
>> ${PP_DEPLOY}/${VIRTUAL_MACHINE_IMAGE_FILE} | cut -d' ' -f1)" >>
>> ${PP_DEPLOY}/${OVA_NAME}.mf echo
>> "SHA${OVA_SHA_ALG}(${OVA_NAME}.ovf)=$(sha${OVA_SHA_ALG}sum
>> ${PP_DEPLOY}/${OVA_NAME}.ovf | cut -d' ' -f1)" >>
>> ${PP_DEPLOY}/${OVA_NAME}.mf
>> - tar -uvf ${PP_DEPLOY}/${OVA_NAME}.ova -C ${PP_DEPLOY}
>> ${OVA_NAME}.mf
>> + tar -uvSf ${PP_DEPLOY}/${OVA_NAME}.ova -C ${PP_DEPLOY}
>> ${OVA_NAME}.mf fi
>> - tar -uvf ${PP_DEPLOY}/${OVA_NAME}.ova -C ${PP_DEPLOY}
>> ${VIRTUAL_MACHINE_IMAGE_FILE}
>> + tar -uvSf ${PP_DEPLOY}/${OVA_NAME}.ova -C ${PP_DEPLOY}
>> ${VIRTUAL_MACHINE_IMAGE_FILE} EOSUDO
>
> Did you get around to testing this on vmware and vbox? Do you think it
> would need to be tested? I could to that.
>
That would be valuable. Just add a sparse file to the image (dd
if=/dev/zero of=sparse.file bs=1G seek=1 count=0) and check if that
bloats the files. And, of course, check if things still boot.
Thanks,
Jan
--
Siemens AG, Technology
Competence Center Embedded Linux
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH 3/3] vm-img: Handle sparse files efficiently
2022-03-07 9:46 ` Jan Kiszka
@ 2022-03-07 14:09 ` Henning Schild
2022-03-07 14:41 ` Jan Kiszka
0 siblings, 1 reply; 14+ messages in thread
From: Henning Schild @ 2022-03-07 14:09 UTC (permalink / raw)
To: Jan Kiszka; +Cc: isar-users
Am Mon, 7 Mar 2022 10:46:34 +0100
schrieb Jan Kiszka <jan.kiszka@siemens.com>:
> On 07.03.22 10:04, Henning Schild wrote:
> > Am Sat, 5 Mar 2022 17:34:50 +0100
> > schrieb Jan Kiszka <jan.kiszka@siemens.com>:
> >
> >> From: Jan Kiszka <jan.kiszka@siemens.com>
> >>
> >> Saves space in case the image contains any sparse files.
> >>
> >> Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
> >> ---
> >> meta/classes/vm-img.bbclass | 6 +++---
> >> 1 file changed, 3 insertions(+), 3 deletions(-)
> >>
> >> diff --git a/meta/classes/vm-img.bbclass
> >> b/meta/classes/vm-img.bbclass index 4bc977b9..27aedf12 100644
> >> --- a/meta/classes/vm-img.bbclass
> >> +++ b/meta/classes/vm-img.bbclass
> >> @@ -93,15 +93,15 @@ do_create_ova() {
> >> export VM_UUID=$(uuidgen)
> >> # create ovf
> >> cat ${PP_WORK}/${OVF_TEMPLATE_STAGE2} | envsubst >
> >> ${PP_DEPLOY}/${OVA_NAME}.ovf
> >> - tar -cvf ${PP_DEPLOY}/${OVA_NAME}.ova -C ${PP_DEPLOY}
> >> ${OVA_NAME}.ovf
> >> + tar -cvSf ${PP_DEPLOY}/${OVA_NAME}.ova -C ${PP_DEPLOY}
> >> ${OVA_NAME}.ovf
> >> # VirtualBox needs here a manifest file. VMware does
> >> accept that format. if [ "${VMDK_SUBFORMAT}" = "monolithicSparse"
> >> ]; then
> >
> > Not sure that already suggested it but we did not do it.
> >
> >> echo
> >> "SHA${OVA_SHA_ALG}(${VIRTUAL_MACHINE_IMAGE_FILE})=$(sha${OVA_SHA_ALG}sum
> >> ${PP_DEPLOY}/${VIRTUAL_MACHINE_IMAGE_FILE} | cut -d' ' -f1)" >>
> >> ${PP_DEPLOY}/${OVA_NAME}.mf echo
> >> "SHA${OVA_SHA_ALG}(${OVA_NAME}.ovf)=$(sha${OVA_SHA_ALG}sum
> >> ${PP_DEPLOY}/${OVA_NAME}.ovf | cut -d' ' -f1)" >>
> >> ${PP_DEPLOY}/${OVA_NAME}.mf
> >> - tar -uvf ${PP_DEPLOY}/${OVA_NAME}.ova -C ${PP_DEPLOY}
> >> ${OVA_NAME}.mf
> >> + tar -uvSf ${PP_DEPLOY}/${OVA_NAME}.ova -C ${PP_DEPLOY}
> >> ${OVA_NAME}.mf fi
> >> - tar -uvf ${PP_DEPLOY}/${OVA_NAME}.ova -C ${PP_DEPLOY}
> >> ${VIRTUAL_MACHINE_IMAGE_FILE}
> >> + tar -uvSf ${PP_DEPLOY}/${OVA_NAME}.ova -C ${PP_DEPLOY}
> >> ${VIRTUAL_MACHINE_IMAGE_FILE} EOSUDO
> >
> > Did you get around to testing this on vmware and vbox? Do you think
> > it would need to be tested? I could to that.
> >
>
> That would be valuable. Just add a sparse file to the image (dd
> if=/dev/zero of=sparse.file bs=1G seek=1 count=0) and check if that
> bloats the files. And, of course, check if things still boot.
NACK, breaks ova import into both vmware and virtualbox. I compared and
it is really this patch which breaks both.
Did not check the vmware logs but vbox says
VERR_TAR_UNSUPPORTED_GNU_HDR_TYPE
So this tarball does not support it because its consumers both do not.
Henning
> Thanks,
> Jan
>
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH 3/3] vm-img: Handle sparse files efficiently
2022-03-07 14:09 ` Henning Schild
@ 2022-03-07 14:41 ` Jan Kiszka
2022-03-07 14:53 ` Henning Schild
0 siblings, 1 reply; 14+ messages in thread
From: Jan Kiszka @ 2022-03-07 14:41 UTC (permalink / raw)
To: Henning Schild; +Cc: isar-users
On 07.03.22 15:09, Henning Schild wrote:
> Am Mon, 7 Mar 2022 10:46:34 +0100
> schrieb Jan Kiszka <jan.kiszka@siemens.com>:
>
>> On 07.03.22 10:04, Henning Schild wrote:
>>> Am Sat, 5 Mar 2022 17:34:50 +0100
>>> schrieb Jan Kiszka <jan.kiszka@siemens.com>:
>>>
>>>> From: Jan Kiszka <jan.kiszka@siemens.com>
>>>>
>>>> Saves space in case the image contains any sparse files.
>>>>
>>>> Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
>>>> ---
>>>> meta/classes/vm-img.bbclass | 6 +++---
>>>> 1 file changed, 3 insertions(+), 3 deletions(-)
>>>>
>>>> diff --git a/meta/classes/vm-img.bbclass
>>>> b/meta/classes/vm-img.bbclass index 4bc977b9..27aedf12 100644
>>>> --- a/meta/classes/vm-img.bbclass
>>>> +++ b/meta/classes/vm-img.bbclass
>>>> @@ -93,15 +93,15 @@ do_create_ova() {
>>>> export VM_UUID=$(uuidgen)
>>>> # create ovf
>>>> cat ${PP_WORK}/${OVF_TEMPLATE_STAGE2} | envsubst >
>>>> ${PP_DEPLOY}/${OVA_NAME}.ovf
>>>> - tar -cvf ${PP_DEPLOY}/${OVA_NAME}.ova -C ${PP_DEPLOY}
>>>> ${OVA_NAME}.ovf
>>>> + tar -cvSf ${PP_DEPLOY}/${OVA_NAME}.ova -C ${PP_DEPLOY}
>>>> ${OVA_NAME}.ovf
>>>> # VirtualBox needs here a manifest file. VMware does
>>>> accept that format. if [ "${VMDK_SUBFORMAT}" = "monolithicSparse"
>>>> ]; then
>>>
>>> Not sure that already suggested it but we did not do it.
>>>
>>>> echo
>>>> "SHA${OVA_SHA_ALG}(${VIRTUAL_MACHINE_IMAGE_FILE})=$(sha${OVA_SHA_ALG}sum
>>>> ${PP_DEPLOY}/${VIRTUAL_MACHINE_IMAGE_FILE} | cut -d' ' -f1)" >>
>>>> ${PP_DEPLOY}/${OVA_NAME}.mf echo
>>>> "SHA${OVA_SHA_ALG}(${OVA_NAME}.ovf)=$(sha${OVA_SHA_ALG}sum
>>>> ${PP_DEPLOY}/${OVA_NAME}.ovf | cut -d' ' -f1)" >>
>>>> ${PP_DEPLOY}/${OVA_NAME}.mf
>>>> - tar -uvf ${PP_DEPLOY}/${OVA_NAME}.ova -C ${PP_DEPLOY}
>>>> ${OVA_NAME}.mf
>>>> + tar -uvSf ${PP_DEPLOY}/${OVA_NAME}.ova -C ${PP_DEPLOY}
>>>> ${OVA_NAME}.mf fi
>>>> - tar -uvf ${PP_DEPLOY}/${OVA_NAME}.ova -C ${PP_DEPLOY}
>>>> ${VIRTUAL_MACHINE_IMAGE_FILE}
>>>> + tar -uvSf ${PP_DEPLOY}/${OVA_NAME}.ova -C ${PP_DEPLOY}
>>>> ${VIRTUAL_MACHINE_IMAGE_FILE} EOSUDO
>>>
>>> Did you get around to testing this on vmware and vbox? Do you think
>>> it would need to be tested? I could to that.
>>>
>>
>> That would be valuable. Just add a sparse file to the image (dd
>> if=/dev/zero of=sparse.file bs=1G seek=1 count=0) and check if that
>> bloats the files. And, of course, check if things still boot.
>
> NACK, breaks ova import into both vmware and virtualbox. I compared and
> it is really this patch which breaks both.
>
> Did not check the vmware logs but vbox says
> VERR_TAR_UNSUPPORTED_GNU_HDR_TYPE
Amazingly primitive format, this OVO...
>
> So this tarball does not support it because its consumers both do not.
>
OK, thanks for testing. We may then consider whether to take patch 2 as
well as it may surprise constrained consumers. Patch 1 is fine as it
only targets Isar itself.
Jan
--
Siemens AG, Technology
Competence Center Embedded Linux
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH 3/3] vm-img: Handle sparse files efficiently
2022-03-07 14:41 ` Jan Kiszka
@ 2022-03-07 14:53 ` Henning Schild
0 siblings, 0 replies; 14+ messages in thread
From: Henning Schild @ 2022-03-07 14:53 UTC (permalink / raw)
To: Jan Kiszka; +Cc: isar-users
Am Mon, 7 Mar 2022 15:41:04 +0100
schrieb Jan Kiszka <jan.kiszka@siemens.com>:
> On 07.03.22 15:09, Henning Schild wrote:
> > Am Mon, 7 Mar 2022 10:46:34 +0100
> > schrieb Jan Kiszka <jan.kiszka@siemens.com>:
> >
> >> On 07.03.22 10:04, Henning Schild wrote:
> >>> Am Sat, 5 Mar 2022 17:34:50 +0100
> >>> schrieb Jan Kiszka <jan.kiszka@siemens.com>:
> >>>
> >>>> From: Jan Kiszka <jan.kiszka@siemens.com>
> >>>>
> >>>> Saves space in case the image contains any sparse files.
> >>>>
> >>>> Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
> >>>> ---
> >>>> meta/classes/vm-img.bbclass | 6 +++---
> >>>> 1 file changed, 3 insertions(+), 3 deletions(-)
> >>>>
> >>>> diff --git a/meta/classes/vm-img.bbclass
> >>>> b/meta/classes/vm-img.bbclass index 4bc977b9..27aedf12 100644
> >>>> --- a/meta/classes/vm-img.bbclass
> >>>> +++ b/meta/classes/vm-img.bbclass
> >>>> @@ -93,15 +93,15 @@ do_create_ova() {
> >>>> export VM_UUID=$(uuidgen)
> >>>> # create ovf
> >>>> cat ${PP_WORK}/${OVF_TEMPLATE_STAGE2} | envsubst >
> >>>> ${PP_DEPLOY}/${OVA_NAME}.ovf
> >>>> - tar -cvf ${PP_DEPLOY}/${OVA_NAME}.ova -C ${PP_DEPLOY}
> >>>> ${OVA_NAME}.ovf
> >>>> + tar -cvSf ${PP_DEPLOY}/${OVA_NAME}.ova -C ${PP_DEPLOY}
> >>>> ${OVA_NAME}.ovf
> >>>> # VirtualBox needs here a manifest file. VMware does
> >>>> accept that format. if [ "${VMDK_SUBFORMAT}" = "monolithicSparse"
> >>>> ]; then
> >>>
> >>> Not sure that already suggested it but we did not do it.
> >>>
> >>>> echo
> >>>> "SHA${OVA_SHA_ALG}(${VIRTUAL_MACHINE_IMAGE_FILE})=$(sha${OVA_SHA_ALG}sum
> >>>> ${PP_DEPLOY}/${VIRTUAL_MACHINE_IMAGE_FILE} | cut -d' ' -f1)" >>
> >>>> ${PP_DEPLOY}/${OVA_NAME}.mf echo
> >>>> "SHA${OVA_SHA_ALG}(${OVA_NAME}.ovf)=$(sha${OVA_SHA_ALG}sum
> >>>> ${PP_DEPLOY}/${OVA_NAME}.ovf | cut -d' ' -f1)" >>
> >>>> ${PP_DEPLOY}/${OVA_NAME}.mf
> >>>> - tar -uvf ${PP_DEPLOY}/${OVA_NAME}.ova -C
> >>>> ${PP_DEPLOY} ${OVA_NAME}.mf
> >>>> + tar -uvSf ${PP_DEPLOY}/${OVA_NAME}.ova -C
> >>>> ${PP_DEPLOY} ${OVA_NAME}.mf fi
> >>>> - tar -uvf ${PP_DEPLOY}/${OVA_NAME}.ova -C ${PP_DEPLOY}
> >>>> ${VIRTUAL_MACHINE_IMAGE_FILE}
> >>>> + tar -uvSf ${PP_DEPLOY}/${OVA_NAME}.ova -C ${PP_DEPLOY}
> >>>> ${VIRTUAL_MACHINE_IMAGE_FILE} EOSUDO
> >>>
> >>> Did you get around to testing this on vmware and vbox? Do you
> >>> think it would need to be tested? I could to that.
> >>>
> >>
> >> That would be valuable. Just add a sparse file to the image (dd
> >> if=/dev/zero of=sparse.file bs=1G seek=1 count=0) and check if that
> >> bloats the files. And, of course, check if things still boot.
> >
> > NACK, breaks ova import into both vmware and virtualbox. I compared
> > and it is really this patch which breaks both.
> >
> > Did not check the vmware logs but vbox says
> > VERR_TAR_UNSUPPORTED_GNU_HDR_TYPE
>
> Amazingly primitive format, this OVO...
Not sure, the ova is just a tar of the disk image and an xml. That disk
image itself can itself be sparse.
see VMDK_SUBFORMAT
> >
> > So this tarball does not support it because its consumers both do
> > not.
>
> OK, thanks for testing. We may then consider whether to take patch 2
> as well as it may surprise constrained consumers. Patch 1 is fine as
> it only targets Isar itself.
I think p2 can default to -S and maybe make the args configurable.
But i think Adriaan might have some generic remarks on the targz-img
anyways.
I usually use it for nfs so for me the S would add value. If i remember
to specify it on unpack ;).
Henning
> Jan
>
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH 0/3] Sparse tar generation
2022-03-05 16:34 [PATCH 0/3] Sparse tar generation Jan Kiszka
` (2 preceding siblings ...)
2022-03-05 16:34 ` [PATCH 3/3] vm-img: " Jan Kiszka
@ 2022-04-21 16:05 ` Jan Kiszka
2022-04-25 6:04 ` Anton Mikanovich
3 siblings, 1 reply; 14+ messages in thread
From: Jan Kiszka @ 2022-04-21 16:05 UTC (permalink / raw)
To: isar-users, Schmidt, Adriaan
On 05.03.22 17:34, Jan Kiszka wrote:
> Just some small optimization: Consistently use tar -S when generating
> archives to avoid expanding sparse files.
>
> Jan
>
> Jan Kiszka (3):
> sstate: Generate image packages using sparse
> targz-img: Handle sparse files efficiently
> vm-img: Handle sparse files efficiently
>
> meta/classes/rootfs.bbclass | 2 +-
> meta/classes/targz-img.bbclass | 2 +-
> meta/classes/vm-img.bbclass | 6 +++---
> meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 2 +-
> 4 files changed, 6 insertions(+), 6 deletions(-)
>
Ping for patch 1 and 2 (3 is dropped due to breaking compatibility),
specifically towards Adriaan.
Jan
--
Siemens AG, Technology
Competence Center Embedded Linux
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH 0/3] Sparse tar generation
2022-04-21 16:05 ` [PATCH 0/3] Sparse tar generation Jan Kiszka
@ 2022-04-25 6:04 ` Anton Mikanovich
0 siblings, 0 replies; 14+ messages in thread
From: Anton Mikanovich @ 2022-04-25 6:04 UTC (permalink / raw)
To: Jan Kiszka, isar-users, Schmidt, Adriaan
21.04.2022 19:05, Jan Kiszka wrote:
> On 05.03.22 17:34, Jan Kiszka wrote:
>> Just some small optimization: Consistently use tar -S when generating
>> archives to avoid expanding sparse files.
>>
>> Jan
>>
>> Jan Kiszka (3):
>> sstate: Generate image packages using sparse
>> targz-img: Handle sparse files efficiently
>> vm-img: Handle sparse files efficiently
>>
>> meta/classes/rootfs.bbclass | 2 +-
>> meta/classes/targz-img.bbclass | 2 +-
>> meta/classes/vm-img.bbclass | 6 +++---
>> meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 2 +-
>> 4 files changed, 6 insertions(+), 6 deletions(-)
>>
> Ping for patch 1 and 2 (3 is dropped due to breaking compatibility),
> specifically towards Adriaan.
>
> Jan
We are ready to merge if there will be no v2 and we'll have an approve.
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH 1/3] sstate: Generate image packages using sparse
2022-03-05 16:34 ` [PATCH 1/3] sstate: Generate image packages using sparse Jan Kiszka
@ 2022-05-24 19:03 ` Jan Kiszka
0 siblings, 0 replies; 14+ messages in thread
From: Jan Kiszka @ 2022-05-24 19:03 UTC (permalink / raw)
To: isar-users, Schmidt, Adriaan; +Cc: Anton Mikanovich
On 05.03.22 17:34, Jan Kiszka wrote:
> From: Jan Kiszka <jan.kiszka@siemens.com>
>
> Avoids needless expansion of sparse files while filling the sstate
> cache.
>
> Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
> ---
> meta/classes/rootfs.bbclass | 2 +-
> meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 2 +-
> 2 files changed, 2 insertions(+), 2 deletions(-)
>
> diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass
> index ba86c60a..927af13f 100644
> --- a/meta/classes/rootfs.bbclass
> +++ b/meta/classes/rootfs.bbclass
> @@ -299,7 +299,7 @@ rootfs_install_sstate_prepare() {
> # so we use some mount magic to prevent that
> mkdir -p ${WORKDIR}/mnt/rootfs
> sudo mount --bind ${WORKDIR}/rootfs ${WORKDIR}/mnt/rootfs -o ro
> - sudo tar -C ${WORKDIR}/mnt -cpf rootfs.tar --one-file-system rootfs
> + sudo tar -C ${WORKDIR}/mnt -cpSf rootfs.tar --one-file-system rootfs
> sudo umount ${WORKDIR}/mnt/rootfs
> sudo chown $(id -u):$(id -g) rootfs.tar
> }
> diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
> index d6f90f63..1b16f874 100644
> --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
> +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
> @@ -400,7 +400,7 @@ SSTATEPOSTINSTFUNCS += "bootstrap_sstate_finalize"
>
> bootstrap_sstate_prepare() {
> # this runs in SSTATE_BUILDDIR, which will be deleted automatically
> - sudo tar -C $(dirname "${ROOTFSDIR}") -cpf bootstrap.tar --one-file-system $(basename "${ROOTFSDIR}")
> + sudo tar -C $(dirname "${ROOTFSDIR}") -cpSf bootstrap.tar --one-file-system $(basename "${ROOTFSDIR}")
> sudo chown $(id -u):$(id -g) bootstrap.tar
> }
>
Ping again - still applies and seems reasonable to me.
Jan
--
Siemens AG, Technology
Competence Center Embedded Linux
^ permalink raw reply [flat|nested] 14+ messages in thread
end of thread, other threads:[~2022-05-24 19:03 UTC | newest]
Thread overview: 14+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-03-05 16:34 [PATCH 0/3] Sparse tar generation Jan Kiszka
2022-03-05 16:34 ` [PATCH 1/3] sstate: Generate image packages using sparse Jan Kiszka
2022-05-24 19:03 ` Jan Kiszka
2022-03-05 16:34 ` [PATCH 2/3] targz-img: Handle sparse files efficiently Jan Kiszka
2022-03-07 9:02 ` Henning Schild
2022-03-07 9:43 ` Jan Kiszka
2022-03-05 16:34 ` [PATCH 3/3] vm-img: " Jan Kiszka
2022-03-07 9:04 ` Henning Schild
2022-03-07 9:46 ` Jan Kiszka
2022-03-07 14:09 ` Henning Schild
2022-03-07 14:41 ` Jan Kiszka
2022-03-07 14:53 ` Henning Schild
2022-04-21 16:05 ` [PATCH 0/3] Sparse tar generation Jan Kiszka
2022-04-25 6:04 ` Anton Mikanovich
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox