Avoid to produce the bmap stuff that double the size

Avoid to produce the bmap stuff that double the size

[Roberto A. Foglietta]

[-- Attachment #1: Type: text/plain, Size: 614 bytes --]

Hi all,

for small footprint embedded system having a bmap is a must and do not
overload the drive but when ISAR is used to build 9.3 GB images, then the
bmap stuff is useless, redundant and takes precious saving space. I can
remove but I wish not produce it in first place:

$ find build/tmp/deploy/ -name \*wic\*
build/tmp/deploy/images/debx86/eval-image-debian-bullseye-debx86.wic
build/tmp/deploy/images/debx86/eval-image-debian-bullseye-debx86.wic.p2
build/tmp/deploy/images/debx86/eval-image-debian-bullseye-debx86.wic.bmap
build/tmp/deploy/images/debx86/eval-image-debian-bullseye-debx86.wic.p1

Thanks, R-

[-- Attachment #2: Type: text/html, Size: 729 bytes --]

Re: Avoid to produce the bmap stuff that double the size

[Jan Kiszka]

On 04.10.22 07:46, Roberto A. Foglietta wrote:
> Hi all,
> 
> for small footprint embedded system having a bmap is a must and do not
> overload the drive but when ISAR is used to build 9.3 GB images, then
> the bmap stuff is useless, redundant and takes precious saving space. I
> can remove but I wish not produce it in first place:
> 
> $ find build/tmp/deploy/ -name \*wic\*
> build/tmp/deploy/images/debx86/eval-image-debian-bullseye-debx86.wic
> build/tmp/deploy/images/debx86/eval-image-debian-bullseye-debx86.wic.p2
> build/tmp/deploy/images/debx86/eval-image-debian-bullseye-debx86.wic.bmap
> build/tmp/deploy/images/debx86/eval-image-debian-bullseye-debx86.wic.p1
> 

I don't think you understood how bmap works and what relates to it. Only
the .bmap file comes from it. And if that is helpful or not depends on
how sparse the image is, not how large. But even with a fully populated
image, bmaptool trumps dd for other reasons.

The listed partition files relate to the discussion in
https://groups.google.com/g/isar-users/c/LtjoygRabns/m/YZsK6S6vCwAJ.

Jan

-- 
Siemens AG, Technology
Competence Center Embedded Linux

RE: Avoid to produce the bmap stuff that double the size

[Moessbauer, Felix]

> From: isar-users@googlegroups.com <isar-users@googlegroups.com> On Behalf Of Roberto A. Foglietta
> Sent: Tuesday, October 4, 2022 1:46 PM
> To: isar-users@googlegroups.com
> Subject: Avoid to produce the bmap stuff that double the size
> 
> Hi all,
> 
> for small footprint embedded system having a bmap is a must and do not overload the drive but when ISAR is used to build 9.3 GB images, then the bmap stuff is useless, redundant and takes precious saving space. I can remove but I wish not produce it in first place:

Hi Roberto,

The bmap itself is just a tiny xml file containing the offsets+sizes in the .wic file that contain relevant data.
This is especially important when adding additional free space at the end of a partition.

The large files are the partitions itself (*.p<x>), and these - unfortunately - do not get compressed even if the IMAGE_FSTYPE is set to "wic.xz".
Last week I sent a patch series that also fixes this:
" [PATCH v7 0/6] use xz and gzip on host (outside chroot)"

Please give it a try and if it works, feel free to ack.
Testers are always welcome 😉

P.S. Please send your emails as plain-text, not as HTML.

Best regards,
Felix

> 
> $ find build/tmp/deploy/ -name \*wic\*
> build/tmp/deploy/images/debx86/eval-image-debian-bullseye-debx86.wic
> build/tmp/deploy/images/debx86/eval-image-debian-bullseye-debx86.wic.p2
> build/tmp/deploy/images/debx86/eval-image-debian-bullseye-debx86.wic.bmap
> build/tmp/deploy/images/debx86/eval-image-debian-bullseye-debx86.wic.p1
> 
> Thanks, R->

Re: Avoid to produce the bmap stuff that double the size

[Roberto A. Foglietta]

Il giorno mar 4 ott 2022 alle ore 08:02 Moessbauer, Felix
<felix.moessbauer@siemens.com> ha scritto:
>
> > From: isar-users@googlegroups.com <isar-users@googlegroups.com> On Behalf Of Roberto A. Foglietta
> > Sent: Tuesday, October 4, 2022 1:46 PM
> > To: isar-users@googlegroups.com
> > Subject: Avoid to produce the bmap stuff that double the size
>
> The large files are the partitions itself (*.p<x>), and these - unfortunately - do not get compressed even if the IMAGE_FSTYPE is set to "wic.xz".
> Last week I sent a patch series that also fixes this:
> " [PATCH v7 0/6] use xz and gzip on host (outside chroot)"
>
> Please give it a try and if it works, feel free to ack.
> Testers are always welcome

Thanks Felix, you got the point. I will give it a try at that patch.

Best, R-

P.S.: finally I found how to send text only e-mail from my google account. :-)

Re: Avoid to produce the bmap stuff that double the size

[Roberto A. Foglietta]

Il giorno mar 4 ott 2022 alle ore 08:00 Jan Kiszka
<jan.kiszka@siemens.com> ha scritto:
>
> On 04.10.22 07:46, Roberto A. Foglietta wrote:
> > Hi all,
> >
> > for small footprint embedded system having a bmap is a must and do not
> > overload the drive but when ISAR is used to build 9.3 GB images, then
> > the bmap stuff is useless, redundant and takes precious saving space. I
> > can remove but I wish not produce it in first place:
> >
> > $ find build/tmp/deploy/ -name \*wic\*
> > build/tmp/deploy/images/debx86/eval-image-debian-bullseye-debx86.wic
> > build/tmp/deploy/images/debx86/eval-image-debian-bullseye-debx86.wic.p2
> > build/tmp/deploy/images/debx86/eval-image-debian-bullseye-debx86.wic.bmap
> > build/tmp/deploy/images/debx86/eval-image-debian-bullseye-debx86.wic.p1
> >
>
> I don't think you understood how bmap works and what relates to it. Only
> the .bmap file comes from it. And if that is helpful or not depends on
> how sparse the image is, not how large. But even with a fully populated
> image, bmaptool trumps dd for other reasons.
>
> The listed partition files relate to the discussion in
> https://groups.google.com/g/isar-users/c/LtjoygRabns/m/YZsK6S6vCwAJ.

$ file eval-image-debian-bullseye-debx86.wic.p?
eval-image-debian-bullseye-debx86.wic.p1: DOS/MBR boot sector, code
offset 0x3c+2, OEM-ID "mkfs.fat", sectors/cluster 4, reserved sectors
4, root entries 512, Media descriptor 0xf8, sectors/FAT 104,
sectors/track 32, heads 8, sectors 106688 (volumes > 32 MB), serial
number 0xc2d97979, label: "efi        ", FAT (16 bit)
eval-image-debian-bullseye-debx86.wic.p2: Linux rev 1.0 ext4
filesystem data, UUID=0e80e944-2eda-43e0-91e1-3b2e163f22cc, volume
name "rootfs" (extents) (64bit) (large files) (huge files)

Dear Jan,

 you are right because despite knowing bmap, there was some kind of
confusion in my question: I mixed up the use of bmap with the creation
/ deletion of the partitions.

 The main point was: I am using ISAR to develop a set of recipes not
for embedded so I do 100 large builds and 1 transfer. Under my PoV, I
like tools like bmap but I am more concerned in speed-up the build.

 For example - supposing that those using ISAR to make a build and
transfer it are 1.000 while those using ISAR to develop recipes are 50
- the most value still stays in speed-up the build rather than the
transfer.

 Especially because those who use ISAR for developing a set of
recipes, are who they are contributing to its development.

 Considering the question and the answers received, I noticed that

 - Jan was focused on technical aspects
 - Felix on the practical aspects like saving space
 - I was pointing out to the value

 The value of a technical tool in order to do certain optimisations
rather than others: fast build vs saving disk space vs transfer speed.

 Confusion is a very bad destinationation but a very good starting point... :-)

 Best regards, R-

 P.S.: the google mail allows the text-only just by the web interface
while the Android application uses HTML always. No, I am writing from
home and I use the web browser.

Re: Avoid to produce the bmap stuff that double the size

[Roberto A. Foglietta]

Il giorno mar 4 ott 2022 alle ore 08:02 Moessbauer, Felix
<felix.moessbauer@siemens.com> ha scritto:
>

> " [PATCH v7 0/6] use xz and gzip on host (outside chroot)"
>
> Please give it a try and if it works, feel free to ack.
> Testers are always welcome 😉
>

Hi Felix,

 this is what I found and fits with what was expecting to find:

$ sudo find build -name \*.wic\*
build/tmp/deploy/images/debx86/eval-image-debian-bullseye-debx86.wic

after having used the next with your patch and having activated it in
the basic image:

diff --git a/recipes-core/images/eval-image-template.inc
b/recipes-core/images/eval-image-template.inc
index 9a9cd3b..48bfc01 100644
--- a/recipes-core/images/eval-image-template.inc
+++ b/recipes-core/images/eval-image-template.inc
@@ -22,3 +22,4 @@ IMAGE_INSTALL += "isar-disable-apt-cache"
 IMAGE_INSTALL += "expand-on-first-boot"

 WIC_IMAGER_INSTALL += "btrfs-progs"
+WIC_DEPLOY_PARTITIONS = "0"

Best regards, R-

Re: Avoid to produce the bmap stuff that double the size

[Roberto A. Foglietta]

[-- Attachment #1: Type: text/plain, Size: 753 bytes --]

Il giorno sab 8 ott 2022 alle ore 10:28 Roberto A. Foglietta
<roberto.foglietta@gmail.com> ha scritto:
>
> Il giorno mar 4 ott 2022 alle ore 08:02 Moessbauer, Felix
> <felix.moessbauer@siemens.com> ha scritto:
> >
>
> > " [PATCH v7 0/6] use xz and gzip on host (outside chroot)"
> >
> > Please give it a try and if it works, feel free to ack.
> > Testers are always welcome 😉
> >
>
> Hi Felix,
>
>  this is what I found and fits with what was expecting to find:
>
> $ sudo find build -name \*.wic\*
> build/tmp/deploy/images/debx86/eval-image-debian-bullseye-debx86.wic

Is it possible to avoid creating a sstate-cache of the {image or rootfs}?
Mine is over 9GB and its cache is useless. Screenshot 12KB attached.

Thanks, R.

[-- Attachment #2: sstate-cache.png --]
[-- Type: image/png, Size: 11903 bytes --]

RE: Avoid to produce the bmap stuff that double the size

[Moessbauer, Felix]

> -----Original Message-----
> From: Roberto A. Foglietta <roberto.foglietta@gmail.com>
> Sent: Saturday, October 8, 2022 5:09 PM
> To: Moessbauer, Felix (T CED SES-DE) <felix.moessbauer@siemens.com>
> Cc: isar-users@googlegroups.com; Kiszka, Jan (T CED)
> <jan.kiszka@siemens.com>
> Subject: Re: Avoid to produce the bmap stuff that double the size
> 
> Il giorno sab 8 ott 2022 alle ore 10:28 Roberto A. Foglietta
> <roberto.foglietta@gmail.com> ha scritto:
> >
> > Il giorno mar 4 ott 2022 alle ore 08:02 Moessbauer, Felix
> > <felix.moessbauer@siemens.com> ha scritto:
> > >
> >
> > > " [PATCH v7 0/6] use xz and gzip on host (outside chroot)"
> > >
> > > Please give it a try and if it works, feel free to ack.
> > > Testers are always welcome 😉
> > >
> >
> > Hi Felix,
> >
> >  this is what I found and fits with what was expecting to find:
> >
> > $ sudo find build -name \*.wic\*
> > build/tmp/deploy/images/debx86/eval-image-debian-bullseye-debx86.wic
> 
> Is it possible to avoid creating a sstate-cache of the {image or rootfs}?
> Mine is over 9GB and its cache is useless. Screenshot 12KB attached.

Hi Roberto,

Please do not mix up topics in this thread.

Disabling sstate cache is possible by passing --no-setscene to bitbake as stated in the docs [1]
Apart from that, I doubt that you do not profit from the cache.
Just have a look at the "match: <X>%" output when bitbake starts.

Felix

[1] https://github.com/ilbers/isar/blob/next/doc/user_manual.md#using-sstate-cache
> 
> Thanks, R.

Re: Avoid to produce the bmap stuff that double the size

[Roberto A. Foglietta]

Il giorno sab 8 ott 2022 alle ore 11:42 Moessbauer, Felix
<felix.moessbauer@siemens.com> ha scritto:

>
> Hi Roberto,
>
> Please do not mix up topics in this thread.
>

ok

> Disabling sstate cache is possible by passing --no-setscene to bitbake as stated in the docs [1]
> Apart from that, I doubt that you do not profit from the cache.
> Just have a look at the "match: <X>%" output when bitbake starts.
>
> Felix
>
> [1] https://github.com/ilbers/isar/blob/next/doc/user_manual.md#using-sstate-cache

ok, now I connected the thread with the subsistem - you are right,
setscene is very important for me.

Thanks, R-