Re: [PATCH v2 3/3] Changes for a faster build using less disk space

[Uladzimir Bely <ubely@ilbers.de>]

[-- Attachment #1: Type: text/plain, Size: 5040 bytes --]

In the email from Friday, 6 January 2023 18:58:26 +03 user Henning Schild 
wrote:
> Am Fri,  6 Jan 2023 07:48:09 +0100
> 
> schrieb Uladzimir Bely <ubely@ilbers.de>:
> > From: "Roberto A. Foglietta" <roberto.foglietta@gmail.com>
> > 
> > Faster build for larger projects with a lot of debian packages
> > 
> > This patch leverages few ways to accomplish the task to optimise
> > the large buildings.
> > 
> > Put a CACHEDIR.TAG file in each cache folder in such a way every
> > tar that uses the option --exclude-caches could be avoided
> > to include debian packages saving a lot of time and disk space.
> > 
> > Signed-off-by: Roberto A. Foglietta <roberto.foglietta@gmail.com>
> > Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
> > ---
> > 
> >  meta/classes/deb-dl-dir.bbclass                     | 2 +-
> >  meta/classes/rootfs.bbclass                         | 3 ++-
> >  meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 7 ++++++-
> >  3 files changed, 9 insertions(+), 3 deletions(-)
> > 
> > diff --git a/meta/classes/deb-dl-dir.bbclass
> > b/meta/classes/deb-dl-dir.bbclass index 7db25251..3afad4e8 100644
> > --- a/meta/classes/deb-dl-dir.bbclass
> > +++ b/meta/classes/deb-dl-dir.bbclass
> > @@ -102,7 +102,7 @@ deb_dl_dir_export() {
> > 
> >              -maxdepth 1 -type f -iname '*\.deb' |\
> >          
> >          while read p; do
> >          
> >              # skip files from a previous export
> > 
> > -            [ -f "${pc}/${p##*/}" ] && continue
> > +            [ -e "${pc}/${p##*/}" ] && continue
> > 
> >              # can not reuse bitbake function here, this is basically
> >              # "repo_contains_package"
> >              package=$(find "${REPO_ISAR_DIR}"/"${DISTRO}" -name
> > 
> > ${p##*/}) diff --git a/meta/classes/rootfs.bbclass
> > b/meta/classes/rootfs.bbclass index 786682d9..f274443d 100644
> > --- a/meta/classes/rootfs.bbclass
> > +++ b/meta/classes/rootfs.bbclass
> > @@ -308,7 +308,8 @@ rootfs_install_sstate_prepare() {
> > 
> >      # so we use some mount magic to prevent that
> >      mkdir -p ${WORKDIR}/mnt/rootfs
> >      sudo mount --bind ${WORKDIR}/rootfs ${WORKDIR}/mnt/rootfs -o ro
> > 
> > -    sudo tar -C ${WORKDIR}/mnt -cpSf rootfs.tar --one-file-system
> > rootfs
> > +    lopts="--one-file-system --exclude-caches"
> > +    sudo tar -C ${WORKDIR}/mnt -cpSf rootfs.tar $lopts rootfs
> > 
> >      sudo umount ${WORKDIR}/mnt/rootfs
> >      sudo chown $(id -u):$(id -g) rootfs.tar
> >  
> >  }
> > 
> > diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
> > b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc index
> > cb0079ec..44019f95 100644 ---
> > a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc +++
> > b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc @@ -395,6
> > +395,10 @@ EOSUDO
> > 
> >      # Cleanup apt cache
> >      sudo -Es chroot "${ROOTFSDIR}" /usr/bin/apt-get -y clean
> > 
> > +
> > +    # Don't include apt cache into sstate cache by marking the
> > directory
> > +    # with a CACHEDIR.TAG file with a special signature
> > +    sudo -Es sh -c "echo Signature: 8a477f597d28d172789f06886806bc55
> 
> What is that "special signature" all about? Does anyone later care to
> remove the file again, or do we risk it ending up in the final rootfs?
> 
> I find this a little weird and would prefer if the calls to tar would
> simply exclude this by path and not by magic tool specific file.
> 

Original patch from Roberto included both `--exclude-caches` and `--
exclude=var/cache/apt/archives"` options for `tar`. First one was expected to 
automatically exclude all directories that contain special CACHEDIR.TAG file 
[1]. But it appeared not to work. Investigating the problem shown that it 
should not be empty file: GNU tar expects file starts with "Signature: 
8a477f597d28d172789f06886806bc55" line.

It means, all the stuff with multiple "touch CACHEDIR.TAG" was useless and the 
actually only option "--exclude=var/cache/apt/archives" did the trick.

I fixed first and removed second option. But you are right, we can do vice 
versa and it should look even simpler.

Final rootfs doesn't have this file since we do "apt-get clean" for it.

[1] https://bford.info/cachedir/

> It could be made a central function so we have all those tar calls in
> one place.
> 
> In this patch i see three places ... hard to maintain.
> 
> Henning
> 

> > > ${ROOTFSDIR}/var/cache/apt/archives/CACHEDIR.TAG" }
> >  
> >  addtask bootstrap before do_build after do_generate_keyrings
> > 
> > @@ -405,7 +409,8 @@ SSTATEPOSTINSTFUNCS += "bootstrap_sstate_finalize"
> > 
> >  bootstrap_sstate_prepare() {
> >  
> >      # this runs in SSTATE_BUILDDIR, which will be deleted
> > 
> > automatically
> > -    sudo tar -C $(dirname "${ROOTFSDIR}") -cpSf bootstrap.tar
> > --one-file-system $(basename "${ROOTFSDIR}")
> > +    lopts="--one-file-system --exclude-caches"
> > +    sudo tar -C $(dirname "${ROOTFSDIR}") -cpSf bootstrap.tar $lopts
> > $(basename "${ROOTFSDIR}") sudo chown $(id -u):$(id -g) bootstrap.tar
> > 
> >  }


[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 488 bytes --]