Re: [PATCH v2 3/3] Changes for a faster build using less disk space

[Henning Schild <henning.schild@siemens.com>]

Am Fri, 6 Jan 2023 20:38:05 +0300
schrieb Uladzimir Bely <ubely@ilbers.de>:

> In the email from Friday, 6 January 2023 18:58:26 +03 user Henning
> Schild wrote:
> > Am Fri,  6 Jan 2023 07:48:09 +0100
> > 
> > schrieb Uladzimir Bely <ubely@ilbers.de>:  
> > > From: "Roberto A. Foglietta" <roberto.foglietta@gmail.com>
> > > 
> > > Faster build for larger projects with a lot of debian packages
> > > 
> > > This patch leverages few ways to accomplish the task to optimise
> > > the large buildings.
> > > 
> > > Put a CACHEDIR.TAG file in each cache folder in such a way every
> > > tar that uses the option --exclude-caches could be avoided
> > > to include debian packages saving a lot of time and disk space.
> > > 
> > > Signed-off-by: Roberto A. Foglietta <roberto.foglietta@gmail.com>
> > > Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
> > > ---
> > > 
> > >  meta/classes/deb-dl-dir.bbclass                     | 2 +-
> > >  meta/classes/rootfs.bbclass                         | 3 ++-
> > >  meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 7 ++++++-
> > >  3 files changed, 9 insertions(+), 3 deletions(-)
> > > 
> > > diff --git a/meta/classes/deb-dl-dir.bbclass
> > > b/meta/classes/deb-dl-dir.bbclass index 7db25251..3afad4e8 100644
> > > --- a/meta/classes/deb-dl-dir.bbclass
> > > +++ b/meta/classes/deb-dl-dir.bbclass
> > > @@ -102,7 +102,7 @@ deb_dl_dir_export() {
> > > 
> > >              -maxdepth 1 -type f -iname '*\.deb' |\
> > >          
> > >          while read p; do
> > >          
> > >              # skip files from a previous export
> > > 
> > > -            [ -f "${pc}/${p##*/}" ] && continue
> > > +            [ -e "${pc}/${p##*/}" ] && continue
> > > 
> > >              # can not reuse bitbake function here, this is
> > > basically # "repo_contains_package"
> > >              package=$(find "${REPO_ISAR_DIR}"/"${DISTRO}" -name
> > > 
> > > ${p##*/}) diff --git a/meta/classes/rootfs.bbclass
> > > b/meta/classes/rootfs.bbclass index 786682d9..f274443d 100644
> > > --- a/meta/classes/rootfs.bbclass
> > > +++ b/meta/classes/rootfs.bbclass
> > > @@ -308,7 +308,8 @@ rootfs_install_sstate_prepare() {
> > > 
> > >      # so we use some mount magic to prevent that
> > >      mkdir -p ${WORKDIR}/mnt/rootfs
> > >      sudo mount --bind ${WORKDIR}/rootfs ${WORKDIR}/mnt/rootfs -o
> > > ro
> > > 
> > > -    sudo tar -C ${WORKDIR}/mnt -cpSf rootfs.tar --one-file-system
> > > rootfs
> > > +    lopts="--one-file-system --exclude-caches"
> > > +    sudo tar -C ${WORKDIR}/mnt -cpSf rootfs.tar $lopts rootfs
> > > 
> > >      sudo umount ${WORKDIR}/mnt/rootfs
> > >      sudo chown $(id -u):$(id -g) rootfs.tar
> > >  
> > >  }
> > > 
> > > diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
> > > b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc index
> > > cb0079ec..44019f95 100644 ---
> > > a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc +++
> > > b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc @@ -395,6
> > > +395,10 @@ EOSUDO
> > > 
> > >      # Cleanup apt cache
> > >      sudo -Es chroot "${ROOTFSDIR}" /usr/bin/apt-get -y clean
> > > 
> > > +
> > > +    # Don't include apt cache into sstate cache by marking the
> > > directory
> > > +    # with a CACHEDIR.TAG file with a special signature
> > > +    sudo -Es sh -c "echo Signature:
> > > 8a477f597d28d172789f06886806bc55  
> > 
> > What is that "special signature" all about? Does anyone later care
> > to remove the file again, or do we risk it ending up in the final
> > rootfs?
> > 
> > I find this a little weird and would prefer if the calls to tar
> > would simply exclude this by path and not by magic tool specific
> > file. 
> 
> Original patch from Roberto included both `--exclude-caches` and `--
> exclude=var/cache/apt/archives"` options for `tar`. First one was
> expected to automatically exclude all directories that contain
> special CACHEDIR.TAG file [1]. But it appeared not to work.
> Investigating the problem shown that it should not be empty file: GNU
> tar expects file starts with "Signature:
> 8a477f597d28d172789f06886806bc55" line.
> 
> It means, all the stuff with multiple "touch CACHEDIR.TAG" was
> useless and the actually only option
> "--exclude=var/cache/apt/archives" did the trick.
> 
> I fixed first and removed second option. But you are right, we can do
> vice versa and it should look even simpler.
> 
> Final rootfs doesn't have this file since we do "apt-get clean" for
> it.

I guess we got lucky. Who knows what future versions of apt-get might
do when strange files hang around.

Henning

> [1] https://bford.info/cachedir/
> 
> > It could be made a central function so we have all those tar calls
> > in one place.
> > 
> > In this patch i see three places ... hard to maintain.
> > 
> > Henning
> >   
> 
> > > > ${ROOTFSDIR}/var/cache/apt/archives/CACHEDIR.TAG" }  
> > >  
> > >  addtask bootstrap before do_build after do_generate_keyrings
> > > 
> > > @@ -405,7 +409,8 @@ SSTATEPOSTINSTFUNCS +=
> > > "bootstrap_sstate_finalize"
> > > 
> > >  bootstrap_sstate_prepare() {
> > >  
> > >      # this runs in SSTATE_BUILDDIR, which will be deleted
> > > 
> > > automatically
> > > -    sudo tar -C $(dirname "${ROOTFSDIR}") -cpSf bootstrap.tar
> > > --one-file-system $(basename "${ROOTFSDIR}")
> > > +    lopts="--one-file-system --exclude-caches"
> > > +    sudo tar -C $(dirname "${ROOTFSDIR}") -cpSf bootstrap.tar
> > > $lopts $(basename "${ROOTFSDIR}") sudo chown $(id -u):$(id -g)
> > > bootstrap.tar
> > > 
> > >  }  
>