Re: [RFC v2][PATCH 2/3] build-rep: Add helper class

[Benedikt Niedermayr <benbrenson89@googlemail.com>]

[-- Attachment #1: Type: text/plain, Size: 5959 bytes --]

Am 24.01.2018 um 22:31 schrieb Jan Kiszka:
> On 2018-01-24 21:53, Benedikt Niedermayr wrote:
>> Am 24.01.2018 um 19:48 schrieb Jan Kiszka:
>>> I also started to look into aptly, which I had on my notes for a while.
>>> One reason is that it could act as a manual workaround for the missing
>>> version-freezing support in Isar (pull into aptly repo, take a snapshot,
>>> and then build from it with Isar).
>>>
>>> Another is that we also need some tooling to manage mirrors and
>>> snapshots on the long run, and reprepro, TBHO, smells strange. On first
>>> glance, it looks to me like the multistrap of the repo management tools:
>>> once a cool thing, but now unmaintained.
>>>
>>> Did you dig further into aptly? I'm currently wondering how it
>>> internally manages snapshots, if it does proper deduplication and how
>>> much of that can be preserved when publishing for consumption by, say,
>>> Isar...
>>>
>>> Jan
>>>
>> Yes,
>>
>> Until now I have implemented two approaches.
>>
>> So I will offer some of my results tomorrow.
>>
>> What I first did, was to use aptly as a complete mirror. Aptly creates a
>> mirror and each package, which is requested by Isar will be added to the
>> mirror and then be downloaded from there.
>>
>> Unfortunately that was not as easy as expected, since aptly cannot
>> download packages directly. Aptly was originally developed for creating
>> complete debian mirrors.
>>
>> It has now the "-filter" option, which can be used for narrowing the
>> packages (and also dependecies) to be mirrored.
>>
>> So for each additional package I extended the filter and updated the
>> mirror afterwards, but that architecture appeared to suffer under less
>> performance, since each extension of the filter
>>
>> and subsequent update caused a download of debians packages.gz file (~
>> 10MB).
>>
> You went down the same path I was exploring the past hours. But I
> dropped as well, also because of such traps:
>
> https://groups.google.com/forum/#!searchin/aptly-discuss/partial$20mirroring%7Csort:date/aptly-discuss/SCMrc0w5Mq0/up4RpnMLBgAJ
>
>> My current approach is to create only a local repo after the first Isar
>> build suceeded. This avoids complexity for now, since Isar and all
>> debian tools are resolving dependencies as usual, when
>>
>> running against upstream repos.
> Would be interested to see the changes you did for that.
>
>> After the first build I create a snapshot of the local repo (both is
>> possible: creating snapshots of local repos and also mirrors).
>>
>>
>> I'm just investigating the deduplication topic and will give you response.
>>
> Deduplication seems to be done by aptly simply by creating hardlinks.
> That even works nicely when publishing to a local directory.
>
> Thinking the aptly path further: I quickly started to dream of a lazy
> mirror. What if we could teach it to create a full mirror but only fetch
> packages when someone requests them?
>
> That means we would first need a live publication services using the
> built-in webserver or something like inotify or even fuse for file-based
> publication. Then we could track accesses and pull from upstream on
> demand. After the first build, we would snapshot the result and use that
> from then on. The impact on Isar would be minimal, no separate
> dependency tracking, no Debian logic duplication, simple user interface.
>
> That's very similar to what Claudius initially proposed, but we would be
> building on top of an existing, mature repo management tool.
>
> Jan
>


Hi,


> Would be interested to see the changes you did for that.
I can only give you my current implementation of my apt-cache.bbclass 
and dpkg-cross.bbclass.

The apt-cache.bbclass contains a simple shell based API for accessing 
the aptly functionality.

I will push that results to my next branch at my isar fork tomorrow.


> Deduplication seems to be done by aptly simply by creating hardlinks.
> That even works nicely when publishing to a local directory.
>
> Thinking the aptly path further: I quickly started to dream of a lazy
> mirror. What if we could teach it to create a full mirror but only fetch
> packages when someone requests them?
>
> That means we would first need a live publication services using the
> built-in webserver or something like inotify or even fuse for file-based
> publication. Then we could track accesses and pull from upstream on
> demand. After the first build, we would snapshot the result and use that
> from then on. The impact on Isar would be minimal, no separate
> dependency tracking, no Debian logic duplication, simple user interface.
>
> That's very similar to what Claudius initially proposed, but we would be
> building on top of an existing, mature repo management tool.
That sounds very nice. I don't know how to find good solution/design for 
that. But maybe a good brain wave might change that :-)


I also thinking about combining buildfragments (debian packages) with 
Bitbake's "setscene" mechanism, in order to create a sstate cache 
functionality, like yocto does.

I also don't know how to achieve that goal, since there is very less 
(means none) documentation about setscene, and the only point to start 
from there is the sstate implementation

of yocto itself. So first of all I have to understand the masses of code 
related to the sstate cache.

It is a big goal but it is also very nice feature and may solve a lot of 
problems related to mirroring or snapshoting a repository, since then a 
direct relationship between task hashes and

build fragments (debian packages) would be given.

If any of the task changes in a recipe, a new/old build fragment would 
be the result. And checking out any commit of a meta-layer would relate 
to matching build fragments.

This would be very helpful when checking out to a sooner commit of Isar 
(or any other layer).

A relationship between your current git repository state and related 
mirror could then be guaranteed.



Benni









[-- Attachment #2: apt-cache.bbclass --]
[-- Type: text/plain, Size: 5045 bytes --]

# This software is a part of ISAR.
# Copyright (C) 2017-2018 Mixed-Mode GmbH
#
# This class implements common functionality for the apt cache.
#
# Todo:
# 1. When a function of this API return an error, bitbake should output the last function produced the error!
#
# 2. Add functionality of creating snapshots of local isar repo.
#
#
CFG_FILE = "${CACHE_CONF_DIR}/${CACHE_CFG_FILE}"
MIRROR = "${DISTRO_SUITE}-mirror"
SNAPSHOT_BASENAME="snapshot"
ISAR_REPO_LOCAL = "isar-repo-local"
ISAR_FIRST_BUILD_DONE = "${TMPDIR}/first_build_done"

# Reproducible build is not default
ISAR_REPRODUCIBLE_BUILD ?= "1"


# Rename a snapshot
# $1: Old name.
# $2: New name.
_rename_snapshot() {
	aptly -config=${CFG_FILE} snapshot rename $1 $2
}


# Create a new repository database for Isar generated packages.
# $1: Repository database name
_repo_db_create() {
    aptly -config=${CFG_FILE} \
          repo create \
          -component="main" \
          -distribution="${DISTRO_SUITE}" \
          $1
}


# Drop a created repository database silenty.
_repo_db_drop() {
    aptly -config=${CFG_FILE} \
          repo drop \
          $1 || bbwarn "drop db failed"

}


# Add packages to the local repository database.
# $1: Repository database name.
# $*: Packages to add.
_repo_db_add_package() {
    repo=$1
    shift
    aptly -config=${CFG_FILE} repo add -force-replace $repo $*
}


# Delete packages from the repository database.
# $1: Repository database name.
# $*: Packages to delete.
_repo_db_delete_package() {
    #
    # Todo: Remove from pool? But maybe this is required for snapshots?
    # We have build fragments saved at the pool! That is good.
    #
    # After 3 rebuilds:
    #
    #./public/isar/pool/main/s/swupdate/swupdate_1.0_armhf.deb
    #./pool/df/88/702fa609fbd90b735d7cdb6dfeef_swupdate_1.0_armhf.deb
    #./pool/f0/6e/21d09a07366d7f9f7f6690ea7656_swupdate_1.0_armhf.deb
    #./pool/bd/c3/974c8903d41666d23239a8330627_swupdate_1.0_armhf.deb

    repo=$1
    shift
    aptly -config=${CFG_FILE} repo remove $repo $*
}


# Publish repository consumed by apt. This function can only be called
# for repositories
# $1: Repository database name
# $2: Path Prefix
_repo_db_publish(){
    aptly -config=${CFG_FILE} \
          publish repo \
          -architectures="${DEB_HOST_ARCH},${DISTRO_ARCH}" \
          $1 \
          $2

}


# Drop an already published repository.
# $1: Path prefix
_repo_db_publish_drop() {
    aptly -config=${CFG_FILE} \
          publish drop \
          ${DISTRO_SUITE} \
          $1 || bbwarn "drop published failed"
}


# Switch already published repository to new repository. This
# will change the content of the published repository. This function
# can only be called on published repos.
# $1: Prefix.
_repo_db_published_update() {
    aptly -config=${CFG_FILE} \
          publish update \
          -force-overwrite \
          ${DISTRO_SUITE} $1
}


# Add packages to the isar cache.
# $1: Repository database name.
# $2: Repository prefix
# $*: Packages to add.
cache_add_package() {
    repo=$1
    prefix=$2
    shift 2
    _repo_db_add_package $repo $*
    _repo_db_published_update $prefix
}


# Delete a package from the apt cache.
# This function is used to be called within do_clean(all) task.
# $1: Repository database name.
# $2: Repository prefix
# $*: Packages to delete.
cache_delete_package() {
    repo=$1
    prefix=$2
    shift 2
    _repo_db_delete_package $repo $*
    _repo_db_published_update $prefix
}


cache_create_snapshot() {
    aptly -config=${CFG_FILE} \
          snapshot create ${SNAPSHOT_BASENAME}_${DATETIME} from repo ${ISAR_REPO_LOCAL}
}


cache_load_snapshot() {
  "${FUNCNAME[1]} not implemented."
}

# Setup the apt cache, only if no cache is available, yet.
do_cache_setup() {
    set -E
    _cleanup() {
        ret=$?
        bbwarn "Error in apt-cache.bblcass: ${BASH_SOURCE[0]}: (Line ${BASH_LINENO[0]}, Func:${FUNCNAME[1]})"
        (exit $ret) || bb_exit_handler
    }
    trap '_cleanup' ERR

    # Very first build of isar. Setting up the cache.
    if [ ! -e "${CFG_FILE}" ]; then
        bbplain "Creating local apt cache for first build."
        sed -e 's|##CACHE_DIR##|${CACHE_DIR}|' \
            ${FILESDIR}/${CACHE_CFG_FILE}.in > ${CFG_FILE}

        _repo_db_create ${ISAR_REPO_LOCAL} ${ISAR_CACHE_LOCAL_PREFIX}
        _repo_db_publish ${ISAR_REPO_LOCAL} ${ISAR_CACHE_LOCAL_PREFIX}
    fi


    if [ -e "${ISAR_FIRST_BUILD_DONE}" ] && [ "${REPRODUCIBLE_BUILD_ENABLED}" == "1" ]; then
        bbplain "Reusing last apt cache snapshot: $(aptly --config=${CFG_FILE} --raw -sort=time snapshot list | tail -n 1)"
        return
    fi

    #bbwarn "Republishing repo of last build"
    #_repo_db_publish_drop ${ISAR_CACHE_LOCAL_PREFIX}
    #_repo_db_drop ${ISAR_REPO_LOCAL}
    #_repo_db_create ${ISAR_REPO_LOCAL} ${ISAR_CACHE_LOCAL_PREFIX}
    #_repo_db_publish ${ISAR_REPO_LOCAL} ${ISAR_CACHE_LOCAL_PREFIX}

}
addtask do_cache_setup before do_rootfs
do_cache_setup[dirs] += "${CACHE_CONF_DIR}"
do_cache_setup[lockfiles] = "${DPKG_LOCK}"

[-- Attachment #3: dpkg-cross.bbclass --]
[-- Type: text/plain, Size: 2089 bytes --]

# This software is a part of ISAR.
# Copyright (C) 2017 Mixed Mode GmbH

# Class implementing common functionalities for running cross builds
# for debianized packages.
# Do not directly inherit this class.
# Instead inherit from cross.bbclass, which is done by setting the BBCLASSEXTEND = "cross" within your recipe.

inherit fetch patch apt-cache

DEPENDS += "cross-buildchroot"
do_unpack[deptask] = "do_install"
do_install[rdeptask] = "do_install"

CHROOT_DIR = "${CROSS_BUILDCHROOT_DIR}"
BUILDROOT = "${CHROOT_DIR}/${PP}"
EXTRACTDIR = "${BUILDROOT}"
S = "${EXTRACTDIR}/${SRC_DIR}"
B = "${BUILDROOT}/${BUILD_DIR}"


python () {
    s = d.getVar('S', True).rstrip('/')
    extract_dir = d.getVar('EXTRACTDIR', True).rstrip('/')

    if s == extract_dir:
        bb.fatal('\nS equals EXTRACTDIR. Maybe SRC_DIR variable was not set.')
}


OPTS = "--tool='apt-get -o Debug::pkgProblemResolver=yes --no-install-recommends -y ${APT_EXTRA_OPTS}'"
do_install_depends() {
    cd ${PPS}
    apt-get update
    mk-build-deps ${OPTS} -i -r debian/control
}
addtask do_install_depends after do_patch before do_build
do_install_depends[lockfiles] = "${DPKG_LOCK}"
do_install_depends[stamp-extra-info] = "${MACHINE}.chroot"
do_install_depends[chroot] = "1"
do_install_depends[id] = "${CROSS_BUILDCHROOT_ID}"


# Build package from sources
do_build() {
    cd ${PPS}
    dpkg-buildpackage ${DEB_SIGN} -pgpg -sn --host-arch=${DEB_ARCH} -Z${DEB_COMPRESSION}
}
do_build[stamp-extra-info] = "${MACHINE}.chroot"
do_build[chroot] = "1"
do_build[id] = "${CROSS_BUILDCHROOT_ID}"


do_install() {
    cache_add_package ${ISAR_REPO_LOCAL} ${ISAR_CACHE_LOCAL_PREFIX} ${BUILDROOT}/*.deb
}
addtask do_install after do_build
do_install[lockfiles] = "${DPKG_LOCK}"
do_install[stamp-extra-info] = "${DISTRO}"
do_install[dirs] += "${DEPLOY_DIR_IMAGE}"


do_clean_cache_pkg() {
    pkgs="$(ls ${BUILDROOT}/*.deb)"
    for pkg in $pkgs ; do
        cache_delete_package ${ISAR_REPO_LOCAL} ${ISAR_CACHE_LOCAL_PREFIX} "$(basename $pkg | sed 's/_.*.deb//')"
    done
}
addtask do_clean_cache_pkg before do_clean do_cleanall