* [PATCH 0/6] Isar apt cache implementation
@ 2017-08-27 15:13 Alexander Smirnov
2017-08-27 15:13 ` [PATCH 1/6] meta-isar-bin: Enable caching of deb packages Alexander Smirnov
` (5 more replies)
0 siblings, 6 replies; 30+ messages in thread
From: Alexander Smirnov @ 2017-08-27 15:13 UTC (permalink / raw)
To: isar-users; +Cc: Alexander Smirnov
Hello all,
this is the first series that introduces apt caching feature. The full
implementation can be found in branch: lenormf/develop-l20170602-dpkg-cross
A few words about apt caching...
The key idea is to have possibility to create custom apt repositories
that could be published and shared between projects. So the packages are
built once, repo is created and the repo is used for further builds.
Also this repo could be used in already running systems via apt-get.
Another requirement for this feature is to follow Debian way, so
reprepro tool is used to manage binary cache.
The binary repo is introduced as a separate layer what simplifies
exporting of just-built repo and importing of existing repo.
The whole implementation is split into two steps to simplify the review:
1. Basic apt implementation that allows to create binary cache. The
feature is introduced as optional (could be enabled by variable
DEBCACHE_ENABLED).
2. Support for binary cache importing, the feature becomes official and
dpkg installation way is deprecated.
Alexander Smirnov (2):
classes/dpkg: Split install for cache
doc/technical_overview: Describe binary cache
Baurzhan Ismagulov (1):
classes/image: Provide /dev/null for Stretch apt
Frank Lenormand (3):
meta-isar-bin: Enable caching of deb packages
meta-isar-bin: Enable apt repo generation for amd64
classes/dpkg: Properly update packages in the cache
doc/technical_overview.md | 59 ++++++-
meta-isar-bin/conf/layer.conf | 18 +++
meta-isar-bin/files/distributions.in | 3 +
meta-isar/conf/bblayers.conf.sample | 1 +
.../images/files/debian-configscript.sh | 8 +
.../images/files/raspbian-configscript.sh | 8 +
meta-isar/recipes-core/images/isar-image-base.bb | 8 +-
meta/classes/dpkg.bbclass | 174 ++++++++++++++++++++-
meta/classes/image.bbclass | 41 +++--
.../buildchroot/files/configscript.sh | 1 +
10 files changed, 297 insertions(+), 24 deletions(-)
create mode 100644 meta-isar-bin/conf/layer.conf
create mode 100644 meta-isar-bin/files/distributions.in
--
2.1.4
^ permalink raw reply [flat|nested] 30+ messages in thread* [PATCH 1/6] meta-isar-bin: Enable caching of deb packages 2017-08-27 15:13 [PATCH 0/6] Isar apt cache implementation Alexander Smirnov @ 2017-08-27 15:13 ` Alexander Smirnov 2017-08-28 15:18 ` Henning Schild 2017-09-06 14:21 ` Henning Schild 2017-08-27 15:13 ` [PATCH 2/6] classes/image: Provide /dev/null for Stretch apt Alexander Smirnov ` (4 subsequent siblings) 5 siblings, 2 replies; 30+ messages in thread From: Alexander Smirnov @ 2017-08-27 15:13 UTC (permalink / raw) To: isar-users; +Cc: Frank Lenormand, Alexander Smirnov From: Frank Lenormand <lenormf@gmail.com> Adds the possibility to create an apt repository for newly built packages and then use that repo for populating the target filesystem. Signed-off-by: Frank Lenormand <lenormf@gmail.com> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de> --- meta-isar-bin/conf/layer.conf | 18 +++ meta-isar-bin/files/distributions.in | 3 + meta-isar/conf/bblayers.conf.sample | 1 + .../images/files/debian-configscript.sh | 8 ++ .../images/files/raspbian-configscript.sh | 8 ++ meta-isar/recipes-core/images/isar-image-base.bb | 8 +- meta/classes/dpkg.bbclass | 126 +++++++++++++++++++-- meta/classes/image.bbclass | 30 +++-- 8 files changed, 186 insertions(+), 16 deletions(-) create mode 100644 meta-isar-bin/conf/layer.conf create mode 100644 meta-isar-bin/files/distributions.in diff --git a/meta-isar-bin/conf/layer.conf b/meta-isar-bin/conf/layer.conf new file mode 100644 index 0000000..24534e1 --- /dev/null +++ b/meta-isar-bin/conf/layer.conf @@ -0,0 +1,18 @@ +# Enable package caching with '1' +DEBCACHE_ENABLED ?= "0" + +# Codename of the repository created by the caching class +DEBDISTRONAME = "isar" + +# Path to the caching repository +DEBCACHEDIR ?= "${LAYERDIR}/apt" + +# Path to the mount point of the repository within the target rootfs, during +# population +DEBCACHEMNT ?= "/opt/cache/apt" + +# Path to the databases used by `reprepro` +DEBDBDIR ?= "${LAYERDIR}/db" + +# Path to the configuration files templates used by `reprepro` +DEBFILESDIR ?= "${LAYERDIR}/files" diff --git a/meta-isar-bin/files/distributions.in b/meta-isar-bin/files/distributions.in new file mode 100644 index 0000000..59f4429 --- /dev/null +++ b/meta-isar-bin/files/distributions.in @@ -0,0 +1,3 @@ +Codename: {DISTRO_NAME} +Architectures: i386 armhf source +Components: main diff --git a/meta-isar/conf/bblayers.conf.sample b/meta-isar/conf/bblayers.conf.sample index 80867e7..53a362b 100644 --- a/meta-isar/conf/bblayers.conf.sample +++ b/meta-isar/conf/bblayers.conf.sample @@ -8,6 +8,7 @@ BBFILES ?= "" BBLAYERS ?= " \ ##ISARROOT##/meta \ ##ISARROOT##/meta-isar \ + ##ISARROOT##/meta-isar-bin \ " BBLAYERS_NON_REMOVABLE ?= " \ ##ISARROOT##/meta \ diff --git a/meta-isar/recipes-core/images/files/debian-configscript.sh b/meta-isar/recipes-core/images/files/debian-configscript.sh index 4ac37d0..b05babb 100644 --- a/meta-isar/recipes-core/images/files/debian-configscript.sh +++ b/meta-isar/recipes-core/images/files/debian-configscript.sh @@ -8,6 +8,8 @@ set -e readonly MACHINE_SERIAL="$1" readonly BAUDRATE_TTY="$2" readonly ROOTFS_DEV="$3" +readonly DEBCACHEMNT="$4" +readonly DEBDISTRONAME="$5" cat >> /etc/default/locale << EOF LANG=en_US.UTF-8 @@ -83,3 +85,9 @@ fi if [ -x "$TARGET/sbin/init" -a -x "$TARGET/usr/sbin/policy-rc.d" ]; then rm -f $TARGET/usr/sbin/policy-rc.d fi + +mkdir -p /etc/apt/sources.list.d/ +cat <<EOF >/etc/apt/sources.list.d/${DEBDISTRONAME}.list +deb file:${DEBCACHEMNT}/ ${DEBDISTRONAME} main +deb-src file:${DEBCACHEMNT}/ ${DEBDISTRONAME} main +EOF diff --git a/meta-isar/recipes-core/images/files/raspbian-configscript.sh b/meta-isar/recipes-core/images/files/raspbian-configscript.sh index 2454481..f995f4d 100644 --- a/meta-isar/recipes-core/images/files/raspbian-configscript.sh +++ b/meta-isar/recipes-core/images/files/raspbian-configscript.sh @@ -8,6 +8,8 @@ set -e readonly MACHINE_SERIAL="$1" readonly BAUDRATE_TTY="$2" readonly ROOTFS_DEV="$3" +readonly DEBCACHEMNT="$4" +readonly DEBDISTRONAME="$5" cat >> /etc/default/locale << EOF LANG=en_US.UTF-8 @@ -89,3 +91,9 @@ KERNEL_IMAGE=`ls /boot | grep vmlinuz` cat > /boot/config.txt << EOF kernel=$KERNEL_IMAGE EOF + +mkdir -p /etc/apt/sources.list.d/ +cat <<EOF >/etc/apt/sources.list.d/${DEBDISTRONAME}.list +deb file:${DEBCACHEMNT}/ ${DEBDISTRONAME} main +deb-src file:${DEBCACHEMNT}/ ${DEBDISTRONAME} main +EOF diff --git a/meta-isar/recipes-core/images/isar-image-base.bb b/meta-isar/recipes-core/images/isar-image-base.bb index b679d97..85e6b27 100644 --- a/meta-isar/recipes-core/images/isar-image-base.bb +++ b/meta-isar/recipes-core/images/isar-image-base.bb @@ -49,8 +49,12 @@ do_rootfs() { sudo multistrap -a ${DISTRO_ARCH} -d "${S}" -f "${WORKDIR}/multistrap.conf" || true # Configure root filesystem - sudo chroot ${S} /configscript.sh ${MACHINE_SERIAL} ${BAUDRATE_TTY} \ - ${ROOTFS_DEV} + sudo chroot ${S} /configscript.sh \ + ${MACHINE_SERIAL} \ + ${BAUDRATE_TTY} \ + ${ROOTFS_DEV} \ + ${DEBCACHEMNT} \ + ${DEBDISTRONAME} sudo rm ${S}/configscript.sh } diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass index 360a95c..b1e201d 100644 --- a/meta/classes/dpkg.bbclass +++ b/meta/classes/dpkg.bbclass @@ -1,5 +1,5 @@ # This software is a part of ISAR. -# Copyright (C) 2015-2016 ilbers GmbH +# Copyright (C) 2015-2017 ilbers GmbH # Add dependency from buildchroot creation DEPENDS += "buildchroot" @@ -55,12 +55,124 @@ do_build() { sudo chroot ${BUILDCHROOT_DIR} /build.sh ${PP}/${SRC_DIR} } - # Install package to dedicated deploy directory -do_install() { - install -m 644 ${BUILDROOT}/*.deb ${DEPLOY_DIR_DEB}/ +do_binary_deb_install() { + readonly DIR_CACHE="${DEBCACHEDIR}/${DISTRO}" + readonly DIR_DB="${DEBDBDIR}/${DISTRO}" + + if [ "${DEBCACHE_ENABLED}" != "0" ]; then + # If `bitbake` is running for the first time, the cache doesn't exist + # yet and needs to be configured using a `distributions` file. + # A template stored in the layer directory is pre-processed to + # generate the configuration file, which is then placed in the + # appropriate directory. + if [ ! -e "${DIR_CACHE}/conf/distributions" ]; then + mkdir -p "${DIR_CACHE}/conf" + sed -e "s#{DISTRO_NAME}#${DEBDISTRONAME}#g" \ + "${DEBFILESDIR}/distributions.in" \ + > "${DIR_CACHE}/conf/distributions" + fi + + # Add binary and source packages to the deb cache + # If the cache doesn't exist yet, it will be created using the + # `distributions` file generated above. + ls -1 "${BUILDROOT}"/*.deb "${BUILDROOT}"/*.dsc | while read -r p; do + reprepro --waitforlock 3 -b "${DIR_CACHE}" --dbdir "${DIR_DB}" \ + -C main "include${p##*.}" "${DEBDISTRONAME}" "${p}" + done + else + # Deb caching is disabled, simply copy all binary packages to the + # deploy directory + mkdir -p "${DEPLOY_DIR_DEB}" + install -m 644 "${BUILDROOT}"/*.deb "${DEPLOY_DIR_DEB}/" + fi } -addtask install after do_build -do_install[dirs] = "${DEPLOY_DIR_DEB}" -do_install[stamp-extra-info] = "${MACHINE}" +addtask binary_deb_install after do_build +do_binary_deb_install[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}" + +# Deb caching lambda run during the parsing phase that checks whether the +# current package has to be rebuilt, or taken from the cache +python __anonymous () { + if d.getVar("DEBCACHE_ENABLED", True) == "0": + # Deb caching is disabled, do nothing + return True + + PN = d.getVar("PN", True) + PV = d.getVar("PV", True) + DISTRO_ARCH = d.getVar("DISTRO_ARCH", True) + DEBCACHEDIR = d.getVar("DEBCACHEDIR", True) + DEBDISTRONAME = d.getVar("DEBDISTRONAME", True) + DEBDBDIR = d.getVar("DEBDBDIR", True) + DISTRO = d.getVar("DISTRO", True) + path_cache = os.path.join(DEBCACHEDIR, DISTRO) + path_databases = os.path.join(DEBDBDIR, DISTRO) + path_distributions = os.path.join(path_cache, "conf", "distributions") + + # The distributions file is needed by `reprepro` to know what types + # of packages are supported, what the distribution name is, etc. + # If it doesn't exist, we have nothing in the cache, do nothing. + if not os.path.exists(path_distributions): + return + + # Anonymous functions are run several times under different contexts + # during the parsing phase, which would let the code that follows be run + # as many times for the same package. + # In order to guarantee that our subroutine only runs once per package, we + # use bitbake's "persist" API in order to have reliable persistent storage + # accross calls of the lambda (using a simple variable in the class won't + # work, as several contexts won't allow fetching its value). + pd = bb.persist_data.persist("DEBCACHE_PACKAGES", d) + if PN in pd and pd[PN] == PV: + return + + import subprocess + try: + # The databases used by `reprepro` are not stored within the cache in + # order to make versioning of only the files needed to use the cache + # as an official Debian repository simpler. + # As such, if a developer uses a peer's cache to speed up their build + # time but have never run bitbake, the database will not have been + # created, so we regenerate them here. + if not os.path.exists(path_databases) and os.path.exists(path_cache): + bb.note("Regenerating the cache databases...") + subprocess.check_call([ + "reprepro", + "--waitforlock", "3", + "-b", path_cache, + "--dbdir", path_databases, + "export", DEBDISTRONAME, + ]) + + # Get a list of the versions of all the packages named after the + # current bitbake package, and check whether the current package + # version is returned. + # As `reprepro` always returns zero with this particular operation, we + # have to use this workaround to check for a package in the cache. + package_version = subprocess.check_output([ + "reprepro", + "--waitforlock", "3", + "-b", path_cache, + "--dbdir", path_databases, + "-C", "main", + "-A", DISTRO_ARCH, + "--list-format", "${version}", + "list", DEBDISTRONAME, PN, + ]) + package_version = package_version.decode("utf-8") + if package_version == PV: + # The below list contains the names of all the tasks are in charge + # of building the package when the cache isn't enabled or if the + # package hasn't been placed in it already. + # As all tasks are enabled by default, we prevent their execution + # by setting the `noexec` flag, which will prevent a rebuild of + # the package when it's cached. + for task in ["fetch", "unpack", "build", "install"]: + d.setVarFlag("do_{}".format(task), "noexec", "1") + + # Cache the results of this command so that subsequent executions + # of this anonymous functions don't run the same code again. + pd[PN] = PV + except subprocess.CalledProcessError as e: + bb.fatal("Unable to check for a candidate for package {0} (errorcode: {1})".format(PN, e.returncode)) +} diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass index a7f0d74..f42aa48 100644 --- a/meta/classes/image.bbclass +++ b/meta/classes/image.bbclass @@ -11,18 +11,34 @@ inherit ${IMAGE_TYPE} do_populate[stamp-extra-info] = "${MACHINE}-${DISTRO}" -# Install Debian packages, that were built from sources +# Install Debian packages from the cache do_populate() { + readonly DIR_CACHE="${DEBCACHEDIR}/${DISTRO}" + if [ -n "${IMAGE_INSTALL}" ]; then - sudo mkdir -p ${S}/deb + if [ "${DEBCACHE_ENABLED}" != "0" ]; then + sudo mkdir -p "${S}/${DEBCACHEMNT}" + sudo mount -o bind "${DIR_CACHE}" "${S}/${DEBCACHEMNT}" + + sudo chroot "${S}" apt-get update -y + for package in ${IMAGE_INSTALL}; do + sudo chroot "${S}" apt-get install -t "${DEBDISTRONAME}" -y \ + --allow-unauthenticated "${package}" + done + + sudo umount "${S}/${DEBCACHEMNT}" + else + sudo mkdir -p ${S}/deb - for p in ${IMAGE_INSTALL}; do - sudo cp ${DEPLOY_DIR_DEB}/${p}_*.deb ${S}/deb - done + for p in ${IMAGE_INSTALL}; do + find "${DEPLOY_DIR_DEB}" -type f -name '*.deb' -exec \ + sudo cp '{}' "${S}/deb/" \; + done - sudo chroot ${S} /usr/bin/dpkg -i -R /deb + sudo chroot ${S} /usr/bin/dpkg -i -R /deb - sudo rm -rf ${S}/deb + sudo rm -rf ${S}/deb + fi fi } -- 2.1.4 ^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [PATCH 1/6] meta-isar-bin: Enable caching of deb packages 2017-08-27 15:13 ` [PATCH 1/6] meta-isar-bin: Enable caching of deb packages Alexander Smirnov @ 2017-08-28 15:18 ` Henning Schild 2017-08-29 6:40 ` Alexander Smirnov 2017-08-31 10:55 ` Claudius Heine 2017-09-06 14:21 ` Henning Schild 1 sibling, 2 replies; 30+ messages in thread From: Henning Schild @ 2017-08-28 15:18 UTC (permalink / raw) To: Alexander Smirnov; +Cc: isar-users, Frank Lenormand This patch is very big, doing a lot of things at a time. The commit message on the other hand is not very verbose. creating the repo is one step - i think using reprepro is a good idea moving do_populate to that repo is another one - i think this task should go away and the repo should be given to multistrap and there are probably more ways to split that into reviewable pieces So this review is not too verbose, we need smaller pieces. More inline Am Sun, 27 Aug 2017 18:13:34 +0300 schrieb Alexander Smirnov <asmirnov@ilbers.de>: > From: Frank Lenormand <lenormf@gmail.com> > > Adds the possibility to create an apt repository for newly built > packages and then use that repo for populating the target filesystem. > > Signed-off-by: Frank Lenormand <lenormf@gmail.com> > Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de> > --- > meta-isar-bin/conf/layer.conf | 18 +++ > meta-isar-bin/files/distributions.in | 3 + > meta-isar/conf/bblayers.conf.sample | 1 + > .../images/files/debian-configscript.sh | 8 ++ > .../images/files/raspbian-configscript.sh | 8 ++ > meta-isar/recipes-core/images/isar-image-base.bb | 8 +- > meta/classes/dpkg.bbclass | 126 > +++++++++++++++++++-- > meta/classes/image.bbclass | 30 +++-- 8 > files changed, 186 insertions(+), 16 deletions(-) create mode 100644 > meta-isar-bin/conf/layer.conf create mode 100644 > meta-isar-bin/files/distributions.in > > diff --git a/meta-isar-bin/conf/layer.conf > b/meta-isar-bin/conf/layer.conf new file mode 100644 > index 0000000..24534e1 > --- /dev/null > +++ b/meta-isar-bin/conf/layer.conf > @@ -0,0 +1,18 @@ > +# Enable package caching with '1' > +DEBCACHE_ENABLED ?= "0" > + > +# Codename of the repository created by the caching class > +DEBDISTRONAME = "isar" > + > +# Path to the caching repository > +DEBCACHEDIR ?= "${LAYERDIR}/apt" > + > +# Path to the mount point of the repository within the target > rootfs, during +# population > +DEBCACHEMNT ?= "/opt/cache/apt" > + > +# Path to the databases used by `reprepro` > +DEBDBDIR ?= "${LAYERDIR}/db" > + > +# Path to the configuration files templates used by `reprepro` > +DEBFILESDIR ?= "${LAYERDIR}/files" > diff --git a/meta-isar-bin/files/distributions.in > b/meta-isar-bin/files/distributions.in new file mode 100644 > index 0000000..59f4429 > --- /dev/null > +++ b/meta-isar-bin/files/distributions.in > @@ -0,0 +1,3 @@ > +Codename: {DISTRO_NAME} > +Architectures: i386 armhf source > +Components: main > diff --git a/meta-isar/conf/bblayers.conf.sample > b/meta-isar/conf/bblayers.conf.sample index 80867e7..53a362b 100644 > --- a/meta-isar/conf/bblayers.conf.sample > +++ b/meta-isar/conf/bblayers.conf.sample > @@ -8,6 +8,7 @@ BBFILES ?= "" > BBLAYERS ?= " \ > ##ISARROOT##/meta \ > ##ISARROOT##/meta-isar \ > + ##ISARROOT##/meta-isar-bin \ > " > BBLAYERS_NON_REMOVABLE ?= " \ > ##ISARROOT##/meta \ > diff --git > a/meta-isar/recipes-core/images/files/debian-configscript.sh > b/meta-isar/recipes-core/images/files/debian-configscript.sh index > 4ac37d0..b05babb 100644 --- > a/meta-isar/recipes-core/images/files/debian-configscript.sh +++ > b/meta-isar/recipes-core/images/files/debian-configscript.sh @@ -8,6 > +8,8 @@ set -e readonly MACHINE_SERIAL="$1" readonly BAUDRATE_TTY="$2" > readonly ROOTFS_DEV="$3" > +readonly DEBCACHEMNT="$4" > +readonly DEBDISTRONAME="$5" > > cat >> /etc/default/locale << EOF > LANG=en_US.UTF-8 > @@ -83,3 +85,9 @@ fi > if [ -x "$TARGET/sbin/init" -a -x "$TARGET/usr/sbin/policy-rc.d" ]; > then rm -f $TARGET/usr/sbin/policy-rc.d > fi > + > +mkdir -p /etc/apt/sources.list.d/ > +cat <<EOF >/etc/apt/sources.list.d/${DEBDISTRONAME}.list > +deb file:${DEBCACHEMNT}/ ${DEBDISTRONAME} main > +deb-src file:${DEBCACHEMNT}/ ${DEBDISTRONAME} main > +EOF > diff --git > a/meta-isar/recipes-core/images/files/raspbian-configscript.sh > b/meta-isar/recipes-core/images/files/raspbian-configscript.sh index > 2454481..f995f4d 100644 --- > a/meta-isar/recipes-core/images/files/raspbian-configscript.sh +++ > b/meta-isar/recipes-core/images/files/raspbian-configscript.sh @@ > -8,6 +8,8 @@ set -e readonly MACHINE_SERIAL="$1" readonly > BAUDRATE_TTY="$2" readonly ROOTFS_DEV="$3" > +readonly DEBCACHEMNT="$4" > +readonly DEBDISTRONAME="$5" > > cat >> /etc/default/locale << EOF > LANG=en_US.UTF-8 > @@ -89,3 +91,9 @@ KERNEL_IMAGE=`ls /boot | grep vmlinuz` > cat > /boot/config.txt << EOF > kernel=$KERNEL_IMAGE > EOF > + > +mkdir -p /etc/apt/sources.list.d/ > +cat <<EOF >/etc/apt/sources.list.d/${DEBDISTRONAME}.list > +deb file:${DEBCACHEMNT}/ ${DEBDISTRONAME} main > +deb-src file:${DEBCACHEMNT}/ ${DEBDISTRONAME} main > +EOF > diff --git a/meta-isar/recipes-core/images/isar-image-base.bb > b/meta-isar/recipes-core/images/isar-image-base.bb index > b679d97..85e6b27 100644 --- > a/meta-isar/recipes-core/images/isar-image-base.bb +++ > b/meta-isar/recipes-core/images/isar-image-base.bb @@ -49,8 +49,12 @@ > do_rootfs() { sudo multistrap -a ${DISTRO_ARCH} -d "${S}" -f > "${WORKDIR}/multistrap.conf" || true > # Configure root filesystem > - sudo chroot ${S} /configscript.sh ${MACHINE_SERIAL} > ${BAUDRATE_TTY} \ > - ${ROOTFS_DEV} > + sudo chroot ${S} /configscript.sh \ > + ${MACHINE_SERIAL} \ > + ${BAUDRATE_TTY} \ > + ${ROOTFS_DEV} \ > + ${DEBCACHEMNT} \ > + ${DEBDISTRONAME} > sudo rm ${S}/configscript.sh > } > > diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass > index 360a95c..b1e201d 100644 > --- a/meta/classes/dpkg.bbclass > +++ b/meta/classes/dpkg.bbclass > @@ -1,5 +1,5 @@ > # This software is a part of ISAR. > -# Copyright (C) 2015-2016 ilbers GmbH > +# Copyright (C) 2015-2017 ilbers GmbH > > # Add dependency from buildchroot creation > DEPENDS += "buildchroot" > @@ -55,12 +55,124 @@ do_build() { > sudo chroot ${BUILDCHROOT_DIR} /build.sh ${PP}/${SRC_DIR} > } > > - > # Install package to dedicated deploy directory > -do_install() { > - install -m 644 ${BUILDROOT}/*.deb ${DEPLOY_DIR_DEB}/ > +do_binary_deb_install() { > + readonly DIR_CACHE="${DEBCACHEDIR}/${DISTRO}" > + readonly DIR_DB="${DEBDBDIR}/${DISTRO}" > + > + if [ "${DEBCACHE_ENABLED}" != "0" ]; then > + # If `bitbake` is running for the first time, the cache > doesn't exist > + # yet and needs to be configured using a `distributions` > file. > + # A template stored in the layer directory is pre-processed > to > + # generate the configuration file, which is then placed in > the > + # appropriate directory. > + if [ ! -e "${DIR_CACHE}/conf/distributions" ]; then > + mkdir -p "${DIR_CACHE}/conf" > + sed -e "s#{DISTRO_NAME}#${DEBDISTRONAME}#g" \ > + "${DEBFILESDIR}/distributions.in" \ > + > "${DIR_CACHE}/conf/distributions" > + fi This step should be in the image and not in every package. I guess it has a race with multiple packages running this init routine. > + # Add binary and source packages to the deb cache > + # If the cache doesn't exist yet, it will be created using > the > + # `distributions` file generated above. > + ls -1 "${BUILDROOT}"/*.deb "${BUILDROOT}"/*.dsc | while read > -r p; do > + reprepro --waitforlock 3 -b "${DIR_CACHE}" --dbdir > "${DIR_DB}" \ > + -C main "include${p##*.}" "${DEBDISTRONAME}" "${p}" > + done > + else > + # Deb caching is disabled, simply copy all binary packages > to the > + # deploy directory > + mkdir -p "${DEPLOY_DIR_DEB}" > + install -m 644 "${BUILDROOT}"/*.deb "${DEPLOY_DIR_DEB}/" > + fi > } > > -addtask install after do_build > -do_install[dirs] = "${DEPLOY_DIR_DEB}" > -do_install[stamp-extra-info] = "${MACHINE}" > +addtask binary_deb_install after do_build > +do_binary_deb_install[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}" Here a task got renamed. This should be another patch ... one that i already tried to get in. > + > +# Deb caching lambda run during the parsing phase that checks > whether the +# current package has to be rebuilt, or taken from the > cache +python __anonymous () { > + if d.getVar("DEBCACHE_ENABLED", True) == "0": > + # Deb caching is disabled, do nothing > + return True > + > + PN = d.getVar("PN", True) > + PV = d.getVar("PV", True) > + DISTRO_ARCH = d.getVar("DISTRO_ARCH", True) > + DEBCACHEDIR = d.getVar("DEBCACHEDIR", True) > + DEBDISTRONAME = d.getVar("DEBDISTRONAME", True) > + DEBDBDIR = d.getVar("DEBDBDIR", True) > + DISTRO = d.getVar("DISTRO", True) > + path_cache = os.path.join(DEBCACHEDIR, DISTRO) > + path_databases = os.path.join(DEBDBDIR, DISTRO) > + path_distributions = os.path.join(path_cache, "conf", > "distributions") + > + # The distributions file is needed by `reprepro` to know what > types > + # of packages are supported, what the distribution name is, etc. > + # If it doesn't exist, we have nothing in the cache, do nothing. > + if not os.path.exists(path_distributions): > + return > + > + # Anonymous functions are run several times under different > contexts > + # during the parsing phase, which would let the code that > follows be run > + # as many times for the same package. > + # In order to guarantee that our subroutine only runs once per > package, we > + # use bitbake's "persist" API in order to have reliable > persistent storage > + # accross calls of the lambda (using a simple variable in the > class won't > + # work, as several contexts won't allow fetching its value). > + pd = bb.persist_data.persist("DEBCACHE_PACKAGES", d) > + if PN in pd and pd[PN] == PV: > + return > + > + import subprocess > + try: > + # The databases used by `reprepro` are not stored within the > cache in > + # order to make versioning of only the files needed to use > the cache > + # as an official Debian repository simpler. > + # As such, if a developer uses a peer's cache to speed up > their build > + # time but have never run bitbake, the database will not > have been > + # created, so we regenerate them here. > + if not os.path.exists(path_databases) and > os.path.exists(path_cache): > + bb.note("Regenerating the cache databases...") > + subprocess.check_call([ > + "reprepro", > + "--waitforlock", "3", > + "-b", path_cache, > + "--dbdir", path_databases, > + "export", DEBDISTRONAME, > + ]) > + > + # Get a list of the versions of all the packages named after > the > + # current bitbake package, and check whether the current > package > + # version is returned. > + # As `reprepro` always returns zero with this particular > operation, we > + # have to use this workaround to check for a package in the > cache. > + package_version = subprocess.check_output([ > + "reprepro", > + "--waitforlock", "3", > + "-b", path_cache, > + "--dbdir", path_databases, > + "-C", "main", > + "-A", DISTRO_ARCH, > + "--list-format", "${version}", > + "list", DEBDISTRONAME, PN, > + ]) > + package_version = package_version.decode("utf-8") > + if package_version == PV: > + # The below list contains the names of all the tasks are > in charge > + # of building the package when the cache isn't enabled > or if the > + # package hasn't been placed in it already. > + # As all tasks are enabled by default, we prevent their > execution > + # by setting the `noexec` flag, which will prevent a > rebuild of > + # the package when it's cached. > + for task in ["fetch", "unpack", "build", "install"]: > + d.setVarFlag("do_{}".format(task), "noexec", "1") > + > + # Cache the results of this command so that subsequent > executions > + # of this anonymous functions don't run the same code > again. > + pd[PN] = PV > + except subprocess.CalledProcessError as e: > + bb.fatal("Unable to check for a candidate for package {0} > (errorcode: {1})".format(PN, e.returncode)) +} > diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass > index a7f0d74..f42aa48 100644 > --- a/meta/classes/image.bbclass > +++ b/meta/classes/image.bbclass > @@ -11,18 +11,34 @@ inherit ${IMAGE_TYPE} > > do_populate[stamp-extra-info] = "${MACHINE}-${DISTRO}" > > -# Install Debian packages, that were built from sources > +# Install Debian packages from the cache > do_populate() { > + readonly DIR_CACHE="${DEBCACHEDIR}/${DISTRO}" > + > if [ -n "${IMAGE_INSTALL}" ]; then > - sudo mkdir -p ${S}/deb > + if [ "${DEBCACHE_ENABLED}" != "0" ]; then > + sudo mkdir -p "${S}/${DEBCACHEMNT}" > + sudo mount -o bind "${DIR_CACHE}" "${S}/${DEBCACHEMNT}" > + > + sudo chroot "${S}" apt-get update -y This will fail if you are behind a proxy. It makes a known issue worse. > + for package in ${IMAGE_INSTALL}; do > + sudo chroot "${S}" apt-get install -t > "${DEBDISTRONAME}" -y \ > + --allow-unauthenticated "${package}" > + done > + > + sudo umount "${S}/${DEBCACHEMNT}" A whole lot of sudo. I do not mind but am still wondering ... Can we replace the bind-mount with a symlink? If anything between the mount and umount fails and you need to run again ... will the mount fail if the old one is still there? > + else > + sudo mkdir -p ${S}/deb > > - for p in ${IMAGE_INSTALL}; do > - sudo cp ${DEPLOY_DIR_DEB}/${p}_*.deb ${S}/deb > - done > + for p in ${IMAGE_INSTALL}; do > + find "${DEPLOY_DIR_DEB}" -type f -name '*.deb' -exec > \ > + sudo cp '{}' "${S}/deb/" \; > + done > > - sudo chroot ${S} /usr/bin/dpkg -i -R /deb > + sudo chroot ${S} /usr/bin/dpkg -i -R /deb > > - sudo rm -rf ${S}/deb > + sudo rm -rf ${S}/deb > + fi > fi > } > ^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [PATCH 1/6] meta-isar-bin: Enable caching of deb packages 2017-08-28 15:18 ` Henning Schild @ 2017-08-29 6:40 ` Alexander Smirnov 2017-08-29 7:51 ` Henning Schild 2017-08-31 10:55 ` Claudius Heine 1 sibling, 1 reply; 30+ messages in thread From: Alexander Smirnov @ 2017-08-29 6:40 UTC (permalink / raw) To: Henning Schild; +Cc: isar-users On 08/28/2017 06:18 PM, Henning Schild wrote: > This patch is very big, doing a lot of things at a time. Ok, I'll think how it can be split. > The commit message on the other hand is not very verbose. Half of the patch is the comments that describe in details what happens in the code, so please questions and proposals. > > creating the repo is one step > - i think using reprepro is a good idea > moving do_populate to that repo is another one > - i think this task should go away and the repo should be given to > multistrap Ok > and there are probably more ways to split that into reviewable pieces Please questions and proposals. > > So this review is not too verbose, we need smaller pieces. > > More inline > > Am Sun, 27 Aug 2017 18:13:34 +0300 > schrieb Alexander Smirnov <asmirnov@ilbers.de>: > >> From: Frank Lenormand <lenormf@gmail.com> >> >> Adds the possibility to create an apt repository for newly built >> packages and then use that repo for populating the target filesystem. >> >> Signed-off-by: Frank Lenormand <lenormf@gmail.com> >> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de> >> --- >> meta-isar-bin/conf/layer.conf | 18 +++ >> meta-isar-bin/files/distributions.in | 3 + >> meta-isar/conf/bblayers.conf.sample | 1 + >> .../images/files/debian-configscript.sh | 8 ++ >> .../images/files/raspbian-configscript.sh | 8 ++ >> meta-isar/recipes-core/images/isar-image-base.bb | 8 +- >> meta/classes/dpkg.bbclass | 126 >> +++++++++++++++++++-- >> meta/classes/image.bbclass | 30 +++-- 8 >> files changed, 186 insertions(+), 16 deletions(-) create mode 100644 >> meta-isar-bin/conf/layer.conf create mode 100644 >> meta-isar-bin/files/distributions.in >> >> diff --git a/meta-isar-bin/conf/layer.conf >> b/meta-isar-bin/conf/layer.conf new file mode 100644 >> index 0000000..24534e1 >> --- /dev/null >> +++ b/meta-isar-bin/conf/layer.conf >> @@ -0,0 +1,18 @@ >> +# Enable package caching with '1' >> +DEBCACHE_ENABLED ?= "0" >> + >> +# Codename of the repository created by the caching class >> +DEBDISTRONAME = "isar" >> + >> +# Path to the caching repository >> +DEBCACHEDIR ?= "${LAYERDIR}/apt" >> + >> +# Path to the mount point of the repository within the target >> rootfs, during +# population >> +DEBCACHEMNT ?= "/opt/cache/apt" >> + >> +# Path to the databases used by `reprepro` >> +DEBDBDIR ?= "${LAYERDIR}/db" >> + >> +# Path to the configuration files templates used by `reprepro` >> +DEBFILESDIR ?= "${LAYERDIR}/files" >> diff --git a/meta-isar-bin/files/distributions.in >> b/meta-isar-bin/files/distributions.in new file mode 100644 >> index 0000000..59f4429 >> --- /dev/null >> +++ b/meta-isar-bin/files/distributions.in >> @@ -0,0 +1,3 @@ >> +Codename: {DISTRO_NAME} >> +Architectures: i386 armhf source >> +Components: main >> diff --git a/meta-isar/conf/bblayers.conf.sample >> b/meta-isar/conf/bblayers.conf.sample index 80867e7..53a362b 100644 >> --- a/meta-isar/conf/bblayers.conf.sample >> +++ b/meta-isar/conf/bblayers.conf.sample >> @@ -8,6 +8,7 @@ BBFILES ?= "" >> BBLAYERS ?= " \ >> ##ISARROOT##/meta \ >> ##ISARROOT##/meta-isar \ >> + ##ISARROOT##/meta-isar-bin \ >> " >> BBLAYERS_NON_REMOVABLE ?= " \ >> ##ISARROOT##/meta \ >> diff --git >> a/meta-isar/recipes-core/images/files/debian-configscript.sh >> b/meta-isar/recipes-core/images/files/debian-configscript.sh index >> 4ac37d0..b05babb 100644 --- >> a/meta-isar/recipes-core/images/files/debian-configscript.sh +++ >> b/meta-isar/recipes-core/images/files/debian-configscript.sh @@ -8,6 >> +8,8 @@ set -e readonly MACHINE_SERIAL="$1" readonly BAUDRATE_TTY="$2" >> readonly ROOTFS_DEV="$3" >> +readonly DEBCACHEMNT="$4" >> +readonly DEBDISTRONAME="$5" >> >> cat >> /etc/default/locale << EOF >> LANG=en_US.UTF-8 >> @@ -83,3 +85,9 @@ fi >> if [ -x "$TARGET/sbin/init" -a -x "$TARGET/usr/sbin/policy-rc.d" ]; >> then rm -f $TARGET/usr/sbin/policy-rc.d >> fi >> + >> +mkdir -p /etc/apt/sources.list.d/ >> +cat <<EOF >/etc/apt/sources.list.d/${DEBDISTRONAME}.list >> +deb file:${DEBCACHEMNT}/ ${DEBDISTRONAME} main >> +deb-src file:${DEBCACHEMNT}/ ${DEBDISTRONAME} main >> +EOF >> diff --git >> a/meta-isar/recipes-core/images/files/raspbian-configscript.sh >> b/meta-isar/recipes-core/images/files/raspbian-configscript.sh index >> 2454481..f995f4d 100644 --- >> a/meta-isar/recipes-core/images/files/raspbian-configscript.sh +++ >> b/meta-isar/recipes-core/images/files/raspbian-configscript.sh @@ >> -8,6 +8,8 @@ set -e readonly MACHINE_SERIAL="$1" readonly >> BAUDRATE_TTY="$2" readonly ROOTFS_DEV="$3" >> +readonly DEBCACHEMNT="$4" >> +readonly DEBDISTRONAME="$5" >> >> cat >> /etc/default/locale << EOF >> LANG=en_US.UTF-8 >> @@ -89,3 +91,9 @@ KERNEL_IMAGE=`ls /boot | grep vmlinuz` >> cat > /boot/config.txt << EOF >> kernel=$KERNEL_IMAGE >> EOF >> + >> +mkdir -p /etc/apt/sources.list.d/ >> +cat <<EOF >/etc/apt/sources.list.d/${DEBDISTRONAME}.list >> +deb file:${DEBCACHEMNT}/ ${DEBDISTRONAME} main >> +deb-src file:${DEBCACHEMNT}/ ${DEBDISTRONAME} main >> +EOF >> diff --git a/meta-isar/recipes-core/images/isar-image-base.bb >> b/meta-isar/recipes-core/images/isar-image-base.bb index >> b679d97..85e6b27 100644 --- >> a/meta-isar/recipes-core/images/isar-image-base.bb +++ >> b/meta-isar/recipes-core/images/isar-image-base.bb @@ -49,8 +49,12 @@ >> do_rootfs() { sudo multistrap -a ${DISTRO_ARCH} -d "${S}" -f >> "${WORKDIR}/multistrap.conf" || true >> # Configure root filesystem >> - sudo chroot ${S} /configscript.sh ${MACHINE_SERIAL} >> ${BAUDRATE_TTY} \ >> - ${ROOTFS_DEV} >> + sudo chroot ${S} /configscript.sh \ >> + ${MACHINE_SERIAL} \ >> + ${BAUDRATE_TTY} \ >> + ${ROOTFS_DEV} \ >> + ${DEBCACHEMNT} \ >> + ${DEBDISTRONAME} >> sudo rm ${S}/configscript.sh >> } >> >> diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass >> index 360a95c..b1e201d 100644 >> --- a/meta/classes/dpkg.bbclass >> +++ b/meta/classes/dpkg.bbclass >> @@ -1,5 +1,5 @@ >> # This software is a part of ISAR. >> -# Copyright (C) 2015-2016 ilbers GmbH >> +# Copyright (C) 2015-2017 ilbers GmbH >> >> # Add dependency from buildchroot creation >> DEPENDS += "buildchroot" >> @@ -55,12 +55,124 @@ do_build() { >> sudo chroot ${BUILDCHROOT_DIR} /build.sh ${PP}/${SRC_DIR} >> } >> >> - >> # Install package to dedicated deploy directory >> -do_install() { >> - install -m 644 ${BUILDROOT}/*.deb ${DEPLOY_DIR_DEB}/ >> +do_binary_deb_install() { >> + readonly DIR_CACHE="${DEBCACHEDIR}/${DISTRO}" >> + readonly DIR_DB="${DEBDBDIR}/${DISTRO}" >> + >> + if [ "${DEBCACHE_ENABLED}" != "0" ]; then >> + # If `bitbake` is running for the first time, the cache >> doesn't exist >> + # yet and needs to be configured using a `distributions` >> file. >> + # A template stored in the layer directory is pre-processed >> to >> + # generate the configuration file, which is then placed in >> the >> + # appropriate directory. >> + if [ ! -e "${DIR_CACHE}/conf/distributions" ]; then >> + mkdir -p "${DIR_CACHE}/conf" >> + sed -e "s#{DISTRO_NAME}#${DEBDISTRONAME}#g" \ >> + "${DEBFILESDIR}/distributions.in" \ >> + > "${DIR_CACHE}/conf/distributions" >> + fi > > This step should be in the image and not in every package. I guess it > has a race with multiple packages running this init routine. Ok. > >> + # Add binary and source packages to the deb cache >> + # If the cache doesn't exist yet, it will be created using >> the >> + # `distributions` file generated above. >> + ls -1 "${BUILDROOT}"/*.deb "${BUILDROOT}"/*.dsc | while read >> -r p; do >> + reprepro --waitforlock 3 -b "${DIR_CACHE}" --dbdir >> "${DIR_DB}" \ >> + -C main "include${p##*.}" "${DEBDISTRONAME}" "${p}" >> + done >> + else >> + # Deb caching is disabled, simply copy all binary packages >> to the >> + # deploy directory >> + mkdir -p "${DEPLOY_DIR_DEB}" >> + install -m 644 "${BUILDROOT}"/*.deb "${DEPLOY_DIR_DEB}/" >> + fi >> } >> >> -addtask install after do_build >> -do_install[dirs] = "${DEPLOY_DIR_DEB}" >> -do_install[stamp-extra-info] = "${MACHINE}" >> +addtask binary_deb_install after do_build >> +do_binary_deb_install[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}" > > Here a task got renamed. This should be another patch ... one that i > already tried to get in. > This implemented in the next patch (to keep this patch in original format), but yes, it should be done here. >> + >> +# Deb caching lambda run during the parsing phase that checks >> whether the +# current package has to be rebuilt, or taken from the >> cache +python __anonymous () { >> + if d.getVar("DEBCACHE_ENABLED", True) == "0": >> + # Deb caching is disabled, do nothing >> + return True >> + >> + PN = d.getVar("PN", True) >> + PV = d.getVar("PV", True) >> + DISTRO_ARCH = d.getVar("DISTRO_ARCH", True) >> + DEBCACHEDIR = d.getVar("DEBCACHEDIR", True) >> + DEBDISTRONAME = d.getVar("DEBDISTRONAME", True) >> + DEBDBDIR = d.getVar("DEBDBDIR", True) >> + DISTRO = d.getVar("DISTRO", True) >> + path_cache = os.path.join(DEBCACHEDIR, DISTRO) >> + path_databases = os.path.join(DEBDBDIR, DISTRO) >> + path_distributions = os.path.join(path_cache, "conf", >> "distributions") + >> + # The distributions file is needed by `reprepro` to know what >> types >> + # of packages are supported, what the distribution name is, etc. >> + # If it doesn't exist, we have nothing in the cache, do nothing. >> + if not os.path.exists(path_distributions): >> + return >> + >> + # Anonymous functions are run several times under different >> contexts >> + # during the parsing phase, which would let the code that >> follows be run >> + # as many times for the same package. >> + # In order to guarantee that our subroutine only runs once per >> package, we >> + # use bitbake's "persist" API in order to have reliable >> persistent storage >> + # accross calls of the lambda (using a simple variable in the >> class won't >> + # work, as several contexts won't allow fetching its value). >> + pd = bb.persist_data.persist("DEBCACHE_PACKAGES", d) >> + if PN in pd and pd[PN] == PV: >> + return >> + >> + import subprocess >> + try: >> + # The databases used by `reprepro` are not stored within the >> cache in >> + # order to make versioning of only the files needed to use >> the cache >> + # as an official Debian repository simpler. >> + # As such, if a developer uses a peer's cache to speed up >> their build >> + # time but have never run bitbake, the database will not >> have been >> + # created, so we regenerate them here. >> + if not os.path.exists(path_databases) and >> os.path.exists(path_cache): >> + bb.note("Regenerating the cache databases...") >> + subprocess.check_call([ >> + "reprepro", >> + "--waitforlock", "3", >> + "-b", path_cache, >> + "--dbdir", path_databases, >> + "export", DEBDISTRONAME, >> + ]) >> + >> + # Get a list of the versions of all the packages named after >> the >> + # current bitbake package, and check whether the current >> package >> + # version is returned. >> + # As `reprepro` always returns zero with this particular >> operation, we >> + # have to use this workaround to check for a package in the >> cache. >> + package_version = subprocess.check_output([ >> + "reprepro", >> + "--waitforlock", "3", >> + "-b", path_cache, >> + "--dbdir", path_databases, >> + "-C", "main", >> + "-A", DISTRO_ARCH, >> + "--list-format", "${version}", >> + "list", DEBDISTRONAME, PN, >> + ]) >> + package_version = package_version.decode("utf-8") >> + if package_version == PV: >> + # The below list contains the names of all the tasks are >> in charge >> + # of building the package when the cache isn't enabled >> or if the >> + # package hasn't been placed in it already. >> + # As all tasks are enabled by default, we prevent their >> execution >> + # by setting the `noexec` flag, which will prevent a >> rebuild of >> + # the package when it's cached. >> + for task in ["fetch", "unpack", "build", "install"]: >> + d.setVarFlag("do_{}".format(task), "noexec", "1") >> + >> + # Cache the results of this command so that subsequent >> executions >> + # of this anonymous functions don't run the same code >> again. >> + pd[PN] = PV >> + except subprocess.CalledProcessError as e: >> + bb.fatal("Unable to check for a candidate for package {0} >> (errorcode: {1})".format(PN, e.returncode)) +} >> diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass >> index a7f0d74..f42aa48 100644 >> --- a/meta/classes/image.bbclass >> +++ b/meta/classes/image.bbclass >> @@ -11,18 +11,34 @@ inherit ${IMAGE_TYPE} >> >> do_populate[stamp-extra-info] = "${MACHINE}-${DISTRO}" >> >> -# Install Debian packages, that were built from sources >> +# Install Debian packages from the cache >> do_populate() { >> + readonly DIR_CACHE="${DEBCACHEDIR}/${DISTRO}" >> + >> if [ -n "${IMAGE_INSTALL}" ]; then >> - sudo mkdir -p ${S}/deb >> + if [ "${DEBCACHE_ENABLED}" != "0" ]; then >> + sudo mkdir -p "${S}/${DEBCACHEMNT}" >> + sudo mount -o bind "${DIR_CACHE}" "${S}/${DEBCACHEMNT}" >> + >> + sudo chroot "${S}" apt-get update -y > > This will fail if you are behind a proxy. It makes a known issue worse. > >> + for package in ${IMAGE_INSTALL}; do >> + sudo chroot "${S}" apt-get install -t >> "${DEBDISTRONAME}" -y \ >> + --allow-unauthenticated "${package}" >> + done >> + >> + sudo umount "${S}/${DEBCACHEMNT}" > > A whole lot of sudo. I do not mind but am still wondering ... Please questions and proposals instead of negative tone. > > Can we replace the bind-mount with a symlink? If anything between the > mount and umount fails and you need to run again ... will the mount > fail if the old one is still there? If we switch to multistrap-based populating, I think this will go. > >> + else >> + sudo mkdir -p ${S}/deb >> >> - for p in ${IMAGE_INSTALL}; do >> - sudo cp ${DEPLOY_DIR_DEB}/${p}_*.deb ${S}/deb >> - done >> + for p in ${IMAGE_INSTALL}; do >> + find "${DEPLOY_DIR_DEB}" -type f -name '*.deb' -exec >> \ >> + sudo cp '{}' "${S}/deb/" \; >> + done >> >> - sudo chroot ${S} /usr/bin/dpkg -i -R /deb >> + sudo chroot ${S} /usr/bin/dpkg -i -R /deb >> >> - sudo rm -rf ${S}/deb >> + sudo rm -rf ${S}/deb >> + fi >> fi >> } >> > ^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [PATCH 1/6] meta-isar-bin: Enable caching of deb packages 2017-08-29 6:40 ` Alexander Smirnov @ 2017-08-29 7:51 ` Henning Schild 2017-08-29 8:20 ` Alexander Smirnov 0 siblings, 1 reply; 30+ messages in thread From: Henning Schild @ 2017-08-29 7:51 UTC (permalink / raw) To: Alexander Smirnov; +Cc: isar-users Am Tue, 29 Aug 2017 09:40:46 +0300 schrieb Alexander Smirnov <asmirnov@ilbers.de>: > On 08/28/2017 06:18 PM, Henning Schild wrote: > > This patch is very big, doing a lot of things at a time. > > Ok, I'll think how it can be split. > > > The commit message on the other hand is not very verbose. > > Half of the patch is the comments that describe in details what > happens in the code, so please questions and proposals. I will comment on that once the patch got split. In general readable good does not require comments. Comments add redundancy and if comments are required it is usually a sign that the code is too complicated. > > > > creating the repo is one step > > - i think using reprepro is a good idea > > moving do_populate to that repo is another one > > - i think this task should go away and the repo should be given to > > multistrap > > Ok > > > and there are probably more ways to split that into reviewable > > pieces > > Please questions and proposals. Yes, for v2. > > > > So this review is not too verbose, we need smaller pieces. > > > > More inline > > > > Am Sun, 27 Aug 2017 18:13:34 +0300 > > schrieb Alexander Smirnov <asmirnov@ilbers.de>: > > > >> From: Frank Lenormand <lenormf@gmail.com> > >> > >> Adds the possibility to create an apt repository for newly built > >> packages and then use that repo for populating the target > >> filesystem. > >> > >> Signed-off-by: Frank Lenormand <lenormf@gmail.com> > >> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de> > >> --- > >> meta-isar-bin/conf/layer.conf | 18 +++ > >> meta-isar-bin/files/distributions.in | 3 + > >> meta-isar/conf/bblayers.conf.sample | 1 + > >> .../images/files/debian-configscript.sh | 8 ++ > >> .../images/files/raspbian-configscript.sh | 8 ++ > >> meta-isar/recipes-core/images/isar-image-base.bb | 8 +- > >> meta/classes/dpkg.bbclass | 126 > >> +++++++++++++++++++-- > >> meta/classes/image.bbclass | 30 +++-- 8 > >> files changed, 186 insertions(+), 16 deletions(-) create mode > >> 100644 meta-isar-bin/conf/layer.conf create mode 100644 > >> meta-isar-bin/files/distributions.in > >> > >> diff --git a/meta-isar-bin/conf/layer.conf > >> b/meta-isar-bin/conf/layer.conf new file mode 100644 > >> index 0000000..24534e1 > >> --- /dev/null > >> +++ b/meta-isar-bin/conf/layer.conf > >> @@ -0,0 +1,18 @@ > >> +# Enable package caching with '1' > >> +DEBCACHE_ENABLED ?= "0" > >> + > >> +# Codename of the repository created by the caching class > >> +DEBDISTRONAME = "isar" > >> + > >> +# Path to the caching repository > >> +DEBCACHEDIR ?= "${LAYERDIR}/apt" > >> + > >> +# Path to the mount point of the repository within the target > >> rootfs, during +# population > >> +DEBCACHEMNT ?= "/opt/cache/apt" > >> + > >> +# Path to the databases used by `reprepro` > >> +DEBDBDIR ?= "${LAYERDIR}/db" > >> + > >> +# Path to the configuration files templates used by `reprepro` > >> +DEBFILESDIR ?= "${LAYERDIR}/files" > >> diff --git a/meta-isar-bin/files/distributions.in > >> b/meta-isar-bin/files/distributions.in new file mode 100644 > >> index 0000000..59f4429 > >> --- /dev/null > >> +++ b/meta-isar-bin/files/distributions.in > >> @@ -0,0 +1,3 @@ > >> +Codename: {DISTRO_NAME} > >> +Architectures: i386 armhf source > >> +Components: main > >> diff --git a/meta-isar/conf/bblayers.conf.sample > >> b/meta-isar/conf/bblayers.conf.sample index 80867e7..53a362b 100644 > >> --- a/meta-isar/conf/bblayers.conf.sample > >> +++ b/meta-isar/conf/bblayers.conf.sample > >> @@ -8,6 +8,7 @@ BBFILES ?= "" > >> BBLAYERS ?= " \ > >> ##ISARROOT##/meta \ > >> ##ISARROOT##/meta-isar \ > >> + ##ISARROOT##/meta-isar-bin \ > >> " > >> BBLAYERS_NON_REMOVABLE ?= " \ > >> ##ISARROOT##/meta \ > >> diff --git > >> a/meta-isar/recipes-core/images/files/debian-configscript.sh > >> b/meta-isar/recipes-core/images/files/debian-configscript.sh index > >> 4ac37d0..b05babb 100644 --- > >> a/meta-isar/recipes-core/images/files/debian-configscript.sh +++ > >> b/meta-isar/recipes-core/images/files/debian-configscript.sh @@ > >> -8,6 +8,8 @@ set -e readonly MACHINE_SERIAL="$1" readonly > >> BAUDRATE_TTY="$2" readonly ROOTFS_DEV="$3" > >> +readonly DEBCACHEMNT="$4" > >> +readonly DEBDISTRONAME="$5" > >> > >> cat >> /etc/default/locale << EOF > >> LANG=en_US.UTF-8 > >> @@ -83,3 +85,9 @@ fi > >> if [ -x "$TARGET/sbin/init" -a -x "$TARGET/usr/sbin/policy-rc.d" > >> ]; then rm -f $TARGET/usr/sbin/policy-rc.d > >> fi > >> + > >> +mkdir -p /etc/apt/sources.list.d/ > >> +cat <<EOF >/etc/apt/sources.list.d/${DEBDISTRONAME}.list > >> +deb file:${DEBCACHEMNT}/ ${DEBDISTRONAME} main > >> +deb-src file:${DEBCACHEMNT}/ ${DEBDISTRONAME} main > >> +EOF > >> diff --git > >> a/meta-isar/recipes-core/images/files/raspbian-configscript.sh > >> b/meta-isar/recipes-core/images/files/raspbian-configscript.sh > >> index 2454481..f995f4d 100644 --- > >> a/meta-isar/recipes-core/images/files/raspbian-configscript.sh +++ > >> b/meta-isar/recipes-core/images/files/raspbian-configscript.sh @@ > >> -8,6 +8,8 @@ set -e readonly MACHINE_SERIAL="$1" readonly > >> BAUDRATE_TTY="$2" readonly ROOTFS_DEV="$3" > >> +readonly DEBCACHEMNT="$4" > >> +readonly DEBDISTRONAME="$5" > >> > >> cat >> /etc/default/locale << EOF > >> LANG=en_US.UTF-8 > >> @@ -89,3 +91,9 @@ KERNEL_IMAGE=`ls /boot | grep vmlinuz` > >> cat > /boot/config.txt << EOF > >> kernel=$KERNEL_IMAGE > >> EOF > >> + > >> +mkdir -p /etc/apt/sources.list.d/ > >> +cat <<EOF >/etc/apt/sources.list.d/${DEBDISTRONAME}.list > >> +deb file:${DEBCACHEMNT}/ ${DEBDISTRONAME} main > >> +deb-src file:${DEBCACHEMNT}/ ${DEBDISTRONAME} main > >> +EOF > >> diff --git a/meta-isar/recipes-core/images/isar-image-base.bb > >> b/meta-isar/recipes-core/images/isar-image-base.bb index > >> b679d97..85e6b27 100644 --- > >> a/meta-isar/recipes-core/images/isar-image-base.bb +++ > >> b/meta-isar/recipes-core/images/isar-image-base.bb @@ -49,8 +49,12 > >> @@ do_rootfs() { sudo multistrap -a ${DISTRO_ARCH} -d "${S}" -f > >> "${WORKDIR}/multistrap.conf" || true > >> # Configure root filesystem > >> - sudo chroot ${S} /configscript.sh ${MACHINE_SERIAL} > >> ${BAUDRATE_TTY} \ > >> - ${ROOTFS_DEV} > >> + sudo chroot ${S} /configscript.sh \ > >> + ${MACHINE_SERIAL} \ > >> + ${BAUDRATE_TTY} \ > >> + ${ROOTFS_DEV} \ > >> + ${DEBCACHEMNT} \ > >> + ${DEBDISTRONAME} > >> sudo rm ${S}/configscript.sh > >> } > >> > >> diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass > >> index 360a95c..b1e201d 100644 > >> --- a/meta/classes/dpkg.bbclass > >> +++ b/meta/classes/dpkg.bbclass > >> @@ -1,5 +1,5 @@ > >> # This software is a part of ISAR. > >> -# Copyright (C) 2015-2016 ilbers GmbH > >> +# Copyright (C) 2015-2017 ilbers GmbH > >> > >> # Add dependency from buildchroot creation > >> DEPENDS += "buildchroot" > >> @@ -55,12 +55,124 @@ do_build() { > >> sudo chroot ${BUILDCHROOT_DIR} /build.sh ${PP}/${SRC_DIR} > >> } > >> > >> - > >> # Install package to dedicated deploy directory > >> -do_install() { > >> - install -m 644 ${BUILDROOT}/*.deb ${DEPLOY_DIR_DEB}/ > >> +do_binary_deb_install() { > >> + readonly DIR_CACHE="${DEBCACHEDIR}/${DISTRO}" > >> + readonly DIR_DB="${DEBDBDIR}/${DISTRO}" > >> + > >> + if [ "${DEBCACHE_ENABLED}" != "0" ]; then > >> + # If `bitbake` is running for the first time, the cache > >> doesn't exist > >> + # yet and needs to be configured using a `distributions` > >> file. > >> + # A template stored in the layer directory is > >> pre-processed to > >> + # generate the configuration file, which is then placed in > >> the > >> + # appropriate directory. > >> + if [ ! -e "${DIR_CACHE}/conf/distributions" ]; then > >> + mkdir -p "${DIR_CACHE}/conf" > >> + sed -e "s#{DISTRO_NAME}#${DEBDISTRONAME}#g" \ > >> + "${DEBFILESDIR}/distributions.in" \ > >> + > "${DIR_CACHE}/conf/distributions" > >> + fi > > > > This step should be in the image and not in every package. I guess > > it has a race with multiple packages running this init routine. > > Ok. > > > > >> + # Add binary and source packages to the deb cache > >> + # If the cache doesn't exist yet, it will be created using > >> the > >> + # `distributions` file generated above. > >> + ls -1 "${BUILDROOT}"/*.deb "${BUILDROOT}"/*.dsc | while > >> read -r p; do > >> + reprepro --waitforlock 3 -b "${DIR_CACHE}" --dbdir > >> "${DIR_DB}" \ > >> + -C main "include${p##*.}" "${DEBDISTRONAME}" > >> "${p}" > >> + done > >> + else > >> + # Deb caching is disabled, simply copy all binary packages > >> to the > >> + # deploy directory > >> + mkdir -p "${DEPLOY_DIR_DEB}" > >> + install -m 644 "${BUILDROOT}"/*.deb "${DEPLOY_DIR_DEB}/" > >> + fi > >> } > >> > >> -addtask install after do_build > >> -do_install[dirs] = "${DEPLOY_DIR_DEB}" > >> -do_install[stamp-extra-info] = "${MACHINE}" > >> +addtask binary_deb_install after do_build > >> +do_binary_deb_install[stamp-extra-info] = > >> "${DISTRO}-${DISTRO_ARCH}" > > > > Here a task got renamed. This should be another patch ... one that i > > already tried to get in. > > > > This implemented in the next patch (to keep this patch in original > format), but yes, it should be done here. > > >> + > >> +# Deb caching lambda run during the parsing phase that checks > >> whether the +# current package has to be rebuilt, or taken from the > >> cache +python __anonymous () { > >> + if d.getVar("DEBCACHE_ENABLED", True) == "0": > >> + # Deb caching is disabled, do nothing > >> + return True > >> + > >> + PN = d.getVar("PN", True) > >> + PV = d.getVar("PV", True) > >> + DISTRO_ARCH = d.getVar("DISTRO_ARCH", True) > >> + DEBCACHEDIR = d.getVar("DEBCACHEDIR", True) > >> + DEBDISTRONAME = d.getVar("DEBDISTRONAME", True) > >> + DEBDBDIR = d.getVar("DEBDBDIR", True) > >> + DISTRO = d.getVar("DISTRO", True) > >> + path_cache = os.path.join(DEBCACHEDIR, DISTRO) > >> + path_databases = os.path.join(DEBDBDIR, DISTRO) > >> + path_distributions = os.path.join(path_cache, "conf", > >> "distributions") + > >> + # The distributions file is needed by `reprepro` to know what > >> types > >> + # of packages are supported, what the distribution name is, > >> etc. > >> + # If it doesn't exist, we have nothing in the cache, do > >> nothing. > >> + if not os.path.exists(path_distributions): > >> + return > >> + > >> + # Anonymous functions are run several times under different > >> contexts > >> + # during the parsing phase, which would let the code that > >> follows be run > >> + # as many times for the same package. > >> + # In order to guarantee that our subroutine only runs once per > >> package, we > >> + # use bitbake's "persist" API in order to have reliable > >> persistent storage > >> + # accross calls of the lambda (using a simple variable in the > >> class won't > >> + # work, as several contexts won't allow fetching its value). > >> + pd = bb.persist_data.persist("DEBCACHE_PACKAGES", d) > >> + if PN in pd and pd[PN] == PV: > >> + return > >> + > >> + import subprocess > >> + try: > >> + # The databases used by `reprepro` are not stored within > >> the cache in > >> + # order to make versioning of only the files needed to use > >> the cache > >> + # as an official Debian repository simpler. > >> + # As such, if a developer uses a peer's cache to speed up > >> their build > >> + # time but have never run bitbake, the database will not > >> have been > >> + # created, so we regenerate them here. > >> + if not os.path.exists(path_databases) and > >> os.path.exists(path_cache): > >> + bb.note("Regenerating the cache databases...") > >> + subprocess.check_call([ > >> + "reprepro", > >> + "--waitforlock", "3", > >> + "-b", path_cache, > >> + "--dbdir", path_databases, > >> + "export", DEBDISTRONAME, > >> + ]) > >> + > >> + # Get a list of the versions of all the packages named > >> after the > >> + # current bitbake package, and check whether the current > >> package > >> + # version is returned. > >> + # As `reprepro` always returns zero with this particular > >> operation, we > >> + # have to use this workaround to check for a package in > >> the cache. > >> + package_version = subprocess.check_output([ > >> + "reprepro", > >> + "--waitforlock", "3", > >> + "-b", path_cache, > >> + "--dbdir", path_databases, > >> + "-C", "main", > >> + "-A", DISTRO_ARCH, > >> + "--list-format", "${version}", > >> + "list", DEBDISTRONAME, PN, > >> + ]) > >> + package_version = package_version.decode("utf-8") > >> + if package_version == PV: > >> + # The below list contains the names of all the tasks > >> are in charge > >> + # of building the package when the cache isn't enabled > >> or if the > >> + # package hasn't been placed in it already. > >> + # As all tasks are enabled by default, we prevent > >> their execution > >> + # by setting the `noexec` flag, which will prevent a > >> rebuild of > >> + # the package when it's cached. > >> + for task in ["fetch", "unpack", "build", "install"]: > >> + d.setVarFlag("do_{}".format(task), "noexec", "1") > >> + > >> + # Cache the results of this command so that subsequent > >> executions > >> + # of this anonymous functions don't run the same code > >> again. > >> + pd[PN] = PV > >> + except subprocess.CalledProcessError as e: > >> + bb.fatal("Unable to check for a candidate for package {0} > >> (errorcode: {1})".format(PN, e.returncode)) +} > >> diff --git a/meta/classes/image.bbclass > >> b/meta/classes/image.bbclass index a7f0d74..f42aa48 100644 > >> --- a/meta/classes/image.bbclass > >> +++ b/meta/classes/image.bbclass > >> @@ -11,18 +11,34 @@ inherit ${IMAGE_TYPE} > >> > >> do_populate[stamp-extra-info] = "${MACHINE}-${DISTRO}" > >> > >> -# Install Debian packages, that were built from sources > >> +# Install Debian packages from the cache > >> do_populate() { > >> + readonly DIR_CACHE="${DEBCACHEDIR}/${DISTRO}" > >> + > >> if [ -n "${IMAGE_INSTALL}" ]; then > >> - sudo mkdir -p ${S}/deb > >> + if [ "${DEBCACHE_ENABLED}" != "0" ]; then > >> + sudo mkdir -p "${S}/${DEBCACHEMNT}" > >> + sudo mount -o bind "${DIR_CACHE}" > >> "${S}/${DEBCACHEMNT}" + > >> + sudo chroot "${S}" apt-get update -y > > > > This will fail if you are behind a proxy. It makes a known issue > > worse. > >> + for package in ${IMAGE_INSTALL}; do > >> + sudo chroot "${S}" apt-get install -t > >> "${DEBDISTRONAME}" -y \ > >> + --allow-unauthenticated "${package}" > >> + done > >> + > >> + sudo umount "${S}/${DEBCACHEMNT}" > > > > A whole lot of sudo. I do not mind but am still wondering ... > > Please questions and proposals instead of negative tone. That was maybe slightly negative. Why? Because that patch was sent at a time where the policy was still "do not add more sudos". And the patch violates more of the groundrules we have agreed on. You posting the patch signals that our groundrules seem to apply to !ilbers-contributions only. > > Can we replace the bind-mount with a symlink? If anything between > > the mount and umount fails and you need to run again ... will the > > mount fail if the old one is still there? > > If we switch to multistrap-based populating, I think this will go. I know, but we have not decided that yet. Henning > > > >> + else > >> + sudo mkdir -p ${S}/deb > >> > >> - for p in ${IMAGE_INSTALL}; do > >> - sudo cp ${DEPLOY_DIR_DEB}/${p}_*.deb ${S}/deb > >> - done > >> + for p in ${IMAGE_INSTALL}; do > >> + find "${DEPLOY_DIR_DEB}" -type f -name '*.deb' > >> -exec \ > >> + sudo cp '{}' "${S}/deb/" \; > >> + done > >> > >> - sudo chroot ${S} /usr/bin/dpkg -i -R /deb > >> + sudo chroot ${S} /usr/bin/dpkg -i -R /deb > >> > >> - sudo rm -rf ${S}/deb > >> + sudo rm -rf ${S}/deb > >> + fi > >> fi > >> } > >> > > ^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [PATCH 1/6] meta-isar-bin: Enable caching of deb packages 2017-08-29 7:51 ` Henning Schild @ 2017-08-29 8:20 ` Alexander Smirnov 0 siblings, 0 replies; 30+ messages in thread From: Alexander Smirnov @ 2017-08-29 8:20 UTC (permalink / raw) To: Henning Schild; +Cc: isar-users On 08/29/2017 10:51 AM, Henning Schild wrote: > Am Tue, 29 Aug 2017 09:40:46 +0300 > schrieb Alexander Smirnov <asmirnov@ilbers.de>: > >> On 08/28/2017 06:18 PM, Henning Schild wrote: >>> This patch is very big, doing a lot of things at a time. >> >> Ok, I'll think how it can be split. >> >>> The commit message on the other hand is not very verbose. >> >> Half of the patch is the comments that describe in details what >> happens in the code, so please questions and proposals. > > I will comment on that once the patch got split. In general readable > good does not require comments. Comments add redundancy and if comments > are required it is usually a sign that the code is too complicated. I think it's a matter of taste, but Ok, I got your position. > >>> >>> creating the repo is one step >>> - i think using reprepro is a good idea >>> moving do_populate to that repo is another one >>> - i think this task should go away and the repo should be given to >>> multistrap >> >> Ok >> >>> and there are probably more ways to split that into reviewable >>> pieces >> >> Please questions and proposals. > > Yes, for v2. Ok. > >>> >>> So this review is not too verbose, we need smaller pieces. >>> >>> More inline >>> >>> Am Sun, 27 Aug 2017 18:13:34 +0300 >>> schrieb Alexander Smirnov <asmirnov@ilbers.de>: >>> >>>> From: Frank Lenormand <lenormf@gmail.com> >>>> >>>> Adds the possibility to create an apt repository for newly built >>>> packages and then use that repo for populating the target >>>> filesystem. >>>> >>>> Signed-off-by: Frank Lenormand <lenormf@gmail.com> >>>> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de> >>>> --- >>>> meta-isar-bin/conf/layer.conf | 18 +++ >>>> meta-isar-bin/files/distributions.in | 3 + >>>> meta-isar/conf/bblayers.conf.sample | 1 + >>>> .../images/files/debian-configscript.sh | 8 ++ >>>> .../images/files/raspbian-configscript.sh | 8 ++ >>>> meta-isar/recipes-core/images/isar-image-base.bb | 8 +- >>>> meta/classes/dpkg.bbclass | 126 >>>> +++++++++++++++++++-- >>>> meta/classes/image.bbclass | 30 +++-- 8 >>>> files changed, 186 insertions(+), 16 deletions(-) create mode >>>> 100644 meta-isar-bin/conf/layer.conf create mode 100644 >>>> meta-isar-bin/files/distributions.in >>>> >>>> diff --git a/meta-isar-bin/conf/layer.conf >>>> b/meta-isar-bin/conf/layer.conf new file mode 100644 >>>> index 0000000..24534e1 >>>> --- /dev/null >>>> +++ b/meta-isar-bin/conf/layer.conf >>>> @@ -0,0 +1,18 @@ >>>> +# Enable package caching with '1' >>>> +DEBCACHE_ENABLED ?= "0" >>>> + >>>> +# Codename of the repository created by the caching class >>>> +DEBDISTRONAME = "isar" >>>> + >>>> +# Path to the caching repository >>>> +DEBCACHEDIR ?= "${LAYERDIR}/apt" >>>> + >>>> +# Path to the mount point of the repository within the target >>>> rootfs, during +# population >>>> +DEBCACHEMNT ?= "/opt/cache/apt" >>>> + >>>> +# Path to the databases used by `reprepro` >>>> +DEBDBDIR ?= "${LAYERDIR}/db" >>>> + >>>> +# Path to the configuration files templates used by `reprepro` >>>> +DEBFILESDIR ?= "${LAYERDIR}/files" >>>> diff --git a/meta-isar-bin/files/distributions.in >>>> b/meta-isar-bin/files/distributions.in new file mode 100644 >>>> index 0000000..59f4429 >>>> --- /dev/null >>>> +++ b/meta-isar-bin/files/distributions.in >>>> @@ -0,0 +1,3 @@ >>>> +Codename: {DISTRO_NAME} >>>> +Architectures: i386 armhf source >>>> +Components: main >>>> diff --git a/meta-isar/conf/bblayers.conf.sample >>>> b/meta-isar/conf/bblayers.conf.sample index 80867e7..53a362b 100644 >>>> --- a/meta-isar/conf/bblayers.conf.sample >>>> +++ b/meta-isar/conf/bblayers.conf.sample >>>> @@ -8,6 +8,7 @@ BBFILES ?= "" >>>> BBLAYERS ?= " \ >>>> ##ISARROOT##/meta \ >>>> ##ISARROOT##/meta-isar \ >>>> + ##ISARROOT##/meta-isar-bin \ >>>> " >>>> BBLAYERS_NON_REMOVABLE ?= " \ >>>> ##ISARROOT##/meta \ >>>> diff --git >>>> a/meta-isar/recipes-core/images/files/debian-configscript.sh >>>> b/meta-isar/recipes-core/images/files/debian-configscript.sh index >>>> 4ac37d0..b05babb 100644 --- >>>> a/meta-isar/recipes-core/images/files/debian-configscript.sh +++ >>>> b/meta-isar/recipes-core/images/files/debian-configscript.sh @@ >>>> -8,6 +8,8 @@ set -e readonly MACHINE_SERIAL="$1" readonly >>>> BAUDRATE_TTY="$2" readonly ROOTFS_DEV="$3" >>>> +readonly DEBCACHEMNT="$4" >>>> +readonly DEBDISTRONAME="$5" >>>> >>>> cat >> /etc/default/locale << EOF >>>> LANG=en_US.UTF-8 >>>> @@ -83,3 +85,9 @@ fi >>>> if [ -x "$TARGET/sbin/init" -a -x "$TARGET/usr/sbin/policy-rc.d" >>>> ]; then rm -f $TARGET/usr/sbin/policy-rc.d >>>> fi >>>> + >>>> +mkdir -p /etc/apt/sources.list.d/ >>>> +cat <<EOF >/etc/apt/sources.list.d/${DEBDISTRONAME}.list >>>> +deb file:${DEBCACHEMNT}/ ${DEBDISTRONAME} main >>>> +deb-src file:${DEBCACHEMNT}/ ${DEBDISTRONAME} main >>>> +EOF >>>> diff --git >>>> a/meta-isar/recipes-core/images/files/raspbian-configscript.sh >>>> b/meta-isar/recipes-core/images/files/raspbian-configscript.sh >>>> index 2454481..f995f4d 100644 --- >>>> a/meta-isar/recipes-core/images/files/raspbian-configscript.sh +++ >>>> b/meta-isar/recipes-core/images/files/raspbian-configscript.sh @@ >>>> -8,6 +8,8 @@ set -e readonly MACHINE_SERIAL="$1" readonly >>>> BAUDRATE_TTY="$2" readonly ROOTFS_DEV="$3" >>>> +readonly DEBCACHEMNT="$4" >>>> +readonly DEBDISTRONAME="$5" >>>> >>>> cat >> /etc/default/locale << EOF >>>> LANG=en_US.UTF-8 >>>> @@ -89,3 +91,9 @@ KERNEL_IMAGE=`ls /boot | grep vmlinuz` >>>> cat > /boot/config.txt << EOF >>>> kernel=$KERNEL_IMAGE >>>> EOF >>>> + >>>> +mkdir -p /etc/apt/sources.list.d/ >>>> +cat <<EOF >/etc/apt/sources.list.d/${DEBDISTRONAME}.list >>>> +deb file:${DEBCACHEMNT}/ ${DEBDISTRONAME} main >>>> +deb-src file:${DEBCACHEMNT}/ ${DEBDISTRONAME} main >>>> +EOF >>>> diff --git a/meta-isar/recipes-core/images/isar-image-base.bb >>>> b/meta-isar/recipes-core/images/isar-image-base.bb index >>>> b679d97..85e6b27 100644 --- >>>> a/meta-isar/recipes-core/images/isar-image-base.bb +++ >>>> b/meta-isar/recipes-core/images/isar-image-base.bb @@ -49,8 +49,12 >>>> @@ do_rootfs() { sudo multistrap -a ${DISTRO_ARCH} -d "${S}" -f >>>> "${WORKDIR}/multistrap.conf" || true >>>> # Configure root filesystem >>>> - sudo chroot ${S} /configscript.sh ${MACHINE_SERIAL} >>>> ${BAUDRATE_TTY} \ >>>> - ${ROOTFS_DEV} >>>> + sudo chroot ${S} /configscript.sh \ >>>> + ${MACHINE_SERIAL} \ >>>> + ${BAUDRATE_TTY} \ >>>> + ${ROOTFS_DEV} \ >>>> + ${DEBCACHEMNT} \ >>>> + ${DEBDISTRONAME} >>>> sudo rm ${S}/configscript.sh >>>> } >>>> >>>> diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass >>>> index 360a95c..b1e201d 100644 >>>> --- a/meta/classes/dpkg.bbclass >>>> +++ b/meta/classes/dpkg.bbclass >>>> @@ -1,5 +1,5 @@ >>>> # This software is a part of ISAR. >>>> -# Copyright (C) 2015-2016 ilbers GmbH >>>> +# Copyright (C) 2015-2017 ilbers GmbH >>>> >>>> # Add dependency from buildchroot creation >>>> DEPENDS += "buildchroot" >>>> @@ -55,12 +55,124 @@ do_build() { >>>> sudo chroot ${BUILDCHROOT_DIR} /build.sh ${PP}/${SRC_DIR} >>>> } >>>> >>>> - >>>> # Install package to dedicated deploy directory >>>> -do_install() { >>>> - install -m 644 ${BUILDROOT}/*.deb ${DEPLOY_DIR_DEB}/ >>>> +do_binary_deb_install() { >>>> + readonly DIR_CACHE="${DEBCACHEDIR}/${DISTRO}" >>>> + readonly DIR_DB="${DEBDBDIR}/${DISTRO}" >>>> + >>>> + if [ "${DEBCACHE_ENABLED}" != "0" ]; then >>>> + # If `bitbake` is running for the first time, the cache >>>> doesn't exist >>>> + # yet and needs to be configured using a `distributions` >>>> file. >>>> + # A template stored in the layer directory is >>>> pre-processed to >>>> + # generate the configuration file, which is then placed in >>>> the >>>> + # appropriate directory. >>>> + if [ ! -e "${DIR_CACHE}/conf/distributions" ]; then >>>> + mkdir -p "${DIR_CACHE}/conf" >>>> + sed -e "s#{DISTRO_NAME}#${DEBDISTRONAME}#g" \ >>>> + "${DEBFILESDIR}/distributions.in" \ >>>> + > "${DIR_CACHE}/conf/distributions" >>>> + fi >>> >>> This step should be in the image and not in every package. I guess >>> it has a race with multiple packages running this init routine. >> >> Ok. >> >>> >>>> + # Add binary and source packages to the deb cache >>>> + # If the cache doesn't exist yet, it will be created using >>>> the >>>> + # `distributions` file generated above. >>>> + ls -1 "${BUILDROOT}"/*.deb "${BUILDROOT}"/*.dsc | while >>>> read -r p; do >>>> + reprepro --waitforlock 3 -b "${DIR_CACHE}" --dbdir >>>> "${DIR_DB}" \ >>>> + -C main "include${p##*.}" "${DEBDISTRONAME}" >>>> "${p}" >>>> + done >>>> + else >>>> + # Deb caching is disabled, simply copy all binary packages >>>> to the >>>> + # deploy directory >>>> + mkdir -p "${DEPLOY_DIR_DEB}" >>>> + install -m 644 "${BUILDROOT}"/*.deb "${DEPLOY_DIR_DEB}/" >>>> + fi >>>> } >>>> >>>> -addtask install after do_build >>>> -do_install[dirs] = "${DEPLOY_DIR_DEB}" >>>> -do_install[stamp-extra-info] = "${MACHINE}" >>>> +addtask binary_deb_install after do_build >>>> +do_binary_deb_install[stamp-extra-info] = >>>> "${DISTRO}-${DISTRO_ARCH}" >>> >>> Here a task got renamed. This should be another patch ... one that i >>> already tried to get in. >>> >> >> This implemented in the next patch (to keep this patch in original >> format), but yes, it should be done here. >> >>>> + >>>> +# Deb caching lambda run during the parsing phase that checks >>>> whether the +# current package has to be rebuilt, or taken from the >>>> cache +python __anonymous () { >>>> + if d.getVar("DEBCACHE_ENABLED", True) == "0": >>>> + # Deb caching is disabled, do nothing >>>> + return True >>>> + >>>> + PN = d.getVar("PN", True) >>>> + PV = d.getVar("PV", True) >>>> + DISTRO_ARCH = d.getVar("DISTRO_ARCH", True) >>>> + DEBCACHEDIR = d.getVar("DEBCACHEDIR", True) >>>> + DEBDISTRONAME = d.getVar("DEBDISTRONAME", True) >>>> + DEBDBDIR = d.getVar("DEBDBDIR", True) >>>> + DISTRO = d.getVar("DISTRO", True) >>>> + path_cache = os.path.join(DEBCACHEDIR, DISTRO) >>>> + path_databases = os.path.join(DEBDBDIR, DISTRO) >>>> + path_distributions = os.path.join(path_cache, "conf", >>>> "distributions") + >>>> + # The distributions file is needed by `reprepro` to know what >>>> types >>>> + # of packages are supported, what the distribution name is, >>>> etc. >>>> + # If it doesn't exist, we have nothing in the cache, do >>>> nothing. >>>> + if not os.path.exists(path_distributions): >>>> + return >>>> + >>>> + # Anonymous functions are run several times under different >>>> contexts >>>> + # during the parsing phase, which would let the code that >>>> follows be run >>>> + # as many times for the same package. >>>> + # In order to guarantee that our subroutine only runs once per >>>> package, we >>>> + # use bitbake's "persist" API in order to have reliable >>>> persistent storage >>>> + # accross calls of the lambda (using a simple variable in the >>>> class won't >>>> + # work, as several contexts won't allow fetching its value). >>>> + pd = bb.persist_data.persist("DEBCACHE_PACKAGES", d) >>>> + if PN in pd and pd[PN] == PV: >>>> + return >>>> + >>>> + import subprocess >>>> + try: >>>> + # The databases used by `reprepro` are not stored within >>>> the cache in >>>> + # order to make versioning of only the files needed to use >>>> the cache >>>> + # as an official Debian repository simpler. >>>> + # As such, if a developer uses a peer's cache to speed up >>>> their build >>>> + # time but have never run bitbake, the database will not >>>> have been >>>> + # created, so we regenerate them here. >>>> + if not os.path.exists(path_databases) and >>>> os.path.exists(path_cache): >>>> + bb.note("Regenerating the cache databases...") >>>> + subprocess.check_call([ >>>> + "reprepro", >>>> + "--waitforlock", "3", >>>> + "-b", path_cache, >>>> + "--dbdir", path_databases, >>>> + "export", DEBDISTRONAME, >>>> + ]) >>>> + >>>> + # Get a list of the versions of all the packages named >>>> after the >>>> + # current bitbake package, and check whether the current >>>> package >>>> + # version is returned. >>>> + # As `reprepro` always returns zero with this particular >>>> operation, we >>>> + # have to use this workaround to check for a package in >>>> the cache. >>>> + package_version = subprocess.check_output([ >>>> + "reprepro", >>>> + "--waitforlock", "3", >>>> + "-b", path_cache, >>>> + "--dbdir", path_databases, >>>> + "-C", "main", >>>> + "-A", DISTRO_ARCH, >>>> + "--list-format", "${version}", >>>> + "list", DEBDISTRONAME, PN, >>>> + ]) >>>> + package_version = package_version.decode("utf-8") >>>> + if package_version == PV: >>>> + # The below list contains the names of all the tasks >>>> are in charge >>>> + # of building the package when the cache isn't enabled >>>> or if the >>>> + # package hasn't been placed in it already. >>>> + # As all tasks are enabled by default, we prevent >>>> their execution >>>> + # by setting the `noexec` flag, which will prevent a >>>> rebuild of >>>> + # the package when it's cached. >>>> + for task in ["fetch", "unpack", "build", "install"]: >>>> + d.setVarFlag("do_{}".format(task), "noexec", "1") >>>> + >>>> + # Cache the results of this command so that subsequent >>>> executions >>>> + # of this anonymous functions don't run the same code >>>> again. >>>> + pd[PN] = PV >>>> + except subprocess.CalledProcessError as e: >>>> + bb.fatal("Unable to check for a candidate for package {0} >>>> (errorcode: {1})".format(PN, e.returncode)) +} >>>> diff --git a/meta/classes/image.bbclass >>>> b/meta/classes/image.bbclass index a7f0d74..f42aa48 100644 >>>> --- a/meta/classes/image.bbclass >>>> +++ b/meta/classes/image.bbclass >>>> @@ -11,18 +11,34 @@ inherit ${IMAGE_TYPE} >>>> >>>> do_populate[stamp-extra-info] = "${MACHINE}-${DISTRO}" >>>> >>>> -# Install Debian packages, that were built from sources >>>> +# Install Debian packages from the cache >>>> do_populate() { >>>> + readonly DIR_CACHE="${DEBCACHEDIR}/${DISTRO}" >>>> + >>>> if [ -n "${IMAGE_INSTALL}" ]; then >>>> - sudo mkdir -p ${S}/deb >>>> + if [ "${DEBCACHE_ENABLED}" != "0" ]; then >>>> + sudo mkdir -p "${S}/${DEBCACHEMNT}" >>>> + sudo mount -o bind "${DIR_CACHE}" >>>> "${S}/${DEBCACHEMNT}" + >>>> + sudo chroot "${S}" apt-get update -y >>> >>> This will fail if you are behind a proxy. It makes a known issue >>> worse. >>>> + for package in ${IMAGE_INSTALL}; do >>>> + sudo chroot "${S}" apt-get install -t >>>> "${DEBDISTRONAME}" -y \ >>>> + --allow-unauthenticated "${package}" >>>> + done >>>> + >>>> + sudo umount "${S}/${DEBCACHEMNT}" >>> >>> A whole lot of sudo. I do not mind but am still wondering ... >> >> Please questions and proposals instead of negative tone. > > That was maybe slightly negative. Why? Because that patch was sent at a > time where the policy was still "do not add more sudos". And the patch > violates more of the groundrules we have agreed on. That is actually not true, because the policy still is "do not add more sudo if you have no use-case for them". I'd like to cover this topic again to try to explain the issue. The thread was started about your comment: > Some security enhancing packages can cause our initrd to be not > readable by a normal user. So we need to copy with sudo. > Also regular cp would destroy ownership and other attributes of files, > possibly creating problems in the future. and in the patch you added three sudo instead of single one for initrd. No problem with sudo for initrd - you provided use-case. But "possibly creating problems in the future" doesn't provide use-case for the remaining two entries. > You posting the patch signals that our groundrules seem to apply > to !ilbers-contributions only. > >>> Can we replace the bind-mount with a symlink? If anything between >>> the mount and umount fails and you need to run again ... will the >>> mount fail if the old one is still there? >> >> If we switch to multistrap-based populating, I think this will go. > > I know, but we have not decided that yet. > > Henning > >>> >>>> + else >>>> + sudo mkdir -p ${S}/deb >>>> >>>> - for p in ${IMAGE_INSTALL}; do >>>> - sudo cp ${DEPLOY_DIR_DEB}/${p}_*.deb ${S}/deb >>>> - done >>>> + for p in ${IMAGE_INSTALL}; do >>>> + find "${DEPLOY_DIR_DEB}" -type f -name '*.deb' >>>> -exec \ >>>> + sudo cp '{}' "${S}/deb/" \; >>>> + done >>>> >>>> - sudo chroot ${S} /usr/bin/dpkg -i -R /deb >>>> + sudo chroot ${S} /usr/bin/dpkg -i -R /deb >>>> >>>> - sudo rm -rf ${S}/deb >>>> + sudo rm -rf ${S}/deb >>>> + fi >>>> fi >>>> } >>>> >>> > ^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [PATCH 1/6] meta-isar-bin: Enable caching of deb packages 2017-08-28 15:18 ` Henning Schild 2017-08-29 6:40 ` Alexander Smirnov @ 2017-08-31 10:55 ` Claudius Heine 2017-08-31 11:20 ` Henning Schild 1 sibling, 1 reply; 30+ messages in thread From: Claudius Heine @ 2017-08-31 10:55 UTC (permalink / raw) To: [ext] Henning Schild, Alexander Smirnov; +Cc: isar-users, Frank Lenormand Hi, On 08/28/2017 05:18 PM, [ext] Henning Schild wrote: > - i think this task should go away and the repo should be given to > multistrap Yes please! Just found another issue why installing all packages that are in one directory is bad. The directory path (DEPLOY_DIR_DEB) is not and cannot be image specific, so building multiple images in the same environment causes packages to be installed just because they are part of an image that was build previously. Cheers, Claudius -- DENX Software Engineering GmbH, Managing Director: Wolfgang Denk HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de ^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [PATCH 1/6] meta-isar-bin: Enable caching of deb packages 2017-08-31 10:55 ` Claudius Heine @ 2017-08-31 11:20 ` Henning Schild 2017-08-31 12:08 ` Claudius Heine 0 siblings, 1 reply; 30+ messages in thread From: Henning Schild @ 2017-08-31 11:20 UTC (permalink / raw) To: Claudius Heine; +Cc: Alexander Smirnov, isar-users, Frank Lenormand Am Thu, 31 Aug 2017 12:55:52 +0200 schrieb Claudius Heine <claudius.heine.ext@siemens.com>: > Hi, > > On 08/28/2017 05:18 PM, [ext] Henning Schild wrote: > > - i think this task should go away and the repo should be given to > > multistrap > > Yes please! Just found another issue why installing all packages that > are in one directory is bad. The directory path (DEPLOY_DIR_DEB) is > not and cannot be image specific, so building multiple images in the > same environment causes packages to be installed just because they > are part of an image that was build previously. This comment does not address the point of feeding the repo into multistrap instead of doing do_populate. Are you sure you are using the do_populate as in master? > for p in ${IMAGE_INSTALL}; do > sudo cp ${DEPLOY_DIR_DEB}/${p}_*.deb ${S}/deb > done > sudo chroot ${S} /usr/bin/dpkg -i -R /deb This looks like not all are installed. But only the mentioned ones. It also looks like you must never have /deb in your rootfs but that is another issue that will get fixed when we get rid of do_populate. Henning > Cheers, > Claudius > ^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [PATCH 1/6] meta-isar-bin: Enable caching of deb packages 2017-08-31 11:20 ` Henning Schild @ 2017-08-31 12:08 ` Claudius Heine 0 siblings, 0 replies; 30+ messages in thread From: Claudius Heine @ 2017-08-31 12:08 UTC (permalink / raw) To: Henning Schild; +Cc: Alexander Smirnov, isar-users, Frank Lenormand Hi, On 08/31/2017 01:20 PM, Henning Schild wrote: > Am Thu, 31 Aug 2017 12:55:52 +0200 > schrieb Claudius Heine <claudius.heine.ext@siemens.com>: > >> Hi, >> >> On 08/28/2017 05:18 PM, [ext] Henning Schild wrote: >>> - i think this task should go away and the repo should be given to >>> multistrap >> >> Yes please! Just found another issue why installing all packages that >> are in one directory is bad. The directory path (DEPLOY_DIR_DEB) is >> not and cannot be image specific, so building multiple images in the >> same environment causes packages to be installed just because they >> are part of an image that was build previously. > > This comment does not address the point of feeding the repo into > multistrap instead of doing do_populate. > > Are you sure you are using the do_populate as in master? Nope, sorry. I used modified code here that handles package dependencies and this is the cause for this issue. But I would love to get rid of these kind of hacks this way! Cheers, Claudius -- DENX Software Engineering GmbH, Managing Director: Wolfgang Denk HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de ^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [PATCH 1/6] meta-isar-bin: Enable caching of deb packages 2017-08-27 15:13 ` [PATCH 1/6] meta-isar-bin: Enable caching of deb packages Alexander Smirnov 2017-08-28 15:18 ` Henning Schild @ 2017-09-06 14:21 ` Henning Schild 2017-09-07 11:13 ` Claudius Heine 1 sibling, 1 reply; 30+ messages in thread From: Henning Schild @ 2017-09-06 14:21 UTC (permalink / raw) To: Alexander Smirnov; +Cc: isar-users, Frank Lenormand Am Sun, 27 Aug 2017 18:13:34 +0300 schrieb Alexander Smirnov <asmirnov@ilbers.de>: > From: Frank Lenormand <lenormf@gmail.com> > > Adds the possibility to create an apt repository for newly built > packages and then use that repo for populating the target filesystem. > > Signed-off-by: Frank Lenormand <lenormf@gmail.com> > Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de> > --- > meta-isar-bin/conf/layer.conf | 18 +++ > meta-isar-bin/files/distributions.in | 3 + > meta-isar/conf/bblayers.conf.sample | 1 + > .../images/files/debian-configscript.sh | 8 ++ > .../images/files/raspbian-configscript.sh | 8 ++ > meta-isar/recipes-core/images/isar-image-base.bb | 8 +- > meta/classes/dpkg.bbclass | 126 > +++++++++++++++++++-- > meta/classes/image.bbclass | 30 +++-- 8 > files changed, 186 insertions(+), 16 deletions(-) create mode 100644 > meta-isar-bin/conf/layer.conf create mode 100644 > meta-isar-bin/files/distributions.in > > diff --git a/meta-isar-bin/conf/layer.conf > b/meta-isar-bin/conf/layer.conf new file mode 100644 > index 0000000..24534e1 > --- /dev/null > +++ b/meta-isar-bin/conf/layer.conf > @@ -0,0 +1,18 @@ > +# Enable package caching with '1' > +DEBCACHE_ENABLED ?= "0" > + > +# Codename of the repository created by the caching class > +DEBDISTRONAME = "isar" > + > +# Path to the caching repository > +DEBCACHEDIR ?= "${LAYERDIR}/apt" > + > +# Path to the mount point of the repository within the target > rootfs, during +# population > +DEBCACHEMNT ?= "/opt/cache/apt" > + > +# Path to the databases used by `reprepro` > +DEBDBDIR ?= "${LAYERDIR}/db" > + > +# Path to the configuration files templates used by `reprepro` > +DEBFILESDIR ?= "${LAYERDIR}/files" > diff --git a/meta-isar-bin/files/distributions.in > b/meta-isar-bin/files/distributions.in new file mode 100644 > index 0000000..59f4429 > --- /dev/null > +++ b/meta-isar-bin/files/distributions.in > @@ -0,0 +1,3 @@ > +Codename: {DISTRO_NAME} > +Architectures: i386 armhf source > +Components: main > diff --git a/meta-isar/conf/bblayers.conf.sample > b/meta-isar/conf/bblayers.conf.sample index 80867e7..53a362b 100644 > --- a/meta-isar/conf/bblayers.conf.sample > +++ b/meta-isar/conf/bblayers.conf.sample > @@ -8,6 +8,7 @@ BBFILES ?= "" > BBLAYERS ?= " \ > ##ISARROOT##/meta \ > ##ISARROOT##/meta-isar \ > + ##ISARROOT##/meta-isar-bin \ > " > BBLAYERS_NON_REMOVABLE ?= " \ > ##ISARROOT##/meta \ > diff --git > a/meta-isar/recipes-core/images/files/debian-configscript.sh > b/meta-isar/recipes-core/images/files/debian-configscript.sh index > 4ac37d0..b05babb 100644 --- > a/meta-isar/recipes-core/images/files/debian-configscript.sh +++ > b/meta-isar/recipes-core/images/files/debian-configscript.sh @@ -8,6 > +8,8 @@ set -e readonly MACHINE_SERIAL="$1" readonly BAUDRATE_TTY="$2" > readonly ROOTFS_DEV="$3" > +readonly DEBCACHEMNT="$4" > +readonly DEBDISTRONAME="$5" > > cat >> /etc/default/locale << EOF > LANG=en_US.UTF-8 > @@ -83,3 +85,9 @@ fi > if [ -x "$TARGET/sbin/init" -a -x "$TARGET/usr/sbin/policy-rc.d" ]; > then rm -f $TARGET/usr/sbin/policy-rc.d > fi > + > +mkdir -p /etc/apt/sources.list.d/ > +cat <<EOF >/etc/apt/sources.list.d/${DEBDISTRONAME}.list > +deb file:${DEBCACHEMNT}/ ${DEBDISTRONAME} main > +deb-src file:${DEBCACHEMNT}/ ${DEBDISTRONAME} main > +EOF > diff --git > a/meta-isar/recipes-core/images/files/raspbian-configscript.sh > b/meta-isar/recipes-core/images/files/raspbian-configscript.sh index > 2454481..f995f4d 100644 --- > a/meta-isar/recipes-core/images/files/raspbian-configscript.sh +++ > b/meta-isar/recipes-core/images/files/raspbian-configscript.sh @@ > -8,6 +8,8 @@ set -e readonly MACHINE_SERIAL="$1" readonly > BAUDRATE_TTY="$2" readonly ROOTFS_DEV="$3" > +readonly DEBCACHEMNT="$4" > +readonly DEBDISTRONAME="$5" > > cat >> /etc/default/locale << EOF > LANG=en_US.UTF-8 > @@ -89,3 +91,9 @@ KERNEL_IMAGE=`ls /boot | grep vmlinuz` > cat > /boot/config.txt << EOF > kernel=$KERNEL_IMAGE > EOF > + > +mkdir -p /etc/apt/sources.list.d/ > +cat <<EOF >/etc/apt/sources.list.d/${DEBDISTRONAME}.list > +deb file:${DEBCACHEMNT}/ ${DEBDISTRONAME} main > +deb-src file:${DEBCACHEMNT}/ ${DEBDISTRONAME} main > +EOF > diff --git a/meta-isar/recipes-core/images/isar-image-base.bb > b/meta-isar/recipes-core/images/isar-image-base.bb index > b679d97..85e6b27 100644 --- > a/meta-isar/recipes-core/images/isar-image-base.bb +++ > b/meta-isar/recipes-core/images/isar-image-base.bb @@ -49,8 +49,12 @@ > do_rootfs() { sudo multistrap -a ${DISTRO_ARCH} -d "${S}" -f > "${WORKDIR}/multistrap.conf" || true > # Configure root filesystem > - sudo chroot ${S} /configscript.sh ${MACHINE_SERIAL} > ${BAUDRATE_TTY} \ > - ${ROOTFS_DEV} > + sudo chroot ${S} /configscript.sh \ > + ${MACHINE_SERIAL} \ > + ${BAUDRATE_TTY} \ > + ${ROOTFS_DEV} \ > + ${DEBCACHEMNT} \ > + ${DEBDISTRONAME} > sudo rm ${S}/configscript.sh > } > > diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass > index 360a95c..b1e201d 100644 > --- a/meta/classes/dpkg.bbclass > +++ b/meta/classes/dpkg.bbclass > @@ -1,5 +1,5 @@ > # This software is a part of ISAR. > -# Copyright (C) 2015-2016 ilbers GmbH > +# Copyright (C) 2015-2017 ilbers GmbH > > # Add dependency from buildchroot creation > DEPENDS += "buildchroot" > @@ -55,12 +55,124 @@ do_build() { > sudo chroot ${BUILDCHROOT_DIR} /build.sh ${PP}/${SRC_DIR} > } > > - > # Install package to dedicated deploy directory > -do_install() { > - install -m 644 ${BUILDROOT}/*.deb ${DEPLOY_DIR_DEB}/ > +do_binary_deb_install() { > + readonly DIR_CACHE="${DEBCACHEDIR}/${DISTRO}" > + readonly DIR_DB="${DEBDBDIR}/${DISTRO}" > + > + if [ "${DEBCACHE_ENABLED}" != "0" ]; then > + # If `bitbake` is running for the first time, the cache > doesn't exist > + # yet and needs to be configured using a `distributions` > file. > + # A template stored in the layer directory is pre-processed > to > + # generate the configuration file, which is then placed in > the > + # appropriate directory. > + if [ ! -e "${DIR_CACHE}/conf/distributions" ]; then > + mkdir -p "${DIR_CACHE}/conf" > + sed -e "s#{DISTRO_NAME}#${DEBDISTRONAME}#g" \ > + "${DEBFILESDIR}/distributions.in" \ > + > "${DIR_CACHE}/conf/distributions" > + fi > + > + # Add binary and source packages to the deb cache > + # If the cache doesn't exist yet, it will be created using > the > + # `distributions` file generated above. > + ls -1 "${BUILDROOT}"/*.deb "${BUILDROOT}"/*.dsc | while read > -r p; do > + reprepro --waitforlock 3 -b "${DIR_CACHE}" --dbdir > "${DIR_DB}" \ > + -C main "include${p##*.}" "${DEBDISTRONAME}" "${p}" > + done > + else > + # Deb caching is disabled, simply copy all binary packages > to the > + # deploy directory > + mkdir -p "${DEPLOY_DIR_DEB}" > + install -m 644 "${BUILDROOT}"/*.deb "${DEPLOY_DIR_DEB}/" > + fi > } > > -addtask install after do_build > -do_install[dirs] = "${DEPLOY_DIR_DEB}" > -do_install[stamp-extra-info] = "${MACHINE}" > +addtask binary_deb_install after do_build > +do_binary_deb_install[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}" > + > +# Deb caching lambda run during the parsing phase that checks > whether the +# current package has to be rebuilt, or taken from the > cache +python __anonymous () { > + if d.getVar("DEBCACHE_ENABLED", True) == "0": > + # Deb caching is disabled, do nothing > + return True > + > + PN = d.getVar("PN", True) > + PV = d.getVar("PV", True) > + DISTRO_ARCH = d.getVar("DISTRO_ARCH", True) > + DEBCACHEDIR = d.getVar("DEBCACHEDIR", True) > + DEBDISTRONAME = d.getVar("DEBDISTRONAME", True) > + DEBDBDIR = d.getVar("DEBDBDIR", True) > + DISTRO = d.getVar("DISTRO", True) > + path_cache = os.path.join(DEBCACHEDIR, DISTRO) > + path_databases = os.path.join(DEBDBDIR, DISTRO) > + path_distributions = os.path.join(path_cache, "conf", > "distributions") + > + # The distributions file is needed by `reprepro` to know what > types > + # of packages are supported, what the distribution name is, etc. > + # If it doesn't exist, we have nothing in the cache, do nothing. > + if not os.path.exists(path_distributions): > + return > + > + # Anonymous functions are run several times under different > contexts > + # during the parsing phase, which would let the code that > follows be run > + # as many times for the same package. > + # In order to guarantee that our subroutine only runs once per > package, we > + # use bitbake's "persist" API in order to have reliable > persistent storage > + # accross calls of the lambda (using a simple variable in the > class won't > + # work, as several contexts won't allow fetching its value). > + pd = bb.persist_data.persist("DEBCACHE_PACKAGES", d) > + if PN in pd and pd[PN] == PV: > + return > + > + import subprocess > + try: > + # The databases used by `reprepro` are not stored within the > cache in > + # order to make versioning of only the files needed to use > the cache > + # as an official Debian repository simpler. > + # As such, if a developer uses a peer's cache to speed up > their build > + # time but have never run bitbake, the database will not > have been > + # created, so we regenerate them here. > + if not os.path.exists(path_databases) and > os.path.exists(path_cache): > + bb.note("Regenerating the cache databases...") > + subprocess.check_call([ > + "reprepro", > + "--waitforlock", "3", > + "-b", path_cache, > + "--dbdir", path_databases, > + "export", DEBDISTRONAME, > + ]) > + > + # Get a list of the versions of all the packages named after > the > + # current bitbake package, and check whether the current > package > + # version is returned. > + # As `reprepro` always returns zero with this particular > operation, we > + # have to use this workaround to check for a package in the > cache. > + package_version = subprocess.check_output([ > + "reprepro", > + "--waitforlock", "3", > + "-b", path_cache, > + "--dbdir", path_databases, > + "-C", "main", > + "-A", DISTRO_ARCH, > + "--list-format", "${version}", > + "list", DEBDISTRONAME, PN, > + ]) > + package_version = package_version.decode("utf-8") > + if package_version == PV: > + # The below list contains the names of all the tasks are > in charge > + # of building the package when the cache isn't enabled > or if the > + # package hasn't been placed in it already. > + # As all tasks are enabled by default, we prevent their > execution > + # by setting the `noexec` flag, which will prevent a > rebuild of > + # the package when it's cached. > + for task in ["fetch", "unpack", "build", "install"]: > + d.setVarFlag("do_{}".format(task), "noexec", "1") > + > + # Cache the results of this command so that subsequent > executions > + # of this anonymous functions don't run the same code > again. > + pd[PN] = PV > + except subprocess.CalledProcessError as e: > + bb.fatal("Unable to check for a candidate for package {0} > (errorcode: {1})".format(PN, e.returncode)) +} > diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass > index a7f0d74..f42aa48 100644 > --- a/meta/classes/image.bbclass > +++ b/meta/classes/image.bbclass > @@ -11,18 +11,34 @@ inherit ${IMAGE_TYPE} > > do_populate[stamp-extra-info] = "${MACHINE}-${DISTRO}" > > -# Install Debian packages, that were built from sources > +# Install Debian packages from the cache > do_populate() { > + readonly DIR_CACHE="${DEBCACHEDIR}/${DISTRO}" > + > if [ -n "${IMAGE_INSTALL}" ]; then > - sudo mkdir -p ${S}/deb > + if [ "${DEBCACHE_ENABLED}" != "0" ]; then > + sudo mkdir -p "${S}/${DEBCACHEMNT}" > + sudo mount -o bind "${DIR_CACHE}" "${S}/${DEBCACHEMNT}" > + > + sudo chroot "${S}" apt-get update -y > + for package in ${IMAGE_INSTALL}; do > + sudo chroot "${S}" apt-get install -t > "${DEBDISTRONAME}" -y \ > + --allow-unauthenticated "${package}" With do_populate using apt-get we now how to deal with package-name collisions. We could have the case where our repo contains a package that already exists in upstream debian. We will have to decide how to deal with them if they occur. My suggestions/first idea would be: - a recipe that creates a .deb with a taken package name fails by default -EBUSY - a recipe can set a variable to force itself in "OVERRIDE_DEBIAN = True" - a recipe that overrides the upstream package has to "win", we need a proper apt Pin-Priority Henning > + done > + > + sudo umount "${S}/${DEBCACHEMNT}" > + else > + sudo mkdir -p ${S}/deb > > - for p in ${IMAGE_INSTALL}; do > - sudo cp ${DEPLOY_DIR_DEB}/${p}_*.deb ${S}/deb > - done > + for p in ${IMAGE_INSTALL}; do > + find "${DEPLOY_DIR_DEB}" -type f -name '*.deb' -exec > \ > + sudo cp '{}' "${S}/deb/" \; > + done > > - sudo chroot ${S} /usr/bin/dpkg -i -R /deb > + sudo chroot ${S} /usr/bin/dpkg -i -R /deb > > - sudo rm -rf ${S}/deb > + sudo rm -rf ${S}/deb > + fi > fi > } > ^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [PATCH 1/6] meta-isar-bin: Enable caching of deb packages 2017-09-06 14:21 ` Henning Schild @ 2017-09-07 11:13 ` Claudius Heine 0 siblings, 0 replies; 30+ messages in thread From: Claudius Heine @ 2017-09-07 11:13 UTC (permalink / raw) To: [ext] Henning Schild, Alexander Smirnov; +Cc: isar-users, Frank Lenormand Hi, On 09/06/2017 04:21 PM, [ext] Henning Schild wrote: >> --- a/meta/classes/image.bbclass >> +++ b/meta/classes/image.bbclass >> @@ -11,18 +11,34 @@ inherit ${IMAGE_TYPE} >> >> do_populate[stamp-extra-info] = "${MACHINE}-${DISTRO}" >> >> -# Install Debian packages, that were built from sources >> +# Install Debian packages from the cache >> do_populate() { >> + readonly DIR_CACHE="${DEBCACHEDIR}/${DISTRO}" >> + >> if [ -n "${IMAGE_INSTALL}" ]; then >> - sudo mkdir -p ${S}/deb >> + if [ "${DEBCACHE_ENABLED}" != "0" ]; then >> + sudo mkdir -p "${S}/${DEBCACHEMNT}" >> + sudo mount -o bind "${DIR_CACHE}" "${S}/${DEBCACHEMNT}" >> + >> + sudo chroot "${S}" apt-get update -y >> + for package in ${IMAGE_INSTALL}; do >> + sudo chroot "${S}" apt-get install -t >> "${DEBDISTRONAME}" -y \ >> + --allow-unauthenticated "${package}" > > With do_populate using apt-get we now how to deal with package-name > collisions. We could have the case where our repo contains a package > that already exists in upstream debian. > > We will have to decide how to deal with them if they occur. My > suggestions/first idea would be: > - a recipe that creates a .deb with a taken package name fails by > default -EBUSY > - a recipe can set a variable to force itself in "OVERRIDE_DEBIAN = > True" > - a recipe that overrides the upstream package has to "win", we need > a proper apt Pin-Priority If all packages that are created by isar have for instance 'isar' in the version, then it should be possible to specify this version explicitly when installing them from the repository. I think pining is only necessary if upgrading packages on the target is a required feature. One option for this would be that another meta package (maybe created in the image recipe) has explicit version dependencies on all packages created by isar. This way apt would not upgrade those packages because it would break the version dependency. Cheers, Claudius -- DENX Software Engineering GmbH, Managing Director: Wolfgang Denk HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de ^ permalink raw reply [flat|nested] 30+ messages in thread
* [PATCH 2/6] classes/image: Provide /dev/null for Stretch apt 2017-08-27 15:13 [PATCH 0/6] Isar apt cache implementation Alexander Smirnov 2017-08-27 15:13 ` [PATCH 1/6] meta-isar-bin: Enable caching of deb packages Alexander Smirnov @ 2017-08-27 15:13 ` Alexander Smirnov 2017-08-28 15:20 ` Henning Schild 2017-08-27 15:13 ` [PATCH 3/6] classes/dpkg: Split install for cache Alexander Smirnov ` (3 subsequent siblings) 5 siblings, 1 reply; 30+ messages in thread From: Alexander Smirnov @ 2017-08-27 15:13 UTC (permalink / raw) To: isar-users; +Cc: Baurzhan Ismagulov, Alexander Smirnov From: Baurzhan Ismagulov <ibr@ilbers.de> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de> Signed-off-by: Baurzhan Ismagulov <ibr@ilbers.de> --- meta/classes/image.bbclass | 3 +++ meta/recipes-devtools/buildchroot/files/configscript.sh | 1 + 2 files changed, 4 insertions(+) diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass index f42aa48..c2ff453 100644 --- a/meta/classes/image.bbclass +++ b/meta/classes/image.bbclass @@ -20,11 +20,14 @@ do_populate() { sudo mkdir -p "${S}/${DEBCACHEMNT}" sudo mount -o bind "${DIR_CACHE}" "${S}/${DEBCACHEMNT}" + # apt-get http method, gpg require /dev/null + # Otherwise, apt http method hangs on Stretch sudo chroot "${S}" apt-get update -y for package in ${IMAGE_INSTALL}; do sudo chroot "${S}" apt-get install -t "${DEBDISTRONAME}" -y \ --allow-unauthenticated "${package}" done + sudo umount "${S}/dev" sudo umount "${S}/${DEBCACHEMNT}" else diff --git a/meta/recipes-devtools/buildchroot/files/configscript.sh b/meta/recipes-devtools/buildchroot/files/configscript.sh index 5080ae1..08c818b 100644 --- a/meta/recipes-devtools/buildchroot/files/configscript.sh +++ b/meta/recipes-devtools/buildchroot/files/configscript.sh @@ -40,6 +40,7 @@ export LC_ALL=C LANGUAGE=C LANG=C /var/lib/dpkg/info/dash.preinst install # apt-get http method, gpg require /dev/null +# Otherwise, apt http method hangs on Stretch mount -t devtmpfs -o mode=0755,nosuid devtmpfs /dev #configuring packages -- 2.1.4 ^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [PATCH 2/6] classes/image: Provide /dev/null for Stretch apt 2017-08-27 15:13 ` [PATCH 2/6] classes/image: Provide /dev/null for Stretch apt Alexander Smirnov @ 2017-08-28 15:20 ` Henning Schild 2017-08-28 15:26 ` Henning Schild 0 siblings, 1 reply; 30+ messages in thread From: Henning Schild @ 2017-08-28 15:20 UTC (permalink / raw) To: Alexander Smirnov; +Cc: isar-users, Baurzhan Ismagulov Am Sun, 27 Aug 2017 18:13:35 +0300 schrieb Alexander Smirnov <asmirnov@ilbers.de>: > From: Baurzhan Ismagulov <ibr@ilbers.de> > > Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de> > Signed-off-by: Baurzhan Ismagulov <ibr@ilbers.de> > --- > meta/classes/image.bbclass | 3 +++ > meta/recipes-devtools/buildchroot/files/configscript.sh | 1 + > 2 files changed, 4 insertions(+) > > diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass > index f42aa48..c2ff453 100644 > --- a/meta/classes/image.bbclass > +++ b/meta/classes/image.bbclass > @@ -20,11 +20,14 @@ do_populate() { > sudo mkdir -p "${S}/${DEBCACHEMNT}" > sudo mount -o bind "${DIR_CACHE}" "${S}/${DEBCACHEMNT}" > > + # apt-get http method, gpg require /dev/null > + # Otherwise, apt http method hangs on Stretch > sudo chroot "${S}" apt-get update -y > for package in ${IMAGE_INSTALL}; do > sudo chroot "${S}" apt-get install -t > "${DEBDISTRONAME}" -y \ --allow-unauthenticated "${package}" > done > + sudo umount "${S}/dev" I see a new umount and a comment, where is the new mount? Henning > sudo umount "${S}/${DEBCACHEMNT}" > else > diff --git a/meta/recipes-devtools/buildchroot/files/configscript.sh > b/meta/recipes-devtools/buildchroot/files/configscript.sh index > 5080ae1..08c818b 100644 --- > a/meta/recipes-devtools/buildchroot/files/configscript.sh +++ > b/meta/recipes-devtools/buildchroot/files/configscript.sh @@ -40,6 > +40,7 @@ export LC_ALL=C LANGUAGE=C > LANG=C /var/lib/dpkg/info/dash.preinst install > # apt-get http method, gpg require /dev/null > +# Otherwise, apt http method hangs on Stretch > mount -t devtmpfs -o mode=0755,nosuid devtmpfs /dev > > #configuring packages ^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [PATCH 2/6] classes/image: Provide /dev/null for Stretch apt 2017-08-28 15:20 ` Henning Schild @ 2017-08-28 15:26 ` Henning Schild 0 siblings, 0 replies; 30+ messages in thread From: Henning Schild @ 2017-08-28 15:26 UTC (permalink / raw) To: Alexander Smirnov; +Cc: isar-users, Baurzhan Ismagulov Am Mon, 28 Aug 2017 17:20:06 +0200 schrieb "[ext] Henning Schild" <henning.schild@siemens.com>: > Am Sun, 27 Aug 2017 18:13:35 +0300 > schrieb Alexander Smirnov <asmirnov@ilbers.de>: > > > From: Baurzhan Ismagulov <ibr@ilbers.de> > > > > Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de> > > Signed-off-by: Baurzhan Ismagulov <ibr@ilbers.de> > > --- > > meta/classes/image.bbclass | 3 +++ > > meta/recipes-devtools/buildchroot/files/configscript.sh | 1 + > > 2 files changed, 4 insertions(+) > > > > diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass > > index f42aa48..c2ff453 100644 > > --- a/meta/classes/image.bbclass > > +++ b/meta/classes/image.bbclass > > @@ -20,11 +20,14 @@ do_populate() { > > sudo mkdir -p "${S}/${DEBCACHEMNT}" > > sudo mount -o bind "${DIR_CACHE}" "${S}/${DEBCACHEMNT}" > > > > + # apt-get http method, gpg require /dev/null > > + # Otherwise, apt http method hangs on Stretch > > sudo chroot "${S}" apt-get update -y > > for package in ${IMAGE_INSTALL}; do > > sudo chroot "${S}" apt-get install -t > > "${DEBDISTRONAME}" -y \ --allow-unauthenticated "${package}" > > done > > + sudo umount "${S}/dev" > > I see a new umount and a comment, where is the new mount? > > Henning > > > sudo umount "${S}/${DEBCACHEMNT}" > > else > > diff --git a/meta/recipes-devtools/buildchroot/files/configscript.sh > > b/meta/recipes-devtools/buildchroot/files/configscript.sh index > > 5080ae1..08c818b 100644 --- > > a/meta/recipes-devtools/buildchroot/files/configscript.sh +++ > > b/meta/recipes-devtools/buildchroot/files/configscript.sh @@ -40,6 > > +40,7 @@ export LC_ALL=C LANGUAGE=C > > LANG=C /var/lib/dpkg/info/dash.preinst install > > # apt-get http method, gpg require /dev/null > > +# Otherwise, apt http method hangs on Stretch > > mount -t devtmpfs -o mode=0755,nosuid devtmpfs /dev Found it ... and it is not new. Can we not get the "mount; { stuff } umount;" into one file? But we should first discuss whether we will drop do_populate alltogether. See "[PATCH 3/3] classes: image: remove populate and replace it with a custom repo" Henning > > #configuring packages > ^ permalink raw reply [flat|nested] 30+ messages in thread
* [PATCH 3/6] classes/dpkg: Split install for cache 2017-08-27 15:13 [PATCH 0/6] Isar apt cache implementation Alexander Smirnov 2017-08-27 15:13 ` [PATCH 1/6] meta-isar-bin: Enable caching of deb packages Alexander Smirnov 2017-08-27 15:13 ` [PATCH 2/6] classes/image: Provide /dev/null for Stretch apt Alexander Smirnov @ 2017-08-27 15:13 ` Alexander Smirnov 2017-08-28 8:00 ` Claudius Heine 2017-08-28 15:30 ` Henning Schild 2017-08-27 15:13 ` [PATCH 4/6] meta-isar-bin: Enable apt repo generation for amd64 Alexander Smirnov ` (2 subsequent siblings) 5 siblings, 2 replies; 30+ messages in thread From: Alexander Smirnov @ 2017-08-27 15:13 UTC (permalink / raw) To: isar-users; +Cc: Alexander Smirnov Split install function into two parts: - install_to_cache: if caching is enabled - install_to_deploy: if caching is disabled This patch brings flexibility to the implementation and makes it possible to move all the caching implementation code to dedicated class. The magic behavior depending on the value of DEBCACHE_ENABLED is now transparent in the bitbake pipeline (can be inspected via "bitbake -g"). Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de> --- meta/classes/dpkg.bbclass | 66 ++++++++++++++++++++++++++-------------------- meta/classes/image.bbclass | 2 +- 2 files changed, 38 insertions(+), 30 deletions(-) diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass index b1e201d..118ba2f 100644 --- a/meta/classes/dpkg.bbclass +++ b/meta/classes/dpkg.bbclass @@ -56,46 +56,53 @@ do_build() { } # Install package to dedicated deploy directory -do_binary_deb_install() { +do_install_to_deploy() { + # Deb caching is disabled, simply copy all binary packages to the deploy + # directory + install -m 644 "${BUILDROOT}"/*.deb "${DEPLOY_DIR_DEB}/" +} + +addtask install_to_deploy after do_build +do_install_to_deploy[dirs] = "${DEPLOY_DIR_DEB}" +do_install_to_deploy[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}" +do_install_to_deploy[noexec] = "1" + +# Install package to dedicated apt repo +do_install_to_cache() { readonly DIR_CACHE="${DEBCACHEDIR}/${DISTRO}" readonly DIR_DB="${DEBDBDIR}/${DISTRO}" - if [ "${DEBCACHE_ENABLED}" != "0" ]; then - # If `bitbake` is running for the first time, the cache doesn't exist - # yet and needs to be configured using a `distributions` file. - # A template stored in the layer directory is pre-processed to - # generate the configuration file, which is then placed in the - # appropriate directory. - if [ ! -e "${DIR_CACHE}/conf/distributions" ]; then - mkdir -p "${DIR_CACHE}/conf" - sed -e "s#{DISTRO_NAME}#${DEBDISTRONAME}#g" \ - "${DEBFILESDIR}/distributions.in" \ - > "${DIR_CACHE}/conf/distributions" - fi - - # Add binary and source packages to the deb cache - # If the cache doesn't exist yet, it will be created using the - # `distributions` file generated above. - ls -1 "${BUILDROOT}"/*.deb "${BUILDROOT}"/*.dsc | while read -r p; do - reprepro --waitforlock 3 -b "${DIR_CACHE}" --dbdir "${DIR_DB}" \ - -C main "include${p##*.}" "${DEBDISTRONAME}" "${p}" - done - else - # Deb caching is disabled, simply copy all binary packages to the - # deploy directory - mkdir -p "${DEPLOY_DIR_DEB}" - install -m 644 "${BUILDROOT}"/*.deb "${DEPLOY_DIR_DEB}/" + # If `bitbake` is running for the first time, the cache doesn't exist + # yet and needs to be configured using a `distributions` file. + # A template stored in the layer directory is pre-processed to + # generate the configuration file, which is then placed in the + # appropriate directory. + if [ ! -e "${DIR_CACHE}/conf/distributions" ]; then + mkdir -p "${DIR_CACHE}/conf" + sed -e "s#{DISTRO_NAME}#${DEBDISTRONAME}#g" \ + "${DEBFILESDIR}/distributions.in" \ + > "${DIR_CACHE}/conf/distributions" fi + + # Add binary and source packages to the deb cache + # If the cache doesn't exist yet, it will be created using the + # `distributions` file generated above. + ls -1 "${BUILDROOT}"/*.deb "${BUILDROOT}"/*.dsc | while read -r p; do + reprepro --waitforlock 3 -b "${DIR_CACHE}" --dbdir "${DIR_DB}" \ + -C main "include${p##*.}" "${DEBDISTRONAME}" "${p}" + done } -addtask binary_deb_install after do_build -do_binary_deb_install[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}" +addtask install_to_cache after do_build +do_install_to_cache[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}" +do_install_to_cache[noexec] = "1" # Deb caching lambda run during the parsing phase that checks whether the # current package has to be rebuilt, or taken from the cache python __anonymous () { if d.getVar("DEBCACHE_ENABLED", True) == "0": - # Deb caching is disabled, do nothing + # Deb caching is disabled + d.delVarFlag('do_install_to_deploy', 'noexec') return True PN = d.getVar("PN", True) @@ -108,6 +115,7 @@ python __anonymous () { path_cache = os.path.join(DEBCACHEDIR, DISTRO) path_databases = os.path.join(DEBDBDIR, DISTRO) path_distributions = os.path.join(path_cache, "conf", "distributions") + d.delVarFlag('do_install_to_cache', 'noexec') # The distributions file is needed by `reprepro` to know what types # of packages are supported, what the distribution name is, etc. diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass index c2ff453..6b1b5eb 100644 --- a/meta/classes/image.bbclass +++ b/meta/classes/image.bbclass @@ -46,4 +46,4 @@ do_populate() { } addtask populate before do_build -do_populate[deptask] = "do_install" +do_populate[deptask] = "do_install_to_cache do_install_to_deploy" -- 2.1.4 ^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [PATCH 3/6] classes/dpkg: Split install for cache 2017-08-27 15:13 ` [PATCH 3/6] classes/dpkg: Split install for cache Alexander Smirnov @ 2017-08-28 8:00 ` Claudius Heine 2017-08-29 7:18 ` Alexander Smirnov 2017-08-28 15:30 ` Henning Schild 1 sibling, 1 reply; 30+ messages in thread From: Claudius Heine @ 2017-08-28 8:00 UTC (permalink / raw) To: Alexander Smirnov, isar-users Hi, On 08/27/2017 05:13 PM, Alexander Smirnov wrote: > Split install function into two parts: > - install_to_cache: if caching is enabled > - install_to_deploy: if caching is disabled > > This patch brings flexibility to the implementation and makes it possible > to move all the caching implementation code to dedicated class. The magic > behavior depending on the value of DEBCACHE_ENABLED is now transparent in > the bitbake pipeline (can be inspected via "bitbake -g"). > > Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de> > --- > meta/classes/dpkg.bbclass | 66 ++++++++++++++++++++++++++-------------------- > meta/classes/image.bbclass | 2 +- > 2 files changed, 38 insertions(+), 30 deletions(-) > > diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass > index b1e201d..118ba2f 100644 > --- a/meta/classes/dpkg.bbclass > +++ b/meta/classes/dpkg.bbclass > @@ -56,46 +56,53 @@ do_build() { > } > > # Install package to dedicated deploy directory > -do_binary_deb_install() { > +do_install_to_deploy() { > + # Deb caching is disabled, simply copy all binary packages to the deploy > + # directory > + install -m 644 "${BUILDROOT}"/*.deb "${DEPLOY_DIR_DEB}/" > +} > + > +addtask install_to_deploy after do_build > +do_install_to_deploy[dirs] = "${DEPLOY_DIR_DEB}" > +do_install_to_deploy[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}" > +do_install_to_deploy[noexec] = "1" > + > +# Install package to dedicated apt repo > +do_install_to_cache() { > readonly DIR_CACHE="${DEBCACHEDIR}/${DISTRO}" > readonly DIR_DB="${DEBDBDIR}/${DISTRO}" > > - if [ "${DEBCACHE_ENABLED}" != "0" ]; then > - # If `bitbake` is running for the first time, the cache doesn't exist > - # yet and needs to be configured using a `distributions` file. > - # A template stored in the layer directory is pre-processed to > - # generate the configuration file, which is then placed in the > - # appropriate directory. > - if [ ! -e "${DIR_CACHE}/conf/distributions" ]; then > - mkdir -p "${DIR_CACHE}/conf" > - sed -e "s#{DISTRO_NAME}#${DEBDISTRONAME}#g" \ > - "${DEBFILESDIR}/distributions.in" \ > - > "${DIR_CACHE}/conf/distributions" > - fi > - > - # Add binary and source packages to the deb cache > - # If the cache doesn't exist yet, it will be created using the > - # `distributions` file generated above. > - ls -1 "${BUILDROOT}"/*.deb "${BUILDROOT}"/*.dsc | while read -r p; do > - reprepro --waitforlock 3 -b "${DIR_CACHE}" --dbdir "${DIR_DB}" \ > - -C main "include${p##*.}" "${DEBDISTRONAME}" "${p}" > - done > - else > - # Deb caching is disabled, simply copy all binary packages to the > - # deploy directory > - mkdir -p "${DEPLOY_DIR_DEB}" > - install -m 644 "${BUILDROOT}"/*.deb "${DEPLOY_DIR_DEB}/" > + # If `bitbake` is running for the first time, the cache doesn't exist > + # yet and needs to be configured using a `distributions` file. > + # A template stored in the layer directory is pre-processed to > + # generate the configuration file, which is then placed in the > + # appropriate directory. > + if [ ! -e "${DIR_CACHE}/conf/distributions" ]; then > + mkdir -p "${DIR_CACHE}/conf" > + sed -e "s#{DISTRO_NAME}#${DEBDISTRONAME}#g" \ > + "${DEBFILESDIR}/distributions.in" \ > + > "${DIR_CACHE}/conf/distributions" > fi > + > + # Add binary and source packages to the deb cache > + # If the cache doesn't exist yet, it will be created using the > + # `distributions` file generated above. > + ls -1 "${BUILDROOT}"/*.deb "${BUILDROOT}"/*.dsc | while read -r p; do > + reprepro --waitforlock 3 -b "${DIR_CACHE}" --dbdir "${DIR_DB}" \ > + -C main "include${p##*.}" "${DEBDISTRONAME}" "${p}" > + done > } > > -addtask binary_deb_install after do_build > -do_binary_deb_install[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}" > +addtask install_to_cache after do_build What was the reason why we put stuff after the 'build' task? In OE the 'build' task is the default task triggering everything necessary for a recipe. In Isar that changed. Why? > +do_install_to_cache[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}" > +do_install_to_cache[noexec] = "1" > > # Deb caching lambda run during the parsing phase that checks whether the > # current package has to be rebuilt, or taken from the cache > python __anonymous () { > if d.getVar("DEBCACHE_ENABLED", True) == "0": > - # Deb caching is disabled, do nothing > + # Deb caching is disabled > + d.delVarFlag('do_install_to_deploy', 'noexec') > return True > > PN = d.getVar("PN", True) > @@ -108,6 +115,7 @@ python __anonymous () { > path_cache = os.path.join(DEBCACHEDIR, DISTRO) > path_databases = os.path.join(DEBDBDIR, DISTRO) > path_distributions = os.path.join(path_cache, "conf", "distributions") > + d.delVarFlag('do_install_to_cache', 'noexec') > > # The distributions file is needed by `reprepro` to know what types > # of packages are supported, what the distribution name is, etc. > diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass > index c2ff453..6b1b5eb 100644 > --- a/meta/classes/image.bbclass > +++ b/meta/classes/image.bbclass > @@ -46,4 +46,4 @@ do_populate() { > } > > addtask populate before do_build > -do_populate[deptask] = "do_install" > +do_populate[deptask] = "do_install_to_cache do_install_to_deploy" I would rather still have a, maybe empty, 'do_install' step that just depends on the 'do_install_to_cache' and 'do_install_to_deploy' tasks. I don't see why ever small change in one class should change the complete pipeline for every recipe. Cheers, Claudius -- DENX Software Engineering GmbH, Managing Director: Wolfgang Denk HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de ^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [PATCH 3/6] classes/dpkg: Split install for cache 2017-08-28 8:00 ` Claudius Heine @ 2017-08-29 7:18 ` Alexander Smirnov 2017-08-30 8:54 ` Claudius Heine 0 siblings, 1 reply; 30+ messages in thread From: Alexander Smirnov @ 2017-08-29 7:18 UTC (permalink / raw) To: Claudius Heine, isar-users On 08/28/2017 11:00 AM, Claudius Heine wrote: > Hi, > > On 08/27/2017 05:13 PM, Alexander Smirnov wrote: >> Split install function into two parts: >> - install_to_cache: if caching is enabled >> - install_to_deploy: if caching is disabled >> >> This patch brings flexibility to the implementation and makes it possible >> to move all the caching implementation code to dedicated class. The magic >> behavior depending on the value of DEBCACHE_ENABLED is now transparent in >> the bitbake pipeline (can be inspected via "bitbake -g"). >> >> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de> >> --- >> meta/classes/dpkg.bbclass | 66 >> ++++++++++++++++++++++++++-------------------- >> meta/classes/image.bbclass | 2 +- >> 2 files changed, 38 insertions(+), 30 deletions(-) >> >> diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass >> index b1e201d..118ba2f 100644 >> --- a/meta/classes/dpkg.bbclass >> +++ b/meta/classes/dpkg.bbclass >> @@ -56,46 +56,53 @@ do_build() { >> } >> # Install package to dedicated deploy directory >> -do_binary_deb_install() { >> +do_install_to_deploy() { >> + # Deb caching is disabled, simply copy all binary packages to the >> deploy >> + # directory >> + install -m 644 "${BUILDROOT}"/*.deb "${DEPLOY_DIR_DEB}/" >> +} >> + >> +addtask install_to_deploy after do_build >> +do_install_to_deploy[dirs] = "${DEPLOY_DIR_DEB}" >> +do_install_to_deploy[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}" >> +do_install_to_deploy[noexec] = "1" >> + >> +# Install package to dedicated apt repo >> +do_install_to_cache() { >> readonly DIR_CACHE="${DEBCACHEDIR}/${DISTRO}" >> readonly DIR_DB="${DEBDBDIR}/${DISTRO}" >> - if [ "${DEBCACHE_ENABLED}" != "0" ]; then >> - # If `bitbake` is running for the first time, the cache >> doesn't exist >> - # yet and needs to be configured using a `distributions` file. >> - # A template stored in the layer directory is pre-processed to >> - # generate the configuration file, which is then placed in the >> - # appropriate directory. >> - if [ ! -e "${DIR_CACHE}/conf/distributions" ]; then >> - mkdir -p "${DIR_CACHE}/conf" >> - sed -e "s#{DISTRO_NAME}#${DEBDISTRONAME}#g" \ >> - "${DEBFILESDIR}/distributions.in" \ >> - > "${DIR_CACHE}/conf/distributions" >> - fi >> - >> - # Add binary and source packages to the deb cache >> - # If the cache doesn't exist yet, it will be created using the >> - # `distributions` file generated above. >> - ls -1 "${BUILDROOT}"/*.deb "${BUILDROOT}"/*.dsc | while read >> -r p; do >> - reprepro --waitforlock 3 -b "${DIR_CACHE}" --dbdir >> "${DIR_DB}" \ >> - -C main "include${p##*.}" "${DEBDISTRONAME}" "${p}" >> - done >> - else >> - # Deb caching is disabled, simply copy all binary packages to >> the >> - # deploy directory >> - mkdir -p "${DEPLOY_DIR_DEB}" >> - install -m 644 "${BUILDROOT}"/*.deb "${DEPLOY_DIR_DEB}/" >> + # If `bitbake` is running for the first time, the cache doesn't >> exist >> + # yet and needs to be configured using a `distributions` file. >> + # A template stored in the layer directory is pre-processed to >> + # generate the configuration file, which is then placed in the >> + # appropriate directory. >> + if [ ! -e "${DIR_CACHE}/conf/distributions" ]; then >> + mkdir -p "${DIR_CACHE}/conf" >> + sed -e "s#{DISTRO_NAME}#${DEBDISTRONAME}#g" \ >> + "${DEBFILESDIR}/distributions.in" \ >> + > "${DIR_CACHE}/conf/distributions" >> fi >> + >> + # Add binary and source packages to the deb cache >> + # If the cache doesn't exist yet, it will be created using the >> + # `distributions` file generated above. >> + ls -1 "${BUILDROOT}"/*.deb "${BUILDROOT}"/*.dsc | while read -r >> p; do >> + reprepro --waitforlock 3 -b "${DIR_CACHE}" --dbdir "${DIR_DB}" \ >> + -C main "include${p##*.}" "${DEBDISTRONAME}" "${p}" >> + done >> } >> -addtask binary_deb_install after do_build >> -do_binary_deb_install[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}" >> +addtask install_to_cache after do_build > > What was the reason why we put stuff after the 'build' task? In OE the > 'build' task is the default task triggering everything necessary for a > recipe. In Isar that changed. Why? Actually AFAIK do_build is not OE tasks, it's default bitbake task. Install is after do_build, because in Isar do_build is used to build the packages, so it's not empty. > >> +do_install_to_cache[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}" >> +do_install_to_cache[noexec] = "1" >> # Deb caching lambda run during the parsing phase that checks >> whether the >> # current package has to be rebuilt, or taken from the cache >> python __anonymous () { >> if d.getVar("DEBCACHE_ENABLED", True) == "0": >> - # Deb caching is disabled, do nothing >> + # Deb caching is disabled >> + d.delVarFlag('do_install_to_deploy', 'noexec') >> return True >> PN = d.getVar("PN", True) >> @@ -108,6 +115,7 @@ python __anonymous () { >> path_cache = os.path.join(DEBCACHEDIR, DISTRO) >> path_databases = os.path.join(DEBDBDIR, DISTRO) >> path_distributions = os.path.join(path_cache, "conf", >> "distributions") >> + d.delVarFlag('do_install_to_cache', 'noexec') >> # The distributions file is needed by `reprepro` to know what types >> # of packages are supported, what the distribution name is, etc. >> diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass >> index c2ff453..6b1b5eb 100644 >> --- a/meta/classes/image.bbclass >> +++ b/meta/classes/image.bbclass >> @@ -46,4 +46,4 @@ do_populate() { >> } >> addtask populate before do_build >> -do_populate[deptask] = "do_install" >> +do_populate[deptask] = "do_install_to_cache do_install_to_deploy" > > I would rather still have a, maybe empty, 'do_install' step that just > depends on the 'do_install_to_cache' and 'do_install_to_deploy' tasks. I > don't see why ever small change in one class should change the complete > pipeline for every recipe. It's a bit difficult question. In general meaning when building something from sources, the install means a storing of build artifacts to some specific location. So in this case, we just copy binary deb package to repo, and there is nothing common with installation of artifacts. Also Isar has in roadmap cross-compilation, so the pipeline will probably look like: fetch - unpack - compile - install - install_to_cache so in this case the install task probably should go before install_to_cache. So probably it makes sense to avoid _install_ in this task, for example use do_populate_deb_cache, what do you think? -- With best regards, Alexander Smirnov ilbers GmbH Baierbrunner Str. 28c D-81379 Munich +49 (89) 122 67 24-0 http://ilbers.de/ Commercial register Munich, HRB 214197 General manager: Baurzhan Ismagulov ^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [PATCH 3/6] classes/dpkg: Split install for cache 2017-08-29 7:18 ` Alexander Smirnov @ 2017-08-30 8:54 ` Claudius Heine 0 siblings, 0 replies; 30+ messages in thread From: Claudius Heine @ 2017-08-30 8:54 UTC (permalink / raw) To: Alexander Smirnov, isar-users Hi, On 08/29/2017 09:18 AM, Alexander Smirnov wrote: > > > On 08/28/2017 11:00 AM, Claudius Heine wrote: >> Hi, >> >> On 08/27/2017 05:13 PM, Alexander Smirnov wrote: >>> Split install function into two parts: >>> - install_to_cache: if caching is enabled >>> - install_to_deploy: if caching is disabled >>> >>> This patch brings flexibility to the implementation and makes it >>> possible >>> to move all the caching implementation code to dedicated class. The >>> magic >>> behavior depending on the value of DEBCACHE_ENABLED is now >>> transparent in >>> the bitbake pipeline (can be inspected via "bitbake -g"). >>> >>> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de> >>> --- >>> meta/classes/dpkg.bbclass | 66 >>> ++++++++++++++++++++++++++-------------------- >>> meta/classes/image.bbclass | 2 +- >>> 2 files changed, 38 insertions(+), 30 deletions(-) >>> >>> diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass >>> index b1e201d..118ba2f 100644 >>> --- a/meta/classes/dpkg.bbclass >>> +++ b/meta/classes/dpkg.bbclass >>> @@ -56,46 +56,53 @@ do_build() { >>> } >>> # Install package to dedicated deploy directory >>> -do_binary_deb_install() { >>> +do_install_to_deploy() { >>> + # Deb caching is disabled, simply copy all binary packages to >>> the deploy >>> + # directory >>> + install -m 644 "${BUILDROOT}"/*.deb "${DEPLOY_DIR_DEB}/" >>> +} >>> + >>> +addtask install_to_deploy after do_build >>> +do_install_to_deploy[dirs] = "${DEPLOY_DIR_DEB}" >>> +do_install_to_deploy[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}" >>> +do_install_to_deploy[noexec] = "1" >>> + >>> +# Install package to dedicated apt repo >>> +do_install_to_cache() { >>> readonly DIR_CACHE="${DEBCACHEDIR}/${DISTRO}" >>> readonly DIR_DB="${DEBDBDIR}/${DISTRO}" >>> - if [ "${DEBCACHE_ENABLED}" != "0" ]; then >>> - # If `bitbake` is running for the first time, the cache >>> doesn't exist >>> - # yet and needs to be configured using a `distributions` file. >>> - # A template stored in the layer directory is pre-processed to >>> - # generate the configuration file, which is then placed in the >>> - # appropriate directory. >>> - if [ ! -e "${DIR_CACHE}/conf/distributions" ]; then >>> - mkdir -p "${DIR_CACHE}/conf" >>> - sed -e "s#{DISTRO_NAME}#${DEBDISTRONAME}#g" \ >>> - "${DEBFILESDIR}/distributions.in" \ >>> - > "${DIR_CACHE}/conf/distributions" >>> - fi >>> - >>> - # Add binary and source packages to the deb cache >>> - # If the cache doesn't exist yet, it will be created using the >>> - # `distributions` file generated above. >>> - ls -1 "${BUILDROOT}"/*.deb "${BUILDROOT}"/*.dsc | while read >>> -r p; do >>> - reprepro --waitforlock 3 -b "${DIR_CACHE}" --dbdir >>> "${DIR_DB}" \ >>> - -C main "include${p##*.}" "${DEBDISTRONAME}" "${p}" >>> - done >>> - else >>> - # Deb caching is disabled, simply copy all binary packages >>> to the >>> - # deploy directory >>> - mkdir -p "${DEPLOY_DIR_DEB}" >>> - install -m 644 "${BUILDROOT}"/*.deb "${DEPLOY_DIR_DEB}/" >>> + # If `bitbake` is running for the first time, the cache doesn't >>> exist >>> + # yet and needs to be configured using a `distributions` file. >>> + # A template stored in the layer directory is pre-processed to >>> + # generate the configuration file, which is then placed in the >>> + # appropriate directory. >>> + if [ ! -e "${DIR_CACHE}/conf/distributions" ]; then >>> + mkdir -p "${DIR_CACHE}/conf" >>> + sed -e "s#{DISTRO_NAME}#${DEBDISTRONAME}#g" \ >>> + "${DEBFILESDIR}/distributions.in" \ >>> + > "${DIR_CACHE}/conf/distributions" >>> fi >>> + >>> + # Add binary and source packages to the deb cache >>> + # If the cache doesn't exist yet, it will be created using the >>> + # `distributions` file generated above. >>> + ls -1 "${BUILDROOT}"/*.deb "${BUILDROOT}"/*.dsc | while read -r >>> p; do >>> + reprepro --waitforlock 3 -b "${DIR_CACHE}" --dbdir >>> "${DIR_DB}" \ >>> + -C main "include${p##*.}" "${DEBDISTRONAME}" "${p}" >>> + done >>> } >>> -addtask binary_deb_install after do_build >>> -do_binary_deb_install[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}" >>> +addtask install_to_cache after do_build >> >> What was the reason why we put stuff after the 'build' task? In OE the >> 'build' task is the default task triggering everything necessary for a >> recipe. In Isar that changed. Why? > > Actually AFAIK do_build is not OE tasks, it's default bitbake task. > Install is after do_build, because in Isar do_build is used to build the > packages, so it's not empty. You are right, its a bitbake task. Since I have much more experience with OE than with just bitbake or isar, I referenced my question to the experience I had using OE. I cannot remember seeing tasks that are added after the do_build task in OE and I still find that very strange. AFAIK the 'build' task in bitbake and OE has a similar meaning to the 'all' target in make files. The idea to add statements to the 'all' target or letting other targets depend on the 'all' target sounds very odd to me. And that is what is done here in isar. Just very strange and unusual for my eyes and I don't see the advantage of this. I would like to hear the reason for this decision. Why have you added statements to the 'build' task and why are other tasks depend on it? >> >>> +do_install_to_cache[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}" >>> +do_install_to_cache[noexec] = "1" >>> # Deb caching lambda run during the parsing phase that checks >>> whether the >>> # current package has to be rebuilt, or taken from the cache >>> python __anonymous () { >>> if d.getVar("DEBCACHE_ENABLED", True) == "0": >>> - # Deb caching is disabled, do nothing >>> + # Deb caching is disabled >>> + d.delVarFlag('do_install_to_deploy', 'noexec') >>> return True >>> PN = d.getVar("PN", True) >>> @@ -108,6 +115,7 @@ python __anonymous () { >>> path_cache = os.path.join(DEBCACHEDIR, DISTRO) >>> path_databases = os.path.join(DEBDBDIR, DISTRO) >>> path_distributions = os.path.join(path_cache, "conf", >>> "distributions") >>> + d.delVarFlag('do_install_to_cache', 'noexec') >>> # The distributions file is needed by `reprepro` to know what >>> types >>> # of packages are supported, what the distribution name is, etc. >>> diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass >>> index c2ff453..6b1b5eb 100644 >>> --- a/meta/classes/image.bbclass >>> +++ b/meta/classes/image.bbclass >>> @@ -46,4 +46,4 @@ do_populate() { >>> } >>> addtask populate before do_build >>> -do_populate[deptask] = "do_install" >>> +do_populate[deptask] = "do_install_to_cache do_install_to_deploy" >> >> I would rather still have a, maybe empty, 'do_install' step that just >> depends on the 'do_install_to_cache' and 'do_install_to_deploy' tasks. >> I don't see why ever small change in one class should change the >> complete pipeline for every recipe. > > It's a bit difficult question. In general meaning when building > something from sources, the install means a storing of build artifacts > to some specific location. So in this case, we just copy binary deb > package to repo, and there is nothing common with installation of > artifacts. > > Also Isar has in roadmap cross-compilation, so the pipeline will > probably look like: > > fetch - unpack - compile - install - install_to_cache > > so in this case the install task probably should go before > install_to_cache. > > So probably it makes sense to avoid _install_ in this task, for example > use do_populate_deb_cache, what do you think? Something like: 'populate_repo' sounds much better for this case. I would try to have more abstract names at first and implement them with more specific names like 'populate_repo_deb'. In OE there already is a 'package' and additional sub tasks like 'package_write_deb'. So a general pipeline for normal recipes like this: fetch, unpack, configure, compile, install, package, populate_repo Of course image recipes would have a slightly changed pipeline. I think the discussion about the build pipeline in isar is very important. Maybe we should make an extra thread about this. Cheers, Claudius -- DENX Software Engineering GmbH, Managing Director: Wolfgang Denk HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de ^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [PATCH 3/6] classes/dpkg: Split install for cache 2017-08-27 15:13 ` [PATCH 3/6] classes/dpkg: Split install for cache Alexander Smirnov 2017-08-28 8:00 ` Claudius Heine @ 2017-08-28 15:30 ` Henning Schild 1 sibling, 0 replies; 30+ messages in thread From: Henning Schild @ 2017-08-28 15:30 UTC (permalink / raw) To: Alexander Smirnov; +Cc: isar-users This is a fixup of 1/6. No need for another patch, just fix 1of6. Henning Am Sun, 27 Aug 2017 18:13:36 +0300 schrieb Alexander Smirnov <asmirnov@ilbers.de>: > Split install function into two parts: > - install_to_cache: if caching is enabled > - install_to_deploy: if caching is disabled > > This patch brings flexibility to the implementation and makes it > possible to move all the caching implementation code to dedicated > class. The magic behavior depending on the value of DEBCACHE_ENABLED > is now transparent in the bitbake pipeline (can be inspected via > "bitbake -g"). > > Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de> > --- > meta/classes/dpkg.bbclass | 66 > ++++++++++++++++++++++++++-------------------- > meta/classes/image.bbclass | 2 +- 2 files changed, 38 insertions(+), > 30 deletions(-) > > diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass > index b1e201d..118ba2f 100644 > --- a/meta/classes/dpkg.bbclass > +++ b/meta/classes/dpkg.bbclass > @@ -56,46 +56,53 @@ do_build() { > } > > # Install package to dedicated deploy directory > -do_binary_deb_install() { > +do_install_to_deploy() { > + # Deb caching is disabled, simply copy all binary packages to > the deploy > + # directory > + install -m 644 "${BUILDROOT}"/*.deb "${DEPLOY_DIR_DEB}/" > +} > + > +addtask install_to_deploy after do_build > +do_install_to_deploy[dirs] = "${DEPLOY_DIR_DEB}" > +do_install_to_deploy[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}" > +do_install_to_deploy[noexec] = "1" > + > +# Install package to dedicated apt repo > +do_install_to_cache() { > readonly DIR_CACHE="${DEBCACHEDIR}/${DISTRO}" > readonly DIR_DB="${DEBDBDIR}/${DISTRO}" > > - if [ "${DEBCACHE_ENABLED}" != "0" ]; then > - # If `bitbake` is running for the first time, the cache > doesn't exist > - # yet and needs to be configured using a `distributions` > file. > - # A template stored in the layer directory is pre-processed > to > - # generate the configuration file, which is then placed in > the > - # appropriate directory. > - if [ ! -e "${DIR_CACHE}/conf/distributions" ]; then > - mkdir -p "${DIR_CACHE}/conf" > - sed -e "s#{DISTRO_NAME}#${DEBDISTRONAME}#g" \ > - "${DEBFILESDIR}/distributions.in" \ > - > "${DIR_CACHE}/conf/distributions" > - fi > - > - # Add binary and source packages to the deb cache > - # If the cache doesn't exist yet, it will be created using > the > - # `distributions` file generated above. > - ls -1 "${BUILDROOT}"/*.deb "${BUILDROOT}"/*.dsc | while read > -r p; do > - reprepro --waitforlock 3 -b "${DIR_CACHE}" --dbdir > "${DIR_DB}" \ > - -C main "include${p##*.}" "${DEBDISTRONAME}" "${p}" > - done > - else > - # Deb caching is disabled, simply copy all binary packages > to the > - # deploy directory > - mkdir -p "${DEPLOY_DIR_DEB}" > - install -m 644 "${BUILDROOT}"/*.deb "${DEPLOY_DIR_DEB}/" > + # If `bitbake` is running for the first time, the cache doesn't > exist > + # yet and needs to be configured using a `distributions` file. > + # A template stored in the layer directory is pre-processed to > + # generate the configuration file, which is then placed in the > + # appropriate directory. > + if [ ! -e "${DIR_CACHE}/conf/distributions" ]; then > + mkdir -p "${DIR_CACHE}/conf" > + sed -e "s#{DISTRO_NAME}#${DEBDISTRONAME}#g" \ > + "${DEBFILESDIR}/distributions.in" \ > + > "${DIR_CACHE}/conf/distributions" > fi > + > + # Add binary and source packages to the deb cache > + # If the cache doesn't exist yet, it will be created using the > + # `distributions` file generated above. > + ls -1 "${BUILDROOT}"/*.deb "${BUILDROOT}"/*.dsc | while read -r > p; do > + reprepro --waitforlock 3 -b "${DIR_CACHE}" --dbdir > "${DIR_DB}" \ > + -C main "include${p##*.}" "${DEBDISTRONAME}" "${p}" > + done > } > > -addtask binary_deb_install after do_build > -do_binary_deb_install[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}" > +addtask install_to_cache after do_build > +do_install_to_cache[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}" > +do_install_to_cache[noexec] = "1" > > # Deb caching lambda run during the parsing phase that checks > whether the # current package has to be rebuilt, or taken from the > cache python __anonymous () { > if d.getVar("DEBCACHE_ENABLED", True) == "0": > - # Deb caching is disabled, do nothing > + # Deb caching is disabled > + d.delVarFlag('do_install_to_deploy', 'noexec') > return True > > PN = d.getVar("PN", True) > @@ -108,6 +115,7 @@ python __anonymous () { > path_cache = os.path.join(DEBCACHEDIR, DISTRO) > path_databases = os.path.join(DEBDBDIR, DISTRO) > path_distributions = os.path.join(path_cache, "conf", > "distributions") > + d.delVarFlag('do_install_to_cache', 'noexec') > > # The distributions file is needed by `reprepro` to know what > types # of packages are supported, what the distribution name is, etc. > diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass > index c2ff453..6b1b5eb 100644 > --- a/meta/classes/image.bbclass > +++ b/meta/classes/image.bbclass > @@ -46,4 +46,4 @@ do_populate() { > } > > addtask populate before do_build > -do_populate[deptask] = "do_install" > +do_populate[deptask] = "do_install_to_cache do_install_to_deploy" ^ permalink raw reply [flat|nested] 30+ messages in thread
* [PATCH 4/6] meta-isar-bin: Enable apt repo generation for amd64 2017-08-27 15:13 [PATCH 0/6] Isar apt cache implementation Alexander Smirnov ` (2 preceding siblings ...) 2017-08-27 15:13 ` [PATCH 3/6] classes/dpkg: Split install for cache Alexander Smirnov @ 2017-08-27 15:13 ` Alexander Smirnov 2017-08-27 15:13 ` [PATCH 5/6] classes/dpkg: Properly update packages in the cache Alexander Smirnov 2017-08-27 15:13 ` [PATCH 6/6] doc/technical_overview: Describe binary cache Alexander Smirnov 5 siblings, 0 replies; 30+ messages in thread From: Alexander Smirnov @ 2017-08-27 15:13 UTC (permalink / raw) To: isar-users; +Cc: Frank Lenormand, Alexander Smirnov From: Frank Lenormand <lenormf@gmail.com> Add amd64 architecture to binary caching. Signed-off-by: Frank Lenormand <lenormf@gmail.com> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de> --- meta-isar-bin/files/distributions.in | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/meta-isar-bin/files/distributions.in b/meta-isar-bin/files/distributions.in index 59f4429..cd214c6 100644 --- a/meta-isar-bin/files/distributions.in +++ b/meta-isar-bin/files/distributions.in @@ -1,3 +1,3 @@ Codename: {DISTRO_NAME} -Architectures: i386 armhf source +Architectures: i386 armhf amd64 source Components: main -- 2.1.4 ^ permalink raw reply [flat|nested] 30+ messages in thread
* [PATCH 5/6] classes/dpkg: Properly update packages in the cache 2017-08-27 15:13 [PATCH 0/6] Isar apt cache implementation Alexander Smirnov ` (3 preceding siblings ...) 2017-08-27 15:13 ` [PATCH 4/6] meta-isar-bin: Enable apt repo generation for amd64 Alexander Smirnov @ 2017-08-27 15:13 ` Alexander Smirnov 2017-08-28 15:32 ` Henning Schild 2017-08-27 15:13 ` [PATCH 6/6] doc/technical_overview: Describe binary cache Alexander Smirnov 5 siblings, 1 reply; 30+ messages in thread From: Alexander Smirnov @ 2017-08-27 15:13 UTC (permalink / raw) To: isar-users; +Cc: Frank Lenormand, Alexander Smirnov From: Frank Lenormand <lenormf@gmail.com> Before installing to cache, existing packages with the same name and version have to be removed. Signed-off-by: Frank Lenormand <lenormf@gmail.com> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de> --- meta/classes/dpkg.bbclass | 46 +++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 43 insertions(+), 3 deletions(-) diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass index 118ba2f..72ae79f 100644 --- a/meta/classes/dpkg.bbclass +++ b/meta/classes/dpkg.bbclass @@ -84,12 +84,52 @@ do_install_to_cache() { > "${DIR_CACHE}/conf/distributions" fi + print_field_value() { + awk "\$1 == \"${1}:\" { print \$2; }" + } + + call_reprepro() { + reprepro --waitforlock 3 -b "${DIR_CACHE}" --dbdir "${DIR_DB}" \ + -C main "$@" + } + # Add binary and source packages to the deb cache # If the cache doesn't exist yet, it will be created using the # `distributions` file generated above. - ls -1 "${BUILDROOT}"/*.deb "${BUILDROOT}"/*.dsc | while read -r p; do - reprepro --waitforlock 3 -b "${DIR_CACHE}" --dbdir "${DIR_DB}" \ - -C main "include${p##*.}" "${DEBDISTRONAME}" "${p}" + ls -1 "${BUILDROOT}"/*.deb | while read -r p; do + name_package=$(dpkg -f "${p}" | print_field_value "Package") + version_package=$(dpkg -f "${p}" | print_field_value "Version") + + # Remove all packages with the same version that were added to the + # repository in previous builds + call_reprepro \ + -A "${DISTRO_ARCH}" \ + removefilter "${DEBDISTRONAME}" \ + "Package (== ${name_package}), \ + Version (== ${version_package})" + call_reprepro \ + -A "${DISTRO_ARCH}" \ + "include${p##*.}" \ + "${DEBDISTRONAME}" \ + "${p}" + done + + ls -1 "${BUILDROOT}"/*.dsc | while read -r p; do + name_package=$(cat "${p}" | print_field_value "Source") + version_package=$(cat "${p}" | print_field_value "Version") + + # Remove all source packages with the same version that were added to + # the repository in previous builds + call_reprepro \ + -A "source" \ + removefilter "${DEBDISTRONAME}" \ + "Package (== ${name_package}), \ + Version (== ${version_package})" + call_reprepro \ + -A "source" \ + "include${p##*.}" \ + "${DEBDISTRONAME}" \ + "${p}" done } -- 2.1.4 ^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [PATCH 5/6] classes/dpkg: Properly update packages in the cache 2017-08-27 15:13 ` [PATCH 5/6] classes/dpkg: Properly update packages in the cache Alexander Smirnov @ 2017-08-28 15:32 ` Henning Schild 2017-08-29 7:20 ` Alexander Smirnov 0 siblings, 1 reply; 30+ messages in thread From: Henning Schild @ 2017-08-28 15:32 UTC (permalink / raw) To: Alexander Smirnov; +Cc: isar-users, Frank Lenormand This is another fixup of 1/6, fold it in. Henning Am Sun, 27 Aug 2017 18:13:38 +0300 schrieb Alexander Smirnov <asmirnov@ilbers.de>: > From: Frank Lenormand <lenormf@gmail.com> > > Before installing to cache, existing packages with the same name and > version have to be removed. > > Signed-off-by: Frank Lenormand <lenormf@gmail.com> > Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de> > --- > meta/classes/dpkg.bbclass | 46 > +++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 43 > insertions(+), 3 deletions(-) > > diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass > index 118ba2f..72ae79f 100644 > --- a/meta/classes/dpkg.bbclass > +++ b/meta/classes/dpkg.bbclass > @@ -84,12 +84,52 @@ do_install_to_cache() { > > "${DIR_CACHE}/conf/distributions" > fi > > + print_field_value() { > + awk "\$1 == \"${1}:\" { print \$2; }" > + } > + > + call_reprepro() { > + reprepro --waitforlock 3 -b "${DIR_CACHE}" --dbdir > "${DIR_DB}" \ > + -C main "$@" > + } > + > # Add binary and source packages to the deb cache > # If the cache doesn't exist yet, it will be created using the > # `distributions` file generated above. > - ls -1 "${BUILDROOT}"/*.deb "${BUILDROOT}"/*.dsc | while read -r > p; do > - reprepro --waitforlock 3 -b "${DIR_CACHE}" --dbdir > "${DIR_DB}" \ > - -C main "include${p##*.}" "${DEBDISTRONAME}" "${p}" > + ls -1 "${BUILDROOT}"/*.deb | while read -r p; do > + name_package=$(dpkg -f "${p}" | print_field_value "Package") > + version_package=$(dpkg -f "${p}" | print_field_value > "Version") + > + # Remove all packages with the same version that were added > to the > + # repository in previous builds > + call_reprepro \ > + -A "${DISTRO_ARCH}" \ > + removefilter "${DEBDISTRONAME}" \ > + "Package (== ${name_package}), \ > + Version (== ${version_package})" > + call_reprepro \ > + -A "${DISTRO_ARCH}" \ > + "include${p##*.}" \ > + "${DEBDISTRONAME}" \ > + "${p}" > + done > + > + ls -1 "${BUILDROOT}"/*.dsc | while read -r p; do > + name_package=$(cat "${p}" | print_field_value "Source") > + version_package=$(cat "${p}" | print_field_value "Version") > + > + # Remove all source packages with the same version that were > added to > + # the repository in previous builds > + call_reprepro \ > + -A "source" \ > + removefilter "${DEBDISTRONAME}" \ > + "Package (== ${name_package}), \ > + Version (== ${version_package})" > + call_reprepro \ > + -A "source" \ > + "include${p##*.}" \ > + "${DEBDISTRONAME}" \ > + "${p}" > done > } > ^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [PATCH 5/6] classes/dpkg: Properly update packages in the cache 2017-08-28 15:32 ` Henning Schild @ 2017-08-29 7:20 ` Alexander Smirnov 2017-08-29 7:57 ` Henning Schild 0 siblings, 1 reply; 30+ messages in thread From: Alexander Smirnov @ 2017-08-29 7:20 UTC (permalink / raw) To: Henning Schild; +Cc: isar-users > This is another fixup of 1/6, fold it in. > It's a good question if we want to keep development history. Because these patches reflect the real history of binary caching development. > > Am Sun, 27 Aug 2017 18:13:38 +0300 > schrieb Alexander Smirnov <asmirnov@ilbers.de>: > >> From: Frank Lenormand <lenormf@gmail.com> >> >> Before installing to cache, existing packages with the same name and >> version have to be removed. >> >> Signed-off-by: Frank Lenormand <lenormf@gmail.com> >> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de> >> --- >> meta/classes/dpkg.bbclass | 46 >> +++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 43 >> insertions(+), 3 deletions(-) >> >> diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass >> index 118ba2f..72ae79f 100644 >> --- a/meta/classes/dpkg.bbclass >> +++ b/meta/classes/dpkg.bbclass >> @@ -84,12 +84,52 @@ do_install_to_cache() { >> > "${DIR_CACHE}/conf/distributions" >> fi >> >> + print_field_value() { >> + awk "\$1 == \"${1}:\" { print \$2; }" >> + } >> + >> + call_reprepro() { >> + reprepro --waitforlock 3 -b "${DIR_CACHE}" --dbdir >> "${DIR_DB}" \ >> + -C main "$@" >> + } >> + >> # Add binary and source packages to the deb cache >> # If the cache doesn't exist yet, it will be created using the >> # `distributions` file generated above. >> - ls -1 "${BUILDROOT}"/*.deb "${BUILDROOT}"/*.dsc | while read -r >> p; do >> - reprepro --waitforlock 3 -b "${DIR_CACHE}" --dbdir >> "${DIR_DB}" \ >> - -C main "include${p##*.}" "${DEBDISTRONAME}" "${p}" >> + ls -1 "${BUILDROOT}"/*.deb | while read -r p; do >> + name_package=$(dpkg -f "${p}" | print_field_value "Package") >> + version_package=$(dpkg -f "${p}" | print_field_value >> "Version") + >> + # Remove all packages with the same version that were added >> to the >> + # repository in previous builds >> + call_reprepro \ >> + -A "${DISTRO_ARCH}" \ >> + removefilter "${DEBDISTRONAME}" \ >> + "Package (== ${name_package}), \ >> + Version (== ${version_package})" >> + call_reprepro \ >> + -A "${DISTRO_ARCH}" \ >> + "include${p##*.}" \ >> + "${DEBDISTRONAME}" \ >> + "${p}" >> + done >> + >> + ls -1 "${BUILDROOT}"/*.dsc | while read -r p; do >> + name_package=$(cat "${p}" | print_field_value "Source") >> + version_package=$(cat "${p}" | print_field_value "Version") >> + >> + # Remove all source packages with the same version that were >> added to >> + # the repository in previous builds >> + call_reprepro \ >> + -A "source" \ >> + removefilter "${DEBDISTRONAME}" \ >> + "Package (== ${name_package}), \ >> + Version (== ${version_package})" >> + call_reprepro \ >> + -A "source" \ >> + "include${p##*.}" \ >> + "${DEBDISTRONAME}" \ >> + "${p}" >> done >> } >> > ^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [PATCH 5/6] classes/dpkg: Properly update packages in the cache 2017-08-29 7:20 ` Alexander Smirnov @ 2017-08-29 7:57 ` Henning Schild 2017-08-29 11:26 ` Jan Kiszka 0 siblings, 1 reply; 30+ messages in thread From: Henning Schild @ 2017-08-29 7:57 UTC (permalink / raw) To: Alexander Smirnov; +Cc: isar-users Am Tue, 29 Aug 2017 10:20:41 +0300 schrieb Alexander Smirnov <asmirnov@ilbers.de>: > > This is another fixup of 1/6, fold it in. > > > > It's a good question if we want to keep development history. Because > these patches reflect the real history of binary caching development. To me the answer is clear. If we find issues in a patch during review, the patch should be fixed before getting merged. If we do not follow that policy you might as well merge all patches without review. Henning > > > > Am Sun, 27 Aug 2017 18:13:38 +0300 > > schrieb Alexander Smirnov <asmirnov@ilbers.de>: > > > >> From: Frank Lenormand <lenormf@gmail.com> > >> > >> Before installing to cache, existing packages with the same name > >> and version have to be removed. > >> > >> Signed-off-by: Frank Lenormand <lenormf@gmail.com> > >> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de> > >> --- > >> meta/classes/dpkg.bbclass | 46 > >> +++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 43 > >> insertions(+), 3 deletions(-) > >> > >> diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass > >> index 118ba2f..72ae79f 100644 > >> --- a/meta/classes/dpkg.bbclass > >> +++ b/meta/classes/dpkg.bbclass > >> @@ -84,12 +84,52 @@ do_install_to_cache() { > >> > "${DIR_CACHE}/conf/distributions" > >> fi > >> > >> + print_field_value() { > >> + awk "\$1 == \"${1}:\" { print \$2; }" > >> + } > >> + > >> + call_reprepro() { > >> + reprepro --waitforlock 3 -b "${DIR_CACHE}" --dbdir > >> "${DIR_DB}" \ > >> + -C main "$@" > >> + } > >> + > >> # Add binary and source packages to the deb cache > >> # If the cache doesn't exist yet, it will be created using > >> the # `distributions` file generated above. > >> - ls -1 "${BUILDROOT}"/*.deb "${BUILDROOT}"/*.dsc | while read > >> -r p; do > >> - reprepro --waitforlock 3 -b "${DIR_CACHE}" --dbdir > >> "${DIR_DB}" \ > >> - -C main "include${p##*.}" "${DEBDISTRONAME}" "${p}" > >> + ls -1 "${BUILDROOT}"/*.deb | while read -r p; do > >> + name_package=$(dpkg -f "${p}" | print_field_value > >> "Package") > >> + version_package=$(dpkg -f "${p}" | print_field_value > >> "Version") + > >> + # Remove all packages with the same version that were > >> added to the > >> + # repository in previous builds > >> + call_reprepro \ > >> + -A "${DISTRO_ARCH}" \ > >> + removefilter "${DEBDISTRONAME}" \ > >> + "Package (== ${name_package}), \ > >> + Version (== ${version_package})" > >> + call_reprepro \ > >> + -A "${DISTRO_ARCH}" \ > >> + "include${p##*.}" \ > >> + "${DEBDISTRONAME}" \ > >> + "${p}" > >> + done > >> + > >> + ls -1 "${BUILDROOT}"/*.dsc | while read -r p; do > >> + name_package=$(cat "${p}" | print_field_value "Source") > >> + version_package=$(cat "${p}" | print_field_value > >> "Version") + > >> + # Remove all source packages with the same version that > >> were added to > >> + # the repository in previous builds > >> + call_reprepro \ > >> + -A "source" \ > >> + removefilter "${DEBDISTRONAME}" \ > >> + "Package (== ${name_package}), \ > >> + Version (== ${version_package})" > >> + call_reprepro \ > >> + -A "source" \ > >> + "include${p##*.}" \ > >> + "${DEBDISTRONAME}" \ > >> + "${p}" > >> done > >> } > >> > > ^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [PATCH 5/6] classes/dpkg: Properly update packages in the cache 2017-08-29 7:57 ` Henning Schild @ 2017-08-29 11:26 ` Jan Kiszka 0 siblings, 0 replies; 30+ messages in thread From: Jan Kiszka @ 2017-08-29 11:26 UTC (permalink / raw) To: [ext] Henning Schild, Alexander Smirnov; +Cc: isar-users On 2017-08-29 09:57, [ext] Henning Schild wrote: > Am Tue, 29 Aug 2017 10:20:41 +0300 > schrieb Alexander Smirnov <asmirnov@ilbers.de>: > >>> This is another fixup of 1/6, fold it in. >>> >> >> It's a good question if we want to keep development history. Because >> these patches reflect the real history of binary caching development. > > To me the answer is clear. If we find issues in a patch during review, > the patch should be fixed before getting merged. > If we do not follow that policy you might as well merge all patches > without review. ACK. This is how many many other projects work as well. Jan -- Siemens AG, Corporate Technology, CT RDA ITP SES-DE Corporate Competence Center Embedded Linux ^ permalink raw reply [flat|nested] 30+ messages in thread
* [PATCH 6/6] doc/technical_overview: Describe binary cache 2017-08-27 15:13 [PATCH 0/6] Isar apt cache implementation Alexander Smirnov ` (4 preceding siblings ...) 2017-08-27 15:13 ` [PATCH 5/6] classes/dpkg: Properly update packages in the cache Alexander Smirnov @ 2017-08-27 15:13 ` Alexander Smirnov 2017-08-28 15:36 ` Henning Schild 5 siblings, 1 reply; 30+ messages in thread From: Alexander Smirnov @ 2017-08-27 15:13 UTC (permalink / raw) To: isar-users; +Cc: Alexander Smirnov Describe Isar binary cache feature. Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de> --- doc/technical_overview.md | 59 +++++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 55 insertions(+), 4 deletions(-) diff --git a/doc/technical_overview.md b/doc/technical_overview.md index 20e08b4..b3265dc 100644 --- a/doc/technical_overview.md +++ b/doc/technical_overview.md @@ -100,6 +100,43 @@ Target filesystem lifecycle can be described as following: - According to the list of custom packages in bitbake recipes, the initial filesystem will be populated by successfully built packages. +## 2.5 Binary Cache + +As mentioned above, apart from fetching binary packages from upstream apt +repositories, Isar is able to build custom packages from sources. To simplify +further custom packages management, Isar implements binary caching. + +Binary cache is a typical apt repository created by Isar to store newly built +packages. This repository has mainly two goals: + + - Simplify image filesystem populating and distributing binary packages to + other systems. + + - Reuse these binaries in next builds to avoid re-building from the + scratch everytime. + +Current Isar tree provides default cache repository: `meta-isar-bin`. +Isar binary cache is described by the following files: + + - `meta-isar-bin/conf/layer.conf` + + - `meta-isar-bin/files/distributions.in` + +After system build is complete, this repo contains binary packages for all the +distros and architectures, requested in multiconfig. So this folder can be +published and used as apt repo for already installed systems. + +An important note, that Isar creates local repositories using dedicated Debian +tool reprepro. This tool takes care about: + + - Maintenance of multiple architectures and distros. + + - Internal packages database management. + + - Repo signatures management. + + - Requests about available packages in repo (name, version etc...). + # 3 Isar Internal Processes ## 3.1 General Overview @@ -174,10 +211,17 @@ contain debian folder. The build process is implemented in 3. Run dpkg-buildpackage -4. Task `do_install`: install successfully built packages +4. Further package processing depends on binary cache option: + + 1. Task `do_install_to_deploy`: install successfully built packages if + binary cache is disabled: `${BUILDCHROOT_DIR}/home/build/${PN}/*.deb` to deploy directory `${DEPLOY_DIR_DEB}` + 2. Task `do_install_to_cache`: install successfully built packages to + binary cache repository: + `${BUILDCHROOT_DIR}/home/build/${PN}/*.deb` to `${DIR_CACHE}` + ## 3.5 Populate Target Filesystem Each target image can be extended by custom packages listed in IMAGE_INSTALL @@ -185,11 +229,18 @@ variable. Task `do_populate` performs the following: 1. Parse IMAGE_INSTALL variable. -2. Find respective packages in `${DEPLOY_DIR_DEB}`. +2. Further steps depends in binary cache option. +If binary cache is disabled: + + 1. Find respective packages in `${DEPLOY_DIR_DEB}`. + + 2. Copy packages to deb folder in dedicated target filesystem. + + 3. Execute dpkg command in chroot for all the copied packages. -3. Copy them to deb folder in dedicated target filesystem. +If binary cache is enabled: -4. Execute dpkg command in chroot for all the copied packages. + 1. Call apt-get install in chroot in target image filesystem. ## 3.6 Generate Bootable Image -- 2.1.4 ^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [PATCH 6/6] doc/technical_overview: Describe binary cache 2017-08-27 15:13 ` [PATCH 6/6] doc/technical_overview: Describe binary cache Alexander Smirnov @ 2017-08-28 15:36 ` Henning Schild 2017-08-29 7:29 ` Alexander Smirnov 2017-08-29 11:29 ` Jan Kiszka 0 siblings, 2 replies; 30+ messages in thread From: Henning Schild @ 2017-08-28 15:36 UTC (permalink / raw) To: Alexander Smirnov; +Cc: isar-users This should be done along the way in every single patch, not afterwards. Henning Am Sun, 27 Aug 2017 18:13:39 +0300 schrieb Alexander Smirnov <asmirnov@ilbers.de>: > Describe Isar binary cache feature. > > Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de> > --- > doc/technical_overview.md | 59 > +++++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 55 > insertions(+), 4 deletions(-) > > diff --git a/doc/technical_overview.md b/doc/technical_overview.md > index 20e08b4..b3265dc 100644 > --- a/doc/technical_overview.md > +++ b/doc/technical_overview.md > @@ -100,6 +100,43 @@ Target filesystem lifecycle can be described as > following: > - According to the list of custom packages in bitbake recipes, the > initial filesystem will be populated by successfully built packages. > > +## 2.5 Binary Cache > + > +As mentioned above, apart from fetching binary packages from > upstream apt +repositories, Isar is able to build custom packages > from sources. To simplify +further custom packages management, Isar > implements binary caching. + > +Binary cache is a typical apt repository created by Isar to store > newly built +packages. This repository has mainly two goals: > + > + - Simplify image filesystem populating and distributing binary > packages to > + other systems. > + > + - Reuse these binaries in next builds to avoid re-building from the > + scratch everytime. > + > +Current Isar tree provides default cache repository: `meta-isar-bin`. > +Isar binary cache is described by the following files: > + > + - `meta-isar-bin/conf/layer.conf` > + > + - `meta-isar-bin/files/distributions.in` > + > +After system build is complete, this repo contains binary packages > for all the +distros and architectures, requested in multiconfig. So > this folder can be +published and used as apt repo for already > installed systems. + > +An important note, that Isar creates local repositories using > dedicated Debian +tool reprepro. This tool takes care about: > + > + - Maintenance of multiple architectures and distros. > + > + - Internal packages database management. > + > + - Repo signatures management. > + > + - Requests about available packages in repo (name, version etc...). > + > # 3 Isar Internal Processes > > ## 3.1 General Overview > @@ -174,10 +211,17 @@ contain debian folder. The build process is > implemented in > 3. Run dpkg-buildpackage > > -4. Task `do_install`: install successfully built packages > +4. Further package processing depends on binary cache option: > + > + 1. Task `do_install_to_deploy`: install successfully built > packages if > + binary cache is disabled: > `${BUILDCHROOT_DIR}/home/build/${PN}/*.deb` to deploy directory > `${DEPLOY_DIR_DEB}` > > + 2. Task `do_install_to_cache`: install successfully built > packages to > + binary cache repository: > + `${BUILDCHROOT_DIR}/home/build/${PN}/*.deb` to `${DIR_CACHE}` > + > ## 3.5 Populate Target Filesystem > > Each target image can be extended by custom packages listed in > IMAGE_INSTALL @@ -185,11 +229,18 @@ variable. Task `do_populate` > performs the following: > 1. Parse IMAGE_INSTALL variable. > > -2. Find respective packages in `${DEPLOY_DIR_DEB}`. > +2. Further steps depends in binary cache option. > +If binary cache is disabled: > + > + 1. Find respective packages in `${DEPLOY_DIR_DEB}`. > + > + 2. Copy packages to deb folder in dedicated target filesystem. > + > + 3. Execute dpkg command in chroot for all the copied packages. > > -3. Copy them to deb folder in dedicated target filesystem. > +If binary cache is enabled: > > -4. Execute dpkg command in chroot for all the copied packages. > + 1. Call apt-get install in chroot in target image filesystem. > > ## 3.6 Generate Bootable Image > ^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [PATCH 6/6] doc/technical_overview: Describe binary cache 2017-08-28 15:36 ` Henning Schild @ 2017-08-29 7:29 ` Alexander Smirnov 2017-08-29 8:06 ` Henning Schild 2017-08-29 11:29 ` Jan Kiszka 1 sibling, 1 reply; 30+ messages in thread From: Alexander Smirnov @ 2017-08-29 7:29 UTC (permalink / raw) To: Henning Schild; +Cc: isar-users > This should be done along the way in every single patch, not afterwards. I don't think that it really should. The requirement to documentation is to describe current code state in repo. So if something has been changed in design - HEAD of branch should contain relevant documentation, what could be achieved by dedicate patch. Moreover this single patch explicitly describes the influence of whole series to current design. > > Henning > > Am Sun, 27 Aug 2017 18:13:39 +0300 > schrieb Alexander Smirnov <asmirnov@ilbers.de>: > >> Describe Isar binary cache feature. >> >> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de> >> --- >> doc/technical_overview.md | 59 >> +++++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 55 >> insertions(+), 4 deletions(-) >> >> diff --git a/doc/technical_overview.md b/doc/technical_overview.md >> index 20e08b4..b3265dc 100644 >> --- a/doc/technical_overview.md >> +++ b/doc/technical_overview.md >> @@ -100,6 +100,43 @@ Target filesystem lifecycle can be described as >> following: >> - According to the list of custom packages in bitbake recipes, the >> initial filesystem will be populated by successfully built packages. >> >> +## 2.5 Binary Cache >> + >> +As mentioned above, apart from fetching binary packages from >> upstream apt +repositories, Isar is able to build custom packages >> from sources. To simplify +further custom packages management, Isar >> implements binary caching. + >> +Binary cache is a typical apt repository created by Isar to store >> newly built +packages. This repository has mainly two goals: >> + >> + - Simplify image filesystem populating and distributing binary >> packages to >> + other systems. >> + >> + - Reuse these binaries in next builds to avoid re-building from the >> + scratch everytime. >> + >> +Current Isar tree provides default cache repository: `meta-isar-bin`. >> +Isar binary cache is described by the following files: >> + >> + - `meta-isar-bin/conf/layer.conf` >> + >> + - `meta-isar-bin/files/distributions.in` >> + >> +After system build is complete, this repo contains binary packages >> for all the +distros and architectures, requested in multiconfig. So >> this folder can be +published and used as apt repo for already >> installed systems. + >> +An important note, that Isar creates local repositories using >> dedicated Debian +tool reprepro. This tool takes care about: >> + >> + - Maintenance of multiple architectures and distros. >> + >> + - Internal packages database management. >> + >> + - Repo signatures management. >> + >> + - Requests about available packages in repo (name, version etc...). >> + >> # 3 Isar Internal Processes >> >> ## 3.1 General Overview >> @@ -174,10 +211,17 @@ contain debian folder. The build process is >> implemented in >> 3. Run dpkg-buildpackage >> >> -4. Task `do_install`: install successfully built packages >> +4. Further package processing depends on binary cache option: >> + >> + 1. Task `do_install_to_deploy`: install successfully built >> packages if >> + binary cache is disabled: >> `${BUILDCHROOT_DIR}/home/build/${PN}/*.deb` to deploy directory >> `${DEPLOY_DIR_DEB}` >> >> + 2. Task `do_install_to_cache`: install successfully built >> packages to >> + binary cache repository: >> + `${BUILDCHROOT_DIR}/home/build/${PN}/*.deb` to `${DIR_CACHE}` >> + >> ## 3.5 Populate Target Filesystem >> >> Each target image can be extended by custom packages listed in >> IMAGE_INSTALL @@ -185,11 +229,18 @@ variable. Task `do_populate` >> performs the following: >> 1. Parse IMAGE_INSTALL variable. >> >> -2. Find respective packages in `${DEPLOY_DIR_DEB}`. >> +2. Further steps depends in binary cache option. >> +If binary cache is disabled: >> + >> + 1. Find respective packages in `${DEPLOY_DIR_DEB}`. >> + >> + 2. Copy packages to deb folder in dedicated target filesystem. >> + >> + 3. Execute dpkg command in chroot for all the copied packages. >> >> -3. Copy them to deb folder in dedicated target filesystem. >> +If binary cache is enabled: >> >> -4. Execute dpkg command in chroot for all the copied packages. >> + 1. Call apt-get install in chroot in target image filesystem. >> >> ## 3.6 Generate Bootable Image >> > ^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [PATCH 6/6] doc/technical_overview: Describe binary cache 2017-08-29 7:29 ` Alexander Smirnov @ 2017-08-29 8:06 ` Henning Schild 0 siblings, 0 replies; 30+ messages in thread From: Henning Schild @ 2017-08-29 8:06 UTC (permalink / raw) To: Alexander Smirnov; +Cc: isar-users Am Tue, 29 Aug 2017 10:29:15 +0300 schrieb Alexander Smirnov <asmirnov@ilbers.de>: > > This should be done along the way in every single patch, not > > afterwards. > > I don't think that it really should. The requirement to documentation > is to describe current code state in repo. So if something has been > changed in design - HEAD of branch should contain relevant > documentation, what could be achieved by dedicate patch. Moreover > this single patch explicitly describes the influence of whole series > to current design. Every commit should leave the whole repo in a consistent state. If you change something and change the doc in a later commit you have inconsistent intermediate states. I already explained that to you in detail when we talked about the order in which the document should appear in the repo. You can argue that humans can deal with such inconsistencies, true. Say there was a change like that, in a language that a machine reads: + mount /foo + umount /foo Would you want those two lines in one or in two patches? If two are OK, should they be merge directly after another or can we put 10 more between them? I think the answer is clear, since you want CI to succeed on every commit. So you want every commit to be consistent in terms of programming language. And i am saying we should do the same for natural/human language. Henning > > > > Henning > > > > Am Sun, 27 Aug 2017 18:13:39 +0300 > > schrieb Alexander Smirnov <asmirnov@ilbers.de>: > > > >> Describe Isar binary cache feature. > >> > >> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de> > >> --- > >> doc/technical_overview.md | 59 > >> +++++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 55 > >> insertions(+), 4 deletions(-) > >> > >> diff --git a/doc/technical_overview.md b/doc/technical_overview.md > >> index 20e08b4..b3265dc 100644 > >> --- a/doc/technical_overview.md > >> +++ b/doc/technical_overview.md > >> @@ -100,6 +100,43 @@ Target filesystem lifecycle can be described > >> as following: > >> - According to the list of custom packages in bitbake recipes, > >> the initial filesystem will be populated by successfully built > >> packages. > >> +## 2.5 Binary Cache > >> + > >> +As mentioned above, apart from fetching binary packages from > >> upstream apt +repositories, Isar is able to build custom packages > >> from sources. To simplify +further custom packages management, Isar > >> implements binary caching. + > >> +Binary cache is a typical apt repository created by Isar to store > >> newly built +packages. This repository has mainly two goals: > >> + > >> + - Simplify image filesystem populating and distributing binary > >> packages to > >> + other systems. > >> + > >> + - Reuse these binaries in next builds to avoid re-building from > >> the > >> + scratch everytime. > >> + > >> +Current Isar tree provides default cache repository: > >> `meta-isar-bin`. +Isar binary cache is described by the following > >> files: + > >> + - `meta-isar-bin/conf/layer.conf` > >> + > >> + - `meta-isar-bin/files/distributions.in` > >> + > >> +After system build is complete, this repo contains binary packages > >> for all the +distros and architectures, requested in multiconfig. > >> So this folder can be +published and used as apt repo for already > >> installed systems. + > >> +An important note, that Isar creates local repositories using > >> dedicated Debian +tool reprepro. This tool takes care about: > >> + > >> + - Maintenance of multiple architectures and distros. > >> + > >> + - Internal packages database management. > >> + > >> + - Repo signatures management. > >> + > >> + - Requests about available packages in repo (name, version > >> etc...). + > >> # 3 Isar Internal Processes > >> > >> ## 3.1 General Overview > >> @@ -174,10 +211,17 @@ contain debian folder. The build process is > >> implemented in > >> 3. Run dpkg-buildpackage > >> > >> -4. Task `do_install`: install successfully built packages > >> +4. Further package processing depends on binary cache option: > >> + > >> + 1. Task `do_install_to_deploy`: install successfully built > >> packages if > >> + binary cache is disabled: > >> `${BUILDCHROOT_DIR}/home/build/${PN}/*.deb` to deploy > >> directory `${DEPLOY_DIR_DEB}` > >> > >> + 2. Task `do_install_to_cache`: install successfully built > >> packages to > >> + binary cache repository: > >> + `${BUILDCHROOT_DIR}/home/build/${PN}/*.deb` to `${DIR_CACHE}` > >> + > >> ## 3.5 Populate Target Filesystem > >> > >> Each target image can be extended by custom packages listed in > >> IMAGE_INSTALL @@ -185,11 +229,18 @@ variable. Task `do_populate` > >> performs the following: > >> 1. Parse IMAGE_INSTALL variable. > >> > >> -2. Find respective packages in `${DEPLOY_DIR_DEB}`. > >> +2. Further steps depends in binary cache option. > >> +If binary cache is disabled: > >> + > >> + 1. Find respective packages in `${DEPLOY_DIR_DEB}`. > >> + > >> + 2. Copy packages to deb folder in dedicated target filesystem. > >> + > >> + 3. Execute dpkg command in chroot for all the copied packages. > >> > >> -3. Copy them to deb folder in dedicated target filesystem. > >> +If binary cache is enabled: > >> > >> -4. Execute dpkg command in chroot for all the copied packages. > >> + 1. Call apt-get install in chroot in target image filesystem. > >> > >> ## 3.6 Generate Bootable Image > >> > > ^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [PATCH 6/6] doc/technical_overview: Describe binary cache 2017-08-28 15:36 ` Henning Schild 2017-08-29 7:29 ` Alexander Smirnov @ 2017-08-29 11:29 ` Jan Kiszka 1 sibling, 0 replies; 30+ messages in thread From: Jan Kiszka @ 2017-08-29 11:29 UTC (permalink / raw) To: [ext] Henning Schild, Alexander Smirnov; +Cc: isar-users On 2017-08-28 17:36, [ext] Henning Schild wrote: > This should be done along the way in every single patch, not afterwards. > I wouldn't be too strict with documentation, though. Generally, updating the docs in one step after a longer series of changes is acceptable as well as long as it happens in the same series. Keeping the code functionality intact in each commit is mandatory (with extremely few exceptions). Jan -- Siemens AG, Corporate Technology, CT RDA ITP SES-DE Corporate Competence Center Embedded Linux ^ permalink raw reply [flat|nested] 30+ messages in thread
end of thread, other threads:[~2017-09-07 11:13 UTC | newest] Thread overview: 30+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2017-08-27 15:13 [PATCH 0/6] Isar apt cache implementation Alexander Smirnov 2017-08-27 15:13 ` [PATCH 1/6] meta-isar-bin: Enable caching of deb packages Alexander Smirnov 2017-08-28 15:18 ` Henning Schild 2017-08-29 6:40 ` Alexander Smirnov 2017-08-29 7:51 ` Henning Schild 2017-08-29 8:20 ` Alexander Smirnov 2017-08-31 10:55 ` Claudius Heine 2017-08-31 11:20 ` Henning Schild 2017-08-31 12:08 ` Claudius Heine 2017-09-06 14:21 ` Henning Schild 2017-09-07 11:13 ` Claudius Heine 2017-08-27 15:13 ` [PATCH 2/6] classes/image: Provide /dev/null for Stretch apt Alexander Smirnov 2017-08-28 15:20 ` Henning Schild 2017-08-28 15:26 ` Henning Schild 2017-08-27 15:13 ` [PATCH 3/6] classes/dpkg: Split install for cache Alexander Smirnov 2017-08-28 8:00 ` Claudius Heine 2017-08-29 7:18 ` Alexander Smirnov 2017-08-30 8:54 ` Claudius Heine 2017-08-28 15:30 ` Henning Schild 2017-08-27 15:13 ` [PATCH 4/6] meta-isar-bin: Enable apt repo generation for amd64 Alexander Smirnov 2017-08-27 15:13 ` [PATCH 5/6] classes/dpkg: Properly update packages in the cache Alexander Smirnov 2017-08-28 15:32 ` Henning Schild 2017-08-29 7:20 ` Alexander Smirnov 2017-08-29 7:57 ` Henning Schild 2017-08-29 11:26 ` Jan Kiszka 2017-08-27 15:13 ` [PATCH 6/6] doc/technical_overview: Describe binary cache Alexander Smirnov 2017-08-28 15:36 ` Henning Schild 2017-08-29 7:29 ` Alexander Smirnov 2017-08-29 8:06 ` Henning Schild 2017-08-29 11:29 ` Jan Kiszka
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox