* [PATCH 0/9] Introduce local apt repo to cache upstream debian packages for offline usage
@ 2018-10-02 12:18 Maxim Yu. Osipov
2018-10-02 12:18 ` [PATCH 1/9] base-apt: Add helper class Maxim Yu. Osipov
` (10 more replies)
0 siblings, 11 replies; 26+ messages in thread
From: Maxim Yu. Osipov @ 2018-10-02 12:18 UTC (permalink / raw)
To: isar-users
Hello everybody,
This series by Alexander Smirnov,
1) Introduces dedicated local apt (base-apt) repo for upstream Debian packages
2) Caches in base-apt repo upstream Debian packages during image generation.
3) After this step, image can be built offline using only base-apt repo.
Usage instructions can be found in the last patch.
TODO:
Sign the repo with gpg
Kind regards,
Maxim.
Alexander Smirnov (8):
base-apt: Add helper class
base-apt: Introduce base implementaiton
isar-boot-strap: Add option to keep cache
image: Add cache_base_repo task
isar-bootstrap: Make possible to reuse the cache
buildchroot: Make it buildable from base-apt
workaround: Use --allow-unauthenticated working with base-apt
local.conf: Add option to use cached base repository
Maxim Yu. Osipov (1):
doc: Creation of local apt repo caching upstream Debian packages
doc/user_manual.md | 39 ++++++++++++++++
meta-isar/conf/layer.conf | 10 ++--
meta-isar/conf/local.conf.sample | 4 ++
meta-isar/recipes-core/images/isar-image-base.bb | 3 +-
meta/classes/base-apt-helper.bbclass | 53 ++++++++++++++++++++++
meta/classes/image.bbclass | 21 +++++++++
meta/classes/isar-bootstrap-helper.bbclass | 13 +++++-
.../isar-bootstrap/files/base-apt-sources | 1 +
.../recipes-core/isar-bootstrap/isar-bootstrap.inc | 32 +++++++++----
meta/recipes-devtools/base-apt/base-apt.bb | 31 +++++++++++++
.../base-apt/files/distributions.in | 3 ++
meta/recipes-devtools/buildchroot/files/deps.sh | 2 +-
12 files changed, 196 insertions(+), 16 deletions(-)
create mode 100644 meta/classes/base-apt-helper.bbclass
create mode 100644 meta/recipes-core/isar-bootstrap/files/base-apt-sources
create mode 100644 meta/recipes-devtools/base-apt/base-apt.bb
create mode 100644 meta/recipes-devtools/base-apt/files/distributions.in
--
2.11.0
^ permalink raw reply [flat|nested] 26+ messages in thread
* [PATCH 1/9] base-apt: Add helper class
2018-10-02 12:18 [PATCH 0/9] Introduce local apt repo to cache upstream debian packages for offline usage Maxim Yu. Osipov
@ 2018-10-02 12:18 ` Maxim Yu. Osipov
2018-10-02 13:39 ` Claudius Heine
2018-10-02 12:19 ` [PATCH 2/9] base-apt: Introduce base implementaiton Maxim Yu. Osipov
` (9 subsequent siblings)
10 siblings, 1 reply; 26+ messages in thread
From: Maxim Yu. Osipov @ 2018-10-02 12:18 UTC (permalink / raw)
To: isar-users
From: Alexander Smirnov <asmirnov@ilbers.de>
base-apt intended to store original upstream debs to re-use them
later offline. This class helps to populate base-apt with the packages
used during build.
Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
---
meta/classes/base-apt-helper.bbclass | 53 ++++++++++++++++++++++++++++++++++++
1 file changed, 53 insertions(+)
create mode 100644 meta/classes/base-apt-helper.bbclass
diff --git a/meta/classes/base-apt-helper.bbclass b/meta/classes/base-apt-helper.bbclass
new file mode 100644
index 0000000..5e3fe36
--- /dev/null
+++ b/meta/classes/base-apt-helper.bbclass
@@ -0,0 +1,53 @@
+# This software is a part of ISAR.
+# Copyright (C) 2018 ilbers GmbH
+
+DISTRO_NAME ?= "${@ d.getVar('DISTRO', True).split('-')[0]}"
+DISTRO_SUITE ?= "${@ d.getVar('DISTRO', True).split('-')[1]}"
+
+populate_base_apt() {
+ search_dir=$1
+
+ for package in $(find $search_dir -name '*.deb'); do
+ # NOTE: due to packages stored by reprepro are not modified, we can
+ # use search by filename to check if package is already in repo. In
+ # addition, m5sums could be compared to ensure, that package is the
+ # same and should not be overwritten. This method is easier and more
+ # robust than querying reprepro by name.
+
+ # Check if this package is taken from Isar-apt, if so - ingore it.
+ isar_package=$(find ${DEPLOY_DIR_APT}/${DISTRO} -name $package)
+ if [ -n "$isar_package" ]; then
+ # Check if MD5 sums are iendtical. This helps to avoid the case
+ # when packages is overriden from another repo.
+ md1=$(md5sum $package)
+ md2=$(md5sum $isar_package)
+ if [ "$md1" = "$md2" ]; then
+ continue
+ fi
+ fi
+
+ # Check if this package is already in base-apt
+ isar_package=$(find ${BASE_APT_DIR}/${DISTRO_NAME} -name $package)
+ if [ -n "$isar_package" ]; then
+ md1=$(md5sum $package)
+ md2=$(md5sum $isar_package)
+ if [ "$md1" = "$md2" ]; then
+ continue
+ fi
+
+ # md5sum differs, so remove the package from base-apt
+ name=$(basename $package | cut -d '_' -f 1)
+ reprepro -b ${BASE_APT_DIR}/${DISTRO_NAME} \
+ --dbdir ${BASE_APT_DB}/${DISTRO_NAME} \
+ -C main -A ${DISTRO_ARCH} \
+ remove ${DISTRO_SUITE} \
+ $name
+ fi
+
+ reprepro -b ${BASE_APT_DIR}/${DISTRO_NAME} \
+ --dbdir ${BASE_APT_DB}/${DISTRO_NAME} \
+ -C main \
+ includedeb ${DISTRO_SUITE} \
+ $package
+ done
+}
--
2.11.0
^ permalink raw reply [flat|nested] 26+ messages in thread
* [PATCH 2/9] base-apt: Introduce base implementaiton
2018-10-02 12:18 [PATCH 0/9] Introduce local apt repo to cache upstream debian packages for offline usage Maxim Yu. Osipov
2018-10-02 12:18 ` [PATCH 1/9] base-apt: Add helper class Maxim Yu. Osipov
@ 2018-10-02 12:19 ` Maxim Yu. Osipov
2018-10-02 14:20 ` Claudius Heine
2018-10-02 12:19 ` [PATCH 3/9] isar-boot-strap: Add option to keep cache Maxim Yu. Osipov
` (8 subsequent siblings)
10 siblings, 1 reply; 26+ messages in thread
From: Maxim Yu. Osipov @ 2018-10-02 12:19 UTC (permalink / raw)
To: isar-users
From: Alexander Smirnov <asmirnov@ilbers.de>
Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
---
meta-isar/conf/layer.conf | 10 ++++---
meta/recipes-devtools/base-apt/base-apt.bb | 31 ++++++++++++++++++++++
.../base-apt/files/distributions.in | 3 +++
3 files changed, 40 insertions(+), 4 deletions(-)
create mode 100644 meta/recipes-devtools/base-apt/base-apt.bb
create mode 100644 meta/recipes-devtools/base-apt/files/distributions.in
diff --git a/meta-isar/conf/layer.conf b/meta-isar/conf/layer.conf
index cd42f06..ae1b3c5 100644
--- a/meta-isar/conf/layer.conf
+++ b/meta-isar/conf/layer.conf
@@ -20,8 +20,10 @@ LAYERDIR_isar = "${LAYERDIR}"
# Codename of the repository created by the caching class
DEBDISTRONAME = "isar"
-# Path to the Isar apt repository
-DEPLOY_DIR_APT ?= "${DEPLOY_DIR}/apt"
+# Isar apt repository paths
+DEPLOY_DIR_APT ?= "${DEPLOY_DIR}/isar-apt/apt"
+DEPLOY_DIR_DB ?= "${DEPLOY_DIR}/isar-apt/db"
-# Path to the Isar databases used by `reprepro`
-DEPLOY_DIR_DB ?= "${DEPLOY_DIR}/db"
+# Base apt repository paths
+BASE_APT_DIR ?= "${DEPLOY_DIR}/base-apt/apt"
+BASE_APT_DB ?= "${DEPLOY_DIR}/base-apt/db"
diff --git a/meta/recipes-devtools/base-apt/base-apt.bb b/meta/recipes-devtools/base-apt/base-apt.bb
new file mode 100644
index 0000000..6ff1164
--- /dev/null
+++ b/meta/recipes-devtools/base-apt/base-apt.bb
@@ -0,0 +1,31 @@
+# This software is a part of ISAR.
+# Copyright (C) 2018 ilbers GmbH
+
+SRC_URI = "file://distributions.in"
+
+inherit base-apt-helper
+
+CACHE_CONF_DIR = "${BASE_APT_DIR}/${DISTRO_NAME}/conf"
+do_cache_config[dirs] = "${CACHE_CONF_DIR}"
+do_cache_config[stamp-extra-info] = "${DISTRO}"
+do_cache_config[lockfiles] = "${BASE_APT_DIR}/isar.lock"
+
+# Generate reprepro config for current distro if it doesn't exist. Once it's
+# generated, this task should do nothing.
+do_cache_config() {
+ if [ ! -e "${CACHE_CONF_DIR}/distributions" ]; then
+ sed -e "s#{DISTRO_NAME}#"${DISTRO_SUITE}"#g" \
+ ${WORKDIR}/distributions.in > ${CACHE_CONF_DIR}/distributions
+ fi
+
+ path_cache="${BASE_APT_DIR}/${DISTRO_NAME}"
+ path_databases="${BASE_APT_DB}/${DISTRO_NAME}"
+
+ if [ ! -d "${path_databases}" ]; then
+ reprepro -b ${path_cache} \
+ --dbdir ${path_databases} \
+ export ${DISTRO_SUITE}
+ fi
+}
+
+addtask cache_config after do_build
diff --git a/meta/recipes-devtools/base-apt/files/distributions.in b/meta/recipes-devtools/base-apt/files/distributions.in
new file mode 100644
index 0000000..cc82c57
--- /dev/null
+++ b/meta/recipes-devtools/base-apt/files/distributions.in
@@ -0,0 +1,3 @@
+Codename: {DISTRO_NAME}
+Architectures: i386 armhf arm64 amd64 source
+Components: main
--
2.11.0
^ permalink raw reply [flat|nested] 26+ messages in thread
* [PATCH 3/9] isar-boot-strap: Add option to keep cache
2018-10-02 12:18 [PATCH 0/9] Introduce local apt repo to cache upstream debian packages for offline usage Maxim Yu. Osipov
2018-10-02 12:18 ` [PATCH 1/9] base-apt: Add helper class Maxim Yu. Osipov
2018-10-02 12:19 ` [PATCH 2/9] base-apt: Introduce base implementaiton Maxim Yu. Osipov
@ 2018-10-02 12:19 ` Maxim Yu. Osipov
2018-10-02 14:49 ` Claudius Heine
2018-10-02 12:19 ` [PATCH 4/9] image: Add cache_base_repo task Maxim Yu. Osipov
` (7 subsequent siblings)
10 siblings, 1 reply; 26+ messages in thread
From: Maxim Yu. Osipov @ 2018-10-02 12:19 UTC (permalink / raw)
To: isar-users
From: Alexander Smirnov <asmirnov@ilbers.de>
The default ISAR behavior assumes to remove all the packages
from apt cache, but we need them to put into base-apt.
Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
---
meta/classes/isar-bootstrap-helper.bbclass | 6 ++++++
1 file changed, 6 insertions(+)
diff --git a/meta/classes/isar-bootstrap-helper.bbclass b/meta/classes/isar-bootstrap-helper.bbclass
index 597f6b1..4a00cb0 100644
--- a/meta/classes/isar-bootstrap-helper.bbclass
+++ b/meta/classes/isar-bootstrap-helper.bbclass
@@ -70,6 +70,7 @@ setup_root_file_system() {
--fstab) FSTAB=$2; shift ;;
--host-arch) ROOTFS_ARCH=${HOST_ARCH} ;;
--host-distro) ROOTFS_DISTRO=${HOST_DISTRO} ;;
+ --keep-apt-cache) KEEP_APT_CACHE=1 ;;
-*) bbfatal "$0: invalid option specified: $1" ;;
*) break ;;
esac
@@ -131,6 +132,11 @@ setup_root_file_system() {
/usr/bin/apt-get purge --yes $pkg
done
if [ ${CLEAN} ]; then
+ if [ ${KEEP_APT_CACHE} ]; then
+ mkdir -p ${WORKDIR}/apt_cache
+ sudo mv $(find $ROOTFSDIR/var/cache/apt -name '*.deb') ${WORKDIR}/apt_cache
+ sudo chown $USER ${WORKDIR}/apt_cache/*
+ fi
sudo -E chroot "$ROOTFSDIR" \
/usr/bin/apt-get autoremove --purge --yes
sudo -E chroot "$ROOTFSDIR" \
--
2.11.0
^ permalink raw reply [flat|nested] 26+ messages in thread
* [PATCH 4/9] image: Add cache_base_repo task
2018-10-02 12:18 [PATCH 0/9] Introduce local apt repo to cache upstream debian packages for offline usage Maxim Yu. Osipov
` (2 preceding siblings ...)
2018-10-02 12:19 ` [PATCH 3/9] isar-boot-strap: Add option to keep cache Maxim Yu. Osipov
@ 2018-10-02 12:19 ` Maxim Yu. Osipov
2018-10-02 12:19 ` [PATCH 5/9] isar-bootstrap: Make possible to reuse the cache Maxim Yu. Osipov
` (6 subsequent siblings)
10 siblings, 0 replies; 26+ messages in thread
From: Maxim Yu. Osipov @ 2018-10-02 12:19 UTC (permalink / raw)
To: isar-users
From: Alexander Smirnov <asmirnov@ilbers.de>
This task puts all the packages using in build to base-apt.
Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
Signed-off-by: Maxim Yu. Osipov <mosipov@ilbers.de>
---
meta-isar/recipes-core/images/isar-image-base.bb | 3 ++-
meta/classes/image.bbclass | 21 +++++++++++++++++++++
2 files changed, 23 insertions(+), 1 deletion(-)
diff --git a/meta-isar/recipes-core/images/isar-image-base.bb b/meta-isar/recipes-core/images/isar-image-base.bb
index 4899593..bf606cc 100644
--- a/meta-isar/recipes-core/images/isar-image-base.bb
+++ b/meta-isar/recipes-core/images/isar-image-base.bb
@@ -41,7 +41,8 @@ devtmpfs /dev devtmpfs mode=0755,nosuid 0 0
# End /etc/fstab
EOF
- setup_root_file_system --clean --fstab "${WORKDIR}/fstab" \
+ setup_root_file_system --clean --keep-apt-cache \
+ --fstab "${WORKDIR}/fstab" \
"${IMAGE_ROOTFS}" ${IMAGE_PREINSTALL} ${IMAGE_INSTALL}
# Configure root filesystem
diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index ea8cbf5..0505a73 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -123,6 +123,27 @@ do_populate_sdk[depends] = "sdkchroot:do_build"
addtask populate_sdk after do_rootfs
+inherit base-apt-helper
+
+do_cache_base_repo[depends] = "base-apt:do_cache_config"
+do_cache_base_repo[stamp-extra-info] = "${MACHINE}-${DISTRO}"
+
+do_cache_base_repo() {
+ if [ -d ${WORKDIR}/apt_cache ]; then
+ populate_base_apt ${WORKDIR}/apt_cache
+ fi
+
+ if [ -d ${BUILDCHROOT_HOST_DIR}/var/cache/apt ]; then
+ populate_base_apt ${BUILDCHROOT_HOST_DIR}/var/cache/apt
+ fi
+
+ if [ -d ${BUILDCHROOT_TARGET_DIR}/var/cache/apt ]; then
+ populate_base_apt ${BUILDCHROOT_TARGET_DIR}/var/cache/apt
+ fi
+}
+
+addtask cache_base_repo after do_rootfs
+
# Imager are expected to run natively, thus will use the target buildchroot.
ISAR_CROSS_COMPILE = "0"
--
2.11.0
^ permalink raw reply [flat|nested] 26+ messages in thread
* [PATCH 5/9] isar-bootstrap: Make possible to reuse the cache
2018-10-02 12:18 [PATCH 0/9] Introduce local apt repo to cache upstream debian packages for offline usage Maxim Yu. Osipov
` (3 preceding siblings ...)
2018-10-02 12:19 ` [PATCH 4/9] image: Add cache_base_repo task Maxim Yu. Osipov
@ 2018-10-02 12:19 ` Maxim Yu. Osipov
2018-11-02 11:40 ` Henning Schild
2018-10-02 12:19 ` [PATCH 6/9] buildchroot: Make it buildable from base-apt Maxim Yu. Osipov
` (5 subsequent siblings)
10 siblings, 1 reply; 26+ messages in thread
From: Maxim Yu. Osipov @ 2018-10-02 12:19 UTC (permalink / raw)
To: isar-users
From: Alexander Smirnov <asmirnov@ilbers.de>
Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
Signed-off-by: Maxim Yu. Osipov <mosipov@ilbers.de>
---
.../isar-bootstrap/files/base-apt-sources | 1 +
.../recipes-core/isar-bootstrap/isar-bootstrap.inc | 32 ++++++++++++++++------
2 files changed, 24 insertions(+), 9 deletions(-)
create mode 100644 meta/recipes-core/isar-bootstrap/files/base-apt-sources
diff --git a/meta/recipes-core/isar-bootstrap/files/base-apt-sources b/meta/recipes-core/isar-bootstrap/files/base-apt-sources
new file mode 100644
index 0000000..594db56
--- /dev/null
+++ b/meta/recipes-core/isar-bootstrap/files/base-apt-sources
@@ -0,0 +1 @@
+deb file:///base-apt/debian {DISTRO} main
diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
index cfad136..9b5a894 100644
--- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
+++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
@@ -12,18 +12,24 @@ SRC_URI = " \
file://isar-apt.conf \
file://isar-apt-fallback.conf \
file://locale \
- file://chroot-setup.sh"
+ file://chroot-setup.sh \
+ file://base-apt-sources"
PV = "1.0"
DEBOOTSTRAP ?= "qemu-debootstrap"
ROOTFSDIR = "${WORKDIR}/rootfs"
APTPREFS = "${WORKDIR}/apt-preferences"
APTSRCS = "${WORKDIR}/apt-sources"
+BASEAPTSRCS = "${WORKDIR}/base-apt-sources"
APTKEYFILES = ""
APTKEYRING = "${WORKDIR}/apt-keyring.gpg"
DEBOOTSTRAP_KEYRING = ""
DEPLOY_ISAR_BOOTSTRAP ?= ""
+DISTRO_APT_PREMIRRORS ?= "${@ "http://ftp\.(\S+\.)?debian.org file:///${BASE_APT_DIR} \n" if bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')) else "" }"
+
+inherit base-apt-helper
+
python () {
from urllib.parse import urlparse
distro_apt_keys = d.getVar("DISTRO_APT_KEYS", False)
@@ -171,6 +177,10 @@ isar_bootstrap() {
esac
shift
done
+ debootstrap_args="--verbose --variant=minbase --include=locales "
+ if [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ]; then
+ debootstrap_args="$debootstrap_args --no-check-gpg"
+ fi
E="${@bb.utils.export_proxies(d)}"
sudo -E flock "${ISAR_BOOTSTRAP_LOCK}" -c "\
set -e
@@ -181,9 +191,7 @@ isar_bootstrap() {
rm -rf "${ROOTFSDIR}"
fi
if [ ${IS_HOST} ]; then
- ${DEBOOTSTRAP} --verbose \
- --variant=minbase \
- --include=locales \
+ ${DEBOOTSTRAP} $debootstrap_args \
${@get_distro_components_argument(d, True)} \
${DEBOOTSTRAP_KEYRING} \
"${@get_distro_suite(d, True)}" \
@@ -191,10 +199,8 @@ isar_bootstrap() {
"${@get_distro_source(d, True)}"
else
- "${DEBOOTSTRAP}" --verbose \
- --variant=minbase \
+ "${DEBOOTSTRAP}" $debootstrap_args \
--arch="${DISTRO_ARCH}" \
- --include=locales \
${@get_distro_components_argument(d, False)} \
${DEBOOTSTRAP_KEYRING} \
"${@get_distro_suite(d, False)}" \
@@ -207,8 +213,16 @@ isar_bootstrap() {
install -v -m644 "${APTPREFS}" \
"${ROOTFSDIR}/etc/apt/preferences.d/bootstrap"
mkdir -p "${ROOTFSDIR}/etc/apt/sources.list.d"
- install -v -m644 "${APTSRCS}" \
- "${ROOTFSDIR}/etc/apt/sources.list.d/bootstrap.list"
+ if [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ]; then
+ sed -i -e "s#{DISTRO}#"${DISTRO_SUITE}"#g" ${BASEAPTSRCS}
+ mkdir -p ${ROOTFSDIR}/base-apt
+ sudo mount --bind ${BASE_APT_DIR} ${ROOTFSDIR}/base-apt
+ install -v -m644 "${BASEAPTSRCS}" \
+ "${ROOTFSDIR}/etc/apt/sources.list.d/base-apt.list"
+ else
+ install -v -m644 "${APTSRCS}" \
+ "${ROOTFSDIR}/etc/apt/sources.list.d/bootstrap.list"
+ fi
rm -f "${ROOTFSDIR}/etc/apt/sources.list"
mkdir -p "${ROOTFSDIR}/etc/apt/apt.conf.d"
install -v -m644 "${WORKDIR}/isar-apt.conf" \
--
2.11.0
^ permalink raw reply [flat|nested] 26+ messages in thread
* [PATCH 6/9] buildchroot: Make it buildable from base-apt
2018-10-02 12:18 [PATCH 0/9] Introduce local apt repo to cache upstream debian packages for offline usage Maxim Yu. Osipov
` (4 preceding siblings ...)
2018-10-02 12:19 ` [PATCH 5/9] isar-bootstrap: Make possible to reuse the cache Maxim Yu. Osipov
@ 2018-10-02 12:19 ` Maxim Yu. Osipov
2018-10-02 12:19 ` [PATCH 7/9] workaround: Use --allow-unauthenticated working with base-apt Maxim Yu. Osipov
` (4 subsequent siblings)
10 siblings, 0 replies; 26+ messages in thread
From: Maxim Yu. Osipov @ 2018-10-02 12:19 UTC (permalink / raw)
To: isar-users
From: Alexander Smirnov <asmirnov@ilbers.de>
Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
Signed-off-by: Maxim Yu. Osipov <mosipov@ilbers.de>
---
meta/classes/isar-bootstrap-helper.bbclass | 5 +++++
1 file changed, 5 insertions(+)
diff --git a/meta/classes/isar-bootstrap-helper.bbclass b/meta/classes/isar-bootstrap-helper.bbclass
index 4a00cb0..9dabc25 100644
--- a/meta/classes/isar-bootstrap-helper.bbclass
+++ b/meta/classes/isar-bootstrap-helper.bbclass
@@ -98,6 +98,11 @@ setup_root_file_system() {
else
sudo mount --bind ${DEPLOY_DIR_APT}/${DISTRO} $ROOTFSDIR/isar-apt
fi
+
+ if [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ]; then
+ sudo mount --bind ${BASE_APT_DIR} ${ROOTFSDIR}/base-apt
+ fi
+
sudo mount -t devtmpfs -o mode=0755,nosuid devtmpfs $ROOTFSDIR/dev
sudo mount -t proc none $ROOTFSDIR/proc
--
2.11.0
^ permalink raw reply [flat|nested] 26+ messages in thread
* [PATCH 7/9] workaround: Use --allow-unauthenticated working with base-apt
2018-10-02 12:18 [PATCH 0/9] Introduce local apt repo to cache upstream debian packages for offline usage Maxim Yu. Osipov
` (5 preceding siblings ...)
2018-10-02 12:19 ` [PATCH 6/9] buildchroot: Make it buildable from base-apt Maxim Yu. Osipov
@ 2018-10-02 12:19 ` Maxim Yu. Osipov
2018-10-02 12:19 ` [PATCH 8/9] local.conf: Add option to use cached base repository Maxim Yu. Osipov
` (3 subsequent siblings)
10 siblings, 0 replies; 26+ messages in thread
From: Maxim Yu. Osipov @ 2018-10-02 12:19 UTC (permalink / raw)
To: isar-users
From: Alexander Smirnov <asmirnov@ilbers.de>
Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
---
meta/classes/isar-bootstrap-helper.bbclass | 2 +-
meta/recipes-devtools/buildchroot/files/deps.sh | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
diff --git a/meta/classes/isar-bootstrap-helper.bbclass b/meta/classes/isar-bootstrap-helper.bbclass
index 9dabc25..b7ab552 100644
--- a/meta/classes/isar-bootstrap-helper.bbclass
+++ b/meta/classes/isar-bootstrap-helper.bbclass
@@ -79,7 +79,7 @@ setup_root_file_system() {
ROOTFSDIR="$1"
shift
PACKAGES="$@"
- APT_ARGS="install --yes -o Debug::pkgProblemResolver=yes"
+ APT_ARGS="install --yes --allow-unauthenticated -o Debug::pkgProblemResolver=yes"
CLEAN_FILES="${ROOTFSDIR}/etc/hostname ${ROOTFSDIR}/etc/resolv.conf"
sudo cp -Trpfx \
diff --git a/meta/recipes-devtools/buildchroot/files/deps.sh b/meta/recipes-devtools/buildchroot/files/deps.sh
index 4bd604f..2fc2f66 100644
--- a/meta/recipes-devtools/buildchroot/files/deps.sh
+++ b/meta/recipes-devtools/buildchroot/files/deps.sh
@@ -10,7 +10,7 @@ source /isar/common.sh
# Notes:
# 1) everything before the -y switch is unchanged from the defaults
# 2) we add -y to go non-interactive
-install_cmd="apt-get -o Debug::pkgProblemResolver=yes --no-install-recommends -y"
+install_cmd="apt-get -o Debug::pkgProblemResolver=yes --no-install-recommends -y --allow-unauthenticated"
# Make sure that we have latest isar-apt content.
# Options meaning:
--
2.11.0
^ permalink raw reply [flat|nested] 26+ messages in thread
* [PATCH 8/9] local.conf: Add option to use cached base repository
2018-10-02 12:18 [PATCH 0/9] Introduce local apt repo to cache upstream debian packages for offline usage Maxim Yu. Osipov
` (6 preceding siblings ...)
2018-10-02 12:19 ` [PATCH 7/9] workaround: Use --allow-unauthenticated working with base-apt Maxim Yu. Osipov
@ 2018-10-02 12:19 ` Maxim Yu. Osipov
2018-10-02 12:19 ` [PATCH 9/9] doc: Creation of local apt repo caching upstream Debian packages Maxim Yu. Osipov
` (2 subsequent siblings)
10 siblings, 0 replies; 26+ messages in thread
From: Maxim Yu. Osipov @ 2018-10-02 12:19 UTC (permalink / raw)
To: isar-users
From: Alexander Smirnov <asmirnov@ilbers.de>
Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
Signed-off-by: Maxim Yu. Osipov <mosipov@ilbers.de>
---
meta-isar/conf/local.conf.sample | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/meta-isar/conf/local.conf.sample b/meta-isar/conf/local.conf.sample
index b8b8495..f3a960a 100644
--- a/meta-isar/conf/local.conf.sample
+++ b/meta-isar/conf/local.conf.sample
@@ -165,3 +165,7 @@ IMAGE_INSTALL = "example-hello example-raw example-module enable-fsck"
# Enable cross-compilation support
# NOTE: this works on build host >= stretch for armhf, arm64 and amd64 targets for now.
ISAR_CROSS_COMPILE ?= "0"
+
+#
+# Uncomment this to enable use of cached base repository
+#ISAR_USE_CACHED_BASE_REPO ?= "1"
--
2.11.0
^ permalink raw reply [flat|nested] 26+ messages in thread
* [PATCH 9/9] doc: Creation of local apt repo caching upstream Debian packages
2018-10-02 12:18 [PATCH 0/9] Introduce local apt repo to cache upstream debian packages for offline usage Maxim Yu. Osipov
` (7 preceding siblings ...)
2018-10-02 12:19 ` [PATCH 8/9] local.conf: Add option to use cached base repository Maxim Yu. Osipov
@ 2018-10-02 12:19 ` Maxim Yu. Osipov
2018-10-02 14:02 ` Claudius Heine
2018-10-02 14:05 ` Claudius Heine
2018-11-02 12:00 ` [PATCH 0/9] Introduce local apt repo to cache upstream debian packages for offline usage Henning Schild
2018-11-04 10:07 ` Jan Kiszka
10 siblings, 2 replies; 26+ messages in thread
From: Maxim Yu. Osipov @ 2018-10-02 12:19 UTC (permalink / raw)
To: isar-users
Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
Signed-off-by: Maxim Yu. Osipov <mosipov@ilbers.de>
---
doc/user_manual.md | 39 +++++++++++++++++++++++++++++++++++++++
1 file changed, 39 insertions(+)
diff --git a/doc/user_manual.md b/doc/user_manual.md
index db58cf1..d69dd0d 100644
--- a/doc/user_manual.md
+++ b/doc/user_manual.md
@@ -19,6 +19,8 @@ Copyright (C) 2016-2017, ilbers GmbH
- [Add a Custom Application](https://github.com/ilbers/isar/blob/master/doc/user_manual.md#add-a-custom-application)
- [Enabling Cross-compilation](https://github.com/ilbers/isar/blob/master/doc/user_manual.md#isar-cross-compilation)
- [Create an ISAR SDK root filesystem](https://github.com/ilbers/isar/blob/master/doc/user_manual.md#create-an-isar-sdk-root-filesystem)
+ - [Creation of local apt repo caching upstream Debian packages](https://github.com/ilbers/isar/blob/master/doc/user_manual.md#creation-repo-caching-upstream-debian)
+
## Introduction
@@ -686,3 +688,40 @@ ii crossbuild-essential-armhf 12.3 all Inf
/usr/share/doc/libhello-dev/copyright
~#
```
+
+## Creation of local apt repo caching upstream Debian packages
+
+### Motivation
+
+Cache upstream debian packages to reduce time for further downloads and to be able to work offline.
+
+### Solution
+
+ - Trigger creation of local apt caching Debian packages during image generation.
+
+```
+bitbake -c cache_base_repo multiconfig:qemuarm-stretch:isar-image-base
+```
+
+ - Set `ISAR_USE_CACHED_BASE_REPO` in `conf/local.conf`:
+
+```
+# Uncomment this to enable use of cached base repository
+#ISAR_USE_CACHED_BASE_REPO ?= "1"
+```
+ - Remove build artifacts to use only local base-apt:
+
+```
+sudo rm -rf tmp/stamps/ tmp/work/ tmp/deploy/isar-apt/ tmp/deploy/images
+
+```
+
+ - Trigger again generation of image (now using local caching repo):
+
+```
+bitbake multiconfig:qemuarm-stretch:isar-image-base
+```
+
+### Limitation
+
+So far the local base-apt repo is not gpg signed.
--
2.11.0
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [PATCH 1/9] base-apt: Add helper class
2018-10-02 12:18 ` [PATCH 1/9] base-apt: Add helper class Maxim Yu. Osipov
@ 2018-10-02 13:39 ` Claudius Heine
0 siblings, 0 replies; 26+ messages in thread
From: Claudius Heine @ 2018-10-02 13:39 UTC (permalink / raw)
To: Maxim Yu. Osipov, isar-users
Hi Maxim,
On 10/2/18 2:18 PM, Maxim Yu. Osipov wrote:
> From: Alexander Smirnov <asmirnov@ilbers.de>
>
> base-apt intended to store original upstream debs to re-use them
> later offline. This class helps to populate base-apt with the packages
> used during build.
>
> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
> ---
> meta/classes/base-apt-helper.bbclass | 53 ++++++++++++++++++++++++++++++++++++
> 1 file changed, 53 insertions(+)
> create mode 100644 meta/classes/base-apt-helper.bbclass
>
> diff --git a/meta/classes/base-apt-helper.bbclass b/meta/classes/base-apt-helper.bbclass
> new file mode 100644
> index 0000000..5e3fe36
> --- /dev/null
> +++ b/meta/classes/base-apt-helper.bbclass
> @@ -0,0 +1,53 @@
> +# This software is a part of ISAR.
> +# Copyright (C) 2018 ilbers GmbH
> +
> +DISTRO_NAME ?= "${@ d.getVar('DISTRO', True).split('-')[0]}"
> +DISTRO_SUITE ?= "${@ d.getVar('DISTRO', True).split('-')[1]}"
> +
> +populate_base_apt() {
> + search_dir=$1
> +
> + for package in $(find $search_dir -name '*.deb'); do
find ... | while read package; do
or
while read package; do ...; done <<< $(find ...)
is better performance wise since the loop is iterated as results come in.
> + # NOTE: due to packages stored by reprepro are not modified, we can
> + # use search by filename to check if package is already in repo. In
> + # addition, m5sums could be compared to ensure, that package is the
> + # same and should not be overwritten. This method is easier and more
> + # robust than querying reprepro by name.
> +
> + # Check if this package is taken from Isar-apt, if so - ingore it.
> + isar_package=$(find ${DEPLOY_DIR_APT}/${DISTRO} -name $package)
Don't you need a `${package##*/}` here to remove the leading path?
> + if [ -n "$isar_package" ]; then
> + # Check if MD5 sums are iendtical. This helps to avoid the case
identical
> + # when packages is overriden from another repo.
overridden
> + md1=$(md5sum $package)
> + md2=$(md5sum $isar_package)
md5sum produces output with like this:
$ md5sum conf/local.conf
c543793c1df3b1f45a5555d92b6f3ee2 conf/local.conf
So you need to remove that before comparing. The same issues are in the
following code as well.
Cheers,
Claudius
> + if [ "$md1" = "$md2" ]; then
> + continue
> + fi
> + fi
> +
> + # Check if this package is already in base-apt
> + isar_package=$(find ${BASE_APT_DIR}/${DISTRO_NAME} -name $package)
> + if [ -n "$isar_package" ]; then
> + md1=$(md5sum $package)
> + md2=$(md5sum $isar_package)
> + if [ "$md1" = "$md2" ]; then
> + continue
> + fi
> +
> + # md5sum differs, so remove the package from base-apt
> + name=$(basename $package | cut -d '_' -f 1)
> + reprepro -b ${BASE_APT_DIR}/${DISTRO_NAME} \
> + --dbdir ${BASE_APT_DB}/${DISTRO_NAME} \
> + -C main -A ${DISTRO_ARCH} \
> + remove ${DISTRO_SUITE} \
> + $name
> + fi
> +
> + reprepro -b ${BASE_APT_DIR}/${DISTRO_NAME} \
> + --dbdir ${BASE_APT_DB}/${DISTRO_NAME} \
> + -C main \
> + includedeb ${DISTRO_SUITE} \
> + $package
> + done
> +}
>
--
DENX Software Engineering GmbH, Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [PATCH 9/9] doc: Creation of local apt repo caching upstream Debian packages
2018-10-02 12:19 ` [PATCH 9/9] doc: Creation of local apt repo caching upstream Debian packages Maxim Yu. Osipov
@ 2018-10-02 14:02 ` Claudius Heine
2018-10-02 14:06 ` Jan Kiszka
2018-10-04 9:03 ` Baurzhan Ismagulov
2018-10-02 14:05 ` Claudius Heine
1 sibling, 2 replies; 26+ messages in thread
From: Claudius Heine @ 2018-10-02 14:02 UTC (permalink / raw)
To: Maxim Yu. Osipov, isar-users
Hi,
On 10/2/18 2:19 PM, Maxim Yu. Osipov wrote:
> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
> Signed-off-by: Maxim Yu. Osipov <mosipov@ilbers.de>
> ---
> doc/user_manual.md | 39 +++++++++++++++++++++++++++++++++++++++
> 1 file changed, 39 insertions(+)
>
> diff --git a/doc/user_manual.md b/doc/user_manual.md
> index db58cf1..d69dd0d 100644
> --- a/doc/user_manual.md
> +++ b/doc/user_manual.md
> @@ -19,6 +19,8 @@ Copyright (C) 2016-2017, ilbers GmbH
> - [Add a Custom Application](https://github.com/ilbers/isar/blob/master/doc/user_manual.md#add-a-custom-application)
> - [Enabling Cross-compilation](https://github.com/ilbers/isar/blob/master/doc/user_manual.md#isar-cross-compilation)
> - [Create an ISAR SDK root filesystem](https://github.com/ilbers/isar/blob/master/doc/user_manual.md#create-an-isar-sdk-root-filesystem)
> + - [Creation of local apt repo caching upstream Debian packages](https://github.com/ilbers/isar/blob/master/doc/user_manual.md#creation-repo-caching-upstream-debian)
> +
>
> ## Introduction
>
> @@ -686,3 +688,40 @@ ii crossbuild-essential-armhf 12.3 all Inf
> /usr/share/doc/libhello-dev/copyright
> ~#
> ```
> +
> +## Creation of local apt repo caching upstream Debian packages
> +
> +### Motivation
> +
> +Cache upstream debian packages to reduce time for further downloads and to be able to work offline.
> +
> +### Solution
> +
> + - Trigger creation of local apt caching Debian packages during image generation.
> +
> +```
> +bitbake -c cache_base_repo multiconfig:qemuarm-stretch:isar-image-base
> +```
> +
> + - Set `ISAR_USE_CACHED_BASE_REPO` in `conf/local.conf`:
> +
> +```
> +# Uncomment this to enable use of cached base repository
> +#ISAR_USE_CACHED_BASE_REPO ?= "1"
No that doesn't make sense to me.
In my opinion its very important to not have such flags or commands.
Caching and reproducible builds should be the default operation,
upgrading or removing the cache should be an extra step.
The cache should be created and used on every image build. On the first
build the cache is empty and on the next build its full, so it will be used.
Use apt preferences to git the cache a higher priority than the upstream
repos to archive that.
Also if the sources.list is different when setting
`ISAR_USE_CACHED_BASE_REPO` than the image is different from the first one.
Also please avoid those binary settings like
`ISAR_USE_CACHED_BASE_REPO`. That just bad design because it leads to
every tiny setting having an own variable. Use FEATURE variables with
entries like:
`ISAR_BUILD_FEATURE = "cross-build use-cache ..."`
Yes cross-builds should have been done like this as well IMO.
Cheers,
Claudius
> +```
> + - Remove build artifacts to use only local base-apt:
> +
> +```
> +sudo rm -rf tmp/stamps/ tmp/work/ tmp/deploy/isar-apt/ tmp/deploy/images
> +
> +```
> +
> + - Trigger again generation of image (now using local caching repo):
> +
> +```
> +bitbake multiconfig:qemuarm-stretch:isar-image-base
> +```
> +
> +### Limitation
> +
> +So far the local base-apt repo is not gpg signed.
>
--
DENX Software Engineering GmbH, Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [PATCH 9/9] doc: Creation of local apt repo caching upstream Debian packages
2018-10-02 12:19 ` [PATCH 9/9] doc: Creation of local apt repo caching upstream Debian packages Maxim Yu. Osipov
2018-10-02 14:02 ` Claudius Heine
@ 2018-10-02 14:05 ` Claudius Heine
1 sibling, 0 replies; 26+ messages in thread
From: Claudius Heine @ 2018-10-02 14:05 UTC (permalink / raw)
To: Maxim Yu. Osipov, isar-users
On 10/2/18 2:19 PM, Maxim Yu. Osipov wrote:
> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
> Signed-off-by: Maxim Yu. Osipov <mosipov@ilbers.de>
> ---
> doc/user_manual.md | 39 +++++++++++++++++++++++++++++++++++++++
> 1 file changed, 39 insertions(+)
>
> diff --git a/doc/user_manual.md b/doc/user_manual.md
> index db58cf1..d69dd0d 100644
> --- a/doc/user_manual.md
> +++ b/doc/user_manual.md
> @@ -19,6 +19,8 @@ Copyright (C) 2016-2017, ilbers GmbH
> - [Add a Custom Application](https://github.com/ilbers/isar/blob/master/doc/user_manual.md#add-a-custom-application)
> - [Enabling Cross-compilation](https://github.com/ilbers/isar/blob/master/doc/user_manual.md#isar-cross-compilation)
> - [Create an ISAR SDK root filesystem](https://github.com/ilbers/isar/blob/master/doc/user_manual.md#create-an-isar-sdk-root-filesystem)
> + - [Creation of local apt repo caching upstream Debian packages](https://github.com/ilbers/isar/blob/master/doc/user_manual.md#creation-repo-caching-upstream-debian)
On another point. Why not use links like?
[Creation of local apt repo caching upstream Debian
packages](#creation-repo-caching-upstream-debian)
> +
>
> ## Introduction
>
> @@ -686,3 +688,40 @@ ii crossbuild-essential-armhf 12.3 all Inf
> /usr/share/doc/libhello-dev/copyright
> ~#
> ```
> +
> +## Creation of local apt repo caching upstream Debian packages
> +
> +### Motivation
> +
> +Cache upstream debian packages to reduce time for further downloads and to be able to work offline.
> +
> +### Solution
> +
> + - Trigger creation of local apt caching Debian packages during image generation.
> +
> +```
> +bitbake -c cache_base_repo multiconfig:qemuarm-stretch:isar-image-base
> +```
> +
> + - Set `ISAR_USE_CACHED_BASE_REPO` in `conf/local.conf`:
> +
> +```
> +# Uncomment this to enable use of cached base repository
> +#ISAR_USE_CACHED_BASE_REPO ?= "1"
> +```
> + - Remove build artifacts to use only local base-apt:
> +
> +```
> +sudo rm -rf tmp/stamps/ tmp/work/ tmp/deploy/isar-apt/ tmp/deploy/images
> +
> +```
> +
> + - Trigger again generation of image (now using local caching repo):
> +
> +```
> +bitbake multiconfig:qemuarm-stretch:isar-image-base
> +```
> +
> +### Limitation
> +
> +So far the local base-apt repo is not gpg signed.
>
--
DENX Software Engineering GmbH, Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [PATCH 9/9] doc: Creation of local apt repo caching upstream Debian packages
2018-10-02 14:02 ` Claudius Heine
@ 2018-10-02 14:06 ` Jan Kiszka
2018-10-04 9:03 ` Baurzhan Ismagulov
1 sibling, 0 replies; 26+ messages in thread
From: Jan Kiszka @ 2018-10-02 14:06 UTC (permalink / raw)
To: [ext] Claudius Heine, Maxim Yu. Osipov, isar-users
On 02.10.18 16:02, [ext] Claudius Heine wrote:
> Also please avoid those binary settings like `ISAR_USE_CACHED_BASE_REPO`. That
> just bad design because it leads to every tiny setting having an own variable.
> Use FEATURE variables with entries like:
>
> `ISAR_BUILD_FEATURE = "cross-build use-cache ..."`
>
> Yes cross-builds should have been done like this as well IMO.
We do have a process for "properly" breaking the recipe API by now... ;)
Jan
--
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [PATCH 2/9] base-apt: Introduce base implementaiton
2018-10-02 12:19 ` [PATCH 2/9] base-apt: Introduce base implementaiton Maxim Yu. Osipov
@ 2018-10-02 14:20 ` Claudius Heine
0 siblings, 0 replies; 26+ messages in thread
From: Claudius Heine @ 2018-10-02 14:20 UTC (permalink / raw)
To: Maxim Yu. Osipov, isar-users
Hi Maxim,
On 10/2/18 2:19 PM, Maxim Yu. Osipov wrote:
> From: Alexander Smirnov <asmirnov@ilbers.de>
>
> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
> ---
> meta-isar/conf/layer.conf | 10 ++++---
> meta/recipes-devtools/base-apt/base-apt.bb | 31 ++++++++++++++++++++++
> .../base-apt/files/distributions.in | 3 +++
> 3 files changed, 40 insertions(+), 4 deletions(-)
> create mode 100644 meta/recipes-devtools/base-apt/base-apt.bb
> create mode 100644 meta/recipes-devtools/base-apt/files/distributions.in
>
> diff --git a/meta-isar/conf/layer.conf b/meta-isar/conf/layer.conf
> index cd42f06..ae1b3c5 100644
> --- a/meta-isar/conf/layer.conf
> +++ b/meta-isar/conf/layer.conf
> @@ -20,8 +20,10 @@ LAYERDIR_isar = "${LAYERDIR}"
> # Codename of the repository created by the caching class
> DEBDISTRONAME = "isar"
>
> -# Path to the Isar apt repository
> -DEPLOY_DIR_APT ?= "${DEPLOY_DIR}/apt"
> +# Isar apt repository paths
> +DEPLOY_DIR_APT ?= "${DEPLOY_DIR}/isar-apt/apt"
> +DEPLOY_DIR_DB ?= "${DEPLOY_DIR}/isar-apt/db"
>
> -# Path to the Isar databases used by `reprepro`
> -DEPLOY_DIR_DB ?= "${DEPLOY_DIR}/db"
> +# Base apt repository paths
> +BASE_APT_DIR ?= "${DEPLOY_DIR}/base-apt/apt"
> +BASE_APT_DB ?= "${DEPLOY_DIR}/base-apt/db"
I don't like those variable names. IMO it would be ok to break the
variable backwards compatibility here and unify that.
How about:
REPO_ISAR_DIR
REPO_ISAR_DB_DIR
REPO_CACHE_DIR
REPO_CACHE_DB_DIR
The db directory is reprepro specific so the main variable should point
the the correct repository path. IMO
> diff --git a/meta/recipes-devtools/base-apt/base-apt.bb b/meta/recipes-devtools/base-apt/base-apt.bb
> new file mode 100644
> index 0000000..6ff1164
> --- /dev/null
> +++ b/meta/recipes-devtools/base-apt/base-apt.bb
> @@ -0,0 +1,31 @@
> +# This software is a part of ISAR.
> +# Copyright (C) 2018 ilbers GmbH
> +
> +SRC_URI = "file://distributions.in"
> +
> +inherit base-apt-helper
> +
> +CACHE_CONF_DIR = "${BASE_APT_DIR}/${DISTRO_NAME}/conf"
> +do_cache_config[dirs] = "${CACHE_CONF_DIR}"
> +do_cache_config[stamp-extra-info] = "${DISTRO}"
> +do_cache_config[lockfiles] = "${BASE_APT_DIR}/isar.lock"
> +
> +# Generate reprepro config for current distro if it doesn't exist. Once it's
> +# generated, this task should do nothing.
> +do_cache_config() {
> + if [ ! -e "${CACHE_CONF_DIR}/distributions" ]; then
> + sed -e "s#{DISTRO_NAME}#"${DISTRO_SUITE}"#g" \
Mixing DISTRO_NAME and DISTRO_SUITE. They are different, try to be
consistent.
Cheers,
Claudius
> + ${WORKDIR}/distributions.in > ${CACHE_CONF_DIR}/distributions
> + fi
> +
> + path_cache="${BASE_APT_DIR}/${DISTRO_NAME}"
> + path_databases="${BASE_APT_DB}/${DISTRO_NAME}"
> +
> + if [ ! -d "${path_databases}" ]; then
> + reprepro -b ${path_cache} \
> + --dbdir ${path_databases} \
> + export ${DISTRO_SUITE}
> + fi
> +}
> +
> +addtask cache_config after do_build
> diff --git a/meta/recipes-devtools/base-apt/files/distributions.in b/meta/recipes-devtools/base-apt/files/distributions.in
> new file mode 100644
> index 0000000..cc82c57
> --- /dev/null
> +++ b/meta/recipes-devtools/base-apt/files/distributions.in
> @@ -0,0 +1,3 @@
> +Codename: {DISTRO_NAME}
> +Architectures: i386 armhf arm64 amd64 source
> +Components: main
>
--
DENX Software Engineering GmbH, Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [PATCH 3/9] isar-boot-strap: Add option to keep cache
2018-10-02 12:19 ` [PATCH 3/9] isar-boot-strap: Add option to keep cache Maxim Yu. Osipov
@ 2018-10-02 14:49 ` Claudius Heine
0 siblings, 0 replies; 26+ messages in thread
From: Claudius Heine @ 2018-10-02 14:49 UTC (permalink / raw)
To: Maxim Yu. Osipov, isar-users
Hi,
what also belongs to this patch are the options that force apt not to
remove anything from the cache like APT::Clean-Installed just to be save
that no automatic cleaning is done by apt.
Cheers,
Claudius
On 10/2/18 2:19 PM, Maxim Yu. Osipov wrote:
> From: Alexander Smirnov <asmirnov@ilbers.de>
>
> The default ISAR behavior assumes to remove all the packages
> from apt cache, but we need them to put into base-apt.
>
> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
> ---
> meta/classes/isar-bootstrap-helper.bbclass | 6 ++++++
> 1 file changed, 6 insertions(+)
>
> diff --git a/meta/classes/isar-bootstrap-helper.bbclass b/meta/classes/isar-bootstrap-helper.bbclass
> index 597f6b1..4a00cb0 100644
> --- a/meta/classes/isar-bootstrap-helper.bbclass
> +++ b/meta/classes/isar-bootstrap-helper.bbclass
> @@ -70,6 +70,7 @@ setup_root_file_system() {
> --fstab) FSTAB=$2; shift ;;
> --host-arch) ROOTFS_ARCH=${HOST_ARCH} ;;
> --host-distro) ROOTFS_DISTRO=${HOST_DISTRO} ;;
> + --keep-apt-cache) KEEP_APT_CACHE=1 ;;
> -*) bbfatal "$0: invalid option specified: $1" ;;
> *) break ;;
> esac
> @@ -131,6 +132,11 @@ setup_root_file_system() {
> /usr/bin/apt-get purge --yes $pkg
> done
> if [ ${CLEAN} ]; then
> + if [ ${KEEP_APT_CACHE} ]; then
> + mkdir -p ${WORKDIR}/apt_cache
> + sudo mv $(find $ROOTFSDIR/var/cache/apt -name '*.deb') ${WORKDIR}/apt_cache
> + sudo chown $USER ${WORKDIR}/apt_cache/*
> + fi
> sudo -E chroot "$ROOTFSDIR" \
> /usr/bin/apt-get autoremove --purge --yes
> sudo -E chroot "$ROOTFSDIR" \
>
--
DENX Software Engineering GmbH, Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [PATCH 9/9] doc: Creation of local apt repo caching upstream Debian packages
2018-10-02 14:02 ` Claudius Heine
2018-10-02 14:06 ` Jan Kiszka
@ 2018-10-04 9:03 ` Baurzhan Ismagulov
2018-10-05 12:09 ` Claudius Heine
1 sibling, 1 reply; 26+ messages in thread
From: Baurzhan Ismagulov @ 2018-10-04 9:03 UTC (permalink / raw)
To: isar-users
On Tue, Oct 02, 2018 at 04:02:54PM +0200, Claudius Heine wrote:
> > +# Uncomment this to enable use of cached base repository
> > +#ISAR_USE_CACHED_BASE_REPO ?= "1"
>
> No that doesn't make sense to me.
>
> In my opinion its very important to not have such flags or commands. Caching
> and reproducible builds should be the default operation, upgrading or
> removing the cache should be an extra step.
The first step is to provide the mechanism which is usable manually or
automatically. Defining policy and setting it as default is best done when the
feature has been stabilized; I'd be reluctant to slap a new feature as default
as soon as it is introduced. I'd suggest to address that in later steps. That
said, the option doesn't take it away from you -- just set it to true in your
configuration.
> Also please avoid those binary settings like `ISAR_USE_CACHED_BASE_REPO`.
> That just bad design because it leads to every tiny setting having an own
> variable. Use FEATURE variables with entries like:
>
> `ISAR_BUILD_FEATURE = "cross-build use-cache ..."`
>
> Yes cross-builds should have been done like this as well IMO.
Hmm, that's an interesting idea. Conceptually, I fail to see any difference
from boolean vars: One still has boolean feature names for every tiny setting.
I personally don't like features since they are difficult to remove from
default settings.
With kind regards,
Baurzhan.
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [PATCH 9/9] doc: Creation of local apt repo caching upstream Debian packages
2018-10-04 9:03 ` Baurzhan Ismagulov
@ 2018-10-05 12:09 ` Claudius Heine
2018-10-11 16:33 ` Baurzhan Ismagulov
0 siblings, 1 reply; 26+ messages in thread
From: Claudius Heine @ 2018-10-05 12:09 UTC (permalink / raw)
To: Baurzhan Ismagulov, isar-users
[-- Attachment #1: Type: text/plain, Size: 3248 bytes --]
Hi Baurzhan,
Quoting Baurzhan Ismagulov (2018-10-04 11:03:11)
> On Tue, Oct 02, 2018 at 04:02:54PM +0200, Claudius Heine wrote:
> > > +# Uncomment this to enable use of cached base repository
> > > +#ISAR_USE_CACHED_BASE_REPO ?= "1"
> >
> > No that doesn't make sense to me.
> >
> > In my opinion its very important to not have such flags or commands. Caching
> > and reproducible builds should be the default operation, upgrading or
> > removing the cache should be an extra step.
>
> The first step is to provide the mechanism which is usable manually or
> automatically. Defining policy and setting it as default is best done when the
> feature has been stabilized; I'd be reluctant to slap a new feature as default
> as soon as it is introduced. I'd suggest to address that in later steps. That
> said, the option doesn't take it away from you -- just set it to true in your
> configuration.
Ok, but then the machanism should be designed that way so that this
feature can be enabled all the time, and the build should work on the first run
with this setting enabled as well.
> > Also please avoid those binary settings like `ISAR_USE_CACHED_BASE_REPO`.
> > That just bad design because it leads to every tiny setting having an own
> > variable. Use FEATURE variables with entries like:
> >
> > `ISAR_BUILD_FEATURE = "cross-build use-cache ..."`
> >
> > Yes cross-builds should have been done like this as well IMO.
>
> Hmm, that's an interesting idea. Conceptually, I fail to see any difference
> from boolean vars: One still has boolean feature names for every tiny setting.
If you do 'bitbake -e ..' you see all used features in one variable
instead of searching for many different variables all over to find what
features are enabled.
> I personally don't like features since they are difficult to remove from
> default settings.
I don't know if I understand you correctly here. I think you meant that
removing an entry from a variable is difficult. If that is the case, then
you could just do:
ISAR_BUILD_FEATURE_remove = "use-cache"
As documented here: [1]
If you meant something different, please explain further.
Cheers,
Claudius
[1] https://www.yoctoproject.org/docs/current/bitbake-user-manual/bitbake-user-manual.html#removing-override-style-syntax
>
>
> With kind regards,
> Baurzhan.
>
> --
> You received this message because you are subscribed to the Google Groups "isar-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
> To post to this group, send email to isar-users@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/20181004090311.GA12890%40yssyq.m.ilbers.de.
> For more options, visit https://groups.google.com/d/optout.
--
DENX Software Engineering GmbH, Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de
PGP key: 6FF2 E59F 00C6 BC28 31D8 64C1 1173 CB19 9808 B153
Keyserver: hkp://pool.sks-keyservers.net
[-- Attachment #2: signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
-----BEGIN PGP SIGNATURE-----
iQIzBAABCAAdFiEEb/LlnwDGvCgx2GTBEXPLGZgIsVMFAlu3VFwACgkQEXPLGZgI
sVOfdxAAgdOSOg1SJJtiuIBva41RvkcX2rSbEGbqBvIpmerOXt12vZkTmQaj0erf
otXh+tjZD5GeU0YvARBNh+dGu3T+NFDqzS6p7AYYz2jr8gdWiMSLpnMC4UyMWDer
1J3dt5KHp/iQZSx7RCyhljqzdn5LMtiETufH7FZ4LfgzykcNrnuILrfvl3k7fdjO
uVLfHFp44elFx2dNe34AR3JRCEJU4AkaeLoVu5vRU+vaOm7cavcxo7aJRDD3RcXD
WEExtJUQy0Xdpl2ZKSVRCPBmflloNtSo/vQF6kNIfAWwaKsIIMG7HYdJPV+JgOP+
REoxOYw+z3BdJCV5ZhNrFC2HOV9tPLJUlHZO8Si91tQHbXmsQvV4xvMbYQKnBfU9
zDMEBUacPNYAH5CnGMGj1zje3LicWSB0HMTDObD6m7fXRn7snCGoB2CO41fRlIhc
LZWtFszIgrrfDk+CRoWLEGj3EzCrYqx5ykuk7q8XpmGy7iL4JXrY9mw60oKkNBGM
j9gLIdRsTECfDHMRBYDKB34V+OKGb4F07nGY9Xk4kNTWKM00jvFIpXQKQm73xWL2
mFiSRhylQJs6aZ9F6iOd2yoBDvr54IwxpPb3u6Fs9/IyuKBuHe+KeMwTs/BOt/Lh
ZGBXq2N2v7qmNyW5LI97YFPyNY/k4Ff71ChUZq7O5zXGzO2ENKg=
=hSnV
-----END PGP SIGNATURE-----
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [PATCH 9/9] doc: Creation of local apt repo caching upstream Debian packages
2018-10-05 12:09 ` Claudius Heine
@ 2018-10-11 16:33 ` Baurzhan Ismagulov
0 siblings, 0 replies; 26+ messages in thread
From: Baurzhan Ismagulov @ 2018-10-11 16:33 UTC (permalink / raw)
To: isar-users
On Fri, Oct 05, 2018 at 02:09:05PM +0200, Claudius Heine wrote:
> Ok, but then the machanism should be designed that way so that this
> feature can be enabled all the time, and the build should work on the first run
> with this setting enabled as well.
The mechanism doesn't preclude that. For this step, our goal is to have it
under user's control, explicitly leaving the policy you'd like to have to the
next steps. When we arrive there, we could decide whether
ISAR_USE_CACHED_BASE_REPO should become the default, and whether we need it to
be configurable. Till then, I'd really like to provide the choice for the user.
> If you do 'bitbake -e ..' you see all used features in one variable
> instead of searching for many different variables all over to find what
> features are enabled.
With a long enough list, it would end up being many different feature names all
over. I guess we'd have to play with that when we implement something and see
ourselves. For now, I don't see any decisive advantage.
> I don't know if I understand you correctly here. I think you meant that
> removing an entry from a variable is difficult. If that is the case, then
> you could just do:
>
> ISAR_BUILD_FEATURE_remove = "use-cache"
Yes, this is what I meant, thanks for the pointer. It does address the problem
of removing and, as I've read, produces also better results when appending.
With kind regards,
Baurzhan.
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [PATCH 5/9] isar-bootstrap: Make possible to reuse the cache
2018-10-02 12:19 ` [PATCH 5/9] isar-bootstrap: Make possible to reuse the cache Maxim Yu. Osipov
@ 2018-11-02 11:40 ` Henning Schild
0 siblings, 0 replies; 26+ messages in thread
From: Henning Schild @ 2018-11-02 11:40 UTC (permalink / raw)
To: Maxim Yu. Osipov; +Cc: isar-users
Am Tue, 2 Oct 2018 14:19:03 +0200
schrieb "Maxim Yu. Osipov" <mosipov@ilbers.de>:
> From: Alexander Smirnov <asmirnov@ilbers.de>
>
> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
> Signed-off-by: Maxim Yu. Osipov <mosipov@ilbers.de>
> ---
> .../isar-bootstrap/files/base-apt-sources | 1 +
> .../recipes-core/isar-bootstrap/isar-bootstrap.inc | 32
> ++++++++++++++++------ 2 files changed, 24 insertions(+), 9
> deletions(-) create mode 100644
> meta/recipes-core/isar-bootstrap/files/base-apt-sources
>
> diff --git a/meta/recipes-core/isar-bootstrap/files/base-apt-sources
> b/meta/recipes-core/isar-bootstrap/files/base-apt-sources new file
> mode 100644 index 0000000..594db56
> --- /dev/null
> +++ b/meta/recipes-core/isar-bootstrap/files/base-apt-sources
> @@ -0,0 +1 @@
> +deb file:///base-apt/debian {DISTRO} main
> diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
> b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc index
> cfad136..9b5a894 100644 ---
> a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc +++
> b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc @@ -12,18
> +12,24 @@ SRC_URI = " \ file://isar-apt.conf \
> file://isar-apt-fallback.conf \
> file://locale \
> - file://chroot-setup.sh"
> + file://chroot-setup.sh \
> + file://base-apt-sources"
> PV = "1.0"
>
> DEBOOTSTRAP ?= "qemu-debootstrap"
> ROOTFSDIR = "${WORKDIR}/rootfs"
> APTPREFS = "${WORKDIR}/apt-preferences"
> APTSRCS = "${WORKDIR}/apt-sources"
> +BASEAPTSRCS = "${WORKDIR}/base-apt-sources"
> APTKEYFILES = ""
> APTKEYRING = "${WORKDIR}/apt-keyring.gpg"
> DEBOOTSTRAP_KEYRING = ""
> DEPLOY_ISAR_BOOTSTRAP ?= ""
>
> +DISTRO_APT_PREMIRRORS ?= "${@ "http://ftp\.(\S+\.)?debian.org
> file:///${BASE_APT_DIR} \n" if
> bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')) else "" }"
> + +inherit base-apt-helper +
> python () {
> from urllib.parse import urlparse
> distro_apt_keys = d.getVar("DISTRO_APT_KEYS", False)
> @@ -171,6 +177,10 @@ isar_bootstrap() {
> esac
> shift
> done
> + debootstrap_args="--verbose --variant=minbase --include=locales "
> + if [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ]; then
> + debootstrap_args="$debootstrap_args --no-check-gpg"
Can this be removed now that the one problematic repo is marked
trusted? Say someone has more upstream repos and uses offline just for
debian, they probably still want gpg verification.
Henning
> + fi
> E="${@bb.utils.export_proxies(d)}"
> sudo -E flock "${ISAR_BOOTSTRAP_LOCK}" -c "\
> set -e
> @@ -181,9 +191,7 @@ isar_bootstrap() {
> rm -rf "${ROOTFSDIR}"
> fi
> if [ ${IS_HOST} ]; then
> - ${DEBOOTSTRAP} --verbose \
> - --variant=minbase \
> - --include=locales \
> + ${DEBOOTSTRAP} $debootstrap_args \
> ${@get_distro_components_argument(d,
> True)} \ ${DEBOOTSTRAP_KEYRING} \
> "${@get_distro_suite(d, True)}" \
> @@ -191,10 +199,8 @@ isar_bootstrap() {
> "${@get_distro_source(d, True)}"
>
> else
> - "${DEBOOTSTRAP}" --verbose \
> - --variant=minbase \
> + "${DEBOOTSTRAP}" $debootstrap_args \
> --arch="${DISTRO_ARCH}" \
> - --include=locales \
> ${@get_distro_components_argument(d,
> False)} \ ${DEBOOTSTRAP_KEYRING} \
> "${@get_distro_suite(d, False)}" \
> @@ -207,8 +213,16 @@ isar_bootstrap() {
> install -v -m644 "${APTPREFS}" \
> "${ROOTFSDIR}/etc/apt/preferences.d/bootstrap"
> mkdir -p "${ROOTFSDIR}/etc/apt/sources.list.d"
> - install -v -m644 "${APTSRCS}" \
> -
> "${ROOTFSDIR}/etc/apt/sources.list.d/bootstrap.list"
> + if [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ]; then
> + sed -i -e "s#{DISTRO}#"${DISTRO_SUITE}"#g"
> ${BASEAPTSRCS}
> + mkdir -p ${ROOTFSDIR}/base-apt
> + sudo mount --bind ${BASE_APT_DIR}
> ${ROOTFSDIR}/base-apt
> + install -v -m644 "${BASEAPTSRCS}" \
> +
> "${ROOTFSDIR}/etc/apt/sources.list.d/base-apt.list"
> + else
> + install -v -m644 "${APTSRCS}" \
> +
> "${ROOTFSDIR}/etc/apt/sources.list.d/bootstrap.list"
> + fi
> rm -f "${ROOTFSDIR}/etc/apt/sources.list"
> mkdir -p "${ROOTFSDIR}/etc/apt/apt.conf.d"
> install -v -m644 "${WORKDIR}/isar-apt.conf" \
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [PATCH 0/9] Introduce local apt repo to cache upstream debian packages for offline usage
2018-10-02 12:18 [PATCH 0/9] Introduce local apt repo to cache upstream debian packages for offline usage Maxim Yu. Osipov
` (8 preceding siblings ...)
2018-10-02 12:19 ` [PATCH 9/9] doc: Creation of local apt repo caching upstream Debian packages Maxim Yu. Osipov
@ 2018-11-02 12:00 ` Henning Schild
2018-11-09 8:04 ` Jan Kiszka
2018-11-04 10:07 ` Jan Kiszka
10 siblings, 1 reply; 26+ messages in thread
From: Henning Schild @ 2018-11-02 12:00 UTC (permalink / raw)
To: Maxim Yu. Osipov; +Cc: isar-users
Hi,
how does series compare to "apt-cacher-ng"? Looks to me like that tool
does something very similar, and it most likely deals with gpg and
other details with more care.
I might be wrong, but if that tool was never considered ... i think it
should be before inventing it again.
Henning
Am Tue, 2 Oct 2018 14:18:58 +0200
schrieb "Maxim Yu. Osipov" <mosipov@ilbers.de>:
> Hello everybody,
>
> This series by Alexander Smirnov,
>
> 1) Introduces dedicated local apt (base-apt) repo for upstream Debian
> packages
>
> 2) Caches in base-apt repo upstream Debian packages during image
> generation.
>
> 3) After this step, image can be built offline using only base-apt
> repo.
>
> Usage instructions can be found in the last patch.
>
> TODO:
> Sign the repo with gpg
>
> Kind regards,
> Maxim.
>
> Alexander Smirnov (8):
> base-apt: Add helper class
> base-apt: Introduce base implementaiton
> isar-boot-strap: Add option to keep cache
> image: Add cache_base_repo task
> isar-bootstrap: Make possible to reuse the cache
> buildchroot: Make it buildable from base-apt
> workaround: Use --allow-unauthenticated working with base-apt
> local.conf: Add option to use cached base repository
>
> Maxim Yu. Osipov (1):
> doc: Creation of local apt repo caching upstream Debian packages
>
> doc/user_manual.md | 39
> ++++++++++++++++ meta-isar/conf/layer.conf |
> 10 ++-- meta-isar/conf/local.conf.sample | 4 ++
> meta-isar/recipes-core/images/isar-image-base.bb | 3 +-
> meta/classes/base-apt-helper.bbclass | 53
> ++++++++++++++++++++++
> meta/classes/image.bbclass | 21 +++++++++
> meta/classes/isar-bootstrap-helper.bbclass | 13
> +++++- .../isar-bootstrap/files/base-apt-sources | 1
> + .../recipes-core/isar-bootstrap/isar-bootstrap.inc | 32
> +++++++++---- meta/recipes-devtools/base-apt/base-apt.bb | 31
> +++++++++++++ .../base-apt/files/distributions.in | 3
> ++ meta/recipes-devtools/buildchroot/files/deps.sh | 2 +- 12
> files changed, 196 insertions(+), 16 deletions(-) create mode 100644
> meta/classes/base-apt-helper.bbclass create mode 100644
> meta/recipes-core/isar-bootstrap/files/base-apt-sources create mode
> 100644 meta/recipes-devtools/base-apt/base-apt.bb create mode 100644
> meta/recipes-devtools/base-apt/files/distributions.in
>
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [PATCH 0/9] Introduce local apt repo to cache upstream debian packages for offline usage
2018-10-02 12:18 [PATCH 0/9] Introduce local apt repo to cache upstream debian packages for offline usage Maxim Yu. Osipov
` (9 preceding siblings ...)
2018-11-02 12:00 ` [PATCH 0/9] Introduce local apt repo to cache upstream debian packages for offline usage Henning Schild
@ 2018-11-04 10:07 ` Jan Kiszka
2018-11-04 20:20 ` Jan Kiszka
10 siblings, 1 reply; 26+ messages in thread
From: Jan Kiszka @ 2018-11-04 10:07 UTC (permalink / raw)
To: Maxim Yu. Osipov, isar-users
On 02.10.18 14:18, Maxim Yu. Osipov wrote:
> Hello everybody,
>
> This series by Alexander Smirnov,
>
> 1) Introduces dedicated local apt (base-apt) repo for upstream Debian packages
>
> 2) Caches in base-apt repo upstream Debian packages during image generation.
>
> 3) After this step, image can be built offline using only base-apt repo.
>
> Usage instructions can be found in the last patch.
>
> TODO:
> Sign the repo with gpg
There are more, in fact:
- enable support for cross-build: dependencies of buildchroot-target are
not cached which breaks the rebuild - probably related to the strange
way of creating the repo via a task. Maybe apt-cacher can help.
- move base_apt out of tmp - it belongs to downloads. That will also
make it easier to purge tmp completely in order trigger a rebuild.
- get rid of the control knobs, at least make them opt-out instead of
opt-in. I see no reason why the cache can't be created by default and
why it can't be used by default if it's there - we do the same with
other downloads.
I'm also curious if the apt-cacher-ng Henning mentioned can help to make things
simpler.
Thanks,
Jan
--
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [PATCH 0/9] Introduce local apt repo to cache upstream debian packages for offline usage
2018-11-04 10:07 ` Jan Kiszka
@ 2018-11-04 20:20 ` Jan Kiszka
0 siblings, 0 replies; 26+ messages in thread
From: Jan Kiszka @ 2018-11-04 20:20 UTC (permalink / raw)
To: Maxim Yu. Osipov, isar-users
On 04.11.18 11:07, Jan Kiszka wrote:> On 02.10.18 14:18, Maxim Yu. Osipov wrote:
>> Hello everybody,
>>
>> This series by Alexander Smirnov,
>>
>> 1) Introduces dedicated local apt (base-apt) repo for upstream Debian packages
>>
>> 2) Caches in base-apt repo upstream Debian packages during image generation.
>>
>> 3) After this step, image can be built offline using only base-apt repo.
>>
>> Usage instructions can be found in the last patch.
>> TODO:
>> Sign the repo with gpg
>
> There are more, in fact:
>
> - enable support for cross-build: dependencies of buildchroot-target are
> not cached which breaks the rebuild - probably related to the strange
> way of creating the repo via a task. Maybe apt-cacher can help.
Played a bit more with it, and it's not cross-build:
diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index ad19f3d..d033cf5 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -143,7 +143,7 @@ do_cache_base_repo() {
fi
}
-addtask cache_base_repo after do_rootfs
+addtask cache_base_repo after do_rootfs do_install_imager_deps
# Imager are expected to run natively, thus will use the target buildchroot.
ISAR_CROSS_COMPILE = "0"
I.e., we simply ignored the dependencies of the imager. Cross-building
is fine now as well.
>
> - move base_apt out of tmp - it belongs to downloads. That will also
> make it easier to purge tmp completely in order trigger a rebuild.
>
> - get rid of the control knobs, at least make them opt-out instead of
> opt-in. I see no reason why the cache can't be created by default and
> why it can't be used by default if it's there - we do the same with
> other downloads.
- make sure you can switch between cached and non-cached build on-the-
fly, without requiring to purge all build results. That's important
when you want to add packages that are not yet cached, at least as
long as we allow to control the mode manually.
>
> I'm also curious if the apt-cacher-ng Henning mentioned can help to make things
> simpler.
>
> Thanks,
> Jan
>
Jan
--
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [PATCH 0/9] Introduce local apt repo to cache upstream debian packages for offline usage
2018-11-02 12:00 ` [PATCH 0/9] Introduce local apt repo to cache upstream debian packages for offline usage Henning Schild
@ 2018-11-09 8:04 ` Jan Kiszka
2018-11-09 9:14 ` Baurzhan Ismagulov
0 siblings, 1 reply; 26+ messages in thread
From: Jan Kiszka @ 2018-11-09 8:04 UTC (permalink / raw)
To: Maxim Yu. Osipov, Baurzhan Ismagulov; +Cc: [ext] Henning Schild, isar-users
On 02.11.18 13:00, [ext] Henning Schild wrote:
> Hi,
>
> how does series compare to "apt-cacher-ng"? Looks to me like that tool
> does something very similar, and it most likely deals with gpg and
> other details with more care.
> I might be wrong, but if that tool was never considered ... i think it
> should be before inventing it again.
Could someone comment on this?
Thanks,
Jan
--
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [PATCH 0/9] Introduce local apt repo to cache upstream debian packages for offline usage
2018-11-09 8:04 ` Jan Kiszka
@ 2018-11-09 9:14 ` Baurzhan Ismagulov
2018-11-29 12:53 ` Henning Schild
0 siblings, 1 reply; 26+ messages in thread
From: Baurzhan Ismagulov @ 2018-11-09 9:14 UTC (permalink / raw)
To: isar-users
On Fri, Nov 09, 2018 at 09:04:43AM +0100, Jan Kiszka wrote:
> > how does series compare to "apt-cacher-ng"? Looks to me like that tool
> > does something very similar, and it most likely deals with gpg and
> > other details with more care.
> > I might be wrong, but if that tool was never considered ... i think it
> > should be before inventing it again.
We had discussed the proxy approach last year. It doesn't support apt's
pluggable fetch schemes and is, being a pure network tool, completely
apt-agnostic. One can get the initial mirroring done with it, but implementing
further use cases like updating the cache without deleting it or selectively
picking just a few packages and, optionally, their dependencies requires apt
introspection. I think adding and maintaining that in a C++ tool designed for a
different purpose is not the way to go. That is why we should be reusing apt
code to stay up-to-date with Debian. libapt + python-apt provide what we need,
and other projects have good experience with that. aptly claims selective
update support, so that is also on our list.
With kind regards,
Baurzhan.
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [PATCH 0/9] Introduce local apt repo to cache upstream debian packages for offline usage
2018-11-09 9:14 ` Baurzhan Ismagulov
@ 2018-11-29 12:53 ` Henning Schild
0 siblings, 0 replies; 26+ messages in thread
From: Henning Schild @ 2018-11-29 12:53 UTC (permalink / raw)
To: Baurzhan Ismagulov; +Cc: isar-users
Am Fri, 9 Nov 2018 10:14:18 +0100
schrieb Baurzhan Ismagulov <ibr@radix50.net>:
> On Fri, Nov 09, 2018 at 09:04:43AM +0100, Jan Kiszka wrote:
> > > how does series compare to "apt-cacher-ng"? Looks to me like that
> > > tool does something very similar, and it most likely deals with
> > > gpg and other details with more care.
> > > I might be wrong, but if that tool was never considered ... i
> > > think it should be before inventing it again.
>
> We had discussed the proxy approach last year. It doesn't support
> apt's pluggable fetch schemes and is, being a pure network tool,
> completely apt-agnostic.
That can also be seen as a benefit. It can in fact be replaced by any
caching proxy, squid ... you name it.
> One can get the initial mirroring done with
> it, but implementing further use cases like updating the cache
> without deleting it or selectively picking just a few packages and,
> optionally, their dependencies requires apt introspection.
Valid point, but is that a strong enough feature to roll our own?
Flushing the cache and rewarming it from scratch does not take too much
space and time. And you do not risk making a mistake. And the
implementation we have in Isar at the moment does not support that
either.
> I think
> adding and maintaining that in a C++ tool designed for a different
> purpose is not the way to go.
We would be using that tool, or any other http/ftp proxy. The language
it is written in or who maintains it does not matter all. If we do not
like it we change .. easy. Implementing our own, we have to maintain
it ...
> That is why we should be reusing apt
> code to stay up-to-date with Debian. libapt + python-apt provide what
> we need, and other projects have good experience with that. aptly
> claims selective update support, so that is also on our list.
Ok maybe aptly. If Isar itself was completely agnostic, people could
choose their cache. Some might go for apt-agnostic some for plain http.
Isar itself could carry documentation and suggestions on how to set
that up.
Henning
> With kind regards,
> Baurzhan.
>
^ permalink raw reply [flat|nested] 26+ messages in thread
end of thread, other threads:[~2018-11-29 12:53 UTC | newest]
Thread overview: 26+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-10-02 12:18 [PATCH 0/9] Introduce local apt repo to cache upstream debian packages for offline usage Maxim Yu. Osipov
2018-10-02 12:18 ` [PATCH 1/9] base-apt: Add helper class Maxim Yu. Osipov
2018-10-02 13:39 ` Claudius Heine
2018-10-02 12:19 ` [PATCH 2/9] base-apt: Introduce base implementaiton Maxim Yu. Osipov
2018-10-02 14:20 ` Claudius Heine
2018-10-02 12:19 ` [PATCH 3/9] isar-boot-strap: Add option to keep cache Maxim Yu. Osipov
2018-10-02 14:49 ` Claudius Heine
2018-10-02 12:19 ` [PATCH 4/9] image: Add cache_base_repo task Maxim Yu. Osipov
2018-10-02 12:19 ` [PATCH 5/9] isar-bootstrap: Make possible to reuse the cache Maxim Yu. Osipov
2018-11-02 11:40 ` Henning Schild
2018-10-02 12:19 ` [PATCH 6/9] buildchroot: Make it buildable from base-apt Maxim Yu. Osipov
2018-10-02 12:19 ` [PATCH 7/9] workaround: Use --allow-unauthenticated working with base-apt Maxim Yu. Osipov
2018-10-02 12:19 ` [PATCH 8/9] local.conf: Add option to use cached base repository Maxim Yu. Osipov
2018-10-02 12:19 ` [PATCH 9/9] doc: Creation of local apt repo caching upstream Debian packages Maxim Yu. Osipov
2018-10-02 14:02 ` Claudius Heine
2018-10-02 14:06 ` Jan Kiszka
2018-10-04 9:03 ` Baurzhan Ismagulov
2018-10-05 12:09 ` Claudius Heine
2018-10-11 16:33 ` Baurzhan Ismagulov
2018-10-02 14:05 ` Claudius Heine
2018-11-02 12:00 ` [PATCH 0/9] Introduce local apt repo to cache upstream debian packages for offline usage Henning Schild
2018-11-09 8:04 ` Jan Kiszka
2018-11-09 9:14 ` Baurzhan Ismagulov
2018-11-29 12:53 ` Henning Schild
2018-11-04 10:07 ` Jan Kiszka
2018-11-04 20:20 ` Jan Kiszka
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox