* [PATCH v2 01/10] base-apt: Add helper class
2018-10-11 16:52 [PATCH v2 00/10] Introduce local apt repo to cache upstream debian packages for offline usage Maxim Yu. Osipov
@ 2018-10-11 16:52 ` Maxim Yu. Osipov
2018-10-23 16:01 ` Henning Schild
2018-10-11 16:52 ` [PATCH v2 02/10] meta: Unify path names to local repositories Maxim Yu. Osipov
` (11 subsequent siblings)
12 siblings, 1 reply; 25+ messages in thread
From: Maxim Yu. Osipov @ 2018-10-11 16:52 UTC (permalink / raw)
To: isar-users
From: Alexander Smirnov <asmirnov@ilbers.de>
base-apt intended to store original upstream debs to re-use them
later offline. This class helps to populate base-apt with the packages
used during build.
Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
Reviewed-by: Claudius Heine <ch@denx.de>
Signed-off-by: Maxim Yu. Osipov <mosipov@ilbers.de>
---
meta-isar/conf/layer.conf | 4 +++
meta/classes/base-apt-helper.bbclass | 54 ++++++++++++++++++++++++++++++++++++
2 files changed, 58 insertions(+)
create mode 100644 meta/classes/base-apt-helper.bbclass
diff --git a/meta-isar/conf/layer.conf b/meta-isar/conf/layer.conf
index cd42f06..22b2ff2 100644
--- a/meta-isar/conf/layer.conf
+++ b/meta-isar/conf/layer.conf
@@ -25,3 +25,7 @@ DEPLOY_DIR_APT ?= "${DEPLOY_DIR}/apt"
# Path to the Isar databases used by `reprepro`
DEPLOY_DIR_DB ?= "${DEPLOY_DIR}/db"
+
+# Base apt repository paths
+REPO_BASE_DIR ?= "${DEPLOY_DIR}/base-apt/apt"
+REPO_BASE_DB_DIR ?= "${DEPLOY_DIR}/base-apt/db"
diff --git a/meta/classes/base-apt-helper.bbclass b/meta/classes/base-apt-helper.bbclass
new file mode 100644
index 0000000..9c03a7e
--- /dev/null
+++ b/meta/classes/base-apt-helper.bbclass
@@ -0,0 +1,54 @@
+# This software is a part of ISAR.
+# Copyright (C) 2018 ilbers GmbH
+
+DISTRO_NAME ?= "${@ d.getVar('DISTRO', True).split('-')[0]}"
+DISTRO_SUITE ?= "${@ d.getVar('DISTRO', True).split('-')[1]}"
+
+populate_base_apt() {
+ search_dir=$1
+
+ find $search_dir -name '*.deb' | while read package; do
+ # NOTE: due to packages stored by reprepro are not modified, we can
+ # use search by filename to check if package is already in repo. In
+ # addition, m5sums could be compared to ensure, that package is the
+ # same and should not be overwritten. This method is easier and more
+ # robust than querying reprepro by name.
+
+ # Check if this package is taken from Isar-apt, if so - ingore it.
+ base_name=${package##*/}
+ isar_package=$(find ${DEPLOY_DIR_APT}/${DISTRO} -name $base_name)
+ if [ -n "$isar_package" ]; then
+ # Check if MD5 sums are identical. This helps to avoid the case
+ # when packages is overridden from another repo.
+ md1=$(md5sum $package | cut -d ' ' -f 1)
+ md2=$(md5sum $isar_package | cut -d ' ' -f 1)
+ if [ "$md1" = "$md2" ]; then
+ continue
+ fi
+ fi
+
+ # Check if this package is already in base-apt
+ isar_package=$(find ${REPO_BASE_DIR}/${DISTRO_NAME} -name $base_name)
+ if [ -n "$isar_package" ]; then
+ md1=$(md5sum $package | cut -d ' ' -f 1)
+ md2=$(md5sum $isar_package | cut -d ' ' -f 1)
+ if [ "$md1" = "$md2" ]; then
+ continue
+ fi
+
+ # md5sum differs, so remove the package from base-apt
+ name=$($base_name | cut -d '_' -f 1)
+ reprepro -b ${REPO_BASE_DIR}/${DISTRO_NAME} \
+ --dbdir ${REPO_BASE_DB_DIR}/${DISTRO_NAME} \
+ -C main -A ${DISTRO_ARCH} \
+ remove ${DISTRO_SUITE} \
+ $name
+ fi
+
+ reprepro -b ${REPO_BASE_DIR}/${DISTRO_NAME} \
+ --dbdir ${REPO_BASE_DB_DIR}/${DISTRO_NAME} \
+ -C main \
+ includedeb ${DISTRO_SUITE} \
+ $package
+ done
+}
--
2.11.0
^ permalink raw reply [flat|nested] 25+ messages in thread* Re: [PATCH v2 01/10] base-apt: Add helper class
2018-10-11 16:52 ` [PATCH v2 01/10] base-apt: Add helper class Maxim Yu. Osipov
@ 2018-10-23 16:01 ` Henning Schild
0 siblings, 0 replies; 25+ messages in thread
From: Henning Schild @ 2018-10-23 16:01 UTC (permalink / raw)
To: Maxim Yu. Osipov; +Cc: isar-users
Am Thu, 11 Oct 2018 18:52:56 +0200
schrieb "Maxim Yu. Osipov" <mosipov@ilbers.de>:
> From: Alexander Smirnov <asmirnov@ilbers.de>
>
> base-apt intended to store original upstream debs to re-use them
> later offline. This class helps to populate base-apt with the packages
> used during build.
>
> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
> Reviewed-by: Claudius Heine <ch@denx.de>
> Signed-off-by: Maxim Yu. Osipov <mosipov@ilbers.de>
> ---
> meta-isar/conf/layer.conf | 4 +++
> meta/classes/base-apt-helper.bbclass | 54
> ++++++++++++++++++++++++++++++++++++ 2 files changed, 58 insertions(+)
> create mode 100644 meta/classes/base-apt-helper.bbclass
>
> diff --git a/meta-isar/conf/layer.conf b/meta-isar/conf/layer.conf
> index cd42f06..22b2ff2 100644
> --- a/meta-isar/conf/layer.conf
> +++ b/meta-isar/conf/layer.conf
> @@ -25,3 +25,7 @@ DEPLOY_DIR_APT ?= "${DEPLOY_DIR}/apt"
>
> # Path to the Isar databases used by `reprepro`
> DEPLOY_DIR_DB ?= "${DEPLOY_DIR}/db"
> +
> +# Base apt repository paths
> +REPO_BASE_DIR ?= "${DEPLOY_DIR}/base-apt/apt"
> +REPO_BASE_DB_DIR ?= "${DEPLOY_DIR}/base-apt/db"
> diff --git a/meta/classes/base-apt-helper.bbclass
> b/meta/classes/base-apt-helper.bbclass new file mode 100644
> index 0000000..9c03a7e
> --- /dev/null
> +++ b/meta/classes/base-apt-helper.bbclass
> @@ -0,0 +1,54 @@
> +# This software is a part of ISAR.
> +# Copyright (C) 2018 ilbers GmbH
> +
> +DISTRO_NAME ?= "${@ d.getVar('DISTRO', True).split('-')[0]}"
> +DISTRO_SUITE ?= "${@ d.getVar('DISTRO', True).split('-')[1]}"
> +
> +populate_base_apt() {
> + search_dir=$1
> +
> + find $search_dir -name '*.deb' | while read package; do
> + # NOTE: due to packages stored by reprepro are not modified,
> we can
> + # use search by filename to check if package is already in
> repo. In
> + # addition, m5sums could be compared to ensure, that package
> is the
> + # same and should not be overwritten. This method is easier
> and more
> + # robust than querying reprepro by name.
> +
> + # Check if this package is taken from Isar-apt, if so -
> ingore it.
> + base_name=${package##*/}
> + isar_package=$(find ${DEPLOY_DIR_APT}/${DISTRO} -name
> $base_name)
> + if [ -n "$isar_package" ]; then
> + # Check if MD5 sums are identical. This helps to avoid
> the case
> + # when packages is overridden from another repo.
> + md1=$(md5sum $package | cut -d ' ' -f 1)
> + md2=$(md5sum $isar_package | cut -d ' ' -f 1)
> + if [ "$md1" = "$md2" ]; then
> + continue
> + fi
This block should maybe be a function we reuse ...
> + fi
> +
> + # Check if this package is already in base-apt
> + isar_package=$(find ${REPO_BASE_DIR}/${DISTRO_NAME} -name
> $base_name)
> + if [ -n "$isar_package" ]; then
> + md1=$(md5sum $package | cut -d ' ' -f 1)
> + md2=$(md5sum $isar_package | cut -d ' ' -f 1)
> + if [ "$md1" = "$md2" ]; then
> + continue
> + fi
here.
Henning
> + # md5sum differs, so remove the package from base-apt
> + name=$($base_name | cut -d '_' -f 1)
> + reprepro -b ${REPO_BASE_DIR}/${DISTRO_NAME} \
> + --dbdir ${REPO_BASE_DB_DIR}/${DISTRO_NAME} \
> + -C main -A ${DISTRO_ARCH} \
> + remove ${DISTRO_SUITE} \
> + $name
> + fi
> +
> + reprepro -b ${REPO_BASE_DIR}/${DISTRO_NAME} \
> + --dbdir ${REPO_BASE_DB_DIR}/${DISTRO_NAME} \
> + -C main \
> + includedeb ${DISTRO_SUITE} \
> + $package
> + done
> +}
^ permalink raw reply [flat|nested] 25+ messages in thread
* [PATCH v2 02/10] meta: Unify path names to local repositories
2018-10-11 16:52 [PATCH v2 00/10] Introduce local apt repo to cache upstream debian packages for offline usage Maxim Yu. Osipov
2018-10-11 16:52 ` [PATCH v2 01/10] base-apt: Add helper class Maxim Yu. Osipov
@ 2018-10-11 16:52 ` Maxim Yu. Osipov
2018-10-11 16:52 ` [PATCH v2 03/10] base-apt: Introduce base implementation Maxim Yu. Osipov
` (10 subsequent siblings)
12 siblings, 0 replies; 25+ messages in thread
From: Maxim Yu. Osipov @ 2018-10-11 16:52 UTC (permalink / raw)
To: isar-users
Rename DEPLOY_DIR_APT, DEPLOY_DIR_APT to REPO_ISAR_DIR and
REPO_ISAR_DB_DIR correspondingly to unify with cached base
repository names REPO_BASE_DIR and REPO_BASE_DB_DIR.
Suggested-by: Claudius Heine <ch@denx.de>
Signed-off-by: Maxim Yu. Osipov <mosipov@ilbers.de>
---
meta-isar/conf/layer.conf | 8 +++-----
meta/classes/base-apt-helper.bbclass | 2 +-
meta/classes/buildchroot.bbclass | 2 +-
meta/classes/dpkg-base.bbclass | 10 +++++-----
meta/classes/dpkg.bbclass | 2 +-
meta/classes/image.bbclass | 4 ++--
meta/classes/isar-bootstrap-helper.bbclass | 4 ++--
meta/recipes-devtools/isar-apt/isar-apt.bb | 8 ++++----
meta/recipes-kernel/linux/linux-custom.inc | 2 +-
9 files changed, 20 insertions(+), 22 deletions(-)
diff --git a/meta-isar/conf/layer.conf b/meta-isar/conf/layer.conf
index 22b2ff2..4aa1cf1 100644
--- a/meta-isar/conf/layer.conf
+++ b/meta-isar/conf/layer.conf
@@ -20,11 +20,9 @@ LAYERDIR_isar = "${LAYERDIR}"
# Codename of the repository created by the caching class
DEBDISTRONAME = "isar"
-# Path to the Isar apt repository
-DEPLOY_DIR_APT ?= "${DEPLOY_DIR}/apt"
-
-# Path to the Isar databases used by `reprepro`
-DEPLOY_DIR_DB ?= "${DEPLOY_DIR}/db"
+# Isar apt repository paths
+REPO_ISAR_DIR ?= "${DEPLOY_DIR}/isar-apt/apt"
+REPO_ISAR_DB_DIR ?= "${DEPLOY_DIR}/isar-apt/db"
# Base apt repository paths
REPO_BASE_DIR ?= "${DEPLOY_DIR}/base-apt/apt"
diff --git a/meta/classes/base-apt-helper.bbclass b/meta/classes/base-apt-helper.bbclass
index 9c03a7e..ff4a3d5 100644
--- a/meta/classes/base-apt-helper.bbclass
+++ b/meta/classes/base-apt-helper.bbclass
@@ -16,7 +16,7 @@ populate_base_apt() {
# Check if this package is taken from Isar-apt, if so - ingore it.
base_name=${package##*/}
- isar_package=$(find ${DEPLOY_DIR_APT}/${DISTRO} -name $base_name)
+ isar_package=$(find ${REPO_ISAR_DIR}/${DISTRO} -name $base_name)
if [ -n "$isar_package" ]; then
# Check if MD5 sums are identical. This helps to avoid the case
# when packages is overridden from another repo.
diff --git a/meta/classes/buildchroot.bbclass b/meta/classes/buildchroot.bbclass
index f67335e..870d27c 100644
--- a/meta/classes/buildchroot.bbclass
+++ b/meta/classes/buildchroot.bbclass
@@ -25,7 +25,7 @@ buildchroot_do_mounts() {
sudo flock ${MOUNT_LOCKFILE} -c ' \
set -e
if ! grep -q ${BUILDCHROOT_DIR}/isar-apt /proc/mounts; then
- mount --bind ${DEPLOY_DIR_APT}/${DISTRO} ${BUILDCHROOT_DIR}/isar-apt
+ mount --bind ${REPO_ISAR_DIR}/${DISTRO} ${BUILDCHROOT_DIR}/isar-apt
mount --bind ${DL_DIR} ${BUILDCHROOT_DIR}/downloads
mount -t devtmpfs -o mode=0755,nosuid devtmpfs ${BUILDCHROOT_DIR}/dev
mount -t proc none ${BUILDCHROOT_DIR}/proc
diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index edd111b..f1b127c 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -76,8 +76,8 @@ CLEANFUNCS += "repo_clean"
repo_clean() {
PACKAGES=$(cd ${S}/..; ls *.deb | sed 's/\([^_]*\).*/\1/')
if [ -n "${PACKAGES}" ]; then
- reprepro -b ${DEPLOY_DIR_APT}/${DISTRO} \
- --dbdir ${DEPLOY_DIR_DB}/${DISTRO} \
+ reprepro -b ${REPO_ISAR_DIR}/${DISTRO} \
+ --dbdir ${REPO_ISAR_DB_DIR}/${DISTRO} \
-C main -A ${DISTRO_ARCH} \
remove ${DEBDISTRONAME} \
${PACKAGES}
@@ -87,8 +87,8 @@ repo_clean() {
# Install package to Isar-apt
do_deploy_deb() {
repo_clean
- reprepro -b ${DEPLOY_DIR_APT}/${DISTRO} \
- --dbdir ${DEPLOY_DIR_DB}/${DISTRO} \
+ reprepro -b ${REPO_ISAR_DIR}/${DISTRO} \
+ --dbdir ${REPO_ISAR_DB_DIR}/${DISTRO} \
-C main \
includedeb ${DEBDISTRONAME} \
${S}/../*.deb
@@ -96,6 +96,6 @@ do_deploy_deb() {
addtask deploy_deb after do_build
do_deploy_deb[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
-do_deploy_deb[lockfiles] = "${DEPLOY_DIR_APT}/isar.lock"
+do_deploy_deb[lockfiles] = "${REPO_ISAR_DIR}/isar.lock"
do_deploy_deb[depends] = "isar-apt:do_cache_config"
do_deploy_deb[dirs] = "${S}"
diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
index da0b40d..f74c9c9 100644
--- a/meta/classes/dpkg.bbclass
+++ b/meta/classes/dpkg.bbclass
@@ -13,7 +13,7 @@ do_install_builddeps() {
addtask install_builddeps after do_prepare_build before do_build
# apt and reprepro may not run in parallel, acquire the Isar lock
-do_install_builddeps[lockfiles] += "${DEPLOY_DIR_APT}/isar.lock"
+do_install_builddeps[lockfiles] += "${REPO_ISAR_DIR}/isar.lock"
do_install_builddeps[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
# Build package from sources using build script
diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index ea8cbf5..e948dea 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -102,7 +102,7 @@ SDKCHROOT_DIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/sdkchroot-${HOST_DISTRO
do_populate_sdk() {
# Copy isar-apt with deployed Isar packages
- sudo cp -Trpfx ${DEPLOY_DIR_APT}/${DISTRO} ${SDKCHROOT_DIR}/rootfs/isar-apt
+ sudo cp -Trpfx ${REPO_ISAR_DIR}/${DISTRO} ${SDKCHROOT_DIR}/rootfs/isar-apt
# Purge apt cache to make image slimmer
sudo rm -rf ${SDKCHROOT_DIR}/rootfs/var/cache/apt/*
@@ -158,7 +158,7 @@ do_install_imager_deps() {
do_install_imager_deps[depends] = "buildchroot-target:do_build"
do_install_imager_deps[deptask] = "do_deploy_deb"
-do_install_imager_deps[lockfiles] += "${DEPLOY_DIR_APT}/isar.lock"
+do_install_imager_deps[lockfiles] += "${REPO_ISAR_DIR}/isar.lock"
do_install_imager_deps[stamp-extra-info] = "${DISTRO}-${MACHINE}"
addtask install_imager_deps before do_build
diff --git a/meta/classes/isar-bootstrap-helper.bbclass b/meta/classes/isar-bootstrap-helper.bbclass
index 597f6b1..f046216 100644
--- a/meta/classes/isar-bootstrap-helper.bbclass
+++ b/meta/classes/isar-bootstrap-helper.bbclass
@@ -93,9 +93,9 @@ setup_root_file_system() {
sudo tee "$ROOTFSDIR/etc/apt/preferences.d/isar" >/dev/null
if [ ${COPYISARAPT} ]; then
- sudo cp -Trpfx ${DEPLOY_DIR_APT}/${DISTRO} $ROOTFSDIR/isar-apt
+ sudo cp -Trpfx ${REPO_ISAR_DIR}/${DISTRO} $ROOTFSDIR/isar-apt
else
- sudo mount --bind ${DEPLOY_DIR_APT}/${DISTRO} $ROOTFSDIR/isar-apt
+ sudo mount --bind ${REPO_ISAR_DIR}/${DISTRO} $ROOTFSDIR/isar-apt
fi
sudo mount -t devtmpfs -o mode=0755,nosuid devtmpfs $ROOTFSDIR/dev
sudo mount -t proc none $ROOTFSDIR/proc
diff --git a/meta/recipes-devtools/isar-apt/isar-apt.bb b/meta/recipes-devtools/isar-apt/isar-apt.bb
index 9c31d12..a959691 100644
--- a/meta/recipes-devtools/isar-apt/isar-apt.bb
+++ b/meta/recipes-devtools/isar-apt/isar-apt.bb
@@ -3,10 +3,10 @@
SRC_URI = "file://distributions.in"
-CACHE_CONF_DIR = "${DEPLOY_DIR_APT}/${DISTRO}/conf"
+CACHE_CONF_DIR = "${REPO_ISAR_DIR}/${DISTRO}/conf"
do_cache_config[dirs] = "${CACHE_CONF_DIR}"
do_cache_config[stamp-extra-info] = "${DISTRO}"
-do_cache_config[lockfiles] = "${DEPLOY_DIR_APT}/isar.lock"
+do_cache_config[lockfiles] = "${REPO_ISAR_DIR}/isar.lock"
# Generate reprepro config for current distro if it doesn't exist. Once it's
# generated, this task should do nothing.
@@ -16,8 +16,8 @@ do_cache_config() {
${WORKDIR}/distributions.in > ${CACHE_CONF_DIR}/distributions
fi
- path_cache="${DEPLOY_DIR_APT}/${DISTRO}"
- path_databases="${DEPLOY_DIR_DB}/${DISTRO}"
+ path_cache="${REPO_ISAR_DIR}/${DISTRO}"
+ path_databases="${REPO_ISAR_DB_DIR}/${DISTRO}"
if [ ! -d "${path_databases}" ]; then
reprepro -b ${path_cache} \
diff --git a/meta/recipes-kernel/linux/linux-custom.inc b/meta/recipes-kernel/linux/linux-custom.inc
index 7c6b624..8c89637 100644
--- a/meta/recipes-kernel/linux/linux-custom.inc
+++ b/meta/recipes-kernel/linux/linux-custom.inc
@@ -36,7 +36,7 @@ do_install_builddeps() {
addtask install_builddeps after do_prepare_build before do_build
# apt and reprepro may not run in parallel, acquire the Isar lock
-do_install_builddeps[lockfiles] += "${DEPLOY_DIR_APT}/isar.lock"
+do_install_builddeps[lockfiles] += "${REPO_ISAR_DIR}/isar.lock"
do_install_builddeps[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
dpkg_runbuild() {
--
2.11.0
^ permalink raw reply [flat|nested] 25+ messages in thread* [PATCH v2 03/10] base-apt: Introduce base implementation
2018-10-11 16:52 [PATCH v2 00/10] Introduce local apt repo to cache upstream debian packages for offline usage Maxim Yu. Osipov
2018-10-11 16:52 ` [PATCH v2 01/10] base-apt: Add helper class Maxim Yu. Osipov
2018-10-11 16:52 ` [PATCH v2 02/10] meta: Unify path names to local repositories Maxim Yu. Osipov
@ 2018-10-11 16:52 ` Maxim Yu. Osipov
2018-10-11 16:52 ` [PATCH v2 04/10] isar-boot-strap: Add option to keep cache Maxim Yu. Osipov
` (9 subsequent siblings)
12 siblings, 0 replies; 25+ messages in thread
From: Maxim Yu. Osipov @ 2018-10-11 16:52 UTC (permalink / raw)
To: isar-users
From: Alexander Smirnov <asmirnov@ilbers.de>
Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
Signed-off-by: Maxim Yu. Osipov <mosipov@ilbers.de>
---
meta/recipes-devtools/base-apt/base-apt.bb | 31 ++++++++++++++++++++++
.../base-apt/files/distributions.in | 3 +++
2 files changed, 34 insertions(+)
create mode 100644 meta/recipes-devtools/base-apt/base-apt.bb
create mode 100644 meta/recipes-devtools/base-apt/files/distributions.in
diff --git a/meta/recipes-devtools/base-apt/base-apt.bb b/meta/recipes-devtools/base-apt/base-apt.bb
new file mode 100644
index 0000000..2370905
--- /dev/null
+++ b/meta/recipes-devtools/base-apt/base-apt.bb
@@ -0,0 +1,31 @@
+# This software is a part of ISAR.
+# Copyright (C) 2018 ilbers GmbH
+
+SRC_URI = "file://distributions.in"
+
+inherit base-apt-helper
+
+CACHE_CONF_DIR = "${REPO_BASE_DIR}/${DISTRO_NAME}/conf"
+do_cache_config[dirs] = "${CACHE_CONF_DIR}"
+do_cache_config[stamp-extra-info] = "${DISTRO}"
+do_cache_config[lockfiles] = "${REPO_BASE_DIR}/isar.lock"
+
+# Generate reprepro config for current distro if it doesn't exist. Once it's
+# generated, this task should do nothing.
+do_cache_config() {
+ if [ ! -e "${CACHE_CONF_DIR}/distributions" ]; then
+ sed -e "s#{DISTRO_NAME}#"${DISTRO_SUITE}"#g" \
+ ${WORKDIR}/distributions.in > ${CACHE_CONF_DIR}/distributions
+ fi
+
+ path_cache="${REPO_BASE_DIR}/${DISTRO_NAME}"
+ path_databases="${REPO_BASE_DB_DIR}/${DISTRO_NAME}"
+
+ if [ ! -d "${path_databases}" ]; then
+ reprepro -b ${path_cache} \
+ --dbdir ${path_databases} \
+ export ${DISTRO_SUITE}
+ fi
+}
+
+addtask cache_config after do_build
diff --git a/meta/recipes-devtools/base-apt/files/distributions.in b/meta/recipes-devtools/base-apt/files/distributions.in
new file mode 100644
index 0000000..cc82c57
--- /dev/null
+++ b/meta/recipes-devtools/base-apt/files/distributions.in
@@ -0,0 +1,3 @@
+Codename: {DISTRO_NAME}
+Architectures: i386 armhf arm64 amd64 source
+Components: main
--
2.11.0
^ permalink raw reply [flat|nested] 25+ messages in thread* [PATCH v2 04/10] isar-boot-strap: Add option to keep cache
2018-10-11 16:52 [PATCH v2 00/10] Introduce local apt repo to cache upstream debian packages for offline usage Maxim Yu. Osipov
` (2 preceding siblings ...)
2018-10-11 16:52 ` [PATCH v2 03/10] base-apt: Introduce base implementation Maxim Yu. Osipov
@ 2018-10-11 16:52 ` Maxim Yu. Osipov
2018-10-23 16:06 ` Henning Schild
2018-10-25 14:28 ` Henning Schild
2018-10-11 16:53 ` [PATCH v2 05/10] image: Add cache_base_repo task Maxim Yu. Osipov
` (8 subsequent siblings)
12 siblings, 2 replies; 25+ messages in thread
From: Maxim Yu. Osipov @ 2018-10-11 16:52 UTC (permalink / raw)
To: isar-users
From: Alexander Smirnov <asmirnov@ilbers.de>
The default ISAR behavior assumes to remove all the packages
from apt cache, but we need them to put into base-apt.
Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
---
meta/classes/isar-bootstrap-helper.bbclass | 6 ++++++
1 file changed, 6 insertions(+)
diff --git a/meta/classes/isar-bootstrap-helper.bbclass b/meta/classes/isar-bootstrap-helper.bbclass
index f046216..b7cd64a 100644
--- a/meta/classes/isar-bootstrap-helper.bbclass
+++ b/meta/classes/isar-bootstrap-helper.bbclass
@@ -70,6 +70,7 @@ setup_root_file_system() {
--fstab) FSTAB=$2; shift ;;
--host-arch) ROOTFS_ARCH=${HOST_ARCH} ;;
--host-distro) ROOTFS_DISTRO=${HOST_DISTRO} ;;
+ --keep-apt-cache) KEEP_APT_CACHE=1 ;;
-*) bbfatal "$0: invalid option specified: $1" ;;
*) break ;;
esac
@@ -131,6 +132,11 @@ setup_root_file_system() {
/usr/bin/apt-get purge --yes $pkg
done
if [ ${CLEAN} ]; then
+ if [ ${KEEP_APT_CACHE} ]; then
+ mkdir -p ${WORKDIR}/apt_cache
+ sudo mv $(find $ROOTFSDIR/var/cache/apt -name '*.deb') ${WORKDIR}/apt_cache
+ sudo chown $USER ${WORKDIR}/apt_cache/*
+ fi
sudo -E chroot "$ROOTFSDIR" \
/usr/bin/apt-get autoremove --purge --yes
sudo -E chroot "$ROOTFSDIR" \
--
2.11.0
^ permalink raw reply [flat|nested] 25+ messages in thread* Re: [PATCH v2 04/10] isar-boot-strap: Add option to keep cache
2018-10-11 16:52 ` [PATCH v2 04/10] isar-boot-strap: Add option to keep cache Maxim Yu. Osipov
@ 2018-10-23 16:06 ` Henning Schild
2018-10-25 14:28 ` Henning Schild
1 sibling, 0 replies; 25+ messages in thread
From: Henning Schild @ 2018-10-23 16:06 UTC (permalink / raw)
To: Maxim Yu. Osipov; +Cc: isar-users
Am Thu, 11 Oct 2018 18:52:59 +0200
schrieb "Maxim Yu. Osipov" <mosipov@ilbers.de>:
> From: Alexander Smirnov <asmirnov@ilbers.de>
>
> The default ISAR behavior assumes to remove all the packages
> from apt cache, but we need them to put into base-apt.
>
> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
> ---
> meta/classes/isar-bootstrap-helper.bbclass | 6 ++++++
> 1 file changed, 6 insertions(+)
>
> diff --git a/meta/classes/isar-bootstrap-helper.bbclass
> b/meta/classes/isar-bootstrap-helper.bbclass index f046216..b7cd64a
> 100644 --- a/meta/classes/isar-bootstrap-helper.bbclass
> +++ b/meta/classes/isar-bootstrap-helper.bbclass
> @@ -70,6 +70,7 @@ setup_root_file_system() {
> --fstab) FSTAB=$2; shift ;;
> --host-arch) ROOTFS_ARCH=${HOST_ARCH} ;;
> --host-distro) ROOTFS_DISTRO=${HOST_DISTRO} ;;
> + --keep-apt-cache) KEEP_APT_CACHE=1 ;;
> -*) bbfatal "$0: invalid option specified: $1" ;;
> *) break ;;
> esac
> @@ -131,6 +132,11 @@ setup_root_file_system() {
> /usr/bin/apt-get purge --yes $pkg
> done
> if [ ${CLEAN} ]; then
> + if [ ${KEEP_APT_CACHE} ]; then
if [ ${KEEP_APT_CACHE} -eq 1 ]; then
> + mkdir -p ${WORKDIR}/apt_cache
> + sudo mv $(find $ROOTFSDIR/var/cache/apt -name '*.deb')
> ${WORKDIR}/apt_cache
> + sudo chown $USER ${WORKDIR}/apt_cache/*
> + fi
> sudo -E chroot "$ROOTFSDIR" \
> /usr/bin/apt-get autoremove --purge --yes
> sudo -E chroot "$ROOTFSDIR" \
^ permalink raw reply [flat|nested] 25+ messages in thread* Re: [PATCH v2 04/10] isar-boot-strap: Add option to keep cache
2018-10-11 16:52 ` [PATCH v2 04/10] isar-boot-strap: Add option to keep cache Maxim Yu. Osipov
2018-10-23 16:06 ` Henning Schild
@ 2018-10-25 14:28 ` Henning Schild
1 sibling, 0 replies; 25+ messages in thread
From: Henning Schild @ 2018-10-25 14:28 UTC (permalink / raw)
To: Maxim Yu. Osipov; +Cc: isar-users
Am Thu, 11 Oct 2018 18:52:59 +0200
schrieb "Maxim Yu. Osipov" <mosipov@ilbers.de>:
> From: Alexander Smirnov <asmirnov@ilbers.de>
>
> The default ISAR behavior assumes to remove all the packages
> from apt cache, but we need them to put into base-apt.
>
> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
> ---
> meta/classes/isar-bootstrap-helper.bbclass | 6 ++++++
> 1 file changed, 6 insertions(+)
>
> diff --git a/meta/classes/isar-bootstrap-helper.bbclass
> b/meta/classes/isar-bootstrap-helper.bbclass index f046216..b7cd64a
> 100644 --- a/meta/classes/isar-bootstrap-helper.bbclass
> +++ b/meta/classes/isar-bootstrap-helper.bbclass
> @@ -70,6 +70,7 @@ setup_root_file_system() {
> --fstab) FSTAB=$2; shift ;;
> --host-arch) ROOTFS_ARCH=${HOST_ARCH} ;;
> --host-distro) ROOTFS_DISTRO=${HOST_DISTRO} ;;
> + --keep-apt-cache) KEEP_APT_CACHE=1 ;;
> -*) bbfatal "$0: invalid option specified: $1" ;;
> *) break ;;
> esac
> @@ -131,6 +132,11 @@ setup_root_file_system() {
> /usr/bin/apt-get purge --yes $pkg
> done
> if [ ${CLEAN} ]; then
> + if [ ${KEEP_APT_CACHE} ]; then
> + mkdir -p ${WORKDIR}/apt_cache
> + sudo mv $(find $ROOTFSDIR/var/cache/apt -name '*.deb')
> ${WORKDIR}/apt_cache
> + sudo chown $USER ${WORKDIR}/apt_cache/*
In my build USER is not set. So we fail with
>> chown: invalid user:
>> '/out/build/tmp/work/debian-stretch-amd64/isar-image-base/apt_cache/adduser_3.115_all.deb'
Where adduser is the first of *. Should be `whoami`.
Henning
> + fi
> sudo -E chroot "$ROOTFSDIR" \
> /usr/bin/apt-get autoremove --purge --yes
> sudo -E chroot "$ROOTFSDIR" \
^ permalink raw reply [flat|nested] 25+ messages in thread
* [PATCH v2 05/10] image: Add cache_base_repo task
2018-10-11 16:52 [PATCH v2 00/10] Introduce local apt repo to cache upstream debian packages for offline usage Maxim Yu. Osipov
` (3 preceding siblings ...)
2018-10-11 16:52 ` [PATCH v2 04/10] isar-boot-strap: Add option to keep cache Maxim Yu. Osipov
@ 2018-10-11 16:53 ` Maxim Yu. Osipov
2018-10-11 16:53 ` [PATCH v2 06/10] isar-bootstrap: Make possible to reuse the cache Maxim Yu. Osipov
` (7 subsequent siblings)
12 siblings, 0 replies; 25+ messages in thread
From: Maxim Yu. Osipov @ 2018-10-11 16:53 UTC (permalink / raw)
To: isar-users
From: Alexander Smirnov <asmirnov@ilbers.de>
This task puts all the packages using in build to base-apt.
Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
Signed-off-by: Maxim Yu. Osipov <mosipov@ilbers.de>
---
meta-isar/recipes-core/images/isar-image-base.bb | 3 ++-
meta/classes/image.bbclass | 21 +++++++++++++++++++++
2 files changed, 23 insertions(+), 1 deletion(-)
diff --git a/meta-isar/recipes-core/images/isar-image-base.bb b/meta-isar/recipes-core/images/isar-image-base.bb
index 4899593..bf606cc 100644
--- a/meta-isar/recipes-core/images/isar-image-base.bb
+++ b/meta-isar/recipes-core/images/isar-image-base.bb
@@ -41,7 +41,8 @@ devtmpfs /dev devtmpfs mode=0755,nosuid 0 0
# End /etc/fstab
EOF
- setup_root_file_system --clean --fstab "${WORKDIR}/fstab" \
+ setup_root_file_system --clean --keep-apt-cache \
+ --fstab "${WORKDIR}/fstab" \
"${IMAGE_ROOTFS}" ${IMAGE_PREINSTALL} ${IMAGE_INSTALL}
# Configure root filesystem
diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index e948dea..05ff06f 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -123,6 +123,27 @@ do_populate_sdk[depends] = "sdkchroot:do_build"
addtask populate_sdk after do_rootfs
+inherit base-apt-helper
+
+do_cache_base_repo[depends] = "base-apt:do_cache_config"
+do_cache_base_repo[stamp-extra-info] = "${MACHINE}-${DISTRO}"
+
+do_cache_base_repo() {
+ if [ -d ${WORKDIR}/apt_cache ]; then
+ populate_base_apt ${WORKDIR}/apt_cache
+ fi
+
+ if [ -d ${BUILDCHROOT_HOST_DIR}/var/cache/apt ]; then
+ populate_base_apt ${BUILDCHROOT_HOST_DIR}/var/cache/apt
+ fi
+
+ if [ -d ${BUILDCHROOT_TARGET_DIR}/var/cache/apt ]; then
+ populate_base_apt ${BUILDCHROOT_TARGET_DIR}/var/cache/apt
+ fi
+}
+
+addtask cache_base_repo after do_rootfs
+
# Imager are expected to run natively, thus will use the target buildchroot.
ISAR_CROSS_COMPILE = "0"
--
2.11.0
^ permalink raw reply [flat|nested] 25+ messages in thread* [PATCH v2 06/10] isar-bootstrap: Make possible to reuse the cache
2018-10-11 16:52 [PATCH v2 00/10] Introduce local apt repo to cache upstream debian packages for offline usage Maxim Yu. Osipov
` (4 preceding siblings ...)
2018-10-11 16:53 ` [PATCH v2 05/10] image: Add cache_base_repo task Maxim Yu. Osipov
@ 2018-10-11 16:53 ` Maxim Yu. Osipov
2018-10-23 16:30 ` Henning Schild
2018-10-11 16:53 ` [PATCH v2 07/10] buildchroot: Make it buildable from base-apt Maxim Yu. Osipov
` (6 subsequent siblings)
12 siblings, 1 reply; 25+ messages in thread
From: Maxim Yu. Osipov @ 2018-10-11 16:53 UTC (permalink / raw)
To: isar-users
From: Alexander Smirnov <asmirnov@ilbers.de>
Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
Signed-off-by: Maxim Yu. Osipov <mosipov@ilbers.de>
---
.../isar-bootstrap/files/base-apt-sources | 1 +
.../recipes-core/isar-bootstrap/isar-bootstrap.inc | 32 ++++++++++++++++------
2 files changed, 24 insertions(+), 9 deletions(-)
create mode 100644 meta/recipes-core/isar-bootstrap/files/base-apt-sources
diff --git a/meta/recipes-core/isar-bootstrap/files/base-apt-sources b/meta/recipes-core/isar-bootstrap/files/base-apt-sources
new file mode 100644
index 0000000..594db56
--- /dev/null
+++ b/meta/recipes-core/isar-bootstrap/files/base-apt-sources
@@ -0,0 +1 @@
+deb file:///base-apt/debian {DISTRO} main
diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
index cfad136..cc1791c 100644
--- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
+++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
@@ -12,18 +12,24 @@ SRC_URI = " \
file://isar-apt.conf \
file://isar-apt-fallback.conf \
file://locale \
- file://chroot-setup.sh"
+ file://chroot-setup.sh \
+ file://base-apt-sources"
PV = "1.0"
DEBOOTSTRAP ?= "qemu-debootstrap"
ROOTFSDIR = "${WORKDIR}/rootfs"
APTPREFS = "${WORKDIR}/apt-preferences"
APTSRCS = "${WORKDIR}/apt-sources"
+BASEAPTSRCS = "${WORKDIR}/base-apt-sources"
APTKEYFILES = ""
APTKEYRING = "${WORKDIR}/apt-keyring.gpg"
DEBOOTSTRAP_KEYRING = ""
DEPLOY_ISAR_BOOTSTRAP ?= ""
+DISTRO_APT_PREMIRRORS ?= "${@ "http://ftp\.(\S+\.)?debian.org file:///${REPO_BASE_DIR} \n" if bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')) else "" }"
+
+inherit base-apt-helper
+
python () {
from urllib.parse import urlparse
distro_apt_keys = d.getVar("DISTRO_APT_KEYS", False)
@@ -171,6 +177,10 @@ isar_bootstrap() {
esac
shift
done
+ debootstrap_args="--verbose --variant=minbase --include=locales "
+ if [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ]; then
+ debootstrap_args="$debootstrap_args --no-check-gpg"
+ fi
E="${@bb.utils.export_proxies(d)}"
sudo -E flock "${ISAR_BOOTSTRAP_LOCK}" -c "\
set -e
@@ -181,9 +191,7 @@ isar_bootstrap() {
rm -rf "${ROOTFSDIR}"
fi
if [ ${IS_HOST} ]; then
- ${DEBOOTSTRAP} --verbose \
- --variant=minbase \
- --include=locales \
+ ${DEBOOTSTRAP} $debootstrap_args \
${@get_distro_components_argument(d, True)} \
${DEBOOTSTRAP_KEYRING} \
"${@get_distro_suite(d, True)}" \
@@ -191,10 +199,8 @@ isar_bootstrap() {
"${@get_distro_source(d, True)}"
else
- "${DEBOOTSTRAP}" --verbose \
- --variant=minbase \
+ "${DEBOOTSTRAP}" $debootstrap_args \
--arch="${DISTRO_ARCH}" \
- --include=locales \
${@get_distro_components_argument(d, False)} \
${DEBOOTSTRAP_KEYRING} \
"${@get_distro_suite(d, False)}" \
@@ -207,8 +213,16 @@ isar_bootstrap() {
install -v -m644 "${APTPREFS}" \
"${ROOTFSDIR}/etc/apt/preferences.d/bootstrap"
mkdir -p "${ROOTFSDIR}/etc/apt/sources.list.d"
- install -v -m644 "${APTSRCS}" \
- "${ROOTFSDIR}/etc/apt/sources.list.d/bootstrap.list"
+ if [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ]; then
+ sed -i -e "s#{DISTRO}#"${DISTRO_SUITE}"#g" ${BASEAPTSRCS}
+ mkdir -p ${ROOTFSDIR}/base-apt
+ sudo mount --bind ${REPO_BASE_DIR} ${ROOTFSDIR}/base-apt
+ install -v -m644 "${BASEAPTSRCS}" \
+ "${ROOTFSDIR}/etc/apt/sources.list.d/base-apt.list"
+ else
+ install -v -m644 "${APTSRCS}" \
+ "${ROOTFSDIR}/etc/apt/sources.list.d/bootstrap.list"
+ fi
rm -f "${ROOTFSDIR}/etc/apt/sources.list"
mkdir -p "${ROOTFSDIR}/etc/apt/apt.conf.d"
install -v -m644 "${WORKDIR}/isar-apt.conf" \
--
2.11.0
^ permalink raw reply [flat|nested] 25+ messages in thread* Re: [PATCH v2 06/10] isar-bootstrap: Make possible to reuse the cache
2018-10-11 16:53 ` [PATCH v2 06/10] isar-bootstrap: Make possible to reuse the cache Maxim Yu. Osipov
@ 2018-10-23 16:30 ` Henning Schild
0 siblings, 0 replies; 25+ messages in thread
From: Henning Schild @ 2018-10-23 16:30 UTC (permalink / raw)
To: Maxim Yu. Osipov; +Cc: isar-users
Am Thu, 11 Oct 2018 18:53:01 +0200
schrieb "Maxim Yu. Osipov" <mosipov@ilbers.de>:
> From: Alexander Smirnov <asmirnov@ilbers.de>
>
> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
> Signed-off-by: Maxim Yu. Osipov <mosipov@ilbers.de>
> ---
> .../isar-bootstrap/files/base-apt-sources | 1 +
> .../recipes-core/isar-bootstrap/isar-bootstrap.inc | 32
> ++++++++++++++++------ 2 files changed, 24 insertions(+), 9
> deletions(-) create mode 100644
> meta/recipes-core/isar-bootstrap/files/base-apt-sources
>
> diff --git a/meta/recipes-core/isar-bootstrap/files/base-apt-sources
> b/meta/recipes-core/isar-bootstrap/files/base-apt-sources new file
> mode 100644 index 0000000..594db56
> --- /dev/null
> +++ b/meta/recipes-core/isar-bootstrap/files/base-apt-sources
> @@ -0,0 +1 @@
> +deb file:///base-apt/debian {DISTRO} main
Try "[trusted=yes]" in here and drop the workaround patch if possible.
Henning
> diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
> b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc index
> cfad136..cc1791c 100644 ---
> a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc +++
> b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc @@ -12,18
> +12,24 @@ SRC_URI = " \ file://isar-apt.conf \
> file://isar-apt-fallback.conf \
> file://locale \
> - file://chroot-setup.sh"
> + file://chroot-setup.sh \
> + file://base-apt-sources"
> PV = "1.0"
>
> DEBOOTSTRAP ?= "qemu-debootstrap"
> ROOTFSDIR = "${WORKDIR}/rootfs"
> APTPREFS = "${WORKDIR}/apt-preferences"
> APTSRCS = "${WORKDIR}/apt-sources"
> +BASEAPTSRCS = "${WORKDIR}/base-apt-sources"
> APTKEYFILES = ""
> APTKEYRING = "${WORKDIR}/apt-keyring.gpg"
> DEBOOTSTRAP_KEYRING = ""
> DEPLOY_ISAR_BOOTSTRAP ?= ""
>
> +DISTRO_APT_PREMIRRORS ?= "${@ "http://ftp\.(\S+\.)?debian.org
> file:///${REPO_BASE_DIR} \n" if
> bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')) else "" }"
> + +inherit base-apt-helper +
> python () {
> from urllib.parse import urlparse
> distro_apt_keys = d.getVar("DISTRO_APT_KEYS", False)
> @@ -171,6 +177,10 @@ isar_bootstrap() {
> esac
> shift
> done
> + debootstrap_args="--verbose --variant=minbase --include=locales "
> + if [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ]; then
> + debootstrap_args="$debootstrap_args --no-check-gpg"
> + fi
> E="${@bb.utils.export_proxies(d)}"
> sudo -E flock "${ISAR_BOOTSTRAP_LOCK}" -c "\
> set -e
> @@ -181,9 +191,7 @@ isar_bootstrap() {
> rm -rf "${ROOTFSDIR}"
> fi
> if [ ${IS_HOST} ]; then
> - ${DEBOOTSTRAP} --verbose \
> - --variant=minbase \
> - --include=locales \
> + ${DEBOOTSTRAP} $debootstrap_args \
> ${@get_distro_components_argument(d,
> True)} \ ${DEBOOTSTRAP_KEYRING} \
> "${@get_distro_suite(d, True)}" \
> @@ -191,10 +199,8 @@ isar_bootstrap() {
> "${@get_distro_source(d, True)}"
>
> else
> - "${DEBOOTSTRAP}" --verbose \
> - --variant=minbase \
> + "${DEBOOTSTRAP}" $debootstrap_args \
> --arch="${DISTRO_ARCH}" \
> - --include=locales \
> ${@get_distro_components_argument(d,
> False)} \ ${DEBOOTSTRAP_KEYRING} \
> "${@get_distro_suite(d, False)}" \
> @@ -207,8 +213,16 @@ isar_bootstrap() {
> install -v -m644 "${APTPREFS}" \
> "${ROOTFSDIR}/etc/apt/preferences.d/bootstrap"
> mkdir -p "${ROOTFSDIR}/etc/apt/sources.list.d"
> - install -v -m644 "${APTSRCS}" \
> -
> "${ROOTFSDIR}/etc/apt/sources.list.d/bootstrap.list"
> + if [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ]; then
> + sed -i -e "s#{DISTRO}#"${DISTRO_SUITE}"#g"
> ${BASEAPTSRCS}
> + mkdir -p ${ROOTFSDIR}/base-apt
> + sudo mount --bind ${REPO_BASE_DIR}
> ${ROOTFSDIR}/base-apt
> + install -v -m644 "${BASEAPTSRCS}" \
> +
> "${ROOTFSDIR}/etc/apt/sources.list.d/base-apt.list"
> + else
> + install -v -m644 "${APTSRCS}" \
> +
> "${ROOTFSDIR}/etc/apt/sources.list.d/bootstrap.list"
> + fi
> rm -f "${ROOTFSDIR}/etc/apt/sources.list"
> mkdir -p "${ROOTFSDIR}/etc/apt/apt.conf.d"
> install -v -m644 "${WORKDIR}/isar-apt.conf" \
^ permalink raw reply [flat|nested] 25+ messages in thread
* [PATCH v2 07/10] buildchroot: Make it buildable from base-apt
2018-10-11 16:52 [PATCH v2 00/10] Introduce local apt repo to cache upstream debian packages for offline usage Maxim Yu. Osipov
` (5 preceding siblings ...)
2018-10-11 16:53 ` [PATCH v2 06/10] isar-bootstrap: Make possible to reuse the cache Maxim Yu. Osipov
@ 2018-10-11 16:53 ` Maxim Yu. Osipov
2018-10-11 16:53 ` [PATCH v2 08/10] workaround: Use --allow-unauthenticated working with base-apt Maxim Yu. Osipov
` (5 subsequent siblings)
12 siblings, 0 replies; 25+ messages in thread
From: Maxim Yu. Osipov @ 2018-10-11 16:53 UTC (permalink / raw)
To: isar-users
From: Alexander Smirnov <asmirnov@ilbers.de>
Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
Signed-off-by: Maxim Yu. Osipov <mosipov@ilbers.de>
---
meta/classes/isar-bootstrap-helper.bbclass | 5 +++++
1 file changed, 5 insertions(+)
diff --git a/meta/classes/isar-bootstrap-helper.bbclass b/meta/classes/isar-bootstrap-helper.bbclass
index b7cd64a..8f5e583 100644
--- a/meta/classes/isar-bootstrap-helper.bbclass
+++ b/meta/classes/isar-bootstrap-helper.bbclass
@@ -98,6 +98,11 @@ setup_root_file_system() {
else
sudo mount --bind ${REPO_ISAR_DIR}/${DISTRO} $ROOTFSDIR/isar-apt
fi
+
+ if [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ]; then
+ sudo mount --bind ${REPO_BASE_DIR} ${ROOTFSDIR}/base-apt
+ fi
+
sudo mount -t devtmpfs -o mode=0755,nosuid devtmpfs $ROOTFSDIR/dev
sudo mount -t proc none $ROOTFSDIR/proc
--
2.11.0
^ permalink raw reply [flat|nested] 25+ messages in thread* [PATCH v2 08/10] workaround: Use --allow-unauthenticated working with base-apt
2018-10-11 16:52 [PATCH v2 00/10] Introduce local apt repo to cache upstream debian packages for offline usage Maxim Yu. Osipov
` (6 preceding siblings ...)
2018-10-11 16:53 ` [PATCH v2 07/10] buildchroot: Make it buildable from base-apt Maxim Yu. Osipov
@ 2018-10-11 16:53 ` Maxim Yu. Osipov
2018-10-23 16:09 ` Henning Schild
2018-10-11 16:53 ` [PATCH v2 09/10] local.conf: Add option to use cached base repository Maxim Yu. Osipov
` (4 subsequent siblings)
12 siblings, 1 reply; 25+ messages in thread
From: Maxim Yu. Osipov @ 2018-10-11 16:53 UTC (permalink / raw)
To: isar-users
From: Alexander Smirnov <asmirnov@ilbers.de>
Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
---
meta/classes/isar-bootstrap-helper.bbclass | 2 +-
meta/recipes-devtools/buildchroot/files/deps.sh | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
diff --git a/meta/classes/isar-bootstrap-helper.bbclass b/meta/classes/isar-bootstrap-helper.bbclass
index 8f5e583..24e1157 100644
--- a/meta/classes/isar-bootstrap-helper.bbclass
+++ b/meta/classes/isar-bootstrap-helper.bbclass
@@ -79,7 +79,7 @@ setup_root_file_system() {
ROOTFSDIR="$1"
shift
PACKAGES="$@"
- APT_ARGS="install --yes -o Debug::pkgProblemResolver=yes"
+ APT_ARGS="install --yes --allow-unauthenticated -o Debug::pkgProblemResolver=yes"
CLEAN_FILES="${ROOTFSDIR}/etc/hostname ${ROOTFSDIR}/etc/resolv.conf"
sudo cp -Trpfx \
diff --git a/meta/recipes-devtools/buildchroot/files/deps.sh b/meta/recipes-devtools/buildchroot/files/deps.sh
index 4bd604f..2fc2f66 100644
--- a/meta/recipes-devtools/buildchroot/files/deps.sh
+++ b/meta/recipes-devtools/buildchroot/files/deps.sh
@@ -10,7 +10,7 @@ source /isar/common.sh
# Notes:
# 1) everything before the -y switch is unchanged from the defaults
# 2) we add -y to go non-interactive
-install_cmd="apt-get -o Debug::pkgProblemResolver=yes --no-install-recommends -y"
+install_cmd="apt-get -o Debug::pkgProblemResolver=yes --no-install-recommends -y --allow-unauthenticated"
# Make sure that we have latest isar-apt content.
# Options meaning:
--
2.11.0
^ permalink raw reply [flat|nested] 25+ messages in thread* Re: [PATCH v2 08/10] workaround: Use --allow-unauthenticated working with base-apt
2018-10-11 16:53 ` [PATCH v2 08/10] workaround: Use --allow-unauthenticated working with base-apt Maxim Yu. Osipov
@ 2018-10-23 16:09 ` Henning Schild
2018-10-25 14:33 ` Henning Schild
0 siblings, 1 reply; 25+ messages in thread
From: Henning Schild @ 2018-10-23 16:09 UTC (permalink / raw)
To: Maxim Yu. Osipov; +Cc: isar-users
Am Thu, 11 Oct 2018 18:53:03 +0200
schrieb "Maxim Yu. Osipov" <mosipov@ilbers.de>:
> From: Alexander Smirnov <asmirnov@ilbers.de>
>
> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
> ---
> meta/classes/isar-bootstrap-helper.bbclass | 2 +-
> meta/recipes-devtools/buildchroot/files/deps.sh | 2 +-
> 2 files changed, 2 insertions(+), 2 deletions(-)
>
> diff --git a/meta/classes/isar-bootstrap-helper.bbclass
> b/meta/classes/isar-bootstrap-helper.bbclass index 8f5e583..24e1157
> 100644 --- a/meta/classes/isar-bootstrap-helper.bbclass
> +++ b/meta/classes/isar-bootstrap-helper.bbclass
> @@ -79,7 +79,7 @@ setup_root_file_system() {
> ROOTFSDIR="$1"
> shift
> PACKAGES="$@"
> - APT_ARGS="install --yes -o Debug::pkgProblemResolver=yes"
> + APT_ARGS="install --yes --allow-unauthenticated -o
> Debug::pkgProblemResolver=yes" CLEAN_FILES="${ROOTFSDIR}/etc/hostname
> ${ROOTFSDIR}/etc/resolv.conf"
> sudo cp -Trpfx \
> diff --git a/meta/recipes-devtools/buildchroot/files/deps.sh
> b/meta/recipes-devtools/buildchroot/files/deps.sh index
> 4bd604f..2fc2f66 100644 ---
> a/meta/recipes-devtools/buildchroot/files/deps.sh +++
> b/meta/recipes-devtools/buildchroot/files/deps.sh @@ -10,7 +10,7 @@
> source /isar/common.sh # Notes:
> # 1) everything before the -y switch is unchanged from the defaults
> # 2) we add -y to go non-interactive
> -install_cmd="apt-get -o Debug::pkgProblemResolver=yes
> --no-install-recommends -y" +install_cmd="apt-get -o
> Debug::pkgProblemResolver=yes --no-install-recommends -y
> --allow-unauthenticated" # Make sure that we have latest isar-apt
> content. # Options meaning:
This should only be done when using the cache ... in fact it should not
be done at all!
Henning
^ permalink raw reply [flat|nested] 25+ messages in thread* Re: [PATCH v2 08/10] workaround: Use --allow-unauthenticated working with base-apt
2018-10-23 16:09 ` Henning Schild
@ 2018-10-25 14:33 ` Henning Schild
0 siblings, 0 replies; 25+ messages in thread
From: Henning Schild @ 2018-10-25 14:33 UTC (permalink / raw)
To: Maxim Yu. Osipov; +Cc: isar-users
Am Tue, 23 Oct 2018 18:09:31 +0200
schrieb "[ext] Henning Schild" <henning.schild@siemens.com>:
> Am Thu, 11 Oct 2018 18:53:03 +0200
> schrieb "Maxim Yu. Osipov" <mosipov@ilbers.de>:
>
> > From: Alexander Smirnov <asmirnov@ilbers.de>
> >
> > Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
> > ---
> > meta/classes/isar-bootstrap-helper.bbclass | 2 +-
> > meta/recipes-devtools/buildchroot/files/deps.sh | 2 +-
> > 2 files changed, 2 insertions(+), 2 deletions(-)
> >
> > diff --git a/meta/classes/isar-bootstrap-helper.bbclass
> > b/meta/classes/isar-bootstrap-helper.bbclass index 8f5e583..24e1157
> > 100644 --- a/meta/classes/isar-bootstrap-helper.bbclass
> > +++ b/meta/classes/isar-bootstrap-helper.bbclass
> > @@ -79,7 +79,7 @@ setup_root_file_system() {
> > ROOTFSDIR="$1"
> > shift
> > PACKAGES="$@"
> > - APT_ARGS="install --yes -o Debug::pkgProblemResolver=yes"
> > + APT_ARGS="install --yes --allow-unauthenticated -o
> > Debug::pkgProblemResolver=yes"
> > CLEAN_FILES="${ROOTFSDIR}/etc/hostname
> > ${ROOTFSDIR}/etc/resolv.conf" sudo cp -Trpfx \
> > diff --git a/meta/recipes-devtools/buildchroot/files/deps.sh
> > b/meta/recipes-devtools/buildchroot/files/deps.sh index
> > 4bd604f..2fc2f66 100644 ---
> > a/meta/recipes-devtools/buildchroot/files/deps.sh +++
> > b/meta/recipes-devtools/buildchroot/files/deps.sh @@ -10,7 +10,7 @@
> > source /isar/common.sh # Notes:
> > # 1) everything before the -y switch is unchanged from the
> > defaults # 2) we add -y to go non-interactive
> > -install_cmd="apt-get -o Debug::pkgProblemResolver=yes
> > --no-install-recommends -y" +install_cmd="apt-get -o
> > Debug::pkgProblemResolver=yes --no-install-recommends -y
> > --allow-unauthenticated" # Make sure that we have latest isar-apt
> > content. # Options meaning:
>
> This should only be done when using the cache ... in fact it should
> not be done at all!
What i was trying to say is, revert this because it breaks integrity
checking of all of Isar. If that breaks the series revert the whole
thing.
Henning
> Henning
>
^ permalink raw reply [flat|nested] 25+ messages in thread
* [PATCH v2 09/10] local.conf: Add option to use cached base repository
2018-10-11 16:52 [PATCH v2 00/10] Introduce local apt repo to cache upstream debian packages for offline usage Maxim Yu. Osipov
` (7 preceding siblings ...)
2018-10-11 16:53 ` [PATCH v2 08/10] workaround: Use --allow-unauthenticated working with base-apt Maxim Yu. Osipov
@ 2018-10-11 16:53 ` Maxim Yu. Osipov
2018-10-11 16:53 ` [PATCH v2 10/10] doc: Creation of local apt repo caching upstream Debian packages Maxim Yu. Osipov
` (3 subsequent siblings)
12 siblings, 0 replies; 25+ messages in thread
From: Maxim Yu. Osipov @ 2018-10-11 16:53 UTC (permalink / raw)
To: isar-users
From: Alexander Smirnov <asmirnov@ilbers.de>
Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
Signed-off-by: Maxim Yu. Osipov <mosipov@ilbers.de>
---
meta-isar/conf/local.conf.sample | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/meta-isar/conf/local.conf.sample b/meta-isar/conf/local.conf.sample
index b8b8495..f3a960a 100644
--- a/meta-isar/conf/local.conf.sample
+++ b/meta-isar/conf/local.conf.sample
@@ -165,3 +165,7 @@ IMAGE_INSTALL = "example-hello example-raw example-module enable-fsck"
# Enable cross-compilation support
# NOTE: this works on build host >= stretch for armhf, arm64 and amd64 targets for now.
ISAR_CROSS_COMPILE ?= "0"
+
+#
+# Uncomment this to enable use of cached base repository
+#ISAR_USE_CACHED_BASE_REPO ?= "1"
--
2.11.0
^ permalink raw reply [flat|nested] 25+ messages in thread* [PATCH v2 10/10] doc: Creation of local apt repo caching upstream Debian packages
2018-10-11 16:52 [PATCH v2 00/10] Introduce local apt repo to cache upstream debian packages for offline usage Maxim Yu. Osipov
` (8 preceding siblings ...)
2018-10-11 16:53 ` [PATCH v2 09/10] local.conf: Add option to use cached base repository Maxim Yu. Osipov
@ 2018-10-11 16:53 ` Maxim Yu. Osipov
2018-10-17 12:13 ` [PATCH v2 00/10] Introduce local apt repo to cache upstream debian packages for offline usage Maxim Yu. Osipov
` (2 subsequent siblings)
12 siblings, 0 replies; 25+ messages in thread
From: Maxim Yu. Osipov @ 2018-10-11 16:53 UTC (permalink / raw)
To: isar-users
Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
Signed-off-by: Maxim Yu. Osipov <mosipov@ilbers.de>
---
doc/user_manual.md | 39 +++++++++++++++++++++++++++++++++++++++
1 file changed, 39 insertions(+)
diff --git a/doc/user_manual.md b/doc/user_manual.md
index db58cf1..d69dd0d 100644
--- a/doc/user_manual.md
+++ b/doc/user_manual.md
@@ -19,6 +19,8 @@ Copyright (C) 2016-2017, ilbers GmbH
- [Add a Custom Application](https://github.com/ilbers/isar/blob/master/doc/user_manual.md#add-a-custom-application)
- [Enabling Cross-compilation](https://github.com/ilbers/isar/blob/master/doc/user_manual.md#isar-cross-compilation)
- [Create an ISAR SDK root filesystem](https://github.com/ilbers/isar/blob/master/doc/user_manual.md#create-an-isar-sdk-root-filesystem)
+ - [Creation of local apt repo caching upstream Debian packages](https://github.com/ilbers/isar/blob/master/doc/user_manual.md#creation-repo-caching-upstream-debian)
+
## Introduction
@@ -686,3 +688,40 @@ ii crossbuild-essential-armhf 12.3 all Inf
/usr/share/doc/libhello-dev/copyright
~#
```
+
+## Creation of local apt repo caching upstream Debian packages
+
+### Motivation
+
+Cache upstream debian packages to reduce time for further downloads and to be able to work offline.
+
+### Solution
+
+ - Trigger creation of local apt caching Debian packages during image generation.
+
+```
+bitbake -c cache_base_repo multiconfig:qemuarm-stretch:isar-image-base
+```
+
+ - Set `ISAR_USE_CACHED_BASE_REPO` in `conf/local.conf`:
+
+```
+# Uncomment this to enable use of cached base repository
+#ISAR_USE_CACHED_BASE_REPO ?= "1"
+```
+ - Remove build artifacts to use only local base-apt:
+
+```
+sudo rm -rf tmp/stamps/ tmp/work/ tmp/deploy/isar-apt/ tmp/deploy/images
+
+```
+
+ - Trigger again generation of image (now using local caching repo):
+
+```
+bitbake multiconfig:qemuarm-stretch:isar-image-base
+```
+
+### Limitation
+
+So far the local base-apt repo is not gpg signed.
--
2.11.0
^ permalink raw reply [flat|nested] 25+ messages in thread* Re: [PATCH v2 00/10] Introduce local apt repo to cache upstream debian packages for offline usage
2018-10-11 16:52 [PATCH v2 00/10] Introduce local apt repo to cache upstream debian packages for offline usage Maxim Yu. Osipov
` (9 preceding siblings ...)
2018-10-11 16:53 ` [PATCH v2 10/10] doc: Creation of local apt repo caching upstream Debian packages Maxim Yu. Osipov
@ 2018-10-17 12:13 ` Maxim Yu. Osipov
2018-10-17 17:11 ` Jan Kiszka
2018-10-23 16:14 ` Henning Schild
12 siblings, 0 replies; 25+ messages in thread
From: Maxim Yu. Osipov @ 2018-10-17 12:13 UTC (permalink / raw)
To: isar-users
On 10/11/18 7:52 PM, Maxim Yu. Osipov wrote:
> Hello everybody,
>
> This series contains fixes/improvements suggested by Claudius Heine
> during v1 series review.
>
> Changes to v2:
> - Fixes/improvements found in base-apt-helper class
> - Unifies path names to local repositories
>
> 1) Introduces dedicated local apt (base-apt) repo for upstream Debian packages
>
> 2) Caches in base-apt repo upstream Debian packages during image generation.
>
> 3) After this step, image can be built offline using only base-apt repo.
>
> Usage instructions can be found in the last patch.
>
> TODO:
> Sign the repo with gpg
Applied to the 'next'.
> Kind regards,
> Maxim.
>
> Alexander Smirnov (8):
> base-apt: Add helper class
> base-apt: Introduce base implementation
> isar-boot-strap: Add option to keep cache
> image: Add cache_base_repo task
> isar-bootstrap: Make possible to reuse the cache
> buildchroot: Make it buildable from base-apt
> workaround: Use --allow-unauthenticated working with base-apt
> local.conf: Add option to use cached base repository
>
> Maxim Yu. Osipov (2):
> meta: Unify path names to local repositories
> doc: Creation of local apt repo caching upstream Debian packages
>
> doc/user_manual.md | 39 ++++++++++++++++
> meta-isar/conf/layer.conf | 10 ++--
> meta-isar/conf/local.conf.sample | 4 ++
> meta-isar/recipes-core/images/isar-image-base.bb | 3 +-
> meta/classes/base-apt-helper.bbclass | 54 ++++++++++++++++++++++
> meta/classes/buildchroot.bbclass | 2 +-
> meta/classes/dpkg-base.bbclass | 10 ++--
> meta/classes/dpkg.bbclass | 2 +-
> meta/classes/image.bbclass | 25 +++++++++-
> meta/classes/isar-bootstrap-helper.bbclass | 17 +++++--
> .../isar-bootstrap/files/base-apt-sources | 1 +
> .../recipes-core/isar-bootstrap/isar-bootstrap.inc | 32 +++++++++----
> meta/recipes-devtools/base-apt/base-apt.bb | 31 +++++++++++++
> .../base-apt/files/distributions.in | 3 ++
> meta/recipes-devtools/buildchroot/files/deps.sh | 2 +-
> meta/recipes-devtools/isar-apt/isar-apt.bb | 8 ++--
> meta/recipes-kernel/linux/linux-custom.inc | 2 +-
> 17 files changed, 213 insertions(+), 32 deletions(-)
> create mode 100644 meta/classes/base-apt-helper.bbclass
> create mode 100644 meta/recipes-core/isar-bootstrap/files/base-apt-sources
> create mode 100644 meta/recipes-devtools/base-apt/base-apt.bb
> create mode 100644 meta/recipes-devtools/base-apt/files/distributions.in
>
--
Maxim Osipov
ilbers GmbH
Maria-Merian-Str. 8
85521 Ottobrunn
Germany
+49 (151) 6517 6917
mosipov@ilbers.de
http://ilbers.de/
Commercial register Munich, HRB 214197
General Manager: Baurzhan Ismagulov
^ permalink raw reply [flat|nested] 25+ messages in thread* Re: [PATCH v2 00/10] Introduce local apt repo to cache upstream debian packages for offline usage
2018-10-11 16:52 [PATCH v2 00/10] Introduce local apt repo to cache upstream debian packages for offline usage Maxim Yu. Osipov
` (10 preceding siblings ...)
2018-10-17 12:13 ` [PATCH v2 00/10] Introduce local apt repo to cache upstream debian packages for offline usage Maxim Yu. Osipov
@ 2018-10-17 17:11 ` Jan Kiszka
2018-10-17 18:15 ` Maxim Yu. Osipov
2018-10-23 16:14 ` Henning Schild
12 siblings, 1 reply; 25+ messages in thread
From: Jan Kiszka @ 2018-10-17 17:11 UTC (permalink / raw)
To: Maxim Yu. Osipov, isar-users
On 11.10.18 18:52, Maxim Yu. Osipov wrote:
> Hello everybody,
>
> This series contains fixes/improvements suggested by Claudius Heine
> during v1 series review.
>
> Changes to v2:
> - Fixes/improvements found in base-apt-helper class
> - Unifies path names to local repositories
>
> 1) Introduces dedicated local apt (base-apt) repo for upstream Debian packages
>
> 2) Caches in base-apt repo upstream Debian packages during image generation.
>
> 3) After this step, image can be built offline using only base-apt repo.
>
> Usage instructions can be found in the last patch.
>
> TODO:
> Sign the repo with gpg
>
> Kind regards,
> Maxim.
>
> Alexander Smirnov (8):
> base-apt: Add helper class
> base-apt: Introduce base implementation
> isar-boot-strap: Add option to keep cache
> image: Add cache_base_repo task
> isar-bootstrap: Make possible to reuse the cache
> buildchroot: Make it buildable from base-apt
> workaround: Use --allow-unauthenticated working with base-apt
> local.conf: Add option to use cached base repository
>
> Maxim Yu. Osipov (2):
> meta: Unify path names to local repositories
> doc: Creation of local apt repo caching upstream Debian packages
>
> doc/user_manual.md | 39 ++++++++++++++++
> meta-isar/conf/layer.conf | 10 ++--
> meta-isar/conf/local.conf.sample | 4 ++
> meta-isar/recipes-core/images/isar-image-base.bb | 3 +-
> meta/classes/base-apt-helper.bbclass | 54 ++++++++++++++++++++++
> meta/classes/buildchroot.bbclass | 2 +-
> meta/classes/dpkg-base.bbclass | 10 ++--
> meta/classes/dpkg.bbclass | 2 +-
> meta/classes/image.bbclass | 25 +++++++++-
> meta/classes/isar-bootstrap-helper.bbclass | 17 +++++--
> .../isar-bootstrap/files/base-apt-sources | 1 +
> .../recipes-core/isar-bootstrap/isar-bootstrap.inc | 32 +++++++++----
> meta/recipes-devtools/base-apt/base-apt.bb | 31 +++++++++++++
> .../base-apt/files/distributions.in | 3 ++
> meta/recipes-devtools/buildchroot/files/deps.sh | 2 +-
> meta/recipes-devtools/isar-apt/isar-apt.bb | 8 ++--
> meta/recipes-kernel/linux/linux-custom.inc | 2 +-
> 17 files changed, 213 insertions(+), 32 deletions(-)
> create mode 100644 meta/classes/base-apt-helper.bbclass
> create mode 100644 meta/recipes-core/isar-bootstrap/files/base-apt-sources
> create mode 100644 meta/recipes-devtools/base-apt/base-apt.bb
> create mode 100644 meta/recipes-devtools/base-apt/files/distributions.in
>
Could it be we have some regression here? After updating to current
next, a build without the cache enabled fails during do_rootfs with
| (Reading database ... 9600 files and directories currently installed.)
| Removing isar-cfg-localepurge (1.0+isar) ...
| find: '/build/tmp/work/debian-stretch-armhf/isar-image-base/rootfs/var/cache/apt/archives/partial': Permission denied
| chown: invalid user: '/build/tmp/work/debian-stretch-armhf/isar-image-base/apt_cache/adduser_3.115_all.deb'
| WARNING: exit code 1 from a shell command.
| ERROR: Function failed: do_rootfs (log file is located at /build/tmp/work/debian-stretch-armhf/isar-image-base/temp/log.do_rootfs.5627)
I do not have the time to bisect, will just jump back for now.
Jan
--
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux
^ permalink raw reply [flat|nested] 25+ messages in thread* Re: [PATCH v2 00/10] Introduce local apt repo to cache upstream debian packages for offline usage
2018-10-17 17:11 ` Jan Kiszka
@ 2018-10-17 18:15 ` Maxim Yu. Osipov
2018-10-23 9:56 ` Maxim Yu. Osipov
0 siblings, 1 reply; 25+ messages in thread
From: Maxim Yu. Osipov @ 2018-10-17 18:15 UTC (permalink / raw)
To: Jan Kiszka, isar-users
On 10/17/18 8:11 PM, Jan Kiszka wrote:
> On 11.10.18 18:52, Maxim Yu. Osipov wrote:
>> Hello everybody,
>>
>> This series contains fixes/improvements suggested by Claudius Heine
>> during v1 series review.
>>
>> Changes to v2:
>> - Fixes/improvements found in base-apt-helper class
>> - Unifies path names to local repositories
>>
>> 1) Introduces dedicated local apt (base-apt) repo for upstream Debian packages
>>
>> 2) Caches in base-apt repo upstream Debian packages during image generation.
>>
>> 3) After this step, image can be built offline using only base-apt repo.
>>
>> Usage instructions can be found in the last patch.
>>
>> TODO:
>> Sign the repo with gpg
>>
>> Kind regards,
>> Maxim.
>>
>> Alexander Smirnov (8):
>> base-apt: Add helper class
>> base-apt: Introduce base implementation
>> isar-boot-strap: Add option to keep cache
>> image: Add cache_base_repo task
>> isar-bootstrap: Make possible to reuse the cache
>> buildchroot: Make it buildable from base-apt
>> workaround: Use --allow-unauthenticated working with base-apt
>> local.conf: Add option to use cached base repository
>>
>> Maxim Yu. Osipov (2):
>> meta: Unify path names to local repositories
>> doc: Creation of local apt repo caching upstream Debian packages
>>
>> doc/user_manual.md | 39 ++++++++++++++++
>> meta-isar/conf/layer.conf | 10 ++--
>> meta-isar/conf/local.conf.sample | 4 ++
>> meta-isar/recipes-core/images/isar-image-base.bb | 3 +-
>> meta/classes/base-apt-helper.bbclass | 54 ++++++++++++++++++++++
>> meta/classes/buildchroot.bbclass | 2 +-
>> meta/classes/dpkg-base.bbclass | 10 ++--
>> meta/classes/dpkg.bbclass | 2 +-
>> meta/classes/image.bbclass | 25 +++++++++-
>> meta/classes/isar-bootstrap-helper.bbclass | 17 +++++--
>> .../isar-bootstrap/files/base-apt-sources | 1 +
>> .../recipes-core/isar-bootstrap/isar-bootstrap.inc | 32 +++++++++----
>> meta/recipes-devtools/base-apt/base-apt.bb | 31 +++++++++++++
>> .../base-apt/files/distributions.in | 3 ++
>> meta/recipes-devtools/buildchroot/files/deps.sh | 2 +-
>> meta/recipes-devtools/isar-apt/isar-apt.bb | 8 ++--
>> meta/recipes-kernel/linux/linux-custom.inc | 2 +-
>> 17 files changed, 213 insertions(+), 32 deletions(-)
>> create mode 100644 meta/classes/base-apt-helper.bbclass
>> create mode 100644 meta/recipes-core/isar-bootstrap/files/base-apt-sources
>> create mode 100644 meta/recipes-devtools/base-apt/base-apt.bb
>> create mode 100644 meta/recipes-devtools/base-apt/files/distributions.in
>>
>
> Could it be we have some regression here? After updating to current
> next, a build without the cache enabled fails during do_rootfs with
>
> | (Reading database ... 9600 files and directories currently installed.)
> | Removing isar-cfg-localepurge (1.0+isar) ...
> | find: '/build/tmp/work/debian-stretch-armhf/isar-image-base/rootfs/var/cache/apt/archives/partial': Permission denied
> | chown: invalid user: '/build/tmp/work/debian-stretch-armhf/isar-image-base/apt_cache/adduser_3.115_all.deb'
> | WARNING: exit code 1 from a shell command.
> | ERROR: Function failed: do_rootfs (log file is located at /build/tmp/work/debian-stretch-armhf/isar-image-base/temp/log.do_rootfs.5627)
>
> I do not have the time to bisect, will just jump back for now.
Hmmm... Of course, I've run CI build before applying to the 'next'
(see http://isar-build.org:8080/job/isar_mosipov_next/61).
I'll check if your problem is reproduced when build is not started from
the scratch.
Regards,
Maxim.
--
Maxim Osipov
ilbers GmbH
Maria-Merian-Str. 8
85521 Ottobrunn
Germany
+49 (151) 6517 6917
mosipov@ilbers.de
http://ilbers.de/
Commercial register Munich, HRB 214197
General Manager: Baurzhan Ismagulov
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [PATCH v2 00/10] Introduce local apt repo to cache upstream debian packages for offline usage
2018-10-17 18:15 ` Maxim Yu. Osipov
@ 2018-10-23 9:56 ` Maxim Yu. Osipov
2018-11-19 8:48 ` Baurzhan Ismagulov
0 siblings, 1 reply; 25+ messages in thread
From: Maxim Yu. Osipov @ 2018-10-23 9:56 UTC (permalink / raw)
To: Jan Kiszka, isar-users
Hi Jan,
On 10/17/18 9:15 PM, Maxim Yu. Osipov wrote:
> On 10/17/18 8:11 PM, Jan Kiszka wrote:
>> On 11.10.18 18:52, Maxim Yu. Osipov wrote:
>>> Hello everybody,
>>>
>>> This series contains fixes/improvements suggested by Claudius Heine
>>> during v1 series review.
>>>
>>> Changes to v2:
>>> - Fixes/improvements found in base-apt-helper class
>>> - Unifies path names to local repositories
>>>
>>> 1) Introduces dedicated local apt (base-apt) repo for upstream Debian
>>> packages
>>>
>>> 2) Caches in base-apt repo upstream Debian packages during image
>>> generation.
>>>
>>> 3) After this step, image can be built offline using only base-apt repo.
>>>
>>> Usage instructions can be found in the last patch.
>>>
>>> TODO:
>>> Sign the repo with gpg
>>>
>>> Kind regards,
>>> Maxim.
>>>
>>> Alexander Smirnov (8):
>>> base-apt: Add helper class
>>> base-apt: Introduce base implementation
>>> isar-boot-strap: Add option to keep cache
>>> image: Add cache_base_repo task
>>> isar-bootstrap: Make possible to reuse the cache
>>> buildchroot: Make it buildable from base-apt
>>> workaround: Use --allow-unauthenticated working with base-apt
>>> local.conf: Add option to use cached base repository
>>>
>>> Maxim Yu. Osipov (2):
>>> meta: Unify path names to local repositories
>>> doc: Creation of local apt repo caching upstream Debian packages
>>>
>>> doc/user_manual.md | 39
>>> ++++++++++++++++
>>> meta-isar/conf/layer.conf | 10 ++--
>>> meta-isar/conf/local.conf.sample | 4 ++
>>> meta-isar/recipes-core/images/isar-image-base.bb | 3 +-
>>> meta/classes/base-apt-helper.bbclass | 54
>>> ++++++++++++++++++++++
>>> meta/classes/buildchroot.bbclass | 2 +-
>>> meta/classes/dpkg-base.bbclass | 10 ++--
>>> meta/classes/dpkg.bbclass | 2 +-
>>> meta/classes/image.bbclass | 25 +++++++++-
>>> meta/classes/isar-bootstrap-helper.bbclass | 17 +++++--
>>> .../isar-bootstrap/files/base-apt-sources | 1 +
>>> .../recipes-core/isar-bootstrap/isar-bootstrap.inc | 32 +++++++++----
>>> meta/recipes-devtools/base-apt/base-apt.bb | 31 +++++++++++++
>>> .../base-apt/files/distributions.in | 3 ++
>>> meta/recipes-devtools/buildchroot/files/deps.sh | 2 +-
>>> meta/recipes-devtools/isar-apt/isar-apt.bb | 8 ++--
>>> meta/recipes-kernel/linux/linux-custom.inc | 2 +-
>>> 17 files changed, 213 insertions(+), 32 deletions(-)
>>> create mode 100644 meta/classes/base-apt-helper.bbclass
>>> create mode 100644
>>> meta/recipes-core/isar-bootstrap/files/base-apt-sources
>>> create mode 100644 meta/recipes-devtools/base-apt/base-apt.bb
>>> create mode 100644
>>> meta/recipes-devtools/base-apt/files/distributions.in
>>>
>>
>> Could it be we have some regression here? After updating to current
>> next, a build without the cache enabled fails during do_rootfs with
>>
>> | (Reading database ... 9600 files and directories currently installed.)
>> | Removing isar-cfg-localepurge (1.0+isar) ...
>> | find:
>> '/build/tmp/work/debian-stretch-armhf/isar-image-base/rootfs/var/cache/apt/archives/partial':
>> Permission denied
>> | chown: invalid user:
>> '/build/tmp/work/debian-stretch-armhf/isar-image-base/apt_cache/adduser_3.115_all.deb'
>>
>> | WARNING: exit code 1 from a shell command.
>> | ERROR: Function failed: do_rootfs (log file is located at
>> /build/tmp/work/debian-stretch-armhf/isar-image-base/temp/log.do_rootfs.5627)
>>
>>
>> I do not have the time to bisect, will just jump back for now.
>
> Hmmm... Of course, I've run CI build before applying to the 'next'
> (see http://isar-build.org:8080/job/isar_mosipov_next/61).
>
> I'll check if your problem is reproduced when build is not started from
> the scratch.
I've built (cross compiled from the scratch) the
multiconfig:qemuarm-stretch:isar-image-base, after that I've checkouted
'next' branch and ran again 'bitbake
multiconfig:qemuarm-stretch:isar-image-base' in the same build
directory. Build was OK. Did you have another test case?
Kind regards,
Maxim.
--
Maxim Osipov
ilbers GmbH
Maria-Merian-Str. 8
85521 Ottobrunn
Germany
+49 (151) 6517 6917
mosipov@ilbers.de
http://ilbers.de/
Commercial register Munich, HRB 214197
General Manager: Baurzhan Ismagulov
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [PATCH v2 00/10] Introduce local apt repo to cache upstream debian packages for offline usage
2018-10-23 9:56 ` Maxim Yu. Osipov
@ 2018-11-19 8:48 ` Baurzhan Ismagulov
2018-11-19 8:58 ` Jan Kiszka
0 siblings, 1 reply; 25+ messages in thread
From: Baurzhan Ismagulov @ 2018-11-19 8:48 UTC (permalink / raw)
To: Jan Kiszka; +Cc: isar-users
Hello Jan,
On Tue, Oct 23, 2018 at 12:56:11PM +0300, Maxim Yu. Osipov wrote:
> > > Could it be we have some regression here? After updating to current
> > > next, a build without the cache enabled fails during do_rootfs with
> > >
> > > | (Reading database ... 9600 files and directories currently installed.)
> > > | Removing isar-cfg-localepurge (1.0+isar) ...
> > > | find: '/build/tmp/work/debian-stretch-armhf/isar-image-base/rootfs/var/cache/apt/archives/partial': Permission denied
> > > | chown: invalid user: '/build/tmp/work/debian-stretch-armhf/isar-image-base/apt_cache/adduser_3.115_all.deb'
> > > | WARNING: exit code 1 from a shell command.
> > > | ERROR: Function failed: do_rootfs (log file is located at /build/tmp/work/debian-stretch-armhf/isar-image-base/temp/log.do_rootfs.5627)
...
> I've built (cross compiled from the scratch) the
> multiconfig:qemuarm-stretch:isar-image-base, after that I've checkouted
> 'next' branch and ran again 'bitbake
> multiconfig:qemuarm-stretch:isar-image-base' in the same build directory.
> Build was OK. Did you have another test case?
This issue blocks reproducibility. How can we reproduce it?
With kind regards,
Baurzhan.
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [PATCH v2 00/10] Introduce local apt repo to cache upstream debian packages for offline usage
2018-11-19 8:48 ` Baurzhan Ismagulov
@ 2018-11-19 8:58 ` Jan Kiszka
0 siblings, 0 replies; 25+ messages in thread
From: Jan Kiszka @ 2018-11-19 8:58 UTC (permalink / raw)
To: isar-users
On 19.11.18 09:48, Baurzhan Ismagulov wrote:
> Hello Jan,
>
> On Tue, Oct 23, 2018 at 12:56:11PM +0300, Maxim Yu. Osipov wrote:
>>>> Could it be we have some regression here? After updating to current
>>>> next, a build without the cache enabled fails during do_rootfs with
>>>>
>>>> | (Reading database ... 9600 files and directories currently installed.)
>>>> | Removing isar-cfg-localepurge (1.0+isar) ...
>>>> | find: '/build/tmp/work/debian-stretch-armhf/isar-image-base/rootfs/var/cache/apt/archives/partial': Permission denied
>>>> | chown: invalid user: '/build/tmp/work/debian-stretch-armhf/isar-image-base/apt_cache/adduser_3.115_all.deb'
>>>> | WARNING: exit code 1 from a shell command.
>>>> | ERROR: Function failed: do_rootfs (log file is located at /build/tmp/work/debian-stretch-armhf/isar-image-base/temp/log.do_rootfs.5627)
> ...
>> I've built (cross compiled from the scratch) the
>> multiconfig:qemuarm-stretch:isar-image-base, after that I've checkouted
>> 'next' branch and ran again 'bitbake
>> multiconfig:qemuarm-stretch:isar-image-base' in the same build directory.
>> Build was OK. Did you have another test case?
>
> This issue blocks reproducibility. How can we reproduce it?
>
Might have been a glitch related to artifacts left over from builds with older
versions. A lot of things changed since then, specifically regarding
permissions. I currently do not see this issue, I rather see others
(bitbake...). Let's just move on.
Jan
--
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [PATCH v2 00/10] Introduce local apt repo to cache upstream debian packages for offline usage
2018-10-11 16:52 [PATCH v2 00/10] Introduce local apt repo to cache upstream debian packages for offline usage Maxim Yu. Osipov
` (11 preceding siblings ...)
2018-10-17 17:11 ` Jan Kiszka
@ 2018-10-23 16:14 ` Henning Schild
2018-10-29 12:28 ` Baurzhan Ismagulov
12 siblings, 1 reply; 25+ messages in thread
From: Henning Schild @ 2018-10-23 16:14 UTC (permalink / raw)
To: Maxim Yu. Osipov; +Cc: isar-users
Hey,
i am kind of missing some context here. Is this really just a cache to
remove deb fetching from the build and allow offline builds? Or is this
somehow targeting reproducable builds?
If the latter is true i would like to see a summary of the earlier
discussions, a detailed description of what is happening and why. All
the commit messages of the series are far from verbose and the doc does
not make it much better.
Henning
Am Thu, 11 Oct 2018 18:52:55 +0200
schrieb "Maxim Yu. Osipov" <mosipov@ilbers.de>:
> Hello everybody,
>
> This series contains fixes/improvements suggested by Claudius Heine
> during v1 series review.
>
> Changes to v2:
> - Fixes/improvements found in base-apt-helper class
> - Unifies path names to local repositories
>
> 1) Introduces dedicated local apt (base-apt) repo for upstream Debian
> packages
>
> 2) Caches in base-apt repo upstream Debian packages during image
> generation.
>
> 3) After this step, image can be built offline using only base-apt
> repo.
>
> Usage instructions can be found in the last patch.
>
> TODO:
> Sign the repo with gpg
>
> Kind regards,
> Maxim.
>
> Alexander Smirnov (8):
> base-apt: Add helper class
> base-apt: Introduce base implementation
> isar-boot-strap: Add option to keep cache
> image: Add cache_base_repo task
> isar-bootstrap: Make possible to reuse the cache
> buildchroot: Make it buildable from base-apt
> workaround: Use --allow-unauthenticated working with base-apt
> local.conf: Add option to use cached base repository
>
> Maxim Yu. Osipov (2):
> meta: Unify path names to local repositories
> doc: Creation of local apt repo caching upstream Debian packages
>
> doc/user_manual.md | 39
> ++++++++++++++++ meta-isar/conf/layer.conf |
> 10 ++-- meta-isar/conf/local.conf.sample | 4 ++
> meta-isar/recipes-core/images/isar-image-base.bb | 3 +-
> meta/classes/base-apt-helper.bbclass | 54
> ++++++++++++++++++++++
> meta/classes/buildchroot.bbclass | 2 +-
> meta/classes/dpkg-base.bbclass | 10 ++--
> meta/classes/dpkg.bbclass | 2 +-
> meta/classes/image.bbclass | 25 +++++++++-
> meta/classes/isar-bootstrap-helper.bbclass | 17
> +++++-- .../isar-bootstrap/files/base-apt-sources | 1
> + .../recipes-core/isar-bootstrap/isar-bootstrap.inc | 32
> +++++++++---- meta/recipes-devtools/base-apt/base-apt.bb | 31
> +++++++++++++ .../base-apt/files/distributions.in | 3
> ++ meta/recipes-devtools/buildchroot/files/deps.sh | 2 +-
> meta/recipes-devtools/isar-apt/isar-apt.bb | 8 ++--
> meta/recipes-kernel/linux/linux-custom.inc | 2 +- 17 files
> changed, 213 insertions(+), 32 deletions(-) create mode 100644
> meta/classes/base-apt-helper.bbclass create mode 100644
> meta/recipes-core/isar-bootstrap/files/base-apt-sources create mode
> 100644 meta/recipes-devtools/base-apt/base-apt.bb create mode 100644
> meta/recipes-devtools/base-apt/files/distributions.in
>
^ permalink raw reply [flat|nested] 25+ messages in thread* Re: [PATCH v2 00/10] Introduce local apt repo to cache upstream debian packages for offline usage
2018-10-23 16:14 ` Henning Schild
@ 2018-10-29 12:28 ` Baurzhan Ismagulov
0 siblings, 0 replies; 25+ messages in thread
From: Baurzhan Ismagulov @ 2018-10-29 12:28 UTC (permalink / raw)
To: isar-users
On Tue, Oct 23, 2018 at 06:14:56PM +0200, Henning Schild wrote:
> i am kind of missing some context here. Is this really just a cache to
> remove deb fetching from the build and allow offline builds? Or is this
> somehow targeting reproducable builds?
>
> If the latter is true i would like to see a summary of the earlier
> discussions, a detailed description of what is happening and why.
Yes, this is just a cache to allow offline builds. And yes, this is a small
step 0 towards reproducibility.
As I see it, previous discussions boil down to:
1. Doing automatic reproducibility right is complex and should be done in
steps.
2. Initial caching is a low-hanging fruit that can be addressed now and is
better than nothing.
This series provides (2).
For (1), I suggest to introduce a mechanism first. It should be usable
manually. It should start from minimal essential use cases (initial creation)
and be possibly extended to further use cases (update all, update single,
update single with deps, remove, etc.) as necessary. If desired, we could look
at optionally building it into the workflow of Isar after the mechanism
stabilizes (with policies such as automatically fetch on the first build,
update on every build, etc.). We can provide more details along the way.
On a related note, we've discussed these issues at ELCE. libapt and python-apt
seem to be what we need for the implementation.
With kind regards,
Baurzhan.
^ permalink raw reply [flat|nested] 25+ messages in thread