* [PATCH] debootstrap: Add fallback apt cache support @ 2018-05-11 15:35 Alexander Smirnov 2018-05-14 9:27 ` Claudius Heine 0 siblings, 1 reply; 6+ messages in thread From: Alexander Smirnov @ 2018-05-11 15:35 UTC (permalink / raw) To: isar-users; +Cc: Alexander Smirnov Add apt config file to switch caching mechanism from MMap to array-based. This needed to provide possibility to build fresh Isar suites on old hosts. For example, building Isar stretch on jessie host fails with the following error: Hit:1 http://security.debian.org stretch/updates InRelease Ign:2 http://ftp.de.debian.org/debian stretch InRelease Hit:3 http://ftp.de.debian.org/debian stretch-updates InRelease Hit:4 http://ftp.de.debian.org/debian stretch Release E: Dynamic MMap ran out of room. Please increase the size of APT::Cache-Start. Current value: 25165824. (man 5 apt.conf) qemu: uncaught target signal 11 (Segmentation fault) - core dumped Segmentation fault Now Isar performs the following: 1. Checking if user attempts to build stretch on old kernels 2. Temporarly installing this apt config file. 3. Removing this file after image rootfs creation So this file will not go to target rootfs. Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de> --- meta-isar/recipes-core/images/isar-image-base.bb | 2 ++ .../isar-bootstrap/files/isar-apt-fallback.conf | 4 ++++ meta/recipes-core/isar-bootstrap/isar-bootstrap.bb | 17 ++++++++++++++++- 3 files changed, 22 insertions(+), 1 deletion(-) create mode 100644 meta/recipes-core/isar-bootstrap/files/isar-apt-fallback.conf diff --git a/meta-isar/recipes-core/images/isar-image-base.bb b/meta-isar/recipes-core/images/isar-image-base.bb index 00e926a..d65b1f1 100644 --- a/meta-isar/recipes-core/images/isar-image-base.bb +++ b/meta-isar/recipes-core/images/isar-image-base.bb @@ -53,4 +53,6 @@ do_rootfs() { sudo rmdir ${IMAGE_ROOTFS}/isar-apt sudo umount -l ${IMAGE_ROOTFS}/dev sudo umount -l ${IMAGE_ROOTFS}/proc + [ -e "${IMAGE_ROOTFS}/etc/apt/apt.conf.d/55isar-fallback.conf" ] && \ + sudo rm "${IMAGE_ROOTFS}/etc/apt/apt.conf.d/55isar-fallback.conf" || true } diff --git a/meta/recipes-core/isar-bootstrap/files/isar-apt-fallback.conf b/meta/recipes-core/isar-bootstrap/files/isar-apt-fallback.conf new file mode 100644 index 0000000..12ddbea --- /dev/null +++ b/meta/recipes-core/isar-bootstrap/files/isar-apt-fallback.conf @@ -0,0 +1,4 @@ +# Switch apt caching mechanism from mmap to array-based. +# This helps to build fresh Isar suites on old hosts. + +APT::Cache-Fallback "1"; diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.bb b/meta/recipes-core/isar-bootstrap/isar-bootstrap.bb index a38dd88..a3a3a2b 100644 --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.bb +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.bb @@ -10,7 +10,9 @@ Description = "Minimal debian root file system" LICENSE = "gpl-2.0" LIC_FILES_CHKSUM = "file://${LAYERDIR_isar}/licenses/COPYING.GPLv2;md5=751419260aa954499f7abaabaa882bbe" FILESPATH_prepend := "${THISDIR}/files:" -SRC_URI = "file://isar-apt.conf" +SRC_URI = " \ + file://isar-apt.conf \ + file://isar-apt-fallback.conf" PV = "1.0" WORKDIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/${PN}" @@ -165,6 +167,13 @@ do_bootstrap() { } addtask bootstrap before do_build after do_apt_config_prepare +def get_host_release(): + import platform + rel = platform.release() + return rel + +HOST_KERN_VER = "${@get_host_release().split('.')[0]}" + do_apt_config_install[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}" do_apt_config_install() { sudo mkdir -p "${ROOTFSDIR}/etc/apt/preferences.d" @@ -177,6 +186,12 @@ do_apt_config_install() { sudo mkdir -p "${ROOTFSDIR}/etc/apt/apt.conf.d" sudo install -v -m644 "${WORKDIR}/isar-apt.conf" \ "${ROOTFSDIR}/etc/apt/apt.conf.d/50isar.conf" + + if [ "${@get_distro_suite(d)}" = "stretch" ] && [ "${HOST_KERN_VER}" -lt "4" ]; then + sudo install -v -m644 "${WORKDIR}/isar-apt-fallback.conf" \ + "${ROOTFSDIR}/etc/apt/apt.conf.d/55isar-fallback.conf" + fi + } addtask apt_config_install before do_build after do_bootstrap -- 2.1.4 ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH] debootstrap: Add fallback apt cache support 2018-05-11 15:35 [PATCH] debootstrap: Add fallback apt cache support Alexander Smirnov @ 2018-05-14 9:27 ` Claudius Heine 2018-05-14 9:40 ` Alexander Smirnov 0 siblings, 1 reply; 6+ messages in thread From: Claudius Heine @ 2018-05-14 9:27 UTC (permalink / raw) To: Alexander Smirnov, isar-users Hi Alex, so I guess that this patch replaces the 'isar-bootstrap: Increase cache room' one right? On 2018-05-11 17:35, Alexander Smirnov wrote: > Add apt config file to switch caching mechanism from MMap to array-based. > This needed to provide possibility to build fresh Isar suites on old hosts. > For example, building Isar stretch on jessie host fails with the following > error: > > Hit:1 http://security.debian.org stretch/updates InRelease > Ign:2 http://ftp.de.debian.org/debian stretch InRelease > Hit:3 http://ftp.de.debian.org/debian stretch-updates InRelease > Hit:4 http://ftp.de.debian.org/debian stretch Release > E: Dynamic MMap ran out of room. Please increase the size of APT::Cache-Start. Current value: 25165824. (man 5 apt.conf) > qemu: uncaught target signal 11 (Segmentation fault) - core dumped > Segmentation fault > > Now Isar performs the following: > 1. Checking if user attempts to build stretch on old kernels > 2. Temporarly installing this apt config file. > 3. Removing this file after image rootfs creation > > So this file will not go to target rootfs. > > Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de> > --- > meta-isar/recipes-core/images/isar-image-base.bb | 2 ++ > .../isar-bootstrap/files/isar-apt-fallback.conf | 4 ++++ > meta/recipes-core/isar-bootstrap/isar-bootstrap.bb | 17 ++++++++++++++++- > 3 files changed, 22 insertions(+), 1 deletion(-) > create mode 100644 meta/recipes-core/isar-bootstrap/files/isar-apt-fallback.conf > > diff --git a/meta-isar/recipes-core/images/isar-image-base.bb b/meta-isar/recipes-core/images/isar-image-base.bb > index 00e926a..d65b1f1 100644 > --- a/meta-isar/recipes-core/images/isar-image-base.bb > +++ b/meta-isar/recipes-core/images/isar-image-base.bb > @@ -53,4 +53,6 @@ do_rootfs() { > sudo rmdir ${IMAGE_ROOTFS}/isar-apt > sudo umount -l ${IMAGE_ROOTFS}/dev > sudo umount -l ${IMAGE_ROOTFS}/proc > + [ -e "${IMAGE_ROOTFS}/etc/apt/apt.conf.d/55isar-fallback.conf" ] && \ > + sudo rm "${IMAGE_ROOTFS}/etc/apt/apt.conf.d/55isar-fallback.conf" || true Why not just 'sudo rm -f "${IMAGE_ROOTFS}/etc/apt/apt.conf.d/55isar-fallback.conf"' ? > } > diff --git a/meta/recipes-core/isar-bootstrap/files/isar-apt-fallback.conf b/meta/recipes-core/isar-bootstrap/files/isar-apt-fallback.conf > new file mode 100644 > index 0000000..12ddbea > --- /dev/null > +++ b/meta/recipes-core/isar-bootstrap/files/isar-apt-fallback.conf > @@ -0,0 +1,4 @@ > +# Switch apt caching mechanism from mmap to array-based. > +# This helps to build fresh Isar suites on old hosts. > + > +APT::Cache-Fallback "1"; > diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.bb b/meta/recipes-core/isar-bootstrap/isar-bootstrap.bb > index a38dd88..a3a3a2b 100644 > --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.bb > +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.bb > @@ -10,7 +10,9 @@ Description = "Minimal debian root file system" > LICENSE = "gpl-2.0" > LIC_FILES_CHKSUM = "file://${LAYERDIR_isar}/licenses/COPYING.GPLv2;md5=751419260aa954499f7abaabaa882bbe" > FILESPATH_prepend := "${THISDIR}/files:" > -SRC_URI = "file://isar-apt.conf" > +SRC_URI = " \ > + file://isar-apt.conf \ > + file://isar-apt-fallback.conf" > PV = "1.0" > > WORKDIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/${PN}" > @@ -165,6 +167,13 @@ do_bootstrap() { > } > addtask bootstrap before do_build after do_apt_config_prepare > > +def get_host_release(): > + import platform > + rel = platform.release() > + return rel > + > +HOST_KERN_VER = "${@get_host_release().split('.')[0]}" If you need this variable in just one place. It might be useful to just add it there. > + > do_apt_config_install[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}" > do_apt_config_install() { > sudo mkdir -p "${ROOTFSDIR}/etc/apt/preferences.d" > @@ -177,6 +186,12 @@ do_apt_config_install() { > sudo mkdir -p "${ROOTFSDIR}/etc/apt/apt.conf.d" > sudo install -v -m644 "${WORKDIR}/isar-apt.conf" \ > "${ROOTFSDIR}/etc/apt/apt.conf.d/50isar.conf" > + > + if [ "${@get_distro_suite(d)}" = "stretch" ] && [ "${HOST_KERN_VER}" -lt "4" ]; then Like this: if [ "${@get_distro_suite(d)}" = "stretch" ] && [ "${@get_host_release().split('.')[0]}" -lt "4" ]; then Or is it useful to overwrite this variable somewhere? > + sudo install -v -m644 "${WORKDIR}/isar-apt-fallback.conf" \ > + "${ROOTFSDIR}/etc/apt/apt.conf.d/55isar-fallback.conf" > + fi > + > } > addtask apt_config_install before do_build after do_bootstrap Apart from those small nitpicks it looks good, Claudius -- DENX Software Engineering GmbH, Managing Director: Wolfgang Denk HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH] debootstrap: Add fallback apt cache support 2018-05-14 9:27 ` Claudius Heine @ 2018-05-14 9:40 ` Alexander Smirnov 2018-05-14 10:33 ` Claudius Heine 0 siblings, 1 reply; 6+ messages in thread From: Alexander Smirnov @ 2018-05-14 9:40 UTC (permalink / raw) To: Claudius Heine, isar-users On 05/14/2018 12:27 PM, Claudius Heine wrote: > Hi Alex, > > so I guess that this patch replaces the 'isar-bootstrap: Increase cache > room' one right? > > On 2018-05-11 17:35, Alexander Smirnov wrote: >> Add apt config file to switch caching mechanism from MMap to array-based. >> This needed to provide possibility to build fresh Isar suites on old >> hosts. >> For example, building Isar stretch on jessie host fails with the >> following >> error: >> >> Hit:1 http://security.debian.org stretch/updates InRelease >> Ign:2 http://ftp.de.debian.org/debian stretch InRelease >> Hit:3 http://ftp.de.debian.org/debian stretch-updates InRelease >> Hit:4 http://ftp.de.debian.org/debian stretch Release >> E: Dynamic MMap ran out of room. Please increase the size of >> APT::Cache-Start. Current value: 25165824. (man 5 apt.conf) >> qemu: uncaught target signal 11 (Segmentation fault) - core dumped >> Segmentation fault >> >> Now Isar performs the following: >> 1. Checking if user attempts to build stretch on old kernels >> 2. Temporarly installing this apt config file. >> 3. Removing this file after image rootfs creation >> >> So this file will not go to target rootfs. >> >> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de> >> --- >> meta-isar/recipes-core/images/isar-image-base.bb | 2 ++ >> .../isar-bootstrap/files/isar-apt-fallback.conf | 4 ++++ >> meta/recipes-core/isar-bootstrap/isar-bootstrap.bb | 17 >> ++++++++++++++++- >> 3 files changed, 22 insertions(+), 1 deletion(-) >> create mode 100644 >> meta/recipes-core/isar-bootstrap/files/isar-apt-fallback.conf >> >> diff --git a/meta-isar/recipes-core/images/isar-image-base.bb >> b/meta-isar/recipes-core/images/isar-image-base.bb >> index 00e926a..d65b1f1 100644 >> --- a/meta-isar/recipes-core/images/isar-image-base.bb >> +++ b/meta-isar/recipes-core/images/isar-image-base.bb >> @@ -53,4 +53,6 @@ do_rootfs() { >> sudo rmdir ${IMAGE_ROOTFS}/isar-apt >> sudo umount -l ${IMAGE_ROOTFS}/dev >> sudo umount -l ${IMAGE_ROOTFS}/proc >> + [ -e "${IMAGE_ROOTFS}/etc/apt/apt.conf.d/55isar-fallback.conf" ] >> && \ >> + sudo rm >> "${IMAGE_ROOTFS}/etc/apt/apt.conf.d/55isar-fallback.conf" || true > > Why not just 'sudo rm -f > "${IMAGE_ROOTFS}/etc/apt/apt.conf.d/55isar-fallback.conf"' ? > Matter of taste, I don't like to remove things that don't exist. > >> } >> diff --git >> a/meta/recipes-core/isar-bootstrap/files/isar-apt-fallback.conf >> b/meta/recipes-core/isar-bootstrap/files/isar-apt-fallback.conf >> new file mode 100644 >> index 0000000..12ddbea >> --- /dev/null >> +++ b/meta/recipes-core/isar-bootstrap/files/isar-apt-fallback.conf >> @@ -0,0 +1,4 @@ >> +# Switch apt caching mechanism from mmap to array-based. >> +# This helps to build fresh Isar suites on old hosts. >> + >> +APT::Cache-Fallback "1"; >> diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.bb >> b/meta/recipes-core/isar-bootstrap/isar-bootstrap.bb >> index a38dd88..a3a3a2b 100644 >> --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.bb >> +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.bb >> @@ -10,7 +10,9 @@ Description = "Minimal debian root file system" >> LICENSE = "gpl-2.0" >> LIC_FILES_CHKSUM = >> "file://${LAYERDIR_isar}/licenses/COPYING.GPLv2;md5=751419260aa954499f7abaabaa882bbe" >> >> FILESPATH_prepend := "${THISDIR}/files:" >> -SRC_URI = "file://isar-apt.conf" >> +SRC_URI = " \ >> + file://isar-apt.conf \ >> + file://isar-apt-fallback.conf" >> PV = "1.0" >> WORKDIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/${PN}" >> @@ -165,6 +167,13 @@ do_bootstrap() { >> } >> addtask bootstrap before do_build after do_apt_config_prepare >> +def get_host_release(): >> + import platform >> + rel = platform.release() >> + return rel >> + >> +HOST_KERN_VER = "${@get_host_release().split('.')[0]}" > If you need this variable in just one place. It might be useful to just > add it there. > I've added the separate variable, because probably some additional version specification could be needed later, like HOST_KERN_MAJOR etc., in this case python inlining could be difficult for reading. But for today I have only one case, so could drop it in v3. Alex >> + >> do_apt_config_install[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}" >> do_apt_config_install() { >> sudo mkdir -p "${ROOTFSDIR}/etc/apt/preferences.d" >> @@ -177,6 +186,12 @@ do_apt_config_install() { >> sudo mkdir -p "${ROOTFSDIR}/etc/apt/apt.conf.d" >> sudo install -v -m644 "${WORKDIR}/isar-apt.conf" \ >> "${ROOTFSDIR}/etc/apt/apt.conf.d/50isar.conf" >> + >> + if [ "${@get_distro_suite(d)}" = "stretch" ] && [ >> "${HOST_KERN_VER}" -lt "4" ]; then > > Like this: > > if [ "${@get_distro_suite(d)}" = "stretch" ] && [ > "${@get_host_release().split('.')[0]}" -lt "4" ]; then > > Or is it useful to overwrite this variable somewhere? > >> + sudo install -v -m644 "${WORKDIR}/isar-apt-fallback.conf" \ >> + >> "${ROOTFSDIR}/etc/apt/apt.conf.d/55isar-fallback.conf" >> + fi >> + >> } >> addtask apt_config_install before do_build after do_bootstrap > > Apart from those small nitpicks it looks good, > Claudius > ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH] debootstrap: Add fallback apt cache support 2018-05-14 9:40 ` Alexander Smirnov @ 2018-05-14 10:33 ` Claudius Heine 2018-05-14 11:11 ` Alexander Smirnov 0 siblings, 1 reply; 6+ messages in thread From: Claudius Heine @ 2018-05-14 10:33 UTC (permalink / raw) To: Alexander Smirnov, isar-users Hi Alex, On 2018-05-14 11:40, Alexander Smirnov wrote: > > On 05/14/2018 12:27 PM, Claudius Heine wrote: >> Hi Alex, >> >> so I guess that this patch replaces the 'isar-bootstrap: Increase >> cache room' one right? >> >> On 2018-05-11 17:35, Alexander Smirnov wrote: >>> Add apt config file to switch caching mechanism from MMap to >>> array-based. >>> This needed to provide possibility to build fresh Isar suites on old >>> hosts. >>> For example, building Isar stretch on jessie host fails with the >>> following >>> error: >>> >>> Hit:1 http://security.debian.org stretch/updates InRelease >>> Ign:2 http://ftp.de.debian.org/debian stretch InRelease >>> Hit:3 http://ftp.de.debian.org/debian stretch-updates InRelease >>> Hit:4 http://ftp.de.debian.org/debian stretch Release >>> E: Dynamic MMap ran out of room. Please increase the size of >>> APT::Cache-Start. Current value: 25165824. (man 5 apt.conf) >>> qemu: uncaught target signal 11 (Segmentation fault) - core dumped >>> Segmentation fault >>> >>> Now Isar performs the following: >>> 1. Checking if user attempts to build stretch on old kernels >>> 2. Temporarly installing this apt config file. >>> 3. Removing this file after image rootfs creation >>> >>> So this file will not go to target rootfs. >>> >>> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de> >>> --- >>> meta-isar/recipes-core/images/isar-image-base.bb | 2 ++ >>> .../isar-bootstrap/files/isar-apt-fallback.conf | 4 ++++ >>> meta/recipes-core/isar-bootstrap/isar-bootstrap.bb | 17 >>> ++++++++++++++++- >>> 3 files changed, 22 insertions(+), 1 deletion(-) >>> create mode 100644 >>> meta/recipes-core/isar-bootstrap/files/isar-apt-fallback.conf >>> >>> diff --git a/meta-isar/recipes-core/images/isar-image-base.bb >>> b/meta-isar/recipes-core/images/isar-image-base.bb >>> index 00e926a..d65b1f1 100644 >>> --- a/meta-isar/recipes-core/images/isar-image-base.bb >>> +++ b/meta-isar/recipes-core/images/isar-image-base.bb >>> @@ -53,4 +53,6 @@ do_rootfs() { >>> sudo rmdir ${IMAGE_ROOTFS}/isar-apt >>> sudo umount -l ${IMAGE_ROOTFS}/dev >>> sudo umount -l ${IMAGE_ROOTFS}/proc >>> + [ -e "${IMAGE_ROOTFS}/etc/apt/apt.conf.d/55isar-fallback.conf" ] >>> && \ >>> + sudo rm >>> "${IMAGE_ROOTFS}/etc/apt/apt.conf.d/55isar-fallback.conf" || true >> >> Why not just 'sudo rm -f >> "${IMAGE_ROOTFS}/etc/apt/apt.conf.d/55isar-fallback.conf"' ? >> > > Matter of taste, I don't like to remove things that don't exist. I guess it is a matter of taste. I do think '|| true' is ugly and testing the existence of the file before removing it redundant. Cheers, Claudius > >> >>> } >>> diff --git >>> a/meta/recipes-core/isar-bootstrap/files/isar-apt-fallback.conf >>> b/meta/recipes-core/isar-bootstrap/files/isar-apt-fallback.conf >>> new file mode 100644 >>> index 0000000..12ddbea >>> --- /dev/null >>> +++ b/meta/recipes-core/isar-bootstrap/files/isar-apt-fallback.conf >>> @@ -0,0 +1,4 @@ >>> +# Switch apt caching mechanism from mmap to array-based. >>> +# This helps to build fresh Isar suites on old hosts. >>> + >>> +APT::Cache-Fallback "1"; >>> diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.bb >>> b/meta/recipes-core/isar-bootstrap/isar-bootstrap.bb >>> index a38dd88..a3a3a2b 100644 >>> --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.bb >>> +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.bb >>> @@ -10,7 +10,9 @@ Description = "Minimal debian root file system" >>> LICENSE = "gpl-2.0" >>> LIC_FILES_CHKSUM = >>> "file://${LAYERDIR_isar}/licenses/COPYING.GPLv2;md5=751419260aa954499f7abaabaa882bbe" >>> >>> FILESPATH_prepend := "${THISDIR}/files:" >>> -SRC_URI = "file://isar-apt.conf" >>> +SRC_URI = " \ >>> + file://isar-apt.conf \ >>> + file://isar-apt-fallback.conf" >>> PV = "1.0" >>> WORKDIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/${PN}" >>> @@ -165,6 +167,13 @@ do_bootstrap() { >>> } >>> addtask bootstrap before do_build after do_apt_config_prepare >>> +def get_host_release(): >>> + import platform >>> + rel = platform.release() >>> + return rel >>> + >>> +HOST_KERN_VER = "${@get_host_release().split('.')[0]}" >> If you need this variable in just one place. It might be useful to >> just add it there. >> > > I've added the separate variable, because probably some additional > version specification could be needed later, like HOST_KERN_MAJOR etc., > in this case python inlining could be difficult for reading. > > But for today I have only one case, so could drop it in v3. > > Alex > >>> + >>> do_apt_config_install[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}" >>> do_apt_config_install() { >>> sudo mkdir -p "${ROOTFSDIR}/etc/apt/preferences.d" >>> @@ -177,6 +186,12 @@ do_apt_config_install() { >>> sudo mkdir -p "${ROOTFSDIR}/etc/apt/apt.conf.d" >>> sudo install -v -m644 "${WORKDIR}/isar-apt.conf" \ >>> >>> "${ROOTFSDIR}/etc/apt/apt.conf.d/50isar.conf" >>> + >>> + if [ "${@get_distro_suite(d)}" = "stretch" ] && [ >>> "${HOST_KERN_VER}" -lt "4" ]; then >> >> Like this: >> >> if [ "${@get_distro_suite(d)}" = "stretch" ] && [ >> "${@get_host_release().split('.')[0]}" -lt "4" ]; then >> >> Or is it useful to overwrite this variable somewhere? >> >>> + sudo install -v -m644 "${WORKDIR}/isar-apt-fallback.conf" \ >>> + "${ROOTFSDIR}/etc/apt/apt.conf.d/55isar-fallback.conf" >>> + fi >>> + >>> } >>> addtask apt_config_install before do_build after do_bootstrap >> >> Apart from those small nitpicks it looks good, >> Claudius >> -- DENX Software Engineering GmbH, Managing Director: Wolfgang Denk HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH] debootstrap: Add fallback apt cache support 2018-05-14 10:33 ` Claudius Heine @ 2018-05-14 11:11 ` Alexander Smirnov 2018-05-14 11:25 ` Claudius Heine 0 siblings, 1 reply; 6+ messages in thread From: Alexander Smirnov @ 2018-05-14 11:11 UTC (permalink / raw) To: Claudius Heine, isar-users On 05/14/2018 01:33 PM, Claudius Heine wrote: > Hi Alex, > > On 2018-05-14 11:40, Alexander Smirnov wrote: >> >> On 05/14/2018 12:27 PM, Claudius Heine wrote: >>> Hi Alex, >>> >>> so I guess that this patch replaces the 'isar-bootstrap: Increase >>> cache room' one right? >>> >>> On 2018-05-11 17:35, Alexander Smirnov wrote: >>>> Add apt config file to switch caching mechanism from MMap to >>>> array-based. >>>> This needed to provide possibility to build fresh Isar suites on old >>>> hosts. >>>> For example, building Isar stretch on jessie host fails with the >>>> following >>>> error: >>>> >>>> Hit:1 http://security.debian.org stretch/updates InRelease >>>> Ign:2 http://ftp.de.debian.org/debian stretch InRelease >>>> Hit:3 http://ftp.de.debian.org/debian stretch-updates InRelease >>>> Hit:4 http://ftp.de.debian.org/debian stretch Release >>>> E: Dynamic MMap ran out of room. Please increase the size of >>>> APT::Cache-Start. Current value: 25165824. (man 5 apt.conf) >>>> qemu: uncaught target signal 11 (Segmentation fault) - core dumped >>>> Segmentation fault >>>> >>>> Now Isar performs the following: >>>> 1. Checking if user attempts to build stretch on old kernels >>>> 2. Temporarly installing this apt config file. >>>> 3. Removing this file after image rootfs creation >>>> >>>> So this file will not go to target rootfs. >>>> >>>> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de> >>>> --- >>>> meta-isar/recipes-core/images/isar-image-base.bb | 2 ++ >>>> .../isar-bootstrap/files/isar-apt-fallback.conf | 4 ++++ >>>> meta/recipes-core/isar-bootstrap/isar-bootstrap.bb | 17 >>>> ++++++++++++++++- >>>> 3 files changed, 22 insertions(+), 1 deletion(-) >>>> create mode 100644 >>>> meta/recipes-core/isar-bootstrap/files/isar-apt-fallback.conf >>>> >>>> diff --git a/meta-isar/recipes-core/images/isar-image-base.bb >>>> b/meta-isar/recipes-core/images/isar-image-base.bb >>>> index 00e926a..d65b1f1 100644 >>>> --- a/meta-isar/recipes-core/images/isar-image-base.bb >>>> +++ b/meta-isar/recipes-core/images/isar-image-base.bb >>>> @@ -53,4 +53,6 @@ do_rootfs() { >>>> sudo rmdir ${IMAGE_ROOTFS}/isar-apt >>>> sudo umount -l ${IMAGE_ROOTFS}/dev >>>> sudo umount -l ${IMAGE_ROOTFS}/proc >>>> + [ -e "${IMAGE_ROOTFS}/etc/apt/apt.conf.d/55isar-fallback.conf" >>>> ] && \ >>>> + sudo rm >>>> "${IMAGE_ROOTFS}/etc/apt/apt.conf.d/55isar-fallback.conf" || true >>> >>> Why not just 'sudo rm -f >>> "${IMAGE_ROOTFS}/etc/apt/apt.conf.d/55isar-fallback.conf"' ? >>> >> >> Matter of taste, I don't like to remove things that don't exist. > > I guess it is a matter of taste. I do think '|| true' is ugly and If you do think so, why then you use it in your isar-debootstrap implementation? > testing the existence of the file before removing it redundant. > For me it's not too valuable issue, so I could drop this check in v3. Alex > Cheers, > Claudius > >> >>> >>>> } >>>> diff --git >>>> a/meta/recipes-core/isar-bootstrap/files/isar-apt-fallback.conf >>>> b/meta/recipes-core/isar-bootstrap/files/isar-apt-fallback.conf >>>> new file mode 100644 >>>> index 0000000..12ddbea >>>> --- /dev/null >>>> +++ b/meta/recipes-core/isar-bootstrap/files/isar-apt-fallback.conf >>>> @@ -0,0 +1,4 @@ >>>> +# Switch apt caching mechanism from mmap to array-based. >>>> +# This helps to build fresh Isar suites on old hosts. >>>> + >>>> +APT::Cache-Fallback "1"; >>>> diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.bb >>>> b/meta/recipes-core/isar-bootstrap/isar-bootstrap.bb >>>> index a38dd88..a3a3a2b 100644 >>>> --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.bb >>>> +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.bb >>>> @@ -10,7 +10,9 @@ Description = "Minimal debian root file system" >>>> LICENSE = "gpl-2.0" >>>> LIC_FILES_CHKSUM = >>>> "file://${LAYERDIR_isar}/licenses/COPYING.GPLv2;md5=751419260aa954499f7abaabaa882bbe" >>>> >>>> FILESPATH_prepend := "${THISDIR}/files:" >>>> -SRC_URI = "file://isar-apt.conf" >>>> +SRC_URI = " \ >>>> + file://isar-apt.conf \ >>>> + file://isar-apt-fallback.conf" >>>> PV = "1.0" >>>> WORKDIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/${PN}" >>>> @@ -165,6 +167,13 @@ do_bootstrap() { >>>> } >>>> addtask bootstrap before do_build after do_apt_config_prepare >>>> +def get_host_release(): >>>> + import platform >>>> + rel = platform.release() >>>> + return rel >>>> + >>>> +HOST_KERN_VER = "${@get_host_release().split('.')[0]}" >>> If you need this variable in just one place. It might be useful to >>> just add it there. >>> >> >> I've added the separate variable, because probably some additional >> version specification could be needed later, like HOST_KERN_MAJOR >> etc., in this case python inlining could be difficult for reading. >> >> But for today I have only one case, so could drop it in v3. >> >> Alex >> >>>> + >>>> do_apt_config_install[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}" >>>> do_apt_config_install() { >>>> sudo mkdir -p "${ROOTFSDIR}/etc/apt/preferences.d" >>>> @@ -177,6 +186,12 @@ do_apt_config_install() { >>>> sudo mkdir -p "${ROOTFSDIR}/etc/apt/apt.conf.d" >>>> sudo install -v -m644 "${WORKDIR}/isar-apt.conf" \ >>>> "${ROOTFSDIR}/etc/apt/apt.conf.d/50isar.conf" >>>> + >>>> + if [ "${@get_distro_suite(d)}" = "stretch" ] && [ >>>> "${HOST_KERN_VER}" -lt "4" ]; then >>> >>> Like this: >>> >>> if [ "${@get_distro_suite(d)}" = "stretch" ] && [ >>> "${@get_host_release().split('.')[0]}" -lt "4" ]; then >>> >>> Or is it useful to overwrite this variable somewhere? >>> >>>> + sudo install -v -m644 "${WORKDIR}/isar-apt-fallback.conf" \ >>>> + "${ROOTFSDIR}/etc/apt/apt.conf.d/55isar-fallback.conf" >>>> + fi >>>> + >>>> } >>>> addtask apt_config_install before do_build after do_bootstrap >>> >>> Apart from those small nitpicks it looks good, >>> Claudius >>> > -- With best regards, Alexander Smirnov ilbers GmbH Baierbrunner Str. 28c D-81379 Munich +49 (89) 122 67 24-0 http://ilbers.de/ Commercial register Munich, HRB 214197 General manager: Baurzhan Ismagulov ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH] debootstrap: Add fallback apt cache support 2018-05-14 11:11 ` Alexander Smirnov @ 2018-05-14 11:25 ` Claudius Heine 0 siblings, 0 replies; 6+ messages in thread From: Claudius Heine @ 2018-05-14 11:25 UTC (permalink / raw) To: Alexander Smirnov, isar-users On 2018-05-14 13:11, Alexander Smirnov wrote: > > > On 05/14/2018 01:33 PM, Claudius Heine wrote: >> Hi Alex, >> >> On 2018-05-14 11:40, Alexander Smirnov wrote: >>> >>> On 05/14/2018 12:27 PM, Claudius Heine wrote: >>>> Hi Alex, >>>> >>>> so I guess that this patch replaces the 'isar-bootstrap: Increase >>>> cache room' one right? >>>> >>>> On 2018-05-11 17:35, Alexander Smirnov wrote: >>>>> Add apt config file to switch caching mechanism from MMap to >>>>> array-based. >>>>> This needed to provide possibility to build fresh Isar suites on >>>>> old hosts. >>>>> For example, building Isar stretch on jessie host fails with the >>>>> following >>>>> error: >>>>> >>>>> Hit:1 http://security.debian.org stretch/updates InRelease >>>>> Ign:2 http://ftp.de.debian.org/debian stretch InRelease >>>>> Hit:3 http://ftp.de.debian.org/debian stretch-updates InRelease >>>>> Hit:4 http://ftp.de.debian.org/debian stretch Release >>>>> E: Dynamic MMap ran out of room. Please increase the size of >>>>> APT::Cache-Start. Current value: 25165824. (man 5 apt.conf) >>>>> qemu: uncaught target signal 11 (Segmentation fault) - core dumped >>>>> Segmentation fault >>>>> >>>>> Now Isar performs the following: >>>>> 1. Checking if user attempts to build stretch on old kernels >>>>> 2. Temporarly installing this apt config file. >>>>> 3. Removing this file after image rootfs creation >>>>> >>>>> So this file will not go to target rootfs. >>>>> >>>>> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de> >>>>> --- >>>>> meta-isar/recipes-core/images/isar-image-base.bb | 2 ++ >>>>> .../isar-bootstrap/files/isar-apt-fallback.conf | 4 ++++ >>>>> meta/recipes-core/isar-bootstrap/isar-bootstrap.bb | 17 >>>>> ++++++++++++++++- >>>>> 3 files changed, 22 insertions(+), 1 deletion(-) >>>>> create mode 100644 >>>>> meta/recipes-core/isar-bootstrap/files/isar-apt-fallback.conf >>>>> >>>>> diff --git a/meta-isar/recipes-core/images/isar-image-base.bb >>>>> b/meta-isar/recipes-core/images/isar-image-base.bb >>>>> index 00e926a..d65b1f1 100644 >>>>> --- a/meta-isar/recipes-core/images/isar-image-base.bb >>>>> +++ b/meta-isar/recipes-core/images/isar-image-base.bb >>>>> @@ -53,4 +53,6 @@ do_rootfs() { >>>>> sudo rmdir ${IMAGE_ROOTFS}/isar-apt >>>>> sudo umount -l ${IMAGE_ROOTFS}/dev >>>>> sudo umount -l ${IMAGE_ROOTFS}/proc >>>>> + [ -e "${IMAGE_ROOTFS}/etc/apt/apt.conf.d/55isar-fallback.conf" >>>>> ] && \ >>>>> + sudo rm >>>>> "${IMAGE_ROOTFS}/etc/apt/apt.conf.d/55isar-fallback.conf" || true >>>> >>>> Why not just 'sudo rm -f >>>> "${IMAGE_ROOTFS}/etc/apt/apt.conf.d/55isar-fallback.conf"' ? >>>> >>> >>> Matter of taste, I don't like to remove things that don't exist. >> >> I guess it is a matter of taste. I do think '|| true' is ugly and > > If you do think so, why then you use it in your isar-debootstrap > implementation? You mean like this? sudo umount -l "${ROOTFSDIR}/dev" || true I still find that ugly, but in may cases unavoidable. Bash/Shell is just a ugly language IMO. Just because something is ugly doesn't mean it should or can be avoided all the time. > >> testing the existence of the file before removing it redundant. >> > > For me it's not too valuable issue, so I could drop this check in v3. Ok, thanks. Claudius > > Alex > >> Cheers, >> Claudius >> >>> >>>> >>>>> } >>>>> diff --git >>>>> a/meta/recipes-core/isar-bootstrap/files/isar-apt-fallback.conf >>>>> b/meta/recipes-core/isar-bootstrap/files/isar-apt-fallback.conf >>>>> new file mode 100644 >>>>> index 0000000..12ddbea >>>>> --- /dev/null >>>>> +++ b/meta/recipes-core/isar-bootstrap/files/isar-apt-fallback.conf >>>>> @@ -0,0 +1,4 @@ >>>>> +# Switch apt caching mechanism from mmap to array-based. >>>>> +# This helps to build fresh Isar suites on old hosts. >>>>> + >>>>> +APT::Cache-Fallback "1"; >>>>> diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.bb >>>>> b/meta/recipes-core/isar-bootstrap/isar-bootstrap.bb >>>>> index a38dd88..a3a3a2b 100644 >>>>> --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.bb >>>>> +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.bb >>>>> @@ -10,7 +10,9 @@ Description = "Minimal debian root file system" >>>>> LICENSE = "gpl-2.0" >>>>> LIC_FILES_CHKSUM = >>>>> "file://${LAYERDIR_isar}/licenses/COPYING.GPLv2;md5=751419260aa954499f7abaabaa882bbe" >>>>> >>>>> FILESPATH_prepend := "${THISDIR}/files:" >>>>> -SRC_URI = "file://isar-apt.conf" >>>>> +SRC_URI = " \ >>>>> + file://isar-apt.conf \ >>>>> + file://isar-apt-fallback.conf" >>>>> PV = "1.0" >>>>> WORKDIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/${PN}" >>>>> @@ -165,6 +167,13 @@ do_bootstrap() { >>>>> } >>>>> addtask bootstrap before do_build after do_apt_config_prepare >>>>> +def get_host_release(): >>>>> + import platform >>>>> + rel = platform.release() >>>>> + return rel >>>>> + >>>>> +HOST_KERN_VER = "${@get_host_release().split('.')[0]}" >>>> If you need this variable in just one place. It might be useful to >>>> just add it there. >>>> >>> >>> I've added the separate variable, because probably some additional >>> version specification could be needed later, like HOST_KERN_MAJOR >>> etc., in this case python inlining could be difficult for reading. >>> >>> But for today I have only one case, so could drop it in v3. >>> >>> Alex >>> >>>>> + >>>>> do_apt_config_install[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}" >>>>> do_apt_config_install() { >>>>> sudo mkdir -p "${ROOTFSDIR}/etc/apt/preferences.d" >>>>> @@ -177,6 +186,12 @@ do_apt_config_install() { >>>>> sudo mkdir -p "${ROOTFSDIR}/etc/apt/apt.conf.d" >>>>> sudo install -v -m644 "${WORKDIR}/isar-apt.conf" \ >>>>> "${ROOTFSDIR}/etc/apt/apt.conf.d/50isar.conf" >>>>> + >>>>> + if [ "${@get_distro_suite(d)}" = "stretch" ] && [ >>>>> "${HOST_KERN_VER}" -lt "4" ]; then >>>> >>>> Like this: >>>> >>>> if [ "${@get_distro_suite(d)}" = "stretch" ] && [ >>>> "${@get_host_release().split('.')[0]}" -lt "4" ]; then >>>> >>>> Or is it useful to overwrite this variable somewhere? >>>> >>>>> + sudo install -v -m644 "${WORKDIR}/isar-apt-fallback.conf" \ >>>>> + "${ROOTFSDIR}/etc/apt/apt.conf.d/55isar-fallback.conf" >>>>> + fi >>>>> + >>>>> } >>>>> addtask apt_config_install before do_build after do_bootstrap >>>> >>>> Apart from those small nitpicks it looks good, >>>> Claudius >>>> >> > -- DENX Software Engineering GmbH, Managing Director: Wolfgang Denk HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2018-05-14 11:25 UTC | newest] Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2018-05-11 15:35 [PATCH] debootstrap: Add fallback apt cache support Alexander Smirnov 2018-05-14 9:27 ` Claudius Heine 2018-05-14 9:40 ` Alexander Smirnov 2018-05-14 10:33 ` Claudius Heine 2018-05-14 11:11 ` Alexander Smirnov 2018-05-14 11:25 ` Claudius Heine
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox