* [PATCH] scripts/ci: enable use of cached base repository for fast build
@ 2018-11-29 7:19 Maxim Yu. Osipov
2018-11-29 7:58 ` Jan Kiszka
` (2 more replies)
0 siblings, 3 replies; 8+ messages in thread
From: Maxim Yu. Osipov @ 2018-11-29 7:19 UTC (permalink / raw)
To: isar-users
Signed-off-by: Maxim Yu. Osipov <mosipovd@ilbers.de>
---
scripts/ci_build.sh | 7 +++++++
1 file changed, 7 insertions(+)
diff --git a/scripts/ci_build.sh b/scripts/ci_build.sh
index 059df21..20b77b7 100755
--- a/scripts/ci_build.sh
+++ b/scripts/ci_build.sh
@@ -86,7 +86,14 @@ fi
if [ -n "$FAST_BUILD" ]; then
# Start build for the reduced set of configurations
# Enforce cross-compilation to speed up the build
+ # Enable use of cached base repository
sed -i -e 's/ISAR_CROSS_COMPILE ?= "0"/ISAR_CROSS_COMPILE ?= "1"/g' conf/local.conf
+ bitbake $BB_ARGS -c cache_base_repo \
+ multiconfig:qemuarm-stretch:isar-image-base \
+ multiconfig:qemuarm64-stretch:isar-image-base \
+ multiconfig:qemuamd64-stretch:isar-image-base
+ sudo rm -rf tmp
+ sed -i -e 's/#ISAR_USE_CACHED_BASE_REPO ?= "1"/ISAR_USE_CACHED_BASE_REPO ?= "1"/g' conf/local.conf
bitbake $BB_ARGS \
multiconfig:qemuarm-stretch:isar-image-base \
multiconfig:qemuarm64-stretch:isar-image-base \
--
2.11.0
^ permalink raw reply [flat|nested] 8+ messages in thread* Re: [PATCH] scripts/ci: enable use of cached base repository for fast build
2018-11-29 7:19 [PATCH] scripts/ci: enable use of cached base repository for fast build Maxim Yu. Osipov
@ 2018-11-29 7:58 ` Jan Kiszka
2018-11-29 11:33 ` Henning Schild
2018-11-29 13:26 ` Maxim Yu. Osipov
2 siblings, 0 replies; 8+ messages in thread
From: Jan Kiszka @ 2018-11-29 7:58 UTC (permalink / raw)
To: Maxim Yu. Osipov, isar-users
On 29.11.18 08:19, Maxim Yu. Osipov wrote:
> Signed-off-by: Maxim Yu. Osipov <mosipovd@ilbers.de>
> ---
> scripts/ci_build.sh | 7 +++++++
> 1 file changed, 7 insertions(+)
>
> diff --git a/scripts/ci_build.sh b/scripts/ci_build.sh
> index 059df21..20b77b7 100755
> --- a/scripts/ci_build.sh
> +++ b/scripts/ci_build.sh
> @@ -86,7 +86,14 @@ fi
> if [ -n "$FAST_BUILD" ]; then
> # Start build for the reduced set of configurations
> # Enforce cross-compilation to speed up the build
> + # Enable use of cached base repository
> sed -i -e 's/ISAR_CROSS_COMPILE ?= "0"/ISAR_CROSS_COMPILE ?= "1"/g' conf/local.conf
> + bitbake $BB_ARGS -c cache_base_repo \
> + multiconfig:qemuarm-stretch:isar-image-base \
> + multiconfig:qemuarm64-stretch:isar-image-base \
> + multiconfig:qemuamd64-stretch:isar-image-base
> + sudo rm -rf tmp
> + sed -i -e 's/#ISAR_USE_CACHED_BASE_REPO ?= "1"/ISAR_USE_CACHED_BASE_REPO ?= "1"/g' conf/local.conf
> bitbake $BB_ARGS \
> multiconfig:qemuarm-stretch:isar-image-base \
> multiconfig:qemuarm64-stretch:isar-image-base \
>
That's a important step.
BTW, what is still missing to do something like this:
diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index d8fbfd5..bfefe20 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -144,7 +144,7 @@ do_cache_base_repo() {
fi
}
-addtask cache_base_repo after do_rootfs do_install_imager_deps
+addtask cache_base_repo after do_rootfs do_install_imager_deps before do_build
# Imager are expected to run natively, thus will use the target buildchroot.
ISAR_CROSS_COMPILE = "0"
That would simplify the usage (I always forget how that special target
is called), including CI.
DISCLAIMER: I didn't test this intensively.
Jan
--
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux
^ permalink raw reply [flat|nested] 8+ messages in thread* Re: [PATCH] scripts/ci: enable use of cached base repository for fast build
2018-11-29 7:19 [PATCH] scripts/ci: enable use of cached base repository for fast build Maxim Yu. Osipov
2018-11-29 7:58 ` Jan Kiszka
@ 2018-11-29 11:33 ` Henning Schild
2018-11-29 12:02 ` Maxim Yu. Osipov
2018-11-29 13:26 ` Maxim Yu. Osipov
2 siblings, 1 reply; 8+ messages in thread
From: Henning Schild @ 2018-11-29 11:33 UTC (permalink / raw)
To: Maxim Yu. Osipov; +Cc: isar-users
Am Thu, 29 Nov 2018 08:19:18 +0100
schrieb "Maxim Yu. Osipov" <mosipov@ilbers.de>:
> Signed-off-by: Maxim Yu. Osipov <mosipovd@ilbers.de>
> ---
> scripts/ci_build.sh | 7 +++++++
> 1 file changed, 7 insertions(+)
>
> diff --git a/scripts/ci_build.sh b/scripts/ci_build.sh
> index 059df21..20b77b7 100755
> --- a/scripts/ci_build.sh
> +++ b/scripts/ci_build.sh
> @@ -86,7 +86,14 @@ fi
> if [ -n "$FAST_BUILD" ]; then
> # Start build for the reduced set of configurations
> # Enforce cross-compilation to speed up the build
> + # Enable use of cached base repository
> sed -i -e 's/ISAR_CROSS_COMPILE ?= "0"/ISAR_CROSS_COMPILE ?=
> "1"/g' conf/local.conf
> + bitbake $BB_ARGS -c cache_base_repo \
> + multiconfig:qemuarm-stretch:isar-image-base \
> + multiconfig:qemuarm64-stretch:isar-image-base \
> + multiconfig:qemuamd64-stretch:isar-image-base
> + sudo rm -rf tmp
That rm is not in line with the user manual, should that be updated as
well?
> + sed -i -e 's/#ISAR_USE_CACHED_BASE_REPO ?=
> "1"/ISAR_USE_CACHED_BASE_REPO ?= "1"/g' conf/local.conf bitbake
I will not propose "ifconfig down" but this here to cover cutting off
the network for the debian side.
# set invalid proxy variables to enforce "offline"
export http_proxy=http://localhost:3128
export https_proxy=$http_proxy
export ftp_proxy=$http_proxy
Henning
> $BB_ARGS \ multiconfig:qemuarm-stretch:isar-image-base \
> multiconfig:qemuarm64-stretch:isar-image-base \
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH] scripts/ci: enable use of cached base repository for fast build
2018-11-29 11:33 ` Henning Schild
@ 2018-11-29 12:02 ` Maxim Yu. Osipov
2018-11-29 12:33 ` Henning Schild
0 siblings, 1 reply; 8+ messages in thread
From: Maxim Yu. Osipov @ 2018-11-29 12:02 UTC (permalink / raw)
To: Henning Schild; +Cc: isar-users
On 11/29/18 2:33 PM, Henning Schild wrote:
> Am Thu, 29 Nov 2018 08:19:18 +0100
> schrieb "Maxim Yu. Osipov" <mosipov@ilbers.de>:
>
>> Signed-off-by: Maxim Yu. Osipov <mosipovd@ilbers.de>
>> ---
>> scripts/ci_build.sh | 7 +++++++
>> 1 file changed, 7 insertions(+)
>>
>> diff --git a/scripts/ci_build.sh b/scripts/ci_build.sh
>> index 059df21..20b77b7 100755
>> --- a/scripts/ci_build.sh
>> +++ b/scripts/ci_build.sh
>> @@ -86,7 +86,14 @@ fi
>> if [ -n "$FAST_BUILD" ]; then
>> # Start build for the reduced set of configurations
>> # Enforce cross-compilation to speed up the build
>> + # Enable use of cached base repository
>> sed -i -e 's/ISAR_CROSS_COMPILE ?= "0"/ISAR_CROSS_COMPILE ?=
>> "1"/g' conf/local.conf
>> + bitbake $BB_ARGS -c cache_base_repo \
>> + multiconfig:qemuarm-stretch:isar-image-base \
>> + multiconfig:qemuarm64-stretch:isar-image-base \
>> + multiconfig:qemuamd64-stretch:isar-image-base
>> + sudo rm -rf tmp
>
> That rm is not in line with the user manual, should that be updated as
> well?
Yes, after commit de68185 "conf: Move base-apt to downloads directory"
one may remove the whole tmp directory. I'll update the user manual.
>> + sed -i -e 's/#ISAR_USE_CACHED_BASE_REPO ?=
>> "1"/ISAR_USE_CACHED_BASE_REPO ?= "1"/g' conf/local.conf bitbake
>
> I will not propose "ifconfig down" but this here to cover cutting off
> the network for the debian side.
> # set invalid proxy variables to enforce "offline"
> export http_proxy=http://localhost:3128
> export https_proxy=$http_proxy
> export ftp_proxy=$http_proxy
Have you tested this in your environment?
If yes, I'll include this in v2.
Thanks,
Maxim.
> Henning
>
>> $BB_ARGS \ multiconfig:qemuarm-stretch:isar-image-base \
>> multiconfig:qemuarm64-stretch:isar-image-base \
>
--
Maxim Osipov
ilbers GmbH
Maria-Merian-Str. 8
85521 Ottobrunn
Germany
+49 (151) 6517 6917
mosipov@ilbers.de
http://ilbers.de/
Commercial register Munich, HRB 214197
General Manager: Baurzhan Ismagulov
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH] scripts/ci: enable use of cached base repository for fast build
2018-11-29 12:02 ` Maxim Yu. Osipov
@ 2018-11-29 12:33 ` Henning Schild
2018-11-29 12:40 ` Jan Kiszka
2018-11-29 13:01 ` Maxim Yu. Osipov
0 siblings, 2 replies; 8+ messages in thread
From: Henning Schild @ 2018-11-29 12:33 UTC (permalink / raw)
To: Maxim Yu. Osipov; +Cc: isar-users
Am Thu, 29 Nov 2018 15:02:06 +0300
schrieb "Maxim Yu. Osipov" <mosipov@ilbers.de>:
> On 11/29/18 2:33 PM, Henning Schild wrote:
> > Am Thu, 29 Nov 2018 08:19:18 +0100
> > schrieb "Maxim Yu. Osipov" <mosipov@ilbers.de>:
> >
> >> Signed-off-by: Maxim Yu. Osipov <mosipovd@ilbers.de>
> >> ---
> >> scripts/ci_build.sh | 7 +++++++
> >> 1 file changed, 7 insertions(+)
> >>
> >> diff --git a/scripts/ci_build.sh b/scripts/ci_build.sh
> >> index 059df21..20b77b7 100755
> >> --- a/scripts/ci_build.sh
> >> +++ b/scripts/ci_build.sh
> >> @@ -86,7 +86,14 @@ fi
> >> if [ -n "$FAST_BUILD" ]; then
> >> # Start build for the reduced set of configurations
> >> # Enforce cross-compilation to speed up the build
> >> + # Enable use of cached base repository
> >> sed -i -e 's/ISAR_CROSS_COMPILE ?= "0"/ISAR_CROSS_COMPILE ?=
> >> "1"/g' conf/local.conf
> >> + bitbake $BB_ARGS -c cache_base_repo \
> >> + multiconfig:qemuarm-stretch:isar-image-base \
> >> + multiconfig:qemuarm64-stretch:isar-image-base \
> >> + multiconfig:qemuamd64-stretch:isar-image-base
> >> + sudo rm -rf tmp
> >
> > That rm is not in line with the user manual, should that be updated
> > as well?
>
> Yes, after commit de68185 "conf: Move base-apt to downloads directory"
> one may remove the whole tmp directory. I'll update the user manual.
>
> >> + sed -i -e 's/#ISAR_USE_CACHED_BASE_REPO ?=
> >> "1"/ISAR_USE_CACHED_BASE_REPO ?= "1"/g' conf/local.conf bitbake
> >
> > I will not propose "ifconfig down" but this here to cover cutting
> > off the network for the debian side.
>
>
> > # set invalid proxy variables to enforce "offline"
> > export http_proxy=http://localhost:3128
> > export https_proxy=$http_proxy
> > export ftp_proxy=$http_proxy
>
> Have you tested this in your environment?
No, this is just the long version of "your patch does not actually go
offline and should not be merged".
Henning
> If yes, I'll include this in v2.
>
> Thanks,
> Maxim.
>
>
> > Henning
> >
> >> $BB_ARGS \ multiconfig:qemuarm-stretch:isar-image-base \
> >> multiconfig:qemuarm64-stretch:isar-image-base \
> >
>
>
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH] scripts/ci: enable use of cached base repository for fast build
2018-11-29 12:33 ` Henning Schild
@ 2018-11-29 12:40 ` Jan Kiszka
2018-11-29 13:01 ` Maxim Yu. Osipov
1 sibling, 0 replies; 8+ messages in thread
From: Jan Kiszka @ 2018-11-29 12:40 UTC (permalink / raw)
To: [ext] Henning Schild, Maxim Yu. Osipov; +Cc: isar-users
On 29.11.18 13:33, [ext] Henning Schild wrote:
> Am Thu, 29 Nov 2018 15:02:06 +0300
> schrieb "Maxim Yu. Osipov" <mosipov@ilbers.de>:
>
>> On 11/29/18 2:33 PM, Henning Schild wrote:
>>> Am Thu, 29 Nov 2018 08:19:18 +0100
>>> schrieb "Maxim Yu. Osipov" <mosipov@ilbers.de>:
>>>
>>>> Signed-off-by: Maxim Yu. Osipov <mosipovd@ilbers.de>
>>>> ---
>>>> scripts/ci_build.sh | 7 +++++++
>>>> 1 file changed, 7 insertions(+)
>>>>
>>>> diff --git a/scripts/ci_build.sh b/scripts/ci_build.sh
>>>> index 059df21..20b77b7 100755
>>>> --- a/scripts/ci_build.sh
>>>> +++ b/scripts/ci_build.sh
>>>> @@ -86,7 +86,14 @@ fi
>>>> if [ -n "$FAST_BUILD" ]; then
>>>> # Start build for the reduced set of configurations
>>>> # Enforce cross-compilation to speed up the build
>>>> + # Enable use of cached base repository
>>>> sed -i -e 's/ISAR_CROSS_COMPILE ?= "0"/ISAR_CROSS_COMPILE ?=
>>>> "1"/g' conf/local.conf
>>>> + bitbake $BB_ARGS -c cache_base_repo \
>>>> + multiconfig:qemuarm-stretch:isar-image-base \
>>>> + multiconfig:qemuarm64-stretch:isar-image-base \
>>>> + multiconfig:qemuamd64-stretch:isar-image-base
>>>> + sudo rm -rf tmp
>>>
>>> That rm is not in line with the user manual, should that be updated
>>> as well?
>>
>> Yes, after commit de68185 "conf: Move base-apt to downloads directory"
>> one may remove the whole tmp directory. I'll update the user manual.
>>
>>>> + sed -i -e 's/#ISAR_USE_CACHED_BASE_REPO ?=
>>>> "1"/ISAR_USE_CACHED_BASE_REPO ?= "1"/g' conf/local.conf bitbake
>>>
>>> I will not propose "ifconfig down" but this here to cover cutting
>>> off the network for the debian side.
>>
>>
>>> # set invalid proxy variables to enforce "offline"
>>> export http_proxy=http://localhost:3128
>>> export https_proxy=$http_proxy
>>> export ftp_proxy=$http_proxy
>>
>> Have you tested this in your environment?
>
> No, this is just the long version of "your patch does not actually go
> offline and should not be merged".
>
FWIW, if we decide to test offline build via invalidating proxies, we need to
have a separate cache test for the same images in place as well. Is that the
case already?
Jan
--
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH] scripts/ci: enable use of cached base repository for fast build
2018-11-29 12:33 ` Henning Schild
2018-11-29 12:40 ` Jan Kiszka
@ 2018-11-29 13:01 ` Maxim Yu. Osipov
1 sibling, 0 replies; 8+ messages in thread
From: Maxim Yu. Osipov @ 2018-11-29 13:01 UTC (permalink / raw)
To: Henning Schild; +Cc: isar-users
On 11/29/18 3:33 PM, Henning Schild wrote:
> Am Thu, 29 Nov 2018 15:02:06 +0300
> schrieb "Maxim Yu. Osipov" <mosipov@ilbers.de>:
>
>> On 11/29/18 2:33 PM, Henning Schild wrote:
>>> Am Thu, 29 Nov 2018 08:19:18 +0100
>>> schrieb "Maxim Yu. Osipov" <mosipov@ilbers.de>:
>>>
>>>> Signed-off-by: Maxim Yu. Osipov <mosipovd@ilbers.de>
>>>> ---
>>>> scripts/ci_build.sh | 7 +++++++
>>>> 1 file changed, 7 insertions(+)
>>>>
>>>> diff --git a/scripts/ci_build.sh b/scripts/ci_build.sh
>>>> index 059df21..20b77b7 100755
>>>> --- a/scripts/ci_build.sh
>>>> +++ b/scripts/ci_build.sh
>>>> @@ -86,7 +86,14 @@ fi
>>>> if [ -n "$FAST_BUILD" ]; then
>>>> # Start build for the reduced set of configurations
>>>> # Enforce cross-compilation to speed up the build
>>>> + # Enable use of cached base repository
>>>> sed -i -e 's/ISAR_CROSS_COMPILE ?= "0"/ISAR_CROSS_COMPILE ?=
>>>> "1"/g' conf/local.conf
>>>> + bitbake $BB_ARGS -c cache_base_repo \
>>>> + multiconfig:qemuarm-stretch:isar-image-base \
>>>> + multiconfig:qemuarm64-stretch:isar-image-base \
>>>> + multiconfig:qemuamd64-stretch:isar-image-base
>>>> + sudo rm -rf tmp
>>>
>>> That rm is not in line with the user manual, should that be updated
>>> as well?
>>
>> Yes, after commit de68185 "conf: Move base-apt to downloads directory"
>> one may remove the whole tmp directory. I'll update the user manual.
>>
>>>> + sed -i -e 's/#ISAR_USE_CACHED_BASE_REPO ?=
>>>> "1"/ISAR_USE_CACHED_BASE_REPO ?= "1"/g' conf/local.conf bitbake
>>>
>>> I will not propose "ifconfig down" but this here to cover cutting
>>> off the network for the debian side.
>>
>>
>>> # set invalid proxy variables to enforce "offline"
>>> export http_proxy=http://localhost:3128
>>> export https_proxy=$http_proxy
>>> export ftp_proxy=$http_proxy
>>
>> Have you tested this in your environment?
>
> No, this is just the long version of "your patch does not actually go
> offline and should not be merged".
So I'll apply this patch as is (at least it will detect problems we
experienced with gpg).
Maxim.
> Henning
>
>> If yes, I'll include this in v2.
>>
>> Thanks,
>> Maxim.
>>
>>
>>> Henning
>>>
>>>> $BB_ARGS \ multiconfig:qemuarm-stretch:isar-image-base \
>>>> multiconfig:qemuarm64-stretch:isar-image-base \
>>>
>>
>>
>
--
Maxim Osipov
ilbers GmbH
Maria-Merian-Str. 8
85521 Ottobrunn
Germany
+49 (151) 6517 6917
mosipov@ilbers.de
http://ilbers.de/
Commercial register Munich, HRB 214197
General Manager: Baurzhan Ismagulov
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH] scripts/ci: enable use of cached base repository for fast build
2018-11-29 7:19 [PATCH] scripts/ci: enable use of cached base repository for fast build Maxim Yu. Osipov
2018-11-29 7:58 ` Jan Kiszka
2018-11-29 11:33 ` Henning Schild
@ 2018-11-29 13:26 ` Maxim Yu. Osipov
2 siblings, 0 replies; 8+ messages in thread
From: Maxim Yu. Osipov @ 2018-11-29 13:26 UTC (permalink / raw)
To: isar-users
On 11/29/18 10:19 AM, Maxim Yu. Osipov wrote:
> Signed-off-by: Maxim Yu. Osipov <mosipovd@ilbers.de>
Applied to the 'next',
Maxim.
> ---
> scripts/ci_build.sh | 7 +++++++
> 1 file changed, 7 insertions(+)
>
> diff --git a/scripts/ci_build.sh b/scripts/ci_build.sh
> index 059df21..20b77b7 100755
> --- a/scripts/ci_build.sh
> +++ b/scripts/ci_build.sh
> @@ -86,7 +86,14 @@ fi
> if [ -n "$FAST_BUILD" ]; then
> # Start build for the reduced set of configurations
> # Enforce cross-compilation to speed up the build
> + # Enable use of cached base repository
> sed -i -e 's/ISAR_CROSS_COMPILE ?= "0"/ISAR_CROSS_COMPILE ?= "1"/g' conf/local.conf
> + bitbake $BB_ARGS -c cache_base_repo \
> + multiconfig:qemuarm-stretch:isar-image-base \
> + multiconfig:qemuarm64-stretch:isar-image-base \
> + multiconfig:qemuamd64-stretch:isar-image-base
> + sudo rm -rf tmp
> + sed -i -e 's/#ISAR_USE_CACHED_BASE_REPO ?= "1"/ISAR_USE_CACHED_BASE_REPO ?= "1"/g' conf/local.conf
> bitbake $BB_ARGS \
> multiconfig:qemuarm-stretch:isar-image-base \
> multiconfig:qemuarm64-stretch:isar-image-base \
>
--
Maxim Osipov
ilbers GmbH
Maria-Merian-Str. 8
85521 Ottobrunn
Germany
+49 (151) 6517 6917
mosipov@ilbers.de
http://ilbers.de/
Commercial register Munich, HRB 214197
General Manager: Baurzhan Ismagulov
^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2018-11-29 13:26 UTC | newest]
Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-11-29 7:19 [PATCH] scripts/ci: enable use of cached base repository for fast build Maxim Yu. Osipov
2018-11-29 7:58 ` Jan Kiszka
2018-11-29 11:33 ` Henning Schild
2018-11-29 12:02 ` Maxim Yu. Osipov
2018-11-29 12:33 ` Henning Schild
2018-11-29 12:40 ` Jan Kiszka
2018-11-29 13:01 ` Maxim Yu. Osipov
2018-11-29 13:26 ` Maxim Yu. Osipov
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox