[PATCH] scripts/ci: enable use of cached base repository for fast build

[PATCH] scripts/ci: enable use of cached base repository for fast build

[Maxim Yu. Osipov]

Signed-off-by: Maxim Yu. Osipov <mosipovd@ilbers.de>
---
 scripts/ci_build.sh | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/scripts/ci_build.sh b/scripts/ci_build.sh
index 059df21..20b77b7 100755
--- a/scripts/ci_build.sh
+++ b/scripts/ci_build.sh
@@ -86,7 +86,14 @@ fi
 if [ -n "$FAST_BUILD" ]; then
     # Start build for the reduced set of configurations
     # Enforce cross-compilation to speed up the build
+    # Enable use of cached base repository
     sed -i -e 's/ISAR_CROSS_COMPILE ?= "0"/ISAR_CROSS_COMPILE ?= "1"/g' conf/local.conf
+    bitbake $BB_ARGS -c cache_base_repo \
+        multiconfig:qemuarm-stretch:isar-image-base \
+        multiconfig:qemuarm64-stretch:isar-image-base \
+        multiconfig:qemuamd64-stretch:isar-image-base
+    sudo rm -rf tmp
+    sed -i -e 's/#ISAR_USE_CACHED_BASE_REPO ?= "1"/ISAR_USE_CACHED_BASE_REPO ?= "1"/g' conf/local.conf
     bitbake $BB_ARGS \
         multiconfig:qemuarm-stretch:isar-image-base \
         multiconfig:qemuarm64-stretch:isar-image-base \
-- 
2.11.0

Re: [PATCH] scripts/ci: enable use of cached base repository for fast build

[Jan Kiszka]

On 29.11.18 08:19, Maxim Yu. Osipov wrote:
> Signed-off-by: Maxim Yu. Osipov <mosipovd@ilbers.de>
> ---
>  scripts/ci_build.sh | 7 +++++++
>  1 file changed, 7 insertions(+)
> 
> diff --git a/scripts/ci_build.sh b/scripts/ci_build.sh
> index 059df21..20b77b7 100755
> --- a/scripts/ci_build.sh
> +++ b/scripts/ci_build.sh
> @@ -86,7 +86,14 @@ fi
>  if [ -n "$FAST_BUILD" ]; then
>      # Start build for the reduced set of configurations
>      # Enforce cross-compilation to speed up the build
> +    # Enable use of cached base repository
>      sed -i -e 's/ISAR_CROSS_COMPILE ?= "0"/ISAR_CROSS_COMPILE ?= "1"/g' conf/local.conf
> +    bitbake $BB_ARGS -c cache_base_repo \
> +        multiconfig:qemuarm-stretch:isar-image-base \
> +        multiconfig:qemuarm64-stretch:isar-image-base \
> +        multiconfig:qemuamd64-stretch:isar-image-base
> +    sudo rm -rf tmp
> +    sed -i -e 's/#ISAR_USE_CACHED_BASE_REPO ?= "1"/ISAR_USE_CACHED_BASE_REPO ?= "1"/g' conf/local.conf
>      bitbake $BB_ARGS \
>          multiconfig:qemuarm-stretch:isar-image-base \
>          multiconfig:qemuarm64-stretch:isar-image-base \
> 

That's a important step.

BTW, what is still missing to do something like this:

diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index d8fbfd5..bfefe20 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -144,7 +144,7 @@ do_cache_base_repo() {
     fi
 }
 
-addtask cache_base_repo after do_rootfs do_install_imager_deps
+addtask cache_base_repo after do_rootfs do_install_imager_deps before do_build
 
 # Imager are expected to run natively, thus will use the target buildchroot.
 ISAR_CROSS_COMPILE = "0"

That would simplify the usage (I always forget how that special target
is called), including CI.

DISCLAIMER: I didn't test this intensively.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

Re: [PATCH] scripts/ci: enable use of cached base repository for fast build

[Henning Schild]

Am Thu, 29 Nov 2018 08:19:18 +0100
schrieb "Maxim Yu. Osipov" <mosipov@ilbers.de>:

> Signed-off-by: Maxim Yu. Osipov <mosipovd@ilbers.de>
> ---
>  scripts/ci_build.sh | 7 +++++++
>  1 file changed, 7 insertions(+)
> 
> diff --git a/scripts/ci_build.sh b/scripts/ci_build.sh
> index 059df21..20b77b7 100755
> --- a/scripts/ci_build.sh
> +++ b/scripts/ci_build.sh
> @@ -86,7 +86,14 @@ fi
>  if [ -n "$FAST_BUILD" ]; then
>      # Start build for the reduced set of configurations
>      # Enforce cross-compilation to speed up the build
> +    # Enable use of cached base repository
>      sed -i -e 's/ISAR_CROSS_COMPILE ?= "0"/ISAR_CROSS_COMPILE ?=
> "1"/g' conf/local.conf
> +    bitbake $BB_ARGS -c cache_base_repo \
> +        multiconfig:qemuarm-stretch:isar-image-base \
> +        multiconfig:qemuarm64-stretch:isar-image-base \
> +        multiconfig:qemuamd64-stretch:isar-image-base
> +    sudo rm -rf tmp

That rm is not in line with the user manual, should that be updated as
well?

> +    sed -i -e 's/#ISAR_USE_CACHED_BASE_REPO ?=
> "1"/ISAR_USE_CACHED_BASE_REPO ?= "1"/g' conf/local.conf bitbake

I will not propose "ifconfig down" but this here to cover cutting off
the network for the debian side.

# set invalid proxy variables to enforce "offline"
export http_proxy=http://localhost:3128
export https_proxy=$http_proxy
export ftp_proxy=$http_proxy

Henning

> $BB_ARGS \ multiconfig:qemuarm-stretch:isar-image-base \
>          multiconfig:qemuarm64-stretch:isar-image-base \

Re: [PATCH] scripts/ci: enable use of cached base repository for fast build

[Maxim Yu. Osipov]

On 11/29/18 2:33 PM, Henning Schild wrote:
> Am Thu, 29 Nov 2018 08:19:18 +0100
> schrieb "Maxim Yu. Osipov" <mosipov@ilbers.de>:
> 
>> Signed-off-by: Maxim Yu. Osipov <mosipovd@ilbers.de>
>> ---
>>   scripts/ci_build.sh | 7 +++++++
>>   1 file changed, 7 insertions(+)
>>
>> diff --git a/scripts/ci_build.sh b/scripts/ci_build.sh
>> index 059df21..20b77b7 100755
>> --- a/scripts/ci_build.sh
>> +++ b/scripts/ci_build.sh
>> @@ -86,7 +86,14 @@ fi
>>   if [ -n "$FAST_BUILD" ]; then
>>       # Start build for the reduced set of configurations
>>       # Enforce cross-compilation to speed up the build
>> +    # Enable use of cached base repository
>>       sed -i -e 's/ISAR_CROSS_COMPILE ?= "0"/ISAR_CROSS_COMPILE ?=
>> "1"/g' conf/local.conf
>> +    bitbake $BB_ARGS -c cache_base_repo \
>> +        multiconfig:qemuarm-stretch:isar-image-base \
>> +        multiconfig:qemuarm64-stretch:isar-image-base \
>> +        multiconfig:qemuamd64-stretch:isar-image-base
>> +    sudo rm -rf tmp
> 
> That rm is not in line with the user manual, should that be updated as
> well?

Yes, after commit de68185 "conf: Move base-apt to downloads directory"
one may remove the whole tmp directory. I'll update the user manual.

>> +    sed -i -e 's/#ISAR_USE_CACHED_BASE_REPO ?=
>> "1"/ISAR_USE_CACHED_BASE_REPO ?= "1"/g' conf/local.conf bitbake
> 
> I will not propose "ifconfig down" but this here to cover cutting off
> the network for the debian side.


> # set invalid proxy variables to enforce "offline"
> export http_proxy=http://localhost:3128
> export https_proxy=$http_proxy
> export ftp_proxy=$http_proxy

Have you tested this in your environment?

If yes, I'll include this in v2.

Thanks,
Maxim.


> Henning
> 
>> $BB_ARGS \ multiconfig:qemuarm-stretch:isar-image-base \
>>           multiconfig:qemuarm64-stretch:isar-image-base \
> 


-- 
Maxim Osipov
ilbers GmbH
Maria-Merian-Str. 8
85521 Ottobrunn
Germany
+49 (151) 6517 6917
mosipov@ilbers.de
http://ilbers.de/
Commercial register Munich, HRB 214197
General Manager: Baurzhan Ismagulov

Re: [PATCH] scripts/ci: enable use of cached base repository for fast build

[Henning Schild]

Am Thu, 29 Nov 2018 15:02:06 +0300
schrieb "Maxim Yu. Osipov" <mosipov@ilbers.de>:

> On 11/29/18 2:33 PM, Henning Schild wrote:
> > Am Thu, 29 Nov 2018 08:19:18 +0100
> > schrieb "Maxim Yu. Osipov" <mosipov@ilbers.de>:
> >   
> >> Signed-off-by: Maxim Yu. Osipov <mosipovd@ilbers.de>
> >> ---
> >>   scripts/ci_build.sh | 7 +++++++
> >>   1 file changed, 7 insertions(+)
> >>
> >> diff --git a/scripts/ci_build.sh b/scripts/ci_build.sh
> >> index 059df21..20b77b7 100755
> >> --- a/scripts/ci_build.sh
> >> +++ b/scripts/ci_build.sh
> >> @@ -86,7 +86,14 @@ fi
> >>   if [ -n "$FAST_BUILD" ]; then
> >>       # Start build for the reduced set of configurations
> >>       # Enforce cross-compilation to speed up the build
> >> +    # Enable use of cached base repository
> >>       sed -i -e 's/ISAR_CROSS_COMPILE ?= "0"/ISAR_CROSS_COMPILE ?=
> >> "1"/g' conf/local.conf
> >> +    bitbake $BB_ARGS -c cache_base_repo \
> >> +        multiconfig:qemuarm-stretch:isar-image-base \
> >> +        multiconfig:qemuarm64-stretch:isar-image-base \
> >> +        multiconfig:qemuamd64-stretch:isar-image-base
> >> +    sudo rm -rf tmp  
> > 
> > That rm is not in line with the user manual, should that be updated
> > as well?  
> 
> Yes, after commit de68185 "conf: Move base-apt to downloads directory"
> one may remove the whole tmp directory. I'll update the user manual.
> 
> >> +    sed -i -e 's/#ISAR_USE_CACHED_BASE_REPO ?=
> >> "1"/ISAR_USE_CACHED_BASE_REPO ?= "1"/g' conf/local.conf bitbake  
> > 
> > I will not propose "ifconfig down" but this here to cover cutting
> > off the network for the debian side.  
> 
> 
> > # set invalid proxy variables to enforce "offline"
> > export http_proxy=http://localhost:3128
> > export https_proxy=$http_proxy
> > export ftp_proxy=$http_proxy  
> 
> Have you tested this in your environment?

No, this is just the long version of "your patch does not actually go
offline and should not be merged".

Henning

> If yes, I'll include this in v2.
> 
> Thanks,
> Maxim.
> 
> 
> > Henning
> >   
> >> $BB_ARGS \ multiconfig:qemuarm-stretch:isar-image-base \
> >>           multiconfig:qemuarm64-stretch:isar-image-base \  
> >   
> 
> 

Re: [PATCH] scripts/ci: enable use of cached base repository for fast build

[Jan Kiszka]

On 29.11.18 13:33, [ext] Henning Schild wrote:
> Am Thu, 29 Nov 2018 15:02:06 +0300
> schrieb "Maxim Yu. Osipov" <mosipov@ilbers.de>:
> 
>> On 11/29/18 2:33 PM, Henning Schild wrote:
>>> Am Thu, 29 Nov 2018 08:19:18 +0100
>>> schrieb "Maxim Yu. Osipov" <mosipov@ilbers.de>:
>>>    
>>>> Signed-off-by: Maxim Yu. Osipov <mosipovd@ilbers.de>
>>>> ---
>>>>    scripts/ci_build.sh | 7 +++++++
>>>>    1 file changed, 7 insertions(+)
>>>>
>>>> diff --git a/scripts/ci_build.sh b/scripts/ci_build.sh
>>>> index 059df21..20b77b7 100755
>>>> --- a/scripts/ci_build.sh
>>>> +++ b/scripts/ci_build.sh
>>>> @@ -86,7 +86,14 @@ fi
>>>>    if [ -n "$FAST_BUILD" ]; then
>>>>        # Start build for the reduced set of configurations
>>>>        # Enforce cross-compilation to speed up the build
>>>> +    # Enable use of cached base repository
>>>>        sed -i -e 's/ISAR_CROSS_COMPILE ?= "0"/ISAR_CROSS_COMPILE ?=
>>>> "1"/g' conf/local.conf
>>>> +    bitbake $BB_ARGS -c cache_base_repo \
>>>> +        multiconfig:qemuarm-stretch:isar-image-base \
>>>> +        multiconfig:qemuarm64-stretch:isar-image-base \
>>>> +        multiconfig:qemuamd64-stretch:isar-image-base
>>>> +    sudo rm -rf tmp
>>>
>>> That rm is not in line with the user manual, should that be updated
>>> as well?
>>
>> Yes, after commit de68185 "conf: Move base-apt to downloads directory"
>> one may remove the whole tmp directory. I'll update the user manual.
>>
>>>> +    sed -i -e 's/#ISAR_USE_CACHED_BASE_REPO ?=
>>>> "1"/ISAR_USE_CACHED_BASE_REPO ?= "1"/g' conf/local.conf bitbake
>>>
>>> I will not propose "ifconfig down" but this here to cover cutting
>>> off the network for the debian side.
>>
>>
>>> # set invalid proxy variables to enforce "offline"
>>> export http_proxy=http://localhost:3128
>>> export https_proxy=$http_proxy
>>> export ftp_proxy=$http_proxy
>>
>> Have you tested this in your environment?
> 
> No, this is just the long version of "your patch does not actually go
> offline and should not be merged".
> 

FWIW, if we decide to test offline build via invalidating proxies, we need to 
have a separate cache test for the same images in place as well. Is that the 
case already?

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

Re: [PATCH] scripts/ci: enable use of cached base repository for fast build

[Maxim Yu. Osipov]

On 11/29/18 3:33 PM, Henning Schild wrote:
> Am Thu, 29 Nov 2018 15:02:06 +0300
> schrieb "Maxim Yu. Osipov" <mosipov@ilbers.de>:
> 
>> On 11/29/18 2:33 PM, Henning Schild wrote:
>>> Am Thu, 29 Nov 2018 08:19:18 +0100
>>> schrieb "Maxim Yu. Osipov" <mosipov@ilbers.de>:
>>>    
>>>> Signed-off-by: Maxim Yu. Osipov <mosipovd@ilbers.de>
>>>> ---
>>>>    scripts/ci_build.sh | 7 +++++++
>>>>    1 file changed, 7 insertions(+)
>>>>
>>>> diff --git a/scripts/ci_build.sh b/scripts/ci_build.sh
>>>> index 059df21..20b77b7 100755
>>>> --- a/scripts/ci_build.sh
>>>> +++ b/scripts/ci_build.sh
>>>> @@ -86,7 +86,14 @@ fi
>>>>    if [ -n "$FAST_BUILD" ]; then
>>>>        # Start build for the reduced set of configurations
>>>>        # Enforce cross-compilation to speed up the build
>>>> +    # Enable use of cached base repository
>>>>        sed -i -e 's/ISAR_CROSS_COMPILE ?= "0"/ISAR_CROSS_COMPILE ?=
>>>> "1"/g' conf/local.conf
>>>> +    bitbake $BB_ARGS -c cache_base_repo \
>>>> +        multiconfig:qemuarm-stretch:isar-image-base \
>>>> +        multiconfig:qemuarm64-stretch:isar-image-base \
>>>> +        multiconfig:qemuamd64-stretch:isar-image-base
>>>> +    sudo rm -rf tmp
>>>
>>> That rm is not in line with the user manual, should that be updated
>>> as well?
>>
>> Yes, after commit de68185 "conf: Move base-apt to downloads directory"
>> one may remove the whole tmp directory. I'll update the user manual.
>>
>>>> +    sed -i -e 's/#ISAR_USE_CACHED_BASE_REPO ?=
>>>> "1"/ISAR_USE_CACHED_BASE_REPO ?= "1"/g' conf/local.conf bitbake
>>>
>>> I will not propose "ifconfig down" but this here to cover cutting
>>> off the network for the debian side.
>>
>>
>>> # set invalid proxy variables to enforce "offline"
>>> export http_proxy=http://localhost:3128
>>> export https_proxy=$http_proxy
>>> export ftp_proxy=$http_proxy
>>
>> Have you tested this in your environment?
> 
> No, this is just the long version of "your patch does not actually go
> offline and should not be merged".

So I'll apply this patch as is (at least it will detect problems we 
experienced with gpg).

Maxim.

> Henning
> 
>> If yes, I'll include this in v2.
>>
>> Thanks,
>> Maxim.
>>
>>
>>> Henning
>>>    
>>>> $BB_ARGS \ multiconfig:qemuarm-stretch:isar-image-base \
>>>>            multiconfig:qemuarm64-stretch:isar-image-base \
>>>    
>>
>>
> 


-- 
Maxim Osipov
ilbers GmbH
Maria-Merian-Str. 8
85521 Ottobrunn
Germany
+49 (151) 6517 6917
mosipov@ilbers.de
http://ilbers.de/
Commercial register Munich, HRB 214197
General Manager: Baurzhan Ismagulov

Re: [PATCH] scripts/ci: enable use of cached base repository for fast build

[Maxim Yu. Osipov]

On 11/29/18 10:19 AM, Maxim Yu. Osipov wrote:
> Signed-off-by: Maxim Yu. Osipov <mosipovd@ilbers.de>

Applied to the 'next',

Maxim.

> ---
>   scripts/ci_build.sh | 7 +++++++
>   1 file changed, 7 insertions(+)
> 
> diff --git a/scripts/ci_build.sh b/scripts/ci_build.sh
> index 059df21..20b77b7 100755
> --- a/scripts/ci_build.sh
> +++ b/scripts/ci_build.sh
> @@ -86,7 +86,14 @@ fi
>   if [ -n "$FAST_BUILD" ]; then
>       # Start build for the reduced set of configurations
>       # Enforce cross-compilation to speed up the build
> +    # Enable use of cached base repository
>       sed -i -e 's/ISAR_CROSS_COMPILE ?= "0"/ISAR_CROSS_COMPILE ?= "1"/g' conf/local.conf
> +    bitbake $BB_ARGS -c cache_base_repo \
> +        multiconfig:qemuarm-stretch:isar-image-base \
> +        multiconfig:qemuarm64-stretch:isar-image-base \
> +        multiconfig:qemuamd64-stretch:isar-image-base
> +    sudo rm -rf tmp
> +    sed -i -e 's/#ISAR_USE_CACHED_BASE_REPO ?= "1"/ISAR_USE_CACHED_BASE_REPO ?= "1"/g' conf/local.conf
>       bitbake $BB_ARGS \
>           multiconfig:qemuarm-stretch:isar-image-base \
>           multiconfig:qemuarm64-stretch:isar-image-base \
> 


-- 
Maxim Osipov
ilbers GmbH
Maria-Merian-Str. 8
85521 Ottobrunn
Germany
+49 (151) 6517 6917
mosipov@ilbers.de
http://ilbers.de/
Commercial register Munich, HRB 214197
General Manager: Baurzhan Ismagulov