[RFC] Interface for installing bootloaders

[RFC] Interface for installing bootloaders

[Jan Kiszka]

Hi all,

for installing a bootloader into an image, a number of packages have to 
be installed into the buildchroot. These can be prebuilt packages or 
packages we build via Isar. Currently, we only handle the case of WIC 
requiring prebuilt Debian packages, and that in a hacky way (no proper 
distro abstraction, unneeded installation of unused dependencies). This 
proposal aims at providing a more holistic solution:

- Introduce new variables that express the two types of dependencies:
    - BOOTLOADER_PREBUILT
    - BOOTLOADER_CUSTOM

   So there is no "WIC" in these, any imaging recipe can use them.

- Imaging recipes (e.g. via classes/wic-img.bbclass) should
    - install all packages from BOOTLOADER_PREBUILT and BOOTLOADER_CUSTOM
      into the buildchroot
    - DEPEND on all recipes in BOOTLOADER_CUSTOM

- Define some common dependencies in conf/distro/${DISTRO}.conf:
    - GRUB_PREBUILT
    - GRUB_PREBUILT_append_${DISTRO_ARCH}
    - SYSLINUX_PREBUILT
    - ...

   Those variables can then be used to initialize BOOTLOADER_PREBUILT as
   needed and also replace those nasty BUILDCHROOT_PREINSTALL_WIC in
   buildchroot-target.bb.

- Set BOOTLOADER_PREBUILT or BOOTLOADER_CUSTOM in the
   conf/machine/${MACHINE}.conf or some overriding conf file - basically
   the same one that selects IMAGE_TYPE and WKS_FILE.

Anything I missed? Better suggestions? Eventually, when we can derive 
bitbake recipe dependencies automatically from package dependencies, 
PREBUILT and CUSTOM could be folded together. Right now, we need that split.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

Re: [RFC] Interface for installing bootloaders

[Claudius Heine]

[-- Attachment #1: Type: text/plain, Size: 2217 bytes --]

Hi Jan,

On 2018-09-04 09:17, [ext] Jan Kiszka wrote:
> Hi all,
> 
> for installing a bootloader into an image, a number of packages have to 
> be installed into the buildchroot. These can be prebuilt packages or 
> packages we build via Isar. Currently, we only handle the case of WIC 
> requiring prebuilt Debian packages, and that in a hacky way (no proper 
> distro abstraction, unneeded installation of unused dependencies). This 
> proposal aims at providing a more holistic solution:
> 
> - Introduce new variables that express the two types of dependencies:
>     - BOOTLOADER_PREBUILT
>     - BOOTLOADER_CUSTOM
> 
>    So there is no "WIC" in these, any imaging recipe can use them.
> 
> - Imaging recipes (e.g. via classes/wic-img.bbclass) should
>     - install all packages from BOOTLOADER_PREBUILT and BOOTLOADER_CUSTOM
>       into the buildchroot
>     - DEPEND on all recipes in BOOTLOADER_CUSTOM
> 
> - Define some common dependencies in conf/distro/${DISTRO}.conf:
>     - GRUB_PREBUILT
>     - GRUB_PREBUILT_append_${DISTRO_ARCH}
>     - SYSLINUX_PREBUILT
>     - ...
> 
>    Those variables can then be used to initialize BOOTLOADER_PREBUILT as
>    needed and also replace those nasty BUILDCHROOT_PREINSTALL_WIC in
>    buildchroot-target.bb.
> 
> - Set BOOTLOADER_PREBUILT or BOOTLOADER_CUSTOM in the
>    conf/machine/${MACHINE}.conf or some overriding conf file - basically
>    the same one that selects IMAGE_TYPE and WKS_FILE.
> 
> Anything I missed? Better suggestions? Eventually, when we can derive 
> bitbake recipe dependencies automatically from package dependencies, 
> PREBUILT and CUSTOM could be folded together. Right now, we need that 
> split.
> 
> Jan

I have a class that I use for the current project that allows installing 
additional packages into the buildchroot from within the image. I didn't 
had time to prepare patches for it for mainline isar. But maybe that 
goes into to right direction.

I attached it.

Cheers,
Claudius

-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de

[-- Attachment #2: buildchroot-depends.bbclass --]
[-- Type: text/plain, Size: 1459 bytes --]

BUILDCHROOT_EXTRA_INSTALL ??= ""

MOUNT_LOCKFILE = "${BUILDCHROOT_DIR}/mount.lock"

BUILDCHROOT_DIR ??= "${BUILDCHROOT_TARGET_DIR}"
BUILDCHROOT_BUILD_DEP ??= "buildchroot-target:do_build"

do_buildchroot_extra_install[depends] = "${BUILDCHROOT_BUILD_DEP}"
do_buildchroot_extra_install[deptask] = "do_deploy_deb"
do_buildchroot_extra_install[stamp-extra-info] = "${DISTRO}-${MACHINE}"
do_buildchroot_extra_install() {
    PACKAGES="${@" ".join(d.getVar("BUILDCHROOT_EXTRA_INSTALL", True).split())}"

    if [ -z "$PACKAGES" ]; then
        bbnote "No packages installed"
        exit
    fi

    sudo flock ${MOUNT_LOCKFILE} -c ' \
        if ! grep -q ${BUILDCHROOT_DIR}/isar-apt /proc/mounts; then
            mount --bind ${DEPLOY_DIR_APT}/${DISTRO} ${BUILDCHROOT_DIR}/isar-apt
            mount --bind ${DL_DIR} ${BUILDCHROOT_DIR}/downloads
            mount -t devtmpfs -o mode=0755,nosuid devtmpfs ${BUILDCHROOT_DIR}/dev
            mount -t proc none ${BUILDCHROOT_DIR}/proc
        fi'

    sudo chroot ${BUILDCHROOT_DIR} \
        apt-get update \
            -o Dir::Etc::sourcelist="sources.list.d/multistrap-isar-apt.list" \
            -o Dir::Etc::sourceparts="-" \
            -o APT::Get::List-Cleanup="0"

    sudo chroot ${BUILDCHROOT_DIR} \
        apt-get -o Debug::pkgProblemResolver=yes --no-install-recommends -y \
            --allow-unauthenticated install \
            $PACKAGES
}

addtask buildchroot_extra_install before do_rootfs

Re: [RFC] Interface for installing bootloaders

[Jan Kiszka]

On 2018-09-04 11:27, Claudius Heine wrote:
> Hi Jan,
> 
> On 2018-09-04 09:17, [ext] Jan Kiszka wrote:
>> Hi all,
>>
>> for installing a bootloader into an image, a number of packages have 
>> to be installed into the buildchroot. These can be prebuilt packages 
>> or packages we build via Isar. Currently, we only handle the case of 
>> WIC requiring prebuilt Debian packages, and that in a hacky way (no 
>> proper distro abstraction, unneeded installation of unused 
>> dependencies). This proposal aims at providing a more holistic solution:
>>
>> - Introduce new variables that express the two types of dependencies:
>>     - BOOTLOADER_PREBUILT
>>     - BOOTLOADER_CUSTOM
>>
>>    So there is no "WIC" in these, any imaging recipe can use them.
>>
>> - Imaging recipes (e.g. via classes/wic-img.bbclass) should
>>     - install all packages from BOOTLOADER_PREBUILT and BOOTLOADER_CUSTOM
>>       into the buildchroot
>>     - DEPEND on all recipes in BOOTLOADER_CUSTOM
>>
>> - Define some common dependencies in conf/distro/${DISTRO}.conf:
>>     - GRUB_PREBUILT
>>     - GRUB_PREBUILT_append_${DISTRO_ARCH}
>>     - SYSLINUX_PREBUILT
>>     - ...
>>
>>    Those variables can then be used to initialize BOOTLOADER_PREBUILT as
>>    needed and also replace those nasty BUILDCHROOT_PREINSTALL_WIC in
>>    buildchroot-target.bb.
>>
>> - Set BOOTLOADER_PREBUILT or BOOTLOADER_CUSTOM in the
>>    conf/machine/${MACHINE}.conf or some overriding conf file - basically
>>    the same one that selects IMAGE_TYPE and WKS_FILE.
>>
>> Anything I missed? Better suggestions? Eventually, when we can derive 
>> bitbake recipe dependencies automatically from package dependencies, 
>> PREBUILT and CUSTOM could be folded together. Right now, we need that 
>> split.
>>
>> Jan
> 
> I have a class that I use for the current project that allows installing 
> additional packages into the buildchroot from within the image. I didn't 
> had time to prepare patches for it for mainline isar. But maybe that 
> goes into to right direction.
> 
> I attached it.

Yeah, can serve as pattern. I'd just like to abstract away from the user 
that fact the bootloader deps go into buildchroot, i.e. the user 
interface shall not be BUILDCHROOT_EXTRA_INSTALL / BUILDCHROOT_BUILD_DEP 
in the context of bootloader installation.

Jan

Re: [RFC] Interface for installing bootloaders

[Jan Kiszka]

On 2018-09-04 11:27, Claudius Heine wrote:
> Hi Jan,
> 
> On 2018-09-04 09:17, [ext] Jan Kiszka wrote:
>> Hi all,
>>
>> for installing a bootloader into an image, a number of packages have 
>> to be installed into the buildchroot. These can be prebuilt packages 
>> or packages we build via Isar. Currently, we only handle the case of 
>> WIC requiring prebuilt Debian packages, and that in a hacky way (no 
>> proper distro abstraction, unneeded installation of unused 
>> dependencies). This proposal aims at providing a more holistic solution:
>>
>> - Introduce new variables that express the two types of dependencies:
>>     - BOOTLOADER_PREBUILT
>>     - BOOTLOADER_CUSTOM
>>
>>    So there is no "WIC" in these, any imaging recipe can use them.
>>
>> - Imaging recipes (e.g. via classes/wic-img.bbclass) should
>>     - install all packages from BOOTLOADER_PREBUILT and BOOTLOADER_CUSTOM
>>       into the buildchroot
>>     - DEPEND on all recipes in BOOTLOADER_CUSTOM
>>
>> - Define some common dependencies in conf/distro/${DISTRO}.conf:
>>     - GRUB_PREBUILT
>>     - GRUB_PREBUILT_append_${DISTRO_ARCH}
>>     - SYSLINUX_PREBUILT
>>     - ...
>>
>>    Those variables can then be used to initialize BOOTLOADER_PREBUILT as
>>    needed and also replace those nasty BUILDCHROOT_PREINSTALL_WIC in
>>    buildchroot-target.bb.
>>
>> - Set BOOTLOADER_PREBUILT or BOOTLOADER_CUSTOM in the
>>    conf/machine/${MACHINE}.conf or some overriding conf file - basically
>>    the same one that selects IMAGE_TYPE and WKS_FILE.
>>
>> Anything I missed? Better suggestions? Eventually, when we can derive 
>> bitbake recipe dependencies automatically from package dependencies, 
>> PREBUILT and CUSTOM could be folded together. Right now, we need that 
>> split.
>>
>> Jan
> 
> I have a class that I use for the current project that allows installing 
> additional packages into the buildchroot from within the image. I didn't 
> had time to prepare patches for it for mainline isar. But maybe that 
> goes into to right direction.
> 
> I attached it.

...

> BUILDCHROOT_EXTRA_INSTALL ??= ""
> 
> MOUNT_LOCKFILE = "${BUILDCHROOT_DIR}/mount.lock"
> 
> BUILDCHROOT_DIR ??= "${BUILDCHROOT_TARGET_DIR}"
> BUILDCHROOT_BUILD_DEP ??= "buildchroot-target:do_build"
> 
> do_buildchroot_extra_install[depends] = "${BUILDCHROOT_BUILD_DEP}"
> do_buildchroot_extra_install[deptask] = "do_deploy_deb"
> do_buildchroot_extra_install[stamp-extra-info] = "${DISTRO}-${MACHINE}"
> do_buildchroot_extra_install() {
>     PACKAGES="${@" ".join(d.getVar("BUILDCHROOT_EXTRA_INSTALL", True).split())}"
> 

Can you explain the magic dance in the line above? I don't see its need yet.

>     if [ -z "$PACKAGES" ]; then

...provided you do "[ -z $VAR ]" here to catch VAR=" ".

>         bbnote "No packages installed"
>         exit
>     fi
> 
>     sudo flock ${MOUNT_LOCKFILE} -c ' \
>         if ! grep -q ${BUILDCHROOT_DIR}/isar-apt /proc/mounts; then
>             mount --bind ${DEPLOY_DIR_APT}/${DISTRO} ${BUILDCHROOT_DIR}/isar-apt
>             mount --bind ${DL_DIR} ${BUILDCHROOT_DIR}/downloads
>             mount -t devtmpfs -o mode=0755,nosuid devtmpfs ${BUILDCHROOT_DIR}/dev
>             mount -t proc none ${BUILDCHROOT_DIR}/proc
>         fi'
> 
>     sudo chroot ${BUILDCHROOT_DIR} \
>         apt-get update \
>             -o Dir::Etc::sourcelist="sources.list.d/multistrap-isar-apt.list" \
>             -o Dir::Etc::sourceparts="-" \
>             -o APT::Get::List-Cleanup="0"
> 
>     sudo chroot ${BUILDCHROOT_DIR} \
>         apt-get -o Debug::pkgProblemResolver=yes --no-install-recommends -y \
>             --allow-unauthenticated install \
>             $PACKAGES
> }
> 
> addtask buildchroot_extra_install before do_rootfs

Thanks,
Jan

Re: [RFC] Interface for installing bootloaders

[Claudius Heine]

Hi Jan,

On 2018-09-05 08:08, Jan Kiszka wrote:
> On 2018-09-04 11:27, Claudius Heine wrote:
>> Hi Jan,
>>
>> On 2018-09-04 09:17, [ext] Jan Kiszka wrote:
>>> Hi all,
>>>
>>> for installing a bootloader into an image, a number of packages have 
>>> to be installed into the buildchroot. These can be prebuilt packages 
>>> or packages we build via Isar. Currently, we only handle the case of 
>>> WIC requiring prebuilt Debian packages, and that in a hacky way (no 
>>> proper distro abstraction, unneeded installation of unused 
>>> dependencies). This proposal aims at providing a more holistic solution:
>>>
>>> - Introduce new variables that express the two types of dependencies:
>>>     - BOOTLOADER_PREBUILT
>>>     - BOOTLOADER_CUSTOM
>>>
>>>    So there is no "WIC" in these, any imaging recipe can use them.
>>>
>>> - Imaging recipes (e.g. via classes/wic-img.bbclass) should
>>>     - install all packages from BOOTLOADER_PREBUILT and 
>>> BOOTLOADER_CUSTOM
>>>       into the buildchroot
>>>     - DEPEND on all recipes in BOOTLOADER_CUSTOM
>>>
>>> - Define some common dependencies in conf/distro/${DISTRO}.conf:
>>>     - GRUB_PREBUILT
>>>     - GRUB_PREBUILT_append_${DISTRO_ARCH}
>>>     - SYSLINUX_PREBUILT
>>>     - ...
>>>
>>>    Those variables can then be used to initialize BOOTLOADER_PREBUILT as
>>>    needed and also replace those nasty BUILDCHROOT_PREINSTALL_WIC in
>>>    buildchroot-target.bb.
>>>
>>> - Set BOOTLOADER_PREBUILT or BOOTLOADER_CUSTOM in the
>>>    conf/machine/${MACHINE}.conf or some overriding conf file - basically
>>>    the same one that selects IMAGE_TYPE and WKS_FILE.
>>>
>>> Anything I missed? Better suggestions? Eventually, when we can derive 
>>> bitbake recipe dependencies automatically from package dependencies, 
>>> PREBUILT and CUSTOM could be folded together. Right now, we need that 
>>> split.
>>>
>>> Jan
>>
>> I have a class that I use for the current project that allows 
>> installing additional packages into the buildchroot from within the 
>> image. I didn't had time to prepare patches for it for mainline isar. 
>> But maybe that goes into to right direction.
>>
>> I attached it.
> 
> ...
> 
>> BUILDCHROOT_EXTRA_INSTALL ??= ""
>>
>> MOUNT_LOCKFILE = "${BUILDCHROOT_DIR}/mount.lock"
>>
>> BUILDCHROOT_DIR ??= "${BUILDCHROOT_TARGET_DIR}"
>> BUILDCHROOT_BUILD_DEP ??= "buildchroot-target:do_build"
>>
>> do_buildchroot_extra_install[depends] = "${BUILDCHROOT_BUILD_DEP}"
>> do_buildchroot_extra_install[deptask] = "do_deploy_deb"
>> do_buildchroot_extra_install[stamp-extra-info] = "${DISTRO}-${MACHINE}"
>> do_buildchroot_extra_install() {
>>     PACKAGES="${@" ".join(d.getVar("BUILDCHROOT_EXTRA_INSTALL", 
>> True).split())}"
>>
> 
> Can you explain the magic dance in the line above? I don't see its need 
> yet.
> 
>>     if [ -z "$PACKAGES" ]; then
> 
> ...provided you do "[ -z $VAR ]" here to catch VAR=" ".

In sh i get:

   $ tets="asdf asdf  asdf asdf"
   $ [ -z $tets ] && echo yes
   sh: 2: [: asdf: unexpected operator

Cheers,
Claudius

-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de

Re: [RFC] Interface for installing bootloaders

[Jan Kiszka]

On 2018-09-05 09:34, Claudius Heine wrote:
> Hi Jan,
> 
> On 2018-09-05 08:08, Jan Kiszka wrote:
>> On 2018-09-04 11:27, Claudius Heine wrote:
>>> Hi Jan,
>>>
>>> On 2018-09-04 09:17, [ext] Jan Kiszka wrote:
>>>> Hi all,
>>>>
>>>> for installing a bootloader into an image, a number of packages have 
>>>> to be installed into the buildchroot. These can be prebuilt packages 
>>>> or packages we build via Isar. Currently, we only handle the case of 
>>>> WIC requiring prebuilt Debian packages, and that in a hacky way (no 
>>>> proper distro abstraction, unneeded installation of unused 
>>>> dependencies). This proposal aims at providing a more holistic 
>>>> solution:
>>>>
>>>> - Introduce new variables that express the two types of dependencies:
>>>>     - BOOTLOADER_PREBUILT
>>>>     - BOOTLOADER_CUSTOM
>>>>
>>>>    So there is no "WIC" in these, any imaging recipe can use them.
>>>>
>>>> - Imaging recipes (e.g. via classes/wic-img.bbclass) should
>>>>     - install all packages from BOOTLOADER_PREBUILT and 
>>>> BOOTLOADER_CUSTOM
>>>>       into the buildchroot
>>>>     - DEPEND on all recipes in BOOTLOADER_CUSTOM
>>>>
>>>> - Define some common dependencies in conf/distro/${DISTRO}.conf:
>>>>     - GRUB_PREBUILT
>>>>     - GRUB_PREBUILT_append_${DISTRO_ARCH}
>>>>     - SYSLINUX_PREBUILT
>>>>     - ...
>>>>
>>>>    Those variables can then be used to initialize 
>>>> BOOTLOADER_PREBUILT as
>>>>    needed and also replace those nasty BUILDCHROOT_PREINSTALL_WIC in
>>>>    buildchroot-target.bb.
>>>>
>>>> - Set BOOTLOADER_PREBUILT or BOOTLOADER_CUSTOM in the
>>>>    conf/machine/${MACHINE}.conf or some overriding conf file - 
>>>> basically
>>>>    the same one that selects IMAGE_TYPE and WKS_FILE.
>>>>
>>>> Anything I missed? Better suggestions? Eventually, when we can 
>>>> derive bitbake recipe dependencies automatically from package 
>>>> dependencies, PREBUILT and CUSTOM could be folded together. Right 
>>>> now, we need that split.
>>>>
>>>> Jan
>>>
>>> I have a class that I use for the current project that allows 
>>> installing additional packages into the buildchroot from within the 
>>> image. I didn't had time to prepare patches for it for mainline isar. 
>>> But maybe that goes into to right direction.
>>>
>>> I attached it.
>>
>> ...
>>
>>> BUILDCHROOT_EXTRA_INSTALL ??= ""
>>>
>>> MOUNT_LOCKFILE = "${BUILDCHROOT_DIR}/mount.lock"
>>>
>>> BUILDCHROOT_DIR ??= "${BUILDCHROOT_TARGET_DIR}"
>>> BUILDCHROOT_BUILD_DEP ??= "buildchroot-target:do_build"
>>>
>>> do_buildchroot_extra_install[depends] = "${BUILDCHROOT_BUILD_DEP}"
>>> do_buildchroot_extra_install[deptask] = "do_deploy_deb"
>>> do_buildchroot_extra_install[stamp-extra-info] = "${DISTRO}-${MACHINE}"
>>> do_buildchroot_extra_install() {
>>>     PACKAGES="${@" ".join(d.getVar("BUILDCHROOT_EXTRA_INSTALL", 
>>> True).split())}"
>>>
>>
>> Can you explain the magic dance in the line above? I don't see its 
>> need yet.
>>
>>>     if [ -z "$PACKAGES" ]; then
>>
>> ...provided you do "[ -z $VAR ]" here to catch VAR=" ".
> 
> In sh i get:
> 
>    $ tets="asdf asdf  asdf asdf"
>    $ [ -z $tets ] && echo yes
>    sh: 2: [: asdf: unexpected operator

Ah, that's what I missed. But then we want to .strip() the variable for 
testing, rather than split/rejoin.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

Re: [RFC] Interface for installing bootloaders

[Claudius Heine]

On 2018-09-05 09:47, Jan Kiszka wrote:
> On 2018-09-05 09:34, Claudius Heine wrote:
>> Hi Jan,
>>
>> On 2018-09-05 08:08, Jan Kiszka wrote:
>>> On 2018-09-04 11:27, Claudius Heine wrote:
>>>> Hi Jan,
>>>>
>>>> On 2018-09-04 09:17, [ext] Jan Kiszka wrote:
>>>>> Hi all,
>>>>>
>>>>> for installing a bootloader into an image, a number of packages 
>>>>> have to be installed into the buildchroot. These can be prebuilt 
>>>>> packages or packages we build via Isar. Currently, we only handle 
>>>>> the case of WIC requiring prebuilt Debian packages, and that in a 
>>>>> hacky way (no proper distro abstraction, unneeded installation of 
>>>>> unused dependencies). This proposal aims at providing a more 
>>>>> holistic solution:
>>>>>
>>>>> - Introduce new variables that express the two types of dependencies:
>>>>>     - BOOTLOADER_PREBUILT
>>>>>     - BOOTLOADER_CUSTOM
>>>>>
>>>>>    So there is no "WIC" in these, any imaging recipe can use them.
>>>>>
>>>>> - Imaging recipes (e.g. via classes/wic-img.bbclass) should
>>>>>     - install all packages from BOOTLOADER_PREBUILT and 
>>>>> BOOTLOADER_CUSTOM
>>>>>       into the buildchroot
>>>>>     - DEPEND on all recipes in BOOTLOADER_CUSTOM
>>>>>
>>>>> - Define some common dependencies in conf/distro/${DISTRO}.conf:
>>>>>     - GRUB_PREBUILT
>>>>>     - GRUB_PREBUILT_append_${DISTRO_ARCH}
>>>>>     - SYSLINUX_PREBUILT
>>>>>     - ...
>>>>>
>>>>>    Those variables can then be used to initialize 
>>>>> BOOTLOADER_PREBUILT as
>>>>>    needed and also replace those nasty BUILDCHROOT_PREINSTALL_WIC in
>>>>>    buildchroot-target.bb.
>>>>>
>>>>> - Set BOOTLOADER_PREBUILT or BOOTLOADER_CUSTOM in the
>>>>>    conf/machine/${MACHINE}.conf or some overriding conf file - 
>>>>> basically
>>>>>    the same one that selects IMAGE_TYPE and WKS_FILE.
>>>>>
>>>>> Anything I missed? Better suggestions? Eventually, when we can 
>>>>> derive bitbake recipe dependencies automatically from package 
>>>>> dependencies, PREBUILT and CUSTOM could be folded together. Right 
>>>>> now, we need that split.
>>>>>
>>>>> Jan
>>>>
>>>> I have a class that I use for the current project that allows 
>>>> installing additional packages into the buildchroot from within the 
>>>> image. I didn't had time to prepare patches for it for mainline 
>>>> isar. But maybe that goes into to right direction.
>>>>
>>>> I attached it.
>>>
>>> ...
>>>
>>>> BUILDCHROOT_EXTRA_INSTALL ??= ""
>>>>
>>>> MOUNT_LOCKFILE = "${BUILDCHROOT_DIR}/mount.lock"
>>>>
>>>> BUILDCHROOT_DIR ??= "${BUILDCHROOT_TARGET_DIR}"
>>>> BUILDCHROOT_BUILD_DEP ??= "buildchroot-target:do_build"
>>>>
>>>> do_buildchroot_extra_install[depends] = "${BUILDCHROOT_BUILD_DEP}"
>>>> do_buildchroot_extra_install[deptask] = "do_deploy_deb"
>>>> do_buildchroot_extra_install[stamp-extra-info] = "${DISTRO}-${MACHINE}"
>>>> do_buildchroot_extra_install() {
>>>>     PACKAGES="${@" ".join(d.getVar("BUILDCHROOT_EXTRA_INSTALL", 
>>>> True).split())}"
>>>>
>>>
>>> Can you explain the magic dance in the line above? I don't see its 
>>> need yet.
>>>
>>>>     if [ -z "$PACKAGES" ]; then
>>>
>>> ...provided you do "[ -z $VAR ]" here to catch VAR=" ".
>>
>> In sh i get:
>>
>>    $ tets="asdf asdf  asdf asdf"
>>    $ [ -z $tets ] && echo yes
>>    sh: 2: [: asdf: unexpected operator
> 
> Ah, that's what I missed. But then we want to .strip() the variable for 
> testing, rather than split/rejoin.

Might work, but I don't like having possible '\n' and '\t' in between 
package names. (Ok, '\n' might not be an issue, since I think bitbake 
already escapes that.)

For me that is the normal code snippet to convert bitbake 'arrays' into 
shell strings.

-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de

Re: [RFC] Interface for installing bootloaders

[Jan Kiszka]

On 2018-09-05 09:57, Claudius Heine wrote:
> On 2018-09-05 09:47, Jan Kiszka wrote:
>> On 2018-09-05 09:34, Claudius Heine wrote:
>>> Hi Jan,
>>>
>>> On 2018-09-05 08:08, Jan Kiszka wrote:
>>>> On 2018-09-04 11:27, Claudius Heine wrote:
>>>>> Hi Jan,
>>>>>
>>>>> On 2018-09-04 09:17, [ext] Jan Kiszka wrote:
>>>>>> Hi all,
>>>>>>
>>>>>> for installing a bootloader into an image, a number of packages 
>>>>>> have to be installed into the buildchroot. These can be prebuilt 
>>>>>> packages or packages we build via Isar. Currently, we only handle 
>>>>>> the case of WIC requiring prebuilt Debian packages, and that in a 
>>>>>> hacky way (no proper distro abstraction, unneeded installation of 
>>>>>> unused dependencies). This proposal aims at providing a more 
>>>>>> holistic solution:
>>>>>>
>>>>>> - Introduce new variables that express the two types of dependencies:
>>>>>>     - BOOTLOADER_PREBUILT
>>>>>>     - BOOTLOADER_CUSTOM
>>>>>>
>>>>>>    So there is no "WIC" in these, any imaging recipe can use them.
>>>>>>
>>>>>> - Imaging recipes (e.g. via classes/wic-img.bbclass) should
>>>>>>     - install all packages from BOOTLOADER_PREBUILT and 
>>>>>> BOOTLOADER_CUSTOM
>>>>>>       into the buildchroot
>>>>>>     - DEPEND on all recipes in BOOTLOADER_CUSTOM
>>>>>>
>>>>>> - Define some common dependencies in conf/distro/${DISTRO}.conf:
>>>>>>     - GRUB_PREBUILT
>>>>>>     - GRUB_PREBUILT_append_${DISTRO_ARCH}
>>>>>>     - SYSLINUX_PREBUILT
>>>>>>     - ...
>>>>>>
>>>>>>    Those variables can then be used to initialize 
>>>>>> BOOTLOADER_PREBUILT as
>>>>>>    needed and also replace those nasty BUILDCHROOT_PREINSTALL_WIC in
>>>>>>    buildchroot-target.bb.
>>>>>>
>>>>>> - Set BOOTLOADER_PREBUILT or BOOTLOADER_CUSTOM in the
>>>>>>    conf/machine/${MACHINE}.conf or some overriding conf file - 
>>>>>> basically
>>>>>>    the same one that selects IMAGE_TYPE and WKS_FILE.
>>>>>>
>>>>>> Anything I missed? Better suggestions? Eventually, when we can 
>>>>>> derive bitbake recipe dependencies automatically from package 
>>>>>> dependencies, PREBUILT and CUSTOM could be folded together. Right 
>>>>>> now, we need that split.
>>>>>>
>>>>>> Jan
>>>>>
>>>>> I have a class that I use for the current project that allows 
>>>>> installing additional packages into the buildchroot from within the 
>>>>> image. I didn't had time to prepare patches for it for mainline 
>>>>> isar. But maybe that goes into to right direction.
>>>>>
>>>>> I attached it.
>>>>
>>>> ...
>>>>
>>>>> BUILDCHROOT_EXTRA_INSTALL ??= ""
>>>>>
>>>>> MOUNT_LOCKFILE = "${BUILDCHROOT_DIR}/mount.lock"
>>>>>
>>>>> BUILDCHROOT_DIR ??= "${BUILDCHROOT_TARGET_DIR}"
>>>>> BUILDCHROOT_BUILD_DEP ??= "buildchroot-target:do_build"
>>>>>
>>>>> do_buildchroot_extra_install[depends] = "${BUILDCHROOT_BUILD_DEP}"
>>>>> do_buildchroot_extra_install[deptask] = "do_deploy_deb"
>>>>> do_buildchroot_extra_install[stamp-extra-info] = 
>>>>> "${DISTRO}-${MACHINE}"
>>>>> do_buildchroot_extra_install() {
>>>>>     PACKAGES="${@" ".join(d.getVar("BUILDCHROOT_EXTRA_INSTALL", 
>>>>> True).split())}"
>>>>>
>>>>
>>>> Can you explain the magic dance in the line above? I don't see its 
>>>> need yet.
>>>>
>>>>>     if [ -z "$PACKAGES" ]; then
>>>>
>>>> ...provided you do "[ -z $VAR ]" here to catch VAR=" ".
>>>
>>> In sh i get:
>>>
>>>    $ tets="asdf asdf  asdf asdf"
>>>    $ [ -z $tets ] && echo yes
>>>    sh: 2: [: asdf: unexpected operator
>>
>> Ah, that's what I missed. But then we want to .strip() the variable 
>> for testing, rather than split/rejoin.
> 
> Might work, but I don't like having possible '\n' and '\t' in between 
> package names. (Ok, '\n' might not be an issue, since I think bitbake 
> already escapes that.)
> 
> For me that is the normal code snippet to convert bitbake 'arrays' into 
> shell strings.

I find this pattern in pure form in exactly one poky recipe. Therefore 
I'm not sure we need it in order to make a list built under bitbake in 
various ways available as parameter string for a shell command. Let's see...

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

Re: [RFC] Interface for installing bootloaders

[Claudius Heine]

On 2018-09-05 10:05, Jan Kiszka wrote:
> On 2018-09-05 09:57, Claudius Heine wrote:
>> On 2018-09-05 09:47, Jan Kiszka wrote:
>>> On 2018-09-05 09:34, Claudius Heine wrote:
>>>> Hi Jan,
>>>>
>>>> On 2018-09-05 08:08, Jan Kiszka wrote:
>>>>> On 2018-09-04 11:27, Claudius Heine wrote:
>>>>>> Hi Jan,
>>>>>>
>>>>>> On 2018-09-04 09:17, [ext] Jan Kiszka wrote:
>>>>>>> Hi all,
>>>>>>>
>>>>>>> for installing a bootloader into an image, a number of packages 
>>>>>>> have to be installed into the buildchroot. These can be prebuilt 
>>>>>>> packages or packages we build via Isar. Currently, we only handle 
>>>>>>> the case of WIC requiring prebuilt Debian packages, and that in a 
>>>>>>> hacky way (no proper distro abstraction, unneeded installation of 
>>>>>>> unused dependencies). This proposal aims at providing a more 
>>>>>>> holistic solution:
>>>>>>>
>>>>>>> - Introduce new variables that express the two types of 
>>>>>>> dependencies:
>>>>>>>     - BOOTLOADER_PREBUILT
>>>>>>>     - BOOTLOADER_CUSTOM
>>>>>>>
>>>>>>>    So there is no "WIC" in these, any imaging recipe can use them.
>>>>>>>
>>>>>>> - Imaging recipes (e.g. via classes/wic-img.bbclass) should
>>>>>>>     - install all packages from BOOTLOADER_PREBUILT and 
>>>>>>> BOOTLOADER_CUSTOM
>>>>>>>       into the buildchroot
>>>>>>>     - DEPEND on all recipes in BOOTLOADER_CUSTOM
>>>>>>>
>>>>>>> - Define some common dependencies in conf/distro/${DISTRO}.conf:
>>>>>>>     - GRUB_PREBUILT
>>>>>>>     - GRUB_PREBUILT_append_${DISTRO_ARCH}
>>>>>>>     - SYSLINUX_PREBUILT
>>>>>>>     - ...
>>>>>>>
>>>>>>>    Those variables can then be used to initialize 
>>>>>>> BOOTLOADER_PREBUILT as
>>>>>>>    needed and also replace those nasty BUILDCHROOT_PREINSTALL_WIC in
>>>>>>>    buildchroot-target.bb.
>>>>>>>
>>>>>>> - Set BOOTLOADER_PREBUILT or BOOTLOADER_CUSTOM in the
>>>>>>>    conf/machine/${MACHINE}.conf or some overriding conf file - 
>>>>>>> basically
>>>>>>>    the same one that selects IMAGE_TYPE and WKS_FILE.
>>>>>>>
>>>>>>> Anything I missed? Better suggestions? Eventually, when we can 
>>>>>>> derive bitbake recipe dependencies automatically from package 
>>>>>>> dependencies, PREBUILT and CUSTOM could be folded together. Right 
>>>>>>> now, we need that split.
>>>>>>>
>>>>>>> Jan
>>>>>>
>>>>>> I have a class that I use for the current project that allows 
>>>>>> installing additional packages into the buildchroot from within 
>>>>>> the image. I didn't had time to prepare patches for it for 
>>>>>> mainline isar. But maybe that goes into to right direction.
>>>>>>
>>>>>> I attached it.
>>>>>
>>>>> ...
>>>>>
>>>>>> BUILDCHROOT_EXTRA_INSTALL ??= ""
>>>>>>
>>>>>> MOUNT_LOCKFILE = "${BUILDCHROOT_DIR}/mount.lock"
>>>>>>
>>>>>> BUILDCHROOT_DIR ??= "${BUILDCHROOT_TARGET_DIR}"
>>>>>> BUILDCHROOT_BUILD_DEP ??= "buildchroot-target:do_build"
>>>>>>
>>>>>> do_buildchroot_extra_install[depends] = "${BUILDCHROOT_BUILD_DEP}"
>>>>>> do_buildchroot_extra_install[deptask] = "do_deploy_deb"
>>>>>> do_buildchroot_extra_install[stamp-extra-info] = 
>>>>>> "${DISTRO}-${MACHINE}"
>>>>>> do_buildchroot_extra_install() {
>>>>>>     PACKAGES="${@" ".join(d.getVar("BUILDCHROOT_EXTRA_INSTALL", 
>>>>>> True).split())}"
>>>>>>
>>>>>
>>>>> Can you explain the magic dance in the line above? I don't see its 
>>>>> need yet.
>>>>>
>>>>>>     if [ -z "$PACKAGES" ]; then
>>>>>
>>>>> ...provided you do "[ -z $VAR ]" here to catch VAR=" ".
>>>>
>>>> In sh i get:
>>>>
>>>>    $ tets="asdf asdf  asdf asdf"
>>>>    $ [ -z $tets ] && echo yes
>>>>    sh: 2: [: asdf: unexpected operator
>>>
>>> Ah, that's what I missed. But then we want to .strip() the variable 
>>> for testing, rather than split/rejoin.
>>
>> Might work, but I don't like having possible '\n' and '\t' in between 
>> package names. (Ok, '\n' might not be an issue, since I think bitbake 
>> already escapes that.)
>>
>> For me that is the normal code snippet to convert bitbake 'arrays' 
>> into shell strings.
> 
> I find this pattern in pure form in exactly one poky recipe. Therefore 
> I'm not sure we need it in order to make a list built under bitbake in 
> various ways available as parameter string for a shell command. Let's 
> see...

Might not be necessary, but I think I started using it, because it makes 
the environment and run.* files easier to read.

-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de