TODOs

TODOs

[Claudius Heine]

Hi,

here is a list of TODOs for isar that I gathered:

- WIC_IMAGER_DEPENDS should be IMAGER_DEPENDS instead
- IMAGE_INSTALL, IMAGE_PREINSTALL, DEPENDS refactoring so that it is 
possible to provide recipes that produce multiple packages. Currently 
all entries in IMAGE_INSTALL are added to DEPENDS, but DEPENDS should 
only contain recipe names and IMAGE_INSTALL only package names.
- base-apt refactoring/redesign:
   - multiple distros/codenames in one base-apt repo. base-apt repos 
should be build specific not specific to distros/codenames.
   - no manual switch via `ISAR_USE_CACHED_REPO`. The cache should be 
build and used at every build
- redesign fstab generation:
   - make it more flexible
   - usable with or without wic and init system independent
- isar-bootstrap refactoring:
   - remove duplicate code
   - split up big functions to allow
   - inc file should be class
   - separate host/target bootstrap settings
   - host/target/build nomenclature fix
- refactor RootFS size calculation, make it similar to the formula of 
open embedded:
     root_fs_partition_size = max(used_space * overhead_factor, 
fixed_rootfs_size) + extra_space
     where overhead_factor, fixed_rootfs_size and extra_space is user 
configurable
     overhead_factor is the IMAGE_OVERHEAD_FACTOR bb variable
     fixed_rootfs_size is IMAGE_ROOTFS_SIZE bb variable
     extra_space is IMAGE_ROOTFS_EXTRA_SPACE bb variable
     all of them in Kbytes
- integration solution for on-first-boot resize rootfs.
   -> investigate systemd-first-boot and systemd-growfs

I don't know if I have enough time to work on all of these, so if 
someone like to do certain tasks, be welcome to them.

regards,
Claudius

-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de

Re: TODOs

[Baurzhan Ismagulov]

Hello Claudius,

thanks for sharing.

On Mon, Jun 03, 2019 at 04:55:39PM +0200, Claudius Heine wrote:
> here is a list of TODOs for isar that I gathered:
> 
> - WIC_IMAGER_DEPENDS should be IMAGER_DEPENDS instead

Is it a new var, or do you by chance mean WIC_IMAGER_INSTALL and
IMAGER_INSTALL?


> - IMAGE_INSTALL, IMAGE_PREINSTALL, DEPENDS refactoring so that it is
> possible to provide recipes that produce multiple packages. Currently all
> entries in IMAGE_INSTALL are added to DEPENDS, but DEPENDS should only
> contain recipe names and IMAGE_INSTALL only package names.

This is tightly coupled with the subject of handling Debian packages with
bitbake, which we have been experimenting with in meta-eid. We had started with
the dependencies. One limitation is that bitbake parses all recipes, then
builds the dependency tree. Debian packages may be local or remote, so they
have to be fetched before the dependency tree is built.

For now, the workaround is to duplicate Build-Depends: in DEPENDS.

Do I understand your use case correctly, you have a.bb producing aa.deb and
ab.deb. You want to have IMAGE_INSTALL += "ab" instead of "a", do you? If yes,
can this be workarounded with PROVIDES = "aa ab"? As I see it, the final
solution without duplication would require solving Debian package handling in
some way.


> - base-apt refactoring/redesign:
>   - multiple distros/codenames in one base-apt repo. base-apt repos should
> be build specific not specific to distros/codenames.

I see that this is desirable for certain use cases. That said, it might also be
a matter of downstream policy, so that either choice is poorly suited for some
other downstreams. Ideally, I'd like to have it configurable.


>   - no manual switch via `ISAR_USE_CACHED_REPO`. The cache should be build
> and used at every build

This can be enabled downstream. This is also a matter of policy: Some projects
always work with the latest stuff, some make snapshots. The ability to turn it
off is useful as a reference if anything breaks.


> - redesign fstab generation:
>   - make it more flexible
>   - usable with or without wic and init system independent
> - isar-bootstrap refactoring:
>   - remove duplicate code
>   - split up big functions to allow
>   - inc file should be class
>   - separate host/target bootstrap settings
>   - host/target/build nomenclature fix

I agree that the inconsistencies exist. The main reason is that compiler people
have to name it differently from the 99% of application developers who
intuitively understand the right thing from "host" and "target". So I'm still
not sure whether the change is worth it.


> - refactor RootFS size calculation, make it similar to the formula of open
> embedded:
>     root_fs_partition_size = max(used_space * overhead_factor,
> fixed_rootfs_size) + extra_space
>     where overhead_factor, fixed_rootfs_size and extra_space is user
> configurable
>     overhead_factor is the IMAGE_OVERHEAD_FACTOR bb variable
>     fixed_rootfs_size is IMAGE_ROOTFS_SIZE bb variable
>     extra_space is IMAGE_ROOTFS_EXTRA_SPACE bb variable
>     all of them in Kbytes
> - integration solution for on-first-boot resize rootfs.
>   -> investigate systemd-first-boot and systemd-growfs


With kind regards,
Baurzhan.

Re: TODOs

[Claudius Heine]

Hi Baurzahn,

On 19/06/2019 17.57, Baurzhan Ismagulov wrote:
> Hello Claudius,
> 
> thanks for sharing.
> 
> On Mon, Jun 03, 2019 at 04:55:39PM +0200, Claudius Heine wrote:
>> here is a list of TODOs for isar that I gathered:
>>
>> - WIC_IMAGER_DEPENDS should be IMAGER_DEPENDS instead
> 
> Is it a new var, or do you by chance mean WIC_IMAGER_INSTALL and
> IMAGER_INSTALL?

Yes, you are right I meant *IMAGER_INSTALL.

> 
> 
>> - IMAGE_INSTALL, IMAGE_PREINSTALL, DEPENDS refactoring so that it is
>> possible to provide recipes that produce multiple packages. Currently all
>> entries in IMAGE_INSTALL are added to DEPENDS, but DEPENDS should only
>> contain recipe names and IMAGE_INSTALL only package names.
> 
> This is tightly coupled with the subject of handling Debian packages with
> bitbake, which we have been experimenting with in meta-eid. We had started with
> the dependencies. One limitation is that bitbake parses all recipes, then
> builds the dependency tree. Debian packages may be local or remote, so they
> have to be fetched before the dependency tree is built.

You would only need the Packages files from the repos in order to figure 
out the dependencies.

> 
> For now, the workaround is to duplicate Build-Depends: in DEPENDS.

That is not really possible since we don't have recipes for every debian 
packages. DEPENDS and Build-Depends are different.

> 
> Do I understand your use case correctly, you have a.bb producing aa.deb and
> ab.deb. You want to have IMAGE_INSTALL += "ab" instead of "a", do you? If yes,
> can this be workarounded with PROVIDES = "aa ab"?

Probably, but I don't know how things like 'virtual/kernel' could be 
handled then. That would possible required all debian packages of the 
kernel to also contain a 'Provides: virtual/kernel' or similar.

If we do that, I see the risk that because we bind recipe names and 
provides and debian package name and provides a bit to tightly together 
that things will not work out and end up with corner cases we cannot model.

> As I see it, the final
> solution without duplication would require solving Debian package handling in
> some way.

Do you mean by that the import from apt repositories into bitbake 
data-store and export from bitbake data-store to control files?

>> - base-apt refactoring/redesign:
>>    - multiple distros/codenames in one base-apt repo. base-apt repos should
>> be build specific not specific to distros/codenames.
> 
> I see that this is desirable for certain use cases. > That said, it might also be
> a matter of downstream policy, so that either choice is poorly suited for some
> other downstreams. Ideally, I'd like to have it configurable.

Well in order to allow both ways, the code needs to be changed anyway.

I did thought about it some more now and I do see reasons to have it 
distro/codename and build specific for different use-cases and it might 
even be ok to leave it as distro/codename specific per default. At least 
I currently don't remember the reasons I thought of when writing that 
todo list.

>>    - no manual switch via `ISAR_USE_CACHED_REPO`. The cache should be build
>> and used at every build
> 
> This can be enabled downstream. This is also a matter of policy: Some projects
> always work with the latest stuff, some make snapshots. The ability to turn it
> off is useful as a reference if anything breaks.

Well IMO the cache should be used per default. If someone wants to 
update a image after the initial creation, then this should be done 
explicitly with full knowledge.

> 
> 
>> - redesign fstab generation:
>>    - make it more flexible
>>    - usable with or without wic and init system independent
>> - isar-bootstrap refactoring:
>>    - remove duplicate code
>>    - split up big functions to allow
>>    - inc file should be class
>>    - separate host/target bootstrap settings
>>    - host/target/build nomenclature fix
> 
> I agree that the inconsistencies exist. The main reason is that compiler people
> have to name it differently from the 99% of application developers who
> intuitively understand the right thing from "host" and "target". So I'm still
> not sure whether the change is worth it.

As I mentioned in previous discussions we could avoid this issue 
altogether by calling what is currently called 'host' -> 'native' and 
remove the word 'target' where possible. 'DISTRO' and 'DISTRO_ARCH' will 
still be called the same and 'HOST_DISTRO' and 'HOST_ARCH' will be 
'NATIVE_DISTRO' and 'NATIVE_ARCH' for instance. Or 'buildchroot' and 
'buildchroot-native'.

regards,
Claudius

-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de