Discussion: Base-apt features

Discussion: Base-apt features

[Vijai Kumar K]

Hi All,

Starting this thread to discuss the base-apt features and limitations.

Here I am listing down some of the issues/features and possibly the
need for them.


1. Support for adding source packages.

Currently we have support only for binaries. The corresponding source
files could also be added. 


2. Support for using password protected keys.

It is a good practice to have the gpg key protected to have an additional
level of security. Right now ISAR does not have provisions to use password
protected keys.


3. Support for specifying the signing key.

Right now, the signing mechanism uses the default gpg key of the system.
This is problematic in many ways. Especially for CI. In the current
implementation, eventhough we specify the key, we are not really using it.


4. Support for adding packages only to base-apt.

Sometimes, we might need a package to be present in base-apt but not in
the target yet. Things like dev & dbg packages. It would be good if we
have something like BASE_APT_INSTALL which contains the list which would
be populated only in base-apt.


5. Refactoring code to consolidate reprepro calls.

Right now, reprepro calls are spread across the build system. Its dependencies
are spread across too(Handling envs like GNUPGHOME, distributions file etc).
My first thought is to have a seperate module implemented to handle these
calls.

Like how https://github.com/openembedded/openembedded-core/blob/master/meta/lib/oe/gpg_sign.py
is used for all signing purpose.

Please add more if you have some features/limitations which needs to be addressed.

Thanks,
Vijai Kumar K

Re: Discussion: Base-apt features

[Baurzhan Ismagulov]

Hello Vijai Kumar,

thanks for summarizing.

On Wed, Sep 25, 2019 at 01:11:22PM +0530, Vijai Kumar K wrote:
> 1. Support for adding source packages.
> 2. Support for using password protected keys.

Yes, I think those are necessary use cases.


> 3. Support for specifying the signing key.
> 
> Right now, the signing mechanism uses the default gpg key of the system.
> This is problematic in many ways. Especially for CI. In the current
> implementation, eventhough we specify the key, we are not really using it.

This is what I'm currently wondering. What do we need to cover signed base-apt?
Is the following enough?

1. debootstrap succeeds.
2. apt-get update in the rootfs succeeds.

IIUC, (1) should be covered as of next 006a6ed "u-boot-custom: Add control for
u-boot-tools package build". (2) seems to be covered in ibr/next fb61019
"scripts: Enable gnupg in ci_build.sh" (subject to rebasing). This has to be
verified, though (feedback welcome). If proven correct, would we be "really
using it"?


> 4. Support for adding packages only to base-apt.
> 
> Sometimes, we might need a package to be present in base-apt but not in
> the target yet. Things like dev & dbg packages. It would be good if we
> have something like BASE_APT_INSTALL which contains the list which would
> be populated only in base-apt.

Sounds useful, also for stuff like strace, optional gcc libs, etc.
Additionally, we may want to download all binary packages of the source
packages we need (for the requested arches and distros only). That would cover
dev and dbg.


> 5. Refactoring code to consolidate reprepro calls.

I'd suggest to evaluate other tools and libs like python-apt and / or aptly.
After looking at Acquire::By-Hash use cases in more detail, I've seen that we
do need it. According to Jan, it isn't supported by reprepro. I think in the
long term, we'll have to use python-apt and touch bitbake to get everything
right.


With kind regards,
Baurzhan.

Re: Discussion: Base-apt features

[Vijai Kumar K]

On Wed, Sep 25, 2019 at 10:12:41AM +0200, Baurzhan Ismagulov wrote:
> Hello Vijai Kumar,
> 
> thanks for summarizing.
> 
> On Wed, Sep 25, 2019 at 01:11:22PM +0530, Vijai Kumar K wrote:
> > 1. Support for adding source packages.
> > 2. Support for using password protected keys.
> 
> Yes, I think those are necessary use cases.
> 
> 
> > 3. Support for specifying the signing key.
> > 
> > Right now, the signing mechanism uses the default gpg key of the system.
> > This is problematic in many ways. Especially for CI. In the current
> > implementation, eventhough we specify the key, we are not really using it.
> 
> This is what I'm currently wondering. What do we need to cover signed base-apt?
> Is the following enough?
> 
> 1. debootstrap succeeds.
> 2. apt-get update in the rootfs succeeds.
> 
> IIUC, (1) should be covered as of next 006a6ed "u-boot-custom: Add control for
> u-boot-tools package build". (2) seems to be covered in ibr/next fb61019
> "scripts: Enable gnupg in ci_build.sh" (subject to rebasing). This has to be
> verified, though (feedback welcome). If proven correct, would we be "really
> using it"?
> 

The problem lies here.
https://github.com/ilbers/isar/blob/006a6ed75286710957b309aa571613d1935e4663/meta/recipes-devtools/base-apt/base-apt.bb#L21
By using "SignWith: yes", we are asking reprepro to use the default key of the system.
BTW, there is a possibility that the key we specified using BASE_REPO_KEY might not be
the default key of the system. 

This was the case in my local jenkins build. I have multiple gpg keys, and have configured 
one of them to be the default key. If I exercise repo signing in my system, then the repo 
would always be signed with my default key and not with the key I gave in BASE_REPO_KEY.

Below is the link to the mailing list discussion.
https://groups.google.com/d/msg/isar-users/CAVHJCuFxRo/WzbRjKdYBwAJ

> 
> > 4. Support for adding packages only to base-apt.
> > 
> > Sometimes, we might need a package to be present in base-apt but not in
> > the target yet. Things like dev & dbg packages. It would be good if we
> > have something like BASE_APT_INSTALL which contains the list which would
> > be populated only in base-apt.
> 
> Sounds useful, also for stuff like strace, optional gcc libs, etc.
> Additionally, we may want to download all binary packages of the source
> packages we need (for the requested arches and distros only). That would cover
> dev and dbg.
> 
> 
> > 5. Refactoring code to consolidate reprepro calls.
> 
> I'd suggest to evaluate other tools and libs like python-apt and / or aptly.
> After looking at Acquire::By-Hash use cases in more detail, I've seen that we
> do need it. According to Jan, it isn't supported by reprepro. I think in the
> long term, we'll have to use python-apt and touch bitbake to get everything
> right.
> 

Thanks for the info. I have been wondering regarding the need to move out of
reprepo. This gives some fair amount of detail. Let me also have a look at
the other tools.

> 
> With kind regards,
> Baurzhan.
> 
> -- 
> You received this message because you are subscribed to the Google Groups "isar-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/20190925081241.bmcblmlsax6xb4i4%40yssyq.m.ilbers.de.

Re: Discussion: Base-apt features

[Baurzhan Ismagulov]

On Wed, Sep 25, 2019 at 02:32:32PM +0530, Vijai Kumar K wrote:
> > > 3. Support for specifying the signing key.
...
> By using "SignWith: yes", we are asking reprepro to use the default key of the system.
> BTW, there is a possibility that the key we specified using BASE_REPO_KEY might not be
> the default key of the system. 

Ah, that's the signing part, thanks for the explanation.


> > > 5. Refactoring code to consolidate reprepro calls.
> > 
> > I'd suggest to evaluate other tools and libs like python-apt and / or aptly.
> > After looking at Acquire::By-Hash use cases in more detail, I've seen that we
> > do need it. According to Jan, it isn't supported by reprepro. I think in the
> > long term, we'll have to use python-apt and touch bitbake to get everything
> > right.
> 
> Thanks for the info. I have been wondering regarding the need to move out of
> reprepo. This gives some fair amount of detail. Let me also have a look at
> the other tools.

In the long term, there is more to that. Currently, we rely on bitbake pipeline
when populating base-apt. To cover all use cases correctly (e.g., updating
base-apt if upstream changes, removing packages not used anymore, etc.), we'd
have to do more stuff statically upfront (including debootstrap). To achieve
that, we might have to extract the functionality to a separate tool using
python-apt, touch bitbake, or both. This functionality may also be necessary
e.g. for determining Build-Depends: of debianized external sources (such as
hello-isar) and reflecting that in the pipeline (currently not possible, since
bitbake determines that before fetching and Debian -- after).

With kind regards,
Baurzhan.

Re: Discussion: Base-apt features

[Claudius Heine]

[-- Attachment #1.1: Type: text/plain, Size: 2722 bytes --]

Hi Vijai,

On 25/09/2019 09.41, Vijai Kumar K wrote:
> Hi All,
> 
> Starting this thread to discuss the base-apt features and limitations.
> 
> Here I am listing down some of the issues/features and possibly the
> need for them.
> 
> 
> 1. Support for adding source packages.
> 
> Currently we have support only for binaries. The corresponding source
> files could also be added. 

Yes. That would be required (AFAIK, but INAL) if the base-apt should be
used to distribute packages to the end-user as per GPL.

> 
> 
> 2. Support for using password protected keys.
> 
> It is a good practice to have the gpg key protected to have an additional
> level of security. Right now ISAR does not have provisions to use password
> protected keys.

There are two use-cases for base-apt AFAIK: Use it to distribute a
repository to the end user and for reproducible/offline build.

I agree that for the former having a password protected key for signing
the repo would be good. For the latter use-case though that might get
cumbersome.

Maybe we should split this use-case and have something different for
reproducible/offline build?

Also 'base-apt' is a bad name...

> 
> 
> 3. Support for specifying the signing key.
> 
> Right now, the signing mechanism uses the default gpg key of the system.
> This is problematic in many ways. Especially for CI. In the current
> implementation, eventhough we specify the key, we are not really using it.

Right, that needs to be fixed.

> 
> 
> 4. Support for adding packages only to base-apt.
> 
> Sometimes, we might need a package to be present in base-apt but not in
> the target yet. Things like dev & dbg packages. It would be good if we
> have something like BASE_APT_INSTALL which contains the list which would
> be populated only in base-apt.

If the base-apt should be used for distributing packages to the
end-user, it might also be useful to exclude certain packages.

> 
> 
> 5. Refactoring code to consolidate reprepro calls.
> 
> Right now, reprepro calls are spread across the build system. Its dependencies
> are spread across too(Handling envs like GNUPGHOME, distributions file etc).
> My first thought is to have a seperate module implemented to handle these
> calls.

Yes, that would be a good refactoring idea.

regards,
Claudius

-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de

           PGP key: 6FF2 E59F 00C6 BC28 31D8 64C1 1173 CB19 9808 B153
                             Keyserver: hkp://pool.sks-keyservers.net


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Discussion: Base-apt features

[Baurzhan Ismagulov]

Hello Claudius,

On Wed, Sep 25, 2019 at 12:14:19PM +0200, Claudius Heine wrote:
> > 2. Support for using password protected keys.
> 
> There are two use-cases for base-apt AFAIK: Use it to distribute a
> repository to the end user and for reproducible/offline build.
> 
> I agree that for the former having a password protected key for signing
> the repo would be good. For the latter use-case though that might get
> cumbersome.
> 
> Maybe we should split this use-case and have something different for
> reproducible/offline build?

This is my understanding. I see all of the following as required:

1. Unsgined base-apt
2. Signed base-apt w/o passphrase
3. Signed base-apt w/passphrase


> Also 'base-apt' is a bad name...

Like with isar-apt, it was difficult to find a short and intuitive one. Do you
have a suggestion?


> > 4. Support for adding packages only to base-apt.
> 
> If the base-apt should be used for distributing packages to the
> end-user, it might also be useful to exclude certain packages.

Could you give an example? After that is a copy of Debian, what is your
concern?


With kind regards,
Baurzhan.

Re: Discussion: Base-apt features

[Claudius Heine]

[-- Attachment #1.1: Type: text/plain, Size: 2892 bytes --]

Hi Baurzahn,

On 25/09/2019 12.26, Baurzhan Ismagulov wrote:
> Hello Claudius,
> 
> On Wed, Sep 25, 2019 at 12:14:19PM +0200, Claudius Heine wrote:
>>> 2. Support for using password protected keys.
>>
>> There are two use-cases for base-apt AFAIK: Use it to distribute a
>> repository to the end user and for reproducible/offline build.
>>
>> I agree that for the former having a password protected key for signing
>> the repo would be good. For the latter use-case though that might get
>> cumbersome.
>>
>> Maybe we should split this use-case and have something different for
>> reproducible/offline build?
> 
> This is my understanding. I see all of the following as required:
> 
> 1. Unsgined base-apt
> 2. Signed base-apt w/o passphrase
> 3. Signed base-apt w/passphrase

Right. It would just be inconvenient to switch between those
configurations for the different use cases that should be satisfiable
simultaneously without change of configuration files or deleting build
results.

I am currently not using Isar, but is something like this possible?

  BASE_REPO_KEY = "" bitbake ...
  BASE_REPO_KEY = "prod.key" bitbake ...

Meaning it should first create the base-apt without key and on the
second command just sign the repo with the 'prod.key' (and possible ask
for password, hardware key or what not). This would allow for a 2-stage
process, first building and testing and second signing the repo to
signify that everything was tested and works.

> 
> 
>> Also 'base-apt' is a bad name...
> 
> Like with isar-apt, it was difficult to find a short and intuitive one. Do you
> have a suggestion?
> 
> 
>>> 4. Support for adding packages only to base-apt.
>>
>> If the base-apt should be used for distributing packages to the
>> end-user, it might also be useful to exclude certain packages.
> 
> Could you give an example? After that is a copy of Debian, what is your
> concern?

Two departments of the same company, one that develops debian bin and
possible src packages and deploys them to a internal repo for other
departments to use, and the other department that takes these packages
and installs them to an image for the customer. Those packages are not
build by isar then and would be placed in the base-apt repo together
with the src package for delivery to the end-customer, which might not
be what was desired.

We should probably just check the license and only deliver packages that
are required to be redistributed.

regards,
Claudius

-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de

           PGP key: 6FF2 E59F 00C6 BC28 31D8 64C1 1173 CB19 9808 B153
                             Keyserver: hkp://pool.sks-keyservers.net


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Discussion: Base-apt features

[Baurzhan Ismagulov]

On Wed, Sep 25, 2019 at 01:56:37PM +0200, Claudius Heine wrote:
> >>> 2. Support for using password protected keys.
> Right. It would just be inconvenient to switch between those
> configurations for the different use cases that should be satisfiable
> simultaneously without change of configuration files or deleting build
> results.
> 
> I am currently not using Isar, but is something like this possible?
> 
>   BASE_REPO_KEY = "" bitbake ...
>   BASE_REPO_KEY = "prod.key" bitbake ...

That would be indeed convenient. We could look whether bitbake supports that.


> >>> 4. Support for adding packages only to base-apt.
> 
> Two departments of the same company, one that develops debian bin and
> possible src packages and deploys them to a internal repo for other
> departments to use, and the other department that takes these packages
> and installs them to an image for the customer. Those packages are not
> build by isar then and would be placed in the base-apt repo together
> with the src package for delivery to the end-customer, which might not
> be what was desired.
> 
> We should probably just check the license and only deliver packages that
> are required to be redistributed.

Thanks for the scenario, we could consider covering this in some way.


With kind regards,
Baurzhan.

Re: Discussion: Base-apt features

[Jan Kiszka]

On 25.09.19 14:19, Baurzhan Ismagulov wrote:
> On Wed, Sep 25, 2019 at 01:56:37PM +0200, Claudius Heine wrote:
>>>>> 2. Support for using password protected keys.
>> Right. It would just be inconvenient to switch between those
>> configurations for the different use cases that should be satisfiable
>> simultaneously without change of configuration files or deleting build
>> results.
>>
>> I am currently not using Isar, but is something like this possible?
>>
>>    BASE_REPO_KEY = "" bitbake ...
>>    BASE_REPO_KEY = "prod.key" bitbake ...
> 
> That would be indeed convenient. We could look whether bitbake supports that.
> 

It's there some env-variable whitelisting for such scenarios?

Jan

> 
>>>>> 4. Support for adding packages only to base-apt.
>>
>> Two departments of the same company, one that develops debian bin and
>> possible src packages and deploys them to a internal repo for other
>> departments to use, and the other department that takes these packages
>> and installs them to an image for the customer. Those packages are not
>> build by isar then and would be placed in the base-apt repo together
>> with the src package for delivery to the end-customer, which might not
>> be what was desired.
>>
>> We should probably just check the license and only deliver packages that
>> are required to be redistributed.
> 
> Thanks for the scenario, we could consider covering this in some way.
> 
> 
> With kind regards,
> Baurzhan.
> 

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

Re: Discussion: Base-apt features

[Claudius Heine]

[-- Attachment #1.1: Type: text/plain, Size: 823 bytes --]

On 25/09/2019 12.26, Baurzhan Ismagulov wrote:
> On Wed, Sep 25, 2019 at 12:14:19PM +0200, Claudius Heine wrote:
>> Also 'base-apt' is a bad name...
> 
> Like with isar-apt, it was difficult to find a short and intuitive one. Do you
> have a suggestion?

When short vs. intuitive then I would rather go with intuitive and
descriptive.

So something like:

isar-download-repo for base-apt and
isar-build-repo for isar-apt

regards,
Claudius

-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de

           PGP key: 6FF2 E59F 00C6 BC28 31D8 64C1 1173 CB19 9808 B153
                             Keyserver: hkp://pool.sks-keyservers.net


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Discussion: Base-apt features

[Henning Schild]

Hi Vijai Kumar K,

that is a lot of points at once, we should probably go step by step
since they are all more complicated than just one bullet-point on a
TODO-list.

Let me start off with a pragmatic "solution" that is easy to implement
and likely to solve a lot or problems at once.

Instead of relying on Isar to get you a partial debian mirror, you you
just mirror upstream and freeze it for a product. You would end up with
a consistent mirror that will contain all you might ever need, all
signed with sources etc. A little big ... yes, but easy.
If you decide to trust in the availability of
https://snapshot.debian.org/ that might also be a path to address some
problems you might be seeing today.

But yes, we probably still want a partial mirroring mechanism inside
Isar.

Am Wed, 25 Sep 2019 13:11:22 +0530
schrieb Vijai Kumar K <vijaikumar.kanagarajan@gmail.com>:

> Hi All,
> 
> Starting this thread to discuss the base-apt features and limitations.
> 
> Here I am listing down some of the issues/features and possibly the
> need for them.
> 
> 
> 1. Support for adding source packages.
> 
> Currently we have support only for binaries. The corresponding source
> files could also be added. 

It goes even further. You probably want the sources for every binary
you install.
+ all the other binaries coming from those sources
+ all build deps, again sources and all binaries
+ recursion of the former

We would have to see how big that gets ;).

> 2. Support for using password protected keys.
> 
> It is a good practice to have the gpg key protected to have an
> additional level of security. Right now ISAR does not have provisions
> to use password protected keys.

Let us put this aside for now, is is technically just something small
on top of signing.

> 3. Support for specifying the signing key.
> 
> Right now, the signing mechanism uses the default gpg key of the
> system. This is problematic in many ways. Especially for CI. In the
> current implementation, eventhough we specify the key, we are not
> really using it.

Did you not have a patch for that "=yes" bug? Anyways signing is not
trivial and we need revisit that anyways.

There are package-level signatures and repo signatures. For the partial
debian-mirror we just need to take care of the repo-level.

All customizations are currently coming out of isar-apt, which is
totally unsigned.

> 4. Support for adding packages only to base-apt.
> 
> Sometimes, we might need a package to be present in base-apt but not
> in the target yet. Things like dev & dbg packages. It would be good
> if we have something like BASE_APT_INSTALL which contains the list
> which would be populated only in base-apt.

I would suggest to have another image to do that. You can do that
today already.

bulky-image_0.1.bb:
require my-image
BASE_APT_INSTALL = "strace gdb"
IMAGE_INSTALL += BASE_APT_INSTALL

Now you build that and ship my-image. I assume that my-image will be
able to apt-get from base-apt later.

If that pattern works we can probably add an example, support-code and
test cases to make that the "supported pattern".

> 5. Refactoring code to consolidate reprepro calls.
> 
> Right now, reprepro calls are spread across the build system. Its
> dependencies are spread across too(Handling envs like GNUPGHOME,
> distributions file etc). My first thought is to have a seperate
> module implemented to handle these calls.

I think that is not too related. The calls for isar-apt are spread and
will probably stay that way, but they should currently not deal with
anything around signing.

For base-apt i envision a post-process task (like implemented today) as
a first implementation. If we decide to build it up on the fly, those
reprepro calls, downloads, signing will shift forward in the buildchain
and will spread like we have it for isar-apt.

Next steps:
- try the idea in 4.
- implement 1 with revisiting [1]

[1]
https://groups.google.com/forum/#!searchin/isar-users/base-apt$20caching$20improvements|sort:date/isar-users/_dqKYWUtTa0/velGfVg0BgAJ

Henning

> Like how
> https://github.com/openembedded/openembedded-core/blob/master/meta/lib/oe/gpg_sign.py
> is used for all signing purpose.
> 
> Please add more if you have some features/limitations which needs to
> be addressed.
> 
> Thanks,
> Vijai Kumar K

Re: Discussion: Base-apt features

[Vijai Kumar K]

On Thu, Sep 26, 2019 at 11:40:26AM +0200, Henning Schild wrote:
> Hi Vijai Kumar K,
> 
> that is a lot of points at once, we should probably go step by step
> since they are all more complicated than just one bullet-point on a
> TODO-list.
> 
> Let me start off with a pragmatic "solution" that is easy to implement
> and likely to solve a lot or problems at once.
> 
> Instead of relying on Isar to get you a partial debian mirror, you you
> just mirror upstream and freeze it for a product. You would end up with
> a consistent mirror that will contain all you might ever need, all
> signed with sources etc. A little big ... yes, but easy.
> If you decide to trust in the availability of
> https://snapshot.debian.org/ that might also be a path to address some
> problems you might be seeing today.

We could do that but would be an overhead considering we have only close to 
200 packages which we need. Also there might be cases where in we patch
a package locally for a CVE(like openssl) and include it in repo. Maintaining
changes like those becomes complicated.

> 
> But yes, we probably still want a partial mirroring mechanism inside
> Isar.
> 
> Am Wed, 25 Sep 2019 13:11:22 +0530
> schrieb Vijai Kumar K <vijaikumar.kanagarajan@gmail.com>:
> 
> > Hi All,
> > 
> > Starting this thread to discuss the base-apt features and limitations.
> > 
> > Here I am listing down some of the issues/features and possibly the
> > need for them.
> > 
> > 
> > 1. Support for adding source packages.
> > 
> > Currently we have support only for binaries. The corresponding source
> > files could also be added. 
> 
> It goes even further. You probably want the sources for every binary
> you install.
> + all the other binaries coming from those sources
> + all build deps, again sources and all binaries
> + recursion of the former
> 
> We would have to see how big that gets ;).

Agreed. The repo size might get compounded. How about we keep it as an optional
feature so in case some one needs it, they could bring in all the sources and
their dependencies.

> 
> > 2. Support for using password protected keys.
> > 
> > It is a good practice to have the gpg key protected to have an
> > additional level of security. Right now ISAR does not have provisions
> > to use password protected keys.
> 
> Let us put this aside for now, is is technically just something small
> on top of signing.

Agreed. We have bigger things to do and this could wait till we piece in the
rest of the things.

> 
> > 3. Support for specifying the signing key.
> > 
> > Right now, the signing mechanism uses the default gpg key of the
> > system. This is problematic in many ways. Especially for CI. In the
> > current implementation, eventhough we specify the key, we are not
> > really using it.
> 
> Did you not have a patch for that "=yes" bug? Anyways signing is not
> trivial and we need revisit that anyways.

Yes. It was proposed by Amy Fong. However I was into the idea of using
KeyID instead of key file. Later decided instead of doing a incremental
change to fix it, will wait for to freeze the new design if any.
 
> 
> There are package-level signatures and repo signatures. For the partial
> debian-mirror we just need to take care of the repo-level.
> 

Yes. We could just have the metadata signed.

> All customizations are currently coming out of isar-apt, which is
> totally unsigned.
> 
> > 4. Support for adding packages only to base-apt.
> > 
> > Sometimes, we might need a package to be present in base-apt but not
> > in the target yet. Things like dev & dbg packages. It would be good
> > if we have something like BASE_APT_INSTALL which contains the list
> > which would be populated only in base-apt.
> 
> I would suggest to have another image to do that. You can do that
> today already.
> 
> bulky-image_0.1.bb:
> require my-image
> BASE_APT_INSTALL = "strace gdb"
> IMAGE_INSTALL += BASE_APT_INSTALL
> 
> Now you build that and ship my-image. I assume that my-image will be
> able to apt-get from base-apt later.

Thanks for the pointer. There is a catch though. Not all images can be
installed simultaneously to the image. Some may conflict.
For example I would not be able to install libssl1.0-dev and libssl-dev
together. But one might need them to co-exist in the repo like how they do
now in upstream. So what I thought was that BASE_APT_INSTALL shall be
processed by a task which calls apt with --download-only to download these
and include it in the local partial repo.

> 
> If that pattern works we can probably add an example, support-code and
> test cases to make that the "supported pattern".
> 
> > 5. Refactoring code to consolidate reprepro calls.
> > 
> > Right now, reprepro calls are spread across the build system. Its
> > dependencies are spread across too(Handling envs like GNUPGHOME,
> > distributions file etc). My first thought is to have a seperate
> > module implemented to handle these calls.
> 
> I think that is not too related. The calls for isar-apt are spread and
> will probably stay that way, but they should currently not deal with
> anything around signing.
> 
> For base-apt i envision a post-process task (like implemented today) as
> a first implementation. If we decide to build it up on the fly, those
> reprepro calls, downloads, signing will shift forward in the buildchain
> and will spread like we have it for isar-apt.

Even if isar-apt and base-apt have their own differences, the tools used
to generate them is same. There might be some differences in options.
Bringing those tools(right now reprepro) under one roof would make the code
easier to maintain. Otherwise things like GNUPGHOME patch[1] would show up
and lead to more duplication and difficult maintainence.

[1]
https://github.com/ilbers/isar/commit/b8a2c6020ccb1b85cc4d0f6b75bc3ac090c24dc7

Thanks,
Vijai Kumar K

> 
> Next steps:
> - try the idea in 4.
> - implement 1 with revisiting [1]
> 
> [1]
> https://groups.google.com/forum/#!searchin/isar-users/base-apt$20caching$20improvements|sort:date/isar-users/_dqKYWUtTa0/velGfVg0BgAJ
> 
> Henning
> 
> > Like how
> > https://github.com/openembedded/openembedded-core/blob/master/meta/lib/oe/gpg_sign.py
> > is used for all signing purpose.
> > 
> > Please add more if you have some features/limitations which needs to
> > be addressed.
> > 
> > Thanks,
> > Vijai Kumar K
> 

Re: Discussion: Base-apt features

[Henning Schild]

Am Thu, 26 Sep 2019 16:32:16 +0530
schrieb Vijai Kumar K <vijaikumar.kanagarajan@gmail.com>:

> On Thu, Sep 26, 2019 at 11:40:26AM +0200, Henning Schild wrote:
> > Hi Vijai Kumar K,
> > 
> > that is a lot of points at once, we should probably go step by step
> > since they are all more complicated than just one bullet-point on a
> > TODO-list.
> > 
> > Let me start off with a pragmatic "solution" that is easy to
> > implement and likely to solve a lot or problems at once.
> > 
> > Instead of relying on Isar to get you a partial debian mirror, you
> > you just mirror upstream and freeze it for a product. You would end
> > up with a consistent mirror that will contain all you might ever
> > need, all signed with sources etc. A little big ... yes, but easy.
> > If you decide to trust in the availability of
> > https://snapshot.debian.org/ that might also be a path to address
> > some problems you might be seeing today.  
> 
> We could do that but would be an overhead considering we have only
> close to 200 packages which we need. Also there might be cases where
> in we patch a package locally for a CVE(like openssl) and include it
> in repo. Maintaining changes like those becomes complicated.

Locally created packages should imho not be mixed with mirrored ones.
So i would expect a modified upstream package to end up in isar-apt,
not base-apt.

> > 
> > But yes, we probably still want a partial mirroring mechanism inside
> > Isar.
> > 
> > Am Wed, 25 Sep 2019 13:11:22 +0530
> > schrieb Vijai Kumar K <vijaikumar.kanagarajan@gmail.com>:
> >   
> > > Hi All,
> > > 
> > > Starting this thread to discuss the base-apt features and
> > > limitations.
> > > 
> > > Here I am listing down some of the issues/features and possibly
> > > the need for them.
> > > 
> > > 
> > > 1. Support for adding source packages.
> > > 
> > > Currently we have support only for binaries. The corresponding
> > > source files could also be added.   
> > 
> > It goes even further. You probably want the sources for every binary
> > you install.
> > + all the other binaries coming from those sources
> > + all build deps, again sources and all binaries
> > + recursion of the former
> > 
> > We would have to see how big that gets ;).  
> 
> Agreed. The repo size might get compounded. How about we keep it as
> an optional feature so in case some one needs it, they could bring in
> all the sources and their dependencies.

Hard to maintain and error-prone to allow toggling switches. I guess
being conservative and storing too much is better than not having
something years later. Storage and bandwidth are cheap enough.

> >   
> > > 2. Support for using password protected keys.
> > > 
> > > It is a good practice to have the gpg key protected to have an
> > > additional level of security. Right now ISAR does not have
> > > provisions to use password protected keys.  
> > 
> > Let us put this aside for now, is is technically just something
> > small on top of signing.  
> 
> Agreed. We have bigger things to do and this could wait till we piece
> in the rest of the things.
> 
> >   
> > > 3. Support for specifying the signing key.
> > > 
> > > Right now, the signing mechanism uses the default gpg key of the
> > > system. This is problematic in many ways. Especially for CI. In
> > > the current implementation, eventhough we specify the key, we are
> > > not really using it.  
> > 
> > Did you not have a patch for that "=yes" bug? Anyways signing is not
> > trivial and we need revisit that anyways.  
> 
> Yes. It was proposed by Amy Fong. However I was into the idea of using
> KeyID instead of key file. Later decided instead of doing a
> incremental change to fix it, will wait for to freeze the new design
> if any.

As far as i remember that was a stand-alone bug fix that made total
sense and did not change the way things generally work. So easy to
merge, even before possibly rebuilding things here.

> > 
> > There are package-level signatures and repo signatures. For the
> > partial debian-mirror we just need to take care of the repo-level.
> >   
> 
> Yes. We could just have the metadata signed.

That is what we do, the packages are signed by their maintainers.

> > All customizations are currently coming out of isar-apt, which is
> > totally unsigned.
> >   
> > > 4. Support for adding packages only to base-apt.
> > > 
> > > Sometimes, we might need a package to be present in base-apt but
> > > not in the target yet. Things like dev & dbg packages. It would
> > > be good if we have something like BASE_APT_INSTALL which contains
> > > the list which would be populated only in base-apt.  
> > 
> > I would suggest to have another image to do that. You can do that
> > today already.
> > 
> > bulky-image_0.1.bb:
> > require my-image
> > BASE_APT_INSTALL = "strace gdb"
> > IMAGE_INSTALL += BASE_APT_INSTALL
> > 
> > Now you build that and ship my-image. I assume that my-image will be
> > able to apt-get from base-apt later.  
> 
> Thanks for the pointer. There is a catch though. Not all images can be
> installed simultaneously to the image. Some may conflict.
> For example I would not be able to install libssl1.0-dev and
> libssl-dev together. But one might need them to co-exist in the repo
> like how they do now in upstream. So what I thought was that
> BASE_APT_INSTALL shall be processed by a task which calls apt with
> --download-only to download these and include it in the local partial
> repo.

Tell the vendor of that libssl1.0 package to finally get that sorted! I
guess i know that crap component too well ;).

Just have many bulky images to address that, in your case two should do.

But you are right, a set of packages going just into the download step
would work as well. I am just afraid of too many variations, of how
things get fetched and need to be mirrored/archived later.

Going back to 1 and to [1] you will need to answer questions like.
- what is the source to a .deb?
- what are sibbling debs for a .deb (same source)
- what are build deps (debs and sources)

And now you would have to answer this for installed and not installed.
Now the worst case is probably a custom debug tool in isar-apt, that
has deps and build-deps from upstream. i.e. the dbgsyms and headers of
your patched openssl

We should go step by step to understand what we need and how to
implement it when needed. I still think the additional image(s) sound
like a good pattern to get that done today.

> > 
> > If that pattern works we can probably add an example, support-code
> > and test cases to make that the "supported pattern".
> >   
> > > 5. Refactoring code to consolidate reprepro calls.
> > > 
> > > Right now, reprepro calls are spread across the build system. Its
> > > dependencies are spread across too(Handling envs like GNUPGHOME,
> > > distributions file etc). My first thought is to have a seperate
> > > module implemented to handle these calls.  
> > 
> > I think that is not too related. The calls for isar-apt are spread
> > and will probably stay that way, but they should currently not deal
> > with anything around signing.
> > 
> > For base-apt i envision a post-process task (like implemented
> > today) as a first implementation. If we decide to build it up on
> > the fly, those reprepro calls, downloads, signing will shift
> > forward in the buildchain and will spread like we have it for
> > isar-apt.  
> 
> Even if isar-apt and base-apt have their own differences, the tools
> used to generate them is same. There might be some differences in
> options. Bringing those tools(right now reprepro) under one roof
> would make the code easier to maintain. Otherwise things like
> GNUPGHOME patch[1] would show up and lead to more duplication and
> difficult maintainence.

Yes i know. The code is also full of calls to apt-get with very similar
args, that are never consolidated anywhere.

If you see room for improvement send patches. But that is just style
and has little to do with the functional points, which to me seem to be
the main subject of this thread. Style is important and Isar needs to
improve here a lot!

Henning

> [1]
> https://github.com/ilbers/isar/commit/b8a2c6020ccb1b85cc4d0f6b75bc3ac090c24dc7
> 
> Thanks,
> Vijai Kumar K
> 
> > 
> > Next steps:
> > - try the idea in 4.
> > - implement 1 with revisiting [1]
> > 
> > [1]
> > https://groups.google.com/forum/#!searchin/isar-users/base-apt$20caching$20improvements|sort:date/isar-users/_dqKYWUtTa0/velGfVg0BgAJ
> > 
> > Henning
> >   
> > > Like how
> > > https://github.com/openembedded/openembedded-core/blob/master/meta/lib/oe/gpg_sign.py
> > > is used for all signing purpose.
> > > 
> > > Please add more if you have some features/limitations which needs
> > > to be addressed.
> > > 
> > > Thanks,
> > > Vijai Kumar K  
> >